Re: [jupyter] Executing script statement-by-statement

[Matthias Bussonnier]

What I would suggest if for you to come up with an "API" by modifying
Jupyter/IPython adding endpoints or if possible as an extension.

I would go for an api that get a document "content" and return a list
of positions that are block of execution code, then see if we can
iterate on it.
That would allow the frontend to show visual indication of what these
"statements" are.

I would also look at how Hydrogen (the Jupyter plugin for atom) is
doing it, as AFAICT it auto-detects blocks.

-- 
M

On Thu, Oct 13, 2016 at 11:39 AM, Tristan Zajonc  wrote:
> Min,
>
> What you suggest is exactly what we do currently.  The downside is that we
> need to implement parsers for each language and put another proxy in front
> of the kernels or kernel gateway and ensure that our parsers are using the
> same underlying runtime as the kernel if they parse a proper AST. So the
> advantage of having a feature like this would be that kernels could
> implement this behavior directly, against the same runtime, and everybody
> could share this implementation easily across both commands like jupyter run
> and kernel gateway.
>
> They very real downside, which you allude to, is there are many approaches
> to chunking a file.  For instance, one might want to use something like Rmd
> or allow some output control in comment pragmas.  This could be accommodated
> by having a style parameter.
>
> Tristan
>
> On Thu, Oct 13, 2016 at 5:52 AM, MinRK  wrote:
>>
>> I don't believe any protocol changes are needed for this. You can already
>> run a script through Jupyter, breaking it up however you want. The trick
>> would be writing a runner for a given script (you would need to write a
>> different one for every language to do the parsing correctly), and then
>> submit the chunks as you see fit. Alternatively, you could rely on simple
>> markers in comments to identify the chunks to split on.
>>
>> There is [a PR](https://github.com/jupyter/jupyter_client/pull/184) adding
>> a `jupyter run` command that executes a script as a whole file. It doesn't
>> record the output into a notebook, but a similar script that does would not
>> be a large project.
>>
>> What do you see as a needed change to the protocol that isn't served by
>> what we already have?
>>
>> -Min
>>
>> On Thu, Oct 13, 2016 at 9:57 AM, Tristan Zajonc 
>> wrote:
>>>
>>> Has anybody considered extending the jupyter messaging protocol to allow
>>> kernels to optionally implement statement-by-statement execution?  This
>>> would allow users to create notebooks from raw scripts, akin to RStudio.
>>> The behavior of the parser would be up to the kernel, but typically it would
>>> parse a code string into statements that could be executed individually.
>>>
>>> I'd be interested in potentially adding this feature -- I've done so
>>> out-of-band for R and Python -- but am curious if there's a preferred path.
>>> I could imagine either extending execute_request to take a "split"
>>> parameter, although that may get confusing since there would then be
>>> multiple execute_results.  The alternative would be to implement a new
>>> message type, execute_statements, then would parse the code string and issue
>>> multiple execute_requests.  This should be backward compatible since the
>>> output messages would be unchanged.
>>>
>>> This could be combined with something like: jupyter notebook run
>>> analysis.py --output=results.ipynb
>>>
>>> Is this worth pursuing as part of an enhancement proposal?
>>>
>>> Tristan
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Project Jupyter" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to jupyter+unsubscr...@googlegroups.com.
>>> To post to this group, send email to jupyter@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jupyter/a3340660-4eb2-4079-b7ee-df7dc4962a35%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Project Jupyter" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/jupyter/SM68P70XzW4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> jupyter+unsubscr...@googlegroups.com.
>> To post to this group, send email to jupyter@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CAHNn8BUbvc2-EyX1VvSLxdcrDH6OwqounZt_Swdbbhz%3De-P8Rg%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+unsubscr...@googlegroups.com.
> To post to this group, send email to jupyter@googlegroups.com.
> To view this discussion on the web vis

Re: [jupyter] Executing script statement-by-statement

[Tristan Zajonc]

Min,

What you suggest is exactly what we do currently.  The downside is that we
need to implement parsers for each language and put another proxy in front
of the kernels or kernel gateway and ensure that our parsers are using the
same underlying runtime as the kernel if they parse a proper AST. So the
advantage of having a feature like this would be that kernels could
implement this behavior directly, against the same runtime, and everybody
could share this implementation easily across both commands like jupyter
run and kernel gateway.

They very real downside, which you allude to, is there are many approaches
to chunking a file.  For instance, one might want to use something like Rmd
or allow some output control in comment pragmas.  This could be
accommodated by having a style parameter.

Tristan

On Thu, Oct 13, 2016 at 5:52 AM, MinRK  wrote:

> I don't believe any protocol changes are needed for this. You can already
> run a script through Jupyter, breaking it up however you want. The trick
> would be writing a runner for a given script (you would need to write a
> different one for every language to do the parsing correctly), and then
> submit the chunks as you see fit. Alternatively, you could rely on simple
> markers in comments to identify the chunks to split on.
>
> There is [a PR](https://github.com/jupyter/jupyter_client/pull/184)
> adding a `jupyter run` command that executes a script as a whole file. It
> doesn't record the output into a notebook, but a similar script that does
> would not be a large project.
>
> What do you see as a needed change to the protocol that isn't served by
> what we already have?
>
> -Min
>
> On Thu, Oct 13, 2016 at 9:57 AM, Tristan Zajonc 
> wrote:
>
>> Has anybody considered extending the jupyter messaging protocol to allow
>> kernels to optionally implement statement-by-statement execution?  This
>> would allow users to create notebooks from raw scripts, akin to RStudio.
>> The behavior of the parser would be up to the kernel, but typically it
>> would parse a code string into statements that could be executed
>> individually.
>>
>> I'd be interested in potentially adding this feature -- I've done so
>> out-of-band for R and Python -- but am curious if there's a preferred
>> path.  I could imagine either extending execute_request to take a "split"
>> parameter, although that may get confusing since there would then be
>> multiple execute_results.  The alternative would be to implement a new
>> message type, execute_statements, then would parse the code string and
>> issue multiple execute_requests.  This should be backward compatible since
>> the output messages would be unchanged.
>>
>> This could be combined with something like: jupyter notebook run
>> analysis.py --output=results.ipynb
>>
>> Is this worth pursuing as part of an enhancement proposal?
>>
>> Tristan
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to jupyter+unsubscr...@googlegroups.com.
>> To post to this group, send email to jupyter@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/jupyter/a3340660-4eb2-4079-b7ee-df7dc4962a35%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Project Jupyter" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/jupyter/SM68P70XzW4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jupyter+unsubscr...@googlegroups.com.
> To post to this group, send email to jupyter@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jupyter/CAHNn8BUbvc2-EyX1VvSLxdcrDH6OwqounZt_
> Swdbbhz%3De-P8Rg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jupyter+unsubscr...@googlegroups.com.
To post to this group, send email to jupyter@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jupyter/CAPVVPNyW-YUPJMP7G5HYLMEsiULDyEtUXdH-CB2nwsMnMqAGJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [jupyter] Executing script statement-by-statement

[MinRK]

I don't believe any protocol changes are needed for this. You can already
run a script through Jupyter, breaking it up however you want. The trick
would be writing a runner for a given script (you would need to write a
different one for every language to do the parsing correctly), and then
submit the chunks as you see fit. Alternatively, you could rely on simple
markers in comments to identify the chunks to split on.

There is [a PR](https://github.com/jupyter/jupyter_client/pull/184) adding
a `jupyter run` command that executes a script as a whole file. It doesn't
record the output into a notebook, but a similar script that does would not
be a large project.

What do you see as a needed change to the protocol that isn't served by
what we already have?

-Min

On Thu, Oct 13, 2016 at 9:57 AM, Tristan Zajonc  wrote:

> Has anybody considered extending the jupyter messaging protocol to allow
> kernels to optionally implement statement-by-statement execution?  This
> would allow users to create notebooks from raw scripts, akin to RStudio.
> The behavior of the parser would be up to the kernel, but typically it
> would parse a code string into statements that could be executed
> individually.
>
> I'd be interested in potentially adding this feature -- I've done so
> out-of-band for R and Python -- but am curious if there's a preferred
> path.  I could imagine either extending execute_request to take a "split"
> parameter, although that may get confusing since there would then be
> multiple execute_results.  The alternative would be to implement a new
> message type, execute_statements, then would parse the code string and
> issue multiple execute_requests.  This should be backward compatible since
> the output messages would be unchanged.
>
> This could be combined with something like: jupyter notebook run
> analysis.py --output=results.ipynb
>
> Is this worth pursuing as part of an enhancement proposal?
>
> Tristan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+unsubscr...@googlegroups.com.
> To post to this group, send email to jupyter@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jupyter/a3340660-4eb2-4079-b7ee-df7dc4962a35%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jupyter+unsubscr...@googlegroups.com.
To post to this group, send email to jupyter@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jupyter/CAHNn8BUbvc2-EyX1VvSLxdcrDH6OwqounZt_Swdbbhz%3De-P8Rg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.