Re: [O] Proposal and RFC for improving ob-python

[Greg Minshall]

hi, John,

> I like the idea behind ob-ipython, hopefully where it might go as
> ob-jupyter, which could make it able to run many other languages via
> the jupyter project and it's kernels.

thanks for the reply.  i'm not really up to speed on Jupyter, et al.
but, from a brief look, my work is probably more directed at a more
plain, vanilla python.  were an [ij]python mode available, i can imagine
being captivated it.  but, if i were to try and hack on any code right
now, it would probably be normal python mode.

cheers, Greg

Re: [O] Proposal and RFC for improving ob-python

[John Kitchin]

now I can not remember if I sent this earlier.

I like the idea behind ob-ipython, hopefully where it might go as
ob-jupyter, which could make it able to run many other languages via the
jupyter project and it's kernels.

The last time I tried it, it only supported sessions, and I had to add
non-session support. I doubt it supported async running either.

Before any major effort went in to a new ob-python, it is probably worth
a serious look at what the jupyter project could offer. 

Greg Minshall writes:

> hi.  i recently got fed up with ob-python.el behavior w.r.t. :session
> handling (startup issues; blank lines [really, a python problem, imho,
> but a workaround maybe could work], and value scraping), looked, and saw
> this discussion from last December, but nothing since then.  has there
> been any progress, either by Ondřej or anyone else?  if my frustration
> level remains high enough, i might try to see if i can do anything.
> cheers, Greg Minshall


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: [O] Proposal and RFC for improving ob-python

[Greg Minshall]

hi.  i recently got fed up with ob-python.el behavior w.r.t. :session
handling (startup issues; blank lines [really, a python problem, imho,
but a workaround maybe could work], and value scraping), looked, and saw
this discussion from last December, but nothing since then.  has there
been any progress, either by Ondřej or anyone else?  if my frustration
level remains high enough, i might try to see if i can do anything.
cheers, Greg Minshall

Re: [O] Proposal and RFC for improving ob-python

[Kyle Meyer]

Ondřej Grover  writes:

[...]

>> But what about when :results != value?  Doesn't your proposal only
>> handle returning the last value?
>>
> You mean :results output ? In that case it could just omit the
> "open(...).write(...) " part and capture anything the console prints before
> the primary prompt appears again.
> Or it could capture stdout into a file, but I think that's needlessly
> complicated:
>
> python -i << HEREDOC_END
> import sys
> sys.stdout = open()   # replace stdout with some file
> _ = block_eval("""
> 
> """)
> sys.stdout.close()
> sys.stdout = sys.__stdout__  # restore stdout
> HEREDOC_END

(I think you should rely on Org babel's shared utilities,
e.g. org-babel-eval and org-babel-eval-read-file, unless there is a
specific reason they won't do.  At any rate, right now I'm just trying
to get a picture of how block_eval would fit in.)

So these are the main options to consider for Python:

| | output | value |
|-++---|
| non-session | a  | b |
| session | c  | d |

a) Using block_eval here seems unhelpful here because the method in
   place is already simple:

   (org-babel-eval org-babel-python-command body)

b) block_eval replaces the need for dumping the code into a function.
   Using block_eval in org-babel-python-wrapper-method would allow you
   to get rid of the odd "return" use.

c) Currently, this is handled by inserting the body lines one at a time
   and then getting the output by splitting on the prompt.  Aside from
   race condition issues, this has limitations with handling blank lines
   in the body.

   block_eval could help here because the whole body could be sent
   through as one line of input to the prompt.  This should resolve most
   issues with prompt splitting, as well as lift the body formatting
   restrictions for sessions.

   However, the main issue I see with this is that I think it would lose
   the main distinction between session and non-session for ":results
   output" that is discussed in (info "(org) Results of evaluation"):

   #+BEGIN_SRC python :results output
print "hello"
2
print "bye"
   #+END_SRC

   #+RESULTS:
   : hello
   : bye

   versus this

   #+BEGIN_SRC python :results output :session
print "hello"
2
print "bye"
   #+END_SRC

   #+RESULTS:
   : hello
   : 2
   : bye

d) The variable _ is assigned the last value by the shell, so the
   current mechanism should work even if block_eval is used as described
   in c.

I've attached a quick-and-dirty patch, just for discussion.  At least
under light testing, it resolves the issues I listed in my previous
response [1].  It does have the change in behavior discussed in c, which
I see as problematic.

It also raises the issue of how to incorporate the block_eval code.  I
think there are two main options.

* Require the user have the module installed and import it.

* Make the module into a snippet to go into the ob-python.el.

The first option is what the current patch does.  The problem is that it
requires users to install an external module.  The second option avoids
that but is uglier and harder to maintain.


[1] There's a remaining issue, which I believe is specific to newer
versions of python.el, where the first time a session block is
executed, it doesn't wait for the shell to load up before sending
input.

>From 642370236f6ca06e70ea32866eedcdba161fd2c4 Mon Sep 17 00:00:00 2001
From: Kyle Meyer 
Date: Thu, 10 Dec 2015 22:09:05 -0500
Subject: [PATCH] WIP lisp/ob-python.el: Use block_eval

---
 lisp/ob-python.el | 108 +++---
 1 file changed, 54 insertions(+), 54 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 95a06f5..baaaf82 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -224,19 +224,18 @@ (defun org-babel-python-initiate-session ( session _params)
 
 (defvar org-babel-python-eoe-indicator "'org_babel_python_eoe'"
   "A string to indicate that evaluation has completed.")
-(defconst org-babel-python-wrapper-method
-  "
-def main():
-%s
 
-open('%s', 'w').write( str(main()) )")
+(defconst org-babel-python-block-eval-import
+  "from block_eval import block_eval as _ob_python_eval")
+
+(defconst org-babel-python-wrapper-method
+  (concat org-babel-python-block-eval-import "
+open('%s', 'w').write(str(_ob_python_eval('%s')))"))
 (defconst org-babel-python-pp-wrapper-method
-  "
+  (concat org-babel-python-block-eval-import "
 import pprint
-def main():
-%s
-
-open('%s', 'w').write( pprint.pformat(main()) )")
+from block_eval import block_eval as _ob_python_eval
+open('%s', 'w').write(pprint.pformat(str(_ob_python_eval('%s'"))
 
 (defun org-babel-python-evaluate
   (session body  result-type result-params preamble)
@@ -247,6 +246,9 @@ (defun org-babel-python-evaluate
 

Re: [O] Proposal and RFC for improving ob-python

[Kyle Meyer]

Ondřej Grover  writes:

[...]

> The ob-ipython project tries to solve this hackiness in a different way by
> using the client-server infrastructure of IPython/Jupyter. That works quite
> well too, but my hope is that improving ob-python.el would also make it
> simpler to use IPython as the python REPL, relying only on the core of the
> Python language.

As I understand it (and if I'm remembering correctly), there are at
least three issues with ob-python sessions:

1. Shell prompt markers leak into the output.

2. Whitespace at the beginning of the output is swallowed.

3. Valid code that has indentation and spaces cannot be sent.

I think the first one is the race conditions that ob-ipython.el's README
refers to.

> It essentially boils down to implementing progn-like eval() function in
> Python which would return the result of the last statement if it is an
> expression.
[...]
> My proposal is to implement an equivalent of the following bash pseudo code
> for non session mode
>
> python -i << HEREDOC_END
> ret = block_eval("""
> 
> """)
> open().write(str(ret))
> HEREDOC_END

So for non-session code and value results, the main idea is to get rid
of the need for "return" at the end?

This seems useful to me.  Requiring a trailing "return", which is a
syntax error when the code is taken by itself, is a quirk of the
implementation (the code ends up being wrapped in a function to get the
return value).  Also, it would allow switching from ":results value" to
":results output" without needing to remove "return" in order to avoid a
syntax error.

> For session mode it would be even simpler, lines containing HEREDOC above
> would be dropped and the rest piped directly into the Python REPL.
>
> This also means that the 'org_babel_python_eoe' string indicator may not be
> necessary anymore because end of evaluation would be simply shown by a new
> line with the primary prompt appearing.

But what about when :results != value?  Doesn't your proposal only
handle returning the last value?

--
Kyle

Re: [O] Proposal and RFC for improving ob-python

[Ondřej Grover]

Thank you for the feedback Kyle.

> As I understand it (and if I'm remembering correctly), there are at
> least three issues with ob-python sessions:

4. session and non-session code differs significantly as you have already
mentioned

> So for non-session code and value results, the main idea is to get rid
> of the need for "return" at the end?
>
Yes, that's the greatest offense to the Python code.

>
> This seems useful to me.  Requiring a trailing "return", which is a
> syntax error when the code is taken by itself, is a quirk of the
> implementation (the code ends up being wrapped in a function to get the
> return value).  Also, it would allow switching from ":results value" to
> ":results output" without needing to remove "return" in order to avoid a
> syntax error.
>
Yes, these would be the obvious benefits.

> But what about when :results != value?  Doesn't your proposal only
> handle returning the last value?
>
You mean :results output ? In that case it could just omit the
"open(...).write(...) " part and capture anything the console prints before
the primary prompt appears again.
Or it could capture stdout into a file, but I think that's needlessly
complicated:
python -i << HEREDOC_END
import sys
sys.stdout = open()   # replace stdout with some file
_ = block_eval("""

""")
sys.stdout.close()
sys.stdout = sys.__stdout__  # restore stdout
HEREDOC_END

After thinking about this, perhaps it is better to capture the block result
into "_" instead of "ret", because
a) it prevents name clashes
b) it is more in line with what "_" is used for

Kind regards,
Ondřej Grover

Re: [O] Proposal and RFC for improving ob-python

[Ondřej Grover]

Hello,

thank you all for confirming that I'm in the right place. If I felt
confident enough that this would work, I wold have already presented a
patch (I'm not an experienced ELisp programmer). That's why I want to
discuss this approach first.

Kyle, I'll be grateful for you feedback.

Kind regards,
Ondřej Grover

On Mon, Dec 7, 2015 at 9:27 PM, Kyle Meyer  wrote:

> Hi Ondřej,
>
> Ondřej Grover  writes:
>
> > TL;DR for those that may have been scared off by the length of my
> previous
> > email:
> > I propose a method of improving ob-python.el by using a progn-like eval()
> > Python function which can wrap and execute source blocks.
> >
> > If this list is not appropriate for discussion of improvements to
> > ob-python.el, could you please give me pointers how to reach people that
> > might be interested in helping and discussing?
>
> Thanks for your proposal.  This is the right place to discuss it.  I've
> only had the chance to briefly look at this, but I'll try to take a
> closer look soon.
>
> --
> Kyle
>

Re: [O] Proposal and RFC for improving ob-python

[Ondřej Grover]

TL;DR for those that may have been scared off by the length of my previous
email:
I propose a method of improving ob-python.el by using a progn-like eval()
Python function which can wrap and execute source blocks.

If this list is not appropriate for discussion of improvements to
ob-python.el, could you please give me pointers how to reach people that
might be interested in helping and discussing?

Kind regards,
Ondřej Grover

On Sat, Dec 5, 2015 at 6:17 PM, Ondřej Grover 
wrote:

> Hello,
>
> I've been playing around with the Org-mode Babel framework and I am
> grateful to all the contributors for making this wonderful library. After
> some time I noticed that Python support seems a little hacky and
> inconsistent and after reading through ob-python.el and consulting Python
> documentation I came up with a proposal for improving it.
>
> The ob-ipython project tries to solve this hackiness in a different way by
> using the client-server infrastructure of IPython/Jupyter. That works quite
> well too, but my hope is that improving ob-python.el would also make it
> simpler to use IPython as the python REPL, relying only on the core of the
> Python language.
>
> It essentially boils down to implementing progn-like eval() function in
> Python which would return the result of the last statement if it is an
> expression. I have come up with a prototype of such a function by diving
> into Python scope internals and its AST capabilities. It was written using
> Org-mode and tangling so it is thoroughly documented and explained, a test
> suite is included. This interesting exercise made me appreciate Lisp even
> more. Here it is
> https://github.com/smartass101/python_block_eval
> I haven't licensed it yet, because I'm not sure what license would be
> appropriate if it was used by org-mode. Any suggestions?
>
> My proposal is to implement an equivalent of the following bash pseudo
> code for non session mode
>
> python -i << HEREDOC_END
> ret = block_eval("""
> 
> """)
> open().write(str(ret))
> HEREDOC_END
>
> For session mode it would be even simpler, lines containing HEREDOC above
> would be dropped and the rest piped directly into the Python REPL.
>
> This also means that the 'org_babel_python_eoe' string indicator may not
> be necessary anymore because end of evaluation would be simply shown by a
> new line with the primary prompt appearing.
>
> The reason why `python -i` (force interactive mode) is used is the IMHO
> poor design choice in the CPython implementation (and other implementations
> have a similar issue AFAIK) to have fast but RO access to local variable
> scope in non-top-level (i.e. functions calling functions) frames/scopes. I
> tried to hack my way around it in the update_locals_after_eval branch of my
> repo to no avail, perhaps some Pythonista among you may know a solution.
>
> I also favor piping input into `python -i` because it means that a
> temporary file does not have to be created.
>
> As explained in the README.org in my repo, this inconsistency can be
> worked around by explicitly first evaluating the side-effect-only part of
> the block and than the last expression with a direct eval() call for each.
> This makes it longer by 2 lines, but has the advantage of properly handling
> variable scope and separating side-effects, which could be used to e.g.
> suppress output. Nevertheless, I think that for Org-mode Babel usage the
> `python -i` and block_eval() approach suffices, unless someone finds a way
> to use the advantages of the alternative approach to improve ob-python.el
> even further.
>
> I'm not a skilled Elisp programmer, so I wanted to ask around as to the
> feasibility of this endeavor or possibly availability of helping hands
> before I devote more time to this.
>
> Kind regards,
> Ondřej Grover
>

Re: [O] Proposal and RFC for improving ob-python

[Kyle Meyer]

Hi Ondřej,

Ondřej Grover  writes:

> TL;DR for those that may have been scared off by the length of my previous
> email:
> I propose a method of improving ob-python.el by using a progn-like eval()
> Python function which can wrap and execute source blocks.
>
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?

Thanks for your proposal.  This is the right place to discuss it.  I've
only had the chance to briefly look at this, but I'll try to take a
closer look soon.

--
Kyle

Re: [O] Proposal and RFC for improving ob-python

[Thomas S . Dye]

Aloha Ondřej Grover,

Ondřej Grover  writes:

> TL;DR for those that may have been scared off by the length of my previous
> email:
> I propose a method of improving ob-python.el by using a progn-like eval()
> Python function which can wrap and execute source blocks.
>
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?
>
> Kind regards,
> Ondřej Grover

This is a good list to discuss improvements to ob-python.

The authors of Babel in Org mode no longer participate in its
development and AFAICT no one has taken over this responsibility.
Instead, there have been changes to one language or another as issues
arise.

I do know that the Babel authors found ob-python difficult to write and
that problems with it have been identified from time to time on this
list.  My sense is that it could be improved, though I don't use it
often enough myself to know.

At any rate, perhaps you could propose a patch and tests?  If the patch
breaks something, then you'll certainly find users interested in
discussing ob-python :)

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [O] Proposal and RFC for improving ob-python

[Achim Gratz]

Ondřej Grover writes:
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?

The best way to contribute to improving ob-python is by showing a patch
that implements your ideas and still passes all existing tests.  Bonus
points if you add new tests for better coverage or any new functionality
or improvements.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

[O] Proposal and RFC for improving ob-python

[Ondřej Grover]

Hello,

I've been playing around with the Org-mode Babel framework and I am
grateful to all the contributors for making this wonderful library. After
some time I noticed that Python support seems a little hacky and
inconsistent and after reading through ob-python.el and consulting Python
documentation I came up with a proposal for improving it.

The ob-ipython project tries to solve this hackiness in a different way by
using the client-server infrastructure of IPython/Jupyter. That works quite
well too, but my hope is that improving ob-python.el would also make it
simpler to use IPython as the python REPL, relying only on the core of the
Python language.

It essentially boils down to implementing progn-like eval() function in
Python which would return the result of the last statement if it is an
expression. I have come up with a prototype of such a function by diving
into Python scope internals and its AST capabilities. It was written using
Org-mode and tangling so it is thoroughly documented and explained, a test
suite is included. This interesting exercise made me appreciate Lisp even
more. Here it is
https://github.com/smartass101/python_block_eval
I haven't licensed it yet, because I'm not sure what license would be
appropriate if it was used by org-mode. Any suggestions?

My proposal is to implement an equivalent of the following bash pseudo code
for non session mode

python -i << HEREDOC_END
ret = block_eval("""

""")
open().write(str(ret))
HEREDOC_END

For session mode it would be even simpler, lines containing HEREDOC above
would be dropped and the rest piped directly into the Python REPL.

This also means that the 'org_babel_python_eoe' string indicator may not be
necessary anymore because end of evaluation would be simply shown by a new
line with the primary prompt appearing.

The reason why `python -i` (force interactive mode) is used is the IMHO
poor design choice in the CPython implementation (and other implementations
have a similar issue AFAIK) to have fast but RO access to local variable
scope in non-top-level (i.e. functions calling functions) frames/scopes. I
tried to hack my way around it in the update_locals_after_eval branch of my
repo to no avail, perhaps some Pythonista among you may know a solution.

I also favor piping input into `python -i` because it means that a
temporary file does not have to be created.

As explained in the README.org in my repo, this inconsistency can be worked
around by explicitly first evaluating the side-effect-only part of the
block and than the last expression with a direct eval() call for each. This
makes it longer by 2 lines, but has the advantage of properly handling
variable scope and separating side-effects, which could be used to e.g.
suppress output. Nevertheless, I think that for Org-mode Babel usage the
`python -i` and block_eval() approach suffices, unless someone finds a way
to use the advantages of the alternative approach to improve ob-python.el
even further.

I'm not a skilled Elisp programmer, so I wanted to ask around as to the
feasibility of this endeavor or possibly availability of helping hands
before I devote more time to this.

Kind regards,
Ondřej Grover