Re: R terminal output does not match src block output due to ">" character in results

[Jack Kamm]

Hi John,

> $ git log
> commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
> origin/master, origin/HEAD)
> Author: Ian Martins 
> Date:   Sat Jan 16 15:52:21 2021 -0500

It looks like the last commit you're on is from about a year ago (Jan
2021), so I suspect you're on an older version of org mode. Maybe
there's an issue with the remote you're pulling from?

I tested out your example R blocks and didn't get any prompt mangling
errors, so I think this issue may have been solved already.

Jack

Re: [BUG] ob-python :results value pp does not working

[Jack Kamm]

Hi Christopher,

> This might need to mentioned in Org manual and in ob-python.el source
> code with comment. WDYT?

The main documentation for this is in the Worg page for ob-python, in
particular see the section "Return values":

https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-python.html

I think that is the appropriate place to put this sort of documentation
(i.e. usage of specific babel languages). However, the clarity of this
documentation could be improved; contributions on that would be very
welcome.

In addition, this is also mentioned in the Org manual on return values
in Org babel, "Results of Evaluation":

For languages like Python, an explicit ‘return’ statement is
mandatory when using ‘:results value’.

Re: [BUG] ob-python :results value pp does not working

[Jack Kamm]

For non-session Python blocks, you need to use "return" on the value, as
if you are in a function.

The following works for me:

#+begin_src python :results value pp
  dic = {'key1': 1, 'b': 2}
  return dic
#+end_src

#+RESULTS:
: {'b': 2, 'key1': 1}

Re: [PATCH] async process in R

[Jack Kamm]

I just noticed one more thing, regarding ess-eval-visibly etc:

>> + (setq user-inject-src-param ess-inject-source)
>>   (setq ess-eval-visibly nil)
>> + (setq  ess-inject-source 'function-and-buffer)
>>   (ess-eval-buffer nil))
>> - (setq ess-eval-visibly tmp)
>> +   (setq ess-eval-visibly tmp)
>> +   (setq ess-inject-source user-inject-src-param)
>
> Rather than using setq, it would be better to let bind these like so:
>
> (let ((ess-eval-visibly nil)
>   (ess-inject-source 'function-and-buffer))
>...code in here...
> )

I noticed these variables are only reset to defaults when ":results
output". It may also be necessary to set them as well for the case of
":results value". In my original implementation [1], I set
"ess-eval-visibly" within the wrapping advice function, so it applied to
all cases.

[1] 
https://github.com/jackkamm/ob-session-async/blob/master/lisp/ob-session-async-R.el

Re: [PATCH] async process in R

[Jack Kamm]

Hi Jeremie and Chuck,

>> But changing to `:async yes', the error aborts in a way that omits the 
>> output.
>
> Interesting, I haven't thought about errors cases enough. Async process
> will be on the 9.5 release and this issue will be next on the todo list.
> Many thanks again for the feedback.

I agree async eval should handle error outputs better. Errors are also
not handled well in ob-python's async eval at the moment. Certainly
something to improve in the future.

Best,
Jack

Re: [PATCH] async process in R

[Jack Kamm]

Hi Jeremie,

Many thanks for bringing this over the finish line! I'm very glad it
made it into Org 9.5.

All the tests passed on my end, and and I successfully ran a few async
R blocks without any issues.

I do have some suggestions for code style below. They apply to both the
original patch, as well as followup fixes that have been committed
since.

> +(defconst ob-session-async-R-indicator "'ob_comint_async_R_%s_%s'")
> +
> +(defun ob-session-async-org-babel-R-evaluate-session

For consistency with the rest of ob-R.el, as well as with the async
functions in ob-python, I suggest using a prefix of
"org-babel-R-async-". So
"ob-session-async-org-babel-R-evaluate-session" would become
"org-babel-R-async-evaluate-session", etc.

> +(ert-deftest ob-session-async-R-simple-session-async-value ()

Again, for consistency I suggest renaming the test functions so they are
prefixed with "test-ob-R/". For example,
"ob-session-async-R-simple-session-async-value" could become
"test-ob-R/async-session-simple-value". This would also allow easily
running all the R tests using "BTEST_RE='.*ob-R.*' make test".

> + (setq user-inject-src-param ess-inject-source)
>   (setq ess-eval-visibly nil)
> + (setq  ess-inject-source 'function-and-buffer)
>   (ess-eval-buffer nil))
> - (setq ess-eval-visibly tmp)
> +   (setq ess-eval-visibly tmp)
> +   (setq ess-inject-source user-inject-src-param)

Rather than using setq, it would be better to let bind these like so:

(let ((ess-eval-visibly nil)
  (ess-inject-source 'function-and-buffer))
   ...code in here...
)

This temporarily sets the variables within the let-block, then resets
them to their original values afterwards. Then you can also remove the
temporary variables "user-inject-src-param" and "ess-eval-visibly-tmp"
as well as their defvars at the top of the file.

Actually, since the code is already in a let-block, you can simply add
these variables to the existing let-statement.

> +   (list (format org-babel-R-write-object-command
> + (if row-names-p "TRUE" "FALSE")
> + (if column-names-p
> + (if row-names-p "NA" "TRUE")
> +   "FALSE")
> + ".Last.value"
> + (org-babel-process-file-name tmp-file 'noquote))

Some parts of ob-session-async-org-babel-R-evaluate-session, such as
the above, are duplicated from org-babel-R-evaluate-session; it would
be good to reduce duplication by abstracting these out to separate
functions.

Thanks again for porting this, and for taking care of ob-R.el in
general.

All the best,
Jack

Re: [PATCH] async process in R

[Jack Kamm]

Hi Jeremie,

> For the parameter  :async without any values assigned to it. I'm
> coordinating with ob-python.el and the orginal package
> https://github.com/jackkamm/ob-session-async.
>
> Jack do you see the need to change it and expect :async to have the
> value yes or no?

I wrote the :async header to use this behavior, following conventions
from other (external) org packages with the :async header.

In particular, below are the packages I know of with ":async" -- I
believe most of them allow using ":async" instead of ":async yes", but
it's been awhile since I've checked, so I may be misremembering:

- ob-async
- ob-ipython
- ob-jupyter
- ob-ein

I can see why requiring an explicit ":async yes" might be preferable
though, and wouldn't oppose the change if there's a consensus for it.

Jack

Re: Bug: ob-python issue in Emacs 28 [9.4.4]

[Jack Kamm]

Hi Augusto,

> In Emacs 28, `python-shell-send-string' prints an additional newline
> between the prompt and output (among other things, so you can tell
> whether or not the inferior process is busy at any given time).
>
> However, this interferes with ob-python, see e.g.
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50514

Thanks for reporting this, and also for your recent improvements to
python.el.

Would you be willing to update your patch to use

(if (version< emacs-version "28") 0 1)

And add a commit message and changelog, so we can push it to master?

> As a permanent solution, I would suggest to replace org-babel-chomp
> and actually strip whitespace from both ends of the string outputted
> from the Python interpreter.  (IPython even has a configuration option
> to add extra whitespace before or after the output.)

I think this would be problematic, because then it would be impossible
to print output with leading or trailing whitespace, which it should be
possible to do.

Best,
Jack

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hi Chuck,

> If you modify the ECM to change `:exports results' to `:exports
> none' and clean older fig[12].png's from the directory, the export
> fails.

Thanks for this example. I had mistakenly thought that code blocks
with ":exports none" would be evaluated for side effects, but you are
right, these blocks are skipped altogether during export.

Instead, I think this use case could be handled by the ":results silent"
header. Blocks with that header are evaluated during export, but the
result is not inserted into the org buffer or the exported document. It
seems like a more general way to evaluate code blocks for side effects
only.

To test this out, you could replace the header arguments in your
example with:

":exports results :results graphics file silent :file fig1.png"

> So if everyone else is determined to make this change I can live
> with it.

I do hope someone submits an RFC, so we can discuss this more
concretely. I still think it would be nice if we could go back to just
":results graphics" to insert a figure. Unfortunately, I'm not currently
able to propose the patch, as I'm still in limbo w.r.t. my employer
agreement.

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hi Chuck,

> Here is an ECM that when exported with `C-c C-e l o y y` (or 'yes RET' for 
> each `y' depending on your setup)

I don't see the example on your last email, could you try re-attaching
it?

Thanks,
Jack

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hello again,

>> A user might like to construct a figure consisting of various subfigures 
>> such as in a subfloat environment.
>>
>> Will this be reasonably simple to accomplish if `:results graphics' (with no 
>> `file' element) automatically inserts a link?
>>
>> Currently, omitting the file element leaves the link out, which I believe is 
>> the most direct way to approach subfloats.
>
> Thanks for bringing up this use case, it hadn't occurred to me before.

Thinking about this more, it occurred to me that the ":exports code" or
":exports none" header should already handle this.

When that header is set, the graphics result won't be added to the latex
document, and the user can construct the subfigure separately in latex.

Then we wouldn't need to support the use-case of ob-R creating a graphic
but not producing a result from it...which still feels a little strange
to me, to be honest.

Or am I missing something still?

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hi Chuck,

> A user might like to construct a figure consisting of various subfigures such 
> as in a subfloat environment.
>
> Will this be reasonably simple to accomplish if `:results graphics' (with no 
> `file' element) automatically inserts a link?
>
> Currently, omitting the file element leaves the link out, which I believe is 
> the most direct way to approach subfloats.

Thanks for bringing up this use case, it hadn't occurred to me before.

I guess this use case is relatively new, since prior to Org 9.3
":results graphics" would insert a link.

But, I can see why this functionality would be useful, and if people
have come to use it, we should try not to break back-compatibility, or
at least provide an alternate mechanism to support it.

I do think that inserting a link for graphics is probably the more
common use case, and should be easier by default.

But clearly, the situation is not quite as simple as I had originally
thought.

Best,
Jack

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Timothy  writes:

> Would it be strange if running the code block with just
>
> :output graphics
>
> Automatically added "link" if *only* graphics is set, and generated a
> file name if no :file is set?

It wouldn't be strange. ob-jupyter [1] auto-generates file names for
images, and it's convenient.

Currently, I believe that if you set global ":file-ext" and
":output-dir" arguments, and also set ":results graphics file" and name
the source block, then org-mode will generate the file name based on the
source block name, output-dir, and file-ext. I think this is similar to
what you want, except for the requirement to name the source
block. Perhaps we could generate the filename from a UUID or hash,
similar to ob-jupyter, if the source block isn't named.

I do think this is more involved, and somewhat orthogonal, to dropping
the ":results file" requirement when ":results graphics" is present. So
it should probably be done in a separate commit.

[1] https://github.com/nnicandro/emacs-jupyter

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hi Jeremie,

>> The requirement for a second file parameter was added in Org 9.3 to
>> support the use case in this thread:
>>
>> https://orgmode.org/list/3ac2f42a-8ff2-1464-fa36-451e2ef0e...@pressure.to/
>>
>> But this syntax is annoyingly verbose for ob-R users, and also broke
>> lots of ob-R examples prior to Org 9.3.
>>
>> A simple fix might be to have the "graphics" flag implicitly add the
>> "file" flag as well. But we would need to first check that this doesn't
>> break other use cases.
>
> I do agree with this solution. If the current specification works, we
> could make it easy for ob-R user by implicitly adding a file flag. But
> as far as I understand, the change will have to be made in ob-core.el.

Hmm, I think you're right -- this would have to be done in ob-core.el.

I think it would still make sense though, and would be beneficial beyond
ob-R. According to [1], the "graphics" and "link" arguments don't do
anything unless used with "file", so it would make sense for them to
automatically add the "file" argument.

[1] https://orgmode.org/manual/Results-of-Evaluation.html#Results-of-Evaluation

Re: [PATCH] ob-R output file with graphics parameter

[Jack Kamm]

Hi all,

> I obviously missing something. The above works for me without the
> patch. Unfortunately, I can't trace back the thread in order to
> understand the context.

I think this is a followup from this mail:

https://orgmode.org/list/87zgxc42qg@gmail.com/

wherein Jeremie states:

> The current patch have been tested for remote connections as well and
> AFAIK, nothing breaks.
> 
> But I'm afraid that the graphical output is broken and has long been
> even before the path. The test for graphical output is compromised and
> does not do the right test. I will suggest new ones.

So I think it's to do with graphical outputs in remote R sessions.

However, I tested Jeremie's example on latest org master and it also
worked fine for me, also on remote sessions.

Jeremie, could you clarify the issue this fixes?

> +(defun org-babel-output-link-path:R (params)
> +  "format org-link to file from PARAMS"
> +  
> +  (format "[[file:%s]]" (concat (cdr (assq :dir params))
> + "/"
> +(cdr (assq :file params)

Rather than concat, I think it is better to use expand-file-name, and
also call file-name-as-directory on the directory component.

Stylistically, I don't think there should be a blank line between the
docstring and the code. The docstring should also start with a capital
letter and end with a period.

> +(setq out-file-path  (concat
> +  (replace-regexp-in-string "/ssh:.*?:" "" (cdr 
> (assq :dir params)))
> + "/"
> +  out-file))

Use org-babel-process-file-name instead of replace-regexp-in-string to
get the local path of the file.

> - Remove second the need for a second file parameter

This would indeed be nice.

The requirement for a second file parameter was added in Org 9.3 to
support the use case in this thread:

https://orgmode.org/list/3ac2f42a-8ff2-1464-fa36-451e2ef0e...@pressure.to/

But this syntax is annoyingly verbose for ob-R users, and also broke
lots of ob-R examples prior to Org 9.3.

A simple fix might be to have the "graphics" flag implicitly add the
"file" flag as well. But we would need to first check that this doesn't
break other use cases.

> Subject: [PATCH 1/4] ob-R.el: Remove redundant argument to function

I think it would be better to squash these changes into a single commit.

Re: [BUG] org babel fails with matlab+python kernel on MacOS-10.15 [9.4.6 (release_9.4.6-541-g52b097 @ /Users/oub/emacs/site-lisp/packages/org/)]

[Jack Kamm]

Hello Uwe,

Uwe Brauer  writes:

> (defalias 'org-babel-execute:matlab 'org-babel-execute:ipython)

ob-ipython is not part of org-mode core, so I don't think this can be
considered a bug here.

While the original ob-ipython is unmaintained, scimax [1] maintains a
fork of it, so you could consider seeking help there.

Also, it looks like you're trying to use org-mode with a jupyter matlab
kernel, so you may also consider ob-jupyter from the emacs-jupyter [2]
project. I have had good success using it to run jupyter kernels from
various languages before.

Also, I would suggest verifying that the jupyter matlab kernel works
outside of Emacs/org-mode, to isolate where the problem is.

> |   ob-ipython--dump-error("Traceback (most recent call last):\n  File 
> \"/Users/...")

This line seems to be the critical one in the error output. If you can
expand out the full error message, it may elucidate what the problem is.

[1] https://github.com/jkitchin/scimax
[2] https://github.com/nnicandro/emacs-jupyter

Re: TMIO Pre-release, request for feedback

[Jack Kamm]

Hi Jeremie,

Jeremie Juste  writes:

> Just a precision async process is already available in R. Thanks again
> to Jack Kamm for this input.

I believe async evaluation in R still requires my external package:

https://github.com/jackkamm/ob-session-async

Are you sure you don't have that package enabled?

I'm currently blocked on porting the R implementation here, while
awaiting paperwork at my current job (I am making headway, but slowly).

In the meantime, feel free to go ahead and port it, if you like. It
should be a very simple copy-paste, changing 1 or 2 variable names. All
that code was written while at my previous job, which had signed the
copyright disclaimer, so is safe to copy.

You can also refer to my port of async sessions to ob-python, which I
wrote before switching jobs:

https://code.orgmode.org/bzg/org-mode/commit/53fd5b774e23406ed351bdb166ab35edd0c44892

https://orgmode.org/list/87h7qi2l2m@gmail.com/

Best,
Jack

Re: literate programming, development log -- ideas?

[Jack Kamm]

Hi Greg,

Greg Minshall  writes:

> but i also feel a need for something that might be called a lab
> notebook, a development log, of ideas, including dead ends, i pursue
> during the development process, with links, etc..  but, i'm not really
> sure how to structure this bit, how to integrate it in the rest of the
> .org file -- i.e., as a separate heading, or related to the code section
> that (originally) was under development when the notes were created.
> or...?  etc.

This is how I have been using org-babel recently. In particulary, I use
it as a journal for my exploratory data analyses (EDA), mainly in Python
and R.

I find that using a month-tree format, similar to org-capture with
:tree-type month [1], works well for organizing my EDA sections/notes.

Figures I create are named like so:

fig/eda/<6-digit-date>-.png

e.g., "fig/eda/210607-celltype-heatmap.png".

I found this workflow keeps things fairly neat, without having to think
too much about how I'm going to organize and name everything. Perfect
for trying out new ideas, iterating quickly, and having an easy to refer
to log of what I've already tried.

[1] https://orgmode.org/manual/Template-elements.html

Re: Moving some lisp/ob-*.el files to org-contrib - your advice?

[Jack Kamm]

Eric S Fraga  writes:

> On Tuesday,  4 May 2021 at 01:49, Timothy wrote:
>> For the future, I'd think Julia actually warrants 1st class inclusion in
>> Org, and I've instigated an effort to write an ob-julia that works well.
>
> +1!  Happy to help test if you wish.  I use Julia as my programming
> language these days.

+1 from me as well, I think Julia passes the "well-established" test and
is an important language for scientific computing.

I like Julia, but only occasionally use it, and it seems to frequently
cause troubles with my Org config whenever I update or move
computers. So I appreciate this effort to support it better -- thank you 
Timothy.

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

Sorry for the noise, replying to add the X-Woof-Patch:applied header.

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

Hi ian,

ian martins  writes:

> I gave this a try and it works for me. One thing I noticed is that if you
> run a call asynchronously, the final result ends up under the source block
> instead of the call.  In the example below both RESULTS were written after
> I ran the call.
>
> #+name: test-call
> #+begin_src python :results output
>   import time
>   time.sleep(5)
>   print("done")
> #+end_src
>
> #+RESULTS: test-call
> : done
>
> #+call: test-call() :session :async
>
> #+RESULTS:
> : 70e844920752b3411170716dc450c50f

Thank you for reporting. I'm adding the X-Woof-Bug header to this thread
so we can track it in updates.orgmode.org.

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

Hi Bastien,

Bastien  writes:

> Please feel free to commit this patch in master so that more people
> can test it, we can test and fix oddities while preparing for 9.5.

OK, I have incorporated the minor fixes from Kyle's review and pushed to
master.

Cheers,
Jack

Re: [POLL] Setting `org-adapt-indentation' to nil by default?

[Jack Kamm]

+1 from me. It is nil in my config, which feels better to me.

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

Hi Timothy,

> This is moving at a glacial pace, but I'd love to see this merged ---
> there's clearly a lot of interest in this from the community if not
> within this mailing list (ob-async which is more limited has 250 stars
> on GitHub).

Yes, this has taken far too long -- sorry about that.

There have been a few things going on in my life recently, among them a job 
change. I am in the process of trying to get my FSF copyright forms approved at 
my new job. I think this will eventually happen, but the process is moving 
slowly.

My last update on this thread was shortly before changing jobs, and I decided 
not to merge until I was sure I'd be able to stick around to maintain it.

If someone is willing to apply the final tweaks and help with maintenance of 
this functionality, please go ahead and merge this in. Otherwise, I'll merge 
this as soon as I've got my paperwork approved and am back in action.

Jack

Re: overloading of internal priority calculations in agenda

[Jack Kamm]

Hi Adam,

> I further noticed that this overloading of the internal priority by
> including timestamp and habit data causes disruption to the behaviour
> I imagine most users would expect from `org-agenda-sorting-strategy'.
> For example, if you have `priority-down' as the first entry in the
> `agenda' section and `category-keep' as the second, then differences
> in the SCHEDULED timestamp are included in the priority calculation
> and can therefore prevent sorting of two adjacent [#B] items by
> category.  This seems like a bug to me, or at least breaks the
> Principle of Least Surprise.

I just ran into this issue you highlight here.

In particular, I was trying to set the org-agenda-sorting-strategy to

(priority-down scheduled-down)

i.e., sorting by priority (highest first), and then within priority,
sorting by scheduled (most recent first).

However, the fact that the priority includes the scheduled timestamp
makes this sorting strategy impossible.

I agree this seems like a bug, in that it contradicts the written
documentation as far as I can tell (for example, the *Help* for
org-agenda-sorting-strategy mentions nothing of the fact that the
priority includes the scheduled timestamp, and I don't see anything
about it in the *Info* either).

I imagine that many have gotten used to the default behavior of sort by
highest priority, then by earliest scheduled timestamp, but we could
keep this default behavior by adding "scheduled-up" after
"priority-down" in org-agenda-sorting-strategy, as you allude.

Jack

Re: ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi Jeremie,

> Many thanks for this package. It seems like a better way all together
> to manipulate python output.

I don't disagree, at least when it comes to handling dataframes.

This is because ob-reticulate makes the block actually executed with
ob-R.el instead of ob-python.el. R has better built-in support for
manipulating dataframes, which ob-R.el can take advantage of when
handling output.

I tried before to improve ob-python handling of dataframes, plots, and
other results, but the code got a bit messy and I decided to put it on
ice. I may return to it, some day.

Cheers,
Jack

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

Hi Timothy,

Many thanks for testing this out.

> I just tried to give this a shot.
> First up, I had to remove the ORG-NEWS part of the patch to be able to
> provide it. It would be nice if you could update the patch so this
> applies cleanly.

I'm attaching an updated patch rebased on master.

> I was initially unable to get this to seem to work, until I changed the
> :results type to "output".
>
> #+begin_src python :async :session blah
> return(a)
> #+end_src
>
> #+RESULTS:
> : /tmp/babel-62cQRX/python-EfJ4o4

ob-python session blocks don't use "return", so this should just be:

#+begin_src python :async :session blah
a
#+end_src

#+RESULTS:
: 2

> Finally, I see that this requires :session to be set in order to work.
> Might it be possible to have this work for non-session blocks too? It
> seems odd that what I'd imagine is the harder case (session blocks) is
> supported, but one-shot (non-session) blocks aren't.

The non-session case is substantially different, and I think it would
probably require a separate implementation.

One possible approach would be to modify ob-eval.el, so that
org-babel--shell-command-on-region uses make-process instead of
process-file.

I agree it would be nice to have, but it would take a bit of work to
figure it all out, and there is already ob-async.el [1] that implements
non-session async for all languages. (I wish it could be brought into
org-mode, but it probably can't, because it depends on the external
async.el.)

> p.s. After this is merged, it would be great to see support for other
> languages grow :)

I also have an async implementation for ob-R that's ready after this is
merged :)

Cheers,
Jack

[1] https://github.com/astahlman/ob-async

>From 864a2377b4eea58df6b0ccd07c4bcba080ecc724 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sun, 28 Feb 2021 13:17:33 -0800
Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation

Adds functionality to ob-comint.el to implement async session eval on
a per-language basis.  Adds a reference implementation for ob-python.

* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.

* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.

*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
---
 etc/ORG-NEWS   |  15 +++
 lisp/ob-comint.el  | 173 +++--
 lisp/ob-python.el  |  56 ++-
 testing/lisp/test-ob-python.el |  61 
 4 files changed, 295 insertions(+), 10 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index f95a568a6..eff75605c 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -169,6 +169,21 @@ tags including from both buffer local and user defined persistent
 global list (~org-tag-alist~ and ~org-tag-persistent-alist~).  Now
 option ~org-complete-tags-always-offer-all-agenda-tags~ is honored.
 
+*** Async session evaluation
+
+The =:async= header argument can be used for asynchronous evaluation
+in session blocks for certain languages.
+
+Currently, async evaluation is supported in Python.  There is also
+functionality to implement async evaluation in other languages that
+use comint, but this needs to be done on a per-language basis.
+
+By default, async evaluation is disabled unless the =:async= header
+argument is present.  You can also set =:async no= to force it off
+(for example if you've set =:async= in a property drawer).
+
+Async evaluation is disabled during export.
+
 ** Miscellaneous
 *** =org-goto-first-child= now works before first heading
 
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index b14849df6..d81ff3edd 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -93,12 +93,7 @@ (defmacro org-babel-comint-with-output (meta  bo

ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi all,

ob-reticulate is now available on MELPA.

You can find more information here:
https://github.com/jackkamm/ob-reticulate

Cheers,
Jack

Re: [PATCH] Persistently save downloaded inline remote images

[Jack Kamm]

> Jack, as the author of the org-display-remote-inline-images commit, what
> do you think about this patch?

This patch takes the same approach as an earlier attempt by Kit in 2014
[1].  That patch was never merged; Michael objected that the approach
seemed too complex, and suggested using file-local-copy and cacheing the
filename instead; he also suggested implementing this cacheing
functionality in Tramp instead of Org.

So, when I submitted a new attempt at this, I chose to cache by simply
opening the image files in Emacs instead, since the implementation is
much simpler, and would hopefully sidestep the objections from the 2014
approach.

While cacheing via tempfiles is more complex, it does have some
benefits: it allows cacheing across Emacs sessions, and it avoids the
side effect of opening the image files in Emacs.

I'm agnostic whether tempfiles or Emacs buffers are ultimately
better. My inclination is to be conservative and stick with the status
quo unless there's a compelling reason to switch.

So, I suggest Ferdinand confirm he's tested the existing "cache" option,
and articulate why he found it unsuitable, before merging this.

If we do proceed in this direction, I'd suggest replacing the current
cache mechanism entirely -- I don't see a good reason to maintain 2
solutions to the same problem. Also, we could combine the "download" and
"cache" options into a single implementation, making them synonyms for
backwards compatibility.

[1] https://lists.gnu.org/archive/html/emacs-orgmode/2014-11/msg00690.html
[2] https://lists.gnu.org/archive/html/emacs-orgmode/2020-01/msg00177.html

Re: Bug report: remote file python src output gives FileNotFound (+ suggested fix)

[Jack Kamm]

Hi Paul,

I'm unable to reproduce the issue, and believe this issue has already
been fixed in the latest version of org-mode. See this patch [1]. Could
you please upgrade to Org 9.4 and test again?

Thanks,
Jack

[1] 
https://orgmode.org/list/87h7t16red.fsf@pc.i-did-not-set--mail-host-address--so-tickle-me/


paul  writes:

> When working remotely, a python SRC block with a session and :results set to 
> output will return a FileNotFoundError.
> To reproduce this bug:
> 1. Open a .org file remotely
> 2. Insert the following
> #+BEGIN_SRC python :results output :session check
> print("a")
> #+END_SRC
> 3. org-ctrl-c-ctrl-c in the code block
>
> I already figured out how to fix it:
> In ob-python.el.gz, the function org-babel-python-evaluate-session the 
> let-variable tmp-src-file is made, which contains tramp-prefix when working 
> remotely. But the tramp-prefix is still there here:
> (format org-babel-python--exec-tmpfile tmp-src-file)
> which causes the remotely executed command to still contain the prefix, and 
> hence it cannot find it.
> To fix, the line above could be replaced with
> (format org-babel-python--exec-tmpfile (or (file-remote-p tmp-src-file 
> 'localname) tmp-src-file))
>
> Hope this helps :) (and many many thanks for org-mode)

Re: ob-python: import local package into a session

[Jack Kamm]

Joost Kremers  writes:

> I haven't really considered the option to install the utility functions as a
> package in the virtual environment, because I expect to change and develop 
> those
> functions together with the rest of the project. If it were a separate 
> package,
> I'd need to reinstall it every time I make changes to it, which will probably
> happen often.

If you install the package using either "python setup.py develop", or
"pip install -e", then Python will install your code via symlinks
instead of copying, so then you don't have to worry about reinstalling
every time you make an edit.

To switch between venv's in emacs, I use pyvenv:
https://github.com/jorgenschaefer/pyvenv

Re: ob-python: import local package into a session

[Jack Kamm]

Jack Kamm  writes:

> You need to make sure your module is either in the working directory you
> started the Python session in, or in your PYTHONPATH, for example by
> adjusting os.env["PYTHONPATH"] before attempting to import the module.

Sorry, this was incorrect, you need to set PYTHONPATH before starting
Python. The correct way to do it from within Python is to use sys.path,
as John points out.

Re: ob-python: import local package into a session

[Jack Kamm]

Hi Joost,

> What I'm trying to do is to import a Python file with a bunch of utility
> functions into the ob-python session. I thought this might be possible if I'd
> structure my code as a regular Python package, because that works if I want to
> import my utility functions into another Python file. But it doesn't seem to
> work for the ob-python session.

You need to make sure your module is either in the working directory you
started the Python session in, or in your PYTHONPATH, for example by
adjusting os.env["PYTHONPATH"] before attempting to import the module.

This shouldn't be ob-python or even Emacs specific. You can test whether
things work by typing "python" in the terminal and attempting to import
your module.

By the way, are you using IPython or vanilla Python? I recently
encountered an issue trying to import modules through a symlink in
IPython, whereas it worked perfectly fine in a vanilla Python session.

Jack

Re: [PATCH] New "project" option for org-link-file-path-type

[Jack Kamm]

Thanks, I've fixed the remaining issues you pointed out and pushed this
in 5371b30fe.

Cheers,
Jack

Re: [PATCH] New "project" option for org-link-file-path-type

[Jack Kamm]

Hi Kyle,

> As a projectile user, I'm tempted to suggest that, instead of the adding
> the `project' value, org-insert-link could learn to call
> org-link-file-path-type if it is a function and, if that returns
> non-nil, do the prefix check.  Then projectile users could set it to
> projectile-project-root.  It seems project.el doesn't have a similar
> function that could be called without any arguments, but I guess we
> could add a simple ol- wrapper.  I'm not sure that's a good idea,
> though.

I like the idea of letting org-link-file-path-type be a function.

However, it struck me that it might be too limiting to just have the
function return the project root. There's a lot more potential for
customization here -- for example, a user might want to combine the
noabbrev option with the adaptive option. If we could instead pass a
function that takes the filename as an argument and returns the path to
insert, that would allow for greater flexibility.

Other benefits are that the implementation is much simpler, and
subjectively I think it's more intuitive to explain the meaning of this
option (as opposed to an option where the user passes a function that
returns the project root).

The downside of this is that the user has to do a bit more work and
write some elisp to take advantage of the option.

I've attached an updated patch in this direction. What do you think? I
think the simplicity and flexibility outweighs the downside, but I'm not
sure.

> The :package-version keyword should be added to signal the change in
> value.

Thanks for the tip, I've added this.

> Functionally I think your current patch would only support Emacs's
> unreleased master, unless the user installed a new project.el via ELPA.
> More on that below.

Good catch. I didn't realize I was using project.el from ELPA but it
turns out I was.

>From d156a9cfcdbfb9be72df3976e2355f48cf10 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Wed, 28 Oct 2020 17:29:04 -0700
Subject: [PATCH] ol.el: New option to set org-link-file-path-type to a
 function

* lisp/ol.el (org-link-file-path-type): Add new option.
(org-insert-link): Handle function option for org-link-file-path-type.
---
 etc/ORG-NEWS | 19 +++
 lisp/ol.el   | 12 ++--
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..891a680ae 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -35,6 +35,25 @@ omit a file description was to omit the header argument entirely,
 which made it difficult/impossible to provide a default value for
 =file-desc=.
 
+*** New option to set ~org-link-file-path-type~ to a function
+
+If ~org-link-file-path-type~ can now be set to a function that takes
+the full filename as an argument and returns the path to link to.
+
+For example, if you use ~project.el~, you can set this function to use
+relative links within a project as follows:
+
+#+begin_src emacs-lisp
+(setq (org-link-file-path-type
+   (lambda (path)
+ (let* ((proj (project-current))
+(root (if proj (project-root proj) default-directory)))
+   (if (string-prefix-p (expand-file-name root) path)
+   (progn
+ (file-relative-name path))
+ (abbreviate-file-name path))
+#+end_src
+
 ** New features
 *** =ob-python= improvements to =:return= header argument 
 
diff --git a/lisp/ol.el b/lisp/ol.el
index 951bb74e7..262a6c5ae 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -212,13 +212,18 @@ (defcustom org-link-file-path-type 'adaptive
 absolute  Absolute path, if possible with ~ for home directory.
 noabbrev  Absolute path, no abbreviation of home directory.
 adaptive  Use relative path for files in the current directory and sub-
-  directories of it.  For other files, use an absolute path."
+  directories of it.  For other files, use an absolute path.
+
+Alternatively, users may supply a custom function that takes the
+full filename as an argument and returns the path."
   :group 'org-link
   :type '(choice
 	  (const relative)
 	  (const absolute)
 	  (const noabbrev)
-	  (const adaptive))
+	  (const adaptive)
+	  (function))
+  :package-version '(Org . "9.5")
   :safe #'symbolp)
 
 (defcustom org-link-abbrev-alist nil
@@ -1876,6 +1881,9 @@ (defun org-insert-link ( complete-file link-location description)
 	(setq path (expand-file-name path)))
 	   ((eq org-link-file-path-type 'relative)
 	(setq path (file-relative-name path)))
+	   ((functionp org-link-file-path-type)
+	(setq path (funcall org-link-file-path-type
+(expand-file-name path
 	   (t
 	(save-match-data
 	  (if (string-match (concat "^" (regexp-quote
-- 
2.29.2

Re: Org mode fontification error in # in python and ipython source blocks

[Jack Kamm]

Hi Sebastian --

> I am having problems with the fontification of python and ipython source
> blocks when the code contains curly brackets "{}" (other course blocks are
> ok). For instance, the following snippet
>
> #+BEGIN_SRC python :results drawer
> import matplotlib.pyplot as plt
> plt.plot([1,2,3])
> plt.show()
> a=1
> print("a={}".format{a})
> #+END_SRC
>
> does not fontify correctly in either python or ipython source blocks.
>
> As a consequence, when exporting the org file to HTML (C-c C-e h h), it
> fails with the following message:
>
> font-lock-fontify-keywords-region: Invalid function: #

I'm unable to reproduce this behavior on emacs 27.1 and git
master. Fontification and export work fine with this code block for me.

Does the error still happen when you use emacs -q ? Or if you use git
master?

Jack

Re: Bug: HTML not formatted correctly from R source code block [9.3.6 (9.3.6-23-g01ee25-elpaplus @ /home/opdfa/.emacs.d/elpa/org-plus-contrib-20200309/)]

[Jack Kamm]

Hi Steven,

Sorry for the delayed response.

> The problem, however, is that what is exported to html and displayed in the
> exported block is either the actual UUID or the tempfile path and not the
> results from evaluating the R code. In the case of the tempfile, the tempfile
> exists but is empty.

Yes, async doesn't work well with export. I think it would be very
challenging to make it work correctly.

For my recent patch to add this functionality into org-mode [0], my
workaround was to simply disable async evaluation during export.

[0] https://orgmode.org/list/87h7qi2l2m@gmail.com/

Re: Thoughts on the standardization of Org

[Jack Kamm]

Hi Timothy,

TEC  writes:

> I feel that this also ties into my earlier idea of putting Emacs
> as/inside an LSP server for Org.  I suspect there may be a a lot 
> of
> potential in making it dead easy to use Emacs as a tool.

I'm too busy to help on this, but I think it's a very good idea and hope
you can pursue it.

In terms of making org-mode more accessible outside emacs, there are 2
main benefits I'd personally be excited about:

1. Being able to collaborate on org-babel computational notebooks with
non-emacs users.
2. Improved mobile apps for org-mode.

Jack

[PATCH] New "project" option for org-link-file-path-type

[Jack Kamm]

The attached patch adds a "project" option for org-link-file-path-type.

When this is set, links to files under the current project root will be
relative, while links elsewhere are absolute.

It relies on project.el, which appears to have been added in emacs 25. I
used fboundp to check whether the functionality is available, and to
silence compiler warnings. I'm not sure if this is the correct way to do
it.

>From c5f9d4043a6cf6a325d122be24214356f36446f1 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Wed, 28 Oct 2020 17:29:04 -0700
Subject: [PATCH] ol.el: New option "project" for org-link-file-path-type

* lisp/ol.el (org-link-file-path-type): Add new option.
(org-insert-link): Handle project option for org-link-file-path-type.
---
 etc/ORG-NEWS |  8 
 lisp/ol.el   | 17 +++--
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..b9adc9089 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -88,6 +88,14 @@ package, to convert pandas Dataframes into orgmode tables:
 | 2 | 3 | 6 |
 #+end_src
 
+*** New option to use relative paths for links in same project
+
+If =org-link-file-path-type= is =project=, inserted links under the
+current project root will use relative paths.
+
+If not in a project, or if =project.el= is not available (as in older
+versions of Emacs), links behave as default (=adaptive=).
+
 * Version 9.4
 ** Incompatible changes
 *** Possibly broken internal file links: please check and fix
diff --git a/lisp/ol.el b/lisp/ol.el
index 951bb74e7..9c48bd9b5 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -212,13 +212,17 @@ (defcustom org-link-file-path-type 'adaptive
 absolute  Absolute path, if possible with ~ for home directory.
 noabbrev  Absolute path, no abbreviation of home directory.
 adaptive  Use relative path for files in the current directory and sub-
-  directories of it.  For other files, use an absolute path."
+  directories of it.  For other files, use an absolute path.
+project   Use relative path for files in the current project and sub-
+  directories of it. For other files, usue an absolute path.
+  If project.el is not available, behave as adaptive."
   :group 'org-link
   :type '(choice
 	  (const relative)
 	  (const absolute)
 	  (const noabbrev)
-	  (const adaptive))
+	  (const adaptive)
+	  (const project))
   :safe #'symbolp)
 
 (defcustom org-link-abbrev-alist nil
@@ -1876,6 +1880,15 @@ (defun org-insert-link ( complete-file link-location description)
 	(setq path (expand-file-name path)))
 	   ((eq org-link-file-path-type 'relative)
 	(setq path (file-relative-name path)))
+	   ((and (fboundp 'project-current)
+		 (fboundp 'project-root)
+		 (project-current)
+		 (eq org-link-file-path-type 'project))
+	(if (string-prefix-p (expand-file-name (project-root
+		(project-current)))
+ (expand-file-name path))
+		(setq path (file-relative-name path))
+	  (setq path (abbreviate-file-name (expand-file-name path)
 	   (t
 	(save-match-data
 	  (if (string-match (concat "^" (regexp-quote
-- 
2.29.1

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Jeremie,

Thanks for volunteering to maintain ob-R.el :)

I'm bumping this patch [0] to see if it could be merged into ob-R.el. It
fixes some long-standing issues with prompt mangling for :session blocks
with :results output.

Elsewhere in the thread, Chuck had a suggestion [1] to allow :results
value to be properly handled in remote sessions. It's not included in
the patch, but worth adding separately IMO.

It may be helpful reading the thread in full -- Chuck had several
comments on ob-R that I found insightful, and may be useful for you to
consider as maintainer. I'd highlight [2] in particular:

> If you do decide to dig into solving this, please be sure that remote
> sessions and graphical outputs are not broken.  test-ob-R.el does not
> cover those cases.  In fact, it is pretty short, so there are probably
> other things that could break without `make test' complaining.

All the best,
Jack

[0] https://orgmode.org/list/87ft7t9wqk@gmail.com/
[1] 
https://orgmode.org/list/1e0046a6-1fab-45b5-9b08-68fe1be98...@health.ucsd.edu/
[2] 
https://orgmode.org/list/352c7149-743f-4944-aca5-7a1242b5a...@health.ucsd.edu/

Re: New website - back to the old unicorn!

[Jack Kamm]

Thank you Timothy! It looks great.

Bastien  writes:

> Dear all,
>
> thanks to the initiative and the patient efforts of Timothy, our
> website has been revamped: new contents, new look and... the old
> unicorn!
>
> Thanks very much to Timothy and to everyone who contributed with
> feedback and ideas.
>
> This is a new basis that we will continue to polish and enhance.
>
> Enjoy :)
>
> https://orgmode.org
>
> -- 
>  Bastien

[PATCH] Async session eval (2nd attempt)

[Jack Kamm]

This patch adds asynchronous evaluation for session blocks in
Python. It also adds functionality to implement async session eval for
other languages using ob-comint.el.

To test the attached patch, add ":async" to a Python session block
with a long computation (or "time.sleep") in it. Upon evaluation, your
Emacs won't freeze to wait for the result -- instead, a placeholder
will be inserted, and replaced with the true result when it's ready.

I'll note how this is different from some related projects. ob-async
implements asynchronous evaluation for Babel, but it doesn't work with
sessions. emacs-jupyter, ein, and ob-ipython all implement
asynchronous session evaluation, but only for Jupyter kernels. Jupyter
is great for some cases, but sometimes I prefer to use the built-in
org-babel languages without jupyter.

The new functionality is mainly implemented in
`org-babel-comint-async-filter', which I've defined in ob-comint.el,
and added as a hook to `comint-output-filter-functions'.  Whenever new
output is added to the comint buffer, the filter scans for an
indicator token (this is inspired by
`org-babel-comint-with-output'). Upon encountering the token, the
filter uses a regular expression to extract a UUID or temp-file
associated with the result, then searches for the appropriate location
to add the result to.

This is my 2nd attempt at this patch [0]. I have also ported it to an
external package [1], but would like to have this functionality in Org
proper, to permit better code reuse between async and sync
implementations. The external package also includes an R
implementation that I regularly use, as well as a Ruby implementation,
but I've left these out to keep this initial patch smaller, and also I
need to confirm copyright assignment on the Ruby implementation which
was externally contributed.

[0] 
https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
[1] https://github.com/jackkamm/ob-session-async

>From 8b7695a148d1831c916737650e115833cb7fc752 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sun, 25 Oct 2020 11:40:10 -0700
Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation

Adds functionality to ob-comint.el to implement async session eval on
a per-language basis.  Adds a reference implementation for ob-python.

* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.

* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.

*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
---
 etc/ORG-NEWS   |  15 +++
 lisp/ob-comint.el  | 172 +++--
 lisp/ob-python.el  |  56 ++-
 testing/lisp/test-ob-python.el |  61 
 4 files changed, 294 insertions(+), 10 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..9d5fbbe30 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -88,6 +88,21 @@ package, to convert pandas Dataframes into orgmode tables:
 | 2 | 3 | 6 |
 #+end_src
 
+*** Async session evaluation
+
+The =:async= header argument can be used for asynchronous evaluation
+in session blocks for certain languages.
+
+Currently, async evaluation is supported in Python.  There is also
+functionality to implement async evaluation in other languages that
+use comint, but this needs to be done on a per-language basis.
+
+By default, async evaluation is disabled unless the =:async= header
+argument is present.  You can also set =:async no= to force it off
+(for example if you've set =:async= in a property drawer).
+
+Async evaluation is disabled during export.
+
 * Version 9.4
 ** Incompatible changes
 *** Possibly broken internal file links: please check and fix
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index d3484bb7c..591754dac 100644
--- a/

Re: [PATCH] ob-python: Rename exec tmpfile handle to prevent conflict

[Jack Kamm]

Thanks Bastien, the Woof! tool looks interesting.

By the way, on seeing this thread again, I realized this patch probably should 
have been applied to the maint branch. So I've cherry picked it into there, and 
merged back into master.

Bastien  writes:

> Hi Jack and Adrian,
>
> Jack Kamm  writes:
>
>> Adrian Kummerländer  writes:
>>
>>> I noticed that after updating to Org 9.4 many of my Python-based Org
>>> files fail to execute with various `io.TextIOWrapper' related error
>>> messages. The reason for this is that opening the exec tmpfile as `f'
>>> shadows this possibly user-defined variable.
>>>
>>> The attached patch fixes this problem for me. As this is my first
>>> time contributing to Org I am especially open for any suggestions!
>>
>> The patch looks good.  I've pushed it to master, after making a couple
>> minor fixes to the commit message (adding a TINYCHANGE cookie and
>> adjusting some of the spacing).
>
> Thanks for applying the patch, I'm marking it as "applied" through
> Woof! adding "X-Woof-Patch: applied" in the headers.
>
> Best,
>
> -- 
>  Bastien

Re: [RFC] ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi all,

I've put ob-reticulate.el on github so it can be more easily used:
https://github.com/jackkamm/ob-reticulate

I plan to submit it to MELPA or GNU ELPA over the coming weeks as well.

Cheers,
Jack

Re: Bug: org-babel python with :results value sends function definition with a statement after a for loop to the shell incorrectly [9.3.6 (9.3.6-elpa @ /home/username/.emacs.d/elpa/org-9.3.6/)]

[Jack Kamm]

Checking back in on this since Org 9.4 has been released. This issue
should be fixed now, and my tests corroborate this. If you're still
experiencing this problem after updating, please let me know.

Best,
Jack

Jack Kamm  writes:

>> I redid the test but loaded the lastest org-mode and there was no
>> error. This means you probably don't need to debug this. It has been
>> fixed in the latest version, but the fix hasn't been updated in the
>> elpa package yet.  you can either wait for the next release or pull
>> the latest code from the repo.
>
> Thanks for checking this Ian, I believe that this issue would have been
> fixed by this patch:
>
> https://orgmode.org/list/87pnfdo88v@gmail.com/
>
> But I've been too swamped to check.

Re: [PATCH] ob-python: Rename exec tmpfile handle to prevent conflict

[Jack Kamm]

Hi Adrian,

Adrian Kummerländer  writes:

> I noticed that after updating to Org 9.4 many of my Python-based Org
> files fail to execute with various `io.TextIOWrapper' related error
> messages. The reason for this is that opening the exec tmpfile as `f'
> shadows this possibly user-defined variable.
>
> The attached patch fixes this problem for me. As this is my first
> time contributing to Org I am especially open for any suggestions!

The patch looks good.  I've pushed it to master, after making a couple
minor fixes to the commit message (adding a TINYCHANGE cookie and
adjusting some of the spacing).

Thanks for your contribution!

Jack

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

Thanks -- I've pushed this to master now.

Jack

Bastien  writes:

> Hi Jack,
>
> Jack Kamm  writes:
>
>> For now, I think I prefer to keep ob-python leaner, so am going to hold
>> off on this.
>
> The leaner the less maintainance ahead :)
>
>> I'll wait a week or so for comments before merging this new, more
>> limited patch into master.
>
> LGTM, thanks!
>
> -- 
>  Bastien

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

After letting it sit, I'm not sure that my patch above is a good idea
anymore. While it would be useful, it also adds substantial complexity
to ob-python.

For now, I think I prefer to keep ob-python leaner, so am going to hold
off on this.

An alternative approach is to have the user handle graphics and
dataframes via noweb or header arguments.

I've added a couple examples on worg, demonstrating how to use noweb
to insert boilerplate code for handling matplotlib figures and pandas
dataframes [0,1].

Additionally, I'm attaching a small patch to make it easier to handle
graphics/dataframes via the :return header argument, as an alternative
to noweb. The patch includes a couple examples in ORG-NEWS
illustrating this.

I'll wait a week or so for comments before merging this new, more
limited patch into master.

[0] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-python.html
[1] worg commit 59e320ad

Cheers,
Jack

>From 118d8b5eb817e9a21e9d84f2f942fcc841ddc51f Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sat, 19 Sep 2020 08:44:30 -0700
Subject: [PATCH] ob-python: Improvements to :return header argument

* lisp/ob-python.el (org-babel-execute:python): Allow return-val to be
non-nil in sessions, and concatenate it after the expanded body.
---
 etc/ORG-NEWS  | 53 +++
 lisp/ob-python.el | 11 ++
 2 files changed, 60 insertions(+), 4 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 0ed626fb7..50a455ad5 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -11,6 +11,59 @@ See the end of the file for license conditions.
 Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.5 (not yet released)
+** New features
+*** =ob-python= improvements to =:return= header argument 
+
+The =:return= header argument in =ob-python= now works for session
+blocks as well as non-session blocks.  Also, it now works with the
+=:epilogue= header argument -- previously, setting the =:return=
+header would cause the =:epilogue= to be ignored.
+
+This change allows more easily moving boilerplate out of the main code
+block and into the header. For example, for plotting, we need to add
+boilerplate to save the figure to a file and return the
+filename. Instead of doing this within the code block, we can now
+handle it through the header arguments as follows:
+
+#+BEGIN_SRC org
+,#+header: :var fname="/home/jack/tmp/plot.svg"
+,#+header: :epilogue plt.savefig(fname)
+,#+header: :return fname
+,#+begin_src python :results value file
+  import matplotlib, numpy
+  import matplotlib.pyplot as plt
+  fig=plt.figure(figsize=(4,2))
+  x=numpy.linspace(-15,15)
+  plt.plot(numpy.sin(x)/x)
+  fig.tight_layout()
+,#+end_src
+
+,#+RESULTS:
+[[file:/home/jack/tmp/plot.svg]]
+#+END_SRC
+
+As another example, we can use =:return= with the external [[https://pypi.org/project/tabulate/][tabulate]]
+package, to convert pandas Dataframes into orgmode tables:
+
+#+begin_src org
+,#+header: :prologue from tabulate import tabulate
+,#+header: :return tabulate(table, headers=table.columns, tablefmt="orgtbl")
+,#+begin_src python :results value raw :session
+  import pandas as pd
+  table = pd.DataFrame({
+  "a": [1,2,3],
+  "b": [4,5,6]
+  })
+,#+end_src
+
+,#+RESULTS:
+|   | a | b |
+|---+---+---|
+| 0 | 1 | 4 |
+| 1 | 2 | 5 |
+| 2 | 3 | 6 |
+#+end_src
+
 * Version 9.4
 ** Incompatible changes
 *** Possibly broken internal file links: please check and fix
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 00a7c1a2d..785b9191b 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -81,13 +81,16 @@ (defun org-babel-execute:python (body params)
 		   (cdr (assq :session params
  (result-params (cdr (assq :result-params params)))
  (result-type (cdr (assq :result-type params)))
-	 (return-val (when (and (eq result-type 'value) (not session))
+	 (return-val (when (eq result-type 'value)
 		   (cdr (assq :return params
 	 (preamble (cdr (assq :preamble params)))
  (full-body
-	  (org-babel-expand-body:generic
-	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
-	   params (org-babel-variable-assignments:python params)))
+	  (concat
+	   (org-babel-expand-body:generic
+	body params
+	(org-babel-variable-assignments:python params))
+	   (when return-val
+	 (format (if session "\n%s" "\nreturn %s") return-val
  (result (org-babel-python-evaluate
 		  session full-body result-type result-params preamble)))
 (org-babel-reassemble-table
-- 
2.28.0

Re: [PATCH] org-add-planning-info: respect caller's given time [9.3.7 (release_9.3.7-716-g3d4876 @ /home/n/.emacs.d/straight/build/org/)]

[Jack Kamm]

Kyle Meyer  writes:

> With Org's master (f17d301e1), the second one hangs for me with Emacs
> 26.3.  Quickly stepping through org-babel-python--send-string, it
> appears to get stuck in the accept-process-output call.
>
> Using an Emacs built from master, there's no hang on my end either.  If
> you'd find it useful, I can try to bisect that to a specific Emacs
> commit tomorrow.
>
> The Emacs 26.3 hang goes away after installing your patch that restores
> the use of a marker.

Thanks Kyle and Nowayman for checking the patch.

I was able to reproduce the problem on emacs-26.3 (but not 27.1). I'll
be more diligent testing on those versions in future.

I pushed an updated version of my patch to master in 939cf16bc. It
solves the problem for me on emacs-26.3. No need to bisect Emacs (but
thanks for offering!). But do let me know if you encounter any more
problems with this.

Best,
Jack

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

> this is already present in `org-babel-R-evaluate-session' in the call to 
> `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines 
> further down in the `(cl-case result-type (value ...))' branch.
>
> The other use of `tmp-file' in that block is the one that requires the 
> prefix, right?
>
> So nothing more to fix.
>
> ??

You're right, I didn't notice that it's already present -- nothing more
to fix.

Jack

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

"Berry, Charles"  writes:

> Also, I wonder if the `:results value' case can be handled in a similar 
> manner, viz. 
>
> -  (let ((tmp-file (org-babel-temp-file "R-")))
> +  (let ((tmp-file (with-current-buffer session (org-babel-temp-file "R-"

Yes, if we did that then tmp-file would have a prefix like
"/scp:user@hostname:", and elisp would then know to read the result file
from the remote host.

Before pasting tmp-file into R code, we should also call

(org-babel-process-file-name tmp-file 'noquote)

to remove the tramp prefix when referring to the file from R.

Re: [PATCH] org-add-planning-info: respect caller's given time [9.3.7 (release_9.3.7-716-g3d4876 @ /home/n/.emacs.d/straight/build/org/)]

[Jack Kamm]

Kyle Meyer  writes:

> That's on a Debian system with the python executable pointing to Python
> 2.7.16.  If I set org-babel-python-command to python3 (3.7.3) at the top
> of test-ob-python.el, I see the same thing.  I haven't dug any farther
> yet.  Jack, presumably you don't see the stall on your end?

No, the tests don't stall on my end (Archlinux with manually compiled
emacs 28). I also tested on a debian10vm and the tests passed there too.

But since we know it's related to 4df12ea39 that gives some
clues...could you try the attached patch to see if it fixes things?

Also, could you try executing some simple ob-python session blocks and
see if they hang? e.g.,

#+begin_src python :session :results output
  print(1+1)
#+end_src

#+begin_src python :session :results value
  1+1
#+end_src

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 1cded4515..a5af55892 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -223,6 +223,9 @@ (defun org-babel-python-initiate-session ( session _params)
 (org-babel-python-session-buffer
  (org-babel-python-initiate-session-by-key session
 
+(defvar org-babel-python-eoe-indicator "org_babel_python_eoe"
+  "A string to indicate that evaluation has completed.")
+
 (defconst org-babel-python-wrapper-method
   "
 def main():
@@ -324,7 +327,9 @@ (defun org-babel-python--send-string (session body)
 	   (comint-output-filter-functions
 	(cons (lambda (text) (setq string-buffer
    (concat string-buffer text)))
-		  comint-output-filter-functions)))
+		  comint-output-filter-functions))
+	   (body (format "%s\nprint('%s')"
+			 body org-babel-python-eoe-indicator)))
   (if (not (eq 'python-mode org-babel-python-mode))
 	  (let ((python-shell-buffer-name
 		 (org-babel-python-without-earmuffs session)))
@@ -333,13 +338,10 @@ (defun org-babel-python--send-string (session body)
 	(py-shell-send-string body (get-buffer-process session)))
   ;; same as `python-shell-comint-end-of-output-p' in emacs-25.1+
   (while (not (string-match
-		   (concat "\r?\n?"
-			   (replace-regexp-in-string
-			(rx string-start ?^) "" comint-prompt-regexp)
-			   (rx eos))
+		   org-babel-python-eoe-indicator
 		   string-buffer))
 	(accept-process-output (get-buffer-process (current-buffer
-  (substring string-buffer 0 (match-beginning 0)
+  (org-babel-chomp (substring string-buffer 0 (match-beginning 0))
 
 (defun org-babel-python-evaluate-session
 (session body  result-type result-params)

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

> I can confirm that this works on my setup in each of the scenarios in which 
> `default-directory' got set correctly in the session buffer.
>
> At some point, my default-directory got reset to drop the tramp prefix 
> "/scp:/user@host:" in one session after faithfully running the src block 
> several times. I am unable to sort out why or even reproduce this.

Thanks for taking the time to test again. If you have any further
feedback at any point, please don't hesitate.

Best,
Jack

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

> This does not work for my remote session.

Thanks for testing this.

The remote tests I originally tried did not cover this case -- I only
tested the case where both org-file and session were remote, but didn't
test the case where org-file was local and session was remote.

I've now tested this case as well, and added some fixes for it. I'm
reattaching the patch in full.

However, this implementation does require the R session to have the
correct default-directory -- see details below.

> The problem is that tempfiles on the remote host are like 
> "/tmp/RtmpeFHudh/file23a66d2fc1f9", but emacs tries to use 
> '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fhgn/T/babel-OSXKNd/R-oNOVVB'
>
> `org-babel-temp-file' doesn't honor remote connections AFAICS.

org-babel-temp-file does honor remote connections, but relies on
default-directory to do so. I've moved the call to org-babel-temp-file
so it happens when the session buffer is current -- that way, the
tempfile will be correctly handled, assuming that the session's
default-directory is on the remote location.

If the session's default-directory isn't at the remote location, then
this will break. But, in this case some other things break as well, for
example inserting links to remote plots.

> Maybe there is some comint or tramp idiom that would solve this, but I do not 
> know what it is.

Here are some ways to start a remote R session with correct
default-directory:

1. Do "M-x R", then when it prompts for a directory, use
"/scp:hostname:/some/path". Optionally, first visit that location with
"C-x C-f /scp:hostname:/some/path", so that the default value is
already there.

2. Alternatively, start the R session by evaluating a source block with
header argument ":dir /scp:hostname:/some/path".

3. If you prefer to use "M-x shell" with "ess-remote",
first visit the remote location with "C-x C-f /scp:hostname:/some/path", before 
calling "M-x shell".

4. If you prefer to start "M-x shell" locally and then
ssh in, it's still possible to have default-directory set, but it
requires some configuration [1]. Or you could use "M-x cd" to set it as well.

[1] 
https://emacs.stackexchange.com/questions/5589/automatically-update-default-directory-when-pwd-changes-in-shell-mode-and-term-m/5592#5592

Best,
Jack

>From e7f1a59167de88fb9a5b96a0e1ac3199f105f600 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts

* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.

Fixes issue reported in
https://orgmode.org/list/875zgjh8wn@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el  | 38 --
 testing/lisp/test-ob-R.el | 13 +
 2 files changed, 33 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..dffbbe112 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
 (declare-function ess-eval-buffer "ext:ess-inf" (vis))
 (declare-function ess-wait-for-process "ext:ess-inf"
 		  ( proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+		  (process string  visibly message _type))
 
 (defconst org-babel-header-args:R
   '((width		 . :any)
@@ -437,24 +439,24 @@ (defun org-babel-R-evaluate-session
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
 (output
- (mapconcat
-  'org-babel-chomp
-  (butlast
-   (delq nil
-	 (mapcar
-	  (lambda (line) (when (> (length line) 0) line))
-	  (mapcar
-	   (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		  "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		  (car (split-string line "\n")))
-		 (substring line (match-end 1))
-		   line))
-	   (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
-		 (inferior-ess-send-input)) "\n"
+ (with-current-buffer session
+   (let* ((tmp-file (org-babel-temp-file "R-"))
+	  (process (get-buffer-process (current-buffer)))
+	  (string-buffer "")
+	  (comint-output-filter-functions
+	   (cons (lambda (text) (setq string-buffer
+	  (concat string-buffer text)

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

I just realized my patch had an issue where it freezes if there is an
error in the source block.

I'm attaching a second patch, to be applied on top of the first one, that fixes 
the issue.

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index b37e3965a..5ddf0ebd1 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
 (output
  (let* ((tmp-file (org-babel-temp-file "R-")))
(with-temp-file tmp-file
-	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+	 (insert body))
(with-current-buffer session
 	 (let* ((process (get-buffer-process (current-buffer)))
 		(string-buffer "")
@@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
 	(concat string-buffer text)))
 		   comint-output-filter-functions)))
 	   (ess-send-string
-	process (format "source('%s', print.eval=TRUE)"
-			(org-babel-process-file-name tmp-file 'noquote)))
+	process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+			(org-babel-process-file-name tmp-file 'noquote)
+			org-babel-R-eoe-indicator))
 	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
  string-buffer))
 	 (accept-process-output process))

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck and Dylan,

"Berry, Charles"  writes:

> This problem has been bugging people for years and previous attempts to solve 
> it have always run up against creating more problems in the process of 
> solving this one.
>
> If you do decide to dig into solving this, please be sure that remote 
> sessions and graphical outputs are not broken.  test-ob-R.el does not cover 
> those cases.  In fact, it is pretty short, so there are probably other things 
> that could break without `make test' complaining.

I think I have found a robust solution to this issue, which I've
attached.

The solution avoids parsing for prompts at all, thus it should be robust
even for the case when the beginning of an output line looks like a
prompt.

I tested that the unit tests all pass, and that remote sessions and
graphical outputs still work. I also added Dylan's examples as unit
tests.

Cheers,
Jack

>From 76d0eaa31506ce8a2f81f64eae43161db5721317 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts

* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.

Fixes issue reported in
https://orgmode.org/list/875zgjh8wn@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el  | 37 +++--
 testing/lisp/test-ob-R.el | 13 +
 2 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..b37e3965a 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
 (declare-function ess-eval-buffer "ext:ess-inf" (vis))
 (declare-function ess-wait-for-process "ext:ess-inf"
 		  ( proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+		  (process string  visibly message _type))
 
 (defconst org-babel-header-args:R
   '((width		 . :any)
@@ -437,24 +439,23 @@ (defun org-babel-R-evaluate-session
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
 (output
- (mapconcat
-  'org-babel-chomp
-  (butlast
-   (delq nil
-	 (mapcar
-	  (lambda (line) (when (> (length line) 0) line))
-	  (mapcar
-	   (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		  "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		  (car (split-string line "\n")))
-		 (substring line (match-end 1))
-		   line))
-	   (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
-		 (inferior-ess-send-input)) "\n"
+ (let* ((tmp-file (org-babel-temp-file "R-")))
+   (with-temp-file tmp-file
+	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+   (with-current-buffer session
+	 (let* ((process (get-buffer-process (current-buffer)))
+		(string-buffer "")
+		(comint-output-filter-functions
+		 (cons (lambda (text) (setq string-buffer
+	(concat string-buffer text)))
+		   comint-output-filter-functions)))
+	   (ess-send-string
+	process (format "source('%s', print.eval=TRUE)"
+			(org-babel-process-file-name tmp-file 'noquote)))
+	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+ string-buffer))
+	 (accept-process-output process))
+	   (substring string-buffer 0 (match-beginning 0
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el
index 7ce340ba4..ff7ea19d5 100644
--- a/testing/lisp/test-ob-R.el
+++ b/testing/lisp/test-ob-R.el
@@ -97,6 +97,19 @@ (ert-deftest test-ob-R/results-file ()
  (org-babel-goto-named-result "TESTSRC") (forward-line 1)
  (should (string= "[[file:junk/test.org]]"
 		  (buffer-substring-no-properties (point-at-bol) (point-at-eol)))
+
+(ert-deftest test-ob-R/prompt-output ()
+  (let (ess-ask-for-ess-directory ess-history-file)
+(org-test-with-temp-text
+ "#+begin_src R :results output :session\nprint(\"  \")\nprint(\"one  three\")\nprint(\"end\")\n#+end_src\n"
+ (should (string= "[1] \"  \"\n[1] \"one  three\"\n[1] \"end\"\n" (org-babel-execute-src-block))
+
+(ert-deftest test-ob-R/output-nonprinted ()
+  (l

Re: [PATCH] ob-python.el: Fix issue with sessions on remote machines

[Jack Kamm]

Hi Bastien,

> - A local maintainer is expected to reply to requests and bug reports
>   regarding the local functionalities he oversees.
>
> - A local maintainer can commit changes directly to the file(s) he
>   maintains (either submitted changes or his own).
>
> - Core maintainers have the final word on any change in any file (so
>   in case of a disagreement with a local maintainer, core maintainers
>   have priority.)
>
> In general, I would like to encourage "optimistic merging" from more
> "local" maintainers.
>
> Does that sound right?  When in doubt, always discuss changes first.

I agree with this policy, and am more or less following it already,
though it's good to be explicit.

I'm keeping an eye out for Python-related mail, but if it looks like
I've missed one, add me to the TO or CC field to ensure it lands in my
inbox.

I appreciate the freedom to make changes to ob-python, but of course
defer to you, Nicolas, and Kyle, and seek out guidance when unsure.

Cheers,
Jack

Re: [PATCH] ob-python.el: Fix issue with sessions on remote machines

[Jack Kamm]

Hi Bastien,

> Would you be okay to add yourself as the ob-python.el maintainer?

Sure, I've added myself as maintainer to the header of ob-python.el.

> I suggest we have a policy that "Org maintainer(s)" have the last
> words on everything in Org's core, but that individual maintainers,
> when known from the header section of an Elisp file, have the very
> "first look" on bug reports and feature suggestions.
>
> WDYT?

I'm trying to review ob-python related patches and mail as I notice
them, and monitor the list for mails with "python" in the subject,
though some may fall through the cracks occasionally, especially when my
workload is heavy.

Should I merge in patches to ob-python.el, as I did here? Or should I
simply review them, and let the core maintainers merge them in after
review?

Jack

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

After taking another look at my patch, I realized that I was not quite
converting dictionaries to proper alists.

Attached is a tweak to do this properly. The printing of dictionaries is
not quite as pretty, in particular it's not a table anymore:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
: ((a . 1) (b . 2))

But, it feels like the right thing to do, since the result handling code
works by converting the result to an elisp value, before passing it to
org-mode to decide how to render it. And the proper elisp conversion of
a dict should be an alist or a plist.

Ideally I wouldn't have to do this from the Python code, and could let
org-babel-script-escape convert the dict objects. It would also be
useful for other languages with similar dictionaries, like
javascript. But it seems fairly complex to implement this from the elisp
side, and I'm not sure I'm up for it.

I also noticed that I had left a couple docstrings as TODOs -- I'll fix
those before finalizing the patch over the next couple weeks.

>From 76a1ad4d50e6638244d9aa17e45895b8b38b3cd0 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sun, 30 Aug 2020 08:51:04 -0700
Subject: [PATCH 2/2] Convert dictionary output to a proper alist

Note: to be squashed with the previous patch before merging
---
 lisp/ob-python.el | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index fb8fe380e..08c1c48e9 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -239,9 +239,14 @@ (defconst org-babel-python--def-format-value "\
 else:
 if not set(result_params).intersection(\
 ['scalar', 'verbatim', 'raw']):
+class alist(dict):
+def __str__(self):
+return '({})'.format(' '.join(['({} . {})'.format(repr(k), repr(v)) for k, v in self.items()]))
+def __repr__(self):
+return self.__str__()
 def dict2alist(res):
 if isinstance(res, dict):
-return [(k, dict2alist(v)) for k, v in res.items()]
+return alist({k: dict2alist(v) for k, v in res.items()})
 elif isinstance(res, list) or isinstance(res, tuple):
 return [dict2alist(x) for x in res]
 else:
-- 
2.28.0

Re: [PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

Hi Kyle,

Thanks for the comments, I'm attaching an updated patch.

Kyle Meyer  writes:

> ModuleNotFoundError wasn't added until Python 3.6, so I think it'd be
> better to use its parent class, ImportError.

I did not know this, thanks for the tip.

> Should handling of Series also be added?

Yes, I've done so now. I'm not sure whether it's better to treat it like
a row or column vector, but since it has an "index", which are the row
names in a DataFrame, I decided to treat it as a column.

>From 40db6b5497de78a9e69de219f4686b405db10c81 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Tue, 25 Aug 2020 21:57:24 -0700
Subject: [PATCH] ob-python: Add results handling for dicts, dataframes,
 arrays, plots

* lisp/ob-python.el (org-babel-execute:python): Parse graphics-file
from params.
(org-babel-python--def-format-value): Python code for formatting
value results before returning.
(org-babel-python--output-graphics-wrapper): Python code for handling
output graphics results.
(org-babel-python--nonsession-value-wrapper): Replaces
org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method.
(org-babel-python--session-output-wrapper): Renamed from
org-babel-python--exec-tmpfile.
(org-babel-python--session-value-wrapper): Renamed and modified from
org-babel-python--eval-ast.
(org-babel-python-evaluate-external-process): New parameter for
graphics file.
(org-babel-python-evaluate-session): New parameter for graphics file.

Added results handling for dictionaries, Pandas and numpy tables, and
matplotlib plots.
---
 etc/ORG-NEWS  |  17 ++-
 lisp/ob-python.el | 126 +++---
 2 files changed, 100 insertions(+), 43 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 10658a970..75c945572 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -66,8 +66,8 @@ to switch to the new signature.
 *** Python session return values must be top-level expression statements
 
 Python blocks with ~:session :results value~ header arguments now only
-return a value if the last line is a top-level expression statement.
-Also, when a None value is returned, "None" will be printed under
+return a value if the last line is a top-level expression statement,
+otherwise the result is None. Also, None will now show up under
 "#+RESULTS:", as it already did with ~:results value~ for non-session
 blocks.
 
@@ -235,6 +235,19 @@ Screen blocks now recognize the =:screenrc= header argument and pass
 its value to the screen command via the "-c" option. The default
 remains =/dev/null= (i.e. a clean screen session)
 
+*** =ob-python.el=: Support for more result types and plotting
+
+=ob-python= now recognizes dictionaries, numpy arrays, and pandas
+dataframes/series, and will convert them to org-mode tables when
+appropriate.
+
+When the header argument =:results graphics= is set, =ob-python= will
+use matplotlib to save graphics. The behavior depends on whether value
+or output results are used. For value results, the last line should
+return a matplotlib Figure object to plot. For output results, the
+current figure (as returned by =pyplot.gcf()=) is cleared before
+evaluation, and then plotted afterwards.
+
 *** =RET= and =C-j= now obey ~electric-indent-mode~
 
 Since Emacs 24.4, ~electric-indent-mode~ is enabled by default.  In
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 44e1b63e0..fb8fe380e 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params)
 	  org-babel-python-command))
 	 (session (org-babel-python-initiate-session
 		   (cdr (assq :session params
+	 (graphics-file (and (member "graphics" (assq :result-params params))
+			 (org-babel-graphical-output-file params)))
  (result-params (cdr (assq :result-params params)))
  (result-type (cdr (assq :result-type params)))
 	 (return-val (when (and (eq result-type 'value) (not session))
@@ -89,7 +91,8 @@ (defun org-babel-execute:python (body params)
 	   (concat body (if return-val (format "\nreturn %s" return-val) ""))
 	   params (org-babel-variable-assignments:python params)))
  (result (org-babel-python-evaluate
-		  session full-body result-type result-params preamble)))
+		  session full-body result-type result-params preamble
+		  graphics-file)))
 (org-babel-reassemble-table
  result
  (org-babel-pick-name (cdr (assq :colname-names params))
@@ -225,67 +228,104 @@ (defun org-babel-python-initiate-session ( session _params)
 (org-babel-python-session-buffer
  (org-babel-python-initiate-session-by-key session
 
-(defconst org-babel-python-wrapper-method
-  "
-def main():
+(defconst org-babel-python--def-format-value "\
+def __org_babel_python_format_value(result, result_file, result_params):
+with open(result_file, 'w') as f:
+if 'graphics' in result_params:
+result.savefig(resul

[PATCH] Expanded ob-python results handling and plotting

[Jack Kamm]

The attached patch adds ob-python value results handling for the
following types of results:

- Dictionaries
- Numpy arrays
- Pandas dataframes
- Matplotlib figures

This is a bigger commit than I'm used to, so I thought I better send it
out before merging, in case someone notices obvious problems I missed.

Overview of changes:

Dictionaries are now transformed into alists before being converted to
lisp. Previously, they had been getting mangled, like so:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
| a | : | 1 | b | : | 2 |

But now they appear like so:

#+begin_src python
  return {"a": 1, "b": 2}
#+end_src

#+RESULTS:
| a | 1 |
| b | 2 |

Numpy arrays and pandas dataframes are also converted to tables
automatically now. Tables converted from Pandas dataframes have row and
column names.

To avoid conversion, you can specify "raw", "verbatim", "scalar", or
"output" in the ":results" header argument.

For plotting, you can specify "graphics" in the ":results"
header. You'll also need to provide a ":file" argument. The behavior
depends on whether using output or value results. For output results,
the current figure (pyplot.gcf) is cleared before evaluating, then the
result saved. For value results, the block is expected to return a
matplotlib Figure, which is saved. To set the figure size, do it from
within Python.

Here is an example of how to plot:

#+begin_src python :results output graphics file :file boxplot.svg
  import matplotlib.pyplot as plt
  import seaborn as sns
  plt.figure(figsize=(5, 5))
  tips = sns.load_dataset("tips")
  sns.boxplot(x="day", y="tip", data=tips)
#+end_src

>From 09f9c42bb629a356e1c36f04f69c8baf795b411b Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Tue, 25 Aug 2020 21:57:24 -0700
Subject: [PATCH] ob-python: Add results handling for dicts, dataframes,
 arrays, plots

* lisp/ob-python.el (org-babel-execute:python): Parse graphics-file
from params.
(org-babel-python--def-format-value): Python code for formatting
value results before returning.
(org-babel-python--output-graphics-wrapper): Python code for handling
output graphics results.
(org-babel-python--nonsession-value-wrapper): Replaces
org-babel-python-wrapper-method, org-babel-python-pp-wrapper-method.
(org-babel-python--session-output-wrapper): Renamed from
org-babel-python--exec-tmpfile.
(org-babel-python--session-value-wrapper): Renamed and modified from
org-babel-python--eval-ast.
(org-babel-python-evaluate-external-process): New parameter for
graphics file.
(org-babel-python-evaluate-session): New parameter for graphics file.

Added results handling for dictionaries, Pandas and numpy tables, and
matplotlib plots.
---
 etc/ORG-NEWS  |  16 +-
 lisp/ob-python.el | 122 +++---
 2 files changed, 96 insertions(+), 42 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 10658a970..4f9863a5b 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -66,8 +66,8 @@ to switch to the new signature.
 *** Python session return values must be top-level expression statements
 
 Python blocks with ~:session :results value~ header arguments now only
-return a value if the last line is a top-level expression statement.
-Also, when a None value is returned, "None" will be printed under
+return a value if the last line is a top-level expression statement,
+otherwise the result is None. Also, None will now show up under
 "#+RESULTS:", as it already did with ~:results value~ for non-session
 blocks.
 
@@ -235,6 +235,18 @@ Screen blocks now recognize the =:screenrc= header argument and pass
 its value to the screen command via the "-c" option. The default
 remains =/dev/null= (i.e. a clean screen session)
 
+*** =ob-python.el=: Support for more result types and plotting
+
+=ob-python= now recognizes dictionaries, numpy arrays, and pandas
+dataframes, and will convert them to org-mode tables when appropriate.
+
+When the header argument =:results graphic= is set, =ob-python= will
+use matplotlib to save graphics. The behavior depends on whether value
+or output results are used. For value results, the last line should
+return a matplotlib Figure object to plot. For output results, the
+current figure (as returned by =pyplot.gcf()=) is cleared and then
+plotted.
+
 *** =RET= and =C-j= now obey ~electric-indent-mode~
 
 Since Emacs 24.4, ~electric-indent-mode~ is enabled by default.  In
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 44e1b63e0..92ca82625 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -79,6 +79,8 @@ (defun org-babel-execute:python (body params)
 	  org-babel-python-command))
 	 (session (org-babel-python-initiate-session
 		   (cdr (assq :session params
+	 (graphics-file (and (member "graphics" (assq :result-params params))
+			 (org

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Dylan,

> The patch does fix that issue -- but it introduces a different bug 
> for code blocks with ~:session~: the R block now only produces 
> output from the last statement evaluated.

Of course, you're right. Good catch.

Here's another attempt. It fixes the issue by modifying the R comint
regular expression, requiring it to match at the beginning of the line.

I think this should fix most cases, including the examples you
sent. Still, it's not totally robust -- for example, it will still
mangle multiline strings, if one of the lines starts with a substring
that looks like a prompt.

I'd be interested to hear if the attached patch works for the common
cases you encounter, such as with tibbles.

As an aside, I personally use an alternative implementation of ob-R
sessions that doesn't suffer from this issue, and also provides some
other benefits like async evaluation [1]. I'm planning to submit these
changes to org-mode someday, but am not ready yet. But you may want to
check it out, it was able to solve the issue in the other thread I
linked as well.

[1] https://github.com/jackkamm/ob-session-async

>From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sat, 29 Aug 2020 00:07:58 -0700
Subject: [PATCH] ob-R: Fix prompt mangling in session output

* lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt
regexp to start at beginning of line, to prevent
org-babel-comint-with-output from splitting mid-line.

Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..10b3b6fe3 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session
 		  (car (split-string line "\n")))
 		 (substring line (match-end 1))
 		   line))
-	   (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
-		 (inferior-ess-send-input)) "\n"
+	   (with-current-buffer session
+		 (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp)))
+		   (org-babel-comint-with-output (session org-babel-R-eoe-output)
+		 (insert (mapconcat 'org-babel-chomp
+	(list body org-babel-R-eoe-indicator)
+	"\n"))
+		 (inferior-ess-send-input "\n"
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.28.0

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Dylan,

I'm able to reproduce your error. I think this same bug has been
previously reported at [1]. As discussed there, the reason is that
substrings ending in ">" get stripped out by
org-babel-comint-with-output because it thinks they are prompts (in
particular, they match comint-prompt-regexp).

I'm attaching a patch to solve the issue. It replaces
org-babel-comint-with-output, with an R call to capture.output. I think
this approach is simpler and more robust, since it doesn't need to do
any parsing to remove prompts or other unwanted outputs.

I haven't committed to ob-R.el before, so thought it would be best to
solicit feedback here before merging this in.

Cheers,
Jack

[1] https://orgmode.org/list/875zgjh8wn@gmail.com/

>From 1dc8e2d2cb01a4e6b82342ea8d8c965df8f5222c Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Fri, 28 Aug 2020 19:16:05 -0700
Subject: [PATCH] ob-R: Fix prompt mangling in session output

* lisp/ob-R.el (org-babel-R-evaluate-session): Replace
call to org-babel-comint-with-output with ess-send-to-buffer and
additional wrapper R code.

Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el | 79 
 1 file changed, 36 insertions(+), 43 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..d69cf23db 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -412,49 +412,42 @@ (defun org-babel-R-evaluate-session
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
 last statement in BODY, as elisp."
-  (cl-case result-type
-(value
- (with-temp-buffer
-   (insert (org-babel-chomp body))
-   (let ((ess-local-process-name
-	  (process-name (get-buffer-process session)))
-	 (ess-eval-visibly-p nil))
-	 (ess-eval-buffer nil)))
- (let ((tmp-file (org-babel-temp-file "R-")))
-   (org-babel-comint-eval-invisibly-and-wait-for-file
-	session tmp-file
-	(format org-babel-R-write-object-command
-		(if row-names-p "TRUE" "FALSE")
-		(if column-names-p
-		(if row-names-p "NA" "TRUE")
-		  "FALSE")
-		".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
-   (org-babel-R-process-value-result
-	(org-babel-result-cond result-params
-	  (with-temp-buffer
-	(insert-file-contents tmp-file)
-	(org-babel-chomp (buffer-string) "\n"))
-	  (org-babel-import-elisp-from-file tmp-file '(16)))
-	column-names-p)))
-(output
- (mapconcat
-  'org-babel-chomp
-  (butlast
-   (delq nil
-	 (mapcar
-	  (lambda (line) (when (> (length line) 0) line))
-	  (mapcar
-	   (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		  "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		  (car (split-string line "\n")))
-		 (substring line (match-end 1))
-		   line))
-	   (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
-		 (inferior-ess-send-input)) "\n"
+  (let ((ess-local-process-name
+	 (process-name (get-buffer-process session)))
+	(ess-eval-visibly-p nil))
+(cl-case result-type
+  (value
+   (with-temp-buffer
+	 (insert (org-babel-chomp body))
+	 (ess-eval-buffer nil))
+   (let ((tmp-file (org-babel-temp-file "R-")))
+	 (org-babel-comint-eval-invisibly-and-wait-for-file
+	  session tmp-file
+	  (format org-babel-R-write-object-command
+		  (if row-names-p "TRUE" "FALSE")
+		  (if column-names-p
+		  (if row-names-p "NA" "TRUE")
+		"FALSE")
+		  ".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
+	 (org-babel-R-process-value-result
+	  (org-babel-result-cond result-params
+	(with-temp-buffer
+	  (insert-file-contents tmp-file)
+	  (org-babel-chomp (buffer-string) "\n"))
+	(org-babel-import-elisp-from-file tmp-file '(16)))
+	  column-names-p)))
+  (output
+   (let ((tmp-file (org-babel-temp-file "R-")))
+	 (with-temp-buffer
+	   (insert (format "capture.output({
+%s
+}, file='%s')"
+			   body tmp-file))
+	   (ess-eval-buffer))
+	 (ess-wait-for-process)
+	 (with-temp-buffer
+	   (insert-file-contents tmp-file)
+	   (buffer-string)))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.28.0

Re: save plot in python babel

[Jack Kamm]

Hi Jeremie,

> I tried to look around and noticed that the documentation regarding
> python with org-mode is aging a bit...  Any help will be
> appreciated. I'll be ready to improve the documentation of org-babel
> python setup as soon as I'm more familiar with it myself.

Historically ob-python has been a bit broken, especially
sessions. Earlier this year, I started taking care of ob-python and
implemented some fixes [1], which will land in 9.4. When that's
released, I plan to update the outdated worg documentation [2].

> I've barely use python with org-mode and I would need some help with
> the python setup. I somehow have some trouble saving python plots. Is
> this the right way to do it?

This part of the documentation [2], at the bottom of the page, is still
up-to-date. Basically, you need to manually call plt.savefig(), then
return the filename.

> In R things seems to be smoother.

Yes, ob-R is in much better shape. In addition to plotting, returning
dataframes works (pandas DataFrames are not yet supported in
ob-python). I also much prefer the way ob-R returns lists as a column
vector instead of a row vector.

> there are ob-* all over the place. [1] ob-ipython does not seem to be
> maintained

For serious work these days, I'm using ob-jupyter [3], which works
really well. Scimax [4] also has a maintained fork of ob-ipython, though
I haven't tried it.

My goal is to make ob-python a serious alternative to these external
ob-* packages eventually. Here are some of the improvements I have in
mind:

- Improved plotting
- Support for pandas DataFrames and numpy Arrays
- Async session evaluation [5]

I've been delayed working on this as I've gotten very busy due to the
pandemic (I am a bioinformaticist working on SARS-CoV-2 sequencing). But
I'm hoping to make a bit of progress this week, as I'm on vacation and
unable to go outside due to the smoke from the California fires.

It's unclear to me if we're in a feature freeze, so I'm currently
pushing ob-python development to a separate branch [6], which I'll
cherry pick into master after the Org 9.4 release.

[1] https://orgmode.org/list/87pnfdo88v@gmail.com/
[2] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-python.html
[3] https://github.com/nnicandro/emacs-jupyter
[4] https://github.com/jkitchin/scimax
[5] https://github.com/jackkamm/ob-session-async
[6] https://code.orgmode.org/jackkamm/org-mode/src/python-develop

[RFC] ob-reticulate: R+Python interface from Babel

[Jack Kamm]

Hi all,

Reticulate is an R package for interfacing between R and Python. It allows 
accessing objects in a Python session from R and vice versa. See 
https://rstudio.github.io/reticulate/ for more info about it.

I've written a small patch for using reticulate from org-babel. It allows 
creating a source block of lang "reticulate", which behaves as Python for font 
highlighting and editing, but is executed in an R session via reticulate.

I'm wondering whether this should go into org-mode, or whether to package this 
separately. I'm also curious whether this would be useful to anyone here. Any 
feedback is appreciated.

The main advantage of reticulate is being able to access Python objects 
directly from R and vice versa, without having to write them to a separate file 
or pass them through the ":var" header argument. For example, we could do the 
following:

#+begin_src reticulate :session
  import pandas as pd

  fib = [0, 1]
  for _ in range(10):
  fib.append(fib[-1] + fib[-2])

  df = pd.DataFrame({
  "i": list(range(len(fib))),
  "F_i": fib
  })
#+end_src

#+begin_src R :session :results graphics value file :file fig.png
  library(reticulate)
  with(py$df, plot(i, F_i))
#+end_src

Reticulate source blocks support both "value" and "output" results, and even 
supports graphics with matplotlib. It's primarily intended to be used in 
sessions, and the ":session" header argument should match between reticulate 
and R source blocks.

Cheers,
Jack

>From 0f691a200cf088c72f93f7552d73caeafb8d588f Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Mon, 24 Aug 2020 08:02:17 -0700
Subject: [PATCH] ob-reticulate: Babel source lang for R+Python reticulate
 package

* lisp/ob-reticulate.el: New babel source block lang for
R's reticulate package for evaluating Python code.
* lisp/org-src.el (org-src-lang-modes): Add reticulate.
---
 lisp/ob-reticulate.el | 50 +++
 lisp/org-src.el   |  1 +
 2 files changed, 51 insertions(+)
 create mode 100644 lisp/ob-reticulate.el

diff --git a/lisp/ob-reticulate.el b/lisp/ob-reticulate.el
new file mode 100644
index 0..7da48681c
--- /dev/null
+++ b/lisp/ob-reticulate.el
@@ -0,0 +1,50 @@
+;;; ob-reticulate.el --- Babel Functions for reticulate -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2020 Free Software Foundation, Inc.
+
+;; Author: Jack Kamm
+;; Keywords: literate programming, reproducible research, R, statistics
+;; Homepage: https://orgmode.org
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Org-Babel support for the R package reticulate.
+
+;;; Code:
+
+(require 'ob-R)
+(require 'ob-python)
+
+(defalias 'org-babel-edit-prep:reticulate 'org-babel-edit-prep:R)
+
+(defun org-babel-execute:reticulate (body params)
+  (let* ((tmp-src-file (org-babel-temp-file "reticulate-"))
+	 (result-type (cdr (assq :result-type params
+(with-temp-file tmp-src-file (insert body))
+(org-babel-execute:R
+ (format (concat "reticulate::py_run_string(\"%s\")"
+		 (when (equal result-type 'value) "
+reticulate::py$`__org_babel_python_final`"))
+	 (format org-babel-python--eval-ast
+		 (org-babel-process-file-name
+		  tmp-src-file 'noquote)))
+ params)))
+
+(provide 'ob-reticulate)
+
+;;; ob-reticulate.el ends here
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 28733d011..1b3d83f87 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -197,6 +197,7 @@ (defcustom org-src-lang-modes
 ("dot" . fundamental)
 ("elisp" . emacs-lisp)
 ("ocaml" . tuareg)
+("reticulate" . python)
 ("screen" . shell-script)
 ("shell" . sh)
 ("sqlite" . sql))
-- 
2.28.0

Re: [PATCH] ob-python.el: Fix issue with sessions on remote machines

[Jack Kamm]

Merged now -- thanks for your contribution.

Cheers,
Jack

Re: [PATCH] ob-python.el: Fix issue with sessions on remote machines

[Jack Kamm]

Hi Christian,

Thanks for reporting this and providing a fix!

> The function `org-babel-python-evaluate-session' doesn't process temp
> file names with `org-babel-process-file-name' before inserting them into
> the Python code blocks. This causes a 'No such file' error when the
> executing the code blocks on a remote directory.

I tested this out and was able to reproduce the bug.

> The attached patch fixes this issue, allowing compilation of Python
> source blocks with a remote directory, such as :dir /ssh:user@server:/.

I tested your patch and it solves the problem.

> This is my first patch ever, so please let me know if there's ways I can
> improve.

The patch is pretty simple and I see no problems with it. Since it is so
small, FSF copyright assignment shouldn't be necessary. Also, it looks
like you followed all the guidelines on
https://orgmode.org/worg/org-contribute.html.

I'm fairly novice myself so will wait a couple days in case one of the
more experienced folks notices any issues here. But I'll merge this to
master by end of week if I don't hear anything.

Re: Bug: org-babel python with :results value sends function definition with a statement after a for loop to the shell incorrectly [9.3.6 (9.3.6-elpa @ /home/username/.emacs.d/elpa/org-9.3.6/)]

[Jack Kamm]

> I redid the test but loaded the lastest org-mode and there was no
> error. This means you probably don't need to debug this. It has been
> fixed in the latest version, but the fix hasn't been updated in the
> elpa package yet.  you can either wait for the next release or pull
> the latest code from the repo.

Thanks for checking this Ian, I believe that this issue would have been
fixed by this patch:

https://orgmode.org/list/87pnfdo88v@gmail.com/

But I've been too swamped to check.

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

Hi Kyle,

Thanks for the code review which was very helpful as always. I've fixed
the style and compile errors that you noticed, and pushed the commit to
master.

Cheers,
Jack

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

Hi,

Here's my second attempt to fix the issue. I'll wait a few days for
comments before merging, to try and avoid causing a bug like last time.

It turns out the problem was deeper than just the :var argument,
multiline strings in the body were also getting mangled. For example,
this block had the same problem as the original report:

#+begin_src python
  text = """a
  b
  c
  """
  return text
#+end_src

#+RESULTS:
: a
:   b
:   c

My fix this time was to use functions from python.el to indent, and to
detect whether we are in a string and shouldn't be indented.

I also added a couple more unit tests, one for multiline strings, and
one for the variable scope/assignment issue that Matt reported.

Cheers,
Jack

>From 179178d39f6216172e1a070f570cf941f99b1a89 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Sat, 6 Jun 2020 10:59:23 -0700
Subject: [PATCH] ob-python.el: Fix multiline strings in non-session :results
 value

* lisp/ob-python.el (org-babel-python-evaluate-external-process): Use
functions from python.el to indent lines, avoiding multiline strings.
* testing/lisp/test-ob-python.el (test-ob-python/multiline-var): Set
test as expected to succeed.
(test-ob-python/multiline-str): Add test for multiline string in body.
(test-ob-python/header-var-assignment): Test that :var is in correct
scope and can be assigned to.

cf. https://orgmode.org/list/87tv009l9a@gmail.com/#t
---
 lisp/ob-python.el  | 14 +-
 testing/lisp/test-ob-python.el | 20 +++-
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index dbcfac08d..622f69ce3 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -296,11 +296,15 @@ (defun org-babel-python-evaluate-external-process
 			(if (member "pp" result-params)
 			org-babel-python-pp-wrapper-method
 			  org-babel-python-wrapper-method)
-			(mapconcat
-			 (lambda (line) (format "\t%s" line))
-			 (split-string (org-remove-indentation (org-trim body))
-   "[\r\n]")
-			 "\n")
+			(with-temp-buffer
+			 (insert body)
+			 (goto-char (point-min))
+			 (while (< (point) (point-max))
+			   (unless (python-syntax-context 'string)
+			 (python-indent-shift-right (line-beginning-position)
+			(line-end-position)))
+			   (forward-line 1))
+			 (buffer-string))
 			(org-babel-process-file-name tmp-file 'noquote
 		 (org-babel-eval-read-file tmp-file))
 (org-babel-result-cond result-params
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index 763942b16..e3f0571a1 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -174,7 +174,6 @@ (ert-deftest test-ob-python/assign-underscore ()
 	(org-babel-execute-src-block)
 
 (ert-deftest test-ob-python/multiline-var ()
-  :expected-result :failed
   (should
(equal "a\nb\nc"
 	  (org-test-with-temp-text "#+begin_src python :var text=\"a\\nb\\nc\"
@@ -182,6 +181,25 @@ (ert-deftest test-ob-python/multiline-var ()
 #+end_src"
 	(org-babel-execute-src-block)
 
+(ert-deftest test-ob-python/multiline-str ()
+  (should
+   (equal "a\nb\nc"
+	  (org-test-with-temp-text "#+begin_src python
+text=\"a\\nb\\nc\"
+return text
+#+end_src"
+	(org-babel-execute-src-block)
+
+(ert-deftest test-ob-python/header-var-assignment ()
+  (should
+   (equal "success"
+	  (org-test-with-temp-text "#+begin_src python :var text=\"failure\"
+text
+text=\"success\"
+return text
+#+end_src"
+	(org-babel-execute-src-block)
+
 (provide 'test-ob-python)
 
 ;;; test-ob-python.el ends here
-- 
2.27.0

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

An update on this -- I decided to revert my fix for multiline Python
variables. I left the unit test I added for this, but marked it as
expected to fail.

I'll try to submit a proper fix for this soon. However I've been swamped
at work so I can't promise when (hopefully a couple weeks). In the
meantime, I think it better to leave the original bug in place, rather
than break any existing ob-python use cases.

When I do get to this, I'll submit it as a patch to the list first, to
make sure I don't accidentally cause a new bug.

Best,
Jack

Jack Kamm  writes:

> Hi Matt --
>
>> A heads up... I believe this changes the scope of the :var variables,
>> since they previously were local to the main() function and now they are
>> declared globally. After this change, some of my existing python source
>> blocks (i.e., ones in which I attempt to assign a new value to a
>> variable defined by :var) now generate the following error:
>>
>> Traceback (most recent call last):
>>   File "", line 223, in 
>>   File "", line 214, in main
>> UnboundLocalError: local variable 'members' referenced before assignment
>
> Thanks for noticing this and pointing it out. This was an oversight on
> my end (I don't really use ":var" or non-session blocks).
>
> Unfortunately, the fix for the original bug will have to be a bit more
> complicated to avoid this error. I'll put it on my todo list, but if
> anyone wants to have a crack at it, that would be very welcome.
>
> We should also add a unit test for this regression so it doesn't happen
> again.
>
>> I hesitate to call this a bug, since it would be fine to think of
>> everything within the source block as local and the header :var
>> declarations as global.
>
> I think it's fair to call this a bug. I think it would be inconvenient
> to be unable to assign to these variables. Also, I did not intend to
> break any existing code with this.
>
>> And I suppose it would be worthwhile to ask: Is this change consistent
>> with other org-babel modules? Is there a canonical way that org-babel
>> handles scope?
>
> IMO everything ought to be in the same scope, and the user shouldn't
> have to think about the scope of the main() function that ob-python
> wraps the body in. Especially since non-session blocks are evaluated
> independently of other blocks, we really shouldn't have to think about
> their scope.
>
> I'm not sure if the Org manual provides any guidance here, but from a
> quick check I didn't see anything about it.
>
> Ideally I think everything should just be in global scope; however
> ob-python needs to use a wrapper function for the "return" keyword.

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

Hi Matt --

> A heads up... I believe this changes the scope of the :var variables,
> since they previously were local to the main() function and now they are
> declared globally. After this change, some of my existing python source
> blocks (i.e., ones in which I attempt to assign a new value to a
> variable defined by :var) now generate the following error:
>
> Traceback (most recent call last):
>   File "", line 223, in 
>   File "", line 214, in main
> UnboundLocalError: local variable 'members' referenced before assignment

Thanks for noticing this and pointing it out. This was an oversight on
my end (I don't really use ":var" or non-session blocks).

Unfortunately, the fix for the original bug will have to be a bit more
complicated to avoid this error. I'll put it on my todo list, but if
anyone wants to have a crack at it, that would be very welcome.

We should also add a unit test for this regression so it doesn't happen
again.

> I hesitate to call this a bug, since it would be fine to think of
> everything within the source block as local and the header :var
> declarations as global.

I think it's fair to call this a bug. I think it would be inconvenient
to be unable to assign to these variables. Also, I did not intend to
break any existing code with this.

> And I suppose it would be worthwhile to ask: Is this change consistent
> with other org-babel modules? Is there a canonical way that org-babel
> handles scope?

IMO everything ought to be in the same scope, and the user shouldn't
have to think about the scope of the main() function that ob-python
wraps the body in. Especially since non-session blocks are evaluated
independently of other blocks, we really shouldn't have to think about
their scope.

I'm not sure if the Org manual provides any guidance here, but from a
quick check I didn't see anything about it.

Ideally I think everything should just be in global scope; however
ob-python needs to use a wrapper function for the "return" keyword.

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

Hello,

Thanks for reporting. I've just fixed this issue in master (commit
6149b6cb6).

The problem was that ob-python adds tab indentation to the code body
before putting it inside a main() function, which adds spurious
indentation to multiline strings passed through :var.

I fixed the issue by moving variable assignment from the code body to
the code preamble, which is executed outside the main() function.

Best,
Jack

Štěpán Němec  writes:

> Recipe:
> ---
>
> emacs -Q
> M-x load-library RET ob-python RET
> M-x org-mode RET
>
> #+begin_src python :var text="a\nb\nc"
> return text
> #+end_src
>
> #+RESULTS:
> : a
> : b
> : c
>
>
> Commentary:
> ---
>
> ob-python seems to prepend a TAB character to every line except for the
> first one.
>
> Emacs  : GNU Emacs 28.0.50 (build 11, x86_64-pc-linux-gnu, GTK+ Version 
> 3.24.14, cairo version 1.17.3)
>  of 2020-03-11
> Package: Org mode version 9.3.6 (release_9.3.6-397-ga08960 @ 
> /home/stepnem/.emacs.d/lib/org/lisp/)

Re: Improving Org Mode for VSCode - Thinking Aloud

[Jack Kamm]

It would be very good indeed for org-babel if it could be ported to
other editors. One of the biggest drawbacks of org-babel notebooks is
that I can't collaborate with my colleagues on them, since I can't
expect them to use Emacs.

Aside from VSCode, I think RStudio would be an excellent target for a
few reasons:

- Literate programming is already popular among R users (see also:
  knitr, sweave, Rmarkdown)
- There is a strong ob-R community here
- There are some prominent Emacs users among the Rstudio developers
  (e.g. Lionel Henry, who I think is both an Rstudio and ESS developer)

However, this would be a massive undertaking, and ultimately would need
a volunteer to step up to the plate. I don't have any bandwidth to do
this in the foreseeable future but dream of working on it one day. The
biggest downside -- it would require spending considerable time outside
Emacs!

Re: Bug: HTML not formatted correctly from R source code block [9.3.6 (9.3.6-23-g01ee25-elpaplus @ /home/opdfa/.emacs.d/elpa/org-plus-contrib-20200309/)]

[Jack Kamm]

Hi Steve,

Steven Delean  writes:

> HTML table produced from a source code block (using R code) does not display
> because html code is not formatted correctly in the exported html output.

I believe this is the same issue as reported here:
https://lists.gnu.org/archive/html/emacs-orgmode/2020-02/msg00267.html

Though, that thread was for ":session" blocks, and it's unclear whether
you're using session or non-session evaluation here.

Anyways, that thread suggests that ":results value html" works
correctly, does switching to that work for you?

If I'm correct, the basic problem is that ob-R tries to remove
interactive shell prompts like ">" from the output. This can also cause
other problems. I think it can be fixed by changing ob-R to use
`ess-eval-region' to evaluate code, I'll try to submit a patch for this
after 9.4 is released.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

As someone who doesn't really use ob-shell, I've made too much noise in
this thread, and am intending this to be my last comment here...

I think it's a bad idea that ":results value", and not specifying
":results", give different behavior.

In my own Org-babel files, I usually have a header line like follows:

#+PROPERTY: header-args :results output

Which sets the default return value to ":results output". Then, I
manually specify ":results value" when I need that instead.

But if explicitly writing ":results value" is different than the
default, then it would no longer be possible to use this pattern
anymore.

I think it would be prudent to revert the change to ":results value"
before the release of 9.4. I think Tom makes a convincing case that
returning the exit code is not only wrong, but also disruptive to
existing users of ob-shell.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Hi Tom,

Tom Gillespie  writes:

> As with many things in emacs, I sometimes feel like I'm loosing my
> mind, or loosing track of just exactly what variables are set

You're not losing your mind. After some further testing I find that
you're right, and my understanding of the situation was incorrect.

The reason I got confused was the initial example that started this
thread:

> #+begin_src sh
>   echo .
> #+end_src
> 
> #+RESULTS:
> : 0

Which looks like it's returning the exit code. However, it turns out
that this is just a coincidence, and it's an entirely unrelated bug.

If I had read the thread more carefully, I would have seen Bastien had
already noted this here:
https://lists.gnu.org/archive/html/emacs-orgmode/2020-02/msg00716.html

I fear my confusion of the issues lead to some misinformed comments on
my part. Sorry about that. I think I wasn't the only commenter confused
about this...

Anyways, it seems the current behavior of ob-shell blocks is:

- If no ":results" specified: returns the output, transformed to a
  table. Same as the old ":results value".
- If ":results value" explicitly specified: returns the exit code. This
  is a change from the old default behavior.

I can see why you, and other regular users of ob-shell, would be annoyed
by this change of behavior.

Now that I understand what's going on, I think it would be better to
stick with the old behavior (no exit code), just fixing the original bug
that prompted this thread.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Hi Tom,

Tom Gillespie  writes:

> First a disclosure, I would be very unhappy if option 1 were selected,
> it would require me to make a whole bunch of changes and try to find
> an option to revert to the current default behavior.

Wasn't option 1 already the default behavior, until the changes made
earlier this month due to this thread? That is the behavior I get when I
check out earlier commits, or maint.

> The principle is that block defaults for results should default to the
> value that the language itself makes the most accessible.

I agree that this would be a good principle.

My understanding is that in 9.3 and earlier, the default was generally
":results value", but now due to this thread we are adding an exception
for ob-shell.

It sounds like after 9.4 is released, there will be a general survey of
how org-babel behaves across languages, after which there will be a
discussion of sensible default behavior.

Re: correct remote path handling

[Jack Kamm]

Hi Felipe,

It looks like you've made some quite substantial changes to ob-shell.

I think it would be a good idea to split this up into 2 patches, and to
start a new thread for the ob-shell patch.

I'm not as familiar with ob-shell, and it's also had some work
lately. So it'd be good to get some more visibility for these changes
with a new thread for it.

Also, I think it'd be a good idea to provide some explanation of what
you've changed in ob-shell, the reason for these changes, and some
examples of how the behavior has changed.

If possible, it'd also be good to split up the ob-shell changes into a
few smaller patches, each addressing a specific issue.

Before starting a new thread, I would recommend waiting until Org 9.4 is
released, which should be any day now.

Below are some specific feedback for your updated patch:

> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
> index dbcfac08d..1afea9cb3 100644
> --- a/lisp/ob-python.el
> +++ b/lisp/ob-python.el
> @@ -327,7 +327,8 @@ last statement in BODY, as elisp."
> "python-")))
>  (with-temp-file tmp-src-file (insert body))
>  (format org-babel-python--exec-tmpfile
> -tmp-src-file))
> +(org-babel-local-file-name
> + tmp-src-file)))

You should use `org-babel-process-file-name' here, not
`org-babel-local-file-name'.

> -tmp-src-file
> -   (org-babel-comint-with-output
> -   (session org-babel-python-eoe-indicator nil body)
> +(org-babel-local-file-name
> + tmp-src-file)
> +(org-babel-comint-with-output
> +(session org-babel-python-eoe-indicator nil body)

Same comment as above.

> diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
> index 347ffedd1..66cdfb94c 100644
> --- a/lisp/ob-shell.el
> +++ b/lisp/ob-shell.el
> @@ -82,12 +82,17 @@ This function is called by `org-babel-execute-src-block'."
>(value-is-exit-status
> (member "value" (cdr (assq :result-params params
>(cmdline (cdr (assq :cmdline params)))
> +  (shebang (cdr (assq :shebang params)))
> +  (padline (not (equal "no" (cdr (assq :padline params)
> +  (result-params (cdr (assq :result-params params)))
>   (full-body (concat
>(org-babel-expand-body:generic
> body params (org-babel-variable-assignments:shell params))
>(when value-is-exit-status "\necho $?"
>  (org-babel-reassemble-table
> - (org-babel-sh-evaluate session full-body params stdin cmdline)
> + (org-babel-sh-evaluate session full-body
> + stdin cmdline shebang value-is-exit-status padline
> + result-params)

These variables used to be let-bound inside `org-babel-sh-evaluate', but
now you've changed the signature of that function to pass them in from
the outside.

I don't see a good reason for this change, since as far as I can tell,
those variables aren't used outside `org-babel-sh-evaluate'.

> -(defun org-babel-sh-evaluate (session body  params stdin cmdline)
> -  "Pass BODY to the Shell process in BUFFER.
> -If RESULT-TYPE equals `output' then return a list of the outputs
> -of the statements in BODY, if RESULT-TYPE equals `value' then
> -return the value of the last statement in BODY."

It looks like you've changed the indentation of this function, which
causes a hard-to-read diff. Please stick to the original indentation so
we can more easily review the changes here.

> + (let* ((block-output-lines
> + (let ((fun-body
> +(format "%s(){\n%s\n}\n%s"
> + org-babel-sh-block-function-name
> + ;; function block
> + (concat
> +  body
> +  "\n"
> +  org-babel-sh-eoe-indicator) ;; mark eoe
> + ;; mark "function has been input"
> + org-babel-sh-eoe-indicator)))

This is a very interesting idea, to wrap the shell block inside a
function.

I think it's a good idea and would fix some issues with leaky prompts,
among other things.

As an aside, I've been working on developing an async evaluation feature
for org-babel blocks, but we were struggling with ob-shell. This idea to
wrap the block in a function might be the solution.

Anyways, it would be good to discuss this change further with motivation
and examples.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Sorry, I was confused about this:

> According to my reading of this thread, most of the commenters were in
> agreement that we should keep the original behavior and return the exit
> code, as we do in 9.3.

Actually, it looks like ":results value" does return the exit code. I
just got confused because shell blocks now return the output when no
":results" is specified.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Hi Bastien,

Bastien  writes:

> thanks for your thoughtful inputs.  I've now removed the option.

I think it's good you removed the option.

However, it looks like the behavior now is to return the output, instead
of the exit code, when ":results value".

According to my reading of this thread, most of the commenters were in
agreement that we should keep the original behavior and return the exit
code, as we do in 9.3.

Re: correct remote path handling

[Jack Kamm]

Hi Felipe,

Felipe Lema  writes:

> I bumped into a problem running src blocks using a remote (tramp) :dir. I've 
> looked into it and found that the problem is that a temporary file is passed 
> as a remote path to the remote process (temp file should be local to remote 
> process).

Thanks for finding this bug and submitting a patch to fix it.

> I'm attaching fixes for python and shell src blocks. I didn't add any tests 
> because that would require more than 15 LOC.

I think it would be difficult to test for this, so I wouldn't worry too
much about it, but if you have ideas for how to do it that'd be great.

> I've signed the necessary papers from (to?) the FSF involving org mode, so 
> I'm 
> ready on my side to add tests and maybe add support for other tramp-related 
> stuff.

Great, we should add you to the list of copyrighted contributors:
https://orgmode.org/worg/org-contribute.html

CC'ing Bastien in case he wants to verify everything by private message
first.

> fix evaluate python code in remote directory
>
> Evaluating an "AST python code" should be local to the process /
> directory.
>
> `file-local-name` will do just this (strip the tramp prefix in path)

Please see https://orgmode.org/worg/org-contribute.html#commit-messages
for information on how to properly format commit messages. You can also
take a look at the git log for examples.

> -tmp-src-file
> +(file-local-name
> + tmp-src-file)

Instead of `file-local-name', the preferred function we use for this is
`org-babel-process-file-name'. It's used in a few other places in
ob-python.el as well.

It looks like this patch only handles the ":session :results value"
case, but ":session :results output" also suffers from this problem,
could you add a fix for that case as well?

For example, I tried to execute the following ":session :results output"
block within a remote Org file, with the following result:

#+begin_src python :session :results output
  x = 1+1
  x
#+end_src

#+RESULTS:
: Traceback (most recent call last):
:   File "", line 1, in 
: IOError: [Errno 2] No such file or directory: '/scp:pi:/tmp/python-NJwf2U'


> -(format "%s %s" shell-file-name script-file))
> +(format "%s %s" shell-file-name
> +(file-local-name script-file)))

As noted above, use `org-babel-process-file-name' instead of
`file-local-name'.

Also, I'm not sure the shell case is totally fixed by this, for example
trying to execute the following shell block within a remote Org file
still yields the error, even after applying your patch:

#+begin_src shell :shebang "#!/usr/bin/bash"
  echo foo
#+end_src

/bin/sh: 1: /scp:pi:/tmp/sh-script-8Z5fw7: not found

Re: ob-python.el: questions about output

[Jack Kamm]

Hi Eric,

"Fraga, Eric"  writes:

> Longer term, better would be an implementation of :prologue and
> :epilogue options for python src blocks like we have for Maxima.  It
> would probably make sense to provide these options for most if not all
> src blocks?

This sounds interesting. Do you know of any documentation or examples
for :prologue and :epilogue? I checked the Worg page for ob-maxima, but
it didn't mention these header arguments.

Re: ob-python.el: questions about output

[Jack Kamm]

Hi RC,

R C  writes:

> My python src blocks often have both graphical output as well as the
> results of some numerical calculation displayed using print statements.
>
> When exported I would like the line: return 'img/ex1.png' not to be
> included in the listing of the src block.
>
> Also, the output of the print statement is not displayed in this case.

The external ob-jupyter [0] accomplishes what you want:

1. It can return multiple types of output, e.g. print statements and plots.
2. It doesn't require the "return" statement.

It's main limitations are:
1. It requires jupyter.
2. It requires using session blocks.

I'd recommend checking it out. It has excellent Python support.

If you prefer a native solution, I think the options are:

1. noweb (as Eric mentioned). Then you only need to write the code
once, but note that it will be executed twice.

2. Using multiple session blocks to return output and value
separately. Since they use the same state, you would only need to
execute the code once.

3. Possibly, you could insert the link to the image separately, create a
named reference to it, and pass it to the Python block via ":var", which
would create at an image at that link. Then use ":results output" to
capture the print statements. I'm not 100% sure if this will work.

[0] https://github.com/dzop/emacs-jupyter

Re: ob-python.el: questions about output

[Jack Kamm]

Sorry, my email had a typo:

> #+begin_src python :session session1 :results output
> x
> #+end_src

That should be ":results value".

Re: ob-python.el: questions about output

[Jack Kamm]

Hi RC,

R C  writes:

> 1. Is it possible to have both types of output displayed from a single src
> block to avoid having to duplicate the src block, using :exports results
> for one and :exports both for the other?

No, there's no option to output both ":results output" and ":results value".

If you want both output and value results, and want to avoid executing
the same code twice, you could consider using a pair of ":session"
blocks, for example: 

#+begin_src python :session session1 :results output
print("Block that does some computations")
x = 1+1
#+end_src

#+begin_src python :session session1 :results output
x
#+end_src

> 2. Is there an option to suppress export of the the last line of the src
> block which is specific to the ob-python implementation.

I'm not sure what you mean by this, could you give an example of what
the observed and desired behaviors are?

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Hi Stefan,

Stefan Nobis  writes:

> What about a third collection option 'none' and make this the default?

A variant on this idea, would be to instead have the third collection
option be 'default', which would stand for a language-specific
default. For example, ":results default" could be equivalent to
":results output" for shell blocks, but equivalent to ":results value"
for emacs-lisp blocks.

Re: Make ob-python.el support ":results pp" pretty print result

[Jack Kamm]

Hi stardiviner,

stardiviner  writes:

> I found ob-python does not support ":results pp" pretty print result. And 
> Python
> has a module "pprint". What about add support this?

":results pp" works for me, here's an example:

> #+begin_src python :results pp
>   return globals()
> #+end_src
> 
> #+RESULTS:
> #+begin_example
> {'__annotations__': {},
>  '__builtins__': ,
>  '__cached__': None,
>  '__doc__': None,
>  '__file__': '',
>  '__loader__': ,
>  '__name__': '__main__',
>  '__package__': None,
>  '__spec__': None,
>  'main': ,
>  'pprint': }
> #+end_example

Here's how it looks by default (without "pp"):

> #+begin_src python
>   return globals()
> #+end_src
> 
> #+RESULTS:
> | __name__ | : | __main__ | __doc__ | : | hline | __package__ | : | hline | 
> __loader__ | : |  | __spec__ | 
> : | hline | __annotations__ | : | nil | __builtins__ | : |  | (built-in) | > | __file__ | : |  | __cached__ | : | hline | main | : 
> |  |

And here's how it looks when using ":results output":

> #+begin_src python :results output
>   print(globals())
> #+end_src
> 
> #+RESULTS:
> : {'__name__': '__main__', '__doc__': None, '__package__': None, 
> '__loader__': , '__spec__': None, 
> '__annotations__': {}, '__builtins__': , 
> '__file__': '', '__cached__': None}

I also tested this on ":session" blocks and the results were the same.

Re: Bug or not a bug? dot expansion in ob-shell

[Jack Kamm]

Hi,

Nick Dokos  writes:

>> That said, we have three solutions:
>>
>> 1. Stick to a strict reading of Org and bash manuals: the absence of a
>>:results header means "return the value, i.e. the exit status".
>>
>> 2. Deviate from this strict reading, introduce an exception for *all*
>>shell blocks: no :results header means "return output" and you need
>>to use :results value to get the exit status (your proposal).
>>
>> 3. Deviate from this strict reading and introduce an option that says:
>>"Don't deviate from the Org's and bash manuals for all src blocks".
>>
>> Obviously, nobody wants the first solution.
>
> I'm not convinced of that: personally, I would be happy with the first
> solution and maybe others would be too.
> 
> My longer term suggestion (which, I realize, does not help to close
> *this* case any time soon):
> 
> I think that the global default setting `:results value' is wrong

Strongly agree with Nick on both points here. I prefer option 1 over
options 2 and 3, I think it is the more consistent and simple behavior.

I also think ":results value" is a problematic default.

Anecdotally, I use Org-Babel as a computational notebook similar to
Jupyter or Rmarkdown, and set ":session :results output" for nearly all
my org-babel blocks.

Of course, there are many ways in which org-babel can be used, and for
some uses, ":results value" is the better default. But for other use
cases, including shell blocks, ":results output" is the more intuitive
default.

And I think this is really the problem here. The result of a shell block
is surprising to the new user, because the user probably wants ":results
output", but ":results value" is the default. I'm not sure of the
solution to this problem, but I don't think it's to change the meaning
of ":results value".

Re: [PATCH] Fix ob-python.el initiate session error with py-shell

[Jack Kamm]

Hi,

Bastien  writes:

> Also, for included ob-*, the idea would be to use the mode that are
> bundled with Emacs core.  For Python, it would mean that ob-python.el
> should support python.el, not python-mode.el.

I agree that ob-python.el should only rely on functionality from
python.el, and shouldn't need to know about other packages. This will
reduce the maintenance burden for ourselves.

At the same time, I would like the user to use whatever Python-related
modes they like when editing the org Python source buffer, such as
python-mode.el, elpy, etc., without issue. That doesn't mean that we'll
put workarounds and references to external packages in ob-python.el, but
rather that we'd keep ob-python simple and easy for external packages to
interop with.

I think this is more or less the case today, and we're on the same page
here. Just wanted to make this goal explicit.

Re: Make ob-python.el support ":results pp" pretty print result

[Jack Kamm]

Hi stardiviner,

stardiviner  writes:

> I found ob-python does not support ":results pp" pretty print result. And 
> Python
> has a module "pprint". What about add support this?

Well, there is code in ob-python.el that uses the pprint module when
":results pp", but I must admit I've never used it and don't know
whether it is currently working. I'm also unsure there's a difference in
how session and non-session blocks deal with this.

I'll plan to test it out later this weekend or next week and see if I
can fix any issues. Or, if you're able, please feel free to submit a
patch (but please let me know if you plan to work on this, so we can
avoid duplicating effort).

Ideally, a patch for this would also include a unit test, to make sure
this doesn't break in future.

Re: [PATCH] Fix ob-python.el initiate session error with py-shell

[Jack Kamm]

Hi stardiviner,

"numbch...@gmail.com"  writes:

> Yes, Jack, as Bastien said, you can format my commit, because my home
> network is broken, I'm using Mobile Phone's 4G network to get online. Can't
> get update immediately.

OK, I've edited the commit message of your patch, and I also fixed the
compilation warning by updating the signature of "(declare-function
py-shell)".

Commit has now been pushed to master :)

> And thanks for tips about `python-mode' is deprecated. I didn't know that.
> I will migrate to `python.el'.

Since python.el is built-in to Emacs, I think that things should
continue to work for python-mode.el users, even if we entirely switch
over to only using python.el.

The only thing I'm not sure about, is whether the equivalent of
"python-shell-send-region" in python-mode.el will work with shells
started by python.el. If not, this could probably be addressed with a
patch to python-mode.el.

You could test this by explicitly setting `org-babel-python-mode' to
`python' (it is probably set to `python-mode' on your system, which is
the default when python-mode.el is detected).

I'll add a TODO for myself to explicitly mark python-mode-related
variables as deprecated. I'm also planning a major update to the Worg
documentation of ob-python when 9.4 comes out, and will mention the
deprecation there as well.

Re: [PATCH] Fix ob-python.el initiate session error with py-shell

[Jack Kamm]

Hi stardiviner,

Sorry for the noise, I have some additional comments below:

When I run "make compile", I get the following message:

In org-babel-python-initiate-session-by-key:
ob-python.el:211:12:Warning: py-shell called with 9 arguments, but accepts
only 0-1

I think this is because the "(declare-function py-shell)" no longer
matches the signature of the upstream py-shell function you're using.

Also, I now see that the commit does have a changelog entry, but not a
commit message. Unless it is standard to take the subject line as the
commit message? I am still a little new to merging patches from email,
and might not have all the proper tooling setup for it.

When you update the patch, please mention that this is specifically for
python-mode.el in the commit message and changelog entry.

Re: [PATCH] Fix ob-python.el initiate session error with py-shell

[Jack Kamm]

Hi stardiviner,

> This minor patch should fixed ~py-shell~ initiate Python session issue.

It looks like this patch only affects users of python-mode.el. Since I
don't use python-mode.el, I cannot test it. But the patch looks
fine. Please add a commit message and changelog entry, and I'll merge it
in.

Now, a few general thoughts on python-mode.el:

I'm a little surprised to see python-mode.el pop up, and that
we have code in ob-python to explicitly support it. I thought it was
obsolete ever since python.el was added to emacs 24. But, it looks like
it's still receiving commits in 2020, so I guess I was wrong.

In a sense, it doesn't seem right to have code in ob-python explicitly
supporting python-mode.el. While at one point it may have been a
contender with python.el as the standard Python mode, at this point it's
just another third-party package like elpy, jedi, etc, none of which get
explicit support.

I don't want to break anyone's setup without further investigation, so
I'll leave the python-mode.el support as is, for now. But a warning,
this may change in future if we ever refactor ob-python.

Re: [PATCH] (Tiny) Tweak Python session null return value

[Jack Kamm]

Hi John,

John Kitchin  writes:

> I can see why you would want to see True/False there, but to get the value,
> you need to specifically return what you want because AFAIK the body is
> wrapped in a function that is evaluated to get the value, it is not simply
> the last thing that gets evaluated.

This is true for non-session blocks, which require explicitly calling
"return". However, session blocks aren't wrapped in functions and don't
use "return" (even before the most recent patches). The problem is that
variables created in a function have local scope, so session blocks
can't be wrapped in functions.

> Your example clarified to me at least why it would be tricky to figure
> it out, you can't rely on the last line, for example.

Since the recent patches, we do extract the last line, using the Python
ast module, however this only works if the last line is a top-level
statement like "f()" or "1+1", not an assignment (like "x = 1+1") or an
indented block (like "if:...else:...").

>  I don't know if there is some special Python variable that contains
> that.

There actually is -- in most Python interpreters, the variable "_"
(underscore) refers to the last statement, unless it's been explicitly
assigned to. This is what was previously relied on. Unfortunately, using
"_" for a dummy variable is a common Python idiom (e.g. "for _ in
range(10)"), and if used would break all subsequent Python session
blocks. So we no longer rely on "_".

In the standard Python interpreter, we can also use "__builtins__._",
but this doesn't work in IPython. Furthermore, this only works for code
explicitly entered in the shell, it won't work for code executed in
"exec()" or "eval()", which we now rely on, because it handles
indentation much more robustly. In particular, ob-python sessions have
had longstanding issues with multiline indented blocks, which are now
solved in the recent patches.

Re: [PATCH] (Tiny) Tweak Python session null return value

[Jack Kamm]

Hi Bastien,

> I've seen you update the NEWS entry, which is good: is there a way to
> present the enhancements in the "* New features" section?  If you feel
> like it, please advertize the enhancements there too.

Given John's feedback, I now think it's better to put off this change to
9.5, if at all. So instead of updating NEWS, I'd prefer to revert this
commit for now. Sorry for the noise! Looking forward to 9.4.

Re: [PATCH] (Tiny) Tweak Python session null return value

[Jack Kamm]

Hi John,

John Kitchin  writes:

> I think None is correct. If you don't specify a return value in Python,
> then a function returns None. I would expect that to happen in a Python
> block too.

Hmm, OK, thanks for your intuition, it's useful feedback.

Working this out loud, I was considering the following sort of block:

#+begin_src python :session :results value
  if some_condition:
  True
  else:
  False
#+end_src

#+RESULTS:
: None

Ideally, it would return True/False, but the current implementation
cannot grab that result unfortunately. (In its defense, it at least
doesn't crash like it did before).

I was thinking not printing anything at all under "#+RESULTS" would be
less surprising than printing "None". But both are admittedly surprising
at first.

I agree your preference of returning "None" is the more technically
consistent behavior though.

I don't want to rush a decision now, I think I need to gather more user
feedback after the 9.4 release. I'll ask Bastien to revert that commit
and put this off to 9.5, rather than writing an updated NEWS entry like
he asked.

Either way, we'll need to update Worg to clearly document what's going
on here.

Re: Bug: R src blocks fail to produce link to graphics file [9.3 org-plus-contrib-20200127]

[Jack Kamm]

Hi Tyler,

>> The documentation for ob-R is now incorrect:
>>
>>   https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-R.html
>
> Yes, also that page could use some other updates, e.g. there are some
> dead links in there.

Just saw that you updated the docs a few days ago. Thanks for taking
care of this!

Best,
Jack

[PATCH] (Tiny) Tweak Python session null return value

[Jack Kamm]

Hi,

Below is a very small patch to Python session blocks, to make them
return a blank result (empty string) instead of None when there is no
return value.

Normally I would push this myself, but since we are so close to 9.4, I
thought it prudent to mail a patch and let the maintainers handle it. It
would be nice to include in 9.4, but not a big deal if I've missed the
window.

Now for an explanation of the patch: 9.4 changes Python session blocks
to fix several bugs and improve robustness overall [0]. However, there
is a cost to these fixes, which is that the session blocks can only
return a result when it is a top-level expression on the last line of
the block. If the last line is not a top-level expression, the block
would previously print "None". However, after some testing, I think this
is a little counter-intuitive, and it would be better if it returned a
blank (empty) result instead of "None".

[0] https://lists.gnu.org/archive/html/emacs-orgmode/2020-01/msg00190.html

Best,
Jack


>From 0b44c3f1c7454e7948cd34eb02995924046b6976 Mon Sep 17 00:00:00 2001
From: Jack Kamm 
Date: Mon, 17 Feb 2020 08:11:49 -0800
Subject: [PATCH] ob-python: Session returns empty string if no return value

* lisp/ob-python.el (org-babel-python--eval-ast): Change sessions to
return an empty string, instead of None, if there is no return value.
---
 etc/ORG-NEWS  | 12 +++-
 lisp/ob-python.el |  2 +-
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index b6ee443e4..1b5429870 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -12,13 +12,15 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.4 (not yet released)
 ** Incompatible changes
-*** Python session return values must be top-level expression statements
+*** Change how Python sessions handle return values
 
 Python blocks with ~:session :results value~ header arguments now only
-return a value if the last line is a top-level expression statement,
-otherwise the result is None. Also, None will now show up under
-"#+RESULTS:", as it already did with ~:results value~ for non-session
-blocks.
+return a value if the last line is a top-level expression
+statement. If the last line is not a top-level expression statement,
+the block will return a blank (empty) result.
+
+Also, a return value of ~None~ will now show up under "#+RESULTS:", as
+it already did for non-session blocks.
 
 *** In HTML export, change on how outline-container-* is set
 
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index de718b04b..b9e36dba0 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -260,7 +260,7 @@ (defconst org-babel-python--eval-ast "\
 __org_babel_python_final.value), '', 'eval'))
 else:
 exec(compile(__org_babel_python_ast, '', 'exec'))
-__org_babel_python_final = None
+__org_babel_python_final = ''
 except Exception:
 from traceback import format_exc
 __org_babel_python_final = format_exc()
-- 
2.25.0

Re: org-babel strange html print in R

[Jack Kamm]

Hi,

Jeremie Juste  writes:

> :results value html  in the following code generates an error for me.
> (Org mode version 9.2.3,R version 3.6.2).
> but :results value output the "html" in an org-table format

Hmm, I can confirm that ":session :results value html" is working for
me, same as for stardiviner. Could you give more info about the error
you're seeing?

I'm using R 3.6.2 and Org 9.4. But I'd be surprised if it doesn't work
for Org 9.2 -- git blame suggests that html output has been supported in
ob-R since at least 2015. But it wouldn't hurt to upgrade to the latest
released version of Org (9.3).