Re: Help debugging R source code block output problem with :session
On Wed, Jan 12, 2022 at 1:50 AM Jeremie Juste wrote:
>
> Hello John,
>
> Many thanks for reporting. I'm short of time right now, but I'll take a
> closer look at the problem over
> the weekend and keep you posted. Could you let me know which version of
> org-mode you are currently using?
Sure thing and thanks for the quick note! I use org from git and
*thought* it used to report the most recent release tag, but I see:
$ git log
commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
origin/master, origin/HEAD)
Author: Ian Martins
Date: Sat Jan 16 15:52:21 2021 -0500
But Mx-version:
Org mode version 9.4.4 (release_9.4.4-186-g7fa817.dirty @
/home/jwhendy/.elisp/org/lisp/)
So perhaps it appends the git commit to the system version?
Btw, it's dirty as I was able to apply Jack's patches (with some mods)
successfully and am a happy camper. In case it's helpful, I attached
the patches as I applied them (first -1, then -2) to this org-version.
Nothing special, they are just Jack's patches with a couple sections
removed (one change was already present, and I removed the adjustments
to the testing file as they were failing).
Thanks so much to all of you who contributed to that, as I'm so happy
that I can at least continue on with my report at this point with good
looking output!
Many thanks,
John
>
> Best regards,
> Jeremie Juste
>
>
> On Tuesday, 11 Jan 2022 at 17:36, John Hendy wrote:
> > Greetings,
> >
> > Apologies for writing off-list, but I don't have the original thread
> > anymore to reply to. I just pulled this group in directly from the
> > email I found... I just got bitten by this as well and burned most of
> > my afternoon trying to figure out what the issue was.
> >
> > Any further thoughts on a patch that could be finalized/applied?
> >
> > Is the one from the thread still the best from your continued experience?
> > https://list.orgmode.org/87zgxc42qg@gmail.com/
> >
> > This r/orgmode post was what finally got me to the existence of this
> > as a previous issue (I'd just sent my own email to list as my
> > searching did not find the above):
> > https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/
> >
> > And it links to a patch as well:
> > https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e
> >
> > Many thanks,
> > John
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..b37e3965a 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -437,24 +439,23 @@ (defun org-babel-R-evaluate-session
(org-babel-import-elisp-from-file tmp-file '(16)))
column-names-p)))
(output
- (mapconcat
- 'org-babel-chomp
- (butlast
- (delq nil
- (mapcar
- (lambda (line) (when (> (length line) 0) line))
- (mapcar
- (lambda (line) ;; cleanup extra prompts left in output
- (if (string-match
- "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
- (car (split-string line "\n")))
- (substring line (match-end 1))
- line))
- (org-babel-comint-with-output (session org-babel-R-eoe-output)
- (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
- (inferior-ess-send-input)) "\n"
+ (let* ((tmp-file (org-babel-temp-file "R-")))
+ (with-temp-file tmp-file
+ (insert (concat body "\n" org-babel-R-eoe-indicator)))
+ (with-current-buffer session
+ (let* ((process (get-buffer-process (current-buffer)))
+ (string-buffer "")
+ (comint-output-filter-functions
+ (cons (lambda (text) (setq string-buffer
+ (concat string-buffer text)))
+ comint-output-filter-functions)))
+ (ess-send-string
+ process (format "source('%s', print.eval=TRUE)"
+ (org-babel-process-file-name tmp-file 'noquote)))
+ (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+ string-buffer))
+ (accept-process-output process))
+ (substring string-buffer 0 (match-beginning 0
(defun org-babel-R-process-value-result (result column-names-p)
"R-specific processing of return value.
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index b37e3965a..5ddf0ebd1 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
(output
(let* ((tmp-file (org-babel-temp-file "R-")))
(with-temp-file tmp-file
- (insert (concat body "\n" org-babel-R-eoe-indicator)))
+ (insert body))
(with-current-buffer session
(let* ((process (get-buffer-process (current-buffer)))
(string-buffer "")
@@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
(concat string-buffer text)))
comint-output-filter-functions)))
(ess-send-string
- process (format "source('%s', print.eval=TRUE)"
- (org-babel-process-file-name tmp-file 'noquote)))
+ process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+ (org-babel-process-file-name tmp-file 'noquote)
+ org-babel-R-eoe-indicator))
(while
Re: Help debugging R source code block output problem with :session
Hello John, Many thanks for reporting. I'm short of time right now, but I'll take a closer look at the problem over the weekend and keep you posted. Could you let me know which version of org-mode you are currently using? Best regards, Jeremie Juste On Tuesday, 11 Jan 2022 at 17:36, John Hendy wrote: > Greetings, > > Apologies for writing off-list, but I don't have the original thread > anymore to reply to. I just pulled this group in directly from the > email I found... I just got bitten by this as well and burned most of > my afternoon trying to figure out what the issue was. > > Any further thoughts on a patch that could be finalized/applied? > > Is the one from the thread still the best from your continued experience? > https://list.orgmode.org/87zgxc42qg@gmail.com/ > > This r/orgmode post was what finally got me to the existence of this > as a previous issue (I'd just sent my own email to list as my > searching did not find the above): > https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/ > > And it links to a patch as well: > https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e > > Many thanks, > John
Re: Help debugging R source code block output problem with :session
Greetings, Apologies for writing off-list, but I don't have the original thread anymore to reply to. I just pulled this group in directly from the email I found... I just got bitten by this as well and burned most of my afternoon trying to figure out what the issue was. Any further thoughts on a patch that could be finalized/applied? Is the one from the thread still the best from your continued experience? https://list.orgmode.org/87zgxc42qg@gmail.com/ This r/orgmode post was what finally got me to the existence of this as a previous issue (I'd just sent my own email to list as my searching did not find the above): https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/ And it links to a patch as well: https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e Many thanks, John
Re: Help debugging R source code block output problem with :session
Hello, I must apologize again for the delay. I'll be more responsive from now on. @Jack, I have applied the patch at the bottom of the mail. It is not your latest patch but it works as well and is able to handle R errors. > https://orgmode.org/list/87ft7t9wqk@gmail.com/ I have also added the 2 test cases with to make sure we are aware if this feature breaks in the future. > https://orgmode.org/list/87h7slgbi5@gmail.com/ > I'd be interested to hear if the attached patch works for the common > cases you encounter, such as with tibbles. @ Ilja, sorry for the delayed reply. Regarding tibbles we are still not out of the woods. But the current patch would work if crayon option is removed. #+BEGIN_SRC R :results output :session "local1" :pre options(crayon.enabled=FALSE) library(tidyr) library(dplyr) as_tibble(iris) #+END_SRC #+RESULTS: #+begin_example # A tibble: 150 x 5 Sepal.Length Sepal.Width Petal.Length Petal.Width Species 1 5.1 3.5 1.4 0.2 setosa 2 4.9 31.4 0.2 setosa 3 4.7 3.2 1.3 0.2 setosa 4 4.6 3.1 1.5 0.2 setosa 5 5 3.6 1.4 0.2 setosa 6 5.4 3.9 1.7 0.4 setosa 7 4.6 3.4 1.4 0.3 setosa 8 5 3.4 1.5 0.2 setosa 9 4.4 2.9 1.4 0.2 setosa 10 4.9 3.1 1.5 0.1 setosa # … with 140 more rows #+end_example > @Chuck > https://orgmode.org/list/352c7149-743f-4944-aca5-7a1242b5a...@health.ucsd.edu/ >If you do decide to dig into solving this, please be sure that remote >sessions and graphical outputs are not broken. test-ob-R.el does not >cover those cases. In fact, it is pretty short, so there are probably >other things that could break without `make test' complaining. The current patch have been tested for remote connections as well and AFAIK, nothing breaks. But I'm afraid that the graphical output is broken and has long been even before the path. The test for graphical output is compromised and does not do the right test. I will suggest new ones. Please feel free to contact me or reply on the mailing list if you have see any improvements to be made. Best regards, Jeremie On Saturday, 29 Aug 2020 at 00:24, Jack Kamm wrote: >>From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001 > From: Jack Kamm > Date: Sat, 29 Aug 2020 00:07:58 -0700 > Subject: [PATCH] ob-R: Fix prompt mangling in session output > > * lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt > regexp to start at beginning of line, to prevent > org-babel-comint-with-output from splitting mid-line. > > Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and > https://orgmode.org/list/87r1rqled0.fsf@havana/ > --- > lisp/ob-R.el | 12 +++- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/lisp/ob-R.el b/lisp/ob-R.el > index 5e9d35f58..10b3b6fe3 100644 > --- a/lisp/ob-R.el > +++ b/lisp/ob-R.el > @@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session > (car (split-string line "\n"))) >(substring line (match-end 1)) > line)) > -(org-babel-comint-with-output (session org-babel-R-eoe-output) > - (insert (mapconcat 'org-babel-chomp > - (list body org-babel-R-eoe-indicator) > - "\n")) > - (inferior-ess-send-input)) "\n" > +(with-current-buffer session > + (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp))) > +(org-babel-comint-with-output (session > org-babel-R-eoe-output) > + (insert (mapconcat 'org-babel-chomp > + (list body org-babel-R-eoe-indicator) > + "\n")) > + (inferior-ess-send-input "\n" > > (defun org-babel-R-process-value-result (result column-names-p) >"R-specific processing of return value. > -- > 2.28.0
Re: Help debugging R source code block output problem with :session
Hello Jack, > Thanks for volunteering to maintain ob-R.el :) Thanks for the encouragements and the suggestions. I need will take some time to dive into the existing code but I will definitely look into it and keep you posted. I've also noticed a [1] bug waiting to be fixed. [1] https://orgmode.org/list/71761e05-7d0a-4fef-8baf-4c776a2fc...@health.ucsd.edu/ Best regards, Jeremie On Wednesday, 28 Oct 2020 at 06:13, Jack Kamm wrote: > Hi Jeremie, > > Thanks for volunteering to maintain ob-R.el :) > > I'm bumping this patch [0] to see if it could be merged into ob-R.el. It > fixes some long-standing issues with prompt mangling for :session blocks > with :results output. > > Elsewhere in the thread, Chuck had a suggestion [1] to allow :results > value to be properly handled in remote sessions. It's not included in > the patch, but worth adding separately IMO. > > It may be helpful reading the thread in full -- Chuck had several > comments on ob-R that I found insightful, and may be useful for you to > consider as maintainer. I'd highlight [2] in particular: > >> If you do decide to dig into solving this, please be sure that remote >> sessions and graphical outputs are not broken. test-ob-R.el does not >> cover those cases. In fact, it is pretty short, so there are probably >> other things that could break without `make test' complaining. > > All the best, > Jack > > [0] https://orgmode.org/list/87ft7t9wqk@gmail.com/ > [1] > https://orgmode.org/list/1e0046a6-1fab-45b5-9b08-68fe1be98...@health.ucsd.edu/ > [2] > https://orgmode.org/list/352c7149-743f-4944-aca5-7a1242b5a...@health.ucsd.edu/ >
Re: Help debugging R source code block output problem with :session
Hi Jeremie, Thanks for volunteering to maintain ob-R.el :) I'm bumping this patch [0] to see if it could be merged into ob-R.el. It fixes some long-standing issues with prompt mangling for :session blocks with :results output. Elsewhere in the thread, Chuck had a suggestion [1] to allow :results value to be properly handled in remote sessions. It's not included in the patch, but worth adding separately IMO. It may be helpful reading the thread in full -- Chuck had several comments on ob-R that I found insightful, and may be useful for you to consider as maintainer. I'd highlight [2] in particular: > If you do decide to dig into solving this, please be sure that remote > sessions and graphical outputs are not broken. test-ob-R.el does not > cover those cases. In fact, it is pretty short, so there are probably > other things that could break without `make test' complaining. All the best, Jack [0] https://orgmode.org/list/87ft7t9wqk@gmail.com/ [1] https://orgmode.org/list/1e0046a6-1fab-45b5-9b08-68fe1be98...@health.ucsd.edu/ [2] https://orgmode.org/list/352c7149-743f-4944-aca5-7a1242b5a...@health.ucsd.edu/
Re: Help debugging R source code block output problem with :session
Thanks. This patch works for my org files. On Tue 08 Sep 2020 at 12:41, Jack Kamm wrote: Hi Chuck, this is already present in `org-babel-R-evaluate-session' in the call to `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines further down in the `(cl-case result-type (value ...))' branch. The other use of `tmp-file' in that block is the one that requires the prefix, right? So nothing more to fix. ?? You're right, I didn't notice that it's already present -- nothing more to fix. Jack --
Re: Help debugging R source code block output problem with :session
Hi Chuck, > this is already present in `org-babel-R-evaluate-session' in the call to > `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines > further down in the `(cl-case result-type (value ...))' branch. > > The other use of `tmp-file' in that block is the one that requires the > prefix, right? > > So nothing more to fix. > > ?? You're right, I didn't notice that it's already present -- nothing more to fix. Jack
Re: Help debugging R source code block output problem with :session
Jack, Maybe I am confused here: > On Sep 8, 2020, at 7:51 AM, Jack Kamm wrote: > > Yes, if we did that then tmp-file would have a prefix like > "/scp:user@hostname:", and elisp would then know to read the result file > from the remote host. > > Before pasting tmp-file into R code, we should also call > > (org-babel-process-file-name tmp-file 'noquote) > > to remove the tramp prefix when referring to the file from R. this is already present in `org-babel-R-evaluate-session' in the call to `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines further down in the `(cl-case result-type (value ...))' branch. The other use of `tmp-file' in that block is the one that requires the prefix, right? So nothing more to fix. ?? Chuck
Re: Help debugging R source code block output problem with :session
"Berry, Charles" writes: > Also, I wonder if the `:results value' case can be handled in a similar > manner, viz. > > - (let ((tmp-file (org-babel-temp-file "R-"))) > + (let ((tmp-file (with-current-buffer session (org-babel-temp-file "R-" Yes, if we did that then tmp-file would have a prefix like "/scp:user@hostname:", and elisp would then know to read the result file from the remote host. Before pasting tmp-file into R code, we should also call (org-babel-process-file-name tmp-file 'noquote) to remove the tramp prefix when referring to the file from R.
Re: Help debugging R source code block output problem with :session
Jack, > On Sep 7, 2020, at 6:06 PM, Jack Kamm wrote: > > Hi Chuck, > >> I can confirm that this works on my setup in each of the scenarios in which >> `default-directory' got set correctly in the session buffer. >> >> At some point, my default-directory got reset to drop the tramp prefix >> "/scp:/user@host:" in one session after faithfully running the src block >> several times. I am unable to sort out why or even reproduce this. > > Thanks for taking the time to test again. If you have any further > feedback at any point, please don't hesitate. > Thanks. This seems like it will help me. Also, I wonder if the `:results value' case can be handled in a similar manner, viz. - (let ((tmp-file (org-babel-temp-file "R-"))) + (let ((tmp-file (with-current-buffer session (org-babel-temp-file "R-" Best, Chuck
Re: Help debugging R source code block output problem with :session
Hi Chuck, > I can confirm that this works on my setup in each of the scenarios in which > `default-directory' got set correctly in the session buffer. > > At some point, my default-directory got reset to drop the tramp prefix > "/scp:/user@host:" in one session after faithfully running the src block > several times. I am unable to sort out why or even reproduce this. Thanks for taking the time to test again. If you have any further feedback at any point, please don't hesitate. Best, Jack
Re: Help debugging R source code block output problem with :session
Jack, I can confirm that this works on my setup in each of the scenarios in which `default-directory' got set correctly in the session buffer. At some point, my default-directory got reset to drop the tramp prefix "/scp:/user@host:" in one session after faithfully running the src block several times. I am unable to sort out why or even reproduce this. Chuck > On Sep 7, 2020, at 1:07 PM, Jack Kamm wrote: > > Hi Chuck, > >> This does not work for my remote session. > > Thanks for testing this. > > The remote tests I originally tried did not cover this case -- I only > tested the case where both org-file and session were remote, but didn't > test the case where org-file was local and session was remote. > > I've now tested this case as well, and added some fixes for it. I'm > reattaching the patch in full. > > However, this implementation does require the R session to have the > correct default-directory -- see details below.
Re: Help debugging R source code block output problem with :session
Hi Chuck,
> This does not work for my remote session.
Thanks for testing this.
The remote tests I originally tried did not cover this case -- I only
tested the case where both org-file and session were remote, but didn't
test the case where org-file was local and session was remote.
I've now tested this case as well, and added some fixes for it. I'm
reattaching the patch in full.
However, this implementation does require the R session to have the
correct default-directory -- see details below.
> The problem is that tempfiles on the remote host are like
> "/tmp/RtmpeFHudh/file23a66d2fc1f9", but emacs tries to use
> '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fhgn/T/babel-OSXKNd/R-oNOVVB'
>
> `org-babel-temp-file' doesn't honor remote connections AFAICS.
org-babel-temp-file does honor remote connections, but relies on
default-directory to do so. I've moved the call to org-babel-temp-file
so it happens when the session buffer is current -- that way, the
tempfile will be correctly handled, assuming that the session's
default-directory is on the remote location.
If the session's default-directory isn't at the remote location, then
this will break. But, in this case some other things break as well, for
example inserting links to remote plots.
> Maybe there is some comint or tramp idiom that would solve this, but I do not
> know what it is.
Here are some ways to start a remote R session with correct
default-directory:
1. Do "M-x R", then when it prompts for a directory, use
"/scp:hostname:/some/path". Optionally, first visit that location with
"C-x C-f /scp:hostname:/some/path", so that the default value is
already there.
2. Alternatively, start the R session by evaluating a source block with
header argument ":dir /scp:hostname:/some/path".
3. If you prefer to use "M-x shell" with "ess-remote",
first visit the remote location with "C-x C-f /scp:hostname:/some/path", before
calling "M-x shell".
4. If you prefer to start "M-x shell" locally and then
ssh in, it's still possible to have default-directory set, but it
requires some configuration [1]. Or you could use "M-x cd" to set it as well.
[1]
https://emacs.stackexchange.com/questions/5589/automatically-update-default-directory-when-pwd-changes-in-shell-mode-and-term-m/5592#5592
Best,
Jack
>From e7f1a59167de88fb9a5b96a0e1ac3199f105f600 Mon Sep 17 00:00:00 2001
From: Jack Kamm
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts
* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.
Fixes issue reported in
https://orgmode.org/list/875zgjh8wn@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
lisp/ob-R.el | 38 --
testing/lisp/test-ob-R.el | 13 +
2 files changed, 33 insertions(+), 18 deletions(-)
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..dffbbe112 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
(declare-function ess-eval-buffer "ext:ess-inf" (vis))
(declare-function ess-wait-for-process "ext:ess-inf"
(&optional proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+ (process string &optional visibly message _type))
(defconst org-babel-header-args:R
'((width . :any)
@@ -437,24 +439,24 @@ (defun org-babel-R-evaluate-session
(org-babel-import-elisp-from-file tmp-file '(16)))
column-names-p)))
(output
- (mapconcat
- 'org-babel-chomp
- (butlast
- (delq nil
- (mapcar
- (lambda (line) (when (> (length line) 0) line))
- (mapcar
- (lambda (line) ;; cleanup extra prompts left in output
- (if (string-match
- "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
- (car (split-string line "\n")))
- (substring line (match-end 1))
- line))
- (org-babel-comint-with-output (session org-babel-R-eoe-output)
- (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
- (inferior-ess-send-input)) "\n"
+ (with-current-buffer session
+ (let* ((tmp-file (org-babel-temp-file "R-"))
+ (process (get-buffer-process (current-buffer)))
+ (string-buffer "")
+ (comint-output-filter-functions
+ (cons (lambda (text) (setq string-buffer
+ (concat string-buffer text)))
+ comint-output-filter-functions)))
+ (with-temp-file tmp-file
+ (insert body))
+ (ess-send-string
+ process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+ (org-babel-process-file-name tm
Re: Help debugging R source code block output problem with :session
Jack,
This does not work for my remote session.
I run from macOS locally and on a Linux host remotely. I use
M-x shell RET
R RET
M-x ess-remote RET R RET
to start the R session.
The problem is that tempfiles on the remote host are like
"/tmp/RtmpeFHudh/file23a66d2fc1f9", but emacs tries to use
'/var/folders/kb/2hchpbyj7lb6z76l0q73w_fhgn/T/babel-OSXKNd/R-oNOVVB'
`org-babel-temp-file' doesn't honor remote connections AFAICS.
Maybe there is some comint or tramp idiom that would solve this, but I do not
know what it is.
Chuck
> On Sep 7, 2020, at 1:45 AM, Jack Kamm wrote:
>
> I just realized my patch had an issue where it freezes if there is an
> error in the source block.
>
> I'm attaching a second patch, to be applied on top of the first one, that
> fixes the issue.
>
> diff --git a/lisp/ob-R.el b/lisp/ob-R.el
> index b37e3965a..5ddf0ebd1 100644
> --- a/lisp/ob-R.el
> +++ b/lisp/ob-R.el
> @@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
> (output
> (let* ((tmp-file (org-babel-temp-file "R-")))
>(with-temp-file tmp-file
> - (insert (concat body "\n" org-babel-R-eoe-indicator)))
> + (insert body))
>(with-current-buffer session
>(let* ((process (get-buffer-process (current-buffer)))
> (string-buffer "")
> @@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
> (concat string-buffer text)))
> comint-output-filter-functions)))
> (ess-send-string
> - process (format "source('%s', print.eval=TRUE)"
> - (org-babel-process-file-name tmp-file 'noquote)))
> + process (format "tryCatch(source('%s', print.eval=TRUE),
> finally=print(%s))"
> + (org-babel-process-file-name tmp-file 'noquote)
> + org-babel-R-eoe-indicator))
> (while (not (string-match (regexp-quote org-babel-R-eoe-output)
>string-buffer))
>(accept-process-output process))
Re: Help debugging R source code block output problem with :session
I just realized my patch had an issue where it freezes if there is an
error in the source block.
I'm attaching a second patch, to be applied on top of the first one, that fixes
the issue.
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index b37e3965a..5ddf0ebd1 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
(output
(let* ((tmp-file (org-babel-temp-file "R-")))
(with-temp-file tmp-file
- (insert (concat body "\n" org-babel-R-eoe-indicator)))
+ (insert body))
(with-current-buffer session
(let* ((process (get-buffer-process (current-buffer)))
(string-buffer "")
@@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
(concat string-buffer text)))
comint-output-filter-functions)))
(ess-send-string
- process (format "source('%s', print.eval=TRUE)"
- (org-babel-process-file-name tmp-file 'noquote)))
+ process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+ (org-babel-process-file-name tmp-file 'noquote)
+ org-babel-R-eoe-indicator))
(while (not (string-match (regexp-quote org-babel-R-eoe-output)
string-buffer))
(accept-process-output process))
Re: Help debugging R source code block output problem with :session
Hi Chuck and Dylan,
"Berry, Charles" writes:
> This problem has been bugging people for years and previous attempts to solve
> it have always run up against creating more problems in the process of
> solving this one.
>
> If you do decide to dig into solving this, please be sure that remote
> sessions and graphical outputs are not broken. test-ob-R.el does not cover
> those cases. In fact, it is pretty short, so there are probably other things
> that could break without `make test' complaining.
I think I have found a robust solution to this issue, which I've
attached.
The solution avoids parsing for prompts at all, thus it should be robust
even for the case when the beginning of an output line looks like a
prompt.
I tested that the unit tests all pass, and that remote sessions and
graphical outputs still work. I also added Dylan's examples as unit
tests.
Cheers,
Jack
>From 76d0eaa31506ce8a2f81f64eae43161db5721317 Mon Sep 17 00:00:00 2001
From: Jack Kamm
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts
* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.
Fixes issue reported in
https://orgmode.org/list/875zgjh8wn@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
lisp/ob-R.el | 37 +++--
testing/lisp/test-ob-R.el | 13 +
2 files changed, 32 insertions(+), 18 deletions(-)
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..b37e3965a 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
(declare-function ess-eval-buffer "ext:ess-inf" (vis))
(declare-function ess-wait-for-process "ext:ess-inf"
(&optional proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+ (process string &optional visibly message _type))
(defconst org-babel-header-args:R
'((width . :any)
@@ -437,24 +439,23 @@ (defun org-babel-R-evaluate-session
(org-babel-import-elisp-from-file tmp-file '(16)))
column-names-p)))
(output
- (mapconcat
- 'org-babel-chomp
- (butlast
- (delq nil
- (mapcar
- (lambda (line) (when (> (length line) 0) line))
- (mapcar
- (lambda (line) ;; cleanup extra prompts left in output
- (if (string-match
- "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
- (car (split-string line "\n")))
- (substring line (match-end 1))
- line))
- (org-babel-comint-with-output (session org-babel-R-eoe-output)
- (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
- (inferior-ess-send-input)) "\n"
+ (let* ((tmp-file (org-babel-temp-file "R-")))
+ (with-temp-file tmp-file
+ (insert (concat body "\n" org-babel-R-eoe-indicator)))
+ (with-current-buffer session
+ (let* ((process (get-buffer-process (current-buffer)))
+ (string-buffer "")
+ (comint-output-filter-functions
+ (cons (lambda (text) (setq string-buffer
+ (concat string-buffer text)))
+ comint-output-filter-functions)))
+ (ess-send-string
+ process (format "source('%s', print.eval=TRUE)"
+ (org-babel-process-file-name tmp-file 'noquote)))
+ (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+ string-buffer))
+ (accept-process-output process))
+ (substring string-buffer 0 (match-beginning 0
(defun org-babel-R-process-value-result (result column-names-p)
"R-specific processing of return value.
diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el
index 7ce340ba4..ff7ea19d5 100644
--- a/testing/lisp/test-ob-R.el
+++ b/testing/lisp/test-ob-R.el
@@ -97,6 +97,19 @@ (ert-deftest test-ob-R/results-file ()
(org-babel-goto-named-result "TESTSRC") (forward-line 1)
(should (string= "[[file:junk/test.org]]"
(buffer-substring-no-properties (point-at-bol) (point-at-eol)))
+
+(ert-deftest test-ob-R/prompt-output ()
+ (let (ess-ask-for-ess-directory ess-history-file)
+(org-test-with-temp-text
+ "#+begin_src R :results output :session\nprint(\" \")\nprint(\"one three\")\nprint(\"end\")\n#+end_src\n"
+ (should (string= "[1] \" \"\n[1] \"one three\"\n[1] \"end\"\n" (org-babel-execute-src-block))
+
+(ert-deftest test-ob-R/output-nonprinted ()
+ (let (ess-ask-for-ess-directory ess-history-file)
+(org-test-with-temp-text
+ "#+begin_src R :results output :session\n4.0 * 3.5\nlog(10)\nlog10(10)\n(3 + 1) * 5\n3^-1\n1/0\n#+end_src\n"
+ (should (string= "[1] 14\n[1] 2.302585\n[1] 1\n[1] 20\n[
Re: Help debugging R source code block output problem with :session
Hi Chuck, "Berry, Charles" writes: > This problem has been bugging people for years and previous attempts to solve > it have always run up against creating more problems in the process of > solving this one. > > If you do decide to dig into solving this, please be sure that remote > sessions and graphical outputs are not broken. test-ob-R.el does not cover > those cases. In fact, it is pretty short, so there are probably other things > that could break without `make test' complaining. Thanks for this info, it's helpful. I do want to dig deeper into this, but probably won't have bandwidth in the near future. But I've captured your note for when I return to this. I'll be sure to test that any solution works with remote sessions and graphics. Best, Jack
Re: Help debugging R source code block output problem with :session
Thanks for the workaround idea of grabbing the results as a
string.
I'm not sure why this issue cropped up for me right now when I did
not run into it in previous years. But I have run into other
:results output issues in the past when ess changed.
Thanks,
Dylan
On Sat 29 Aug 2020 at 15:35, Berry, Charles
wrote:
This problem has been bugging people for years and previous
attempts to solve it have always run up against creating more
problems in the process of solving this one.
This workaround gives the same results with or without `:session
"NEW"' and same as OPs first src block:
#+header: :prologue "capture.output({" :epilogue "})"
#+begin_src R :results value verbatim :session "NEW"
print(" ")
print("one three")
print("end")
#+end_src
If you do decide to dig into solving this, please be sure that
remote sessions and graphical outputs are not broken.
test-ob-R.el does not cover those cases. In fact, it is pretty
short, so there are probably other things that could break
without `make test' complaining.
HTH,
Chuck
On Aug 29, 2020, at 12:24 AM, Jack Kamm
wrote:
Hi Dylan,
The patch does fix that issue -- but it introduces a different
bug
for code blocks with ~:session~: the R block now only produces
output from the last statement evaluated.
Of course, you're right. Good catch.
Here's another attempt. It fixes the issue by modifying the R
comint
regular expression, requiring it to match at the beginning of
the line.
--
Re: Help debugging R source code block output problem with :session
Yes, the modified comint-prompt-regexp works for my use cases and I'm back to expected output and what I had last time I taught the course. This was my instinct for a simple solution as well, but I've not tested against remote sessions and graphical outputs as Charles Berry warns. I'll take a look at your ob-session-async, thanks! -Dylan On Sat 29 Aug 2020 at 02:24, Jack Kamm wrote: Hi Dylan, The patch does fix that issue -- but it introduces a different bug for code blocks with ~:session~: the R block now only produces output from the last statement evaluated. Of course, you're right. Good catch. Here's another attempt. It fixes the issue by modifying the R comint regular expression, requiring it to match at the beginning of the line. I think this should fix most cases, including the examples you sent. Still, it's not totally robust -- for example, it will still mangle multiline strings, if one of the lines starts with a substring that looks like a prompt. I'd be interested to hear if the attached patch works for the common cases you encounter, such as with tibbles. As an aside, I personally use an alternative implementation of ob-R sessions that doesn't suffer from this issue, and also provides some other benefits like async evaluation [1]. I'm planning to submit these changes to org-mode someday, but am not ready yet. But you may want to check it out, it was able to solve the issue in the other thread I linked as well. [1] https://github.com/jackkamm/ob-session-async From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001 From: Jack Kamm Date: Sat, 29 Aug 2020 00:07:58 -0700 Subject: [PATCH] ob-R: Fix prompt mangling in session output * lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt regexp to start at beginning of line, to prevent org-babel-comint-with-output from splitting mid-line. Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and https://orgmode.org/list/87r1rqled0.fsf@havana/ --- lisp/ob-R.el | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lisp/ob-R.el b/lisp/ob-R.el index 5e9d35f58..10b3b6fe3 100644 --- a/lisp/ob-R.el +++ b/lisp/ob-R.el @@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session (car (split-string line "\n"))) (substring line (match-end 1)) line)) - (org-babel-comint-with-output (session org-babel-R-eoe-output) -(insert (mapconcat 'org-babel-chomp -(list body org-babel-R-eoe-indicator) - "\n")) -(inferior-ess-send-input)) "\n" + (with-current-buffer session + (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp))) + (org-babel-comint-with-output (session org-babel-R-eoe-output) +(insert (mapconcat 'org-babel-chomp + (list body org-babel-R-eoe-indicator) + "\n")) +(inferior-ess-send-input "\n" (defun org-babel-R-process-value-result (result column-names-p) "R-specific processing of return value. --
Re: Help debugging R source code block output problem with :session
This problem has been bugging people for years and previous attempts to solve
it have always run up against creating more problems in the process of solving
this one.
This workaround gives the same results with or without `:session "NEW"' and
same as OPs first src block:
#+header: :prologue "capture.output({" :epilogue "})"
#+begin_src R :results value verbatim :session "NEW"
print(" ")
print("one three")
print("end")
#+end_src
If you do decide to dig into solving this, please be sure that remote sessions
and graphical outputs are not broken. test-ob-R.el does not cover those cases.
In fact, it is pretty short, so there are probably other things that could
break without `make test' complaining.
HTH,
Chuck
> On Aug 29, 2020, at 12:24 AM, Jack Kamm wrote:
>
> Hi Dylan,
>
>> The patch does fix that issue -- but it introduces a different bug
>> for code blocks with ~:session~: the R block now only produces
>> output from the last statement evaluated.
>
> Of course, you're right. Good catch.
>
> Here's another attempt. It fixes the issue by modifying the R comint
> regular expression, requiring it to match at the beginning of the line.
Re: Help debugging R source code block output problem with :session
Hi Dylan, > The patch does fix that issue -- but it introduces a different bug > for code blocks with ~:session~: the R block now only produces > output from the last statement evaluated. Of course, you're right. Good catch. Here's another attempt. It fixes the issue by modifying the R comint regular expression, requiring it to match at the beginning of the line. I think this should fix most cases, including the examples you sent. Still, it's not totally robust -- for example, it will still mangle multiline strings, if one of the lines starts with a substring that looks like a prompt. I'd be interested to hear if the attached patch works for the common cases you encounter, such as with tibbles. As an aside, I personally use an alternative implementation of ob-R sessions that doesn't suffer from this issue, and also provides some other benefits like async evaluation [1]. I'm planning to submit these changes to org-mode someday, but am not ready yet. But you may want to check it out, it was able to solve the issue in the other thread I linked as well. [1] https://github.com/jackkamm/ob-session-async >From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001 From: Jack Kamm Date: Sat, 29 Aug 2020 00:07:58 -0700 Subject: [PATCH] ob-R: Fix prompt mangling in session output * lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt regexp to start at beginning of line, to prevent org-babel-comint-with-output from splitting mid-line. Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and https://orgmode.org/list/87r1rqled0.fsf@havana/ --- lisp/ob-R.el | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lisp/ob-R.el b/lisp/ob-R.el index 5e9d35f58..10b3b6fe3 100644 --- a/lisp/ob-R.el +++ b/lisp/ob-R.el @@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session (car (split-string line "\n"))) (substring line (match-end 1)) line)) - (org-babel-comint-with-output (session org-babel-R-eoe-output) - (insert (mapconcat 'org-babel-chomp -(list body org-babel-R-eoe-indicator) -"\n")) - (inferior-ess-send-input)) "\n" + (with-current-buffer session + (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp))) + (org-babel-comint-with-output (session org-babel-R-eoe-output) + (insert (mapconcat 'org-babel-chomp + (list body org-babel-R-eoe-indicator) + "\n")) + (inferior-ess-send-input "\n" (defun org-babel-R-process-value-result (result column-names-p) "R-specific processing of return value. -- 2.28.0
Re: Help debugging R source code block output problem with :session
Hi Jack, The patch does fix that issue -- but it introduces a different bug for code blocks with ~:session~: the R block now only produces output from the last statement evaluated. #+begin_src R :results output :exports both :session 4.0 * 3.5 log(10) # natural log log10(10) (3 + 1) * 5 3^-1 1/0 #+end_src #+results: : [1] Inf Without the ~:session header~ the output is as it was before the patch: #+results: : [1] 14 : [1] 2.302585 : [1] 1 : [1] 20 : [1] 0.333 : [1] Inf -Dylan
Re: Help debugging R source code block output problem with :session
Hi Dylan,
I'm able to reproduce your error. I think this same bug has been
previously reported at [1]. As discussed there, the reason is that
substrings ending in ">" get stripped out by
org-babel-comint-with-output because it thinks they are prompts (in
particular, they match comint-prompt-regexp).
I'm attaching a patch to solve the issue. It replaces
org-babel-comint-with-output, with an R call to capture.output. I think
this approach is simpler and more robust, since it doesn't need to do
any parsing to remove prompts or other unwanted outputs.
I haven't committed to ob-R.el before, so thought it would be best to
solicit feedback here before merging this in.
Cheers,
Jack
[1] https://orgmode.org/list/875zgjh8wn@gmail.com/
>From 1dc8e2d2cb01a4e6b82342ea8d8c965df8f5222c Mon Sep 17 00:00:00 2001
From: Jack Kamm
Date: Fri, 28 Aug 2020 19:16:05 -0700
Subject: [PATCH] ob-R: Fix prompt mangling in session output
* lisp/ob-R.el (org-babel-R-evaluate-session): Replace
call to org-babel-comint-with-output with ess-send-to-buffer and
additional wrapper R code.
Fixes https://orgmode.org/list/875zgjh8wn@gmail.com/ and
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
lisp/ob-R.el | 79
1 file changed, 36 insertions(+), 43 deletions(-)
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..d69cf23db 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -412,49 +412,42 @@ (defun org-babel-R-evaluate-session
If RESULT-TYPE equals `output' then return standard output as a
string. If RESULT-TYPE equals `value' then return the value of the
last statement in BODY, as elisp."
- (cl-case result-type
-(value
- (with-temp-buffer
- (insert (org-babel-chomp body))
- (let ((ess-local-process-name
- (process-name (get-buffer-process session)))
- (ess-eval-visibly-p nil))
- (ess-eval-buffer nil)))
- (let ((tmp-file (org-babel-temp-file "R-")))
- (org-babel-comint-eval-invisibly-and-wait-for-file
- session tmp-file
- (format org-babel-R-write-object-command
- (if row-names-p "TRUE" "FALSE")
- (if column-names-p
- (if row-names-p "NA" "TRUE")
- "FALSE")
- ".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
- (org-babel-R-process-value-result
- (org-babel-result-cond result-params
- (with-temp-buffer
- (insert-file-contents tmp-file)
- (org-babel-chomp (buffer-string) "\n"))
- (org-babel-import-elisp-from-file tmp-file '(16)))
- column-names-p)))
-(output
- (mapconcat
- 'org-babel-chomp
- (butlast
- (delq nil
- (mapcar
- (lambda (line) (when (> (length line) 0) line))
- (mapcar
- (lambda (line) ;; cleanup extra prompts left in output
- (if (string-match
- "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
- (car (split-string line "\n")))
- (substring line (match-end 1))
- line))
- (org-babel-comint-with-output (session org-babel-R-eoe-output)
- (insert (mapconcat 'org-babel-chomp
-(list body org-babel-R-eoe-indicator)
-"\n"))
- (inferior-ess-send-input)) "\n"
+ (let ((ess-local-process-name
+ (process-name (get-buffer-process session)))
+ (ess-eval-visibly-p nil))
+(cl-case result-type
+ (value
+ (with-temp-buffer
+ (insert (org-babel-chomp body))
+ (ess-eval-buffer nil))
+ (let ((tmp-file (org-babel-temp-file "R-")))
+ (org-babel-comint-eval-invisibly-and-wait-for-file
+ session tmp-file
+ (format org-babel-R-write-object-command
+ (if row-names-p "TRUE" "FALSE")
+ (if column-names-p
+ (if row-names-p "NA" "TRUE")
+ "FALSE")
+ ".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
+ (org-babel-R-process-value-result
+ (org-babel-result-cond result-params
+ (with-temp-buffer
+ (insert-file-contents tmp-file)
+ (org-babel-chomp (buffer-string) "\n"))
+ (org-babel-import-elisp-from-file tmp-file '(16)))
+ column-names-p)))
+ (output
+ (let ((tmp-file (org-babel-temp-file "R-")))
+ (with-temp-buffer
+ (insert (format "capture.output({
+%s
+}, file='%s')"
+ body tmp-file))
+ (ess-eval-buffer))
+ (ess-wait-for-process)
+ (with-temp-buffer
+ (insert-file-contents tmp-file)
+ (buffer-string)))
(defun org-babel-R-process-value-result (result column-names-p)
"R-specific processing of return value.
--
2.28.0
Help debugging R source code block output problem with :session
I returned to an org file this week and found that I am getting
some strange source code block output for R code when I use the
:session header. I have been able to duplicate this with a minimal
init file. I strongly suspect it is some problem at my end but
perhaps folks here can help me know here to look? It occurs with a
minimal emacs initialization.
First, here is a small org file example of the problem. In the
second results block the ">" appears to be treated as a prompt
line to strip from output but only when a session is started. This
happens with any ">" in output when the :session header occurs.
the problem is dramatic when printing tibbles where the column
modes are enclosed in angle brackets.
test.org
** Test org babel R output prompt problem
#+begin_src R :results output
print(" ")
print("one three")
print("end")
#+end_src
#+RESULTS:
: [1] " "
: [1] "one three"
: [1] "end"
#+begin_src R :results output :session "NEW"
print(" ")
print("one three")
print("end")
#+end_src
#+RESULTS:
: [1] "<
: <
: "
: [1] "one <
: three"
: [1] "end"
<<<
The second block produces the bad output. This occurs with a
minimal setup.
I start emacs with
emacs -Q -l "~/debugorg.el"
where debugorg.el is:
require 'package)
add-to-list 'package-archives '("melpa" .
"https://melpa.org/packages/";) t)
add-to-list 'package-archives '("org" .
"https://orgmode.org/elpa/";) t)
package-initialize)
require 'ess-site)
(org-babel-do-load-languages
'org-babel-load-languages
'((R . t)))
<<
emacs version 26.3
org-mode version 9.3.7
ess version 18.10.3snapshot
--
