Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

> [...]
 > > > >
 > > > [...]
 > > > >
 > > > > I've wondered if there is not a better way for Babel to share an
 > > > > interactive session with the user.  For instance, if we wanted to
 > > > > support a new form of results processing on addition to value and
 > > > > output Namely, "transcript", wherein the results of evaluating a
 > > > > source block would be a transcript of the season with statements
 > > > > interwoven with their respective outputs I suspect that this would
 > > > > not be easy extension of the current approach since it would require
 > > > > parsing the source into statements that get evaluated sequentially with
 > > > > visible results echoed into the transcript.
 > > >
 > > > You can do stuff like this using babel and some R code.
 > > >
 > > > Here is a start:
 > > >
 > 
 > [snip]
 > 
 > > Sourcing a textConnection on a :noweb interpolated block will not
 > > handle embedded quotes in the source block correctly.  Adding an
 > > assignment of a string to a variable in my-block reveals this
 > > (i.e. `b<-"asdf"`)
 > 
 > I know. That is why I said it is a 'start'. You can work around this
 > with more Babel if that is the only issue. Write a src block `good-fmt'
 > that will render the body of another block as you want it and then use
 > <> to insert it.
 > 
 > 
 > > Nor does it extend to my underspecified conception of what
 > > :transcript output would be.  I intended that :transcript would
 > > generate a colorized source blocks separated by results for
 > > statements which generated visible results.  You implementation makes
 > > the source and results undifferentiated.  My mistake for
 > > underspecifying my intention.  I think I might be able to cobble what
 > > I want using the 'evaluate' package
 > > (https://cran.rstudio.com/web/packages/evaluate/evaluate.pdf) with an
 > > output handler to format source as an R code block and results as R
 > > results block.
 > 
 > See Aaron Ecay's patches from around 08/2014. And discussions on this
 > list from about that time.  He had some of this working, but there
 > were some issues about handling remote calls, IIRC.
 > 
 > > Probably not worth the effort.  Or rather, probably
 > > already done within the knitr/rmarkdown.
 > 
 > By default knitr interlaces the code and output in a frame with
 > the output lines prefixed by '##'. The code is highlighted by default.
 > 
 > In *.Rnw this leads to a single block with a background color and
 > colorized code.
 > 
 > In *.Rhtml you get source code blocks colored per 
 > and results blocks uncolored per .
 > 
 > Using *.org via ox-ravel --> *.Rhtml via knitr --> *.html will get you
 > that far.
 > 

Indeed it does.  I have previously used ox-ravel to superb effect allowing me 
to compose R lessons as *.org and distribute as *.Rmd for use within RStudio 
server.  I must remember to remember this option!

 > Maybe customizing the classes to your taste will finish it.
 > 
 > Best,
 > 
 > Chuck
 > 
 > p.s. If you go with ox-ravel, I recommend the ravel-lang branch:
 > https://github.com/chasberry/orgmode-accessories/tree/ravel-lang 

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

 > -Original Message-
 > From: Charles C. Berry [mailto:ccbe...@ucsd.edu]
 > Sent: Saturday, November 07, 2015 6:33 PM
 > To: Cook, Malcolm 
 > Cc: emacs-orgmode@gnu.org
 > Subject: Re: BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-
 > session over aggressively performs "; ; cleanup extra prompts left in output"
 > and a possible workaround
 > 
 > On Sat, 7 Nov 2015, Cook, Malcolm wrote:
 > 
 > > Thanks Chuck for setting this through.
 > >
 > [...]
 > >
 > > I've wondered if there is not a better way for Babel to share an
 > > interactive session with the user.  For instance, if we wanted to
 > > support a new form of results processing on addition to value and
 > > output Namely, "transcript", wherein the results of evaluating a
 > > source block would be a transcript of the season with statements
 > > interwoven with their respective outputs I suspect that this would
 > > not be easy extension of the current approach since it would require
 > > parsing the source into statements that get evaluated sequentially with
 > > visible results echoed into the transcript.
 > 
 > You can do stuff like this using babel and some R code.
 > 
 > Here is a start:
 > 
 > --8<---cut here---start->8---
 > #+NAME: my-block
 > #+BEGIN_SRC R :eval no
 > ls()
 > a <- 1
 > sqrt(a+2)
 > ls()
 > #+END_SRC
 > 
 > 
 > #+BEGIN_SRC R :session :noweb yes :results value raw :wrap example
 >  capture.output(
 >  source(textConnection("
 ><>
 >"),
 >echo=T,print=T))
 > #+END_SRC
 > 
 > #+RESULTS:
 > #+BEGIN_example
 > 
 > > ls()
 > [1] "a"
 > 
 > > a <- 1
 > 
 > > sqrt(a+2)
 > [1] 1.732051
 > 
 > > ls()
 > [1] "a"
 > #+END_example
 > 
 > 
 > --8<---cut here---end--->8---
 > 
 > I suppose you would want `:exports results'.

Hi Chuck,

Sourcing a textConnection on a :noweb interpolated block will not handle 
embedded quotes in the source block correctly.  Adding an assignment of a 
string to a variable in my-block reveals this (i.e. `b<-"asdf"`)

Nor does it extend to my underspecified conception of what :transcript output 
would be.  I  intended that :transcript would generate a colorized source 
blocks separated by results for statements which generated visible results.  
You implementation makes the source and results undifferentiated.   My mistake 
for underspecifying my intention.  I think I might be able to cobble what I 
want using the 'evaluate' package 
(https://cran.rstudio.com/web/packages/evaluate/evaluate.pdf) with an output 
handler to format source as an R code block and results as R results block.  
Probably not worth the effort.  Or rather, probably already done within the 
knitr/rmarkdown.

Thanks,

Malcolm

 > 
 > Best,
 > Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Mon, 9 Nov 2015, Cook, Malcolm wrote:
[...]

> >
> [...]
> >
> > I've wondered if there is not a better way for Babel to share an
> > interactive session with the user.  For instance, if we wanted to
> > support a new form of results processing on addition to value and
> > output Namely, "transcript", wherein the results of evaluating a
> > source block would be a transcript of the season with statements
> > interwoven with their respective outputs I suspect that this would
> > not be easy extension of the current approach since it would require
> > parsing the source into statements that get evaluated sequentially with
> > visible results echoed into the transcript.
>
> You can do stuff like this using babel and some R code.
>
> Here is a start:
>


[snip]


Sourcing a textConnection on a :noweb interpolated block will not
handle embedded quotes in the source block correctly.  Adding an
assignment of a string to a variable in my-block reveals this
(i.e. `b<-"asdf"`)


I know. That is why I said it is a 'start'. You can work around this
with more Babel if that is the only issue. Write a src block `good-fmt' 
that will render the body of another block as you want it and then use

<> to insert it.



Nor does it extend to my underspecified conception of what
:transcript output would be.  I intended that :transcript would
generate a colorized source blocks separated by results for
statements which generated visible results.  You implementation makes
the source and results undifferentiated.  My mistake for
underspecifying my intention.  I think I might be able to cobble what
I want using the 'evaluate' package
(https://cran.rstudio.com/web/packages/evaluate/evaluate.pdf) with an
output handler to format source as an R code block and results as R
results block.


See Aaron Ecay's patches from around 08/2014. And discussions on this
list from about that time.  He had some of this working, but there
were some issues about handling remote calls, IIRC.


Probably not worth the effort.  Or rather, probably
already done within the knitr/rmarkdown.


By default knitr interlaces the code and output in a frame with
the output lines prefixed by '##'. The code is highlighted by default.

In *.Rnw this leads to a single block with a background color and
colorized code.

In *.Rhtml you get source code blocks colored per 
and results blocks uncolored per .

Using *.org via ox-ravel --> *.Rhtml via knitr --> *.html will get you 
that far.


Maybe customizing the classes to your taste will finish it.

Best,

Chuck

p.s. If you go with ox-ravel, I recommend the ravel-lang branch:
https://github.com/chasberry/orgmode-accessories/tree/ravel-lang

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Nicolas Goaziou]

"Charles C. Berry"  writes:

> On Thu, 5 Nov 2015, Nicolas Goaziou wrote:
> [...]
>> Would you want to provide a patch for that (and commit it while you're
>> at it)?
>>
>
> Done.

Thank you.

Regards,

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Thu, 5 Nov 2015, Nicolas Goaziou wrote:
[...]

Would you want to provide a patch for that (and commit it while you're
at it)?



Done.

Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Sat, 7 Nov 2015, Cook, Malcolm wrote:


Thanks Chuck for setting this through.


[...]


I've wondered if there is not a better way for Babel to share an 
interactive session with the user.  For instance, if we wanted to 
support a new form of results processing on addition to value and 
output Namely, "transcript", wherein the results of evaluating a 
source block would be a transcript of the season with statements 
interwoven with their respective outputs I suspect that this would 
not be easy extension of the current approach since it would require 
parsing the source into statements that get evaluated sequentially with 
visible results echoed into the transcript.


You can do stuff like this using babel and some R code.

Here is a start:

--8<---cut here---start->8---
#+NAME: my-block
#+BEGIN_SRC R :eval no
ls()
a <- 1
sqrt(a+2)
ls()
#+END_SRC


#+BEGIN_SRC R :session :noweb yes :results value raw :wrap example
capture.output(
source(textConnection("
  <>
  "),
  echo=T,print=T))
#+END_SRC

#+RESULTS:
#+BEGIN_example


ls()

[1] "a"


a <- 1



sqrt(a+2)

[1] 1.732051


ls()

[1] "a"
#+END_example


--8<---cut here---end--->8---

I suppose you would want `:exports results'.

Best,
Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

Thanks Chuck for setting this through.

I've never been happy at how Babel arranged the inter process communication 
with R  including the need for cleaning up the output with scary regular 
expressions like this.

Also there are some of uses of temp files that sometime cause of bugs 
especially when running R under tramp on a remote host. I'll be sure to report 
it next time I see it.

I've wondered if there is not a better way for Babel to share an interactive 
session with the user.   For instance, if we wanted to support a new form of 
results processing on addition to value and output Namely, "transcript", 
wherein the results of evaluating a source block would be a transcript of the 
season with statements interwoven with their respective outputs I suspect 
that this would not be easy extension of the current approach since it would 
require parsing the source into statements that get evaluated sequentially with 
visible results echoed into the transcript.

But I digress.

Thanks again for running with my workaround

On Nov 7, 2015 3:31 PM, "Charles C. Berry"  wrote:
On Thu, 5 Nov 2015, Nicolas Goaziou wrote:
[...]
> Would you want to provide a patch for that (and commit it while you're
> at it)?
>

Done.

Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Thu, 5 Nov 2015, Nicolas Goaziou wrote:


Hello,

"Charles C. Berry"  writes:


I have traced this back to its source, and the culprit appears to be me.

See
  commit dc92eaa08d89e4bc9556f868ae65633196157a8d
  Author: Eric Schulte 
  Date:   Wed Jul 28 08:05:30 2010 -0600
and subsequent modifications.

I believe the leading blank(s) in the regex were copied from ess, but
nowadays I only see this in code in ess-traceback.el that is commented
out.

So I guess there would be no harm in pushing the latter regex.


Would you want to provide a patch for that (and commit it while you're
at it)?


Yes, but Babel seems to be broken - at least on my box - by this

commit 40356ae3765d123fd9950a961718530219fa5cb8
Author: Aaron Ecay 
Date:   Thu Nov 5 15:51:06 2015 +

Once that is resolved, I'll do something.

Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Nicolas Goaziou]

Hello,

"Charles C. Berry"  writes:

> I have traced this back to its source, and the culprit appears to be me.
>
> See
>   commit dc92eaa08d89e4bc9556f868ae65633196157a8d
>   Author: Eric Schulte 
>   Date:   Wed Jul 28 08:05:30 2010 -0600
> and subsequent modifications.
>
> I believe the leading blank(s) in the regex were copied from ess, but
> nowadays I only see this in code in ess-traceback.el that is commented
> out.
>
> So I guess there would be no harm in pushing the latter regex.

Would you want to provide a patch for that (and commit it while you're
at it)?

Thank you for the investigation.


Regards,

-- 
Nicolas Goaziou

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Thu, 1 Oct 2015, Cook, Malcolm wrote:


I am not sure what the best solution is, but, in my hands using

> > Org-mode version 8.3.2-elpa org-20150929 the reg-expt used to "cleanup
> > extra prompts left in output" is over-aggressive and will trim session
> > :output at lines consisting exclusively of blanks and periods such as
> > produced when printing a BioConductor 'Views' object which wants to
> > appear as
>


[snip]


I offer as a possible workaround the following:

> >
> > So far, I have had good success having removed provision for allowing
> leading whitespace by changing the regexp  org-babel-R-evaluate-session
> from
> >  "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
> > to
> >  "^\\([>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
> >
> > But I don't know all the test cases so, YMMV


I have traced this back to its source, and the culprit appears to be me.

See
  commit dc92eaa08d89e4bc9556f868ae65633196157a8d
  Author: Eric Schulte 
  Date:   Wed Jul 28 08:05:30 2010 -0600
and subsequent modifications.

I believe the leading blank(s) in the regex were copied from ess, but 
nowadays I only see this in code in ess-traceback.el that is commented 
out.


So I guess there would be no harm in pushing the latter regex.

Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

> On Thu, 1 Oct 2015, Cook, Malcolm wrote:
 > 
 > >>> I am not sure what the best solution is, but, in my hands using
 > > > > Org-mode version 8.3.2-elpa org-20150929 the reg-expt used to
 > > > > "cleanup extra prompts left in output" is over-aggressive and will
 > > > > trim session :output at lines consisting exclusively of blanks and
 > > > > periods such as produced when printing a BioConductor 'Views'
 > > > > object which wants to appear as
 > > >
 > 
 > [snip]
 > 
 > >>> I offer as a possible workaround the following:
 > > > >
 > > > > So far, I have had good success having removed provision for
 > > > > allowing
 > > > leading whitespace by changing the regexp
 > > > org-babel-R-evaluate-session from
 > > > >  "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
 > > > > to
 > > > >  "^\\([>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
 > > > >
 > > > > But I don't know all the test cases so, YMMV
 > 
 > I have traced this back to its source, and the culprit appears to be me.
 > 
 > See
 >commit dc92eaa08d89e4bc9556f868ae65633196157a8d
 >Author: Eric Schulte 
 >Date:   Wed Jul 28 08:05:30 2010 -0600
 > and subsequent modifications.
 > 
 > I believe the leading blank(s) in the regex were copied from ess, but 
 > nowadays
 > I only see this in code in ess-traceback.el that is commented out.
 > 
 > So I guess there would be no harm in pushing the latter regex.


Hi Chuck.  Thanks for the history lesson, and taking this on in the first 
place.  Yay!~Malcolm

 > 
 > Chuck

[O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

Hello ,

I am not sure what the best solution is, but, in my hands using Org-mode 
version 8.3.2-elpa org-20150929 the reg-expt  used to "cleanup extra prompts 
left in output" is over-aggressive and will trim session :output at lines 
consisting exclusively of blanks and periods such as produced when printing a 
BioConductor 'Views' object which wants to appear as

#+RESULTS:
#+begin_example
  Views on a 23011544-letter DNAString subject
subject: 
CGACAATGCACGACAGAGGAAGCAGAACAGATATTTAGATTGCCTCTCACTCTCCCATATTATAGGGAGAAATATGATCGCGTATGCGAGAGTAGTGCCAACATATTGTGCTCTTTGATTGGCAACCCTGGTGGCGGATGAACGAGATGATAATATATTCAAGTTGCCGCTAATCAGAAATAAATTCATTGCAACGTTAAATACAGCACAATATATGATCGCGTATGCGAGAGTAGTGCCAACATATTGTGCTAATGAGTGCCTCTCGTTCTCTGTCTTATA...TATCTTTCAAAGATGACACTAGCAATGCGTTAACCCAAATAATGATTTCCCTAAATCCTTCCGTAAATATTAACTGGCTCCACCCAAATTTCGGTCATTAAATAATCAGTTGCCACAACTAATTGTCTGTGGAATGTCATATCTCGATGAGCTCATAATTAAATTTACAATCAAACTGTGTTCGAGAGCTAACATTTGGCATATTTGCAAAGATGAACCTTTCAAA
views:
 start  end width
  [1]   344766   344773 8 [CATGAGGC]
  [2]   563564   563571 8 [CATGAGGC]
  [3]   641027   641034 8 [CATGAGGC]
  [4]   656168   656175 8 [CATGAGGC]
  [5]   709112   709119 8 [CATGAGGC]
  ...  ...  ...   ... ...
[141] 22209984 22209991 8 [CATGAGGC]
[142] 22371543 22371550 8 [CATGAGGC]
[143] 22554991 22554998 8 [CATGAGGC]
[144] 22618578 22618585 8 [CATGAGGC]
[145] 22897728 22897735 8 [CATGAGGC]
#+end_example

But alas rather appears as:

#+RESULTS:
#+begin_example
  ...  ...  ...   ... ...
[141] 22209984 22209991 8 [CATGAGGC]
[142] 22371543 22371550 8 [CATGAGGC]
[143] 22554991 22554998 8 [CATGAGGC]
[144] 22618578 22618585 8 [CATGAGGC]
[145] 22897728 22897735 8 [CATGAGGC]
#+end_example


I offer as a possible workaround the following:

So far, I have had good success having removed provision for allowing leading 
whitespace by changing the regexp  org-babel-R-evaluate-session  from
  "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
 to
  "^\\([>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"

But I don't know all the test cases so, YMMV

HTH,

~Malcolm (who, FWIW, has never really liked the way ob-R communicated with the 
inferior R session in the first place)

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Charles C. Berry]

On Thu, 1 Oct 2015, Cook, Malcolm wrote:


Hello ,



I am not sure what the best solution is, but, in my hands using Org-mode 
version 8.3.2-elpa org-20150929 the reg-expt used to "cleanup extra 
prompts left in output" is over-aggressive and will trim session :output 
at lines consisting exclusively of blanks and periods such as produced 
when printing a BioConductor 'Views' object which wants to appear as


I'd try (for example):

#+BEGIN_SRC R :results value raw drawer :session
require(GenomicRanges)
capture.output(show(GRanges(seqnames=rep("a",100),
IRanges(start=1:100,width=1),strand=rep("*",100
#+END_SRC

rather than messing with the regexp.




I offer as a possible workaround the following:

So far, I have had good success having removed provision for allowing leading 
whitespace by changing the regexp  org-babel-R-evaluate-session  from
 "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
to
 "^\\([>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"

But I don't know all the test cases so, YMMV


Just guessing that the things that `(setq ess-eval-visibly nil)' generates 
(e.g., + + + + + > ) won't behave using your version.


HTH,

Chuck

Re: [O] BUG: emacs orgmode ob-R.el function org-babel-R-evaluate-session over aggressively performs "; ; cleanup extra prompts left in output" and a possible workaround

[Cook, Malcolm]

> > I am not sure what the best solution is, but, in my hands using
 > > Org-mode version 8.3.2-elpa org-20150929 the reg-expt used to "cleanup
 > > extra prompts left in output" is over-aggressive and will trim session
 > > :output at lines consisting exclusively of blanks and periods such as
 > > produced when printing a BioConductor 'Views' object which wants to
 > > appear as
 > 
 > I'd try (for example):
 > 
 > #+BEGIN_SRC R :results value raw drawer :session
 > require(GenomicRanges)
 > capture.output(show(GRanges(seqnames=rep("a",100),
 >  IRanges(start=1:100,width=1),strand=rep("*",100
 > #+END_SRC

Thanks for the thought, but, I'm trying to engage in "literate programming" for 
which in my opinion I should not have to re-write my code (using capture.output 
and show) to get the output of evaluation captured.

We're I to sweave my code I would get all this un-wanted cruft.

I think org-bable-R-evaluate-FOO should be handling all this for us.

 > rather than messing with the regexp.

Hmm not sure I understand your preference here.   I'm providing an example 
of some that should "just work", but does not.   I too prefer not to mess with 
the regexp other than to identify it as the source of the issue.  Perhaps the 
"real solution" is to recode org-bable-R-evaluate-FOO.

> > I offer as a possible workaround the following:
 > >
 > > So far, I have had good success having removed provision for allowing
 > leading whitespace by changing the regexp  org-babel-R-evaluate-session
 > from
 > >  "^\\([ ]*[>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
 > > to
 > >  "^\\([>+\\.][ ]?\\)+\\([[0-9]+\\|[ ]\\)"
 > >
 > > But I don't know all the test cases so, YMMV
 > 
 > Just guessing that the things that `(setq ess-eval-visibly nil)' generates 
 > (e.g., +
 > + + + + > ) won't behave using your version.
 > 

In fact it does behave.  I tried it.  I think that ess-eval-visibly effects how 
ob-R interacts with the ess session.

 > HTH,

I wish it did ;)  Thanks for your efforts.

 > 
 > Chuck