Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Eric Schulte]

Eric Schulte  writes:

> Aaron Ecay  writes:
>
>> 2012ko azaroak 17an, Eric Schulte-ek idatzi zuen:
>>
>>> Oh!, thanks for catching this, I just pushed up a fix.
>>
>> This is no longer a (complete) fix for the original problem, though.  (A
>> large part of) the slowdown comes from reading the results from a temp
>> file and transforming them into an elispy format, which is handled by
>> the backends.  Your code only prevents them from being echoed to the
>> minibuffer.
>>
>> You can see this by trying your original “seq” tests.  You should see
>> them taking the same (very long) amount of time.
>>
>> If you set debug-on-quit to t (before evaluating the block) and
>> interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes
>> through ‘org-babel-execute:sh’ and then
>> ‘org-babel-import-elisp-from-file’.  The presence of the former in the
>> call stack is why I claim that per-backend fixes are ultimately needed.
>> The latter is where my patch addresses the problem in a temporary way.
>>
>
> I apologize for rushing through your previous email and missing
> important portions of the content.
>
>>
>>> 
>>> I may be outvoted, but I find this approach too be overly complicated.
>>> Also, size may frequently not be the best proxy for the time which
>>> Emacs will take to ingest the results.
>>
>> I agree that my patch is a stopgap.  But until per-backend fixes are
>> available, this allows certain code to run that otherwise wouldn’t (at
>> least not without hacks, such as putting NULL at the end of an R source
>> block so that the “.Last.value” in the block is trivial).
>
> I'm attaching a new patch which should serve as a more permanent
> solution, however, given that it touches almost every backend I want to
> give the list a chance to try it out before committing it.  This patch
> introduces a new macro (`org-babel-result-cond') which attempts to unify
> the results processing.  Not only does this check for ":results none"
> and ignore all results in that case, but it also checks whether the
> results should be parsed as a scalar or a vector.  Hopefully this will
> serve as useful refactoring and reduce the total amount of code as well
> as provide for more uniform results processing across languages.
>
> Please let me know if this looks like a good permanent solution.
>
> I see that I'm now getting 5 test failures with this patch applied
> locally, but I won't have time to look at this further until early next
> week at the earliest, so I'm sharing the existing patch now.
>
> Thanks,

Alright, I've now changed the above patch so that as many tests are
passing locally as were passing without the patch.  Given that, I've
applied this patch to the main git repository.  Everything which worked
previously should still be working, only now the ":results none" options
should have the desired effect.

Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Eric Schulte]

Aaron Ecay  writes:

> 2012ko azaroak 17an, Eric Schulte-ek idatzi zuen:
>
>> Oh!, thanks for catching this, I just pushed up a fix.
>
> This is no longer a (complete) fix for the original problem, though.  (A
> large part of) the slowdown comes from reading the results from a temp
> file and transforming them into an elispy format, which is handled by
> the backends.  Your code only prevents them from being echoed to the
> minibuffer.
>
> You can see this by trying your original “seq” tests.  You should see
> them taking the same (very long) amount of time.
>
> If you set debug-on-quit to t (before evaluating the block) and
> interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes
> through ‘org-babel-execute:sh’ and then
> ‘org-babel-import-elisp-from-file’.  The presence of the former in the
> call stack is why I claim that per-backend fixes are ultimately needed.
> The latter is where my patch addresses the problem in a temporary way.
>

I apologize for rushing through your previous email and missing
important portions of the content.

>
>> 
>> I may be outvoted, but I find this approach too be overly complicated.
>> Also, size may frequently not be the best proxy for the time which
>> Emacs will take to ingest the results.
>
> I agree that my patch is a stopgap.  But until per-backend fixes are
> available, this allows certain code to run that otherwise wouldn’t (at
> least not without hacks, such as putting NULL at the end of an R source
> block so that the “.Last.value” in the block is trivial).

I'm attaching a new patch which should serve as a more permanent
solution, however, given that it touches almost every backend I want to
give the list a chance to try it out before committing it.  This patch
introduces a new macro (`org-babel-result-cond') which attempts to unify
the results processing.  Not only does this check for ":results none"
and ignore all results in that case, but it also checks whether the
results should be parsed as a scalar or a vector.  Hopefully this will
serve as useful refactoring and reduce the total amount of code as well
as provide for more uniform results processing across languages.

Please let me know if this looks like a good permanent solution.

I see that I'm now getting 5 test failures with this patch applied
locally, but I won't have time to look at this further until early next
week at the earliest, so I'm sharing the existing patch now.

Thanks,

>From a2878534832923e42aef49176b87d9993c04af71 Mon Sep 17 00:00:00 2001
From: Eric Schulte 
Date: Sun, 18 Nov 2012 18:02:09 -0700
Subject: [PATCH] org-babel-result-cond - unified handling of results

* lisp/ob.el (org-babel-result-cond): This function should now be used
  by all language backends to handle the processing of raw code block
  output into scalar results, table results, or ignored results
  depending on the value of result-params.

* lisp/ob-C.el (org-babel-C-execute): Use org-babel-result-cond.
* lisp/ob-R.el (org-babel-R-evaluate-external-process): Use
  org-babel-result-cond.
  (org-babel-R-evaluate-session): Use org-babel-result-cond.
* lisp/ob-awk.el (org-babel-execute:awk): Use org-babel-result-cond.
* lisp/ob-clojure.el (org-babel-execute:clojure): Use
  org-babel-result-cond.
* lisp/ob-emacs-lisp.el (org-babel-execute:emacs-lisp): Use
  org-babel-result-cond.
* lisp/ob-fortran.el (org-babel-execute:fortran): Use
  org-babel-result-cond.
* lisp/ob-io.el (org-babel-io-evaluate): Use org-babel-result-cond.
* lisp/ob-java.el (org-babel-execute:java): Use org-babel-result-cond.
* lisp/ob-lisp.el (org-babel-execute:lisp): Use org-babel-result-cond.
* lisp/ob-maxima.el (org-babel-execute:maxima): Use
  org-babel-result-cond.
* lisp/ob-picolisp.el (org-babel-execute:picolisp): Use
  org-babel-result-cond.
* lisp/ob-python.el (org-babel-python-evaluate-external-process): Use
  org-babel-result-cond.
  (org-babel-python-evaluate-session): Use org-babel-result-cond.
* lisp/ob-scala.el (org-babel-scala-evaluate): Use org-babel-result-cond.
* lisp/ob-sh.el (org-babel-sh-evaluate): Use org-babel-result-cond.
* lisp/ob-shen.el (org-babel-execute:shen): Use org-babel-result-cond.
* lisp/ob-sql.el (org-babel-execute:sql): Use org-babel-result-cond.
* lisp/ob-sqlite.el (org-babel-execute:sqlite): Use
  org-babel-result-cond.
---
 lisp/ob-C.el  | 10 +-
 lisp/ob-R.el  | 18 --
 lisp/ob-awk.el|  6 ++
 lisp/ob-clojure.el|  5 ++---
 lisp/ob-emacs-lisp.el |  9 -
 lisp/ob-fortran.el| 10 +-
 lisp/ob-io.el |  4 ++--
 lisp/ob-java.el   |  8 
 lisp/ob-lisp.el   |  4 ++--
 lisp/ob-maxima.el |  6 ++
 lisp/ob-picolisp.el   |  9 ++---
 lisp/ob-python.el | 14 --
 lisp/ob-scala.el  |  4 ++--
 lisp/ob-sh.el |  6 ++
 lisp/ob-shen.el   |  5 ++---
 lisp/ob-sql.el|  8 ++--
 lisp/ob-sqlite.el | 32 +++-
 lisp/ob.el| 14 ++

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Aaron Ecay]

2012ko azaroak 17an, Eric Schulte-ek idatzi zuen:

> Oh!, thanks for catching this, I just pushed up a fix.

This is no longer a (complete) fix for the original problem, though.  (A
large part of) the slowdown comes from reading the results from a temp
file and transforming them into an elispy format, which is handled by
the backends.  Your code only prevents them from being echoed to the
minibuffer.

You can see this by trying your original “seq” tests.  You should see
them taking the same (very long) amount of time.

If you set debug-on-quit to t (before evaluating the block) and
interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes
through ‘org-babel-execute:sh’ and then
‘org-babel-import-elisp-from-file’.  The presence of the former in the
call stack is why I claim that per-backend fixes are ultimately needed.
The latter is where my patch addresses the problem in a temporary way.

> 
> I may be outvoted, but I find this approach too be overly complicated.
> Also, size may frequently not be the best proxy for the time which
> Emacs will take to ingest the results.

I agree that my patch is a stopgap.  But until per-backend fixes are
available, this allows certain code to run that otherwise wouldn’t (at
least not without hacks, such as putting NULL at the end of an R source
block so that the “.Last.value” in the block is trivial).

-- 
Aaron Ecay

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Eric Schulte]

Aaron Ecay  writes:

> 2012ko azaroak 16an, Eric Schulte-ek idatzi zuen:
>
>> 
>> The attached patch adds a "really-silent" results header argument.  To
>> see the impact compare the running time of the following two code
>> blocks.
>
> Unfortunately, the attached patch doesn’t work correctly.  This can be
> seen by replacing the “seq” command in your examples with a command that
> has side effects – notify-send, aplay, etc.  In the :results none” case,
> the command is not run at all.
>
> That’s because the “(funcall cmd body params)” call at l. 574 of ob.el
> (patched version) has been put in a branch that is only run if :results
> != none.  That funcall is responsible for actually evaluating the code
> block.
>

Oh!, thanks for catching this, I just pushed up a fix.

>
> (The indentation of the patch as applied isn’t correct – the two
> branches of the if on l. 565 are indented at the same depth as each
> other, and as the if.  So it’s possible that the problem is due to a
> paren in the wrong place.  But I cannot see a way to make this approach
> work.)
>

Yes, sometimes I find this approach to be preferable to make the diffs
more readable and to avoid over-indentation in very large functions.

>
> The code generating the slowdown is in backend-specific files.  So, for
> example, a new branch needs to be added to the case statements in
> org-babel-R-evaluate(-session), to detect the :results none case and not
> read the result.  I suspect that each backend will need this treatment
> (unless some of them share code).
>
> In the meantime, attached to this email is a patch that implements a
> size check on reading results.  If the results file is over 10kb, it
> asks the user whether to proceed.  You can test this by evaluating:
>
> #+begin_src sh :results silent
>   seq 1
> #+end_src
>
> 10kb of results actually doesn’t result in a very serious hang (~5sec on
> my system).  But I chose the value as more of a sanity check – odds are
> (?) that very few people want to see 10k of results in the (mini)buffer.
> The value could be made customizable.
>
> I also chose the polarity of the y-or-n-p so that picking the default
> (yes) option does the sensible thing of not hanging emacs, although it
> thus does discard data.  I’m not sure which is the worse problem.
>

I may be outvoted, but I find this approach too be overly complicated.
Also, size may frequently not be the best proxy for the time which Emacs
will take to ingest the results.

Best,

>
> From 1053f3acfc21f24fc994ae85adff6779838b0ce7 Mon Sep 17 00:00:00 2001
> From: Aaron Ecay 
> Date: Sat, 17 Nov 2012 19:26:43 -0500
> Subject: [PATCH] lisp/ob.el: add a size check to
>  `org-babel-import-elisp-from-file'
>
> Reading large results can cause emacs to hang for a long time.  Ask the
> user whether to proceed in such cases.
>
> Signed-off-by: Aaron Ecay 
> ---
>  lisp/ob.el | 42 --
>  1 file changed, 24 insertions(+), 18 deletions(-)
>
> diff --git a/lisp/ob.el b/lisp/ob.el
> index bf4b455..b2385e9 100644
> --- a/lisp/ob.el
> +++ b/lisp/ob.el
> @@ -2462,24 +2462,30 @@ appropriate."
>  (defun org-babel-import-elisp-from-file (file-name &optional separator)
>"Read the results located at FILE-NAME into an elisp table.
>  If the table is trivial, then return it as a scalar."
> -  (let (result)
> -(save-window-excursion
> -  (with-temp-buffer
> - (condition-case err
> - (progn
> -   (org-table-import file-name separator)
> -   (delete-file file-name)
> -   (setq result (mapcar (lambda (row)
> -  (mapcar #'org-babel-string-read row))
> -(org-table-to-lisp
> -   (error (message "Error reading results: %s" err) nil)))
> -  (if (null (cdr result)) ;; if result is trivial vector, then scalarize 
> it
> -   (if (consp (car result))
> -   (if (null (cdr (car result)))
> -   (caar result)
> - result)
> - (car result))
> - result
> +  (let* ((file-size (nth 7 (file-attributes file-name)))
> + (can-load (or (< file-size (* 10 1024)) ; 10kb
> +   (not (y-or-n-p (concat "Displaying the block's large "
> +  "results may hang emacs; skip "
> +  "reading them?"))
> +(when can-load
> +  (let (result)
> +(save-window-excursion
> +  (with-temp-buffer
> +(condition-case err
> +(progn
> +  (org-table-import file-name separator)
> +  (delete-file file-name)
> +  (setq result (mapcar (lambda (row)
> + (mapcar #'org-babel-string-read 
> row))
> +   (org-table-to-lisp
> +  (error (message "Error reading results: %s" err) nil)))
> +  (if (null (cdr resul

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Aaron Ecay]

2012ko azaroak 16an, Eric Schulte-ek idatzi zuen:

> 
> The attached patch adds a "really-silent" results header argument.  To
> see the impact compare the running time of the following two code
> blocks.

Unfortunately, the attached patch doesn’t work correctly.  This can be
seen by replacing the “seq” command in your examples with a command that
has side effects – notify-send, aplay, etc.  In the :results none” case,
the command is not run at all.

That’s because the “(funcall cmd body params)” call at l. 574 of ob.el
(patched version) has been put in a branch that is only run if :results
!= none.  That funcall is responsible for actually evaluating the code
block.

(The indentation of the patch as applied isn’t correct – the two
branches of the if on l. 565 are indented at the same depth as each
other, and as the if.  So it’s possible that the problem is due to a
paren in the wrong place.  But I cannot see a way to make this approach
work.)

The code generating the slowdown is in backend-specific files.  So, for
example, a new branch needs to be added to the case statements in
org-babel-R-evaluate(-session), to detect the :results none case and not
read the result.  I suspect that each backend will need this treatment
(unless some of them share code).

In the meantime, attached to this email is a patch that implements a
size check on reading results.  If the results file is over 10kb, it
asks the user whether to proceed.  You can test this by evaluating:

#+begin_src sh :results silent
  seq 1
#+end_src

10kb of results actually doesn’t result in a very serious hang (~5sec on
my system).  But I chose the value as more of a sanity check – odds are
(?) that very few people want to see 10k of results in the (mini)buffer.
The value could be made customizable.

I also chose the polarity of the y-or-n-p so that picking the default
(yes) option does the sensible thing of not hanging emacs, although it
thus does discard data.  I’m not sure which is the worse problem.

>From 1053f3acfc21f24fc994ae85adff6779838b0ce7 Mon Sep 17 00:00:00 2001
From: Aaron Ecay 
Date: Sat, 17 Nov 2012 19:26:43 -0500
Subject: [PATCH] lisp/ob.el: add a size check to
 `org-babel-import-elisp-from-file'

Reading large results can cause emacs to hang for a long time.  Ask the
user whether to proceed in such cases.

Signed-off-by: Aaron Ecay 
---
 lisp/ob.el | 42 --
 1 file changed, 24 insertions(+), 18 deletions(-)

diff --git a/lisp/ob.el b/lisp/ob.el
index bf4b455..b2385e9 100644
--- a/lisp/ob.el
+++ b/lisp/ob.el
@@ -2462,24 +2462,30 @@ appropriate."
 (defun org-babel-import-elisp-from-file (file-name &optional separator)
   "Read the results located at FILE-NAME into an elisp table.
 If the table is trivial, then return it as a scalar."
-  (let (result)
-(save-window-excursion
-  (with-temp-buffer
-	(condition-case err
-	(progn
-	  (org-table-import file-name separator)
-	  (delete-file file-name)
-	  (setq result (mapcar (lambda (row)
- (mapcar #'org-babel-string-read row))
-   (org-table-to-lisp
-	  (error (message "Error reading results: %s" err) nil)))
-  (if (null (cdr result)) ;; if result is trivial vector, then scalarize it
-	  (if (consp (car result))
-	  (if (null (cdr (car result)))
-		  (caar result)
-		result)
-	(car result))
-	result
+  (let* ((file-size (nth 7 (file-attributes file-name)))
+ (can-load (or (< file-size (* 10 1024)) ; 10kb
+   (not (y-or-n-p (concat "Displaying the block's large "
+  "results may hang emacs; skip "
+  "reading them?"))
+(when can-load
+  (let (result)
+(save-window-excursion
+  (with-temp-buffer
+(condition-case err
+(progn
+  (org-table-import file-name separator)
+  (delete-file file-name)
+  (setq result (mapcar (lambda (row)
+ (mapcar #'org-babel-string-read row))
+   (org-table-to-lisp
+  (error (message "Error reading results: %s" err) nil)))
+  (if (null (cdr result)) ;; if result is trivial vector, then scalarize it
+  (if (consp (car result))
+  (if (null (cdr (car result)))
+  (caar result)
+result)
+(car result))
+result))
 
 (defun org-babel-string-read (cell)
   "Strip nested \"s from around strings."
-- 
1.8.0



-- 
Aaron Ecay

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Eric Schulte]

Achim Gratz  writes:

> Andreas Leha writes:
>> I would suggest simply "none" for the new "really-silent".
>
> Yes please, especially since I tried ":results none" two days ago to
> achieve that effect and wondered why it wouldn't work.  :-)
>

Great, since "none" seems like the obvious choice I just committed a
version of this patch which places this behavior behind ":results none".

At some point documentation should be added to the relevant portion of
the manual, but at least we now have this for those reading the mailing
list and for those who simply try "none" with this behavior in mind.

Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Achim Gratz]

Andreas Leha writes:
> I would suggest simply "none" for the new "really-silent".

Yes please, especially since I tried ":results none" two days ago to
achieve that effect and wondered why it wouldn't work.  :-)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Fri, Nov 16, 2012 at 11:47 AM, Andreas Leha <
andreas.l...@med.uni-goettingen.de> wrote:

> Hi Eric,
>
> Eric Schulte  writes:
>
> > Andreas Leha  writes:
> >
> >> t...@tsdye.com (Thomas S. Dye) writes:
> >>
> >>> Aloha Aaron,
> >>>
> >>> Welcome to Org-mode.
> >>>
> >>> Aaron Ecay  writes:
> >>>
>  2012ko azaroak 13an, John Hendy-ek idatzi zuen:
> 
>  [...]
> > Crazy. I really wondered if it had something to do with trying to
> spit
> > out the results into the minibuffer. Why is that behavior included?
> 
>  “:results silent” just means “silent except for the minibuffer”; IDK
>  why.
> >>>
> >>
> >> [...]
> >>
> >> Just for the record:
> >> I would also love to see the "really-silent results".
> >>
> >
> > Yes, the existing ":results silent" option still echos the results to
> > the minibuffer.  It was originally added in the case where one does not
> > want to change the Org-mode buffer (but would still like to see code
> > block output).
> >
> > The attached patch adds a "really-silent" results header argument.  To
> > see the impact compare the running time of the following two code
> > blocks.
> >
> > #+begin_src sh :results really-silent
> >   seq 1
> > #+end_src
> > #+begin_src sh :results silent
> >   seq 1
> > #+end_src
> >
> > Before such a patch is applied it would need corresponding
> > documentation, and the "really-silent" moniker may need to be
> > reconsidered in favor of something more informative (the main purpose of
> > this header argument is not silence but is rather avoiding all post
> > processing), and ideally shorter as well.  These new header arguments
> > are of no value if they are not easily discover-able and easily
> > remember-able to help others avoid the same issue that launched this
> > thread.
> >
> > Cheers,
>
> this is simply great!  Thank a lot.  I just tested and like it a lot!
>
>
I'll have to give this a try.


> With respect to the name: I think choosing a new name is hard because it
> really should be "silent".  What is now "silent" would better be named
> "minibuffer" or sth. similar.  But since changing that would break quite
> some (of my) code, I would suggest simply "none" for the new
> "really-silent".
>

I like the "none" idea. Or another approach would be a variable/in-buffer
setting that defines the meaning of "silent"? Something like
=org-babel-echo-results-in-minibuffer= or something like that? Though
perhaps folks would like to mix and match silent and none within the same
document.

Thanks for creating the patch, Eric. I'll try shortly.

Best regards,
John


>
> Just my 2ct.
>
> Thanks again.  I'll start using it right now.
>
> Regards,
> Andreas
>
>
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Andreas Leha]

Hi Eric,

Eric Schulte  writes:

> Andreas Leha  writes:
>
>> t...@tsdye.com (Thomas S. Dye) writes:
>>
>>> Aloha Aaron,
>>>
>>> Welcome to Org-mode.
>>>
>>> Aaron Ecay  writes:
>>>
 2012ko azaroak 13an, John Hendy-ek idatzi zuen:

 [...]
> Crazy. I really wondered if it had something to do with trying to spit
> out the results into the minibuffer. Why is that behavior included?

 “:results silent” just means “silent except for the minibuffer”; IDK
 why.
>>>
>>
>> [...]
>>
>> Just for the record:
>> I would also love to see the "really-silent results".
>>
>
> Yes, the existing ":results silent" option still echos the results to
> the minibuffer.  It was originally added in the case where one does not
> want to change the Org-mode buffer (but would still like to see code
> block output).
>
> The attached patch adds a "really-silent" results header argument.  To
> see the impact compare the running time of the following two code
> blocks.
>
> #+begin_src sh :results really-silent
>   seq 1
> #+end_src
> #+begin_src sh :results silent
>   seq 1
> #+end_src
>
> Before such a patch is applied it would need corresponding
> documentation, and the "really-silent" moniker may need to be
> reconsidered in favor of something more informative (the main purpose of
> this header argument is not silence but is rather avoiding all post
> processing), and ideally shorter as well.  These new header arguments
> are of no value if they are not easily discover-able and easily
> remember-able to help others avoid the same issue that launched this
> thread.
>
> Cheers,

this is simply great!  Thank a lot.  I just tested and like it a lot!

With respect to the name: I think choosing a new name is hard because it
really should be "silent".  What is now "silent" would better be named
"minibuffer" or sth. similar.  But since changing that would break quite
some (of my) code, I would suggest simply "none" for the new
"really-silent".

Just my 2ct.

Thanks again.  I'll start using it right now.

Regards,
Andreas

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Eric Schulte]

Andreas Leha  writes:

> t...@tsdye.com (Thomas S. Dye) writes:
>
>> Aloha Aaron,
>>
>> Welcome to Org-mode.
>>
>> Aaron Ecay  writes:
>>
>>> 2012ko azaroak 13an, John Hendy-ek idatzi zuen:
>>>
>>> [...]
 Crazy. I really wondered if it had something to do with trying to spit
 out the results into the minibuffer. Why is that behavior included?
>>>
>>> “:results silent” just means “silent except for the minibuffer”; IDK
>>> why.
>>
>
> [...]
>
> Just for the record:
> I would also love to see the "really-silent results".
>

Yes, the existing ":results silent" option still echos the results to
the minibuffer.  It was originally added in the case where one does not
want to change the Org-mode buffer (but would still like to see code
block output).

The attached patch adds a "really-silent" results header argument.  To
see the impact compare the running time of the following two code
blocks.

#+begin_src sh :results really-silent
  seq 1
#+end_src

#+begin_src sh :results silent
  seq 1
#+end_src

Before such a patch is applied it would need corresponding
documentation, and the "really-silent" moniker may need to be
reconsidered in favor of something more informative (the main purpose of
this header argument is not silence but is rather avoiding all post
processing), and ideally shorter as well.  These new header arguments
are of no value if they are not easily discover-able and easily
remember-able to help others avoid the same issue that launched this
thread.

Cheers,

>From 8ae5b59acdaaa271c797756bfe7f9479e1f9a277 Mon Sep 17 00:00:00 2001
From: Eric Schulte 
Date: Fri, 16 Nov 2012 08:40:46 -0700
Subject: [PATCH] "really-silent" header avoids result post-process

This patch introduces a new header argument which hide all code block
results from Emacs so that they are not processed into Emacs-lisp
objects; a process which may take a great deal of time for large
objects.

* lisp/ob.el (org-babel-common-header-args-w-values): Adding the new
  "really-silent" header argument.
  (org-babel-execute-src-block): Don't do *any* result processing if the
  "really-silent" header argument has been specified.
  (org-babel-sha1-hash): Remove the really-silent header argument from
  header arg lists when calculating cache sums.
  (org-babel-insert-result): Flesh out some documentation.
---
 lisp/ob.el | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/lisp/ob.el b/lisp/ob.el
index 69a5c58..167fe1a 100644
--- a/lisp/ob.el
+++ b/lisp/ob.el
@@ -421,7 +421,7 @@ then run `org-babel-pop-to-session'."
 (padline	. ((yes no)))
 (results	. ((file list vector table scalar verbatim)
 		   (raw html latex org code pp drawer)
-		   (replace silent append prepend)
+		   (replace silent really-silent append prepend)
 		   (output value)))
 (rownames	. ((no yes)))
 (sep	. :any)
@@ -562,6 +562,8 @@ block."
 		(message "executing %s code block%s..."
 			 (capitalize lang)
 			 (if (nth 4 info) (format " (%s)" (nth 4 info)) ""))
+		(if (member "really-silent" result-params)
+		(message "result silenced")
 		(setq result
 		  ((lambda (result)
 			 (if (and (eq (cdr (assoc :result-type params)) 'value)
@@ -581,7 +583,8 @@ block."
 		(org-babel-insert-result
 		 result result-params info new-hash indent lang)
 		(run-hooks 'org-babel-after-execute-hook)
-		result))
+		result
+		)))
 	  (setq call-process-region 'org-babel-call-process-region-original))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
@@ -1028,7 +1031,8 @@ the current subtree."
 	  (sort (copy-sequence (nth 2 info))
 		(lambda (a b) (string< (car a) (car b)
 (let* ((rm (lambda (lst)
-		 (dolist (p '("replace" "silent" "append" "prepend"))
+		 (dolist (p '("replace" "silent" "really-silent"
+			  "append" "prepend"))
 		   (setq lst (remove p lst)))
 		 lst))
 	   (norm (lambda (arg)
@@ -1851,7 +1855,10 @@ RESULT-PARAMS can take the following values:
 replace - (default option) insert results after the source block
   replacing any previously inserted results
 
-silent -- no results are inserted
+silent -- no results are inserted into the Org-mode buffer but
+  the results are echoed to the minibuffer and are
+  ingested by Emacs (a potentially time consuming
+  process)
 
 file  the results are interpreted as a file path, and are
   inserted into the buffer using the Org-mode file syntax
-- 
1.8.0


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Andreas Leha]

t...@tsdye.com (Thomas S. Dye) writes:

> Aloha Aaron,
>
> Welcome to Org-mode.
>
> Aaron Ecay  writes:
>
>> 2012ko azaroak 13an, John Hendy-ek idatzi zuen:
>>
>> [...]
>>> Crazy. I really wondered if it had something to do with trying to spit
>>> out the results into the minibuffer. Why is that behavior included?
>>
>> “:results silent” just means “silent except for the minibuffer”; IDK
>> why.
>

[...]

Just for the record:
I would also love to see the "really-silent results".

Regards,
Andreas

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Thomas S. Dye]

Aloha Aaron,

Welcome to Org-mode.

Aaron Ecay  writes:

> 2012ko azaroak 13an, John Hendy-ek idatzi zuen:
>
> [...]
>> Crazy. I really wondered if it had something to do with trying to spit
>> out the results into the minibuffer. Why is that behavior included?
>
> “:results silent” just means “silent except for the minibuffer”; IDK
> why.

I think "silent" has to do with the Org buffer that contains the source
code block.  Non-silent blocks do something in the buffer.  This was,
for me, one of the amazing things about babel when it was being
developed.  One could "speak" any language and have sensible results
written back to a plain text file.

>
>> Are you sure it's only on file import? I was generating some large
>> output when I first experienced this.
>
> Yes, that’s what I meant (and didn’t explain well).  Org writes R’s
> output to a file, and then imports it into Emacs (to print in the
> minibuffer).  But it also does an expensive transformation before
> printing it – whence the slowdown.
>
>> I have to
>> change to the R buffer to see if there were any errors.
>
> I agree that this is sub-optimal.
>
> I seem to remember that ob-R.el is unmaintained (or was at some point in
> the past).  Once there is consensus on what a fix should look like, I
> can attempt to produce a patch implementing it.  And I’ll take a crack
> at error reporting in the minibuffer too.

Yes, ob-R.el is currently without a maintainer. My sense and experience
is that Eric S. is happy to entertain patches to the various ob language
files.  My experience is also that active maintainers welcome
patches--I don't think there is a barrier to patch proposals.

I can help with documentation if your patch results in some substantive
changes to R support in babel.  Please don't hesitate to ask if need be.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Aaron Ecay]

2012ko azaroak 13an, John Hendy-ek idatzi zuen:

[...]
> Crazy. I really wondered if it had something to do with trying to spit
> out the results into the minibuffer. Why is that behavior included?

“:results silent” just means “silent except for the minibuffer”; IDK
why.

> Are you sure it's only on file import? I was generating some large
> output when I first experienced this.

Yes, that’s what I meant (and didn’t explain well).  Org writes R’s
output to a file, and then imports it into Emacs (to print in the
minibuffer).  But it also does an expensive transformation before
printing it – whence the slowdown.

> I have to
> change to the R buffer to see if there were any errors.

I agree that this is sub-optimal.

I seem to remember that ob-R.el is unmaintained (or was at some point in
the past).  Once there is consensus on what a fix should look like, I
can attempt to produce a patch implementing it.  And I’ll take a crack
at error reporting in the minibuffer too.

-- 
Aaron Ecay

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Tue, Nov 13, 2012 at 9:27 PM, Aaron Ecay  wrote:

> I just ran into this problem.  It is caused by the calls to
> ‘org-babel-import-elisp-from-file’ on lines 310 and 343 of ‘ob-R.el’
> (for non-session and session code blocks respectively).  I determined
> this by setting ‘debug-on-quit’ to t in Emacs, then hitting C-g when cpu
> usage spiked, and examining the backtrace.
>
>
Yes! I'm not crazy! I'm not very familiar with the debug options in emacs
and had a heck of a time setting it up right the last time. Thanks so much
for checking this out.


> It seems like Emacs tries to read in the last value of the R command, to
> print it in the minibuffer.  It gets bogged down doing so.  As a
> stopgap, putting NULL as the last line of the source block fixes the
> problem (the value “returned” by the block is just NULL).  I believe
> setting “:results output” (as opposed to “value”) would also make the
> bug disappear, since the output will be very small.  This is likely what
> made cberry’s test differ from John’s.
>
>
Crazy. I really wondered if it had something to do with trying to spit out
the results into the minibuffer. Why is that behavior included?


> I’m not sure what the right strategy would be for a fix.  Two ideas
> would be:
> 1) adding a sanity check to o-b-import-elisp-from-file, which asks the
>user whether to really import very large files (or just ignores such
>files)
>

Are you sure it's only on file import? I was generating some large output
when I first experienced this. It was only to avoid running that code that
I wrote it to file once and decided to just read it once per session
afterward (thinking reading would be faster than generating the input on
demand). I was generating a grid of interactions with something akin to:

#+begin_src R
dat <- expand.grid(var1 = seq(0, 4, by = 200), var2 = seq(0, 4, by
= 200))
#+end_src

I can check this tomorrow if you'd like, but I remember it lagging as it
tried to print all of that into the minibuffer.


> 2) adding a new :results handling header arg “really-silent” (probably
>with a better name...) that discards the result entirely, not even
>printing it to the minibuffer.
>
>
I like this, in addition to changing the minibuffer output as it is. It
would be fantastic to have the minibuffer say more than it does already
(something akin to "Block execution complete"). I have to change to the R
buffer to see if there were any errors. I like how the LaTeX exporter tells
you if PDF export failed. It'd be nice to see something like (in addition
to no results in the minibuffer), "Block execution complete but with
errors."

Thanks again, and I'm glad you were able to track this down.


John


> --
> Aaron Ecay
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Aaron Ecay]

I just ran into this problem.  It is caused by the calls to
‘org-babel-import-elisp-from-file’ on lines 310 and 343 of ‘ob-R.el’
(for non-session and session code blocks respectively).  I determined
this by setting ‘debug-on-quit’ to t in Emacs, then hitting C-g when cpu
usage spiked, and examining the backtrace.

It seems like Emacs tries to read in the last value of the R command, to
print it in the minibuffer.  It gets bogged down doing so.  As a
stopgap, putting NULL as the last line of the source block fixes the
problem (the value “returned” by the block is just NULL).  I believe
setting “:results output” (as opposed to “value”) would also make the
bug disappear, since the output will be very small.  This is likely what
made cberry’s test differ from John’s.

I’m not sure what the right strategy would be for a fix.  Two ideas
would be:
1) adding a sanity check to o-b-import-elisp-from-file, which asks the
   user whether to really import very large files (or just ignores such
   files)
2) adding a new :results handling header arg “really-silent” (probably
   with a better name...) that discards the result entirely, not even
   printing it to the minibuffer.

-- 
Aaron Ecay

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Nick Dokos]

John Hendy  wrote:

> On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos  wrote:
> 
> John Hendy  wrote:
>
> >     o run top (or whatever equivalent is available on your OS) and see
> >       whether the CPU (or one of the CPUs) gets pegged at 100% 
> utilization
> >       and stays there. If yes, that's an indication of an infinite loop
> >       somewhere.
> >
> > - quit any other instances of emacs/R
> > - start `top` in terminal
> > - execute block
> > - Use '<' '>' to sort back and forth between cpu and ram
> >
> > Observations
> > - R is at 80-100% cpu for about 5sec
> > - Then emacs shifts to fairly constant ~100% cpu usage 
> > - After about a minute, the minibuffer expands to ~1/3 of the window 
> height and fills with the
> csv
> > data
> > - Finished after ~5min total time
> > - So, R took about 5sec, emacs took another 5min to finish
> >  
> >
>
> So not an infinite loop. That's progress ;-)
>
> Perhaps emacs is thrashing? If you are on linux, use swapon -s
> or (perhaps better) iostat, or (perhaps even better, at least if
> you are on the Gnome 2.x desktop[fn:1]), run the system monitor
> applet, click Properties, enable all the monitored resources
> (cpu, mem, net, swap, load, disk) and watch it while you are
> running the test: in particular, check memory and swap. If you
> are swapping even a little bit, that's enough to cause severe
> performacne problems[fn:2], which can be cured by using less memory
> (so stop any memory hogs from running) or by adding memory.
> 
> Oh, and I'm conscious of R using RAM, so I tried to run this stuff without 
> browsers open, no other R
> sessions, etc. Prior to running R and without a browser, I'm typically around 
> 10-15% RAM utilized...
> 
> I have 4gb, so that should be enough to read this, especially per Chuck's 
> results! This is also an i7
> work computer. It should fly (and in the ESS session itself, it does).
> 

Check /var/log/syslog (or thereabouts) to see if the kernel is seeing errors 
with
some blocks on the disk and keeps retrying. Assuming SMART is enabled, run 
smartctl -a
as well and look for problems.

Nick

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos  wrote:

> John Hendy  wrote:
>
> > o run top (or whatever equivalent is available on your OS) and see
> >   whether the CPU (or one of the CPUs) gets pegged at 100%
> utilization
> >   and stays there. If yes, that's an indication of an infinite loop
> >   somewhere.
> >
> > - quit any other instances of emacs/R
> > - start `top` in terminal
> > - execute block
> > - Use '<' '>' to sort back and forth between cpu and ram
> >
> > Observations
> > - R is at 80-100% cpu for about 5sec
> > - Then emacs shifts to fairly constant ~100% cpu usage
> > - After about a minute, the minibuffer expands to ~1/3 of the window
> height and fills with the csv
> > data
> > - Finished after ~5min total time
> > - So, R took about 5sec, emacs took another 5min to finish
> >
> >
>
> So not an infinite loop. That's progress ;-)
>
> Perhaps emacs is thrashing? If you are on linux, use swapon -s
> or (perhaps better) iostat, or (perhaps even better, at least if
> you are on the Gnome 2.x desktop[fn:1]), run the system monitor
> applet, click Properties, enable all the monitored resources
> (cpu, mem, net, swap, load, disk) and watch it while you are
> running the test: in particular, check memory and swap. If you
> are swapping even a little bit, that's enough to cause severe
> performacne problems[fn:2], which can be cured by using less memory
> (so stop any memory hogs from running) or by adding memory.
>

Oh, and I'm conscious of R using RAM, so I tried to run this stuff without
browsers open, no other R sessions, etc. Prior to running R and without a
browser, I'm typically around 10-15% RAM utilized...

I have 4gb, so that should be enough to read this, especially per Chuck's
results! This is also an i7 work computer. It should fly (and in the ESS
session itself, it does).


John


>
> Top can also show you memory and swap so maybe that's the quickest
> way to check.
>
> Nick
>
> Footnotes:
>
> [fn:1] You can pobably do this in other desktops but I have no
>experience with them, so no guidance to give.
>
> [fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix
>to index my mail: it would take 30 minutes or so to finish.  My
>new(er) laptop has 4GB and finishes in 30 seconds: memory usage
>peaks at about 2.5 GB.
>
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos  wrote:

> John Hendy  wrote:
>
> > o run top (or whatever equivalent is available on your OS) and see
> >   whether the CPU (or one of the CPUs) gets pegged at 100%
> utilization
> >   and stays there. If yes, that's an indication of an infinite loop
> >   somewhere.
> >
> > - quit any other instances of emacs/R
> > - start `top` in terminal
> > - execute block
> > - Use '<' '>' to sort back and forth between cpu and ram
> >
> > Observations
> > - R is at 80-100% cpu for about 5sec
> > - Then emacs shifts to fairly constant ~100% cpu usage
> > - After about a minute, the minibuffer expands to ~1/3 of the window
> height and fills with the csv
> > data
> > - Finished after ~5min total time
> > - So, R took about 5sec, emacs took another 5min to finish
> >
> >
>
> So not an infinite loop. That's progress ;-)
>
> Perhaps emacs is thrashing? If you are on linux, use swapon -s
> or (perhaps better) iostat, or (perhaps even better, at least if
> you are on the Gnome 2.x desktop[fn:1]), run the system monitor
> applet, click Properties, enable all the monitored resources
> (cpu, mem, net, swap, load, disk) and watch it while you are
> running the test: in particular, check memory and swap. If you
> are swapping even a little bit, that's enough to cause severe
> performacne problems[fn:2], which can be cured by using less memory
> (so stop any memory hogs from running) or by adding memory.
>
> Top can also show you memory and swap so maybe that's the quickest
> way to check.
>
>
It's not swap... :)

$ free
total   used   free sharedbuffers cached
Mem:   397886424271321551732  0 122072 509140
-/+ buffers/cache:17959202182944
Swap:0  0  0



Best regards,
John


> Nick
>
> Footnotes:
>
> [fn:1] You can pobably do this in other desktops but I have no
>experience with them, so no guidance to give.
>
> [fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix
>to index my mail: it would take 30 minutes or so to finish.  My
>new(er) laptop has 4GB and finishes in 30 seconds: memory usage
>peaks at about 2.5 GB.
>
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Nick Dokos]

John Hendy  wrote:

> o run top (or whatever equivalent is available on your OS) and see
>   whether the CPU (or one of the CPUs) gets pegged at 100% utilization
>   and stays there. If yes, that's an indication of an infinite loop
>   somewhere.
> 
> - quit any other instances of emacs/R
> - start `top` in terminal
> - execute block
> - Use '<' '>' to sort back and forth between cpu and ram
> 
> Observations
> - R is at 80-100% cpu for about 5sec
> - Then emacs shifts to fairly constant ~100% cpu usage 
> - After about a minute, the minibuffer expands to ~1/3 of the window height 
> and fills with the csv
> data
> - Finished after ~5min total time
> - So, R took about 5sec, emacs took another 5min to finish
>  
> 

So not an infinite loop. That's progress ;-)

Perhaps emacs is thrashing? If you are on linux, use swapon -s
or (perhaps better) iostat, or (perhaps even better, at least if
you are on the Gnome 2.x desktop[fn:1]), run the system monitor
applet, click Properties, enable all the monitored resources
(cpu, mem, net, swap, load, disk) and watch it while you are
running the test: in particular, check memory and swap. If you
are swapping even a little bit, that's enough to cause severe
performacne problems[fn:2], which can be cured by using less memory
(so stop any memory hogs from running) or by adding memory.

Top can also show you memory and swap so maybe that's the quickest
way to check.

Nick

Footnotes:

[fn:1] You can pobably do this in other desktops but I have no
   experience with them, so no guidance to give.

[fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix
   to index my mail: it would take 30 minutes or so to finish.  My
   new(er) laptop has 4GB and finishes in 30 seconds: memory usage
   peaks at about 2.5 GB.

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Wed, Oct 31, 2012 at 5:53 PM, Nick Dokos  wrote:

> John Hendy  wrote:
>
> > On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
> >
> > John Hendy  writes:
> >
> > > On Wed, Oct 31, 2012 at 11:41 AM,   cbe...@tajo.ucsd.edu> wrote:
> > > John Hendy  writes:
> > >
> > >> I edited the subject to be more concise/clear.I let orgmode chug
> away
> > >> on reading in some ~10-30mb csv files for nearly 30min.
> > >
> > > [rest deleted]
> > >
> > > You need an ECM.I did my best to provide one, other than the file,
> which I offered to provide
> > if others requested that I upload it somewhere. Since you have done
> so, so have I:
> > > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> >
> > > Let me know if that doesn't work. I put it on Google docs and
> sometimes have issues with
> > the sharing settings...
> >
> > Not an ECM in my book, but ...
> >
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
>
> Smaller set of data I'd guess :-) But it does not seem to be the
> size of the data that matters.
>
> >
> >
> > On my 4 year old MacBook:
> >
> > ,
> > |
> > | #+PROPERTY: session *R*
> > |
> > | #+name: bigcsv
> > | #+begin_src R
> > | bigcsv <- Sys.glob("~/Downloads/*.csv")
> > | #+end_src
> > |
> > | #+RESULTS: bigcsv
> > | : /Users/cberry/Downloads/test-file.csv
> > |
> > | #+name: readbig
> > | #+begin_src R :results output
> > |   system.time(
> > | tmp <- read.csv(bigcsv)
> > | )
> > |
> > | #+end_src
> > |
> > | #+RESULTS: readbig
> > | :user  system elapsed
> > | :   5.679   0.306   6.002
> > |
> > `
> >
> > About the same as running from ESS.
> >
> > Not sure what to say. Looking for ways to troubleshoot or confirm. Since
> you can't confirm, any
> > suggestions on where I should look for my issue? I can't explain it! All
> I know is that org chugs
> > and chugs and the direct execution in ESS session is lightning fast.
> >
>
> A few things to try in no particular order:
>
>
This was extremely helpful. Thanks for the suggestions.

Here's my attempt at an ECM, though I'm going to keep using the big file
since that's what's actually doing it an I've already uploaded it :)
- Using emacs config here: http://pastebin.com/raw.php?i=iTbRtCE9
- Using this org-mode file:

#+begin_src org

* headline

#+begin_src R :session r :results silent
# file here:
https://docs.google.com/uc?export=download&confirm=no_antivirus&id=0BzQupOSnvw08WHdabHh5VVczRGM
data <- read.csv("path/to/file.csv")
#+end_src

#+end_src org

- Execute block with C-c C-c after downloading and changing path

 o run top (or whatever equivalent is available on your OS) and see
>   whether the CPU (or one of the CPUs) gets pegged at 100% utilization
>   and stays there. If yes, that's an indication of an infinite loop
>   somewhere.
>
>
- quit any other instances of emacs/R
- start `top` in terminal
- execute block
- Use '<' '>' to sort back and forth between cpu and ram

Observations
- R is at 80-100% cpu for about 5sec
- Then emacs shifts to fairly constant ~100% cpu usage
- After about a minute, the minibuffer expands to ~1/3 of the window height
and fills with the csv data
- Finished after ~5min total time
- So, R took about 5sec, emacs took another 5min to finish


> o run vmstat (or equivalent) and see if any of the counters are out of
> whack.
>   That requires some experience though.
>
>
I'll skip for now; no experience with that.


> o use elp-instrument-package to instrument org and run the test, getting
>   a profile. I'm not sure whether the results will be useful, since you
>   are going to interrupt the test when you run out of patience, but it
>   cannot hurt and it might tell you something useful.
>
> o run your ECM on a different computer/OS/emacs installation. Being able
>   to compare things side by side is often very useful.
>
> o Halve your file and run the test on each half (but that's probably not
>   the problem given Chuck's results).
>
> o Reinstall org from scratch - you might have some corruption in one of
>   the compiled files that's causing it to go into an infinite loop.
>
>
- `cd ~/.elisp`
- `sudo rm -r org.git`
- `git clone http://git://orgmode.org/org-mode.git org.git`
- cd org.git && make clean && make && make doc
- Quit previous emacs instance; reopen
- Remove (require 'org-install) per prompt; restart again
- Repeat `top` experiment

Results:
- Didn't even see R flash on the screen this time; emacs just jumped to 100%
- After 1min 10sec, the minibuffer filled with data
- At that point I quit, as I think it will be a repeat of the above


> o Turn on debug-on-quit, start your test, wait a bit and then interrupt
>   it. Check the backtrace.  Do it again and check whether the backtrace
>   looks the same. That's

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Nick Dokos]

Thomas S. Dye  wrote:

> Hi John,
> 
> Have you tried wrapping your R read in system.time()?  If you are right
> about :results silent eating up lots of time, then this should fix the
> problem.  system.time yields just a bit of output, so shouldn't slow
> things down if writing out the data is indeed the problem as you
> suspect. 
> 

I missed this in the thread so I thought I'd add one more thing to try:

o run iostat (or equivalent) to check how your disks are doing.
  on linux, iostat is part of a package usually called sysstat.

Nick

> 
> 
> John Hendy  writes:
> 
> > On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
> >
> >> John Hendy  writes:
> >>
> >> > On Wed, Oct 31, 2012 at 11:41 AM,   >> cbe...@tajo.ucsd.edu> wrote:
> >> > John Hendy  writes:
> >> >
> >> >> I edited the subject to be more concise/clear.I let orgmode chug away
> >> >> on reading in some ~10-30mb csv files for nearly 30min.
> >> >
> >> > [rest deleted]
> >> >
> >> > You need an ECM.I did my best to provide one, other than the file, which
> >> I offered to provide if others requested that I upload it somewhere. Since
> >> you have done so, so have I:
> >> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> >>
> >> > Let me know if that doesn't work. I put it on Google docs and
> >> sometimes have issues with the sharing settings...
> >>
> >> Not an ECM in my book, but ...
> >>
> >>
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
> >
> >
> >> On my 4 year old MacBook:
> >>
> >> ,
> >> |
> >> | #+PROPERTY: session *R*
> >> |
> >> | #+name: bigcsv
> >> | #+begin_src R
> >> | bigcsv <- Sys.glob("~/Downloads/*.csv")
> >> | #+end_src
> >> |
> >> | #+RESULTS: bigcsv
> >> | : /Users/cberry/Downloads/test-file.csv
> >> |
> >> | #+name: readbig
> >> | #+begin_src R :results output
> >> |   system.time(
> >> | tmp <- read.csv(bigcsv)
> >> | )
> >> |
> >> | #+end_src
> >> |
> >> | #+RESULTS: readbig
> >> | :user  system elapsed
> >> | :   5.679   0.306   6.002
> >> |
> >> `
> >>
> >> About the same as running from ESS.
> >>
> >>
> > Not sure what to say. Looking for ways to troubleshoot or confirm. Since
> > you can't confirm, any suggestions on where I should look for my issue? I
> > can't explain it! All I know is that org chugs and chugs and the direct
> > execution in ESS session is lightning fast.
> >
> >
> >
> > FYI,
> >>
> >> > dim(tmp)
> >> [1] 964805  9
> >>
> >>
> >>
> > Was that just to show the size, or is something wrong with my file?
> >
> >
> >> Chuck
> >>
> >>
> >
> > Thanks for attempting to replicate!
> > John
> >
> >
> >> [ rest deleted ]
> >>
> >>
> >>
> >>
> > On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
> >
> > John Hendy  writes:
> > 
> > > On Wed, Oct 31, 2012 at 11:41 AM,   > dir="ltr"> wrote:
> > > John Hendy  writes:
> > >
> > >> I edited the subject to be more concise/clear.I let orgmode
> > chug away
> > >> on reading in some ~10-30mb csv files for nearly 30min.
> > >
> > > [rest deleted]
> > >
> > > You need an ECM.I did my best to provide one, other than the
> > file, which I offered to provide if others requested that I upload
> > it somewhere. Since you have done so, so have I:
> > > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> > 
> > > Let me know if that doesn't work. I put it on Google docs
> > and sometimes have issues with the sharing settings...
> > 
> > Not an ECM in my book, but ...
> > 
> > 
> >
> > What else would you like? I provided:
> > - the config
> > - the data
> > - how to [attempt to] reproduce
> > - the org-mode text
> >
> >  
> >
> > On my 4 year old MacBook:
> > 
> > ,
> > |
> > | #+PROPERTY: session *R*
> > |
> > | #+name: bigcsv
> > | #+begin_src R
> > | bigcsv <- Sys.glob("~/Downloads/*.csv")
> > | #+end_src
> > |
> > | #+RESULTS: bigcsv
> > | : /Users/cberry/Downloads/test-file.csv
> > |
> > | #+name: readbig
> > | #+begin_src R :results output
> > |   system.time(
> > |     tmp <- read.csv(bigcsv)
> > |     )
> > |
> > | #+end_src
> > |
> > | #+RESULTS: readbig
> > | :    user  system elapsed
> > | :   5.679   0.306   6.002
> > |
> > `
> > 
> > About the same as running from ESS.
> > 
> > 
> >
> > Not sure what to say. Looking for ways to troubleshoot or confirm.
> > Since you can't confirm, any suggestions on where I should look for my
> > issue? I can't explain it! All I know is that org chugs and chugs and
> > the direct execution in ESS session is lightning fast.
> >
> >
> >
> > FYI,
> > 
> > > dim(tmp)
> > [1] 964805      9
> > 
> > 
> > 
> >
> > Was that just to show the size, or is something wrong with my file?
> 

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Nick Dokos]

John Hendy  wrote:

> On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
> 
> John Hendy  writes:
>
> > On Wed, Oct 31, 2012 at 11:41 AM,   dir="ltr"> wrote:
> > John Hendy  writes:
> >
> >> I edited the subject to be more concise/clear.I let orgmode chug away
> >> on reading in some ~10-30mb csv files for nearly 30min.
> >
> > [rest deleted]
> >
> > You need an ECM.I did my best to provide one, other than the file, 
> which I offered to provide
> if others requested that I upload it somewhere. Since you have done so, 
> so have I:
> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
>
> > Let me know if that doesn't work. I put it on Google docs and 
> sometimes have issues with
> the sharing settings...
>
> Not an ECM in my book, but ...
> 
> What else would you like? I provided:
> - the config
> - the data
> - how to [attempt to] reproduce
> - the org-mode text
> 

Smaller set of data I'd guess :-) But it does not seem to be the
size of the data that matters.

>  
> 
> On my 4 year old MacBook:
>
> ,
> |
> | #+PROPERTY: session *R*
> |
> | #+name: bigcsv
> | #+begin_src R
> | bigcsv <- Sys.glob("~/Downloads/*.csv")
> | #+end_src
> |
> | #+RESULTS: bigcsv
> | : /Users/cberry/Downloads/test-file.csv
> |
> | #+name: readbig
> | #+begin_src R :results output
> |   system.time(
> |     tmp <- read.csv(bigcsv)
> |     )
> |
> | #+end_src
> |
> | #+RESULTS: readbig
> | :    user  system elapsed
> | :   5.679   0.306   6.002
> |
> `
>
> About the same as running from ESS.
> 
> Not sure what to say. Looking for ways to troubleshoot or confirm. Since you 
> can't confirm, any
> suggestions on where I should look for my issue? I can't explain it! All I 
> know is that org chugs
> and chugs and the direct execution in ESS session is lightning fast.
> 

A few things to try in no particular order:

o run top (or whatever equivalent is available on your OS) and see
  whether the CPU (or one of the CPUs) gets pegged at 100% utilization
  and stays there. If yes, that's an indication of an infinite loop
  somewhere.

o run vmstat (or equivalent) and see if any of the counters are out of whack.
  That requires some experience though.

o use elp-instrument-package to instrument org and run the test, getting
  a profile. I'm not sure whether the results will be useful, since you
  are going to interrupt the test when you run out of patience, but it
  cannot hurt and it might tell you something useful.

o run your ECM on a different computer/OS/emacs installation. Being able
  to compare things side by side is often very useful.

o Halve your file and run the test on each half (but that's probably not
  the problem given Chuck's results).

o Reinstall org from scratch - you might have some corruption in one of
  the compiled files that's causing it to go into an infinite loop.

o Turn on debug-on-quit, start your test, wait a bit and then interrupt
  it. Check the backtrace.  Do it again and check whether the backtrace
  looks the same. That's often an indication of an infinite loop
  (inferring an infinite loop from a two element sample is statistically
  suspect of course, but surprisingly effective nevertheless). The point
  here is that the infinite loop is in emacs and the backtrace tells you
  something about the parties involved.

These are obviously not independent and the results of one experiment will
have to guide you in what you try next.

Good luck,
Nick

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[Thomas S. Dye]

Hi John,

Have you tried wrapping your R read in system.time()?  If you are right
about :results silent eating up lots of time, then this should fix the
problem.  system.time yields just a bit of output, so shouldn't slow
things down if writing out the data is indeed the problem as you
suspect. 

hth,
Tom


John Hendy  writes:

> On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
>
>> John Hendy  writes:
>>
>> > On Wed, Oct 31, 2012 at 11:41 AM,  > cbe...@tajo.ucsd.edu> wrote:
>> > John Hendy  writes:
>> >
>> >> I edited the subject to be more concise/clear.I let orgmode chug away
>> >> on reading in some ~10-30mb csv files for nearly 30min.
>> >
>> > [rest deleted]
>> >
>> > You need an ECM.I did my best to provide one, other than the file, which
>> I offered to provide if others requested that I upload it somewhere. Since
>> you have done so, so have I:
>> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
>>
>> > Let me know if that doesn't work. I put it on Google docs and
>> sometimes have issues with the sharing settings...
>>
>> Not an ECM in my book, but ...
>>
>>
> What else would you like? I provided:
> - the config
> - the data
> - how to [attempt to] reproduce
> - the org-mode text
>
>
>
>> On my 4 year old MacBook:
>>
>> ,
>> |
>> | #+PROPERTY: session *R*
>> |
>> | #+name: bigcsv
>> | #+begin_src R
>> | bigcsv <- Sys.glob("~/Downloads/*.csv")
>> | #+end_src
>> |
>> | #+RESULTS: bigcsv
>> | : /Users/cberry/Downloads/test-file.csv
>> |
>> | #+name: readbig
>> | #+begin_src R :results output
>> |   system.time(
>> | tmp <- read.csv(bigcsv)
>> | )
>> |
>> | #+end_src
>> |
>> | #+RESULTS: readbig
>> | :user  system elapsed
>> | :   5.679   0.306   6.002
>> |
>> `
>>
>> About the same as running from ESS.
>>
>>
> Not sure what to say. Looking for ways to troubleshoot or confirm. Since
> you can't confirm, any suggestions on where I should look for my issue? I
> can't explain it! All I know is that org chugs and chugs and the direct
> execution in ESS session is lightning fast.
>
>
>
> FYI,
>>
>> > dim(tmp)
>> [1] 964805  9
>>
>>
>>
> Was that just to show the size, or is something wrong with my file?
>
>
>> Chuck
>>
>>
>
> Thanks for attempting to replicate!
> John
>
>
>> [ rest deleted ]
>>
>>
>>
>>
> On Wed, Oct 31, 2012 at 3:12 PM,  wrote:
>
> John Hendy  writes:
> 
> > On Wed, Oct 31, 2012 at 11:41 AM,   dir="ltr"> wrote:
> > John Hendy  writes:
> >
> >> I edited the subject to be more concise/clear.I let orgmode
> chug away
> >> on reading in some ~10-30mb csv files for nearly 30min.
> >
> > [rest deleted]
> >
> > You need an ECM.I did my best to provide one, other than the
> file, which I offered to provide if others requested that I upload
> it somewhere. Since you have done so, so have I:
> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
> 
> > Let me know if that doesn't work. I put it on Google docs
> and sometimes have issues with the sharing settings...
> 
> Not an ECM in my book, but ...
> 
> 
>
> What else would you like? I provided:
> - the config
> - the data
> - how to [attempt to] reproduce
> - the org-mode text
>
>  
>
> On my 4 year old MacBook:
> 
> ,
> |
> | #+PROPERTY: session *R*
> |
> | #+name: bigcsv
> | #+begin_src R
> | bigcsv <- Sys.glob("~/Downloads/*.csv")
> | #+end_src
> |
> | #+RESULTS: bigcsv
> | : /Users/cberry/Downloads/test-file.csv
> |
> | #+name: readbig
> | #+begin_src R :results output
> |   system.time(
> |     tmp <- read.csv(bigcsv)
> |     )
> |
> | #+end_src
> |
> | #+RESULTS: readbig
> | :    user  system elapsed
> | :   5.679   0.306   6.002
> |
> `
> 
> About the same as running from ESS.
> 
> 
>
> Not sure what to say. Looking for ways to troubleshoot or confirm.
> Since you can't confirm, any suggestions on where I should look for my
> issue? I can't explain it! All I know is that org chugs and chugs and
> the direct execution in ESS session is lightning fast.
>
>
>
> FYI,
> 
> > dim(tmp)
> [1] 964805      9
> 
> 
> 
>
> Was that just to show the size, or is something wrong with my file?
>  
>
> Chuck
> 
> 
>
>
> Thanks for attempting to replicate!
> John
>  
>
> [ rest deleted ]
> 
> 
> 
> 
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Wed, Oct 31, 2012 at 3:12 PM,  wrote:

> John Hendy  writes:
>
> > On Wed, Oct 31, 2012 at 11:41 AM,   cbe...@tajo.ucsd.edu> wrote:
> > John Hendy  writes:
> >
> >> I edited the subject to be more concise/clear.I let orgmode chug away
> >> on reading in some ~10-30mb csv files for nearly 30min.
> >
> > [rest deleted]
> >
> > You need an ECM.I did my best to provide one, other than the file, which
> I offered to provide if others requested that I upload it somewhere. Since
> you have done so, so have I:
> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM
>
> > Let me know if that doesn't work. I put it on Google docs and
> sometimes have issues with the sharing settings...
>
> Not an ECM in my book, but ...
>
>
What else would you like? I provided:
- the config
- the data
- how to [attempt to] reproduce
- the org-mode text



> On my 4 year old MacBook:
>
> ,
> |
> | #+PROPERTY: session *R*
> |
> | #+name: bigcsv
> | #+begin_src R
> | bigcsv <- Sys.glob("~/Downloads/*.csv")
> | #+end_src
> |
> | #+RESULTS: bigcsv
> | : /Users/cberry/Downloads/test-file.csv
> |
> | #+name: readbig
> | #+begin_src R :results output
> |   system.time(
> | tmp <- read.csv(bigcsv)
> | )
> |
> | #+end_src
> |
> | #+RESULTS: readbig
> | :user  system elapsed
> | :   5.679   0.306   6.002
> |
> `
>
> About the same as running from ESS.
>
>
Not sure what to say. Looking for ways to troubleshoot or confirm. Since
you can't confirm, any suggestions on where I should look for my issue? I
can't explain it! All I know is that org chugs and chugs and the direct
execution in ESS session is lightning fast.



FYI,
>
> > dim(tmp)
> [1] 964805  9
>
>
>
Was that just to show the size, or is something wrong with my file?


> Chuck
>
>

Thanks for attempting to replicate!
John


> [ rest deleted ]
>
>
>
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[cberry]

John Hendy  writes:

> On Wed, Oct 31, 2012 at 11:41 AM,   dir="ltr"> wrote:
> John Hendy  writes:
>
>> I edited the subject to be more concise/clear.I let orgmode chug away
>> on reading in some ~10-30mb csv files for nearly 30min.
>
> [rest deleted]
>
> You need an ECM.I did my best to provide one, other than the file, which I 
> offered to provide if others requested that I upload it somewhere. Since you 
> have done so, so have I:
> - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM

> Let me know if that doesn't work. I put it on Google docs and sometimes 
> have issues with the sharing settings...

Not an ECM in my book, but ...

On my 4 year old MacBook:

,
| 
| #+PROPERTY: session *R*
| 
| #+name: bigcsv
| #+begin_src R 
| bigcsv <- Sys.glob("~/Downloads/*.csv")
| #+end_src
| 
| #+RESULTS: bigcsv
| : /Users/cberry/Downloads/test-file.csv
| 
| #+name: readbig
| #+begin_src R :results output
|   system.time(
| tmp <- read.csv(bigcsv)
| )
|   
| #+end_src
| 
| #+RESULTS: readbig
| :user  system elapsed 
| :   5.679   0.306   6.002
| 
`

About the same as running from ESS.

FYI,

> dim(tmp)
[1] 964805  9


Chuck

[ rest deleted ]

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

On Wed, Oct 31, 2012 at 11:41 AM,  wrote:

> John Hendy  writes:
>
> > I edited the subject to be more concise/clear.I let orgmode chug away
> > on reading in some ~10-30mb csv files for nearly 30min.
>
> [rest deleted]
>
> You need an ECM.
>

I did my best to provide one, other than the file, which I offered to
provide if others requested that I upload it somewhere. Since you have done
so, so have I:
- https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM

Let me know if that doesn't work. I put it on Google docs and sometimes
have issues with the sharing settings...

You will find the rest needed for a complete ECM of my setup in the
original email:
- minimal config I used with `emacs -Q --load=file`
- org file with minimal babel R block

Let me know if you can reproduce with the linked file. I'm open to
suggestions on how to bisect a file, though it's a column of strings and
several columns of numbers, I'm not exactly sure what to look for.


Best regards,
John


>
> I cannot reproduce your issue.
>
> This runs in the same amount of time, whether I execute the src block or
> run the code from ESS:
>
>
> #+begin_src R :session
> write.csv(diag(10)[rep(1:10,10),],file="abc.csv")
> #+end_src
>
> #+begin_src R :session :results output
>   system.time(read.csv("abc.csv"))
> #+end_src
>
>
> abc.csv is >25MB
>
> If the behavior you describe only happens with some files, I suggest you
> try to bisect them to find the issue.
>
> HTH,
>
> Chuck
>
>
>

Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[cberry]

John Hendy  writes:

> I edited the subject to be more concise/clear.I let orgmode chug away
> on reading in some ~10-30mb csv files for nearly 30min. 

[rest deleted]

You need an ECM. 

I cannot reproduce your issue.

This runs in the same amount of time, whether I execute the src block or
run the code from ESS:


#+begin_src R :session 
write.csv(diag(10)[rep(1:10,10),],file="abc.csv")
#+end_src

#+begin_src R :session :results output
  system.time(read.csv("abc.csv"))
#+end_src


abc.csv is >25MB

If the behavior you describe only happens with some files, I suggest you
try to bisect them to find the issue.

HTH,

Chuck

[O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly

[John Hendy]

I edited the subject to be more concise/clear.

I let orgmode chug away on reading in some ~10-30mb csv files for nearly
30min. I finally gave up with Ctrl + g. Out of curiosity, I copied my babel
block code (four simple lines in the form of `var <- read.csv("./file.csv",
header = T, sep = ",")` into the ESS session already started by my first
block execution, and it finished in less than 10 seconds.

Something seems seriously wrong with however org + babel is executing. I'm
suspicious of it printing the output to the minibuffer. I have about 200^2
- 400^2 lines in the csv files. Was shocked that I'd been waiting for org
this whole time and could have just run the code in the ESS R session
directly in a minuscule fraction of the time.

I could post a reproducible csv file if others would like to try this?

I tried the following to make sure it wasn't my setup:

1) from terminal: `emacs -Q --load=~/Desktop/basic-config`, where
basic-config contains:

;; set load paths
;; set load dirs and global config options
(add-to-list 'load-path "~/.elisp/site-lisp/")
(add-to-list 'load-path "~/.elisp/org.git/contrib/lisp/")
(add-to-list 'load-path "~/.elisp/org.git/lisp/")
(add-to-list 'load-path "~/.elisp/site-lisp/ess/lisp/")

;; setup org
(require 'org-install)

;; setup babel languages
(require 'ess-site)
(org-babel-do-load-languages
 'org-babel-load-languages
 '((R . t)))

2) Created a simple org file:

--
* headline

#+begin_src R :session r :results silent
data <-
read.csv("/media/vault/misc/benefit-plan-analysis/2012/contour_table.csv")
#+end_src
--

3) Run =C-c C-c YES RET= on block

4) Change to R session and enter the same line manually.

The results are unbelievably different. I didn't let the org session finish
as once it took longer than 30sec I figured I'd get the same results as
with my actual file in this example. Changing to the R session and it was
done in less than 5sec with a ~70mb csv.

Any suggestions on troubleshooting this?


Best regards,
John

On Tue, Oct 30, 2012 at 2:46 AM, John Hendy  wrote:

> I'm working with a large dataset. I try to minimize overhead by doing
> some bigger data rearranging and calculations in a single block that I
> execute once per session and then switch :eval to "no."
>
> This block is set with :results silent, but I know with this option
> the results are still printed to the minibuffer. Usually this isn't a
> problem, but I've had some hangs with emacs and started wondering if
> this could be due to Orgmode regurgitating the results. Could this be
> the case? If so, could Org be made to stop this behavior?
>
> If I accidentally type the name of a data frame in the R session
> buffer, even when I use C-c C-c  to stop the print out, it still takes
> quite a while to catch up and stop itself. It made me wonder if trying
> to print all of those results is increasing my compilation times
> unnecessarily.
>
>
> Thanks for any thoughts,
> John
>