Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> Thanks!
> Now applied onto main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=192742c9c5a4c6a1470674343a09e5ba1f1bc440
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8901fd2261a8a16917b32eb9e43c7ea43b51

Thank you for all your help!

>From 7+ minutes back to 14-second exports.

Onward and upward!

R+

-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
-- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

>> (I'd like this patches to be tested on real export first as it is
>> touching on very basic export routines).
>
> For what is worth, I have tried the patch.  With the patch, Org exports
> my elaborate notebook as fast as without it (+/- 1 second) and the tests
> pass on macOS and Emacs 29.  Just FYI.

Thanks!
Now applied onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=192742c9c5a4c6a1470674343a09e5ba1f1bc440
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8901fd2261a8a16917b32eb9e43c7ea43b51

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> The patch is only working by accident because [...]

TIL!

> (I'd like this patches to be tested on real export first as it is
> touching on very basic export routines).

For what is worth, I have tried the patch.  With the patch, Org exports
my elaborate notebook as fast as without it (+/- 1 second) and the tests
pass on macOS and Emacs 29.  Just FYI.

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

>>> (1) re-opening the file after export always re-opens it, and
>>
>> I am not sure how to test this part.
>
> How about the attached patch?
>
> It mirrors the manual steps (find-file, export, kill-buffer, find-file),
> except it calls just the 'org-export-copy-buffer' procedure instead of
> exporting to HTML.  The test fails without your fix and passes with it.

The test in the patch is erroneous. `org-export-copy-buffer', unless it
kills the generated buffer, does nothing about `buffer-file-name' local
variable.  The patch is only working by accident because `find-file'
opens the first buffer with matching `buffer-file-name'. If there are
multiple such buffers, the return value depends on their order.

I have updated `org-export-with-buffer-copy' macro to take care about
this `buffer-file-name' issue uniformly without a need to put
workarounds into every caller. I have also updated your patch using a
more appropriate testing assertions. It now uses the updated macro.

See the attached patches.
(I'd like this patches to be tested on real export first as it is touching
on very basic export routines).

>From e8e4065da6e166e9859e14c95558e385d320dcbe Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sun, 9 Oct 2022 14:16:41 +0800
Subject: [PATCH 1/2] org-export-with-buffer-copy: Allow safe reusable buffer
 copy
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ox.el (org-export--generate-copy-script):
(org-export-copy-buffer):  Accept cl-style key arguments.
* lisp/ox.el (org-export-with-buffer-copy): Allow optional cl-style
key arguments for fine-grained control on what to copy from the target
buffer and which buffer to use as a target.  Do not kill the target
buffer upon finished, when the target is provided.  Make sure that
remaining buffer copy does not preserve `buffer-file-name' from the
original buffer and that it never overwrites the file associated with
the original buffer.
* lisp/ox-html.el (org-html-format-latex): Use the new version of
`org-export-with-buffer-copy' instead of managing edge cases by
itself.

Reported-by: Rudolf Adamkovič 
Link: https://orgmode.org/list/87zge8j5iu.fsf@localhost
---
 lisp/ox-html.el | 27 +---
 lisp/ox.el  | 82 ++---
 2 files changed, 59 insertions(+), 50 deletions(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 3f55be2ee..cad06aebf 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -2879,25 +2879,14 @@ (defun org-html-format-latex (latex-frag processing-type info)
 	;; temporary buffer so that dvipng/imagemagick can properly
 	;; turn the fragment into an image.
 	(setq latex-frag (concat latex-header latex-frag
-(with-current-buffer
-(org-export-copy-buffer
- (get-buffer-create " *Org HTML Export LaTeX*")
- 'drop-visible 'drop-narrowing 'drop-contents)
-  (erase-buffer)
-  (insert latex-frag)
-  (org-format-latex cache-relpath nil nil cache-dir nil
-		"Creating LaTeX Image..." nil processing-type)
-  ;; Present save dialogue to be shown for this buffer and prevent
-  ;; Emacs to jump into this buffer when opening
-  ;; `buffer-file-name' file.  We need this because
-  ;; `org-export-copy-buffer' copies `buffer-file-name' local
-  ;; variable thus making Emacs think that the buffer copy is
-  ;; associated with file.  Note that despite `buffer-file-name',
-  ;; `org-export-copy-buffer' arranges saving to not perform
-  ;; actual writing onto the disk.
-  (setq buffer-file-name nil)
-  (restore-buffer-modified-p nil)
-  (buffer-string
+(org-export-with-buffer-copy
+ :to-buffer (get-buffer-create " *Org HTML Export LaTeX*")
+ :drop-visibility t :drop-narrowing t :drop-contents t
+ (erase-buffer)
+ (insert latex-frag)
+ (org-format-latex cache-relpath nil nil cache-dir nil
+		   "Creating LaTeX Image..." nil processing-type)
+ (buffer-string
 
 (defun org-html--wrap-latex-environment (contents _  caption label)
   "Wrap CONTENTS string within appropriate environment for equations.
diff --git a/lisp/ox.el b/lisp/ox.el
index e059983fb..69a95ffe5 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2544,9 +2544,9 @@ ;;; Core functions
 ;; a default template (or a back-end specific template) at point or in
 ;; current subtree.
 
-(defun org-export-copy-buffer ( buffer drop-visibility
- drop-narrowing drop-contents
- drop-locals)
+(cl-defun org-export-copy-buffer ( to-buffer drop-visibility
+   drop-narrowing drop-contents
+   drop-locals)
   "Return a copy of the current buffer.
 The copy preserves Org buffer-local variables, visibility and
 narrowing.
@@ -2561,61 +2561,81 @@ (defun org-export-copy-buffer ( buffer drop-visibility
 Emacs 

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> I do not see how it is related.  Do you see the error with your config
> or with clean Emacs?

Good point.  Just with my config.  Ignoring for now!

> Oops. Another side-effect of `buffer-file-name'. Fixed now.

Thank you!

>> (2) no corruption happens to the original file after export.
>
> Done.

Superb!

>> (1) re-opening the file after export always re-opens it, and
>
> I am not sure how to test this part.

How about the attached patch?

It mirrors the manual steps (find-file, export, kill-buffer, find-file),
except it calls just the 'org-export-copy-buffer' procedure instead of
exporting to HTML.  The test fails without your fix and passes with it.

Rudy

>From 8bc0f1e807df653634e77e5e5fc543a2e82191c2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rudolf=20Adamkovi=C4=8D?= 
Date: Fri, 7 Oct 2022 16:04:28 +0200
Subject: [PATCH] Add a regression test for 'org-export-copy-buffer'

* testing/lisp/test-ox.el (test-org-export/org-export-copy-buffer):
Make sure that 'find-file' after 'org-export-copy-buffer' re-opens the
original file instead of the copy buffer.
---
 testing/lisp/test-ox.el | 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/testing/lisp/test-ox.el b/testing/lisp/test-ox.el
index 90a9eb3aa..60bf04316 100644
--- a/testing/lisp/test-ox.el
+++ b/testing/lisp/test-ox.el
@@ -63,20 +63,31 @@ variable, and communication channel under `info'."
 
 (ert-deftest test-org-export/org-export-copy-buffer ()
   "Test `org-export-copy-buffer' specifications."
-  ;; The buffer copy must not cause overwriting the original file
-  ;; buffer under any circumstances.
+  ;; The copy must not overwrite the original file.
   (org-test-with-temp-text-in-file
   "* Heading"
 (let ((file (buffer-file-name)))
   (with-current-buffer (org-export-copy-buffer)
-(insert "This must not go into actual file.")
+(insert "This must not go into the original file.")
 (save-buffer)
 (should
  (equal
   "* Heading"
   (with-temp-buffer
 (insert-file-contents file)
-(buffer-string
+(buffer-string)))
+  ;; The copy must not show when re-opening the original file.
+  (org-test-with-temp-text-in-file
+  "* Heading"
+(let ((file (buffer-file-name)))
+  (with-current-buffer (org-export-copy-buffer)
+(insert "This must not show as the original file.")
+(save-buffer))
+  (should
+   (equal "* Heading"
+  (with-temp-buffer
+(find-file file)
+(buffer-string)))
 
 (ert-deftest test-org-export/bind-keyword ()
   "Test reading #+BIND: keywords."
-- 
2.37.3

-- 
"Chop your own wood and it will warm you twice."
-- Henry Ford; Francis Kinloch, 1819; Henry David Thoreau, 1854

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> Ihor Radchenko  writes:
>
>> Rudolf, please kindly confirm that the export performance is back to
>> satisfactory on the latest main.
>
> Performance back to normal!  Though, I noticed two issues.
>
> *** ISSUE 1: Error when exporting. ***
>
> While exporting, I see the error:
>
> "Org mode fontification error in # at 10"
>
> The error does not bother me, but I mention it in the case it relates to
> the next issue.

I do not see how it is related.
Do you see the error with your config or with clean Emacs?

> *** ISSUE 2: Problems re-opening the notebook. ***
>
> Reproduction steps:
>
> 1. open 'notebook.org' (C-x C-f ~/notebook.org RET)
> 2. export to HTML and open (C-c C-e h o)
> 3. kill the notebook buffer (C-x k RET)
> 4. re-open notebook (C-x C-f ~/notebook.org RET)
>
> Expected:
>
> 'notebook.org' opens
>
> Actual:
>
> *Org HTML Export LaTeX* opens, with the following content:
>
> \begin{align*}
>   \frac{dy}{dx}
>   & = y \left( -1 - 2 \tan x - \frac{2x + 1}{x^2 + x + 1} \right)
>   \\
>   & = \frac{e^{-x} {\cos}^2 x}{x^2 + x + 1}
> \left( -1 - 2 \tan x - \frac{2x + 1}{x^2 + x + 1} \right)
> \end{align*}

Oops. Another side-effect of `buffer-file-name'. Fixed now.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=86b6d843f6e8d33c0dcf414f7fda2b1ac6492550

> P.S. The original file remains intact, but given the way we implement
> exports in Org, I think we should add two tests to the test suite to
> verify that:

> (2) no corruption happens to the original file after export.

Done.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3de74bbeff704a1e297d0f8c445790928e71344f

> (1) re-opening the file after export always re-opens it, and

I am not sure how to test this part.
I instead documented the caveats in `org-export-copy-buffer' docstring
and in the `org-html-format-latex'.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=17708ab3fa4f2a0f118bf86cebd4753f50e8bb26
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=86b6d843f6e8d33c0dcf414f7fda2b1ac6492550

Though maybe the code in `org-html-format-latex' should be factored out
into a separate macro like `org-export-with-persistent-buffer-copy'. The
macro will accept buffer name as an argument and do everything necessary
to not keep the `buffer-file-name' around.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> Rudolf, please kindly confirm that the export performance is back to
> satisfactory on the latest main.

Performance back to normal!  Though, I noticed two issues.

*** ISSUE 1: Error when exporting. ***

While exporting, I see the error:

"Org mode fontification error in # at 10"

The error does not bother me, but I mention it in the case it relates to
the next issue.

*** ISSUE 2: Problems re-opening the notebook. ***

Reproduction steps:

1. open 'notebook.org' (C-x C-f ~/notebook.org RET)
2. export to HTML and open (C-c C-e h o)
3. kill the notebook buffer (C-x k RET)
4. re-open notebook (C-x C-f ~/notebook.org RET)

Expected:

'notebook.org' opens

Actual:

*Org HTML Export LaTeX* opens, with the following content:

\begin{align*}
  \frac{dy}{dx}
  & = y \left( -1 - 2 \tan x - \frac{2x + 1}{x^2 + x + 1} \right)
  \\
  & = \frac{e^{-x} {\cos}^2 x}{x^2 + x + 1}
\left( -1 - 2 \tan x - \frac{2x + 1}{x^2 + x + 1} \right)
\end{align*}

P.S. The original file remains intact, but given the way we implement
exports in Org, I think we should add two tests to the test suite to
verify that:

(1) re-opening the file after export always re-opens it, and
(2) no corruption happens to the original file after export.

Rudy
-- 
"Genius is 1% inspiration and 99% perspiration."
-- Thomas Alva Edison, 1932

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

After further off-list discussion and online debugging, we have managed
to find out the culprit.

Thanks, Rudolf, for the kind assistance with debugging!

The slowdown mostly came from the new version of `org-html-format-latex'
that copies the parent Org buffer environment when generating LaTeX
images. There were a couple of bottlenecks.

In addition, we have found a critical bug caused by `buffer-file-name'
being non-nil in the buffer copies during export. The bug existed
forever but was previously much harder to be triggered.

Now, everything should be fixed on main via
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5a167f6ef20e337c9b1063097ebe64624ad932a6
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=aa90e5b59cc0fe41072cd6f78968e0b8d5ac6939
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=4cb1b8ebd006e0659a34c942ab5594f177d6bf49
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=0551eba40ca1cdeac8645c8614747566025a7e94
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=b45911d41e9d81bc23b93d7e75d0a7611ae18f38
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=09fd5f886a81af686ac0b69af19eb7a8d2007284
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=514e3a9cd26e150f6ee7251433e14701b62f7e98
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=dcdf98a564d17f918afbb947dfc6c96244b94abb
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=0be36ac13eee17b3bf26ea55ba596bf61b71602d

Export copy buffer now can never be saved to disk and will never corrupt
the source Org file.

Rudolf, please kindly confirm that the export performance is back to
satisfactory on the latest main.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> Can you execute [...] after loading org and try exporting again?


* EXPERIMENT 1 - TREATMENT *


*** VERSIONS:

- Emacs 29 (adaa2fc90e)
- Org (ebbc2ffaa)

*** CONFIGURATION:

;;; -*- lexical-binding: t; -*-
;;; .emacs --- Emacs configuration

;; Make the Org mode less dependent on the arrow keys.
;; From the documentation: "You must set it before loading Org."
(setq-default org-use-extra-keys t)

;; Minimal configuration.
(setq org-export-use-babel nil
  org-export-with-broken-links 'mark
  org-cite-csl-styles-dir "~/"
  org-cite-export-processors '((t csl)))
(with-eval-after-load 'ox
  (require 'oc-csl))

;; EXPERIMENT
;; Can you please set org-element-use-cache to nil and org-fold-core-style
;; to 'overlays in your config and try the export again?
(setq org-element-use-cache nil
  org-fold-core-style 'overlays)

;; Load the latest Org mode.
(add-to-list 'load-path "~/src/org-mode/lisp/")

;; EXPERIMENT
;; Can you execute [...] after loading org and try exporting again?
(with-eval-after-load 'org
  (defun org-export-copy-buffer ()
"Return a copy of the current buffer.
The copy preserves Org buffer-local variables, visibility and
narrowing."
(let ((copy-buffer-fun (org-export--generate-copy-script (current-buffer)))
  (new-buf (generate-new-buffer (buffer-name) 'inhibit-hooks)))
  (with-current-buffer new-buf
(funcall copy-buffer-fun)
(set-buffer-modified-p nil))
  new-buf)))

*** EXPORT:

completed, took 1 minute and 56 seconds

*** NOTES:

I could not capture the profile.  This time, after both
'profiler-report' or 'profiler-stop', I got:

cl--assertion-failed: Assertion failed: (>= match imatch)

**
* EXPERIMENT 2 - CONTROL *
**

*** VERSIONS:

- Emacs 29 (adaa2fc90e)
- Org (ebbc2ffaa)

*** CONFIGURATION:

;;; -*- lexical-binding: t; -*-
;;; .emacs --- Emacs configuration

;; Make the Org mode less dependent on the arrow keys.
;; From the documentation: "You must set it before loading Org."
(setq-default org-use-extra-keys t)

;; Minimal configuration.
(setq org-export-use-babel nil
  org-export-with-broken-links 'mark
  org-cite-csl-styles-dir "~/"
  org-cite-export-processors '((t csl)))
(with-eval-after-load 'ox
  (require 'oc-csl))

*** EXPORT:

completed, took 12 seconds

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
-- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

>> May you try with Emacs version that is not bleeding edge? Like Emacs
>> 28.
>
> So, I did the following:
>
> + removed Emacs 29
> + installed Emacs 28.2
> ...
> + waited for 1 minute and 35 seconds
> + saved 'emacs-28-org-export.profile'

Thanks!

Can you execute

(defun org-export-copy-buffer ()
  "Return a copy of the current buffer.
The copy preserves Org buffer-local variables, visibility and
narrowing."
  (let ((copy-buffer-fun (org-export--generate-copy-script (current-buffer)))
(new-buf (generate-new-buffer (buffer-name) 'inhibit-hooks)))
(with-current-buffer new-buf
  (funcall copy-buffer-fun)
  (set-buffer-modified-p nil))
new-buf))

after loading org and try exporting again?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> All right, I did the following 10 experiments.

Thanks again!

> RESULT:
>
>   Exporting crashes Emacs.

*shrug*
May you try with Emacs version that is not bleeding edge? Like Emacs 28. 

> *
> *** EXPERIMENT 10 ***
> *
>
> CONFIGURATION:
>
>   The same as Experiment 3-8 and 9.
>
> RESULT:
>
>   HTML exported in 1 minute 31 seconds.
>
> NOTES:
>
>   The profile attached to this message as 'export-3-cache.profile'.

There is something very strange going on there.
A lot of time is taken running org-mode-hook. But (!) org-mode-hook is
explicitly let-bound to nil in this context.
A lot of time is taken parsing the buffer, when the functions must (!)
accept already parsed element.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> UPDATE.
>
> I tried exporting again (after some document clean up; most probably
> unrelated); and I found that exporting sometimes works!
>
> Four attempts, chronologically:
>
> 1. New Org, nil cache + core overlays -- 3 minutes 21 seconds
> 2. New Org -- 3 minutes 22 seconds
> 3. Old Org -- 12 seconds
> 4. New Org (again) -- aborted (C-g) after 5 minutes and 30 seconds
>
> 3 minutes and 22 seconds make exports unbearably slow, but at least
> possible.  What gives?

Thanks for the update!
What if you run M-x org-element-cache-reset after exiting the hang?

Also, if you can get the export finish, can you also share the profiler
report? (M-x profiler-report-write-profile from inside the profiler
report buffer).

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Rudolf Adamkovič  writes:

> No luck!

UPDATE.

I tried exporting again (after some document clean up; most probably
unrelated); and I found that exporting sometimes works!

Four attempts, chronologically:

1. New Org, nil cache + core overlays -- 3 minutes 21 seconds
2. New Org -- 3 minutes 22 seconds
3. Old Org -- 12 seconds
4. New Org (again) -- aborted (C-g) after 5 minutes and 30 seconds

3 minutes and 22 seconds make exports unbearably slow, but at least
possible.  What gives?

Rudy
-- 
"Programming reliably -- must be an activity of an undeniably
mathematical nature […] You see, mathematics is about thinking, and
doing mathematics is always trying to think as well as possible."
-- Edsger W. Dijkstra, 1981

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

Thank you for helping me, Ihor.

> Let's try to approach this problem from another direction.
>
> Can you please set org-element-use-cache to nil and org-fold-core-style
> to 'overlays in your config and try the export again?

No luck!  I used the following configuration in '.emacs', FYI:

;; Make the Org mode less dependent on the arrow keys.
;; From the documentation: "You must set it before loading Org."
(setq-default org-use-extra-keys t)

;; Minimal configuration.
(setq org-export-use-babel nil
  org-export-with-broken-links 'mark
  org-cite-csl-styles-dir "~/"
  org-cite-export-processors '((t csl)))
(with-eval-after-load 'ox
  (require 'oc-csl))

;; EXPERIMENT
(setq org-element-use-cache nil
  org-fold-core-style 'overlays)

;; Load the latest Org mode.
(add-to-list 'load-path "~/src/org-mode/lisp/")

> If you see not improvement, we may arrange a screen session for more
> interactive debugging.

Amazing!  I can pay for your time, of course.

Rudy
-- 
"'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and
if it were so, it would be; but as it isn't, it ain't.  That's logic.'"
-- Lewis Carroll, Through the Looking Glass, 1871/1872

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Max Nikulin]

On 28/09/2022 19:55, Rudolf Adamkovič wrote:

Rudolf Adamkovič writes:

Also, I forgot to mention that Org mode from Org/Git 'main' quietly
deletes all COMMENT headings from my notebook, including their content
(!), and inlines all macros.  I have no idea why, but if I did not use
Git, I would have lost important data.  Scary!


Export as org could generate such result, but it adds additional .org 
suffix to the file name, so it is strange if your file was overwritten.

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Rudolf Adamkovič  writes:

Also, I forgot to mention that Org mode from Org/Git 'main' quietly
deletes all COMMENT headings from my notebook, including their content
(!), and inlines all macros.  I have no idea why, but if I did not use
Git, I would have lost important data.  Scary!

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
-- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

Thank you for helping me out.

Context:

With all this talk about Emacs 29 around the corner, I became scared
that the upcoming Org will render my Org notes unusable.  Hence, I
decided to ask now, before it happens.

> Can you
> 1. M-x debug-on-entry  org-export--generate-copy-script 
> 2. Start export
> 3. Step into the function (use "d", not "c" binding)
> 4. Try to identify which part (if any) is being slow

FYI: I had to do "(require 'ox)" before this worked.

I spent about 30 minutes pressing "d".  I saw Org sorting some character
encodings in a large number of steps, and then it started doing a large
number of various tasks, spending time in the AVL tree, etc.  I have no
idea what else to do.

I could not figure out how to step through N times at once for N > 1.  I
tried 'C-9 C-9 d' to step through 99 times, but that did not work.  The
documentation for 'debugger-step-through' contains just two sentences,
and I do not even understand the second one:

"Proceed, stepping through subexpressions of this expression.  Enter
another debugger on next entry to eval, apply or funcall."

> If the above does not work, you can also try
> 1. M-x trace-function  org-export--generate-copy-script 
> 2. M-x elp-instrument-function  org-export--generate-copy-script
>
> 2. Run export, possibly terminating it with C-g in the process
> 3. M-x elp-results 
> 4. Share the contents of the trace buffer and the contents of the
>elp-results buffer.

So, I ran the export for 17 minutes and then terminated with 'C-g'.  The
trace buffer included a very long line, which made Emacs froze, and I
had to kill it, after waiting for 5 minutes, losing all data in the
process.

I then tried to run the export for 3 minutes, and I managed to copy the
trace buffer, but upon pasting it into another buffer, Emacs froze
again, and I had to kill it again, losing data once again.

I then ran the export for 3 minutes once again, this time enabling just
the 'elp-instrument-function' to avoid freezing.  The 'elp-results'
output:

> org-export--generate-copy-script 1221 2.826854 0.0023151957

(The buffer also contains a header line, with e.g. "Function Name", but
Emacs did not allow me to select and copy it for you.)

Rudy
-- 
"Mathematics takes us still further from what is human into the region
of absolute necessity, to which not only the actual world, but every
possible world, must conform."
-- Bertrand Russell, 1902

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> Rudolf Adamkovič  writes:
>
> Also, I forgot to mention that Org mode from Org/Git 'main' quietly
> deletes all COMMENT headings from my notebook, including their content
> (!), and inlines all macros.  I have no idea why, but if I did not use
> Git, I would have lost important data.  Scary!

I assure you that there is nothing in Org code that intentionally
deletes things in files without user issuing explicit command.

The only thing that might mess up edits is terribly misbehaving
org-element-cache, but it is safeguarded by self-verification.

Having said that, do note that Emacs 29 currently contains a risky new
feature aiming to optimize buffers with long lines. This can badly
affect Org parser (bug#57207).

May it help if you set long-line-threshold to nil in your config?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

>> Can you
>> 1. M-x debug-on-entry  org-export--generate-copy-script 
>> 2. Start export
>> 3. Step into the function (use "d", not "c" binding)
>> 4. Try to identify which part (if any) is being slow
>
> FYI: I had to do "(require 'ox)" before this worked.
>
> I spent about 30 minutes pressing "d".  I saw Org sorting some character
> encodings in a large number of steps, and then it started doing a large
> number of various tasks, spending time in the AVL tree, etc.  I have no
> idea what else to do.

That's quite an involved stepping through. Thanks for your time, and sorry
for not being clear enough about what you can do!

The basic idea of this kind of test is use "d" to step-in once and then
continue with "c" until you run into a long-running function. Then, you
restart the process but also "d" into that identified function and again
continue with "c". There is no need to dig to the bottom. At least, I
expected to know if and individual call to
org-export--generate-copy-script is slow and what inside it may be
causing the slowdown.

In any case, your elp results combined with previous message when you
used "c" imply that (1) org-export--generate-copy-script is called
recursively; (2) the top-level call is taking a lot of time somehow.

This is all very strange.

Let's try to approach this problem from another direction.

Can you please set org-element-use-cache to nil and org-fold-core-style
to 'overlays in your config and try the export again?

These are two major new features that are complex enough to cause
potential issues.

If you see not improvement, we may arrange a screen session for more
interactive debugging.

> I could not figure out how to step through N times at once for N > 1.  I
> tried 'C-9 C-9 d' to step through 99 times, but that did not work.  The
> documentation for 'debugger-step-through' contains just two sentences,
> and I do not even understand the second one:
>
> "Proceed, stepping through subexpressions of this expression.  Enter
> another debugger on next entry to eval, apply or funcall."

Usually, "d" (step-in) is combined with "c" (step through).

> (The buffer also contains a header line, with e.g. "Function Name", but
> Emacs did not allow me to select and copy it for you.)

That's because it is not a part of buffer text. Just like modeline.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> "Fraga, Eric"  writes:
>
>> Try M-x toggle-debug-on-quit RET and then export, hitting C-g when it
>> gets stuck.  That'll give you a backtrace which might help debug this.
>
> Thank you Eric.  I tried it and stopped after more than 3 minutes of
> waiting.  The output:
>
> ;; Debugger entered--Lisp error: (quit)
> ;;   org-export--generate-copy-script(#>)
> ;;   org-export-copy-buffer()
> ;;   org-html-format-latex( ...

This does give some useful information.

Can you
1. M-x debug-on-entry  org-export--generate-copy-script 
2. Start export
3. Step into the function (use "d", not "c" binding)
4. Try to identify which part (if any) is being slow

If the above does not work, you can also try
1. M-x trace-function  org-export--generate-copy-script 
2. M-x elp-instrument-function  org-export--generate-copy-script 
2. Run export, possibly terminating it with C-g in the process
3. M-x elp-results 
4. Share the contents of the trace buffer and the contents of the
   elp-results buffer.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Ihor Radchenko  writes:

> Profiler reports after C-g are not reliable.

TIL!

> In addition to debug-on-quit, you can M-x debug-on-entry org-export-as
> and try to identify which part of that function is lagging.

I tried it and it says ...

Debugger entered--entering a function:
* org-export-as(html nil nil nil (:output-file "notebook.html"))
  org-export-to-file(html "notebook.html" nil nil nil nil nil)
  org-html-export-to-html(nil nil nil nil)
  (org-open-file (org-html-export-to-html nil s v b))
  (if a (org-html-export-to-html t s v b) (org-open-file 
(org-html-export-to-html nil s v b)))
  (lambda (a s v b) (if a (org-html-export-to-html t s v b) (org-open-file 
(org-html-export-to-html nil s v b(nil nil nil nil)
  org-export-dispatch(nil)
  funcall-interactively(org-export-dispatch nil)
  call-interactively(org-export-dispatch nil nil)
  command-execute(org-export-dispatch)

... and then just "Continuing." and nothing else.  I waited for more
than 3 minutes.

Rudy
-- 
"The whole science is nothing more than a refinement of everyday
thinking."
-- Albert Einstein, 1879-1955

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: The Org mode in the Org Git does not export

[Rudolf Adamkovič]

"Fraga, Eric"  writes:

> Try M-x toggle-debug-on-quit RET and then export, hitting C-g when it
> gets stuck.  That'll give you a backtrace which might help debug this.

Thank you Eric.  I tried it and stopped after more than 3 minutes of
waiting.  The output:

;; Debugger entered--Lisp error: (quit)
;;   org-export--generate-copy-script(#>)
;;   org-export-copy-buffer()
;;   org-html-format-latex("\\begin{equation*}\n  (V_A^R, V_A^..." mathjax 
(:export-options nil :back-end #s(org-export-backend :name html :parent nil 
:transcoders ((bold . org-html-bold) (center-block . org-html-center-block) 
(clock . org-html-clock) (code . org-html-code) (drawer . org-html-drawer) 
(dynamic-block . org-html-dynamic-block) (entity . org-html-entity) 
(example-block . org-html-example-block) (export-block . org-html-export-block) 
(export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) 
(footnote-reference . org-html-footnote-reference) (headline . 
org-html-headline) (horizontal-rule . org-html-horizontal-rule) 
(inline-src-block . org-html-inline-src-block) (inlinetask . 
org-html-inlinetask) (inner-template . org-html-inner-template) (italic . 
org-html-italic) (item . org-html-item) (keyword . org-html-keyword) 
(latex-environment . org-html-latex-environment) (latex-fragment . 
org-html-latex-fragment) (line-break . org-html-line-break) (link . 
org-html-link) (node-property . org-html-node-property) (paragraph . 
org-html-paragraph) (plain-list . org-html-plain-list) (plain-text . 
org-html-plain-text) (planning . org-html-planning) (property-drawer . 
org-html-property-drawer) (quote-block . org-html-quote-block) (radio-target . 
org-html-radio-target) ...) :options ((:html-doctype "HTML_DOCTYPE" nil 
org-html-doctype) (:html-container "HTML_CONTAINER" nil 
org-html-container-element) (:html-content-class "HTML_CONTENT_CLASS" nil 
org-html-content-class) (:description "DESCRIPTION" nil nil newline) (:keywords 
"KEYWORDS" nil nil space) (:html-html5-fancy nil "html5-fancy" 
org-html-html5-fancy) (:html-link-use-abs-url nil "html-link-use-abs-url" 
org-html-link-use-abs-url) (:html-link-home "HTML_LINK_HOME" nil 
org-html-link-home) (:html-link-up "HTML_LINK_UP" nil org-html-link-up) 
(:html-mathjax "HTML_MATHJAX" nil "" space) (:html-equation-reference-format 
"HTML_EQUATION_REFERENCE_FORMAT" nil org-html-equation-reference-format t) 
(:html-postamble nil "html-postamble" org-html-postamble) (:html-preamble nil 
"html-preamble" org-html-preamble) (:html-head "HTML_HEAD" nil org-html-head 
newline) (:html-head-extra "HTML_HEAD_EXTRA" nil org-html-head-extra newline) 
(:subtitle "SUBTITLE" nil nil parse) (:html-head-include-default-style nil 
"html-style" org-html-head-include-default-style) (:html-head-include-scripts 
nil "html-scripts" org-html-head-include-scripts) 
(:html-allow-name-attribute-in-anchors nil nil 
org-html-allow-name-attribute-in-anchors) (:html-divs nil nil org-html-divs) 
(:html-checkbox-type nil nil org-html-checkbox-type) (:html-extension nil nil 
org-html-extension) (:html-footnote-format nil nil org-html-footnote-format) 
(:html-footnote-separator nil nil org-html-footnote-separator) 
(:html-footnotes-section nil nil org-html-footnotes-section) 
(:html-format-drawer-function nil nil org-html-format-drawer-function) 
(:html-format-headline-function nil nil org-html-format-headline-function) 
(:html-format-inlinetask-function nil nil org-html-format-inlinetask-function) 
(:html-home/up-format nil nil org-html-home/up-format) (:html-indent nil nil 
org-html-indent) (:html-infojs-options nil nil org-html-infojs-options) 
(:html-infojs-template nil nil org-html-infojs-template) ...) :filters 
((:filter-options . org-html-infojs-install-script) (:filter-parse-tree . 
org-html-image-link-filter) (:filter-final-output . org-html-final-function)) 
:blocks nil :menu (104 "Export to HTML" (... ... ...))) :translate-alist ((bold 
. org-html-bold) (center-block . org-html-center-block) (clock . 
org-html-clock) (code . org-html-code) (drawer . org-html-drawer) 
(dynamic-block . org-html-dynamic-block) (entity . org-html-entity) 
(example-block . org-html-example-block) (export-block . org-html-export-block) 
(export-snippet . org-html-export-snippet) (fixed-width . org-html-fixed-width) 
(footnote-reference . org-html-footnote-reference) (headline . 
org-html-headline) (horizontal-rule . org-html-horizontal-rule) 
(inline-src-block . org-html-inline-src-block) (inlinetask . 
org-html-inlinetask) (inner-template . org-html-inner-template) (italic . 
org-html-italic) (item . org-html-item) (keyword . org-html-keyword) 
(latex-environment . org-html-latex-environment) (latex-fragment . 
org-html-latex-fragment) (line-break . org-html-line-break) (link . 
org-html-link) (node-property . org-html-node-property) (paragraph . 
org-html-paragraph) (plain-list . org-html-plain-list) (plain-text . 
org-html-plain-text) (planning . org-html-planning) (property-drawer . 

Re: The Org mode in the Org Git does not export

[Ihor Radchenko]

Rudolf Adamkovič  writes:

> When I attach the profiler before exporting and then abort with C-g, the
> profiler reports that Org spends almost all of the time in
> 'org-export--generate-copy-script'.

Profiler reports after C-g are not reliable.

> Any ideas?

In addition to debug-on-quit, you can M-x debug-on-entry org-export-as
and try to identify which part of that function is lagging.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: The Org mode in the Org Git does not export

[Fraga, Eric]

On Sunday, 25 Sep 2022 at 18:40, Rudolf Adamkovič wrote:
> When I attach the profiler before exporting and then abort with C-g, the
> profiler reports that Org spends almost all of the time in
> 'org-export--generate-copy-script'.

Try M-x toggle-debug-on-quit RET and then export, hitting C-g when it
gets stuck.  That'll give you a backtrace which might help debug this.

-- 
: Eric S Fraga, with org release_9.5.5-815-gae2140 in Emacs 29.0.50

The Org mode in the Org Git does not export

[Rudolf Adamkovič]

Hello smart people!

The Org mode from Git does not export my notebook, while the Org mode
that comes with Emacs 29 does.  Is that normal?  My 1.6 MB notebook
contains 2078 headings and lots of prose, mathematics, code, citations,
tables, ... virtually everything.

The minimal configuration used for testing:

| (setq-default org-use-extra-keys t)
| (setq org-export-use-babel nil ; <--- NOTE no Babel execution
|   org-export-with-broken-links 'mark
|   org-cite-csl-styles-dir "~/"
|   org-cite-export-processors '((t csl)))
| (with-eval-after-load 'ox
|   (require 'oc-csl))

The Org mode in Emacs 29 (Emacs Git, e0565e3896) exports the notebook in
about 13 seconds, but the Org mode from Git (Org Git, ebbc2ffaa) keeps
exporting forever.  (FYI, the new Org announces that the buffer is
syntactically correct.)

When I attach the profiler before exporting and then abort with C-g, the
profiler reports that Org spends almost all of the time in
'org-export--generate-copy-script'.

P.S. A while ago, I tried to binary-search the problematic part in my
notebook, but I did not find it.  The export times got progressively
better as I cut the parts.  Worst-case, I can try this again.

Any ideas?

Rudy
-- 
"The whole science is nothing more than a refinement of everyday
thinking."
-- Albert Einstein, 1879-1955

Rudolf Adamkovič  [he/him]
Studenohorská 25
84103 Bratislava
Slovakia