[O] footnote fontify causing massive slowdown
While diagnosing a server condition, I was listing parts of a system log via a babel expression. The 130 lines in the babel output are wrapped in an example block. This block caused massive slowdown of scrolling and other operations. Using the emacs profiler I see: - redisplay_internal (C function) 8232 88% - jit-lock-function 8226 88% - jit-lock-fontify-now 8226 88% - funcall 8226 88% - # 8226 88% - run-hook-with-args 8226 88% - font-lock-fontify-region 8226 88% - font-lock-default-fontify-region 8226 88% - font-lock-fontify-keywords-region 8226 88% - org-activate-footnote-links 8158 87% - org-footnote-next-reference-or-definition 8158 87% - byte-code 8158 87% - org-footnote-at-reference-p 4114 44% - org-footnote-in-valid-context-p 4106 44% + org-inside-LaTeX-fragment-p 2380 25% + org-in-block-p 1563 16% + org-in-verbatim-emphasis 159 1% org-at-comment-p 4 0% Checking for footnote pattern matches (org-footnote-re) in the wrapped block, I see that every line matches based on the very simple and trivial pattern of number enclosed in angular brackets, so all the process numbers following the "sshd" in these lines like "sshd[1234]" do match and cause load. # /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:41 some-host sshd[20089]: pam_ldap: error trying to bind as user "x" (Invalid credentials) # Since this kind of pattern is so common in logs and 130 lines are really not a large number, it makes it hard to use org for this purpose. Can this be turned off selectively, or can it be prevented in example blocks? Using [8.3.2 (8.3.2-37-gd45217-elpaplus @ /home/dfeich/.emacs.d/elpa/org-plus-contrib-20151116/)] with GNU Emacs 24.5.1. Best regards, Derek -- Paul Scherrer Institut Dr. Derek Feichtinger Phone: +41 56 310 47 33 Section Head Scientific Computing Email: derek.feichtin...@psi.ch Building/Room No. WHGA/U126 CH-5232 Villigen PSI
Re: [O] footnote fontify causing massive slowdown
Hello, Thomas S. Dyewrites: > William Denton writes: > >> On 5 December 2015, Nicolas Goaziou wrote: >> >>> However, this report raises an interesting question about footnotes: >>> should we still support plain (e.g., "[1]") footnotes in Org documents? >>> >>> The pattern is very common an regularly introduces false positives. >>> Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode >>> buffers), to provide some common features with "footnote.el" library. >>> >>> I think we could remove this kind of footnotes, and yet preserve >>> `org-footnote-normalize' to change Org footnotes into these ones, for >>> foreign documents. >> >> +1. The false positives are a common problem for me, and [fn:1] works >> cleanly >> and clearly. > > Same here. +1. Thanks for the feedback. I'll send a patch to the ML by the end of the next week. Regards, -- Nicolas Goaziou
Re: [O] footnote fontify causing massive slowdown
Hello, Aaron Ecaywrites: > The way I read the report, org-footnote-in-valid-context-p takes 44% > of the cumulative time, which is composed of > org-inside-LaTeX-fragment-p (25%) + org-in-block-p (16%) + other stuff > (3%). So org-inside-LaTeX-fragment-p accounts for >50% of the time > spent in org-footnote-in-valid-context-p. Interesting. I thought that percents were cumulative, so `org-in-block-p' accounted for 16% - 1% and `org-inside-LaTeX-fragment-p' 25% - 16% of the total time. > I see. Eventually it sounds like a good idea, indeed. Maybe we should > use something a bit stronger than org-lint to warn of the deprecation. > What if opening a document with plain footnotes generated a warning? Generate a warning about plain footnotes deprecation ? How do you turn it off ? Do you require the user to set some variable to a non-nil value before Org's initialization ? IIRC Magit did that at some point and I found it annoying. Note that we cannot turn it off automatically due to false positives (i.e., we cannot decide if there are obsolete footnotes in the document or something looking like them and yet perfectly valid in the document). Regards, -- Nicolas Goaziou
Re: [O] footnote fontify causing massive slowdown
Hello, Derek Feichtingerwrites: > While diagnosing a server condition, I was listing parts of a system log > via a babel expression. The 130 lines in the babel output are wrapped in > an example block. This block caused massive slowdown of scrolling and > other operations. > > Using the emacs profiler I see: > > - redisplay_internal (C function) 8232 88% > - jit-lock-function 8226 88% > - jit-lock-fontify-now 8226 88% >- funcall 8226 88% > - # 8226 88% > - run-hook-with-args 8226 88% > - font-lock-fontify-region 8226 88% >- font-lock-default-fontify-region 8226 88% > - font-lock-fontify-keywords-region 8226 88% > - org-activate-footnote-links 8158 87% > - org-footnote-next-reference-or-definition 8158 87% >- byte-code 8158 87% > - org-footnote-at-reference-p 4114 44% > - org-footnote-in-valid-context-p 4106 44% > + org-inside-LaTeX-fragment-p 2380 25% > + org-in-block-p 1563 16% > + org-in-verbatim-emphasis 159 1% > org-at-comment-p 4 0% > > Checking for footnote pattern matches (org-footnote-re) in the wrapped > block, I see that > every line matches based on the very simple and trivial pattern of > number enclosed in angular brackets, so all the process numbers > following the "sshd" in these lines like "sshd[1234]" do match and cause load. > > # > /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts > authentication refused for userXYZ: bad ownership or modes for home directory. > /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts > authentication refused for userXYZ: bad ownership or modes for home directory. > /var/log/secure-20151129:Nov 23 02:25:41 some-host sshd[20089]: > pam_ldap: error trying to bind as user "x" (Invalid credentials) > # > > Since this kind of pattern is so common in logs and 130 lines are really not > a large number, it makes it hard to use > org for this purpose. Can this be turned off selectively, or can it be > prevented in example blocks? This is a limitation of our current way to fontify a buffer. Changing it implies some serious work, which I'd rather spend on switching to syntax-based (instead of regexp-based) fontification. However, this report raises an interesting question about footnotes: should we still support plain (e.g., "[1]") footnotes in Org documents? The pattern is very common an regularly introduces false positives. Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode buffers), to provide some common features with "footnote.el" library. I think we could remove this kind of footnotes, and yet preserve `org-footnote-normalize' to change Org footnotes into these ones, for foreign documents. WDYT? Regards, -- Nicolas Goaziou
Re: [O] footnote fontify causing massive slowdown
Hi Nicolas, 2015ko abenudak 5an, Nicolas Goaziou-ek idatzi zuen: > > This is a limitation of our current way to fontify a buffer. Changing it > implies some serious work, which I'd rather spend on switching to > syntax-based (instead of regexp-based) fontification. Indeed. However, this code was needlessly slow because it failed to take advantage of short-circuit evaluation. I pushed a fix in 046310d. > > However, this report raises an interesting question about footnotes: > should we still support plain (e.g., "[1]") footnotes in Org documents? > > The pattern is very common an regularly introduces false positives. > Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode > buffers), to provide some common features with "footnote.el" library. > > I think we could remove this kind of footnotes, and yet preserve > `org-footnote-normalize' to change Org footnotes into these ones, for > foreign documents. > > WDYT? Do [1]-type footnotes present other performance problems today? I’d rather see if simple solutions to those can be effective before going for a breaking change to syntax. Then there’s the fact that syntax fontification (incl. org-elements cache) is going to have such different performance characteristics I’m not sure we can predict where the bottlenecks will be. -- Aaron Ecay
Re: [O] footnote fontify causing massive slowdown
Hello, Aaron Ecaywrites: > Indeed. However, this code was needlessly slow because it failed to > take advantage of short-circuit evaluation. According to the profile report, I don't understand the logic of your patch. >>> - org-footnote-in-valid-context-p 4106 44% >>> + org-inside-LaTeX-fragment-p 2380 25% >>> + org-in-block-p 1563 16% >>> + org-in-verbatim-emphasis 159 1% ISTM that `org-in-block-p' is marginally slower (15%) than `org-inside-LaTeX-fragment-p' (9%). Of course, in OP's report, `org-in-block-p' is the test returning early, so pushing it earlier is faster since you don't call `org-inside-LaTeX-fragment-p', but this is only a very specific optimization made at the expense of other cases (and contradicts your commit message). Am I missing something? I don't understand either the benefit of adding `not' calls all over the place. I generally use de Morgan's law the other way and save a few primcalls. > Do [1]-type footnotes present other performance problems today? The main problem of plain footnotes isn't really their performance, but false positives'. Anytime something looks like a footnote in a buffer, you get a performance hit. This happens much more often with plain footnotes than with other footnote types. Moreover, false positives can introduce not-so-subtle problems during export (see for example 2c66e40c). Note that suggesting to not use them (the default value, actually, per `org-footnote-auto-label') doesn't help, since false positives are the problem, not real footnotes. Eventually, removing them doesn't remove any feature to Org. Of course, this is an incompatible change, and some users will need to update documents using plain footnotes. We can provide a function for that, and use `org-lint' to check for obsolete footnotes. The benefit is to avoid a whole class of problems. Regards, -- Nicolas Goaziou
Re: [O] footnote fontify causing massive slowdown
Hi Nicolas, 2015ko abenudak 5an, Nicolas Goaziou-ek idatzi zuen: > > Hello, > > Aaron Ecaywrites: > >> Indeed. However, this code was needlessly slow because it failed to >> take advantage of short-circuit evaluation. > > According to the profile report, I don't understand the logic of your > patch. > - org-footnote-in-valid-context-p 4106 44% + org-inside-LaTeX-fragment-p 2380 25% + org-in-block-p 1563 16% + org-in-verbatim-emphasis 159 1% > > ISTM that `org-in-block-p' is marginally slower (15%) than > `org-inside-LaTeX-fragment-p' (9%). I’m not sure where 15 and 9 come from. The way I read the report, org-footnote-in-valid-context-p takes 44% of the cumulative time, which is composed of org-inside-LaTeX-fragment-p (25%) + org-in-block-p (16%) + other stuff (3%). So org-inside-LaTeX-fragment-p accounts for >50% of the time spent in org-footnote-in-valid-context-p. > > Of course, in OP's report, `org-in-block-p' is the test returning early, > so pushing it earlier is faster since you don't call > `org-inside-LaTeX-fragment-p', but this is only a very specific > optimization made at the expense of other cases (and contradicts your > commit message). Am I missing something? ...no, you’re not missing anything. I looked at my patch again, and it seems completely dumb. I should not write code before finishing my morning cup of tea. I reverted in a198d81. > > I don't understand either the benefit of adding `not' calls all over the > place. I generally use de Morgan's law the other way and save a few > primcalls. > >> Do [1]-type footnotes present other performance problems today? > > The main problem of plain footnotes isn't really their performance, but > false positives'. Anytime something looks like a footnote in a buffer, > you get a performance hit. This happens much more often with plain > footnotes than with other footnote types. > > Moreover, false positives can introduce not-so-subtle problems during > export (see for example 2c66e40c). > > Note that suggesting to not use them (the default value, actually, per > `org-footnote-auto-label') doesn't help, since false positives are the > problem, not real footnotes. > > Eventually, removing them doesn't remove any feature to Org. Of course, > this is an incompatible change, and some users will need to update > documents using plain footnotes. We can provide a function for that, and > use `org-lint' to check for obsolete footnotes. The benefit is to avoid > a whole class of problems. I see. Eventually it sounds like a good idea, indeed. Maybe we should use something a bit stronger than org-lint to warn of the deprecation. What if opening a document with plain footnotes generated a warning? Thanks, -- Aaron Ecay
Re: [O] footnote fontify causing massive slowdown
On 05/12/15 23:58, Nicolas Goaziou wrote: Hello, Derek Feichtingerwrites: While diagnosing a server condition, I was listing parts of a system log via a babel expression. The 130 lines in the babel output are wrapped in an example block. This block caused massive slowdown of scrolling and other operations. Using the emacs profiler I see: - redisplay_internal (C function) 8232 88% - jit-lock-function 8226 88% - jit-lock-fontify-now 8226 88% - funcall 8226 88% - # 8226 88% - run-hook-with-args 8226 88% - font-lock-fontify-region 8226 88% - font-lock-default-fontify-region 8226 88% - font-lock-fontify-keywords-region 8226 88% - org-activate-footnote-links 8158 87% - org-footnote-next-reference-or-definition 8158 87% - byte-code 8158 87% - org-footnote-at-reference-p 4114 44% - org-footnote-in-valid-context-p 4106 44% + org-inside-LaTeX-fragment-p 2380 25% + org-in-block-p 1563 16% + org-in-verbatim-emphasis 159 1% org-at-comment-p 4 0% Checking for footnote pattern matches (org-footnote-re) in the wrapped block, I see that every line matches based on the very simple and trivial pattern of number enclosed in angular brackets, so all the process numbers following the "sshd" in these lines like "sshd[1234]" do match and cause load. # /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:41 some-host sshd[20089]: pam_ldap: error trying to bind as user "x" (Invalid credentials) # Since this kind of pattern is so common in logs and 130 lines are really not a large number, it makes it hard to use org for this purpose. Can this be turned off selectively, or can it be prevented in example blocks? This is a limitation of our current way to fontify a buffer. Changing it implies some serious work, which I'd rather spend on switching to syntax-based (instead of regexp-based) fontification. However, this report raises an interesting question about footnotes: should we still support plain (e.g., "[1]") footnotes in Org documents? The pattern is very common an regularly introduces false positives. Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode buffers), to provide some common features with "footnote.el" library. I think we could remove this kind of footnotes, and yet preserve `org-footnote-normalize' to change Org footnotes into these ones, for foreign documents. WDYT? Regards, I would be delighted to see the 'plain' footnote format abolished. I use org for writing legal text which often has things like Bank of New South Wales v Laing [1954] AC 135. Rasmus helped me with a patch to ignore these kinds of references, but they remain a nuisance. Special case, I know, but +1 for getting rid of the things. Cheers, Alan -- Alan L Tyreehttp://www2.austlii.edu.au/~alan Tel: 04 2748 6206 sip:typh...@iptel.org
Re: [O] footnote fontify causing massive slowdown
as with others, i am ok with nixing those [1] footnotes if nobody objects. on the other hand, i strongly want inline footnotes to work again with multiple paragraphs, including in fontifying. i know export is incompatible with post-8.0 paragraphs and will accept a filter or something if needed as long as it isn't too kludgy.
Re: [O] footnote fontify causing massive slowdown
On 5 December 2015, Nicolas Goaziou wrote: However, this report raises an interesting question about footnotes: should we still support plain (e.g., "[1]") footnotes in Org documents? The pattern is very common an regularly introduces false positives. Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode buffers), to provide some common features with "footnote.el" library. I think we could remove this kind of footnotes, and yet preserve `org-footnote-normalize' to change Org footnotes into these ones, for foreign documents. +1. The false positives are a common problem for me, and [fn:1] works cleanly and clearly. Bill -- William Denton ↔ Toronto, Canada ↔ https://www.miskatonic.org/
Re: [O] footnote fontify causing massive slowdown
William Dentonwrites: > On 5 December 2015, Nicolas Goaziou wrote: > >> However, this report raises an interesting question about footnotes: >> should we still support plain (e.g., "[1]") footnotes in Org documents? >> >> The pattern is very common an regularly introduces false positives. >> Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode >> buffers), to provide some common features with "footnote.el" library. >> >> I think we could remove this kind of footnotes, and yet preserve >> `org-footnote-normalize' to change Org footnotes into these ones, for >> foreign documents. > > +1. The false positives are a common problem for me, and [fn:1] works > cleanly > and clearly. Same here. +1. All the best, Tom -- Thomas S. Dye http://www.tsdye.com
Re: [O] footnote fontify causing massive slowdown
Nicolas Goaziouwrites: > However, this report raises an interesting question about footnotes: > should we still support plain (e.g., "[1]") footnotes in Org documents? > > The pattern is very common an regularly introduces false positives. > Also, IIRC, it was introduced for non-Org buffers (e.g., in Message mode > buffers), to provide some common features with "footnote.el" library. > > I think we could remove this kind of footnotes, and yet preserve > `org-footnote-normalize' to change Org footnotes into these ones, for > foreign documents. > > WDYT? I'd be happy to kill [n] style footnotes. I've had issues with them in the past. Rasmus -- Not everything that goes around comes back around, you know
Re: [O] footnote fontify causing massive slowdown
Alan L Tyreewrites: > I would be delighted to see the 'plain' footnote format abolished. I > use org for writing legal text which often has things like Bank of New > South Wales v Laing [1954] AC 135. Rasmus helped me with a patch to > ignore these kinds of references, but they remain a nuisance. > > Special case, I know, but +1 for getting rid of the things. +1 here as well. A lot of text I clip from the web has numbers in brackets. Matt
[O] footnote fontify causing massive slowdown
While diagnosing a server condition, I was listing parts of a system log via a babel expression. The 130 lines in the babel output are wrapped in an example block. This block caused massive slowdown of scrolling and other operations. Using the emacs profiler I see: - redisplay_internal (C function) 8232 88% - jit-lock-function 8226 88% - jit-lock-fontify-now 8226 88% - funcall 8226 88% - # 8226 88% - run-hook-with-args 8226 88% - font-lock-fontify-region 8226 88% - font-lock-default-fontify-region 8226 88% - font-lock-fontify-keywords-region 8226 88% - org-activate-footnote-links 8158 87% - org-footnote-next-reference-or-definition 8158 87% - byte-code 8158 87% - org-footnote-at-reference-p 4114 44% - org-footnote-in-valid-context-p 4106 44% + org-inside-LaTeX-fragment-p 2380 25% + org-in-block-p 1563 16% + org-in-verbatim-emphasis 159 1% org-at-comment-p 4 0% Checking for footnote pattern matches (org-footnote-re) in the wrapped block, I see that every line matches based on the very simple and trivial pattern of number enclosed in angular brackets, so all the process numbers following the "sshd" in these lines like "sshd[1234]" do match and cause load. # /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:36 some-host sshd[20089]: Rhosts authentication refused for userXYZ: bad ownership or modes for home directory. /var/log/secure-20151129:Nov 23 02:25:41 some-host sshd[20089]: pam_ldap: error trying to bind as user "x" (Invalid credentials) # Since this kind of pattern is so common in logs and 130 lines are really not a large number, it makes it hard to use org for this purpose. Can this be turned off selectively, or can it be prevented in example blocks? Using [8.3.2 (8.3.2-37-gd45217-elpaplus @ /home/dfeich/.emacs.d/elpa/org-plus-contrib-20151116/)] with GNU Emacs 24.5.1. Best regards, Derek