Re: Table alignment problem

[Ihor Radchenko]

Fr Ml  writes:

> Hello,
> there is an old problem with table alignment. It's mentioned here:
> https://emacs.stackexchange.com/q/30495/11498
>
> It occurs as far as I know only in 4 cases (last 4 rows):
>
> | 2 latin letters | ab | (2 glyphs)   
>  |
> | 2 arabic letters    | من | ok (2 glyphs)
>  |
> | same but with 2 diacritics  | مِنْ | also ok  (2 
> glyphs)   |
> | the arabich letter ا and then ل isn't a problem | ال | also ok (2 glyphs)   
>  |
> | but ل and then ا is a problem (case1)   | لا | not ok (it's 1 
> glyph) |
> | also ل and then أ (case2)   | لأ | " (it's 1 glyph) 
>  |
> | also ل and then إ (case3)   | لإ | " (it's 1 glyph) 
>  |
> | also ل and then آ (case4)   | لآ | " (it's 1 glyph) 
>  |

Can you try with org-fold branch [1]? It implements a new way to
calculate string width.

[1] https://github.com/yantar92/org

Best,
Ihor

Re: Need help using the dev version on windows

[Ihor Radchenko]

Denis Maier  writes:
> Then, I get this message ...
> ==
> = Invoke "make help" for a synopsis of make targets. =
> = Created a default local.mk template.   =
> = Setting "oldorg" as the default target.=
> = Please adapt local.mk to your local setup! =
> ==
> ... and that's it ...
> Nothing happens.
>
> What am I missing?

Most likely you need to add emacs to your %PATH% Windows system
variable. Essentially, you should be able to run "emacs" command from
the terminal.

Best,
Ihor

Re: displaying equations with ob-latex

[Ihor Radchenko]

michael-franz...@gmx.com writes:

> Ok, I got some progress, I did "C-C C-x C-l" and got the equation.
>
> The equation in extremely small though.

By default, it should have the same height with you text line.
You can change it though. I have the following snippet in my config:

(setq org-format-latex-options
  (quote
   (:foreground default :background default :scale 2.0 :justify center 
:html-foreground "Black" :html-background "Transparent" :html-scale 1.0 
:matchers
("begin" "$1" "$" "$$" "\\(" "\\[";; 2x height of 
formulas

Re: [PATCH] Use cache in org-up-heading-safe

[Ihor Radchenko]

Maxim Nikulin  writes:

> Though I still have not tested the patch, I think, it is an improvement 
> and it is helpful in its current form. I am unable to tell if it follows 
> code style.

FYI, this patch may also slightly improve the performance of
org-get-outline-path.

> My bad, you mentioned tags earlier, but I grepped org-agenda.el only.
> My new idea is that org-get-tags may have its own cache as well. Unsure 
> if it should be tried.

I am trying it now ;) No benchmarks yet, but it also provides a
subjective improvement. However, I am trying to reuse org-element-cache
for tags. org-element-cache is smart enough to update itself partially
on buffer changes. However, there are (known) bugs in org-element-cache.
I managed to track down some, but still need to test thoughtfully before
submitting upstream.

> Did you just replace gethash by avl-tree?

Yes

> Likely my idea is based on a 
> wrong assumption. I hoped that having positions of headers it is 
> possible to avoid jumps (goto-char ...) preceded or followed by regexp 
> matching almost completely. Previous header for arbitrary initial 
> position can be found using binary search through structure obtained 
> during scan.

Sorry, I cannot follow what you mean. The call to goto-char in
org-up-heading-safe is required by function docstring - we need to move
point to the parent heading.

>> +(re-search-backward
>> + (format "^\\*\\{1,%d\\} " level-up) nil t)
>> +(funcall outline-level
>
> Unsure concerning the following optimization from the point of 
> readability and reliability in respect to future modifications. Outline 
> level can be derived from the length of matched string without the 
> funcall requiring extra regexp.

I am not sure here. outline-level also consults outline-heading-alist,
though I found no references to that variable in Org mode code.
Otherwise, outline-level already reuses match-data. There is no regexp
matching there. 

P.S. I just found that hash-tables are created by reference, so we need
to call make-hash-table separately in each buffer with cache. Updated
patch attached.

>From 08a175752b14f84767a71665773aa64807606af4 Mon Sep 17 00:00:00 2001
Message-Id: <08a175752b14f84767a71665773aa64807606af4.1620316036.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Thu, 6 May 2021 14:13:20 +0800
Subject: [PATCH] Use cache in org-up-heading-safe

* lisp/org.el (org-up-heading-safe): Use buffer-local cache to store
positions of parent headings.  The cache is invalidated when buffer
text is changed, according to `buffer-chars-modified-tick'.
(org--up-heading-cache):  Buffer-local hash-table storing the cache.
(org--up-heading-cache-tick):  The buffer modification state for
currently active `org--up-heading-cache'.

The optimisation is critical when running agenda or org-entry-get
queries using property/tag inheritance.  In such scenarios
`org-up-heading-safe' can be called thousands of times.  For example,
building todo agenda on large number of headings lead to the following
benchmark results:

Benchmark:

1. (elp-instrument-function #'org-up-heading-safe)
2. Run agenda
3. (elp-results) ;; function, # calls, total time, avg time

Without cache:
org-up-heading-safe  27555   8.4234025759  0.0003056941

With cache, first run:
org-up-heading-safe  23227   0.5300747539  2.282...e-05

With cache, second run on unchanged buffer:
org-up-heading-safe  23227   0.1447754880  6.233...e-06

The final speedup is 1-2 orders of magnitude (~15-56 times).
---
 lisp/org.el | 30 ++
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index 0ff13c79c..7c58f0e2e 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -20440,6 +20440,10 @@ (defun org-up-heading-all (arg)
 With argument, move up ARG levels."
   (outline-up-heading arg t))
 
+(defvar-local org--up-heading-cache nil
+  "Buffer-local `org-up-heading-safe' cache.")
+(defvar-local org--up-heading-cache-tick nil
+  "Buffer `buffer-chars-modified-tick' in `org--up-heading-cache'.")
 (defun org-up-heading-safe ()
   "Move to the heading line of which the present line is a subheading.
 This version will not throw an error.  It will return the level of the
@@ -20449,10 +20453,28 @@ (defun org-up-heading-safe ()
 because it relies on stars being the outline starters.  This can really
 make a significant difference in outlines with very many siblings."
   (when (ignore-errors (org-back-to-heading t))
-(let ((level-up (1- (funcall outline-level
-  (and (> level-up 0)
-	   (re-search-backward (format "^\\*\\{1,%d\\} " level-up) nil t)
-	   (funcall outline-level)
+(let (level-cache)
+  (unless org--up-heading-cache
+(setq org--up-heading-cache (make-hash-table)))
+  (if

Re: [PATCH] Use cache in org-up-heading-safe

[Ihor Radchenko]

Maxim Nikulin  writes:
> I have not tested the patch due to I do not use agenda. My interest was 
> stimulated solely by my attempts to make org-refile-get-targets faster.

Thanks for the feedback!

> I consider it as an improvement. I have noticed the only thing that must 
> be fixed: comments with old variant of the function. Reaction to my 
> other comments is optional.

Oops. Fixed.

> In org-agenda.el org-up-heading-safe function is called only from 
> org-find-top-headline.

That's probably not the slowest part. org-agenda also calls
org-up-heading-safe indirectly. In particular, org-get-tags is calling
org-up-heading-safe to get inherited tags. It is org-get-tags that is
taking >50% time of agenda generation in my agendas. And
org-up-heading-safe was the largest contributor to that >50% time.

> ...  So the purpose of the dance with backward 
> searches is to get top level headings. My expectation is that scan 
> through the whole buffer and storing result in a structure that allows 
> binary search of position (array, red-black tree, etc) may be even
> faster.

Scan through the whole buffer could be faster, but it is not always
desired. Most of Org code only need information for current headline.
Re-scanning the whole buffer would be costly.

Also, I tried to compare avl-tree with hash table. However, avl-tree
does not give much benefit for my test case of an agenda with ~12000
todo items. Since avl-tree requires much more custom code, hash table
should be better.

;; No cache
;; org-up-heading-safe  180493  285.37713697  0.0015810980
;; org-up-heading-safe  134876  199.44584854  0.0014787349
;; org-up-heading-safe  134872  198.23494205  0.0014698005

;; Hash cache
;; org-up-heading-safe  180493  5.0327451709  2.788...e-05
;; org-up-heading-safe  134872  1.3568663460  1.006...e-05
;; org-up-heading-safe  134872  0.7527555679  5.581...e-06

;; AVL cache
;; org-up-heading-safe  180500  5.1044455589  2.827...e-05
;; org-up-heading-safe  134872  1.2781428280  9.476...e-06
;; org-up-heading-safe  134872  1.2866258809  9.539...e-06

> Alternatively lazily obtained position of top heading could be stored in 
> cache to reduce number of iterations on org-find-top-line.

That one is used for org-agenda-filter-by-top-headline. I do not really
use it, so I have no clue if there is much need to optimise it
specifically. Yet, caching org-up-heading-safe will indeed speed it up
as well.

>> +(let ((level-cache (gethash (point) org--up-heading-cache)))
>> +  (if (and level-cache
>> +   (eq (buffer-chars-modified-tick) org--up-heading-cache-tick))
>
> If buffer-chars-modified-tick is faster than gethash then reordering 
> might result in very slight improvement of performance.

Sure. Done.

>> +;; Parent is inside accessible part of the buffer.
>> +(progn (goto-char level-cache)
>> +   (funcall outline-level)))
>
> I do not see any reason why outline level can not be cached in pair with 
> position.

Hmm. You are right. Benchmarking with and without additional caching
shows that it can give extra ~2x improvement:

;; Hash cache
;; org-up-heading-safe  180493  5.0327451709  2.788...e-05
;; org-up-heading-safe  134872  1.3568663460  1.006...e-05
;; org-up-heading-safe  134872  0.7527555679  5.581...e-06

;; Hash cache + outline-level cache
;; org-up-heading-safe  180507  4.3326449169  2.400...e-05
;; org-up-heading-safe  134872  0.4952472380  3.671...e-06
;; org-up-heading-safe  134879  0.5298789350  3.928...e-06

So, outline-level is also cached in the new version of the patch.

>> +  (let (result)
>> +(setq result
>
> I am not a lisp guru, so from my point of view this can be reduced to
> (let ((result ...

Sure.

>> +(format "^\\*\\{1,%d\\} " level-up) nil t)
>
> \t as an alternative to " " is used in org-refile.el,
> e.g. "^\\*+[ \t]+". Unsure which variant is canonical one and I see that 
> regexp is taken from original function, so no regression is expected.

I do not know either. Actually, I wish if Org mode code base used less
literal strings and more regexp variables from org-element.el and
org.el.

Best,
Ihor

>From d40b2bbb1dc5113d3085be2fae52ebe5ac1b023d Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Thu, 6 May 2021 14:13:20 +0800
Subject: [PATCH] Use cache in org-up-heading-safe

* lisp/org.el (org-up-heading-safe): Use buffer-local cache to store
positions of parent headings.  The cache is invalidated when buffer
text is changed, according to `buffer-chars-modified-tick'.
(org--up-heading-cache):  Buffer-local hash-table storing the cache.
(org--up-heading-cache-tick):  The buffer modification state for
currently active `org--up-heading-cache'.

The o

Re: babel output seems to drop anything before % (in session)

[Ihor Radchenko]

"Nicholas Savage"  writes:

> I can confirm this too on the latest master.
>
> I took a quick peek this morning, and my suspicion is that the problem is 
> somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm 
> not positive at this point.

I confirm as well. I also saw an anomaly in the comint buffer. Note that
all the output lines, except "five 0% six" are after the shell prompt.
As I remember, the code expects the result to be exactly at the prompt
line. So, for some reason the first command ("echo five 0% six") of the
second block does not get inserted at the empty line.

echo one 0% two
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% 
two
echo tree 0% four
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 
0% four
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ 
org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo 
five 0% six
five 0% six
echo seven 0% eight
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 
0% eight
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ 
org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ 

Best,
Ihor

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

William Xu  writes:

> I think I'm still seeing the issue. For example, if i change (M-x
> org-agenda-todo) a TODO item into next state ONGOING, which i have made
> prettified:
>
>   (push '("ONGOING" . "" ) prettify-symbols-alist)
>
> So far so good. But as soon as I call org-agenda-redo-all, after the
> agenda is refreshed, it changes back to text 'ONGOING'. 

I was able to reproduce using prettify-symbols-mode (though not using
pretty-symbols-mode). Should be fixed now in the attached patch.

Best,
Ihor

>From 46d6c93cd59d56959e0835c958e3c822aa5308d6 Mon Sep 17 00:00:00 2001
Message-Id: <46d6c93cd59d56959e0835c958e3c822aa5308d6.1620267219.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Tue, 4 May 2021 20:33:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text properties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org--string-from-props): Removed.  It was only used by old
`org-string-width' code.

(org-buffer-substring-fontified): New function. Like
`buffer-substring', but make sure that the substring is fontified.

(org-looking-at-fontified): New function.  Like `looking-at', but make
sure that the match is fontified.

* lisp/org.el (org-get-heading): Make sure that heading is fontified.

(org--get-local-tags, org-get-tags, org-scan-tags): Add optional
argument `fontified'.  When non-nil, the returned tags are fontified.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled,
org-agenda-fix-displayed-tags, org-search-view, org-agenda-get-todos,
org-agenda-get-timestamps, org-agenda-get-sexps,
org-agenda-get-deadlines, org-agenda-get-progress,
org-agenda-get-blocks, org-tags-view, org-agenda-list, org-todo-list,
org-agenda-highlight-todo): Make sure that fontification is the same
with original Org buffers.  All the buffer-substring and match-data
queries are changed to ensure that region of interest is fontified.
Also, preserve composition properties, used i.e. by
`prettify-symbols-mode'.  The composition is usually set to be removed
on text change, so we do the changes inside
`with-silent-modifications'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixel-wise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.
---
 lisp/org-agenda.el | 126 +-
 lisp/org-macs.el   | 134 +++--
 lisp/org.el|  36 
 3 files changed, 171 insertions(+), 125 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 4c34ca5fe..546a03bc3 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -3984,7 +3984,7 @@ (defun org-agenda-finalize ()
 		  (put-text-property (point-at-bol) (point-at-eol)
  'tags
  (org-with-point-at mrk
-   (org-get-tags
+   (org-get-tags nil nil t
 	(setq org-agenda-represented-tags nil
 	  org-agenda-represented-categories nil)
 	(when org-agenda-top-headline-filter
@@ -,9 +,12 @@ (defun org-agenda-list ( arg start-day span with-hour)
 	(put-text-property s (1- (point)) 'org-today t))
 	  (setq rtnall
 		(org-agenda-add-time-grid-maybe rtnall ndays todayp))
-	  (when rtnall (insert ;; all entries
-			(org-agenda-finalize-entries rtnall 'agenda)
-			"\n"))
+  (with-silent-modifications
+;; Composition property in entries may be self-destructed
+;; on change.  Suppress the self-destruction.
+	(when rtnall (insert ;; all entries
+			  (org-agenda-finalize-entries rtnall 'agenda)
+			  "\n")))
 	  (put-text-property s (1- (point)) 'day d)
 	  (put-text-property s (1- (point)) 'org-day-cnt day-cnt)))
   (when (and org-agenda-clockreport-mode clocktable-start)
@@ -4778,10 +4781,11 @@ (defun org-search-view ( todo-only string edit-at)
   (and (eq org-agenda-show-inherited-tags t)
    (or (eq org-agenda-use-tag-inheritance t)
 	   (memq 'todo org-agenda-use-tag-inheritance
-			  tags (org-get-tags nil (not inherited-tags))
+			  tags (org-get-tags
+

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

> William Xu  writes:
>> The only issue I still see, is that when you org-agenda-redo-all, or
>> org-agenda-log-mode (which triggers org-agenda-redo-all), the
>> prettify gets lost again. Maybe org-buffer-substring-fontified call is
>> also required somewhere during org-agenda-redo-all? 
>
> I managed to reproduce it. This time, I went through all the agenda.el
> and updated places where the strings are fetched from Org buffers into
> agenda. The updated patch is attached.

Still forgot to update fontification in agenda tags view. Yet another
update...

>From ffc418abbf3169424ccfda059e7456dc33613a38 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Tue, 4 May 2021 20:33:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text properties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org--string-from-props): Removed.  It was only used by old
`org-string-width' code.

(org-buffer-substring-fontified): New function. Like
`buffer-substring', but make sure that the substring is fontified.

(org-looking-at-fontified): New function.  Like `looking-at', but make
sure that the match is fontified.

* lisp/org.el (org-get-heading): Make sure that heading is fontified.

(org--get-local-tags, org-get-tags, org-scan-tags): Add optional
argument `fontified'.  When non-nil, the returned tags are fontified.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled,
org-agenda-fix-displayed-tags, org-search-view, org-agenda-get-todos,
org-agenda-get-timestamps, org-agenda-get-sexps,
org-agenda-get-deadlines, org-agenda-get-progress,
org-agenda-get-blocks, org-tags-view): Make sure that fontification is
the same with original Org buffers.

(org-agenda-highlight-todo): Preserve composition property used,
i.e. by `prettify-symbols-mode'.  The composition is usually set to be
removed on text change, so we do the changes inside
`with-silent-modifications'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixel-wise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.
---
 lisp/org-agenda.el |  96 ++--
 lisp/org-macs.el   | 134 +++--
 lisp/org.el|  36 
 3 files changed, 150 insertions(+), 116 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 4c34ca5fe..47cb7a936 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -3984,7 +3984,7 @@ (defun org-agenda-finalize ()
 		  (put-text-property (point-at-bol) (point-at-eol)
  'tags
  (org-with-point-at mrk
-   (org-get-tags
+   (org-get-tags nil nil t
 	(setq org-agenda-represented-tags nil
 	  org-agenda-represented-categories nil)
 	(when org-agenda-top-headline-filter
@@ -4778,10 +4778,11 @@ (defun org-search-view ( todo-only string edit-at)
   (and (eq org-agenda-show-inherited-tags t)
    (or (eq org-agenda-use-tag-inheritance t)
 	   (memq 'todo org-agenda-use-tag-inheritance
-			  tags (org-get-tags nil (not inherited-tags))
+			  tags (org-get-tags
+nil (not inherited-tags) t)
 			  txt (org-agenda-format-item
    ""
-   (buffer-substring-no-properties
+   (org-buffer-substring-fontified
 beg1 (point-at-eol))
    level category tags t))
 			(org-add-props txt props
@@ -5001,7 +5002,9 @@ (defun org-tags-view ( todo-only match)
 		(widen))
 		  (setq rtn (org-scan-tags 'agenda
 	   matcher
-	   org--matcher-tags-todo-only))
+	   org--matcher-tags-todo-only
+   nil
+   'fontify))
 		  (setq rtnall (append rtnall rtn
   (org-agenda--insert-overriding-header
 (with-temp-buffer
@@ -5562,7 +5565,8 @@ (defun org-agenda-get-todos ()
 	  ts-date-pair (org-agenda-entry-get-agenda-timestamp (point))
 	  ts-date (car ts-date-pair)
 	  ts-date-type (cdr ts-date-pair)
-	  txt (org-trim (buffer-substring (match-beginning 2) (match-end 0)))
+

Re: Bug: org-store-link uses CUSTOM_ID instead of target point [9.4.4 (release_9.4.4 @ /usr/share/emacs/27.2/lisp/org/)]

[Ihor Radchenko]

Fr Ml  writes:
> If the cursor is on a <> and the item has a CUSTOM_ID then a
> link to the headline (with the CUSTOM_ID) is stored - and not the link to the 
> target.
> This behavior is not always the same.

Confirmed on master.

Recipe:

1. Create the following org file:

* test
:PROPERTIES:
:CUSTOM_ID: testasd
:END:

<>

2. Put point at the target
3. org-store-link
4. org-insert-link
5. The inserted link is pointing to the custom ID
[[#testasd][file:/tmp/4.org::targetasd]]

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Maxim Nikulin  writes:

> helm-org-ql.el that is a part of org-ql does not use cache when calling 
> org-get-outline-path. helm-org performs sequential scan similar to 
> org-refile-get-targets.

Hmm. You are right. But they could. I myself ran into an issue with
helm-org-ql exactly because formatting displayed headings was slow. If
org-refile-use-cache is deprecated, there would be no option to use
cache at all.

Best,
Ihor

[PATCH] Use cache in org-up-heading-safe

[Ihor Radchenko]

Hello,

While testing another patch for agenda fontification, I noticed that
agenda can spend up to half!! time doing org-up-heading-safe. Mostly
inside queries for inherited tags and properties.

I managed to make org-up-heading-safe up to 50x faster using position
cache.

If this patch also works for others, org-agenda-use-tag-inheritance
might become much less of an issue.

Benchmark:

1. (elp-instrument-function #'org-up-heading-safe)
2. Run agenda
3. (elp-results) ;; function, # calls, total time, avg time

Without cache:
org-up-heading-safe  27555   8.4234025759  0.0003056941

With cache, first run:
org-up-heading-safe  23227   0.5300747539  2.282...e-05

With cache, second run on unchanged buffer:
org-up-heading-safe  23227   0.1447754880  6.233...e-06

Best,
Ihor

>From fe1e7094a7b76ba6bf282c5c4409a32b36827d56 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Tue, 4 May 2021 22:55:14 +0800
Subject: [PATCH] Use cache in org-up-heading-safe

* lisp/org.el (org-up-heading-safe): Use buffer-local cache to store
positions of parent headings.  The cache is invalidated when buffer
text is changed, according to `buffer-chars-modified-tick'.
(org--up-heading-cache):  Buffer-local hash-table storing the cache.
(org--up-heading-cache-tick):  The buffer modification state for
currently active `org--up-heading-cache'.

The optimisation is critical when running agenda or org-entry-get
queries using property/tag inheritance.  In such scenarios
`org-up-heading-safe' can be called thousands of times.  For example,
building todo agenda on large number of headings lead to the following
benchmark results:

Benchmark:

1. (elp-instrument-function #'org-up-heading-safe)
2. Run agenda
3. (elp-results) ;; function, # calls, total time, avg time

Without cache:
org-up-heading-safe  27555   8.4234025759  0.0003056941

With cache, first run:
org-up-heading-safe  23227   0.5300747539  2.282...e-05

With cache, second run on unchanged buffer:
org-up-heading-safe  23227   0.1447754880  6.233...e-06

The final speedup is 1-2 orders of magnitude (~15-56 times).
---
 lisp/org.el | 43 +++
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index b8678359f..4385903ee 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -19434,6 +19434,10 @@ (defun org-up-heading-all (arg)
 With argument, move up ARG levels."
   (outline-up-heading arg t))
 
+(defvar-local org--up-heading-cache (make-hash-table)
+  "Buffer-local `org-up-heading-safe' cache.")
+(defvar-local org--up-heading-cache-tick nil
+  "Buffer `buffer-chars-modified-tick' in `org--up-heading-cache'.")
 (defun org-up-heading-safe ()
   "Move to the heading line of which the present line is a subheading.
 This version will not throw an error.  It will return the level of the
@@ -19443,10 +19447,41 @@ (defun org-up-heading-safe ()
 because it relies on stars being the outline starters.  This can really
 make a significant difference in outlines with very many siblings."
   (when (ignore-errors (org-back-to-heading t))
-(let ((level-up (1- (funcall outline-level
-  (and (> level-up 0)
-	   (re-search-backward (format "^\\*\\{1,%d\\} " level-up) nil t)
-	   (funcall outline-level)
+(let ((level-cache (gethash (point) org--up-heading-cache)))
+  (if (and level-cache
+   (eq (buffer-chars-modified-tick) org--up-heading-cache-tick))
+  (when (<= (point-min) level-cache (point-max))
+;; Parent is inside accessible part of the buffer.
+(progn (goto-char level-cache)
+   (funcall outline-level)))
+;; Buffer modified.  Invalidate cache.
+(when level-cache
+  (clrhash org--up-heading-cache)
+  (setq-local org--up-heading-cache-tick
+  (buffer-chars-modified-tick)))
+(let ((level-up (1- (funcall outline-level)))
+  (pos (point)))
+  (let (result)
+(setq result
+  (and (> level-up 0)
+	   (re-search-backward
+(format "^\\*\\{1,%d\\} " level-up) nil t)
+	   (funcall outline-level)))
+(when result
+  (puthash pos (point) org--up-heading-cache
+;; (defun org-up-heading-safe ()
+;;   "Move to the heading line of which the present line is a subheading.
+;; This version will not throw an error.  It will return the level of the
+;; headline found, or nil if no higher level is found.
+
+;; Also, this function will be a lot faster than `outline-up-heading',
+;; because it relies on stars being the outline starters.  This can really
+;; make a significant difference in outlines with very many siblings."
+;;   (when (ignore-errors (org-back-to-heading t))
+;; (let ((level-up (1- (funcall outline-level
+;;   (and (> level-up 0)
+;; 	  

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

Ihor Radchenko  writes:

> Bastien  writes:
>> Could it slow down agenda generation for some configurations?

> The total slowdown is ~30%, though the second part will only be slow
> before the headings are fontified first time by
> org-buffer-substring-fontified. Subsequent agenda rebuilds will be
> faster.

I have updated the code to avoid creating temporary org buffers. Now, I
got 8% slowdown during first agenda run. The slowdown diminishes as the
headlines contributing to agenda get fontified (i.e. for all next
org-agenda-redo).

> Please move the comments after the change log themselves.

Done.

> Here and for the rest of the patch: please try to keep lines below 80
> characters.  I'm aware this is not always feasible, especially given
> long functions with many nested s-exps, but let's try to come as close
> as possible to 80.

Done.

William Xu  writes:
> The only issue I still see, is that when you org-agenda-redo-all, or
> org-agenda-log-mode (which triggers org-agenda-redo-all), the
> prettify gets lost again. Maybe org-buffer-substring-fontified call is
> also required somewhere during org-agenda-redo-all? 

I managed to reproduce it. This time, I went through all the agenda.el
and updated places where the strings are fetched from Org buffers into
agenda. The updated patch is attached.

Best,
Ihor

>From 8a6f629669772ff4561180ace320eb0a6365969f Mon Sep 17 00:00:00 2001
Message-Id: <8a6f629669772ff4561180ace320eb0a6365969f.1620134057.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Tue, 4 May 2021 20:33:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text properties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org--string-from-props): Removed.  It was only used by old
`org-string-width' code.

(org-buffer-substring-fontified): New function. Like
`buffer-substring', but make sure that the substring is fontified.

(org-looking-at-fontified): New function.  Like `looking-at', but make
sure that the match is fontified.

* lisp/org.el (org-get-heading): Make sure that heading is fontified.

(org--get-local-tags, org-get-tags): Add optional argument
`fontified'.  When non-nil, the returned tags are fontified.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled,
org-agenda-fix-displayed-tags, org-search-view, org-agenda-get-todos,
org-agenda-get-timestamps, org-agenda-get-sexps,
org-agenda-get-deadlines, org-agenda-get-progress,
org-agenda-get-blocks): Make sure that fontification is the same with
original Org buffers.

(org-agenda-highlight-todo): Preserve composition property used,
i.e. by `prettify-symbols-mode'.  The composition is usually set to be
removed on text change, so we do the changes inside
`with-silent-modifications'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixel-wise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.
---
 lisp/org-agenda.el |  92 ++-
 lisp/org-macs.el   | 134 +++--
 lisp/org.el|  24 +---
 3 files changed, 138 insertions(+), 112 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 4c34ca5fe..420579ecf 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -3984,7 +3984,7 @@ (defun org-agenda-finalize ()
 		  (put-text-property (point-at-bol) (point-at-eol)
  'tags
  (org-with-point-at mrk
-   (org-get-tags
+   (org-get-tags nil nil t
 	(setq org-agenda-represented-tags nil
 	  org-agenda-represented-categories nil)
 	(when org-agenda-top-headline-filter
@@ -4778,10 +4778,11 @@ (defun org-search-view ( todo-only string edit-at)
   (and (eq org-agenda-show-inherited-tags t)
    (or (eq org-agenda-use-tag-inheritance t)
 	   (memq 'todo org-agenda-use-tag-inheritance
-			  tags (org-get-tags nil (not inherited-tags))
+			  tags (org-get-tags
+nil (not inherited-tags) t)
 			  txt (org-agenda-format-item
    ""
-   (buffer-substring-no-properties
+   (org-buffer-substring-fo

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

Bastien  writes:
> Could it slow down agenda generation for some configurations?

Yes, it can. Specifically, fontifying tags can be costly. For
illustration, below if profiler report for a very large agenda buffer
(1468 entries):

19820  95%   - org-agenda
...
4401  21%   - org-agenda-fix-displayed-tags
3711  17%+ org-mode
...
1410   6%  - org-buffer-substring-fontified
1380   6%   + jit-lock-fontify-now
(full profiler report at the end of the message)

The total slowdown is ~30%, though the second part will only be slow
before the headings are fontified first time by
org-buffer-substring-fontified. Subsequent agenda rebuilds will be
faster.

The first part is harder. It is related to tag fontification. Currently,
agenda fetches tag list as unfontified strings via org-get-tags. So, I
had to re-fontify tags manual is temporary org-mode buffer. You can see
that running org-mode in fresh buffer takes a lot of time. Alternative
approach would be modifying org-get-tags to return fontified strings,
but I am not sure if it is safe to do - unrelated parts of org
might be affected. Or I can write something like org-get-tags-fontified.
What do you think?

Best,
Ihor
20147  96% - command-execute
20147  96%  - funcall-interactively
19820  95%   - org-agenda
19820  95%- apply
19820  95% - ad-Advice-org-agenda
19820  95%  - #
19820  95%   - apply
19820  95%- #
19761  94% - call-interactively
19761  94%  - funcall-interactively
19741  94%   - org-todo-list
13331  63%- org-agenda-get-day-entries
13331  63% - apply
13331  63%  - #
13331  63%   - org-agenda-get-todos
13331  63%- apply
13331  63% - #
8115  38%  - org-agenda-format-item
4401  21%   - org-agenda-fix-displayed-tags
3711  17%+ org-mode
409   1%+ font-lock-ensure
161   0%+ org-fold-core--fix-folded-region
40   0%+ #
3554  17%   - eval
3554  17%- format
3554  17% - format
3544  17%  - org-eval
3463  16%   + yant/format-time-balance-multiplier
81   0%   + yant/format-summary-for-agenda
10   0%  + if
40   0% replace-regexp-in-string
10   0% org-get-time-of-day
3185  15%  - org-get-tags
3095  14%   + org-up-heading-safe
20   0% org--get-local-tags
20   0%   + org-before-first-heading-p
10   0%   + mapcar
10   0%   + org-remove-uninherited-tags
1410   6%  - org-buffer-substring-fontified
1380   6%   + jit-lock-fontify-now
362   1%org-get-priority
39   0%org-add-props
20   0%org-get-todo-state
10   0%org-agenda-skip
10   0%replace-regexp-in-string
9   0%org-agenda-new-marker
5400  25%- org-agenda-finalize
2920  14% + org-get-tags
1380   6% + run-hooks
610   2%   org-agenda-fontify-priorities
340   1% + org-agenda-align-tags
30   0% + org-activate-links
20   0% + org-agenda-mark-clocking-task
730   3%+ org-agenda-prepare
150   0%+ org-agenda-finalize-entries
80   0%+ org-fold-core--fix-folded-region
59   0% + org-agenda-get-restriction-and-command

Re: Checkboxes in headings - opinions wanted

[Ihor Radchenko]

Arthur Miller  writes:
> ... Example can be seen in attached screenshot from
> my init file where I use org headings to form a list of packages to
> install, and checkboxes to indicate if a package configuration is used
> or not. 

Depending on details of your workflow, you might also use
org-toggle-comment for indication of configuration usage. As a bonus, if
you use literate config, commenting the heading will also disable
tangling for that heading.

Ihor
Best,

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

William Xu  writes:

> Now I try to test it extensively. Even with all your changes, I find
> when I use org-agenda-todo to change the todo-state inside the agenda
> buffer, the new state isn't always prettified. 
>
> Do you see the same behaviour? 

Oops. Yes, I do see the same behaviour. I only did light testing on
master and I had some unrelated changes nullifying the problem you
observe on my testing branch.

See the updated patch.

>From 06a2d8ab328721835866bf97f0344cce15cd1dee Mon Sep 17 00:00:00 2001
Message-Id: <06a2d8ab328721835866bf97f0344cce15cd1dee.1619960107.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Sat, 1 May 2021 20:09:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixelwise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text proerpties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org-buffer-substring-fontified): New function getting fontified
substring from current buffer.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled): Use
org-buffer-substring-fontified to get fontified heading.

(org-agenda-fix-displayed-tags): Fontify tags.

(org-agenda-highlight-todo): Preserve composition property used,
i.e. by `prettify-symbols-mode'.  The composition is usually set to be
removed on text change, so we do the changes inside
`with-silent-modifications'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.

* lisp/org.el (org-get-heading): Make sure that heading is fontified.
---
 lisp/org-agenda.el |  63 --
 lisp/org-macs.el   | 108 ++---
 lisp/org.el|   2 +
 3 files changed, 85 insertions(+), 88 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index bd9d466a6..5add0e092 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -5562,7 +5562,7 @@ (defun org-agenda-get-todos ()
 	  ts-date-pair (org-agenda-entry-get-agenda-timestamp (point))
 	  ts-date (car ts-date-pair)
 	  ts-date-type (cdr ts-date-pair)
-	  txt (org-trim (buffer-substring (match-beginning 2) (match-end 0)))
+	  txt (org-trim (org-buffer-substring-fontified (match-beginning 2) (match-end 0)))
 	  inherited-tags
 	  (or (eq org-agenda-show-inherited-tags 'always)
 		  (and (listp org-agenda-show-inherited-tags)
@@ -5973,7 +5973,7 @@ (defun org-agenda-get-progress ()
 	  clockp (not (or closedp statep))
 	  state (and statep (match-string 2))
 	  category (org-get-category (match-beginning 0))
-	  timestr (buffer-substring (match-beginning 0) (point-at-eol)))
+	  timestr (org-buffer-substring-fontified (match-beginning 0) (point-at-eol)))
 	(when (string-match "\\]" timestr)
 	  ;; substring should only run to end of time stamp
 	  (setq rest (substring timestr (match-end 0))
@@ -6254,7 +6254,7 @@ (defun org-agenda-get-deadlines ( with-hour)
 	(let* ((category (org-get-category))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (inherited-tags
 		(or (eq org-agenda-show-inherited-tags 'always)
 			(and (listp org-agenda-show-inherited-tags)
@@ -6469,7 +6469,7 @@ (defun org-agenda-get-scheduled ( deadlines with-hour)
 		   (tags (org-get-tags nil (not inherited-tags)))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (time
 		(cond
 		 ;; No time of day designation if it is only a
@@ -6856,6 +6856,15 @@ (defun org-agenda-fix-displayed-tags (txt tags add-inherited hide-re)
 			   x))
 			   tags ":")
 			  (if have-i "::" ":"))
+  (let ((tag-string (when (string-match org-tag-group-re txt)
+  (match-string 0 txt
+(when tag-string
+  (with-temp-buffer
+(save-match-data
+  (let ((org-inhibit-startup t)) (org-mode))
+

Re: Separator "::" -> what is it for in headlines?

[Ihor Radchenko]

Ypo  writes:

> Same issue:
> * *To **the left of* :: it's bold

I can reproduce. The issue appears only in level 1 headlines because of
wrong regexp in font-lock-keywords. The attach patch should fix the
problem.

Best,
Ihor

>From 791471a5f28723f5d3b34a9345795e0f2902b9b6 Mon Sep 17 00:00:00 2001
Message-Id: <791471a5f28723f5d3b34a9345795e0f2902b9b6.1619957998.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Sun, 2 May 2021 20:16:44 +0800
Subject: [PATCH] Fix description list item regexp when fontifying "*" items

* lisp/org.el (org-set-font-lock-defaults): Avoid fontifying headlines
with "::" as description list items.  Lists can start with "*", but
"*" must not be at the beginning of line.  Old regexp did not require
whitespace before "*" in description list items.
---
 lisp/org.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org.el b/lisp/org.el
index 5cc37a57e..53ff5d463 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -5582,7 +5582,7 @@ (defun org-set-font-lock-defaults ()
 	 '("\\[\\([0-9]*%\\)\\]\\|\\[\\([0-9]*\\)/\\([0-9]*\\)\\]"
 	   (0 (org-get-checkbox-statistics-face) prepend)))
 	   ;; Description list items
-	   '("^[ \t]*[-+*][ \t]+\\(.*?[ \t]+::\\)\\([ \t]+\\|$\\)"
+   '("\\(?:^[ \t]*[-+]\\|^[ \t]+[*]\\)[ \t]+\\(.*?[ \t]+::\\)\\([ \t]+\\|$\\)"
 	 1 'org-list-dt prepend)
;; Inline export snippets
'("\\(@@\\)\\([a-z-]+:\\).*?\\(@@\\)"
-- 
2.26.3

[PATCH] org-test: Fix wrong-numbre-of-arguments error on master+native-comp

[Ihor Radchenko]

Running make test on master using Emacs master compiled with native-comp
support throws error:

decode-time((22352 22528) nil nil)
  time-to-days((22352 22528))
  org-time-stamp-to-now("2016-06-03 Fri")
  org-deadline-close-p("2016-06-03 Fri" 0)
  apply(org-deadline-close-p ("2016-06-03 Fri" 0))

As discussed in Emacs bug#48133, this is because of outdated cl-letf
statement in org-test. The attached patch is fixing the problem.

Best,
Ihor

>From f973eb6cf635ccce86b865001d0bca5fba29c898 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sun, 2 May 2021 16:29:02 +0800
Subject: [PATCH] org-test: Fix wrong-numbre-of-arguments error on
 master+native-comp

* testing/org-test.el (org-test-at-time): Use correct number of
arguments in 'decode-time cl-letf binding. `decode-time' accepts up to
3 arguments on master.

The wrong-numbre-of-arguments error is raised on Emacs master
configured with native-comp support when running make test on Org mode
master.  Native-comp modifies function calls with optional arguments
in the way that omitted arguments are still provided as `nil'.  For
example, `decode-time' called as (decode-time time) in lisp source may
be compiled into (decode-time time nil nil) call by native-compiler.
If redefined `decode-time' does not accept 3 arguments, it result in
error.

See Emacs bug#48133.
---
 testing/org-test.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testing/org-test.el b/testing/org-test.el
index ade212440..30adb977e 100644
--- a/testing/org-test.el
+++ b/testing/org-test.el
@@ -466,8 +466,8 @@ (defmacro org-test-at-time (time  body)
 	   (apply ,(symbol-function 'current-time-zone)
 		  (or time ,at) args)))
 	((symbol-function 'decode-time)
-	 (lambda ( time) (funcall ,(symbol-function 'decode-time)
-	   (or time ,at
+	 (lambda ( time zone form) (funcall ,(symbol-function 'decode-time)
+	(or time ,at) zone form)))
 	((symbol-function 'encode-time)
 	 (lambda (time  args)
 	   (apply ,(symbol-function 'encode-time) (or time ,at) args)))
-- 
2.26.3

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Tim Cross  writes:

> I suspect the reason it is not done automatically is that getting that
> right for all use cases is very hard to do without adding adverse impact
> to performance. A cache which is marked as 'dirty' too often results in
> too frequent cache refresh operations, but at the same time, determining
> what changes in an org file actually invalidate the cache can be a
> process intensive operation. Allowing the user to force cache refresh
> when needed is likely a reasonable compromise. 

Not really. `helm-org-ql' uses cache that is invalidated by _any_ change
in buffer + updating cache only when results are queried. Performance is
much better compared with built-in Org solutions.

Also, there is `org-element-use-cache' doing more granular cache
invalidation. You can enable it and test the performance. It is not
really degraded (except some hard-to-catch bugs existing in the
org-element-cache code).

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Maxim Nikulin  writes:

> Some additions. org-outline-path-cache is used solely by 
> org-refile-get-targets (maybe there are some calls in other packages) 
> but it efficiency is questionable. It was not clear for me earlier that 
> the cache is reset before each scan through a buffer. So if 
> org-refile-use is disabled, org-outline-path-cache from previous run of 
> org-refile or org-goto is not used as well. A query to 
> org-outline-path-cache requires at least one backward search and hash 
> lookup. During sequential scan in org-refile-get-targets it is enough to 
> have previous heading path to update it when new heading is found. I 
> think, org-outline-path-cache should be deprecated.

At least helm-org-ql and helm-org are using `org-get-outline-path' with
cache. I do think it is useful. Though, indeed, not as fast as
single-pass scan. Probably, we can rewrite the code of
`org-get-outline-path' instead of deprecating it?

>> Just cleanup heading text:
>
> I have realized what is wrong with this benchmark. It runs so fast 
> because it matches no headings, so it never spent time for cleaning them up.

Oh. You are right. I tested with fixed benchmark and it is indeed much slower.

> On 29/04/2021 21:12, Ihor Radchenko wrote:
>> For the cleaned heading text, I do not think that re-calculating the
>> heading text on each change is a good idea. It may degrade typing
>> latency. Yet, an acceptable approach could be simply invalidating cache
>> for the changed headings. Then, outline paths can be re-calculated on
>> changed headings when needed.
>
> I agree that it is enough to invalidate cleaned heading on edit to 
> refresh it in org-refile-get-targets. On the other hand, I still prefer 
> text properties since they could be fetched even if some lines have been 
> added or removed before. Position-based cache is useless in such cases.

Note that `org-refresh-category-properties' is using pretty much same
idea. However, there is no automatic invalidation and one needs to run
org-refresh-category-properties to ensure that cache is up to date.

Alternatively, markers can be used here. They will not be invalidated by
changes in buffer.

> Concerning typing latency, it should be postponed and resumed when no 
> new edits is performed for certain period of time (~1s). However I am 
> unsure if it is possible to accurately track all affected lines since 
> later changes can add/remove lines before the line scheduled for 
> invalidation.

As one option, we can accumulate the changed regions in some variable
and process them on timer. As I remember, org-element-cache is already
doing something similar. In fact, org-element-cache might be modified to
handle outline-path as well. The problem is that org-element-cache has
some difficult bugs. I tried using it multiple times and ran into issues
I was not able to debug. I may work on this in future.

Best,
Ihor

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

William Xu  writes:

>> (org-agenda-highlight-todo): Preserve composition property used,
>> i.e. by `prettify-symbols-mode'.
>
> It looks like this change is not really needed, my emacs is built from
> git master.  Maybe the 'composition property is now preserved
> automatically in the buffer?

This change is needed, for example, when you change todo-state using
`org-agenda-todo'. Refreshing the agenda line in
org-agenda-highlight-todo involves

(insert (format org-agenda-todo-keyword-format s))

`insert' will destroy the 'composition as 'composition is set by
pretty-symbols to be self-destructed on change.

Having said that, I can now see an easier approach to deal with the
problem: simply wrap insert into `with-silent-modifications'

Amended patch with some minor refactoring is attached.

Best,
Ihor

>From b6edd24d4477467e05b69258eccef58d48b39676 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 1 May 2021 20:09:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixelwise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text proerpties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org-buffer-substring-fontified): New function getting fontified
substring from current buffer.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled): Use
org-buffer-substring-fontified to get fontified heading.

(org-agenda-fix-displayed-tags): Fontify tags.

(org-agenda-highlight-todo): Preserve composition property used,
i.e. by `prettify-symbols-mode'.  The composition is usually set to be
removed on text change, so we do the changes inside
`with-silent-modifications'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.
---
 lisp/org-agenda.el |  63 --
 lisp/org-macs.el   | 108 ++---
 2 files changed, 83 insertions(+), 88 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index bd9d466a6..5add0e092 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -5562,7 +5562,7 @@ (defun org-agenda-get-todos ()
 	  ts-date-pair (org-agenda-entry-get-agenda-timestamp (point))
 	  ts-date (car ts-date-pair)
 	  ts-date-type (cdr ts-date-pair)
-	  txt (org-trim (buffer-substring (match-beginning 2) (match-end 0)))
+	  txt (org-trim (org-buffer-substring-fontified (match-beginning 2) (match-end 0)))
 	  inherited-tags
 	  (or (eq org-agenda-show-inherited-tags 'always)
 		  (and (listp org-agenda-show-inherited-tags)
@@ -5973,7 +5973,7 @@ (defun org-agenda-get-progress ()
 	  clockp (not (or closedp statep))
 	  state (and statep (match-string 2))
 	  category (org-get-category (match-beginning 0))
-	  timestr (buffer-substring (match-beginning 0) (point-at-eol)))
+	  timestr (org-buffer-substring-fontified (match-beginning 0) (point-at-eol)))
 	(when (string-match "\\]" timestr)
 	  ;; substring should only run to end of time stamp
 	  (setq rest (substring timestr (match-end 0))
@@ -6254,7 +6254,7 @@ (defun org-agenda-get-deadlines ( with-hour)
 	(let* ((category (org-get-category))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (inherited-tags
 		(or (eq org-agenda-show-inherited-tags 'always)
 			(and (listp org-agenda-show-inherited-tags)
@@ -6469,7 +6469,7 @@ (defun org-agenda-get-scheduled ( deadlines with-hour)
 		   (tags (org-get-tags nil (not inherited-tags)))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (time
 		(cond
 		 ;; No time of day designation if it is only a
@@ -6856,6 +6856,15 @@ (defun org-agenda-fix-displayed-tags (txt tags add-inherited hide-re)
 			   x))
 			   tags ":")
 			  (if have-i "::" ":"))
+  (let ((tag-string (when (string-mat

Re: Bug: Org mode fails to compile using Emacs 24.5-r10 [9.4.5 (9.4.5-g3ea248 @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Bastien  writes:

> could you provide a patch for these two warnings for the maint branch?

>> In end of data:
>> ob-gnuplot.el:299:1:Warning: the function ‘file-local-name’ is not known to 
>> be

This does not appear on maint. Only on master. The patch for master attached.

>> In org-display-inline-images:
>> org.el:16554:57:Warning: reference to free variable ‘image-map’

Kyle already dealt with it.

Best,
Ihor

>From d57d51007392f41645b78a28e8ef14132b42c324 Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Sat, 1 May 2021 21:54:13 +0800
Subject: [PATCH] Use org version of file-local-name for compatibility with
 Emacs 25

---
 lisp/ob-gnuplot.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index c0a9ff13a..a9b8e65e5 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -101,7 +101,7 @@ (defun org-babel-gnuplot-process-vars (params)
 org-babel-temporary-directory
 "/gnuplot/"
 (file-remote-p val 'host)
-(file-local-name val
+(org-babel-local-file-name val
 		  (if (and (file-exists-p local-name) ;; only download file if remote is newer
 			   (file-newer-than-file-p local-name val))
 		  local-name
-- 
2.26.3

Re: Bug: Org mode fails to compile using Emacs 24.5-r10 [9.4.5 (9.4.5-g3ea248 @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Kyle Meyer  writes:

> Here are a few notes on ones present in maint that I've glanced at.

I confirm that there are no warning using Emacs 25.3.1 on maint.

For Emacs 24.5.1, I see that following in addition to what you
mentioned:

Compiling /home/yantar92/Git/org-mode/lisp/ob-python.el...
Compiler-macro error for python-syntax-context: (void-function 
python-syntax--context-compiler-macro)

Compiling /home/yantar92/Git/org-mode/lisp/ob-C.el...

In toplevel form:
ob-C.el:477:1:Error: Unknown upattern `(quote integerp)'

Re: prettify-symbols-mode in org agenda?

[Ihor Radchenko]

Bastien  writes:

> Thanks for bringing this idea up.
>
> If allowing prettify-symbols-mode in Org agenda mode does not slow
> down the agenda display and does not create spacing problems, then
> yes, why not.

Here is the patch. It will be great if other people test it first, as I
rewrote it from advised functions in my personal config.

Best,
Ihor

>From 787181ac85c75b2a99e3098b066f9086536c4aa6 Mon Sep 17 00:00:00 2001
Message-Id: <787181ac85c75b2a99e3098b066f9086536c4aa6.1619872197.git.yanta...@gmail.com>
From: Ihor Radchenko 
Date: Sat, 1 May 2021 20:09:10 +0800
Subject: [PATCH] Make sure that fontification is preserved in agenda

Preserve fontification and composition of headlines and tags in
agenda.  If the headlines/tags are not yet fontified when building
agenda, make sure that they are fontified in the original Org mode
buffers first.

In addition, tags alignment is now done pixelwise to avoid alignment
issues with variable-pitch symbols that may appear in fontified Org
mode buffers.  The alignment is utilising :align-to specification,
which means that the alignment will be automatically updated as the
agenda buffer is resized.

* lisp/org-macs.el (org-string-width): Refactor old code and add
optional argument to return pixel width.  The old code used manual
parsing of text proerpties to find which parts of string are visible.
The new code defers this work to Emacs display engine via
`window-text-pixel-size'.  The visibility settings of current buffer
are taken into account.

(org-buffer-substring-fontified): New function getting fontified
substring from current buffer.

* lisp/org-agenda.el (org-agenda-get-todos, org-agenda-get-progress,
org-agenda-get-deadlines, org-agenda-get-scheduled): Use
org-buffer-substring-fontified to get fontified heading.

(org-agenda-fix-displayed-tags): Fontify tags.

(org-agenda-highlight-todo): Preserve composition property used,
i.e. by `prettify-symbols-mode'.

(org-agenda-align-tags): Use pixel width and (space . :align-to)
'display property to align tags in agenda.
---
 lisp/org-agenda.el |  65 +--
 lisp/org-macs.el   | 108 ++---
 2 files changed, 86 insertions(+), 87 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index bd9d466a6..b7699afa1 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -5562,7 +5562,7 @@ (defun org-agenda-get-todos ()
 	  ts-date-pair (org-agenda-entry-get-agenda-timestamp (point))
 	  ts-date (car ts-date-pair)
 	  ts-date-type (cdr ts-date-pair)
-	  txt (org-trim (buffer-substring (match-beginning 2) (match-end 0)))
+	  txt (org-trim (org-buffer-substring-fontified (match-beginning 2) (match-end 0)))
 	  inherited-tags
 	  (or (eq org-agenda-show-inherited-tags 'always)
 		  (and (listp org-agenda-show-inherited-tags)
@@ -5973,7 +5973,7 @@ (defun org-agenda-get-progress ()
 	  clockp (not (or closedp statep))
 	  state (and statep (match-string 2))
 	  category (org-get-category (match-beginning 0))
-	  timestr (buffer-substring (match-beginning 0) (point-at-eol)))
+	  timestr (org-buffer-substring-fontified (match-beginning 0) (point-at-eol)))
 	(when (string-match "\\]" timestr)
 	  ;; substring should only run to end of time stamp
 	  (setq rest (substring timestr (match-end 0))
@@ -6254,7 +6254,7 @@ (defun org-agenda-get-deadlines ( with-hour)
 	(let* ((category (org-get-category))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (inherited-tags
 		(or (eq org-agenda-show-inherited-tags 'always)
 			(and (listp org-agenda-show-inherited-tags)
@@ -6469,7 +6469,7 @@ (defun org-agenda-get-scheduled ( deadlines with-hour)
 		   (tags (org-get-tags nil (not inherited-tags)))
 		   (level (make-string (org-reduced-level (org-outline-level))
    ?\s))
-		   (head (buffer-substring (point) (line-end-position)))
+		   (head (org-buffer-substring-fontified (point) (line-end-position)))
 		   (time
 		(cond
 		 ;; No time of day designation if it is only a
@@ -6856,6 +6856,15 @@ (defun org-agenda-fix-displayed-tags (txt tags add-inherited hide-re)
 			   x))
 			   tags ":")
 			  (if have-i "::" ":"))
+  (let ((tag-string (when (string-match org-tag-group-re txt)
+  (match-string 0 txt
+(when tag-string
+  (with-temp-buffer
+(save-match-data
+  (let ((org-inhibit-startup t)) (org-mode))
+  (insert "* X" tag-string)
+  (font-lock-ensure))
+(setf (substring txt (match-beginning 0) (match-end 0)) (buffer-substring 4 (point-max))
   txt)
 
 (defvar org-agenda-sorting-strategy) ;; because the def is in a let form
@@ -7110,7 +7119,8 @@ (defun org-agenda-limit-

Re: Big problem: org agenda freezes my process

[Ihor Radchenko]

torys.ander...@gmail.com (Tory S. Anderson) writes:

> Lately, when I try to view an orgmode agenda it seems that two things make it 
> cause my emacs CPU threat to spin to max and freeze up, which is a huge 
> problem since I'm an exwm user and live in my agenda view. As far as I've 
> been able to tell, when an item from today is clocked into multiple times in 
> the same day, it prompts most agenda actions to result in a freeze and some 
> kind of loop that is binding up my system. It has some gaps, so if I spam C-g 
> I will eventually, usually, escape given 30 seconds to 2 minutes.

Few generic things that may produce behaviour when Emacs is not reacting
to C-g:

Do you use any kind of fontification packages/customisation dealing with
org agenda? Does disabling font-lock-mode in agenda help? Do you have
any mode line customisation that may be affected? 

Best,
Ihor

Re: org-attach-attach in an org-capture template?

[Ihor Radchenko]

Tim Visher  writes:

> I also believe I could do this with one of the org-capture hooks but
> examining them I didn't see the obvious right one to add my function to. I
> would think it would be the org-capture-prepare-finalize-hook and I may
> just give that a try.

Capture hooks should be the right place to do what your want. I
recommend org-capture-before-finalize-hook since it will not run if you
abort the capture for any reason. Otherwise, you may attach the file,
abort capture (via C-c C-k or because of some error), and your attached
file will hang somewhere unreferenced. Your first idea with
%()-expansion have the same problem.

> ... I further assume then that I'd need to to apply the
> advice here
> 
> and
> check for what template I'm in with (plist-get org-capture-plist :key) or
> similar.

I recommend using doct [1] with :before-finalize keyword. It will take
care that your function runs for the right template.

[1] https://github.com/progfolio/doct#hooks

> So my question is whether there's anything I can do to get org-attach-attach 
> to
> recognize the file it's in. I'm assuming that when it's expanding it's in a
> temporary buffer of some kind which is why it's failing.

You are right, template expansion is done in separate "*Capture*" buffer
using org-capture-fill-template. The buffer is not associated with any
file.

Best,
Ihor

Re: [WDYT, mini] key h in agenda for quick help

[Ihor Radchenko]

Samuel Wales  writes:

> links to the info manual from help pages and vice-versa would be keen.

See helpful [1] package. It provides links from help buffers to manual.
For links from manual, there is always  v/f/.

[1] https://github.com/Wilfred/helpful

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Maxim Nikulin  writes:

> Curiously my experience is that avoiding this lazy cache with 
> backtracking and maintaining custom structure during sequential scan of 
> the buffer works several times faster.

My experience is exactly opposite. Or maybe I miss something. Can you
elaborate?

> ... However it is appropriate time to 
> populate the cache you mentioned. Unfortunately it is still necessary to 
> cleanup heading text, and it consumes significant time.

Did you do any benchmarks? I just tried

Outline path without cache:

(benchmark-run 1
  (goto-char (point-min))
  (while (re-search-forward "^\\*+" nil t)
(org-get-outline-path t nil))) => (6.051079914 1 0.286472487869)

Outline path with cache:

(benchmark-run 1
  (goto-char (point-min))
  (while (re-search-forward "^\\*+" nil t)
(org-get-outline-path t nil))) => (1.658461165 0 0.0)

Just cleanup heading text:

(benchmark-run 1
  (goto-char (point-min))
  (while (re-search-forward "^\\*+" nil t)
(let ((case-fold-search nil))
  (looking-at org-complex-heading-regexp)
  (if (not (match-end 4)) ""
;; Remove statistics cookies.
(org-trim
 (org-link-display-format
  (replace-regexp-in-string
   "\\[[0-9]+%\\]\\|\\[[0-9]+/[0-9]+\\]" ""
   (match-string-no-properties 4 => (0.013364877 0 0.0)

Best,
Ihor

Re: Concerns about community contributor support

[Ihor Radchenko]

D  writes:
> TL;DR: Maybe we could improve the visibility of patches by having a
> dedicated mailing list for them? This would also allow for a greater
> deal of automation in the way we deal with patches.

What about https://updates.orgmode.org/? Or do you refer to something
else?

Best,
Ihor

Re: [PATCH] Allow multiple %(expression) instances in org-agenda-prefix-format

[Ihor Radchenko]

Bastien  writes:

> Applied as 0260d2fcf, thanks!

Looking at the patch again, I notice that it will break code like
"%((this (has (many parenthesis". Though I have no clue how to match
balanced parenthesis using Elisp regexps.

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Maxim Nikulin  writes:

> Maybe I could avoid org-goto as well. Actual reason to use it was that 
> it does not ask for file name as the first step in the case of 
> (org-refile-use-outline-path 'file). It took enough time to me to 
> realize how to jump/refile to non-leaf heading without such settings.

Note that Org mode has integration with imenu. It is an alternative you
can use to jump around current buffer.

> Just an idea. Is it possible to implement some specific text property 
> for heading lines, namely cleaned out heading text (no cookies, tags, 
> hidden parts of links), that is updated after each editing (likely 
> something like font locks)? It could significantly speed up scanning 
> buffer for goto/refile targets. Unfortunately it would not help for 
> files that have not opened yet.

There is org-outline-path-cache used by org-get-outline-cache. It avoids
computing parent's outline path multiple times, which is already a great
improvement.

For the cleaned heading text, I do not think that re-calculating the
heading text on each change is a good idea. It may degrade typing
latency. Yet, an acceptable approach could be simply invalidating cache
for the changed headings. Then, outline paths can be re-calculated on
changed headings when needed. This should be an improvement compared to
blanket (setq org-outline-path-cache nil) in org-refile-get-targets.

> To be clear, org-refile and org-goto share the same cache and it is the 
> source of the problem.

I would say that the problem is rather org-goto (ab)using org-refile
incorrectly. I have the following piece of elisp using org-refile
mechanism for headline selection:

(let ((org-refile-history (append saved-id parent-project))
  (org-refile-cache nil)
  (org-refile-target-verify-function #'org-att-id-skip-function))
  (let ((prompt-ans (org-refile-get-location "Link to attachment from")))
(prog1
(org-id-get (seq-find #'markerp
  prompt-ans)
'create

org-refile-cache can be simply let-bound to nil (in my case) or
alternative cache variable, if the alternative cache should persist.

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Samuel Wales  writes:

> would it be more useful if it automaticaly generated the cache instead
> of telling you to runt he command to do so?

I think so. To be frank, I do not understand the reason why it is not
done by default.

> if a solid, perhaps unified, cache existed, would org-id use it too?

Sure. Why not? I imagine such cache can store the following info:

org files in system -> per-file cache -> per-heading cache -> ...

Best,
Ihor

Re: [PATCH] Bug: fragile org refile cache

[Ihor Radchenko]

Samuel Wales  writes:

> long ago i used to use the refile cache.  i think it is probably not
> widely used, or maybe even not at all.

At least, I do use it. A lot. I rely on it.

I do not observe the breakage as described in the first message, mostly
because I use refile cache exclusively for org-refile. Yet, I often see
"Please regenerate the refile cache with C-0 C-c C-w" when I do a lot of
batch refiling. A faster, more reliable, caching would be certainly
welcome.

In general, various parts of Org mode code base implement different
types of caches in parallel. I am aware at least about org-element,
org-scan-tags, org-agenda, org-refile, and org-goto. Probably Org mode
could benefit from unified caching mechanism? A good implementation
coming to my mind is org-ql [1]. It implements tag caches, outline path
caches, and can even be used to cache results of an arbitrary function
with point at heading. Basically, all (except org-element) types of
caches Org mode uses now are already implemented in org-ql in unified
way.

WDYT?

[1] https://github.com/alphapapa/org-ql

Best,
Ihor

Re: Bug: Org mode fails to compile using Emacs 24.5-r10 [9.4.5 (9.4.5-g3ea248 @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Timothy  writes:

> Maybe this is a good time to start a discussion about moving Org's
> minimum supported Emacs to 25...?

I have no objections here, since I am on Emacs master anyway.

In any case, not all the warnings go away even using Emacs 25.3:

Compiling /home/yantar92/Git/org-mode/lisp/ob-gnuplot.el...

In end of data:
ob-gnuplot.el:299:1:Warning: the function ‘file-local-name’ is not known to be
defined.

Compiling /home/yantar92/Git/org-mode/lisp/org-plot.el...

In end of data:
org-plot.el:727:1:Warning: the function ‘ignore-error’ is not known to be
defined.

Compiling /home/yantar92/Git/org-mode/lisp/org.el...

In org-display-inline-images:
org.el:16554:57:Warning: reference to free variable ‘image-map’

Best,
Ihor

Re: org-goto and org-store-link/org-id-get-create

[Ihor Radchenko]

Bastien  writes:

> Can you provide a patch for this?

Sure. Attached.

>From d914acea52d251e2099681ac9541e4cb42e0953f Mon Sep 17 00:00:00 2001
Message-Id: 
From: Ihor Radchenko 
Date: Wed, 28 Apr 2021 22:51:53 +0800
Subject: [PATCH] Bypass read-only state in org-entry-put

* lisp/org.el (org-entry-put): Ignore read-only state of the buffer.

Fixes bug when ID property is not insered when creating ID in an
indirect read-only org-goto buffer. [1]

[1] https://orgmode.org/list/8ffe2da5-e2cb-f44c-0a46-b19873c0b...@gmx.de/
---
 lisp/org.el | 113 ++--
 1 file changed, 57 insertions(+), 56 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index eb4b2db88..dbc245534 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -12340,62 +12340,63 @@ (defun org-entry-put (pom property value)
 	((not (stringp value)) (error "Properties values should be strings"))
 	((not (org--valid-property-p property))
 	 (user-error "Invalid property name: \"%s\"" property)))
-  (org-with-point-at pom
-(if (or (not (featurep 'org-inlinetask)) (org-inlinetask-in-task-p))
-	(org-back-to-heading-or-point-min t)
-  (org-with-limited-levels (org-back-to-heading-or-point-min t)))
-(let ((beg (point)))
-  (cond
-   ((equal property "TODO")
-	(cond ((not (org-string-nw-p value)) (setq value 'none))
-	  ((not (member value org-todo-keywords-1))
-	   (user-error "\"%s\" is not a valid TODO state" value)))
-	(org-todo value)
-	(org-align-tags))
-   ((equal property "PRIORITY")
-	(org-priority (if (org-string-nw-p value) (string-to-char value) ?\s))
-	(org-align-tags))
-   ((equal property "SCHEDULED")
-	(forward-line)
-	(if (and (looking-at-p org-planning-line-re)
-		 (re-search-forward
-		  org-scheduled-time-regexp (line-end-position) t))
-	(cond ((string= value "earlier") (org-timestamp-change -1 'day))
-		  ((string= value "later") (org-timestamp-change 1 'day))
-		  ((string= value "") (org-schedule '(4)))
-		  (t (org-schedule nil value)))
-	  (if (member value '("earlier" "later" ""))
-	  (call-interactively #'org-schedule)
-	(org-schedule nil value
-   ((equal property "DEADLINE")
-	(forward-line)
-	(if (and (looking-at-p org-planning-line-re)
-		 (re-search-forward
-		  org-deadline-time-regexp (line-end-position) t))
-	(cond ((string= value "earlier") (org-timestamp-change -1 'day))
-		  ((string= value "later") (org-timestamp-change 1 'day))
-		  ((string= value "") (org-deadline '(4)))
-		  (t (org-deadline nil value)))
-	  (if (member value '("earlier" "later" ""))
-	  (call-interactively #'org-deadline)
-	(org-deadline nil value
-   ((member property org-special-properties)
-	(error "The %s property cannot be set with `org-entry-put'" property))
-   (t
-	(let* ((range (org-get-property-block beg 'force))
-	   (end (cdr range))
-	   (case-fold-search t))
-	  (goto-char (car range))
-	  (if (re-search-forward (org-re-property property nil t) end t)
-	  (progn (delete-region (match-beginning 0) (match-end 0))
-		 (goto-char (match-beginning 0)))
-	(goto-char end)
-	(insert "\n")
-	(backward-char))
-	  (insert ":" property ":")
-	  (when value (insert " " value))
-	  (org-indent-line)
-(run-hook-with-args 'org-property-changed-functions property value)))
+  (org-no-read-only
+   (org-with-point-at pom
+ (if (or (not (featurep 'org-inlinetask)) (org-inlinetask-in-task-p))
+	 (org-back-to-heading-or-point-min t)
+   (org-with-limited-levels (org-back-to-heading-or-point-min t)))
+ (let ((beg (point)))
+   (cond
+((equal property "TODO")
+	 (cond ((not (org-string-nw-p value)) (setq value 'none))
+	   ((not (member value org-todo-keywords-1))
+	(user-error "\"%s\" is not a valid TODO state" value)))
+	 (org-todo value)
+	 (org-align-tags))
+((equal property "PRIORITY")
+	 (org-priority (if (org-string-nw-p value) (string-to-char value) ?\s))
+	 (org-align-tags))
+((equal property "SCHEDULED")
+	 (forward-line)
+	 (if (and (looking-at-p org-planning-line-re)
+		  (re-search-forward
+		   org-scheduled-time-regexp (line-end-position) t))
+	 (cond ((string= value "earlier") (org-timestamp-change -1 'day))
+		   ((string= value "later") (org-timestamp-change 1 'day))
+		   ((string= value "") (org-schedule '(4)))
+		   (t (org-schedule nil value)))
+	   (if (member value '("earlier" "later" ""))
+	   (call-interactively #'org-schedule)
+	 (org-schedule nil value
+((equal property "DEADLINE")
+	 (forward-line)
+	 (if (and (looking

Bug: Org mode fails to compile using Emacs 24.5-r10 [9.4.5 (9.4.5-g3ea248 @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

I was recently testing Org mode using old Emacs versions. Running make
on master fails with the following errors and warnings:

Compiling /home/yantar92/Git/org-mode/lisp/ob-C.el...
Eager macro-expansion failure: (error "Vector QPatterns not implemented yet")

Compiling /home/yantar92/Git/org-mode/lisp/ob-clojure.el...

In end of data:
ob-clojure.el:255:1:Warning: the function `funcall-interactively' is not known
to be defined.

Compiling /home/yantar92/Git/org-mode/lisp/ob-core.el...

In toplevel form:
ob-core.el:649:1:Error: Vector QPatterns not implemented yet

Compiling /home/yantar92/Git/org-mode/lisp/ob-gnuplot.el...

In end of data:
ob-gnuplot.el:299:1:Warning: the function `file-local-name' is not known to be
defined.

Compiling /home/yantar92/Git/org-mode/lisp/ob-python.el...
Compiler-macro error for python-syntax-context: (void-function 
python-syntax--context-compiler-macro)

Compiling /home/yantar92/Git/org-mode/lisp/ol-eww.el...

In org-eww-store-link:
ol-eww.el:76:36:Warning: reference to free variable `eww-data'

In end of data:
ol-eww.el:182:1:Warning: the function `eww-current-url' is not known to be
defined.

Compiling /home/yantar92/Git/org-mode/lisp/org-agenda.el...

In toplevel form:
org-agenda.el:5400:1:Warning: Unused lexical argument `e'

In end of data:
org-agenda.el:10851:1:Warning: the following functions are not known to be
defined: funcall-interactively, window-font-width

Compiling /home/yantar92/Git/org-mode/lisp/org-plot.el...

In org-plot/gnuplot:
org-plot.el:633:4:Warning: `(dump-func (plist-get type :data-dump))' is a
malformed function
org-plot.el:682:17:Warning: reference to free variable `dump-func'

In end of data:
org-plot.el:727:1:Warning: the following functions are not known to be
defined: seq-group-by, if-let, ignore-error

Compiling /home/yantar92/Git/org-mode/lisp/org.el...

In org-display-inline-images:
org.el:16554:57:Warning: reference to free variable `image-map'

In end of data:
org.el:21292:1:Warning: the function `make-process' is not known to be
defined.

Best,
Ihor

Re: Bug: Logbook drawer and org-adapt-indentation with value headline-data [9.4 (9.4-7-g3eccc5-elpaplus @ /home/gustavo/.emacs.d/elpa/org-plus-contrib-20200921/)]

[Ihor Radchenko]

Bastien  writes:

> This should be fixed with commit 730a05f78.

> ;; FIXME: when storing a note in a LOGBOOK drawer,
> ;; `org-store-log-note' needs to insert a new line before
> ;; the newly inserted note, thus the `type' at point will
> ;; return `paragraph' instead of the expected `drawer', so
> ;; we need to manually detect the drawer.

Maybe you can use

(eq (org-element-type (car (org-element-lineage element))) 'drawer)

A quick test showed that org-element-lineage can detect drawer even when
element is a note inside LOGBOOK drawer.

Best,
Ihor

Re: Bug: org-plot gives Invalid function error

[Ihor Radchenko]

Timothy  writes:

> That's why the reset command is run:
> https://code.orgmode.org/bzg/org-mode/src/master/lisp/org-plot.el#L560

Sorry, I missed that. It should indeed make things much less likely to
break. One exception is when plot depends on settings defined in
.gnuplot file. reset command clears user customisations in .gnuplot.

Of course, the described bug may be related to something else.
[after some testing...]
Of course, it is:

> plot '/tmp/org-plotiPs0To' using 1:3 with histograms title 'H-index'

This is not valid gnuplot command to create histograms. One needs to use:

plot '/tmp/org-plotiPs0To' using 3:tics(1) with histograms title 'H-index'

Best,
Ihor

Re: On using to-do lists efficiently

[Ihor Radchenko]

Bastien  writes:

> Slightly offtopic but I sat down this week-end trying to grasp with
> very few words what I learned on how to use to-do lists efficiently
> over the years, and here it is:

I am wondering if we can incorporate such or similar tips into Org mode
manual. Similar to Elisp manual section info:elisp#tips, Org mode may
introduce recommended usage of Agenda, TODO-lists, and other major Org
mode components.

Similar idea was also discussed in
https://orgmode.org/list/x9e%2ffv%2f9hd4bl...@protected.rcdrun.com/ 

> I'm curious if this resonates with your experience.

I have very similar experience, though I have several subtle comments on
some points you raise.

> Write less to-do items and more notes.

Having less TODO items is really helpful. At some point, I found myself
cherry-picking tasks that are easy to do, but not important in place of
tasks that I really needed to do. Moving "wish to" tasks away from
visibility really helps to focus as the size of todo-lists grows to
thousands of tasks.

Yet, I found it helpful to have few "wish to" notes as actual tasks. When
you want to take a short break or have some free time it is handy to
have some "light" task within reach. Otherwise, social media tends to
fill all the free time slots.
I have a setup to quickly move groups of "wish to" tasks between notes
and tasks depending on my workload.

> Your to-do list should be a list of things to do, not to remember

While I generally agree with this, removing things to remember from
visibility most often results in forgetting them, especially as the
number remember notes grows to thousands. Thousands of notes are hard to
review regularly. I find it useful to bring such notes to my attention
from time to time using spaced repetition.

> don't mix notes and tasks

I agree with one exception - notes on literature/articles. For my work,
I need to deal with a lot of reading. It is common that some book may
need to be read multiple times looking for different kinds of
information. Having common summary notes right in the "reading" task is
quite helpful to get started, especially if the previous reading time
was months ago.

> Write precise, concise, atomic tasks

This is a great suggestion. Vague tasks tends to be ignored or
postponed. Having a very concrete action as a task makes it easy to do.
Yet, assigning concrete action to some tasks may itself take significant
time, especially for complex tasks requiring some research. For such
tasks, I often add one simple action required to get started on the
task. This does not require spending much time on planning each step to
complete the task, yet making complex tasks look less intimidating.

Best,
Ihor

Re: Bug: org-plot gives Invalid function error

[Ihor Radchenko]

Timothy  writes:

> From a quick test myself, this appears reproducible, though I have no
> idea what's going on (yet). Please let me know if you find anything.

I remember seeing similar gnuplot errors using ob-gnuplot. They tend to
disappear upon restarting gnuplot process (M-x gnuplot-kill-gunplot-buffer),
probably because ob-gnuplot does not refresh gnuplot session.

A quick look through org-plot code shows that gnuplot session is also
preserved:

(when (get-buffer "*gnuplot*") ; reset *gnuplot* if it already running
  (with-current-buffer "*gnuplot*"
(goto-char (point-max

So, I suspect that the problem I encountered may also be relevant in
org-plot.

Hope it helps.

Best,
Ihor

Re: wip-cite status question and feedback

[Ihor Radchenko]

Nicolas Goaziou  writes:

> In my mind, "opening" leads to the bibliography reference, not to the
> original document. IIUC, in this situation, the location does not matter
> much, does it?
>
> In any case, Org has no clue about the "location" of the citation. It
> can provide the suffix associated to the citation key, but it is up to
> the citation processor to extract page information out of it. If this is
> desirable, we need to provide the full citation object instead of the
> key. I don't know if it is worth the trouble.

>From my experience using org-ref for citations, going to the
bibliography reference is rarely useful. Most of time, I want to jump to
the document I am citing (or even page in the document).

I think that full citation object should be worth passing to the
citation backend.

Best,
Ihor

Re: Bug: org mode rendering linked heading wrongly [9.4.4 (9.4.4-4-g99eafe-elpaplus @ /Users/cvl/.config/emacs/elpa/28.0/develop/org-plus-contrib-20210104/)]

[Ihor Radchenko]

Nicolas Goaziou  writes:

>> [[**heading to be linked to*]]
>>
>> Expected rendered result:
>
> I had troubles understanding your report. What does "rendering" means
> here? Is it about `org-insert-link', fontification, export?

I think the report referred to fontification. When
org-hide-emphasis-markers is non-nil, the heading is rendered in bold
and the "*" markers are hidden. However, "*" emphasis markers are not
hidden in the link - inconsistent with what users could expect from
non-nil org-hide-emphasis-markers setting.

Best,
Ihor

Re: BUG?: Are TAGs case sensitive? ('org-agenda-show-tags')

[Ihor Radchenko]

> Is it a bug that 'org-agenda-show-tags' does this?

It is a bug. Agenda is downcasing all the tags. Also, see
https://orgmode.org/list/87k0rftf8t.fsf@localhost/

Someone needs to study the agenda code to understand the need of
downcasing and how to remove it safely. Patches are welcome.

Best,
Ihor

David Masterson  writes:

> David Masterson  writes:
>
>> I have been using upper-case TAGs and just noticed that
>> 'org-agenda-show-tags' is reporting them in lower-case which is not
>> right since I have another TAG that is the lower-case version of the
>> upper-case TAG (sort of a visual importance indicator).
>
> In case my wording wasn't good, I meant:
>
> 1. I have upper case tags attached to several task headers,
> 2. 'org-agenda-show-tags' reports them as lower case tags.
> 3. I have unused tags in 'org-tag-alist' that are the lower case forms
> of the upper case tags (as well as the upper case ones)
>
> Is it a bug that 'org-agenda-show-tags' does this?
>
> -- 
> David Masterson

Re: [feature request] The :pre header argument

[Ihor Radchenko]

Hi,

Rodrigo Morales  writes:

> I've provided more relevant information on this feature request [[
> https://codeberg.org/rdrg109/gists/src/branch/main/feature-request-pre-header-argument.org][here]].
> Please consider reading that instead of the first message on this thread.

I think the origin intention of the :post header argument is slightly
different from your use-case. It is intended to be used for
post-processing the results:

> The ‘post’ header argument is for post-processing results from block
> evaluation.  When ‘post’ has any value, Org binds the results to
> ‘*this*’ variable for easy passing to ‘var’ header argument
> specifications (see *note Environment of a Code Block::).  That makes
> results available to other code blocks, or even for direct Emacs Lisp
> code execution.

If you want to execute some arbitrary code before/after your code block,
it is probably more canonical to use noweb references. Using your
example:

#+begin_src emacs-lisp
# Cleanup first using experiments/clean-dir(): "<>"
# Create dir structure using
# minimal-reproducible-example/create-dir-structure:
# "<>"
# Finally, execute `tree' using experiments/execute-tree:
(message "<>")
#+end_src

It will not produce spurious variable definitions.

Best,
Ihor

Re: Why is there no inline-src syntax highlighting?

[Ihor Radchenko]

Timothy  writes:

> Update: I've been using
> https://tecosaur.github.io/emacs-config/config.html#fontifying-inline-src
> for a while now and it seems to work out, there's just one thing I
> haven't been able to work out --- how to have font-lock apply to
> *multiple* inline src blocks on one line.
>
> If anyone has any ideas, please let me know. Once I've got this sorted
> I'll probably make a patch.

I think you just need to replace `when' with `while' in the following line:

(when (re-search-forward "\\_

Re: How to undo all items, including headings?

[Ihor Radchenko]

Jean Louis  writes:

> I would like to undo all TODO/DONE/PENDING tags and list items in a
> list of lists similar like this below.
> ...
> Does similar function exist already?

org-checklist.el from contrib may be what you need.

Best,
Ihor

Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Updated version of the patch.

Ihor Radchenko  writes:

> Nicolas Goaziou  writes:
>> Actually, this is (was?) intentional. By forbidding parenthesis in
>> a plain URL, you allow one to type, e.g., (https://orgmode.org), which
>> is, IMO, a more frequent need than having to deal with parenthesis in
>> the URL.
>
> The patch correctly recognises situations like this.
> https://orgmode.org is recognised correctly without parenthesis.
>
> I guess this example may be another addition to the tests.
>
> Best,
> Ihor

>From 08efc990a578c925d42315c45e0b9b76536b92af Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Wed, 24 Mar 2021 21:27:24 +0800
Subject: [PATCH] Improve org-link-plain-re

(org-link-plain-re): Update docstring.  Now, the docstring explicitly
mentions that the regexp must contain groups for the link type and the
path.

* lisp/ol.el (org-link-make-regexps): Allow URLs with up to two levels
of nested brackets.  Now, URLs like [1] can be matched.  The new
regexp is based on [2].

* testing/lisp/test-ol.el: Add tests for the plain link regexp

[1] https://doi.org/10.1016/0160-791x(79)90023-x
[2] https://daringfireball.net/2010/07/improved_regex_for_matching_urls
---
 lisp/ol.el  |  41 +++
 testing/lisp/test-ol.el | 110 
 2 files changed, 141 insertions(+), 10 deletions(-)

diff --git a/lisp/ol.el b/lisp/ol.el
index b8bd7d234..550c0cff6 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -519,7 +519,10 @@ links more efficient."
   "Matches link with angular brackets, spaces are allowed.")
 
 (defvar org-link-plain-re nil
-  "Matches plain link, without spaces.")
+  "Matches plain link, without spaces.
+Group 1 must contain the link type (i.e. https).
+Group 2 must contain the link path (i.e. //example.com).
+Used by `org-element-link-parser'.")
 
 (defvar org-link-bracket-re nil
   "Matches a link in double brackets.")
@@ -807,15 +810,33 @@ This should be called after the variable `org-link-parameters' has changed."
 	  (format "<%s:\\([^>\n]*\\(?:\n[ \t]*[^> \t\n][^>\n]*\\)*\\)>"
 		  types-re)
 	  org-link-plain-re
-	  (concat
-	   "\\<" types-re ":"
-	   "\\([^][ \t\n()<>]+\\(?:([[:word:]0-9_]+)\\|\\([^[:punct:] \t\n]\\|/\\)\\)\\)")
-	  ;;	 "\\([^]\t\n\r<>() ]+[^]\t\n\r<>,.;() ]\\)")
-	  org-link-bracket-re
-	  (rx (seq "[["
-		   ;; URI part: match group 1.
-		   (group
-		(one-or-more
+  (let* ((non-space-bracket "[^][ \t\n()<>]")
+	 (parenthesis
+		  `(seq "("
+		(0+ (or (regex ,non-space-bracket)
+			(seq "("
+ (0+ (regex ,non-space-bracket))
+ ")")))
+		")")))
+	;; Heuristics for an URL link inspired by
+	;; https://daringfireball.net/2010/07/improved_regex_for_matching_urls
+	(rx-to-string
+	 `(seq word-start
+   ;; Link type: match group 1.
+		   (regexp ,types-re)
+		   ":"
+   ;; Link path: match group 2.
+   (group
+		(1+ (or (regex ,non-space-bracket)
+			,parenthesis))
+		(or (regexp "[^[:punct:] \t\n]")
+		?/
+		,parenthesis)
+  org-link-bracket-re
+  (rx (seq "[["
+	   ;; URI part: match group 1.
+	   (group
+	(one-or-more
  (or (not (any "[]\\"))
 			 (and "\\" (zero-or-more "") (any "[]"))
 			 (and (one-or-more "\\") (not (any "[]"))
diff --git a/testing/lisp/test-ol.el b/testing/lisp/test-ol.el
index 5b7dc513b..ddcc570b3 100644
--- a/testing/lisp/test-ol.el
+++ b/testing/lisp/test-ol.el
@@ -491,5 +491,115 @@
 	  (org-previous-link))
 	(buffer-substring (point) (line-end-position))
 
+
+;;; Link regexps
+
+
+(defmacro test-ol-parse-link-in-text (text)
+  "Return list of :type and :path of link parsed in TEXT.
+\"\" string must be at the beginning of the link to be parsed."
+  (declare (indent 1))
+  `(org-test-with-temp-text ,text
+ (list (org-element-property :type (org-element-link-parser))
+   (org-element-property :path (org-element-link-parser)
+
+(ert-deftest test-ol/plain-link-re ()
+  "Test `org-link-plain-re'."
+  (should
+   (equal
+'("https" "//example.com")
+(test-ol-parse-link-in-text
+"(https://example.com)")))
+  (should
+   (equal
+'("https" "//example.com/qwe()")
+(test-ol-parse-link-in-text
+"(Some text https://example.com/qwe())")))
+  (should
+   (equal
+'("https" "//doi.org/10.1016/0160-791x(79)90023-x")
+(test-ol-parse-link-in-text
+"https://doi.org

Re: [PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Maxim Nikulin  writes:

> Example for #+STARTUP: overview:
>
> org-activate-links  560 0.028971085   5.173...e-05
>
> For content number of calls is 410, without special settings (all) 120,
> let me remind that it is for 10 find-file invocations. Another example
>
> org-activate-links  410 0.1384633219  0.0003377154

I repeated your benchmark on my largest working file (~14k links):

;; with patch
;; org-activate-links  400 0.0259791009  6.494...e-05
;; org-activate-links  400 0.0114822140  2.870...e-05
;; org-activate-links  400 0.0255080609  6.377...e-05
;; without patch
;; org-activate-links  400 0.0297167870  7.429...e-05
;; org-activate-links  400 0.0149334709  3.733...e-05
;; org-activate-links  400 0.0105385180  2.634...e-05

There is not much difference indeed. I guess, there was something about
my config and external packages.

> I see such variations in both cases with and without the patch, but 
> these numbers are negligible in my opinion.

Your benchmark is measuring is jit-lock - there will be no reliable
result as jit-lock is timer-based.

I did more reliable version as well:

(progn
  (require 'elp)
  (require 'org-element)
  (setq elp-function-list (list #'org-activate-links))
  (elp-instrument-list nil)
  (dolist (i (number-sequence 1 10))
(message "iter %d" i)
(find-file "~/Org/notes.org")
(font-lock-ensure) ;; Force fontification in all the buffer
(sit-for 1)
(kill-buffer "notes.org")
(sit-for 1))
  (elp-results))

Results are not different though (time-per-single-call):

;; with patch + font-lock-ensure
;; org-activate-links  163290  9.720667509   5.953...e-05
;; org-activate-links  163290  9.8090518640  6.007...e-05
;; without patch + font-lock-ensure
;; org-activate-links  163290  9.9175657860  6.073...e-05
;; org-activate-links  163290  10.073281878  6.168...e-05

This latter case was what was happening with my config. Some package was
causing full buffer fontification.

> In my opinion, combining changes related to white spaces and meaningful 
> modifications makes commits less clear, especially when reading email. 
> However the following recommendation has certainly more weight:
>
> https://orgmode.org/list/87zh2hosex@bzg.fr/ From: Bastien
>> Also, the convention in Emacs is to avoid whitespaces-only commits,
>> you need to fix whitespaces within other non-whitespaces changes in
>> a commit.

Actually, I feel confused now. I remember that message from Bastien, but
now I cannot recall what is considered "fix whitespaces". Do we use
tab-convention or space-convention?

I think I will better clear the whitespace staff in the patch before I
understand the whitespace policy more clearly. At least, the patch will
be more readable.

Best,
Ihor

Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Nicolas Goaziou  writes:
> Actually, this is (was?) intentional. By forbidding parenthesis in
> a plain URL, you allow one to type, e.g., (https://orgmode.org), which
> is, IMO, a more frequent need than having to deal with parenthesis in
> the URL.

The patch correctly recognises situations like this.
https://orgmode.org is recognised correctly without parenthesis.

I guess this example may be another addition to the tests.

Best,
Ihor

Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Ihor Radchenko]

Hello,

This is another update about the status of the patch.

I am mostly happy with the current state of the code, got rid of most of
the bugs, and did not get any new bug reports in github for a while.

I would like to start the process of applying the patch on master.
As a first step, I would like to submit the core folding library
(org-fold-core) for review.

org-fold-core is pretty much independent from org-mode code base and
does not affect anything if applied as is. It will be used by
org-specific org-fold.el I will finalise and send later.

For now, I would like to hear any suggestions about API and
implementation of org-fold-core.el. I tried to document all the details
in the code.

Looking forward for the feedback.

Best,
Ihor



org-fold-core.el
Description: application/emacs-lisp

Re: [PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Maxim Nikulin  writes:

> To be clear, I do not ask for any changes. It is great to have some 
> tests even in the current form. I just have never tried ert before, so I 
> have some questions.

Note that I have no experience with tests since I never did programming
professionally. I wrote the tests in the patch simply by looking at
other examples in the file.

> Am I right that just one failure will be reported if a change in the 
> regexp causes problems with several test inputs? I am afraid, it is 
> rather inconvenient since this tests are rather for quality of 
> heuristics than exact requirements. To find proper balance of accuracy 
> and speed/regexp complexity, it is better to have all failures at once. 
> I am looking up for a feature like EXPECT_EQ (delayed failure) in 
> addition to strict ASSERT_EQ in googletest or at least like 
> TestCase.subTest in puthon unittest. For this particular case even 
> parametrized tests are enough (googletest, pytest).

I cannot find anything like delayed failure in the info node for ERT and
in the source code. There are should, should-not, should-error, and
skip-unless asserts.

> I have tried implement something similar to illustrate my expectations. 
> I think, for experienced lisp programmers the code looks ugly

Using a macro is certainly more efficient than my copy-paste in the
patch :) Though you may probably use more backquoting (see Backquote in
Elisp manual) when writing macros. That would look more readable.

> I do not know if it is possible to implement "might" (in addition to 
> "should") that using restart or some other technique will prevent 
> immediate abandoning of the test but will mark whole test as failed at 
> the end.

It is indeed possible and maybe even welcome in emacs-devel.

> Actually I hope to get response that I am trying to reinvent a wheel and 
> org or emacs has an established way to write tests feeding the same 
> function with list of cases

Well... Example from ERT info page:

 (ert-deftest ert-test-mismatch ()
   (should (eql (cl-mismatch "" "") nil))
   (should (eql (cl-mismatch "" "a") 0))
   (should (eql (cl-mismatch "a" "a") nil))
   (should (eql (cl-mismatch "ab" "a") 1))
   (should (eql (cl-mismatch "Aa" "aA") 0))
   (should (eql (cl-mismatch '(a b c) '(a b d)) 2)))

> ... and to get all failures in a reasonably readable report.

When running ert interactively, things should get a little more
manageable. See Running Tests Interactively node of ERT manual.

Hope it helps.

Best,
Ihor

Re: [PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Maxim Nikulin  writes:

> I could not guess how to benchmark font-lock. I have tried to open file 
> (to get everything loaded), kill the buffer,

I usually just use profiler-start/open buffer/profiler-report. However,
there is also https://github.com/Lindydancer/font-lock-profiler for more
fine-grained benchmark. Also, you may instrument org-activate-links with
elp.el (built-in).

> but I see some changes in the buffer after 0.19 is reported (both with 
> and without the patch). However I have not converted bracketed links 
> into plain ones yet. I was going to try if some tricks could improve 
> performance. E.g. I am curious if it will work noticeably faster when no 
> nested parenthesis are allowed, but single ones may be at any position, 
> not necessary at the end.

I am currently looking into somewhat orthogonal approach. Instead of
tweaking individual regexps (there were similar issues with priority
regexp in the past), I am trying to use custom
font-lock-fontify-region-function. Once we avoid fontifying folded text,
startup time drops several times at least. I can see the difference
immediately. It comes at the cost though - the behaviour of some Org API
functions changes. They will not always return fontified text. AFAIK, at
least helm-org and helm-org-ql do reply on such fontification.

> Are changes in white spaces below actually modified lines in your patch 
> intended?

They are generated by aggressive-indent. Since org files mix indentation
styles I keep getting those whitespace changes in my patches. Forgot to
remove this time. Should I?

Best,
Ihor

Re: [PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Maxim Nikulin  writes:

> I have tried to compare current, your, and a little modified your 
> regexps on several synthetic examples:

Thanks! Your version is indeed cleaner. I updated the patch. It uses
your version now with one group specification that I missed earlier. I
also added tests from your email, the post introducing regexp, and few
more examples I came up with.

I am testing the new regexp for a few days now. Because the regexp is
quite complex and because font-lock apparently fontifies even invisible
(folded) text, loading time on large org files with many links became
noticeably longer. Though it was 7.2Mb file with ~13k links in it.

Best,
Ihor

>From 6eb2208a67745e3150024e7b72509115b97fcfa3 Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Tue, 16 Mar 2021 20:20:32 +0800
Subject: [PATCH] Improve org-link-plain-re

(org-link-plain-re): Update docstring.  Now, the docstring explicitly
mentions that the regexp must contain groups for the link type and the
path.

* lisp/ol.el (org-link-make-regexps): Allow URLs with up to two levels
of nested brackets.  Now, URLs like [1] can be matched.  The new
regexp is based on [2].

[1] https://doi.org/10.1016/0160-791x(79)90023-x
[2] https://daringfireball.net/2010/07/improved_regex_for_matching_urls

* testing/lisp/test-ol.el: Add tests for plain links.
---
 lisp/ol.el  |  61 +--
 testing/lisp/test-ol.el | 132 
 2 files changed, 173 insertions(+), 20 deletions(-)

diff --git a/lisp/ol.el b/lisp/ol.el
index b8bd7d234..d3a07a3ed 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -519,7 +519,10 @@ links more efficient."
   "Matches link with angular brackets, spaces are allowed.")
 
 (defvar org-link-plain-re nil
-  "Matches plain link, without spaces.")
+  "Matches plain link, without spaces.
+Group 1 must contain the link type (i.e. https).
+Group 2 must contain the link path (i.e. //example.com).
+Used by `org-element-link-parser'.")
 
 (defvar org-link-bracket-re nil
   "Matches a link in double brackets.")
@@ -807,26 +810,44 @@ This should be called after the variable `org-link-parameters' has changed."
 	  (format "<%s:\\([^>\n]*\\(?:\n[ \t]*[^> \t\n][^>\n]*\\)*\\)>"
 		  types-re)
 	  org-link-plain-re
-	  (concat
-	   "\\<" types-re ":"
-	   "\\([^][ \t\n()<>]+\\(?:([[:word:]0-9_]+)\\|\\([^[:punct:] \t\n]\\|/\\)\\)\\)")
-	  ;;	 "\\([^]\t\n\r<>() ]+[^]\t\n\r<>,.;() ]\\)")
-	  org-link-bracket-re
-	  (rx (seq "[["
-		   ;; URI part: match group 1.
-		   (group
-		(one-or-more
+  (let* ((non-space-bracket "[^][ \t\n()<>]")
+	 (parenthesis
+		  `(seq "("
+		(0+ (or (regex ,non-space-bracket)
+			(seq "("
+ (0+ (regex ,non-space-bracket))
+ ")")))
+		")")))
+	;; Heuristics for an URL link inspired by
+	;; https://daringfireball.net/2010/07/improved_regex_for_matching_urls
+	(rx-to-string
+	 `(seq word-start
+   ;; Link type: match group 1.
+		   (regexp ,types-re)
+		   ":"
+   ;; Link path: match group 2.
+   (group
+		(1+ (or (regex ,non-space-bracket)
+			,parenthesis))
+		(or (regexp "[^[:punct:] \t\n]")
+		?/
+		,parenthesis)
+  org-link-bracket-re
+  (rx (seq "[["
+	   ;; URI part: match group 1.
+	   (group
+	(one-or-more
  (or (not (any "[]\\"))
-			 (and "\\" (zero-or-more "") (any "[]"))
-			 (and (one-or-more "\\") (not (any "[]"))
-		   "]"
-		   ;; Description (optional): match group 2.
-		   (opt "[" (group (+? anything)) "]")
-		   "]"))
-	  org-link-any-re
-	  (concat "\\(" org-link-bracket-re "\\)\\|\\("
-		  org-link-angle-re "\\)\\|\\("
-		  org-link-plain-re "\\)"
+		 (and "\\" (zero-or-more "") (any "[]"))
+		 (and (one-or-more "\\") (not (any "[]"))
+	   "]"
+	   ;; Description (optional): match group 2.
+	   (opt "[" (group (+? anything)) "]")
+	   "]"))
+  org-link-any-re
+  (concat "\\(" org-link-bracket-re "\\)\\|\\("
+	  org-link-angle-re "\\)\\|\\("
+	  org-link-plain-re "\\)"
 
 (defun org-link-complete-file ( arg)
   "Create a file link using completion."
diff --git a/testing/lisp/test-ol.el b/testing/lisp/test-ol.el
index 5b7dc513b..e6208cd38 100644
--- a/testing/lisp/test-ol.el
+++ b/testing/lisp/test-ol.el
@@ -491,5 +491,1

Re: org-capture: question about function to create template

[Ihor Radchenko]

Joost Kremers  writes:

> ... I was wondering if there's any
> way for this function to access the state of the ongoing capture process.
> Specifically, it would be useful for me if the function can find out the key 
> of
> the template that the user selected.

See org-capture-plist.

Also, you might find https://github.com/progfolio/doct useful.

Best,
Ihor

[PATCH] Allow multiple %(expression) instances in org-agenda-prefix-format

[Ihor Radchenko]

>From ae1e4041f77d056649b8fa90d12e2cd1354b78f3 Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Mon, 15 Mar 2021 15:18:20 +0800
Subject: [PATCH] Allow multiple %(expression) instances in
 org-agenda-prefix-format

* lisp/org-agenda.el (org-compile-prefix-format): Use non-greedy match
for %(expression).

Previously, format like "%-12.12s%(expr1) %(expr2)" would not be
parsed correctly because of greedy "(.+)" regexp used to match the
expressions.
---
 lisp/org-agenda.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 50a9b3dbd..8bd328e6d 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -6929,7 +6929,7 @@ and stored in the variable `org-prefix-format-compiled'."
 	(t "  %-12:c%?-12t% s")))
 	(start 0)
 	varform vars var c f opt) ;; e
-(while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+)\\)"
+(while (string-match "%\\(\\?\\)?\\([-+]?[0-9.]*\\)\\([ .;,:!?=|/<>]?\\)\\([cltseib]\\|(.+?)\\)"
 			 s start)
   (setq var (or (cdr (assoc (match-string 4 s)
 '(("c" . category) ("t" . time) ("l" . level) ("s" . extra)
-- 
2.26.2

Re: [PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Sorry, attached wrong version of the patch.

Here is the right one

>From 0acb961d916d4bfde505c9f5eb16d1e8851b6c8f Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Sat, 13 Mar 2021 13:16:06 +0800
Subject: [PATCH] Improve org-link-plain-re

* lisp/ol.el (org-link-make-regexps): Allow URLs with up to two levels
of nested brackets.  Now, URLs like [1] can be matched.  The new
regexp is taken from [2].

[1] https://doi.org/10.1016/0160-791x(79)90023-x
[2] https://daringfireball.net/2010/07/improved_regex_for_matching_urls
---
 lisp/ol.el | 55 +++---
 1 file changed, 36 insertions(+), 19 deletions(-)

diff --git a/lisp/ol.el b/lisp/ol.el
index 8b9755b51..7ce7a4798 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -829,26 +829,43 @@ This should be called after the variable `org-link-parameters' has changed."
 	  (format "<%s:\\([^>\n]*\\(?:\n[ \t]*[^> \t\n][^>\n]*\\)*\\)>"
 		  types-re)
 	  org-link-plain-re
-	  (concat
-	   "\\<" types-re ":"
-	   "\\([^][ \t\n()<>]+\\(?:([[:word:]0-9_]+)\\|\\([^[:punct:] \t\n]\\|/\\)\\)\\)")
-	  ;;	 "\\([^]\t\n\r<>() ]+[^]\t\n\r<>,.;() ]\\)")
-	  org-link-bracket-re
-	  (rx (seq "[["
-		   ;; URI part: match group 1.
-		   (group
-		(one-or-more
+	  (let ((non-space-bracket "[^][ \t\n()<>]+"))
+;; Heiristics for an URL link.  Source:
+;; https://daringfireball.net/2010/07/improved_regex_for_matching_urls
+(rx-to-string
+ `(seq (regexp "\\<")
+   (regexp ,types-re)
+   ":"
+   (1+ (or (regex ,non-space-bracket)
+   (seq "("
+(* (or (regex ,non-space-bracket)
+   (seq "("
+(regex ,non-space-bracket)
+")")))
+")")))
+   (or (seq "("
+(* (or (regex ,non-space-bracket)
+   (seq "("
+(regex ,non-space-bracket)
+")")))
+")")
+   (regexp "\\([^[:punct:] \t\n]\\|/\\)")
+  org-link-bracket-re
+  (rx (seq "[["
+	   ;; URI part: match group 1.
+	   (group
+	(one-or-more
  (or (not (any "[]\\"))
-			 (and "\\" (zero-or-more "") (any "[]"))
-			 (and (one-or-more "\\") (not (any "[]"))
-		   "]"
-		   ;; Description (optional): match group 2.
-		   (opt "[" (group (+? anything)) "]")
-		   "]"))
-	  org-link-any-re
-	  (concat "\\(" org-link-bracket-re "\\)\\|\\("
-		  org-link-angle-re "\\)\\|\\("
-		  org-link-plain-re "\\)"
+		 (and "\\" (zero-or-more "") (any "[]"))
+		 (and (one-or-more "\\") (not (any "[]"))
+	   "]"
+	   ;; Description (optional): match group 2.
+	   (opt "[" (group (+? anything)) "]")
+	   "]"))
+  org-link-any-re
+  (concat "\\(" org-link-bracket-re "\\)\\|\\("
+	  org-link-angle-re "\\)\\|\\("
+	  org-link-plain-re "\\)"
 
 (defun org-link-complete-file ( arg)
   "Create a file link using completion."
-- 
2.26.2

[PATCH] Re: Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Kyle Meyer  writes:

> This case was discussed at
> <https://orgmode.org/list/5faf0bd7-b114-9723-773e-7f3da1660...@grinta.net/T/#u>.
> There's mention of an improved regexp that might be worth trying.

The improved regexp appears to work for me. I tried my best to transform
the regexp provided in the link to rx notation. The patch is attached.

Best,
Ihor

>From 79eaa3b5b71f2aba02d4f17b860f67a6a77255fc Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Sat, 13 Mar 2021 13:16:06 +0800
Subject: [PATCH] Improve org-link-plain-re

* lisp/ol.el (org-link-make-regexps): Allow URLs with up to two levels
of nested brackets.  Now, URLs like [1] can be matched.  The new
regexp is taken from [2].

[1] https://doi.org/10.1016/0160-791x(79)90023-x
[2] https://daringfireball.net/2010/07/improved_regex_for_matching_urls
---
 lisp/ol.el | 30 +-
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/lisp/ol.el b/lisp/ol.el
index 8b9755b51..0e166de38 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -829,15 +829,27 @@ This should be called after the variable `org-link-parameters' has changed."
 	  (format "<%s:\\([^>\n]*\\(?:\n[ \t]*[^> \t\n][^>\n]*\\)*\\)>"
 		  types-re)
 	  org-link-plain-re
-	  (concat
-	   "\\<" types-re ":"
-	   "\\([^][ \t\n()<>]+\\(?:([[:word:]0-9_]+)\\|\\([^[:punct:] \t\n]\\|/\\)\\)\\)")
-	  ;;	 "\\([^]\t\n\r<>() ]+[^]\t\n\r<>,.;() ]\\)")
-	  org-link-bracket-re
-	  (rx (seq "[["
-		   ;; URI part: match group 1.
-		   (group
-		(one-or-more
+	  (let ((non-space-bracket "[^][ \t\n()<>]+"))
+;; Heiristics for an URL link.  Source:
+;; https://daringfireball.net/2010/07/improved_regex_for_matching_urls
+(rx-to-string
+ `(seq (regexp "\\<")
+   (regexp ,types-re)
+   ":"
+   (1+ (or (regex ,non-space-bracket)
+   (seq "("
+(* (or (regex ,non-space-bracket)
+   (seq "("
+(regex ,non-space-bracket)
+")")))
+")")))
+   (or (seq "("
+(* (or (regex ,non-space-bracket)
+   (seq "("
+(regex ,non-space-bracket)
+")")))
+")")
+   (regexp "\\([^[:punct:] \t\n]\\|/\\)")
  (or (not (any "[]\\"))
 			 (and "\\" (zero-or-more "") (any "[]"))
 			 (and (one-or-more "\\") (not (any "[]"))
-- 
2.26.2

Bug: Plain https links with brackets are not recognised [9.4.4 (release_9.4.4-625-g763c7a @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

I have a https link like "https://doi.org/10.1016/0160-791x(79)90023-x".
If I put this link into an org file (using emacs -Q or Org mode master),
only "https://doi.org/10.1016/0160-791x(79)" part of the link is treated
as a link. I guess, it should not happen.

Best,
Ihor

Re: archiving speed [was Re: Tips on maintaining history in Org Mode]

[Ihor Radchenko]

Samuel Wales  writes:

> that makes sense.  but why would appending to an archive as the result
> of bulk archiving lag?  if the problem is large archive files, which
> i'd bet is the case for a lot of users and not just me, then could org
> in principle be changed so that all it does is append?  thus not lag?
> like, build the entry in a temporary buffer?

Looking into the source code, the culprit appears to be the call to
org-show-all inside org-archive-subtree. The call to org-show-all on
master calls org-cycle-hide-drawers, which goes through every single
drawer (= every archived heading in archive file) and folds it manually
(just to unfold it later). Do not ask.

> as i see it, having more than one archive file per org file is good
> for speed, but doesn't work in existing org, because iirc e.g. v A in
> the agenda goes org agenda file -> corresponding archive file and will
> miss the archive files that do not have a corresponding org file with
> exactly the same basename sans extension.

The code from my config takes care about this. v A will work. I plan to
suggest it for master in future, but I am focused on the org-fold branch
for now. If you wish, feel free to propose a patch based on this code.

>> 3. (untested) Put #+STARTUP: showeverything at the beginning of the
>>archives, so that nothing is going to be folded
>
> good idea.  my included-by-agenda archive files do seem to be in
> showeveryting mode already for some reason.  but perhaps not when bulk
> archiving.
>
> would it be a silly idea for an fr that org make this an option for
> bulk archiving?  hmm or for archive files in general?

Upon reviewing the source of org-archive-subtree, this should not be
needed. Actually, org-mode already disables startup visibility and
various hooks when opening archives. As you can see, it is not
sufficient, since org-archive shoots its own leg later by changing
visibility every time you archive a subtree.

>> Sorry, the config is actually not yet formatted for public use. You can
>> search for the code block containing "defun org-archive--compute-location".
>
> firefox find does not seem to find it.

This is odd. All I can suggest then is cloning the repo and searching
using Emacs. Emacs is more reliable when opening org files ;)

git clone https://github.com/yantar92/emacs-config
emacs emacs-config/config.org

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales  writes:

> Hi Ihor,
>
> it never occurred to me that bulk archiving could be sped up by
> changing anything about the archive files.  i assumed that archiving
> worked by appending to those files, so once the initial find-file was
> performed, there should be no additional slowness.

Disregarding the initial find-file (which may also be very slow on large
archives), the slowness is happening because many org-mode commands are
using line-motion functions that respect visibility. Those commands
become slow when there are many folded elements (see [1] for technical
details why). So, many org commands tend to lag on large archives.

The lags can be solved in several ways:
1. Reduce the archive file size
2. Use optimised folding mechanism [1] (this will speed up org-mode in
general as well)
3. (untested) Put #+STARTUP: showeverything at the beginning of the
   archives, so that nothing is going to be folded

> i will keep in mind disabling font lock in archive files.  any
> suggested code for that?

Note that it will mostly affect find-file performance. To disable
font-lock, you will need to customise font-lock-global-modes, so that it
does not auto-start in org-mode. The value should be something like
'(not org-mode), but check the current value - you may have other
customisation in place.

Then, you will need to have a custom org-mode-hook that activates
font-lock only when the org file is not the archive file. Something like

(defun yant/font-lock-activate-maybe ()
"Activate font-lock in non-archive org buffers."
(when (org-all-archive-files) (font-lock-mode +1)))
(add-hook 'org-mode-hook #'yant/font-lock-activate-maybe)

> your config is impressively [and for me
> dauntingly] comprehensive.  the Archiving anchor does not seem to have
> a referent.  I searched for archiving, but didn't seem to find what
> you were probably pointing me to.

Sorry, the config is actually not yet formatted for public use. You can
search for the code block containing "defun org-archive--compute-location". 
You will need that code block and the following code block.

[1] https://github.com/yantar92/org

Best,
Ihor

Re: Set archive location relative to property

[Ihor Radchenko]

Florian Lindner  writes:
> Ok, I expected that's the way to go. Unfortunately, I wasn't able to 
> find how to set a custom archive function. Could you give me a pointer?

You will need to advice org-archive--compute-location

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales  writes:
>   [currently the archiver is so slow i can't use it]

Are your existing archives very big (few Mbs)? If so, you may try to
speed up the archiving using feature/org-fold branch [1]. If that is not
enough, I recommend splitting archives on yearly basis [2] or disabling
font-lock in archive files.

Best,
Ihor

[1] https://github.com/yantar92/org
[2] https://github.com/yantar92/emacs-config/blob/master/config.org#archiving

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales  writes:

> perhaps also changing org-archive-file-header-format to allow a format
> thingie for a timestamp would allow you to take parts of an archive
> file and move them into one per year without having to put the date in
> each archived entry.

FYI: I have implemented automatic per-year archiving, which is correctly
handled by other org commands in my personal config:
https://github.com/yantar92/emacs-config/blob/master/config.org#archiving

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

David Masterson  writes:

> Interesting, but then how do you get the list?  I mean is there an
> agenda to use?

Generally yes, you can use agenda. Or you can use sparse tree (more manual).
For agenda, if you customise org-log-done, you can use
org-agenda-log-mode ("v l" or "v L" in an agenda buffer). You can add
archived items as well with "v a" or/and "v A".

Just org-agenda-log-mode will show everything, not just calls. Narrowing
to calls only will depend on how you define a todo, which is a call.

If you use something like PHONE or CALL todo keywords, it might be
slightly tricky. You will need to customise org-todo-keywords, so that
your CALL->DONE changes are recorded (see the org-todo-keywords
docstring). You will also need to filter displayed items in agenda by
regexp involving the keyword you use to define the call.

An easier way could be marking your calls with a tag. Then, you can
filter your org-agenda-log by that tag to show only calls.

Hope it helps.

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

David Masterson  writes:
> My question is, if you have meetings/phone calls as TODOs, what is the
> preferred way to handle when they move into history so that, *much*
> later, you can easily produce a list of all of the meetings/phone calls
> with dates and times of them?  The issue (I think) is, when you mark the
> TODO as DONE, you lose the info of what the TODO was originally.

See Org manual :: 5.3 Progress Logging

Re: Source Block header arguments

[Ihor Radchenko]

Russell Adams  writes:

> I know I can easily create templates for frequently used code blocks.
>
> My question is, is there a completion or in-place documentation for
> valid header arguments to code blocks? The options are rather buried
> in the manual.

I use helm-info-org for this.

Best,
Ihor

Re: encryption problems using org-mode

[Ihor Radchenko]

Colin Baxter  writes:

> Odd. The problem has gone away and all is now well. I am tracking the
> master branch of emacs-28; however, I thought I had checked and got the
> same error using emacs-27.1.

This is because .elc files are in .gitignore. Checking out other branch
would leave all the compiled .elc files in place and they have a loading
priority, thus you effectively did nothing when checked out to other
branch without running make bootstrap or make clean.

Best,
Ihor

Re: encryption problems using org-mode

[Ihor Radchenko]

Colin Baxter  writes:

> Debugger entered--Lisp error: (void-variable minor-modes)

This looks like master Emacs issue. Did you happen to update Emacs to
master as well? There was a recent commit related to minor-modes
variable. You can search for "0bd846c 1/2: Rename minor-modes to
local-minor-modes" in emacs-devel. I suspect that you may need to
re-compile local and built-in Emacs packages.

Best,
Ihor

Re: Content below current line lost

[Ihor Radchenko]

Dmitry Knyaginin  writes:

> By the way, I have started to test my theory that the bug has to do with
> the file size, but so far I have not been able to reproduce the bug.

Another guess about your case: I have experienced deleted parts of
buffer and lost history when using undo-tree. Do you happen to use it?

Best,
Ihor

Re: Is it possible to optimize Org Mode org-activate-links ?

[Ihor Radchenko]

Christopher Miles  writes:
> Should I use the following options? I saw the warning. Does this freeze 
> happens often? I decide to try it.

Setting org-element-use-cache to t should be enough to see differences.

Best,
Ihor

Re: Is it possible to optimize Org Mode org-activate-links ?

[Ihor Radchenko]

Christopher Miles  writes:
> I checked org-element-context source code, it's not so long and complex. Why 
> it caused so many items in Memory profiler result? Is it possible to optimize 
> it?

You can try to use org-element-cache. That might help.

Best,
Ihor

Re: Typing latency

[Ihor Radchenko]

Maxim Nikulin  writes:

> It is not namely typing latency, but I have noticed lags while moving 
> over collapsed headings with "up" key (with "down" it is not so 
> apparent) e.g. in overview view. It has happened after linux upgrade, 
> emacs version changed from 25.2 to 25.3, system package with org mode 
> updated as well. The org file has significant size: 50k lines, 2Mb. I 
> have created a LXC container to compare performance with older emacs. It 
> is quite strange. With org version from git, emacs version is not really 
> important, flyspell mode is irrelevant.

You can try feature/org-fold branch aiming to address issues with
performance on large files https://github.com/yantar92/org

Best,
Ihor

[PATCH] Allow tags containing capital letters in org-agenda-filter

[Ihor Radchenko]

Hi,

I recently noticed that org-agenda-filter does not match tags with
capital letters because all the stored tags in agenda are downcased. The
attached patch is fixing the issue. Though, ideally, it would be better
if agenda filter were case-sensitive for tags.

Best,
Ihor
>From 9d7a966497458bdb0ab5e5171d2bab1fa8612bc5 Mon Sep 17 00:00:00 2001
From: Ihor Radchenko 
Date: Thu, 11 Feb 2021 12:03:15 +0800
Subject: [PATCH] Allow tags containing capital letters in org-agenda-filter

* lisp/org-agenda.el (org-agenda-filter): Downcase tags in the search
string provided by user.  This is needed because all the tags stored
in 'tags text property are downcased.

Example when old code did not work is a tag like COMMON.  The user
would not expect a need to input +|-common in the agenda filter
instead of +|-COMMON.  The latter would only result in
"COMMON filter ignored because tag/category is not represented".
---
 lisp/org-agenda.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index 90920ef41..0845d0ca6 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -7767,8 +7767,8 @@ the variable `org-agenda-auto-exclude-function'."
 	  (setq s (replace-regexp-in-string ; Remove the temporary special string.
 		   "~~~" "-" (match-string 3 f-string)))
 	  (cond
-	   ((member s tag-list)
-	(add-to-list 'ft (concat pm s) 'append 'equal))
+	   ((member (downcase s) tag-list)
+	(add-to-list 'ft (concat pm (downcase s)) 'append 'equal))
 	   ((member s category-list)
 	(add-to-list 'fc (concat pm ; Remove temporary double quotes.
  (replace-regexp-in-string "\"\\(.*\\)\"" "\\1" s))
-- 
2.26.2

Re: Typing latency

[Ihor Radchenko]

Timothy  writes:
>> (setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")
>
> If this doesn't produce regressions, is this worth patching into Org?

I believe so, but I am not sure about regressions. Current master allow
placing priority cookies anywhere, while my version only limits priority
to headline.

Best,
Ihor

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel  writes:

> the delay, which must be caused by the already fontified parts.
> I guess the issue must be wih emacs' internals.

I had some issues with performance when displaying unicode symbols in
the past for certain (large) fonts. That time, the following helped to
improve Emacs performance drastically:

(setq inhibit-compacting-font-caches t)

That is the only thing I can think about in regard to the Emacs display
engine slowness.

Best,
Ihor

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel  writes:

> It reports that most of the time is spent in org-do-latex-and-related, 
> and some 20% in something related to the priority faces (despite the 
> lack of priority cookies).

I also had issues with priority faces. It is related to sub-optimal
regexp used to detect priority cookies.

I have the following in my config to speed things up:

(setq org-priority-regexp "^\\*+.*\\(\\[#\\([A-Z0-9]+\\)\\] ?\\)")

For the latex fontification, I never had issues, but you might play with
org-highlight-latex-and-related.

Hope it helps.

Best,
Ihor

Re: Typing latency

[Ihor Radchenko]

Sébastien Miquel  writes:

> AFAIU, the elisp profiler is useless in this case (fontification being
> slow). At least my runs don't seem to report anything relevant.

You can try font-lock-profiler package
(https://github.com/Lindydancer/font-lock-profiler)

Best,
Ihor

Re: org-contacts problem

[Ihor Radchenko]

Raoul Comninos  writes:
> Can someone perhaps guide me?

You need to tell Emacs to load org-contacts:

(require 'org-contacts)

Best,
Ihor

 

Re: org-attach-git don't automatically commit changes

[Ihor Radchenko]

Juan Manuel Macías  writes:

> Do you think a possible patch could simply consist of replacing (as I
> have done) `(expand-file-name org-attach-id-dir)' by `(org-attach-dir)'
> in `org-attach-git-commit'? This would allow you to handle attachment
> dirs as individual git repos, regardless of :ID: or :DIR: properties,
> since "[(org-attach-dir)] Return the directory associated with the
> current outline node. First check for DIR property, then ID property
> [...]".

All the instances of (expand-file-name org-attach-id-dir) should be
replaced. There are 3. That's not a big deal.

What is more tricky is making sure that workflow for people using the
old behaviour is not broken. Just changing to (org-attach-dir) will
break existing git repos in org-attach-id-dir. This should also not be
too hard (say, we can introduce a custom variable defaulting to old
behaviour), but exact details may need to be discussed.

In any case, if you provide a patch, it will encourage the org
maintainers to join this discussion and proceed with changes.

Best,
Ihor

Re: org-attach-git don't automatically commit changes

[Ihor Radchenko]

Juan Manuel Macías  writes:

> But if I evaluate this, I get an 'incomplete' path:

I think you misunderstood how org-attach-git works.

org-attach-git is:

;; An extension to org-attach.  If `org-attach-id-dir' is initialized
;; as a Git repository, then org-attach-git will automatically commit
;; changes when it sees them.  Requires git-annex.

So, it is not designed to work when your actual attachment directory is
a git repo, but rather when org-attach-id-dir is a git repo (initialised
manually).

The other thing is that `org-attach-id-dir' makes much less sense from
the time :DIR: property was introduced. So, I believe that
org-attach-git should be updated. Probably, handling attachment dirs as
individual git repos can be one of the ways to upgrade the current
version. I guess, patches welcome.

Best,
Ihor

Re: org-goto and org-store-link/org-id-get-create

[Ihor Radchenko]

Peter Klenner  writes:

> Calling either org-store-link or org-id-get-create in an indirect
> org-goto buffer results in an empty ID-property drawer with (setq
> org-id-link-to-org-use-id t).

Confirmed.

This is because org-goto buffer is in read-only state, which is ignored
when inserting empty property drawer, but respected when trying to
insert :ID: property, which is indeed inconsistent.

Currently, the following functions disregard the read-only state in org
buffers: org-store-log-note, org-insert-property-drawer,
org-agenda-undo, org-agenda-todo, org-agenda-add-note,
org-agenda-priority, org-agenda-set-tags, org-agenda-set-property,
org-agenda-set-effort, org-agenda-toggle-archive-tag,
and org-columns-store-format.

A fix to this particular issue could be using org-no-read-only in
org-entry-put. Though more functions may suffer from similar issues in
read-only org buffers.

Best,
Ihor

Re: org-attach-git don't automatically commit changes

[Ihor Radchenko]

Juan Manuel Macías  writes:

> The default value of `org-attach-id-dir' is "data/", and if I evaluate
> `(expand-file-name org-attach-id-dir)' on my current node, it returns a
> wrong path to the attached folder. `org-attach-sync' only works for me
> if I set in `org-attach-git-commit' the variable like this:

Does it mean that your attachment folder is set in :DIR: property?

> #+begin_src emacs-lisp
> ;; ...
>   (let* ((dir (expand-file-name org-attach-id-dir))
> ;; ...
> #+end_src

I suspect that it is a leftover from the major changes in org-attach
when :DIR: property was introduced. The org-attach-git presumes that all
the attachments in current file are stored in sub-directories located
inside org-attach-id-dir, which is no longer guaranteed. In fact, the
existing approach to treat all the attachments to all headings in
current file as files in a single git repo cannot be used. I can see two
possible fixes:
1. Treat each attachment dir as individual git repo (breaking change for
   those who are using :ID: property to build the attachment dirs)
2. Treat attachment dirs defined by :DIR: property individually and
   leave the :ID:-defined attachments as they were treated before
   (inconsistent).

I am in favour of the first approach since I do not like the idea of
keeping all the attachments in the whole file in a single git repo.
I think feedback from other is needed to decide what we need to do here.

P.S. Marking this as a bug.

Best,
Ihor

Re: tickler file & recurring events

[Ihor Radchenko]

Saša Janiška  writes:

> so wonder if
> it's possible for such tasks to be added to agenda view **only**  when
> they're due and not before?

Check org-agenda-show-future-repeats
( v org-agenda-show-future-repeats ).

Best,
Ihor

Re: org-attach-git don't automatically commit changes

[Ihor Radchenko]

Juan Manuel Macías  writes:

> I don't see that org-attach "automatically commit changes when it sees
> them". Only when I run in the attached directory `M-:
> (org-attach-git-commit) RET' it works fine.

Note that org "sees" changes in the attachment dir only when you use
M-x org-attach command to manage the attachments. Directly changing the
attachment directory will not be noticed. You can force org-mode to
check for changes in attachment dir by running "C-c C-a z".

I guess using something like inotify might be a good new feature to
make org detect changes automatically.

Best,
Ihor

Re: [PATCH] Org Agenda Support Argument Collection for Custom Bulk Functions (was: Custom Bulk Functions With Prompt)

[Ihor Radchenko]

Kevin Foley  writes:

> Attached patch should allow user to specify a function to collect
> arguments when calling a custom bulk function such that those arguments
> are only collected once and used for each entry.

Looks good for me. Maybe you can also add an entry to ORG-NEWS.

Marking this thread as a patch for maintainer review.

Best,
Ihor

Re: Bug: [enhancement] "Insert Link" (C-c C-l) to propose completion (dired-listing?) of local files when link=local-file [9.1.9 (release_9.1.9-65-g5e4542 @ /usr/share/emacs/26.3/lisp/org/)]

[Ihor Radchenko]

Guillaume MULLER  writes:

> This is a simple enhancement proposition for C-c C-l (insert link): when
> user starts the link with ./ or / or file: , it would be great if org-mode 
> could
> propose completions based on some local-files listing (dired?)

org-mode does it already. Try C-c C-l file:  

Best,
Ihor

Re: logrepeat only logging LAST_REPEAT

[Ihor Radchenko]

Michael Heerdegen  writes:

> Do you think others may want the same and it could be worth to implement
> this feature?

It should be not too hard to implement. For example, one can add another
possible value of org-log-repeat. Patches welcome ;)

Best,
Ihor

Re: logrepeat only logging LAST_REPEAT

[Ihor Radchenko]

Michael Heerdegen  writes:

> As far as I understand, the logging is implemented in
> `org-auto-repeat-maybe', and the thing I do not want is caused by the
> `org-add-log-setup' call, and that is done unconditionally unless
> `org-log-repeat's value is nil.  Which I don't want because this would
> also disable the "LAST_REPEAT" property setting.

You are right. Since this is hard-coded, you may have to use :around
advice to disable org-log-setup:

(cl-letf (((symbol-function org-log-setup) (lambda ( _) nil))) )

Best,
Ihor

Re: Custom Bulk Functions With Prompt

[Ihor Radchenko]

Kevin Foley  writes:

> '((?R (set-category set-category-args))
>   (?C bulk-cut))
>
> I mostly went with this because I'm not very familiar with `pcase'.
> I've confirmed it works as expected but let me know if you still think
> another approach is better.

I understand now. I thought you wanted to use different format:
(?R set-category set-category-args)

Then, your code looks reasonable. Just need to document the new custom
bulk action format.

Best,
Ihor

Re: logrepeat only logging LAST_REPEAT

[Ihor Radchenko]

Michael Heerdegen  writes:

> Is there a way to get only the :LAST_REPEAT: prop logged, without the
> ever-growing list?

I guess you can try to play with org-log-done and org-log-repeat variables.

Best,
Ihor

Re: Region-based meta-notes

[Ihor Radchenko]

Lawrence Bottorff  writes:

> So yes, simply being able to select regions and make side notes about them
> could give you a very fine level of control over metadata in a file. Is
> there such a thing in the Emacs/org-mode world?

I use inlinetasks for this. You can add notes to inlinetask just as you
add notes to a normal headline. Also, inlinetasks can be created around
active region:

Signature
(org-inlinetask-insert-task  NO-STATE)

Documentation
Insert an inline task.

If prefix arg NO-STATE is set, ignore org-inlinetask-default-state.
If there is a region wrap it inside the inline task.

Key Bindings
org-mode-map C-c C-x t

Best,
Ihor

Re: Why are (UU)IDs limited to the headline level?

[Ihor Radchenko]

"Dr. Arne Babenhauserheide"  writes:

> You can use <> to make it possible to link to any
> place in a document.

It does not work globally though. One cannot just put
[[id:some-anchor-name]] link in other file.

Best,
Ihor

Re: Custom Bulk Functions With Prompt

[Ihor Radchenko]

Kevin Foley  writes:

> +(`(,_ ,f)
> +(when (listp f)
> +  (let ((args (funcall (nth 1 f)))
> +(func (nth 0 f)))
> +(setq f (apply #'apply-partially func args
> +(setq cmd f) (setq redo-at-end t))

That will not work. pcase will not even match list of 3+ elements using
`(,_ ,f):

(pcase '(a b c)
  (`(,_ ,f)
   (if (listp f)
   "List!"
 "Not list!"))
  (_ "No match!")) => "No match!"

You need to add extra matcher `(,_ ,argf ,f).

Best,
Ihor

Re: Feature request

[Ihor Radchenko]

Raoul Comninos  writes:

> With your permission can I post your solution to stack-exchange,
> crediting you?

Sure. The public message URL is
https://orgmode.org/list/87sg70vsvy.fsf@localhost/

Best,
Ihor

Re: Feature request

[Ihor Radchenko]

Raoul Comninos  writes:

> It works now and its awesome! I cannot believe that they had this
> feature and removed it. I am a very happy man and I cannot thank you enough.

This feature creates a lot of junk text when attachment folder gets very
large. For example, I keep my travel photos as attachments. There can be
hundreds of photos in a single attachment folder. Showing all the photo
names in a property would be not very useful.

On the other hand, if more people are interested, this feature can be
resurrected as a user-option (disabled by default).

Best,
Ihor

Re: Custom Bulk Functions With Prompt

[Ihor Radchenko]

Kevin Foley  writes:

> I took a look this morning and came up with the attached patch.
> Essentially if the custom function is a list then the first element is
> the bulk function and the second is the argument function.
>
> If that looks reasonable I can add some documentation and submit a
> proper patch.
>
> Kevin Foley

That sounds like a reasonable option.

Note that attachment in the previous email appears to be empty. Can you
resend? I will take a look then.

Best,
Ihor

Re: [feature request] A new cookie type [!] showing the last note taken

[Ihor Radchenko]

Christopher Miles  writes:

> I think this is possible through read last note text in logbook, then display 
> it
> in headline through text-property overlay.
>
> That's what I did in this package 
> [[https://github.com/stardiviner/org-link-beautify]]
>
> Should not be that hard to implement it.

Thanks for the suggestion. However, I think that it would be more
consistent to keep the same approach with other cookie types in org.
They do not use overlays, but change text directly.

Best,
Ihor

Re: Feature request

[Ihor Radchenko]

Raoul Comninos  writes:

> I have copied the code to my dot Emacs, but now when I try to add an 
> attachment now, it generates this error:
>
> run-hook-with-args: Wrong number of arguments: (lambda nil "Save list of
> attachments to ORG_ATTACH_FILES property." (when-let* ((dir
> (org-attach-dir)) (files (org-attach-file-list dir))) (org-set-property
> "ORG_ATTACH_FILES" (mapconcat #'identity files ", ", 1
>
> Can you help me with this?

You can try:

(defun org-attach-save-file-list-to-property (dir)
  "Save list of attachments to ORG_ATTACH_FILES property."
  (when-let* ((files (org-attach-file-list dir)))
(org-set-property "ORG_ATTACH_FILES" (mapconcat #'identity files ", "
(add-hook 'org-attach-after-change-hook #'org-attach-save-file-list-to-property)

Also, note that the list of files will only be updated if you
attach/delete files calling org-attach. If you change the attachment
folder manually, you will need to run M-x org-attach-sync.

Best,
Ihor

Re: Feature request

[Ihor Radchenko]

I think something like the following will do (untested):

(defun org-attach-save-file-list-to-property ()
  "Save list of attachments to ORG_ATTACH_FILES property."
  (when-let* ((dir (org-attach-dir))
  (files (org-attach-file-list dir)))
(org-set-property "ORG_ATTACH_FILES" (mapconcat #'identity files ", "
(add-hook 'org-attach-after-change-hook #'org-attach-save-file-list-to-property)

Best,
Ihor

Re: Feature request

[Ihor Radchenko]

Raoul Comninos  writes:

>> Listing all the attached files used to be the built-in, but it was
>> removed a few years ago.
> Oh, that's a pity. Thanks for responding. 

You can still implement it on your side for personal use. There is
org-attach-after-change-hook where you can put a custom function saving
the attached file list (org-attach-file-list) into a property.

Best,
Ihor

Re: Custom Bulk Functions With Prompt

[Ihor Radchenko]

Kevin Foley  writes:

> I'd like to setup a custom bulk function using
> `org-agenda-bulk-custom-functions', however I'd like to receive one
> prompt when I select the action and then have the action performed on
> every entry without being prompted again.

Generally, you can set a variable indicating if your custom function is
invoked first time during the bulk action. This variable can be set to
nil in :before advice for org-agenda-bulk-action. Then, you can check
the variable value in your custom function. If it is nil, it is the
first invocation and you run the interactive version then setting the
variable to 't. The variable will have 't value in all the following
invocations and your function can run non-interactively.

Instead of advice, you can also provide a simple patch implementing the
described functionality in org-agenda-bulk-action. I do support adding
this functionality to org.

Best,
Ihor