[PATCH] ox-latex: Allow percent sign in 'src-block' caption

[Maxim Nikulin]

On 03/09/2021 19:17, Maxim Nikulin wrote:
> On 30/07/2021 22:00, Charest, Luc wrote:
>>
>> org-latex-src-block: Not enough arguments for format string
> 
> Confirmed
> 
> User input is combined with format string in `org-latex-src-block'
> so percent character in caption is treated as a format specifier.> Maybe 
> emacs versions newer than 26 does not throw an error
> on extra specifier with missed argument. =:latex-cation-above nil=
> might still cause corrupted LaTeX code.

I have decided that it is better to provide a partial fix than to leave
the problem as is.  This patch covers only plain verbatim environment,
more advanced formatting using listings or minted should be addressed
by another change (Anybody?).  Do not close the bug after applying
this patch.
>From 5e6930e3c3d30f13e6e33a6fc504b445cacc579d Mon Sep 17 00:00:00 2001
In-Reply-To: 
References: 
From: Max Nikulin 
Date: Tue, 7 Sep 2021 19:01:11 +0700
Subject: [PATCH] ox-latex: Allow percent sign in 'src-block' caption

* lisp/ox-latex.el (org-latex-src-block): Prevent leak of percent sign
from caption to `format' first argument causing export failure due to
insufficient argument number.  The fix covers only the case of verbatim
environment.
* testing/lisp/test-ox.el
(test-org-export/latex-src-block-verbatim-caption): New test for src
block LaTeX export with cases of various formatting of caption and
verbatim environment.  Check that percent signs in caption and source
block body do not lead to errors.

Reported-by: Charest, Luc 
<https://lists.gnu.org/archive/html/emacs-orgmode/2021-08/msg00017.html>
---
 lisp/ox-latex.el| 30 ++---
 testing/lisp/test-ox.el | 97 +
 2 files changed, 111 insertions(+), 16 deletions(-)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 99076f8b9..b325b8892 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -2994,22 +2994,20 @@ contextual information."
   (cond
;; Case 1.  No source fontification.
((or (not lang) (not listings))
-	(let* ((caption-str (org-latex--caption/label-string src-block info))
-	   (float-env
-		(cond ((string= "multicolumn" float)
-		   (format "\\begin{figure*}[%s]\n%s%%s\n%s\\end{figure*}"
-			   (plist-get info :latex-default-figure-position)
-			   (if caption-above-p caption-str "")
-			   (if caption-above-p "" caption-str)))
-		  (caption (concat
-(if caption-above-p caption-str "")
-"%s"
-(if caption-above-p "" (concat "\n" caption-str
-		  (t "%s"
-	  (format
-	   float-env
-	   (concat (format "\\begin{verbatim}\n%s\\end{verbatim}"
-			   (org-export-format-code-default src-block info))
+	(let ((caption-str (org-latex--caption/label-string src-block info))
+  (verbatim (format "\\begin{verbatim}\n%s\\end{verbatim}"
+(org-export-format-code-default src-block info
+  (cond ((string= "multicolumn" float)
+ (format "\\begin{figure*}[%s]\n%s%s\n%s\\end{figure*}"
+ (plist-get info :latex-default-figure-position)
+ (if caption-above-p caption-str "")
+ verbatim
+ (if caption-above-p "" caption-str)))
+(caption (concat
+  (if caption-above-p caption-str "")
+  verbatim
+  (if caption-above-p "" (concat "\n" caption-str
+(t verbatim
;; Case 2.  Custom environment.
(custom-env
 	(let ((caption-str (org-latex--caption/label-string src-block info))
diff --git a/testing/lisp/test-ox.el b/testing/lisp/test-ox.el
index 3f39645af..54558041b 100644
--- a/testing/lisp/test-ox.el
+++ b/testing/lisp/test-ox.el
@@ -3907,6 +3907,103 @@ Another text. (ref:text)
 	  (org-export-format-code-default
 	   (org-element-map tree 'src-block #'identity info t) info))
 
+(ert-deftest test-org-export/latex-src-block-verbatim-caption ()
+  "Test `org-latex-src-block' caption for verbatim environment.
+Check that percent sign does not become a part of format.
+This test does not cover listings and custom environments."
+  (let ((export
+	 (lambda (buffer-text)
+	   (org-test-with-parsed-data
+	   buffer-text
+	 (let* ((backend (org-export-get-backend 'latex))
+		(info (org-combine-plists
+			   (org-export--get-export-attributes backend)
+			   (org-export-get-environment backend)))
+		(result (org-latex-src-block
+			 (org-element-map tree 'src-block #'identity info t)
+			 t info)))
+	   ;; Remove properties to make failure reports more clear.
+	   (set-text-properties 0 (length result) nil 

Re: Bug: PDF Export of Link fails (emphasis ends inside link target)

[Maxim Nikulin]

On 03/09/2021 21:33, Timothy wrote:


Perhaps a “fix” could be auto-escaping problematic characters in urls when
entering links via `org-insert-link'.

[/query/] 


Just to be clear: while comma may be percent-escaped (`org-lint' perhaps 
would not be happy), "/?" are interpreted by HTTP server, so such 
characters must be preserved.


`org-insert-link' still may detect emphasis markers around the point and 
add extra marks. Unfortunately heuristics to detect start marker only 
(while typing text when no end marker yet) will be prone to false positives.

Re: Bug: PDF Export of Link fails (emphasis ends inside link target)

[Maxim Nikulin]

On 03/09/2021 12:17, Dr. Arne Babenhauserheide wrote:


Using the following text I get failing pdf export: The link is not
recognized correctly but ends after the comma and the markup persists in
the PDF.

/Diesen Text habe ich leicht abgewandelt schon am Montag per E-Mail
ans [[https://km-bw.de/,Lde/startseite/service/kontakt][Kultusministerium BW]] 
geschickt. Da hatten wir noch 2 Wochen./


It was discussed earlier.

On Tue, 20 Apr 2021 22:37:31 +0200, Nicolas Goaziou wrote
Re: [Patch] to correctly sort the items with emphasis marks in a list
https://orgmode.org/list/874kg0ae0k@nicolasgoaziou.fr/ :

Maxim Nikulin writes:


Surprisingly there are still cases when the old approach works better:

(let ((s (org-sort-remove-invisible
"A /wrapping [[https://orgmode.org/?a=b=d#e][link]] emphasis/")))
  (set-text-properties 0 (length s) nil s)
  s)
"A wrapping [[https://orgmode.org?a=b=d#e][link]] emphasis/"

I expect "A wrapping link emphasis".


Yet, your expectations are wrong. There is no link in the text above.
Emphasized text starts at "/wrapping" and ends before "/?".

Granted, this is a situation where the Org syntax is not very intuitive.
Anyway, the new function is more accurate.

Maybe we should require a space after punctuation following emphasized
text. I don't know. This is orthogonal to the current discussion.


In your particular case it is possible to use "%2C" instead of comma:

/Link 
[[https://km-bw.de/%2CLde/startseite/service/kontakt][kontakt]] link/


As a more general approach, additional emphasis marks can be added 
around link borders


/Test/ [[https://orgmode.org/?q=query][/query/]] /test/

While I believe that during parsing of link target lookup for end of 
emphasis should be suppressed, I am not sure that it is feasible with 
current implementation. That is why I consider such behavior as a price 
for lightweight markup and its current parser.

Re: Bug: Percentage in caption (even escaped) does not work in LaTeX export

[Maxim Nikulin]

On 30/07/2021 22:00, Charest, Luc wrote:

    I simplified the problem down to :

#+CAPTION: Org Mode works 99.99\% of the time.
#+BEGIN_SRC -n
// this is only a proof of concept
#+END_SRC

As soon as I put the percentage sign in the caption, the LaTeX export 
backend fails with this message :


org-latex-src-block: Not enough arguments for format string


Confirmed

User input is combined with format string in `org-latex-src-block' so 
percent character in caption is treated as a format specifier. Maybe 
emacs versions newer than 26 does not throw an error on extra specifier 
with missed argument. =:latex-cation-above nil= might still cause 
corrupted LaTeX code.


Code of `org-latex-src-block' is quite convoluted to provide a quick 
fix. Besides particular reported case it is better to check variants 
with listings and minted packages, custom environment.


Absence of language and line numbering option are irrelevant. Backslash 
before percent likely should not be used in Org files. My minimal example


#+caption: User input "%S" is combined with format string
#+begin_src emacs-lisp
  t
#+end_src

P.S. "Not enough arguments for format string" error usually means that a 
string provided by user is passed as format or incorrect number of 
arguments is specified for a format-like function. The former case is a 
quite frequent mistake:

- https://orgmode.org/list/87bl5tzof2@posteo.net/,
- 
https://orgmode.org/list/caj51etqzm+8rwcomymh2mexe8si0nzn2wq3k55n041-raxh...@mail.gmail.com/ 
(vs. correct variant 
https://emacs.stackexchange.com/questions/38276/get-description-of-an-org-mode-link).


Use

(defun my-safe-fun (msg)
 (message "%s" msg))

not just

(defun not-a-fun (msg)
 (message msg))

Re: Bug: "DEFINITION NOT FOUND" for footnote in Org manual

[Maxim Nikulin]

On 21/07/2021 22:26, Timothy wrote:

Maxim Nikulin writes:


It seems, the footnote was unintentionally lost in the merge commit with a lot
of conflicts due to renumbering in both branches:

https://code.orgmode.org/bzg/org-mode/commit/9410fbe0606a53b0582cc175b2cc525d556b23fe#diff-6498bd2213e2e527d904942bb214768a1913776L21749


Many thanks for tracking that down! I'm inclined to just add back the
text lost in that merge unless you'd recommend anything else.


I think, it is better to restore the footnote text than to leave it in 
its current state "DEFINITION NOT FOUND".

Re: Org table: numeric locale

[Maxim Nikulin]

On 28/08/2021 14:07, Jarmo Hurri wrote:


I wonder if it is possible to get org table formulas to respect numeric
locale.


Even to add some functions to Emacs that respect numeric locales, it is 
necessary to use e.g. dynamic modules (warning: do not try to change 
global locale by setlocale(3), almost certainly something in Emacs will 
be broken, a library that allows local locale objects is strongly 
preferred).

https://www.gnu.org/software/emacs/manual/html_node/elisp/Writing-Dynamic-Modules.html


Emacs documentation says that Emacs uses the value of LC_NUMERIC:

https://www.gnu.org/software/emacs/manual/html_node/emacs/General-Variables.html


It is a confusing statement, see:
- https://debbugs.gnu.org/cgi/bugreport.cgi?bug=29645
  #29645 Feature Request: Locale aware formatting
  Date: Sun, 10 Dec 2017 15:05:02 UTC
- https://lists.gnu.org/archive/html/emacs-devel/2021-06/msg00327.html
  https://lists.gnu.org/archive/html/emacs-devel/2021-06/msg00139.html

The state of numeric locales in Emacs is a little worse than just 
unsupported. Mostly "C" is forced for LC_NUMERIC, so e.g. grouping is 
not used. However if your locale uses comma "," as decimal separator,

you may be confused by calc results:

M-x calc RET dg RET 123456
123,456

Re: ob-lilypond.el doesn't generate midi files

[Maxim Nikulin]

On 28/08/2021 02:54, Nick Dokos wrote:


Also check if `org-babel-lilypond-arrange-mode' is t or nil: behavior
is very different depending on that.

Final thought: `xdg-open' starts whatever app it is supposed to run
in the background and returns immediately. That may cause problems with
babel.


To be precise, xdg-open is merely a wrapper. It does not decide whether 
a viewer should be started in background or in foreground, it just 
delegates decision concerning particular application and a way to start 
it to desktop environment (if any is running). Gnome's gio open and 
kde-open5 launch appropriate viewer in background. In my opinion they 
are free to do so.


It is a decade-old problem that emacs kills such processes. I disagree 
that it is e.g. Gnome bug.

https://lists.gnu.org/archive/html/emacs-devel/2009-07/msg00279.html

ob-lilypond.el suffers from this problem in its original form, and it 
should be fixed.


Approach taken in Org mode for compatibility reasons:
https://code.orgmode.org/bzg/org-mode/commit/869b7a21b94ed112f6640c8f2711c2a68b661dea
Emacs mailcap:
http://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/net/mailcap.el#n1180

Pipe process instead of pty one is not a really robust way. The sour 
truth is that emacs API does allow reliable "shoot and forget" approach 
for external applications. Currently viewers might make Emacs CPU hungry 
for no reason (unsure if applications with such behavior exist in real 
world however). Eli Zaretskii suspects some incompatibility due to 
switching from pty to pipe processes, but he refused to provide details

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=12972#38

Re: org-capture config broken

[Maxim Nikulin]

On 26/08/2021 16:48, Orm Finnendahl wrote:


Am Mittwoch, den 25. August 2021 um 18:16:15 Uhr (+0200) schrieb Maxim
Nikulin:


Unfortunately you did not specify version of Org you are currently using.


Sorry, it's 9.4.4.


The changes, I were writing about, landed in 9.4.6. If you were using 
the latest version, perhaps you would not experience a problem with 
old-style URI.



$ emacsclient org-protocol:/capture?template=m?url=URL=TITLE=BODY
Waiting for Emacs...
*ERROR*: No capture template referred to by "m?url" keys

Thanks for the hint! If this example is still in the master branch
maybe that should get fixed (or the code if it should work with a
question mark).


I just took the example from the manual at the web site. Then I realized 
that it looked strange and fixed it. Unfortunately I missed one question 
mark. The example in master and maint branches is correct. Thank you for 
drawing attention to this issue.

Re: Manual on web site is not the latest version

[Maxim Nikulin]

On 28/04/2021 12:35, Bastien wrote:

Do you still see differences between the doc/org-manual.org and the
one published on https://orgmode.org/manual/ ?

If so, please point at one or two differences.  Thanks!


curl -I https://orgmode.org/manual/The-capture-protocol.html
date: Thu, 26 Aug 2021 13:54:39 GMT
last-modified: Wed, 24 Mar 2021 08:04:13 GMT

emacsclient org-protocol://capture?template=X?url=URL?title=TITLE?body=BODY

git show release_9.4.6:doc/org-manual.org | grep template=
: emacsclient 
"org-protocol://capture?template=X=URL=TITLE=BODY"

Re: org-capture config broken (org-protocol URI syntax)

[Maxim Nikulin]

On 25/08/2021 19:05, Orm Finnendahl wrote:


  I use a custom script (dating back to a post from 2013) to capture my
mutt Emails using org-capture. It worked flawlessly but stopped
working some time ago (I don't exactly know, when, probably last
year).


Unfortunately you did not specify version of Org you are currently using.


I assume the code has changed and doesn't accept the syntax of the
org-protocol line any more.

emacsclient 
org-protocol:/capture:/m/mutt:ysyuf6skfsbwt...@example.com/mail/%20%20Subject%3A%20test%0A%20%20%20%20From%3A%20Unknown%20%3Cdummy%40example.com%3E


This the old style of Org protocol URI. See 
https://orgmode.org/manual/The-capture-protocol.html for an example of 
modern variant (it has typos fixed in master branch, some "?" should be 
"&"). I am unsure concerning order of parameters in old-style URLs. 
Older syntax should still be supported (a warning requesting update may 
be issued however), unfortunately there were some issues, see e.g. 
https://orgmode.org/list/s4cjck$156g$2...@ciao.gmane.io


Some details: https://orgmode.org/Changes_old.html#org3da1425 (news for 
version 9.0)



The emacsclient commmand above is generated with a perl script using
uri_escape for everything after /mail/. The single slashes in the
protocol line of the emacsclient command seem strange, but I couldn't
find any hint in the internet how the syntax is supposed to be for a
certain org-capture-template.


I suggest you to update perl script to generate URI with key-value 
parameters in query part.


Concerning number of slashes, I suppose, the following should minimize 
issues with desktop integration and intermediate bugs in various 
versions of Org


org-protocol:/capture?template=X?url=URL=TITLE=BODY

namely no double slash after org-protocol otherwise in some cases slash 
might be inserted before question mark or second colon might be dropped 
in old-style URI. With double slash subprotocol might be interpreted as 
host name causing further normalization as dropping colon that is not 
followed by port number or adding path "/" before query part. Such 
considerations however are not applied for direct invocation of emacsclient.


Emacsclient considers org-protocol URI as relative path, so it prepends 
argument with current working directory and squashes multiple slashes 
into single ones. That is why it does not matter how much consequent 
slashes are used in emacsclient command, however desktop-wide handlers 
may modify URI depending on number of slashes.


Latest releases should tolerate URIs with slash before question mark like

org-protocol://capture/?template=X?url=URL=TITLE=BODY

Re: Bug: org-src-font-lock-fontify-block should be wrapped with save-match-data [9.3.7 (9.3.7-4-gba6ca7-elpaplus @ mixed installation! /mnt/c/Users/toz/Weiterbildung/Soft/Emacs/ and /mnt/c/Users/toz/.

[Maxim Nikulin]

In my previous message I forgot to mention the subject of the initial 
bug report:


Bug: org-src-font-lock-fontify-block should be wrapped with 
save-match-data [9.3.7 (9.3.7-4-gba6ca7-elpaplus @ mixed installation! 
/mnt/c/Users/toz/Weiterbildung/Soft/Emacs/ and 
/mnt/c/Users/toz/.emacs.d/elpa/org-plus-contrib-20200615/)


Notice "mixed installation!". Often it is source of obscure errors.
https://orgmode.org/worg/org-faq.html#mixed-install

Maybe link to FAQ item should be added to bug report template when mixed 
installation is detected. Since mixed installation can be detected, I 
would consider issuing a message during loading of Org mode.


On 24/08/2021 23:57, Maxim Nikulin wrote:

On 23/08/2021 14:45, Timothy wrote:


Thanks for your efforts. I have prepared a patch accordingly that wraps
org-src-font-lock-fontify-block’s body with save-match-data (attached).

>
The following question may be dumb since I am not familiar how font lock 
works in Emacs. Is it necessary to wrap whole function? I do not see 
explicit operation with regexps. The only suspecting line is


   (org-font-lock-ensure)


I think, now it is irrelevant, so just for completeness: initialization 
of major mode might affect match data as well


(unless (eq major-mode lang-mode) (funcall lang-mode))

Re: What happened to ./contrib?

[Maxim Nikulin]

On 25/08/2021 15:07, Martin Steffen wrote:


Note that the documentation

   https://orgmode.org/worg/org-contrib/

(still) mentions currently a directory contrib, but in my latest pulled
version, it's gone (see also https://github.com/bzg/org-mode).


There is a patch that should fix links to raw files on worg:
https://orgmode.org/list/sc4k26$uf6$1...@ciao.gmane.io

See https://orgmode.org/list/874kf4fanb@gnu.org/ or
https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00769.html
for the announce of org-contrib package:

Bastien writes:

You can now install org-contrib as a new NonGNU ELPA package:
https://elpa.nongnu.org/nongnu/


P.S. Please, do not use "reply" to ask new question since it merges 
unrelated discussions into single thread. It is better to compose new 
message.

Re: Bug: org-src-font-lock-fontify-block should be wrapped with save-match-data [9.3.7 (9.3.7-4-gba6ca7-elpaplus @ mixed installation! /mnt/c/Users/toz/Weiterbildung/Soft/Emacs/ and /mnt/c/Users/toz/.

[Maxim Nikulin]

On 23/08/2021 14:45, Timothy wrote:


Thanks for your efforts. I have prepared a patch accordingly that wraps
org-src-font-lock-fontify-block’s body with save-match-data (attached).

If I don’t hear anything bad about it in the next few days, I’ll push it :)
Please let me know if my commit message agrees with your understanding of the
issue.


Timothy, are you able to reproduce the issue with performance? Due to 
the thread
https://orgmode.org/list/cafhswegab_im1wpxp3xsffxcoahkyycm4gaqrin0suxxd0g...@mail.gmail.com/ 
"Large source block causes org-mode to be unusable" I assume that it 
could be quite severe. On the other hand I tried to insert an .el file 
into org-guide.org and I have not noticed slow down. Actually my 
curiosity is caused by other performance issues, so I am interested if 
the effect may be broader.


The following question may be dumb since I am not familiar how font lock 
works in Emacs. Is it necessary to wrap whole function? I do not see 
explicit operation with regexps. The only suspecting line is


  (org-font-lock-ensure)

I am not sure whether the body of `org-src-font-lock-fontify-block' 
should be wrapped or its call site.


Feel free to disregard my questions since fontification is far aside 
from my experience.

Re: Internal link broken when publishing (was org-id with ox-html)

[Maxim Nikulin]

On 23/08/2021 03:42, inkbottle wrote:

Links between two different files, like
`[[id:e54113f9-2ad7-4a86-94be-68ffc696de0b][hello]]`, get broken when
publishing. (whatever the settings).

I haven't found any workaround.

Being able to move around entries, functionality provided by `org-id-link-to-
org-use-id`, is pivotal, IMO.


From my point of view it looks like a bug.
https://code.orgmode.org/bzg/org-mode/src/master/lisp/ox.el#L4381
checks for ID property.
https://code.orgmode.org/bzg/org-mode/src/master/lisp/ox-html.el#L1659
queries CUSTOM_ID only. I suppose, ID should be here as well. A subtle 
point is which one (ID or CUSTOM_ID) should be used if both are defined 
for some headers.


Maybe it is possible to create workaround as a custom config without 
touching of Org code. I guess, "nicer" ids may be replaced by values of 
ID property. Examples (I tried none of them):
- 
https://tecosaur.github.io/emacs-config/config.html#nicer-generated-heading 
https://orgmode.org/list/e1jxajq-0004dk...@lists.gnu.org/ (TEC. 
[Interest] Determanistic Org IDs. Sun, 19 Jul 2020 22:27:31 +0800)
- 
https://github.com/alphapapa/unpackaged.el#export-to-html-with-useful-anchors 
https://orgmode.org/list/ca+g3_pocytkr6m1k8xupzusg5ehd5tv4fe4x-6mvadcsjsk...@mail.gmail.com/ 
(Tom Gillespie. Tue, 8 Dec 2020 20:39:08 -0500, Re: stability of toc links)
- 
https://github.com/zzamboni/dot-doom/blob/master/doom.org#capturing-and-creating-internal-org-links 
https://orgmode.org/list/cagy83ecfexvco6tutoqiywgddo0cxe+db828leiddimxbro...@mail.gmail.com/

(Diego Zamboni. Wed, 9 Dec 2020 09:45:52 +0100. Re: stability of toc links)
- 
http://ivanmalison.github.io/dotfiles/#usemyowndefaultnamingschemefororgheadings 
https://orgmode.org/list/4698cfeb2e415f88bd25e71267f29...@isnotmyreal.name/
(TRS-80. Sat, 12 Dec 2020 20:25:15 -0500. Re: Better heading links in 
org-mode exports)

- https://writequit.org/articles/emacs-org-mode-generate-ids.html

Re: Still errors with Emacs 26.3

[Maxim Nikulin]

On 16/08/2021 11:49, Axel Kielhorn wrote:

Thanks for fixing the errors in org cite.


I hope, what you report about are no more than warnings, not errors.


Compiling /Users/axel/src/Emacs/org-mode/lisp/org-agenda.el...

In end of data:
org-agenda.el:10873:1:Warning: the function ‘string-empty-p’ is not known to
 be defined.


A patch for this particular issue:
https://orgmode.org/list/seeb8h$364$1...@ciao.gmane.io
Unfortunately it is not tracked at https://updates.orgmode.org/ for some 
unclear reason.



Compiling /Users/axel/src/Emacs/org-mode/lisp/org-plot.el...

In org-plot/gnuplot:
org-plot.el:634:4:Warning: ‘(dump-func (plist-get type :data-dump))’ is a
 malformed function
org-plot.el:685:17:Warning: reference to free variable ‘dump-func’


Try "make single" to get even more warnings...

Re: org-attach-sync uses directory-empty-p (new in Emacs 28)

[Maxim Nikulin]

On 15/08/2021 16:11, Marco Wahl wrote:

Maxim Nikulin  writes:


Compiling single /home/ubuntu/org-mode/lisp/org-compat.el...

In end of data:
org-compat.el:1255:1:Warning: the function ‘directory-empty-p’ is not
known to
  be defined.



I have never fought with this kind of problem. After looking at the
other constructs in the org-compat.el file, I have tried

(if (fboundp 'directory-empty-p)
 (defalias 'org-directory-empty-p #'directory-empty-p)
   (defun org-directory-empty-p (dir)
 ; ...


The latter suggestion landed in master a few minutes ago.


Thank you. The warning has disappeared for current master HEAD and 
Emacs-25.2.


Maybe a brief comment in org-compat.el clarifying usage of version 
comparison vs. `fboundp' would be helpful to avoid similar issues in 
future. E.g. I am unsure concerning simple < vs. version< (whether 
attempt of version check could cause a compatibility issue). Fortunately 
latest change does not use any of them.

Re: Smart quotes for Greek (questions for a possible patch)

[Maxim Nikulin]

On 14/08/2021 22:06, Juan Manuel Macías wrote:


I would like to submit a patch to add smart quotes for Greek in
`org-export-smart-quotes-alist', but I have a couple of questions.

First of all, I am not a native speaker of Greek


Disclaimer: I am not a Greek speaker at all.


Yannis Haralambous explains in "From Unicode to
Typography, a Case Study: the Greek Script" (1999, p. 20:
http://web.archive.org/web/20120229131933/http://omega.enstb.org/yannis/pdf/boston99.pdf).

Fortunately these quotes are provided by the Unicode standard (*U+201F*
and U+201D, the latter being the same closing double quotes as in
English);


There is no Greek page for
https://en.wikipedia.org/wiki/Quotation_mark
As to English page, notice
https://en.wikipedia.org/wiki/Quotation_mark#Summary_table
https://en.wikipedia.org/wiki/Quotation_mark#Greek
sections. It cites
http://publications.europa.eu/code/el/el-4100107el.htm
that uses “ U+0201C (, , or ) and ” U+0201D (, 
, or ) as inner quotes. If it is acceptable to use this 
pair, LaTeX commands likely may be \textquotedblleft and 
\textquotedblright (or simply `` and ''). I have not noticed anything 
more appropriate in "The Comprehensive LATEX Symbol List". Even
https://www.ctan.org/pkg/babel-greek does not suggest proper way for 
inner quotes.


Even if precise variant is uncertain, maybe any reasonable choice is 
better than currently used defaults. Anyway it may be corrected later.



#+begin_src emacs-lisp
(secondary-opening :utf-8 "‟" :html ";" :latex "??" :texinfo "??")
#+end_src


It seems, there is no named entity for U+0201F
https://html.spec.whatwg.org/multipage/named-characters.html#named-character-references

For some unclear reason (there is a badge requesting references), 
Russian page

https://ru.wikipedia.org/wiki/Кавычки
mentions
- «…» as usually used outer,
- ‹…› as usually used inner,
- “…” as alternative outer,
- ‘…’ as alternative inner
quotes. Perhaps, it is just a mistake.

Re: return column from table as a column

[Maxim Nikulin]

On 13/08/2021 21:17, Roger Mason wrote:


I need to extract a column from a table to use as input to a source
block.  I want the extracted column to be returned as a column but it is
returned as a row.  The following illustrates the problem:

#+name: s1
| scale |  scale1 |   scale3 |  jid |
| - | 1.00402 | 0.952329 | 1632 |
| - | 1.00402 | 0.962247 | 1633 |

#+begin_src emacs-lisp :var data=s1[,3]
data
#+end_src

#+RESULTS:
| jid | 1632 | 1633 |

I want:

|  jid |
| 1632 |
| 1633 |

Is there some means of changing 'data=s1[,3]' to accomplish this?


Notice that you can get vertical representation of results as a list
#+begin_src emacs-lisp :results list

The following is a variation of a recipe suggested by Juan Manuel:

#+name: to-column
#+begin_src emacs-lisp :var lst=()
  (mapcar #'list lst)
#+end_src

#+begin_src emacs-lisp :var data=s1[,3] :post to-column(lst=*this*)
  data
#+end_src

To transform more than one column, the following idea may be useful

(defun rotate (list-of-lists)
  (apply #'mapcar #'list list-of-lists))

https://stackoverflow.com/questions/3513128/transposing-lists-in-common-lisp

Re: BUG: org-lint false missing arguments for macro with optional arguments

[Maxim Nikulin]

On 30/06/2021 20:30, Michael Dauer wrote:


I've the following:
(defun ct-action-plan ( up recently-done)
...
(push '("action_plan" . "(eval (ct-action-plan $1 $2))") 
org-export-global-macros)


Didn't you report that adding macros directly to 
`org-export-global-macros' are not supported by `org-lint' at all?


On 30/06/2021 16:40, Michael Dauer wrote:

I have a lot of global macros defined in org-export-global-macros.
Wherever they are used in my org files org-lint reports them as undefined.

end of 30/06/2021 16:40 message

The arguments of the function are specified as optional. The macro just 
call it with $1 and $2, which are nil if not stated like in the following:

{{{action_plan}}}

But here org-lint reports:
327 low   Missing arguments in macro "action_plan"


I can not reproduce it with current master and macros defined inside an 
org document


#+begin_src emacs-lisp :results silent :eval never-export
  (defun tst-macro-wrapper (args)
(apply #'tst-macro args))
  (defun tst-macro ( one two)
(format "one: %S two: %S." one two))
#+end_src

#+macro: mcropt (eval (tst-macro-wrapper _))
#+macro: mcrarg (eval (tst-macro $1 $2))

- opt ::
  + {{{mcropt}}}
  + {{{mcropt(onlyone)}}}
  + {{{mcropt(first,second)}}}

- arg ::
  + {{{mcrarg}}}
  + {{{mcrarg(onlyone)}}}
  + {{{mcrarg(first,second)}}}

`org-lint' reports no issues. mcropt relies on undocumented details of 
implementation.



Is this a bug? Any ideas for a work-around to avoid confusion for my users.


Since eval expression may be arbitrary complex, it is a challenge to 
parse a general form to find whether arguments are optional at 
particular call site. More precise analysis of macro arguments would be 
a great feature. Unsure if it is feasible.


P.S. Various static code analyzers for programming languages allows to 
suppress particular warnings at specific lines by adding special comments.


P.P.S "Rest" arguments substitution in macros (e.g. $_) may mitigate an 
issue with phrases including commas as an argument 
https://orgmode.org/list/s7gfc6$hj1$1...@ciao.gmane.io

Re: org-attach-sync uses directory-empty-p (new in Emacs 28)

[Maxim Nikulin]

On 12/08/2021 04:17, Marco Wahl wrote:

Maxim Nikulin writes:


Compiling single /home/ubuntu/org-mode/lisp/org-compat.el...

In end of data:
org-compat.el:1255:1:Warning: the function ‘directory-empty-p’ is not
known to
 be defined.

...


How make the compiler happy?


I have never fought with this kind of problem. After looking at the 
other constructs in the org-compat.el file, I have tried


(if (fboundp 'directory-empty-p)
(defalias 'org-directory-empty-p #'directory-empty-p)
  (defun org-directory-empty-p (dir)
; ...

I have no idea if there are any drawbacks of such approach, I can only 
say that such form suppresses the warning.


It seems, version comparison is sensible when an existing function gets 
new arguments, not in the case of completely new function.

Re: citations: rx problems with emacs-26.3

[Maxim Nikulin]

On 11/08/2021 16:33, Nicolas Goaziou wrote:

Maxim Nikulin writes:


There are a couple of issues with "make single" for emacs-25.2:


All fixed. Thank you.


Thank you, Nicolas. It seems org-cite can be cleanly loaded or compiled 
by Emacs-25 and Emacs-26 now.

Re: org-attach-sync uses directory-empty-p (new in Emacs 28)

[Maxim Nikulin]

On 11/08/2021 03:52, Marco Wahl wrote:

Kyle Meyer  writes:


In 61e083732 (org-attach: Possibly delete empty attach directory,
2021-07-09), you added a call to directory-empty-p.  This function was
introduced in Emacs's 0806075520 (Add directory-empty-p and new argument
COUNT for directory-files-*, 2020-11-02) and hasn't yet made it into a
release.

Could you update org-attach-sync to avoid using directory-empty-p (e.g.,
by inlining it or by adding a compatibility alias)?


Starting from Arthur's suggestion and Kyle's hint to the compatibility
alias I put org-directory-empty-p into org-compat.el.  So there is
org-directory-empty-p now which provides the functionality of
directory-empty-p from Emacs 28 for smaller version Emacsen.


Unfortunately current code causes a compiler warning at least when 
Emacs-25.2 is used:


Compiling single /home/ubuntu/org-mode/lisp/org-compat.el...

In end of data:
org-compat.el:1255:1:Warning: the function ‘directory-empty-p’ is not 
known to

be defined.

Re: citations: rx problems with emacs-26.3

[Maxim Nikulin]

On 08/08/2021 21:14, Nicolas Goaziou wrote:

Maxim Nikulin writes:


I have noticed another problem with
a lot of pcase branches in `org-cite-natbib--style-to-command' in and
my experience is not enough to resolve it. On attempt to
byte-compile-file
the following error is reported after significant delay (and CPU fan
becomes more noisy)

oc-natbib.el:108:9:Error: Bytecode overflow


Fixed. Thank you.


Nice. Perhaps `org-cite-biblatex-export-citation' should be fixed in a 
similar way. There is no error yet but compiled file is significantly 
blown up:


-rw-rw-r-- 1 ubuntu ubuntu 13383 Aug  9 02:17 lisp/oc-biblatex.el
-rw-rw-r-- 1 ubuntu ubuntu 48906 Aug  9 17:06 lisp/oc-biblatex.elc

There are a couple of issues with "make single" for emacs-25.2:

Compiling single /home/ubuntu/org-mode/lisp/oc-basic.el...

In end of data:
oc-basic.el:772:1:Warning: the function ‘buffer-hash’ is not known to be
defined.
Compiling single /home/ubuntu/org-mode/lisp/oc-biblatex.el...
Compiling single /home/ubuntu/org-mode/lisp/oc-csl.el...

In toplevel form:
oc-csl.el:90:1:Error: Cannot open load file: No such file or directory, 
org-cite

Makefile:59: recipe for target 'oc-csl.elc' failed
make[2]: [oc-csl.elc] Error 1 (ignored)

Re: citations: rx problems with emacs-26.3

[Maxim Nikulin]

On 08/08/2021 03:27, Nicolas Goaziou wrote:


Maxim Nikulin writes:


It seems, rx e.g. in emacs-26.3 does not support all features used in
oc.el and oc-csl.el. Loading an org file using git master, I get
a warning


Eager macro-expansion failure: (error "rx form ‘regexp’ requires args satisfying 
‘stringp’")


Thanks. Could you send the patch again with a proper commit message,
using git format-patch?


With the attached patches I do not see warnings any more while org is 
loading. On the other hand I am not an org-cite user, so I am not sure 
that nothing is broken by these patches.



>From 151191d0283cb55e719c9c01459de02dfb584e02 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Mon, 9 Aug 2021 23:59:41 +0700
Subject: [PATCH 1/2] oc: rx compatibility with Emacs < 27.1

* lisp/oc.el (org-cite-adjust-note):
* lisp/oc-basic.el (org-cite-basic-goto):
* lisp/oc-biblatex.el (org-cite-biblatex-prepare-preamble):
* lisp/oc-csl.el (org-cite-csl--label-regexp): Avoid `rx' forms
introduced in Emacs-27.1 to keep compatibility with older versions.

Use `rx-to-string' instead of `rx' due to lack of support of  `regexp'
and `literal' forms with a variable as an argument.
---
 lisp/oc-basic.el|  4 +++-
 lisp/oc-biblatex.el |  2 +-
 lisp/oc-csl.el  | 11 +++
 lisp/oc.el  | 21 ++---
 4 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/lisp/oc-basic.el b/lisp/oc-basic.el
index c46e7a715..6d625e4a5 100644
--- a/lisp/oc-basic.el
+++ b/lisp/oc-basic.el
@@ -667,7 +667,9 @@ present in the citation."
 (org-open-file file '(4))
 (pcase (file-name-extension file)
   ("json"
-   (let ((regexp (rx "\"id\":" (0+ (any "[ \t]")) "\"" (literal key) "\"")))
+   ;; `rx' can not be used with Emacs <27.1 since `literal' form
+   ;; is not supported.
+   (let ((regexp (rx-to-string `(seq "\"id\":" (0+ (any "[ \t]")) "\"" ,key "\"") t)))
  (goto-char (point-min))
  (re-search-forward regexp)
  (search-backward "{")))
diff --git a/lisp/oc-biblatex.el b/lisp/oc-biblatex.el
index 5a9c304a3..c82a77046 100644
--- a/lisp/oc-biblatex.el
+++ b/lisp/oc-biblatex.el
@@ -260,7 +260,7 @@ to the document, and set styles."
   ;; Then set options.
   (goto-char (match-beginning 0))
   (let ((re (rx "\\usepackage"
-(opt (group "[" (*? anychar) "]"))
+(opt (group "[" (*? anything) "]"))
 "{biblatex}")))
 (cond
  ;; No "biblatex" package loaded.  Insert "usepackage" command
diff --git a/lisp/oc-csl.el b/lisp/oc-csl.el
index 5e0e27211..c08adc430 100644
--- a/lisp/oc-csl.el
+++ b/lisp/oc-csl.el
@@ -253,10 +253,13 @@ If nil then the Chicago author-date style is used as a fallback.")
   "Alist mapping locator names to locators.")
 
 (defconst org-cite-csl--label-regexp
-  (rx word-start
-  (regexp (regexp-opt (mapcar #'car org-cite-csl--label-alist) t))
-  (0+ digit)
-  (or word-start line-end (any ?\s ?\t)))
+  ;; Prior to Emacs-27.1 argument of `regexp' form must be a string literal.
+  ;; It is the reason why `rx' is avoided here.
+  (rx-to-string `(seq word-start
+  (regexp ,(regexp-opt (mapcar #'car org-cite-csl--label-alist) t))
+  (0+ digit)
+  (or word-start line-end (any ?\s ?\t)))
+t)
   "Regexp matching a label in a citation reference suffix.
 Label is in match group 1.")
 
diff --git a/lisp/oc.el b/lisp/oc.el
index 338348116..53eafb3bd 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -878,14 +878,19 @@ the same object, call `org-cite-adjust-punctuation' first."
   (when org-cite-adjust-note-numbers
 (pcase-let* ((rule (or rule (org-cite--get-note-rule info)))
  (punct-re (regexp-opt (or punct org-cite-punctuation-marks)))
+ ;; with Emacs <27.1. Argument of `regexp' form (PUNCT-RE this case)
+ ;; must be a string literal.
  (previous-punct-re
-  (rx (opt (group (0+ (any blank ?\n)) (regexp punct-re)))
-  (opt (0+ (any blank ?\n)) (group ?\"))
-  (opt (group (1+ (any blank ?\n
-  string-end))
+  (rx-to-string `(seq (opt (group (regexp ,(rx (0+ (any blank ?\n
+  (regexp ,punct-re)))
+  (regexp ,(rx (opt (0+ (any blank ?\n)) (group ?\"))
+   (opt (group (1+ (any blank ?\n
+   string-end)))
+t))
  (next

Re: citations: rx problems with emacs-26.3

[Maxim Nikulin]

On 08/08/2021 03:27, Nicolas Goaziou wrote:

Maxim Nikulin writes:


It seems, rx e.g. in emacs-26.3 does not support all features used in
oc.el and oc-csl.el. Loading an org file using git master, I get
a warning


Eager macro-expansion failure: (error "rx form ‘regexp’ requires args satisfying 
‘stringp’")



diff --git a/lisp/oc-basic.el b/lisp/oc-basic.el


Thanks. Could you send the patch again with a proper commit message,
using git format-patch?


I going to do it. For a while, I have noticed another problem with a lot 
of pcase branches in `org-cite-natbib--style-to-command' in and my 
experience is not enough to resolve it. On attempt to byte-compile-file
the following error is reported after significant delay (and CPU fan 
becomes more noisy)


oc-natbib.el:108:9:Error: Bytecode overflow

I have tried to comment out some conditions

ls -l lisp/oc-natbib.elc
-rw-rw-r-- 1 ubuntu ubuntu 241882 Aug  8 13:23 lisp/oc-natbib.elc

If more branches are commented out

-rw-rw-r-- 1 ubuntu ubuntu 8525 Aug  8 13:20 lisp/oc-natbib.el

Looks like a kind of combinatorics explosion. It seems, if Emacs-25.2 
and 26.3 are considered as supported, this function should be rewritten 
to avoid issues with byte compilation.

Re: [PATCH] org-agenda.el: Avoid dependency on subr-x

[Maxim Nikulin]

On 05/08/2021 19:59, Timothy wrote:



Restore compatibility with Emacs-24.3.


FYI Emacs 24 is no longer Org’s minimum version, from the next release onward
it will be Emacs 25.


Timothy, I have seen that discussion. For me it does not matter in which 
way org-agenda.el will be fixed: by adding (require 'subr-x) or by using 
string= instead of string-empty-p. Both variants work with Emacs-26.3. 
The point is that "make single" should work without warnings and in the 
case of mixed install users should not get confusing errors that 
resemble real issues.


The only reason why I suggested namely to avoid string-empty-p is that 
compatibility with Emacs-24.3 is still declared in Org sources.

Re: [PATCH] org-agenda.el: Avoid dependency on subr-x

[Maxim Nikulin]

On 05/08/2021 04:41, Вячеслав Гришин wrote:

Now I don't know if I need to do something extra, because the
problem  arose because of my mistake.


Vyacheslav, you do not need to react to my patch, it is for Org 
maintainers. There is neither continuous integration server nor ready to 
use configuration (e.g. a set of docker containers) to verify commits in 
clean environment. As a result some files may have minor issues that are 
unnoticeable by most of users. However it is better to fix 
inconsistencies like missed `require', etc.



When I removed the second version of Org and the extra code from .emacs,
everything worked. I don't know how to explain it, because I'm just
starting to learn emacs and lisp


Notice that you can have more than one org mode version installed. E.g. 
in Debian and Ubuntu in addition to the version bundled with Emacs, it 
is possible to install elpa-org deb package. The version however is 
anyway noticeably older than current stable Org release, so people often 
use various package managers for Emacs.


The problem you have met is not uncommon, example of usual response:
https://orgmode.org/list/87im5gszqs@kyleam.com/
Unfortunately symptoms are usually quite obscure, they may reveal 
another issue. I do not know if it is possible to implement a trap to 
explicitly notify user that parts of org are loaded from different 
directories.


There are may be some pitfalls on the way to have newer org version:
- You should not require any org modules before `load-path' is adjusted.
- Do not forget to recompile after update of sources. It seems, Emacs do 
not have means to detect stale .elc file similar to e.g. Python.
- Attempt to try alternative version without modification of init.el may 
result in unexpected result. Usually "-Q" or at least "-q" option is 
required to suppress initialization. "-L" option is processed after 
init.el. So "emacs -L ~/src/org-mode/lisp" can easily result in mixed 
install. E.g. "emacs -Q -L ~/src/org-mode/lisp -l ~/.emacs.d/init.el" 
mostly work but there are some edge cases since it is not full 
equivalent of real initialization. There is no way to specify custom 
init file through command line option, it is necessary to create a part 
of parallel universe and pass it as --user option argument, HOME or 
XDG_CONFIG_HOME (for latest emacs version) environment:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=15539#121

Do not be afraid by such complications too much. ELPA package is the 
recommended way to have latest stable version:

https://orgmode.org/install.html
Alternatively ensure that (add-to-list 'load-path "~/src/org-mode/lisp") 
is early enough in your init.el.

Re: [PATCH] Rename headline to heading

[Maxim Nikulin]

On 05/08/2021 19:12, André A. Gomes wrote:

@@ -963,7 +963,7 @@ Store them in the capture property list."
(org-capture-put-target-region-and-position)
(goto-char position))
   (_ (error "Cannot find target ID \"%s\"" id
-   (`(file+headline ,path ,headline)
+   (`(file+heading ,path ,heading)


I am sorry if is a false alarm. Does it mean that all users having 
"file+headline" in their init.el files will have to immediately update 
configuration? Shouldn't both variants be supported during transitional 
period?


P.S. https://orgmode.org/worg/dev/org-syntax.html consistently uses 
"headline".

Re: [PATCH] org-agenda.el: Avoid dependency on subr-x

[Maxim Nikulin]

The patch is created for the maint branch.

I have reordered some part of the quoted message.

On 05/08/2021 05:11, Tim Cross wrote:
> I don't think this patch is required, especially since we have not had
> any bug reported relating to that version.

org-agenda.el (and some other files) should be fixed anyway.

make single
...
Compiling single /home/ubuntu/org-mode/lisp/org-agenda.el...

In end of data:
org-agenda.el:10769:1:Warning: the function ‘string-empty-p’ is not known to
be defined.

(log for emacs-25.2.2)


Note that compatibility with 24.3 was not the cause of the OP original
issue (they were running 26.3).


I just do not have enough experience with elisp to suggest a patch to 
explicitly notify users that they have mixed org install. On the other 
hand, particular symptom from the report can be easily mitigated.



Do we really want to maintain compatibility with Emacs 24.3?


There is the only place in org where `string-empty-p' is used. There is 
no (require 'subr-x) in org-agenda.el. Notice


lisp/org.el:10: ;; Package-Requires: ((emacs "24.3"))

Till this line is updated, I suppose, patches should respect it, 
especially for the maint branch. Unfortunately new org cite feature 
recently merged to master relies on some rx features unavailable even in 
Emacs-26.3 (Ubuntu-20.04 LTS focal).



That
version was released over 8 years ago and I suspect there are other
areas of org which are not compatible with that version. Note also that
the subr-x and string-empty-p were introduced in Emacs 24.4. If we are
going to support Emacs 24, I think it is reasonable it be 24.4 rather
than aiming for all 24 releases.


It seems, RedHat-7 and thus CentOS-7 were released with Emacs-24.3. I 
have no idea how many users run these flavors on desktops and are 
permitted to update Org to the latest version. Accordingly to

https://wiki.centos.org/About/Product
end of lifetime dates for CentOS-7 are 2020-08-06 for full updates and 
2024-06-30 for maintenance updates.

[PATCH] org-agenda.el: Avoid dependency on subr-x

[Maxim Nikulin]

* lisp/org-agenda.el (org-agenda-highlight-todo): Do not use `string-empty-p'.

Restore compatibility with Emacs-24.3.
---
 lisp/org-agenda.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index c3fbabc01..b6cd895bc 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -7080,7 +7080,7 @@ The optional argument TYPE tells the agenda type."
(setq x
  (concat
   (substring x 0 (match-end 1))
-   (unless (string-empty-p org-agenda-todo-keyword-format)
+   (unless (string= org-agenda-todo-keyword-format "")
 (format org-agenda-todo-keyword-format
 (match-string 2 x)))
;; Remove `display' property as the icon could leak
-- 
2.25.1

Re: Multilingual quotes inside paragraphs

[Maxim Nikulin]

On 28/07/2021 20:49, Juan Manuel Macías wrote:

Links (or macros) are a good user-level solution to solve certain
specific cases, IMHO. At a native level (in case Org someday implement
consistent and native multilingual support), I don't know what would be
the best solution for multilingual chunks of text inside paragraphs.
Maybe some kind of inline `quote' block?


My personal impression is that lightweight Org markup is suitable for 
simple documents. Problems grow as a snowball as soon as a document 
becomes more complex. If syntax were extended too much, Org would loose 
its simplicity that is one of its advantage. Complex documents require 
more strict and flexible thus more verbose markup format.


It seems, your have tried all possible variants for inline quotes
https://orgmode.org/worg/dev/org-syntax.html#Objects
macros, src_org{}, even links and nothing is perfect
https://orgmode.org/list/875yzwce28@posteo.net
Code for any approach may be polished to some extent, but some 
limitations will remain.

bug#12972: 24.3.50; Move `org-open-file' and associated code out of Org mode

[Maxim Nikulin]

On 30/07/2021 19:01, Lars Ingebrigtsen wrote:

Maxim Nikulin writes:


I am attaching a patch similar to proposed to Org mode that should
help to avoid obscure failures of viewers due to unnecessary terminal
sessions.


Thanks; makes sense to me (and seems to fix these persistent issues with
xgd-open etc), so I've applied it to Emacs 28.


Thanks for looking into this issue. Please, consider the following 
additional change:


- 8< -
diff --git a/lisp/net/mailcap.el b/lisp/net/mailcap.el
index f64897ac9b..aeeb9bd8d3 100644
--- a/lisp/net/mailcap.el
+++ b/lisp/net/mailcap.el
@@ -1186,7 +1186,6 @@ See \"~/.mailcap\", `mailcap-mime-data' and 
related files and variables."

 (make-process
  :name "mailcap-view-file"
  :connection-type 'pipe
- :noquery t
  :buffer nil ; "*Messages*" may be suitable for debugging
  :sentinel (lambda (proc event)
  (when (and (memq (process-status proc) '(exit signal))
 >8 

I did not update the patch since Eli had some objections (unclear to 
me). I wrongly added :noquery expecting to get a process completely 
detached from Emacs, something like result of "synchronous"


(process-file-shell-command command nil 0 nil)

on linux (on stackoverflow I have seen a note that w32 requires special 
treatment).


Notice that "pipe" asynchronous Emacs processes have a problem with CPU 
consumption if a process does something like


#!/bin/sh
exec 1>&-
exec 2>&-
sleep 30

Though I believe even silently killed on exit children and CPU-hungry 
Emacs in rare cases are better than decade old pseudoterminal+SIGHUP 
problem with xdg-open. I suppose, it is an Emacs bug, note Gnome or KDE 
issue.

https://lists.gnu.org/archive/html/emacs-devel/2009-07/msg00279.html

Re: bug processing non emacs-lisp blocks

[Maxim Nikulin]

On 29/07/2021 11:13, Tim Cross wrote:

dmg writes:


org-babel-load-file will try to tangle any source block that contains
the substring emacs-lisp or elisp in their language.

For example, the following code block will be tangled:

#+begin_src emacs-lispDONOT
(use-package "org-sidebar")
#+end_src

I think this is a regression. I used to comment out blocks from my
.org init files by simply adding a XXX (as in emacs-lisp). In
current org these blocks are tangled :(


I think your right. The regexp should only match on complete string.


I agree as well. Related code changed in 9.4:
- 
https://orgmode.org/list/caloshornnlxjma6jnrt-ikie-b9f9o3clqdxfekrzvyiyze...@mail.gmail.com/T/#u

- https://code.orgmode.org/bzg/org-mode/commit/be93859c78

The following is just my opinion, not a request for an update of the patch.

Maybe it will be better to change `org-babel-tangle-collect-blocks' 
function in lisp/ob-tangle.el instead and to wrap passed argument in the 
suggested way (at first I thought of non-capturing group "\\(?:" "\\)", 
but it is not necessary since the regexp is passed to `string-match-p'). 
I suppose, it is more reliable default. If anyone really needs partial 
match, he may express it explicitly by adding ".*", "?", etc. I hope, 
such change will break no existing code.

Re: Multilingual quotes inside paragraphs

[Maxim Nikulin]

On 26/07/2021 16:25, Juan Manuel Macías wrote:


I'm experimenting with `org-link-set-parameters' to create multilingual
quotes (chunks of text) inside paragraphs. Although I focus mainly on
the export to LaTeX with babel and csquotes packages, I also want extend
support for HTML and odt output. I leave here some sketches.


Are you intentionally avoiding macros {{{lang(ru, текст)}}}? It seems, 
you are abusing links a bit. Though it allows to hide "target" part and 
thus to reduce "noise" in the text. On the other hand, links may be 
broadly considered as customizable element ("interactive" property is 
not necessary for such snippets) since there are no other similar 
objects in Org syntax. One problem is support in 3rd party tools: 
pandoc, various renderers, e.g. (ruby?) on github.


Another issue is that someone will almost immediately want to put such 
quote inside link or vice versa to make some fragment of a quote a 
regular link. A workaround is possible: 
lang:de?href=https%3A%2F%2Forgmode.org%2F with some code for handling of 
unescaped target. I expect complains that it is not user-friendly to 
require splitting fragment at the borders of inner link. Org does not 
support nested objects of the same type.



or explicitly preceded by
a "!": "!german" (the reason for the latter is that in babel we can
define new languages):


Unsure that ODT and HTML allows to define languages. I would consider 
some lisp structures describing mapping of customizable languages codes 
to some set of properties.

Re: Comments break up a paragraph when writing one-setence-per-line

[Maxim Nikulin]

On 21/07/2021 22:22, Eric S Fraga wrote:

On Wednesday, 21 Jul 2021 at 21:48, Maxim Nikulin wrote:

It is at least fragile. HTML export results in separate paragraphs:


Yes, I said "if exporting to LaTeX"... ;-)


I am unsure, but "#+latex:" probably belongs to "Keywords" that are
elements, so it should end a paragraph. If such interpretation is
correct than either LaTeX exporter has a bug or syntax description
should be justified.


No, I don't think of it as a keyword and it cannot end a paragraph as it
would then break any document that uses such constructs.  Not sure what
to call it but it falls into the same category as inline directives like
@@latex:...@@.  And, in fact, this would work as well as the first
example, again failing for HTML export:

...

(untested) :-)


"@@backend:@@" aka Export Snippet is an "object", not "element"
thus is should not start paragraph
https://orgmode.org/worg/dev/org-syntax.html#Export_Snippets

Though... With LaTeX you should be careful

 >8 
One sentence is here.
@@html:inline HTML@@
Second sentence.
 8< 

Export to LaTeX:
 >8 
One sentence is here.

Second sentence.
 8< 

"@@latex:@@" line in HTML export generates empty line but does not break 
the paragraph. "#+html:" construct splits paragraphs:


One sentence is here.
#+html: 
Second sentence.


One sentence is here.



Second sentence.


I find behavior of HTML exporter more consistent but partially the 
difference originates in distinct interpretation of empty lines.


P.S. There are two types of comments: ones that should be exported as 
comments and that should be completely removed. "#+latex: %" comments 
leak to LaTeX, though they will be stripped during generation of PDF. So 
depending of what format you should provide, such behavior may be 
acceptable or not.

Re: Bug: "DEFINITION NOT FOUND" for footnote in Org manual

[Maxim Nikulin]

On 21/07/2021 17:36, Timothy wrote:



At https://orgmode.org/manual/Export-Settings.html#DOCF124
and (renumbered to "fn:123") in doc/org-manual.org, a foot-
note is defined as "DEFINITION NOT FOUND":


Well that certainly looks dodgy!


The original footnote seems to have been added with commit
e30aed8f0c62e74633f7a0398340f9bd7bdfa3cc (but personally I
don't find that text very helpful).


It seems, the footnote was unintentionally lost in the merge commit with 
a lot of conflicts due to renumbering in both branches:


https://code.orgmode.org/bzg/org-mode/commit/9410fbe0606a53b0582cc175b2cc525d556b23fe#diff-6498bd2213e2e527d904942bb214768a1913776L21749

Re: Comments break up a paragraph when writing one-setence-per-line

[Maxim Nikulin]

On 19/07/2021 21:03, Eric S Fraga wrote:

On Friday, 16 Jul 2021 at 12:06, William Denton wrote:

People who write one-sentence-per-line, have you had this problem, and if so how
did you handle it?


If I will be exporting to LaTeX, I do the following:

--8<---cut here---start->8---
One sentence is here.
#+latex: % a sentence that has been commented out.
The third sentence is here and should be in the same paragraph as the first.
--8<---cut here---end--->8---

The LaTeX directive is a LaTeX comment (% at the start) and so replaces
what would otherwise be a blank line causing a new paragraph to start.

It may not be pretty but it works just fine.


It is at least fragile. HTML export results in separate paragraphs:


One sentence is here.


The third sentence is here and should be in the same paragraph as the 
first.


https://orgmode.org/worg/dev/org-syntax.html#Paragraphs
> Empty lines and other elements end paragraphs.

I am unsure, but "#+latex:" probably belongs to "Keywords" that are 
elements, so it should end a paragraph. If such interpretation is 
correct than either LaTeX exporter has a bug or syntax description 
should be justified.

Bug: duplicated \texttt in LaTeX export

[Maxim Nikulin]

It seems, something goes wrong with LaTeX export at least in git master

 >8 
#+PROPERTY: header-args :eval never-export :exports code :results silent

src_elisp{(delete-dups nil)}
 8< 

Export as LaTeX buffer:

\texttt{\texttt{(delete-dups nil)}}

I see no reason why \texttt should be doubled this case. Expectation: e.g.

\texttt{(delete-dups nil)}

Re: Comments break up a paragraph when writing one-setence-per-line

[Maxim Nikulin]

On 16/07/2021 23:06, William Denton wrote:

#+begin_src org

In this paragraph I introduce an idea.
# Here is something I'm not sure about yet.
But I am sure about this.
And here is my conclusion.

#+end_src org

When this is exported, it becomes two paragraphs, as though the 
commented line was a blank line.  This was unexpected, because to me 
this is one paragraph with one sentence hidden.


People who write one-sentence-per-line, have you had this problem, and 
if so how did you handle it?


This is unexpected and inconvenient for me as well. However it may be 
considered consistent with


https://orgmode.org/worg/dev/org-syntax.html#Paragraphs

Empty lines and other elements end paragraphs.


I do not write one-sentence-per-line, but I do not think it matters at 
all. As a workaround I can suggest the following hack


#+macro: comment

Another test
{{{comment(some text
)}}} with macro comment.

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 17/07/2021 01:34, Juan Manuel Macías wrote:

Maxim Nikulin writes:


I think that low level implementation in browser or in some underlying
library is much faster

 
   LM Roman 12
   abc абв…с
   LM Roman 12, CMU Serif
   abc абв…с
 


They are two different scenarios: web publishing and book typesetting


Juan Manuel, it seems you are too inclined to book typesetting. It is 
important case and it should be supported by org. I have repeated two 
times in this thread that there is another case, namely routine quick 
notes. Such documents have another balance concerning time required to 
get acceptable result. TeX takes responsibility for a lot of defaults 
such as what spaces should be added around "=" and what ones around 
right angle bracket. Users do not need to make decisions concerning such 
design details to get accurately typeset equations.


At the age of custom charsets (often 8bit) and encodings the problem of 
multilingual texts was obvious. Unicode and UTF-8 alleviate many issues. 
It happened that Cyrillic is an edge case for Unicode TeX engines. Since 
~2000 it works out of the box for LaTeX and PdfLaTeX. Last years I did 
not need to adjust config files and regenerate formats. Hyphenation, 
default fonts (OK, "apt install cm-super" to avoid rasterized fonts is a 
bit behind defaults though no manual config is required) just work. Each 
document needs a few universal lines to setup Russian. Some users may 
have specific style files but generally source documents are quite portable.


Default fonts for LuaTeX and XeTeX do not include Cyrillic any more. 
Every user have to do a decision which fonts should be used even if one 
does not care concerning particular fonts. It increases probability to 
get a file with font configuration that is specific to Mac or Windows.


I do not actively use characters from other Unicode planes, however 
sometimes I do it for fun. Getting them completely missing is less 
preferred than displaying them with low quality font. Specifying all 
fonts requires lengthy config, probably different for LuaTeX and XeTeX. 
At first it is necessary to realize which fonts are available for 
particular glyph. Finally it makes *source* document less portable.


"font-family: 'LM Roman 12', 'CMU Serif'" above is a dumb example of 
what I consider relatively high-level config for routine documents that 
do not need to be handcrafted. Unavailable glyph or even font is not an 
error, it just causes lookup of candidates in the following items. For 
TeX maybe it is reasonable to consider fallback to complete set of fonts 
at ones (roman + mono + math) if such combination is available.



If I want to use the GFS Porson as italics from
another font, a Didot typeface for example, I can do this:


If it is not a book or at the stage of early draft another scenario is 
possible. Text should just appear in the compiled document, particular 
font does not matter, its choice is postponed since text content has 
higher priority. Minimal setup in invaluable.


At least with minimal examples I faced another issue: characters 
silently disappears, no warning is generated. Adding babel changes it, 
but I still believe that especially for documents with carefully chosen 
fonts. It is a serious hidden error to get "invalid char glyph" instead 
of all missed characters.



[1] If you want to have fallback fonts, you can also do it in
LuaTeX by adding some Lua code:
https://tex.stackexchange.com/questions/514940/define-fallback-font-for-missing-glyphs-in-lualatex


Would you recommend such code as default for Org? Let's assume that some 
information concerning system fonts are available. I suspect, the 
accepted answer is not fool-proof. In addition, XeLaTeX requires 
something different.


luaotfload provides fallback feature close to what I expect, however it 
is necessary to explicitly specify script that I would prefer to avoid. 
Moreover it significantly increases compilation time. Sometimes LuaLaTeX 
starts to eat CPU with no progress, emoji does not work for some reason.
I am unsure concerning particular "Noto Sans CJK" since several ones are 
available.


\documentclass{article}
\usepackage{fontspec}
\usepackage{unicode-math}
\directlua{luaotfload.add_fallback
  ("seriffallback",
{
  "Noto Serif CJK SC:mode=harf;script=sc;",
  "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
})
}
% TwemojiMozilla is not shown by viewers, visible in pdftotext
  %"Noto Color Emoji:mode=harf;"
% or
  %"file:/usr/share/fonts/truetype/noto/NotoColorEmoji.ttf:mode=harf;"
% % ! error:  (file /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf) 
(ttf): loca

% table not found
% !  ==> Fatal error occurred, no output PDF file produced!
\setmainfont{CMU Serif}[RawFeature={fallback=seriffallback}]

\directlua{luaotfload.add_fallback
  (

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 16/07/2021 02:40, Juan Manuel Macías wrote:

Maxim Nikulin writes:


In CSS it is possible to specify a list of fonts and a glyph is taken
from the first font where it is present. Despite particular fonts have
limited coverage, I see wide range of Unicode characters on web pages,
that is why I am almost sure that system font libraries combine fonts.


In LuaTeX you can associate a font family to a range or a group of
characters.

   texto = unicode.utf8.gsub ( text, "([\u{e000}-\u{f8ff}])", "\\puatext{%1}" )


I expect it is terribly inefficient for long spans of text in particular 
language. Command should not be per-character, preferably per-paragraph 
or at least per-word


"([\u{e000}-\u{f8ff}]+)"

However maybe you just do not use sequences of symbols from private use 
area.


I think that low level implementation in browser or in some underlying 
library is much faster



  LM Roman 12
  abc абв…с
  LM Roman 12, CMU Serif
  abc абв…с

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 15/07/2021 02:05, Stefan Nobis wrote:

Maxim Nikulin writes:


There are cm-super fonts for at least of 15 years.


There are many tradeoffs in many aspects. No single font pleases
everyone. So you want to say: Your requirements are more
important/common/stylish/whatever that the requirements of other
people?


I hope, it is possible to make Org export to PDF working out of the box 
for more people. Surprisingly Unicode TeX engines besides alleviating of 
some limitations of earlier implementation bring new burden. I see no 
problem to fix support of Cyrillic in PdfTeX. I have not realized if it 
can be done for LuaTeX or XeTeX despite original expectation that 
support of Unicode means that there should be no problem for any 
character of any script. I started to discuss Russian because it is a 
language for which most of problems are apparent for me. Perhaps a 
similar approach can be used for other scripts.


Before UTF-8 became wide spread on Linux, there were guides how to 
enable support of Cyrillic in e.g. Netscape Navigator. Now such problems 
are forgotten. Is it achievable for TeX?


In CSS it is possible to specify a list of fonts and a glyph is taken 
from the first font where it is present. Despite particular fonts have 
limited coverage, I see wide range of Unicode characters on web pages, 
that is why I am almost sure that system font libraries combine fonts.


Do Unicode TeX engines support such combination of fonts? Is it 
efficient enough to be used by default? Is it possible to write 
reasonably complete config for such purpose? It is preferable to have 
such config as a LaTeX style file, but it may be implemented in Org code 
as well.


There are two quite distinct cases: documents with carefully chosen 
fonts (e.g. a book) and everyday notes when particular font is not so 
important. For the former case a glyph taken from wrong font is a 
serious error. For the latter one, likely even low quality font is 
significantly better than a missed character. I think, both cases should 
be supported however.



[unicode-math]

Thank you for the hint. Do you think Org should use it by default?
Are there any caveats?


Yes, unicode-math should be seen as must have for lualatex and xelatex
(if math is used). As far as I know there are no downsides and it
should be part of the default packages (but only for lualatex and
xelatex, not for pdflatex).


Maybe it is possible to scan the document for presence of character from 
specific category, range, etc., to avoid overhead when the package is 
not necessary.



Is it possible to detect lualatex and xelatex in runtime?


Yes, that is possible. The LaTeX package iftex provides macros to
execute commands based on the running engine (see
https://www.ctan.org/pkg/iftex?lang=en).


I mean detection if LuaLaTeX or XeLaTeX is usable from Org code. The 
intention is to minimize customization required before successful 
export. Ideally Org should guess reasonable configuration form content 
of the document and available external tools (and maybe freeze it by 
adding properties to the document to make next compilations faster). 
Certainly user should have a way to force fixed values by disabling 
autoconfiguration as a whole or only for particular settings.



Is it possible to provide reasonable defaults for fonts?


I do not think so. You want Cyrillic. But what about Japanese,
Chinese, Devanagari, Tamil, Arabic etc? I doubt that there exists a
single font that supports all these scripts satisfactorily.


I hope, single font is not necessary since other applications can show 
all scripts simultaneously. Of course, my example was not complete, feel 
free to extend it, e.g.


Randomly chosen examples: "日本", "多喝水", "".

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 14/07/2021 13:44, Stefan Nobis wrote:


The main point: utf8x and the associated package ucs are not
maintained for quite some time (utf8x seems to be last changed in
2004) and as far as I understand have always been more of a workaround
than a solution. But I'm not an expert in this regard.

Nowadays the LaTeX kernel and input encodings like (plain) utf8 are
much more powerful and extensible and do play much better with other
packages.


I was unlucky to notice the utf8x option when ≠ character was discussed 
on this mail list in the context of pseudocode listings. I am afraid, 
year 2004 is impressive in other sense than you expected. "Better" 
solution still unable to handle ≠ (unlike deprecated one). Actually 
necessity to explicitly specify several fonts reminds me end of previous 
century when Cyrillic fonts was not in default package set and required 
manual adjustment of config files (not to mention missed support in 
babel and inputenc, so several partially incompatible variants were used 
at that time).



You need to specify all fonts that you want to use and that deviate
from the default (Latin Modern in the case of lualatex). How else
should the system now that you want something else?


There are cm-super fonts for at least of 15 years. OK, they are Type1, 
not otf or ttf and conversion from metafont was not lossless. (Actually 
the only real problem I noticed is that some rare printers ignore 
hairlines.) Why is the Latin Modern family used if CMU is available? 
Nowadays most of applications have no problem with wide range of Unicode 
characters. However it is a kind of trade-off to preserve traditional 
Computer Modern font (a kind of feature of TeX) or to pick a system font 
with more characters.



The package "unicode-math" should always be used with lualatex and
xelatex, in order to support unicode math input.


Thank you for the hint. Do you think Org should use it by default? Are 
there any caveats?



In your minimal example neither polyglossia nor babel are required


They are almost unavoidable in any real document unless it is preview of 
e.g. particular equation.


On 14/07/2021 00:53, Juan Manuel Macías wrote:> And here I add that 
feature to Linux Libertine font:


\setmainfont{Linux Libertine O}[RawFeature=+mysub]


It is wonderful that custom font can be chosen so easily. I was never 
brave enough and I did not have strong enough reason to follow lengthy 
guides how to use ttf font in latex+dvips+ps2pdf workflow. However 
custom fonts are for special documents. It perfectly suits for e.g. a 
book when camera ready variant is required. For routine notes it is 
better to keep from defaults as minimal as possible to minimize problems 
that may arise a decade later. I would prefer to avoid Linux Libertine 
if I am going to send source file to a colleague having another OS. I 
prefer to do fine tuning at the last stages of preparation of a 
document. It is sad that default fonts are often unusable for me.



For multilingual management I recommend using Babel instead of
Polyglossia.


I have no experience with polyglossia yet. I added it just because most 
of examples for LuaLaTeX or XeLaTeX use it.


TeX takes responsibility for a lot of things and it allows to get rather 
pleasant results with minimal efforts due to reasonable defaults. Unlike 
Apache FO processor for general formatting. Equations were looking 
disgusting in MS Word till ~2010.


This topic started from question concerning multilingual support out of 
the box. I can not help with Japanese quotes. However some problems can 
be noticed with e.g. (sorry for some raw LaTeX):


 >8 
Test¹ of superscript and ½ fraction.

*«Теорема».* /Пусть/ $α → ∞$ и $\beta \to \infty$.

=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α. µm.

Text utf8x ≠ utf8 and math $8 ≠ x$.
 8< -

Current default in Org is pdflatex. It requires at least adjusting of 
fontspec by adding T2A option.


If LuaTeX and XeLaTeX handles Unicode better, is it possible to make any 
of them the default option and to leave pdflatex as a fallback? Is it 
possible to detect lualatex and xelatex in runtime? I have noticed that 
/usr/bin/lualatex belongs to texlive-latex-base package. Originally I 
did not have texlive-luatex package installed, so likely lualatex was 
rather broken despite presence of the binary in the system.


Should some packages for lualatex and xelatex be added to default list 
to minimize user problems and at the same time keeping configuration 
safe? (unicode-math, etc.)


Is it possible to provide reasonable defaults for fonts? Since lmodern 
is hardcoded in luatex and xetex, it may be done either by some usually 
available latex package or by org code and custom variables.


If some defaults can not be determined (e.g. \setmainfont) likely they 
should be explicitly mentioned in the org manual.

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 10/07/2021 23:44, Stefan Nobis wrote:

Maxim Nikulin writes:


(add-to-list 'org-latex-inputenc-alist '("utf8" . "utf8x"))
so I am unaware whether \usepackage[utf8x]{inputenc} has any drawbacks.


Do not do this. Both, utf8x and ucs, are obsolete and deprecated for
quite some time.


Maybe, I have seen such warnings. However I have tested neither utf8 nor 
utf8x on real examples. That is why I am unaware what can be broken in 
particular. For small examples with various symbols outside of ASCII, 
utf8x may give better support.


Last time I did something serious with LaTeX, bibtex was 8bit internally 
(unicode version had not reached stable Linux distributions, pybtex was 
buggy), so that time I used 8bit charset.



For proper unicode support, switch from pdflatex to lualatex or
xelatex. With these newer backends (and proper adjustments for the
LaTeX preamble generated by Org) Unicode should work out of the box
(if the font supports the requested Unicode characters).


Maybe my expectation from proper unicode support was too optimistic. I 
see the reason why preamble is necessary with 8bit encodings. I 
appreciate possibility to easily specify particular ttf or otf font. I 
do not like that it is necessary to specify *all* fonts, otherwise some 
characters are missed or compilation fails with error. I do not know if 
new engines allows to get list of available fonts and to choose a set of 
fonts with better coverage than lmodern.


I have tried LuaLaTeX with the following file. From some examples I have 
found, I expect, further tuning is required. Apparent problem is missed 
unicode characters in math mode. I admit that polyglossia is unavoidable 
for any real text but I do not like that \set…font commands are mandatory.


\documentclass{article}
\usepackage{fontspec}
\usepackage{polyglossia}
\setdefaultlanguage{russian}
\setotherlanguage{english}
\setmainfont{CMU Serif} % breaks ∞
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}

\begin{document}
Test¹ of superscript and ½ fraction.

\textbf{Теорема.} \emph{Пусть} $α → ∞$ и $\beta \to \infty$.

\verb=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α.
\end{document}



(setq org-latex-default-packages-alist
 '(("AUTO" "inputenc" t ("pdflatex"))
   ("T1" "fontenc" t ("pdflatex"))


I just have realized that fontenc behavior should be similar to inputenc 
and babel, e.g. something like \usepackage[T1,T2A]{fontenc} should be 
used for Russian.

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 23:00, Ihor Radchenko wrote:

Maxim Nikulin writes:


Ihor, did you get that numbers using emacs -Q? If not, I suspect you may
have twice more markers in the buffer than `org-refile-get-targets'
usually generates. If my guess is correct it may be necessary to file a
bug that org-ql or some other package abuses markers by generation to
much of them.


With emacs -Q things are twice better, though still not good enough:

...

org-ql might be one of the reasons of slowdown. Without it, the nm-tst
after clearing the cache is slightly faster (was 13 sec):


13.8 - 10.1 = 3.7 that is comparable with "emacs -Q" that is 5.3. So 
contribution of org-ql is not major one, but it is still significant.



However, markers are also created by agendas for example. The last
result is also on fresh Emacs. Fresh Emacs tends to be faster in general.

Do you know a good way to find out all the markers associated with a buffer?


I was going to ask you, but forgot to do it. Perhaps it is impossible by 
design.


(length org-agenda-markers)

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 21:20, Ihor Radchenko wrote:

Maxim Nikulin writes > | nm-tst |   22.599429478 | 4 | 
2.078571420924 |
| org-refile-cache-clear |0.003903494 | 0 |0.0 |
| nm-tst |   13.969909913 | 4 |   2.0694318153 |


Certainly 13 seconds is no go especially for interactive usage.


Yeah. So, I do not use it for now until me or someone else fixes the
issue.


Ihor, did you get that numbers using emacs -Q? If not, I suspect you may 
have twice more markers in the buffer than `org-refile-get-targets' 
usually generates. If my guess is correct it may be necessary to file a 
bug that org-ql or some other package abuses markers by generation to 
much of them.


P.S. Maybe generating of custom IDs automatically is a viable idea. 
Before I have not considered such variant believing that they would be 
noticeably longer and would have weaker associations.

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 11/07/2021 18:49, Ihor Radchenko wrote:


Your test on my 12M chars 18k headings org file (Emacs master, Org mode
master). I cannot use org-set-property even without org-refile-cache :(
  
| nm-tst |   13.969909913 | 4 |   2.0694318153 |


Certainly 13 seconds is no go especially for interactive usage. I use 
`org-set-property' to add CUSTOM_ID when necessary. I do not like purely 
random ID. Do you have an alternative for adding properties? 
Unfortunately `org-set-property-and-value' C-c C-x P does not allow to 
set even fixed list of options for completion.

citations: rx problems with emacs-26.3

[Maxim Nikulin]

I think citation support is a great feature. (Sorry, I do not work with 
references now, so I can tell nothing concerning implementation for org.)


It seems, rx e.g. in emacs-26.3 does not support all features used in 
oc.el and oc-csl.el. Loading an org file using git master, I get a warning



Eager macro-expansion failure: (error "rx form ‘regexp’ requires args satisfying 
‘stringp’")


It looks like (rx (regexp ...)) require namely literal, variables are 
not supported yet. In additional (rx (literal ...)) is not supported as 
well.



Eager macro-expansion failure: (error "Unknown rx form ‘literal’")


In a couple of places `rx-to-string' likely could be used instead of 
just `rx'. I have no idea yet what could be done with (pcase-let* ((rx 
...))) in `org-cite-adjust-note'.



diff --git a/lisp/oc-basic.el b/lisp/oc-basic.el
index 4e9d2e562..2ac601f34 100644
--- a/lisp/oc-basic.el
+++ b/lisp/oc-basic.el
@@ -656,7 +656,7 @@ present in the citation."
 (org-open-file file '(4))
 (if (not (equal "json" (file-name-extension file)))
 (bibtex-search-entry key)
-  (let ((regexp (rx "\"id\":" (0+ (any "[ \t]")) "\"" (literal key) "\"")))
+  (let ((regexp (rx-to-string `(seq "\"id\":" (0+ (any "[ \t]")) "\"" ,key 
"\"") t)))
 (goto-char (point-min))
 (re-search-forward regexp)
 (search-backward "{")
diff --git a/lisp/oc-csl.el b/lisp/oc-csl.el
index a7a2a6042..f3c2fd901 100644
--- a/lisp/oc-csl.el
+++ b/lisp/oc-csl.el
@@ -248,10 +248,10 @@ If nil then the Chicago author-date style is used as a 
fallback.")
   "Alist mapping locator names to locators.")
 
 (defconst org-cite-csl--label-regexp

-  (rx word-start
-  (regexp (regexp-opt (mapcar #'car org-cite-csl--label-alist) t))
-  (0+ digit)
-  (or word-start line-end (any ?\s ?\t)))
+  (rx-to-string `(seq word-start
+  (regexp ,(regexp-opt (mapcar #'car 
org-cite-csl--label-alist) t))
+  (0+ digit)
+  (or word-start line-end (any ?\s ?\t))) t)
   "Regexp matching a label in a citation reference suffix.
 Label is in match group 1.")
 
diff --git a/lisp/oc.el b/lisp/oc.el

index 1d704fb03..43a3bb444 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -924,7 +924,7 @@ the same object, call `org-cite-adjust-punctuation' first."
  (concat final-punct "\"") previous nil nil 2))
(new-next
 (replace-regexp-in-string
- (rx string-start (literal final-punct))
+ (rx-to-string `(seq string-start ,final-punct) t)
  "" next)))
(org-element-set-element previous new-prev)
(org-element-set-element next new-next)


Maybe I have missed other similar problems.

Does someone have better ideas how to restore compatibility with older 
emacs versions? 24.3 is declared in org.el. Ubuntu 20.04 (latest long 
time support release) has 26.3.

Re: [BUG] org-goto slows down org-set-property

[Maxim Nikulin]

On 10/07/2021 22:47, Maxim Nikulin wrote:
It is rather annoying that C-c C-x p `org-set-property' may be quite 
slow if a file is large enough. (I do not consider it huge however.) 
There is noticeable delay before the prompt for property name appears in 
minibuffer. Accidentally I have noticed that sometimes it is fast.


Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
a jump is not required, it is enough to get jump (refile) targets.


It is strange. Merely 3500 markers created by org-refile-cache in 2.2Mb 
file ruin performance of regexps in that buffer. I would expect either 
no influence of markers at all since search is a read-only operation or 
some negligible penalty proportional to fraction of markers in relation 
to total amount of characters.


Look at the numbers for nm-test that is just some regexp searches from 
`org-buffer-property-keys'. Performance is restored after clearing of 
the cache.


#+begin_src elisp
  (defun nm-tst ()
(org-with-wide-buffer
 (goto-char (point-min))
 (while (re-search-forward org-property-start-re nil t)
   (org-get-property-block)
   (outline-next-heading))
 nil))

  (org-refile-cache-clear)

  (let ((org-refile-targets '((nil :maxlevel . 6)))
(org-refile-use-outline-path 'file)
(org-refile-use-cache t)
(iters 10))
(cons
 (list (org-version))
 (mapcar (lambda (f)
   (append (list f) (benchmark-run iters (funcall f
 (list #'nm-tst ; #'org-buffer-property-keys
   #'org-refile-get-targets
   #'nm-tst ; #'org-buffer-property-keys
   #'org-refile-cache-clear
   #'nm-tst
#+end_src

#+RESULTS:
| 9.4.6  | |   ||
| nm-tst | 0.044981942 | 0 |0.0 |
| org-refile-get-targets | 1.748572638 | 8 | 0.387660137993 |
| nm-tst | 0.393022201 | 0 |0.0 |
| org-refile-cache-clear | 0.001787078 | 0 |0.0 |
| nm-tst | 0.035731803 | 0 |0.0 |

It seems, it is better to drop org-refile-cache completely. It has some 
design problems making it rather fragile. At the same time a patch can 
be applied that improves performance of `org-refile-get-targets' by a 
factor of at least 2 for (org-refile-use-outline-path 'file).


Emacs-26.3

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 10/07/2021 20:52, Juan Manuel Macías wrote:


I agree with you that Org should have multilingual support. A few months
ago I started this thread here, with some proposals:
https://orgmode.org/list/87o8d95pvo@posteo.net/


I tried to draw more attention to support of locale-aware formatting of 
numbers on emacs-devel mail list. Certainly a question concerning 
mixed-language buffers were raised. Result of attempts to discuss text 
properties to mark regions having alternative languages was just "Which 
property will help here? we don't have such properties" from Eli Zaretskii.



 > Jean-Christophe Helary writes:


I guess it is an issue with the Latex backend and could have been
solved with the proper Latex settings, but it seems weird that the
default settings do not allow for out-of-the-box multilingual
support.


(add-to-list 'org-latex-inputenc-alist '("utf8" . "utf8x"))

might help. I am not an active LaTeX user last years, so I am unaware 
whether \usepackage[utf8x]{inputenc} has any drawbacks.

[BUG] org-goto slows down org-set-property

[Maxim Nikulin]

It is rather annoying that C-c C-x p `org-set-property' may be quite 
slow if a file is large enough. (I do not consider it huge however.) 
There is noticeable delay before the prompt for property name appears in 
minibuffer. Accidentally I have noticed that sometimes it is fast.


Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
a jump is not required, it is enough to get jump (refile) targets.


I hope, the following numbers are convincing enough. Second attempt to 
get list of properties is much slower than the first one.


#+begin_src elisp
  (require 'org-refile)
  (let ((org-refile-targets '((nil :maxlevel . 6)))
(org-refile-use-outline-path 'file)
(org-refile-use-cache t)
(iters 10))
(cons
 (list (org-version))
 (mapcar (lambda (f)
   (append (list f) (benchmark-run iters (funcall f
 (list #'org-buffer-property-keys
   #'org-refile-get-targets
   #'org-buffer-property-keys
#+end_src

git master:

#+RESULTS:
| 9.4.6|  || |
| org-buffer-property-keys | 0.065695 |  0 | 0.0 |
| org-refile-get-targets   | 1.836473 | 12 | 0.3949867119985 |
| org-buffer-property-keys | 0.563227 |  0 | 0.0 |

elpa-org system package for Ubuntu-20.04 is even more slower:

#+RESULTS:
| 9.3.1| || |
| org-buffer-property-keys | 0.242574778 |  0 | 0.0 |
| org-refile-get-targets   | 2.669112479 | 13 | 0.285175928 |
| org-buffer-property-keys | 1.231223427 |  0 | 0.0 |

The effect is noticeable for the doc/org-manual.org file from org 
sources though absolute numbers are well below delay that makes 
interaction incomfortable.


In the configuration above effect of `org-refile-use-cache' is crucial, 
contribution of `org-refile-use-outline-path' is noticeable but not so 
important.


My expectation is that neither `org-refile-use-cache' nor `org-goto' 
command should make adding property slower.

Re: A requires/provides approach to linking source code blocks

[Maxim Nikulin]

On 09/07/2021 02:32, Tim Cross wrote:

Marko Schuetz-Schmuck writes:


I would find it useful to have a more declarative way for specifying
sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
capture dependency and then have the exporter compute a sequence
satisfying these. I would think that this makes the maintenance of the
dependencies more convenient.


My concern here is with the additional complexity. This is already a
somewhat complex aspect of org mode and the behaviour you describe can
effectively be done using noweb, although as you say, not as
declarative in style.


There was a feature request for dependencies between code blocks to run 
them in proper order a half of a year ago.


I think, it is better to try such feature as an extension of org, as a 
separate package. I suspect that some non-obvious issue may arise. 
Likely, to be convenient, it will be desired to autofill 
requires/provides attributes using some tools related to particular 
language.


The question is whether org code is organized in such way, so extensions 
of this kind are possible without code duplication. Maybe API could be 
adjusted. On the other hand, even maintaining of stable semi-internal 
API sometimes is significant burden.


P.S. I am reading the mail list through NNTP news.gmain.io. I have 
noticed that personal copies sometimes arrives without mail list 
address. Sorry if you will have to manually add it to reply.

[PATCH] worg/org-contrib: fix links to raw files

[Maxim Nikulin]

On 07/07/2021 22:31, Maxim Nikulin wrote:

On 07/07/2021 14:23, Michael Maurer wrote:

On Tue, 6 Jul 2021 at 08:07, Michael Maurer wrote:


in my config, but making a camel case headline does not create a link
when repeating the word in a paragraph. I've also tried activating
org-wikinodes with m-x customize, but no difference.


"The contrib/ directory now lives outside of Org's repository" is 
announced on https://updates.orgmode.org/


Maybe the new repository and the package should be mentioned on the 
install page to avoid confusion.


Even link to the source file on https://orgmode.org/worg/org-contrib/ is 
incorrect. See the patch.
>From 8ef6e13d10c1ecf03fd3d1b49bcfc53fe03598c7 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Wed, 7 Jul 2021 23:02:56 +0700
Subject: [PATCH] org-contrib/index.org: update org-contrib raw file links

Contrib were removed from the main repository, new location is
https://git.sr.ht/~bzg/org-contrib/
See https://orgmode.org/list/874kf4fanb@gnu.org/
---
 org-contrib/index.org | 113 +-
 1 file changed, 57 insertions(+), 56 deletions(-)

diff --git a/org-contrib/index.org b/org-contrib/index.org
index 0bb3af0c..19a50209 100644
--- a/org-contrib/index.org
+++ b/org-contrib/index.org
@@ -8,6 +8,7 @@
 #+LANGUAGE:   en
 #+CATEGORY:   worg
 #+LINK:   repofile https://code.orgmode.org/bzg/org-mode/raw/master/
+#+LINK:   contribfile https://git.sr.ht/~bzg/org-contrib/blob/master/
 #+HTML_LINK_HOME: https://orgmode.org/worg/
 #+HTML_LINK_UP: ../index.html
 
@@ -37,33 +38,33 @@ hopefully have some documentation.
 - /org-annotate-file.el/ -- annotate a file with org syntax ::
   The annotation is in a separate file, with links back to the
   annotated file.  Written by /Philip Jackson/.
-  [[repofile:contrib/lisp/org-annotate-file.el][Link to raw file]].
+  [[contribfile:lisp/org-annotate-file.el][Link to raw file]].
 
 - /org-bibtex-extras.el/ -- extras for working with org-bibtex entries ::
   Written by /Eric Schulte/.
-  [[repofile:contrib/lisp/org-bibtex-extras.el][Link to raw file]].
+  [[contribfile:lisp/org-bibtex-extras.el][Link to raw file]].
 
 - /org-bookmark.el/ -- support for links to Emacs bookmarks ::
   Written by /Tokuya Kameshima/.
-  [[repofile:contrib/lisp/org-bookmark.el][Link to raw file]].
+  [[contribfile:lisp/org-bookmark.el][Link to raw file]].
 
 - [[file:org-checklist.org][/org-checklist.el/ -- org functions for checklist handling]] ::
   Reset checklists in repeating entries.  Written by /James TD Smith/.
-  [[repofile:contrib/lisp/org-checklist.el][Link to raw file]].
+  [[contribfile:lisp/org-checklist.el][Link to raw file]].
 
 - [[file:org-choose.org][/org-choose.el/ -- decision management for org-mode]] ::
   Org-choose helps documenting a decision-making process by using
   TODO keywords for different degrees of /chosenness/, and by
   automatically keeping a set of alternatives in a consistent state.
   Writen by /Tom Breton/.
-  [[repofile:contrib/lisp/org-choose.el][Link to raw file]].
+  [[contribfile:lisp/org-choose.el][Link to raw file]].
 
 - [[file:org-collector.org][/org-collector.el/ -- collect properties into tables]] ::
   Collect and process properties into a table.
   Written by /Eric Schulte/.
-  [[repofile:contrib/lisp/org-collector.el][Link to raw file]].
+  [[contribfile:lisp/org-collector.el][Link to raw file]].
 
-- [[repofile:contrib/lisp/org-contacts.el][/org-contacts.el/ -- manage contacts (org-plus-contrib)]] ::
+- [[contribfile:lisp/org-contacts.el][/org-contacts.el/ -- manage contacts (org-plus-contrib)]] ::
   Written by /Julien Danjou/, now in Org contrib.
 
 - [[file:org-depend.org][/org-depend.el/ -- TODO dependencies for Org-mode]] ::
@@ -71,41 +72,41 @@ hopefully have some documentation.
   be blocked by the state of another entry.  Also, easily create
   chains of TODO items with exactly one active item at any time.
   Written by /Carsten Dominik/.
-  [[repofile:contrib/lisp/org-depend.el][Link to raw file]].
+  [[contribfile:lisp/org-depend.el][Link to raw file]].
 
 - /org-elisp-symbol.el/ -- Org links to emacs-lisp symbols. ::
   This can create annotated links that exactly point to the definition
   location of a variable of function.
   Written by /Bastien Guerry/.
-  [[repofile:contrib/lisp/org-elisp-symbol.el][Link to raw file]].
+  [[contribfile:lisp/org-elisp-symbol.el][Link to raw file]].
 
 - /org-expiry.el/ -- expiry mechanism for Org entries ::
   Written by /Bastien Guerry/.
-  [[repofile:contrib/lisp/org-expiry.el][Link to raw file]].
+  [[contribfile:lisp/org-expiry.el][Link to raw file]].
 
 - [[file:org-git-link.org][/org-git-link.el/ -- link to files under git version control]] ::
   This package adds new link types to link to specific versions of a
   file, which will be checked out when the link is activated.
   Written by /Reimar Finken/.
-  [[repofile:contrib/lisp/org-git-link.el][Link to raw file]].
+  [[contribfile:lisp/or

Re: unable to activate org-wikinodes

[Maxim Nikulin]

On 07/07/2021 14:23, Michael Maurer wrote:

On Tue, 6 Jul 2021 at 08:07, Michael Maurer  wrote:


in my config, but making a camel case headline does not create a link
when repeating the word in a paragraph. I've also tried activating
org-wikinodes with m-x customize, but no difference.


Is this deprecated and no longer part of org? I searched the
org-folder for any file resembling wikinodes, but couldn't find
anything.


org-wikinodes.el is a part of org-contrib

"The contrib/ directory now lives outside of Org's repository" is 
announced on https://updates.orgmode.org/


Maybe the new repository and the package should be mentioned on the 
install page to avoid confusion.


https://orgmode.org/list/874kf4fanb@gnu.org/ or
https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00769.html
On 15/05/2021 19:23, Bastien wrote:

Bastien writes:


The files previously stored in the contrib/ directory of Org's repo
now lives here: https://git.sr.ht/~bzg/org-contrib


You can now install org-contrib as a new NonGNU ELPA package:
https://elpa.nongnu.org/nongnu/

Re: [PATCH] make org-notify support for macOS desktop notification

[Maxim Nikulin]

On 06/07/2021 08:45, stardiviner wrote:

Here is the new patch which invokes notifications though Emacs built-in API 
`ns-do-applescript`.


I suspect, you considered your original patch as small and simple, 
didn't you?



+  (format "display notification \"%s\" with title \"Org mode 
notification\""
+  (replace-regexp-in-string "\"" "#" notification


Last character of notification might be "\" so it might escape the 
closing quote. Some less common quote character like "´" may be less 
confusing than "#" in this confusing if proper escaping is a challenge. 
I am unsure how many people are still using 8-bit charsets.


It seems there is no way to pass argv through `ns-do-applescript'. Is 
there at least a helper to escape strings similar to 
`shell-quote-argument'? Otherwise it is not friendly to developers since 
everyone has to do it himself.


On 06/07/2021 07:06, Tim Cross wrote:

I suspect that any code
which 'encourages' the use of melpa packages will not be acceptabl
If there are license problems with `osx-lib-notify2', I expect, it can 
be utilized through `org-show-notification-handler'. However users have 
to manually add such customization that is not really convenient.


If `org-show-notification' depends on runtime configuration (whether 
DBus socket is available), maybe it is reasonable to rewrite the 
function and to add a convention that e.g. 'default return value causes 
attempt to call next handler in the responsibility chain. The intention 
is to allow fallback from `org-show-notification-handler' to standard 
notifiers or graceful degradation from `notifications-notify' to 
`message' if dbus connection is not available (e.g. Emacs is running in 
a minimal container). Such changes however out of scope of this patch.

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

By mistake I sent the message below as private one at first. However it 
actually does not add anything new to my previous comments to the bug.


On 04/07/2021 20:49, Eli Zaretskii wrote:

From: Maxim Nikulin Date: Sun, 4 Jul 2021 20:37:24 +0700


Sorry, I'm not sure I understand what this is all about.  Are you
still talking about the patch you proposed?


Yes, I am. It is about proper way to a launch viewer in 
`mailcap-view-file'. Original `start-process-shell-command' with 'pty 
connection type prematurely kills children of kde-open5 or gio open. 
With 'pipe connection type it or `make-process' might make emacs CPU 
hungry if a child decides to close stdout and stderr:



#!/bin/sh
exec 1>&-
exec 2>&-
sleep 30


and finally `process-file-shell-command' does not allow to report 
failure. Moreover you suspect another secret compatibility problem with 
'pipe.

Re: [PATCH] make org-notify support for macOS desktop notification

[Maxim Nikulin]

On 05/07/2021 10:50, stardiviner wrote:

I updated the patch, I found the package `osx-lib` contains solution.
So I removed the directly osascript process invocation.


I have no objections any more. On the other hand I have no access to 
macOS, so I have not tested this patch. Feel free to ignore comments 
from this message, they are mostly matter of taste.


I expect that a simple script "notify-send" may allow to avoid 
modification of code. Something like (untested, unsure concerning 
"quoted form of ...")


#!/usr/bin/env osascript
display notification (item 1 of argv)

However if osx-lib in is installed automatically, it may be more 
convenient. Unsure if some of currently supported linux distributions 
have notify-send that can not handle title as the first argument.



-   ((fboundp 'notifications-notify)
+   ((and (eq system-type 'gnu/linux) (fboundp 'notifications-notify))


Does it mean that `notifications-notify' is bound but it does not work 
on macOS? If so, maybe it is better to put new clause for 'darwin above 
and to drop 'gnu/linux here. From my point of view, it is preferable to 
avoid additional requirement for `notifications-notify'. If someone will 
create a feature request for `notifications-notify' for macOS, it will 
just work without installing of additional packages as soon as such 
feature is implemented.

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

On 03/07/2021 18:56, Eli Zaretskii wrote:

From: Maxim Nikulin Date: Sat, 3 Jul 2021 18:29:30 +0700

I am giving up with this issue.


That's too bad.  I see no reason to give up, and I urge you to
reconsider, please.


Sorry, but the space of your assumptions and maybe confusions has too 
high number of dimension to realize what you actually mean and what you 
consider as a problem that should be fixed. So any further steps are 
impossible.



Your patch unconditionally assumes that every handler will immediately
exit, and that it doesn't care about the connection type with the
parent Emacs process, but that is not guaranteed to be true.


There is no intention of such assumption and it *works* even for 
handlers that does not exit immediately.


I admit that I wrongly added ":noquery t", for some reason I believed 
that it allows to choose whether processes are allowed to exist longer 
than emacs or it is preferred to kill them with emacs. Actually 
asynchronous processes are killed always and the option manages the 
query only. This option should be dropped to restore compatibility with 
previous variant.


I have not found a way to detach asynchronous process from emacs. 
Surprisingly it is possible for synchronous processes but it is 
impossible to detect failure (thus to allow a user to figure out what 
has happened)


(process-file-shell-command command nil 0 nil)

So process API in emacs is a kind of a short blanket.

Accidentally I have created an example of program that is incompatible 
with 'pipe asynchronous processes in emacs


#!/bin/sh
exec 1>&-
exec 2>&-
sleep 30

(let ((command "cpu-stress-test")
  (process-connection-type nil))
  (start-process-shell-command command nil command))

And emacs (at least 26.3) consumes 100% CPU for the specified amount of 
time. I do not see any reason to do so since the program does not do 
anything ugly. I have not found a way to explicitly force emacs to close 
pipes. That is why I consider it as an outstanding bug. Emacs must 
properly handle closed pipes.


So `process-file-shell-command' ... 0 is better than current 
`start-process-shell-command' but it does not allow to add error handling.


So besides that I still have no guess what problem you suspect, now I 
know that emacs may become mad in response to purely innocent action of 
a child process.

Re: [PATCH] make org-notify support for macOS desktop notification

[Maxim Nikulin]

On 04/07/2021 07:23, stardiviner wrote:

I found `org-notify` does not support macOS desktop notification. So I write a 
small patch for this.


I am surprised that there is no OS-agnostic function in Emacs that sends 
simple notification, suitable when no advanced feature are necessary. 
Only OS-dependent variants are implemented for Linux and Windows.



+(format "'display notification \"%s\" with title \"title\"'" 
notification "Org mode message")))


Unsafe substitution of the argument. There is no guarantee that 
notification has no quote characters. I do not know, which link you 
would prefer:

- old https://xkcd.com/327/ "Robert'); DROP TABLE"
- recent 
https://arstechnica.com/gadgets/2021/06/mass-data-wipe-in-my-book-devices-prompts-warning-from-western-digital/ 
Wipe data from NAS (accordingly to some sources, device can be protected 
by firewall, it is enough to open in a browser a page with a malicious 
 element, e.g. in a comment of an earlier visitor)


The preferred way is to pass such parameters as separate arguments of 
`start-process'. I am not familiar with osascript, I hope, it does not 
additionally interpret strings passed to "display notification" to do 
something fancy things. Example with sh:


Current unsafe variant:


   sh -c "`printf 'echo "%s: %s - %s"' 'some-command' '"; echo another action ; echo 
"' 'second arg'
With parameters passed as separate arguments to avoid interpretation of 
special characters:



   sh -c 'echo "$0: $1 - $2"' 'some-command' '"; echo another action ; echo "' 
'second arg'

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

I am giving up with this issue.

My summary. New function `mailcap-view-file' has a problem similar to 
Bug#44824 (kde-open5 and "gio open" called directly or through xdg-open 
are unreliable in Emacs, I consider it as fixed in Org mode) that was 
reported by several users and refused by developers as not reproducible 
at first. I tried my best to draw attention to this problem and then to 
suggest a change that fixes the obscure problem and improves error 
handling. I am unaware of any real negative consequences of my patch. It 
is up to Emacs developers to leave the new bug as is or to fix it in a 
way they like


On 03/07/2021 00:28, Eli Zaretskii wrote:

From: Maxim Nikulin Date: Fri, 2 Jul 2021 23:24:23 +0700

On 02/07/2021 19:37, Eli Zaretskii wrote:

From: Maxim Nikulin Date: Fri, 2 Jul 2021 19:21:55 +0700


And with other handlers, this could be an
incompatible behavior change if the handler behaves differently when
its standard handles are connected to a pipe rather than a terminal
device.


Example of such handler, please.


Why do you need one?


Because of I can not imagine such case for mailcap handler in Emacs yet
and, accordingly to you, "this could be an incompatible behavior change".


You don't need to imagine it, you just need to trust me that I know
what I'm talking about: it would be an incompatible change.


Is it a kind of Church of Emacs that I have to just believe in you? 
Previous time you were trying to convince me that unconditional 'pipe is 
perfectly safe when I was unsure concerning behavior on Windows.


You prefer to keep reasons of your objections unveiled. I see no issue 
with the patch. It can be by a few lines shorter but the price is worse 
user experience. I have no idea how to move further.


Finally, the patch touches month-old unreleased code, so I see no point 
to discuss that it is "incompatible".


P.S. It was my fault to use `make-process' in Org since the function is 
not available in Emacs-24. I'm sorry for that incompatibility.

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

On 02/07/2021 19:37, Eli Zaretskii wrote:

From: Maxim Nikulin
Date: Fri, 2 Jul 2021 19:21:55 +0700
Cc: Lars Ingebrigtsen


And with other handlers, this could be an
incompatible behavior change if the handler behaves differently when
its standard handles are connected to a pipe rather than a terminal
device.


Example of such handler, please.


Why do you need one?


Because of I can not imagine such case for mailcap handler in Emacs yet 
and, accordingly to you, "this could be an incompatible behavior change".

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

On 02/07/2021 01:38, Eli Zaretskii wrote:

From: Maxim Nikulin Date: Fri, 2 Jul 2021 00:01:59 +0700

--- a/lisp/net/mailcap.el
+++ b/lisp/net/mailcap.el
-(start-process-shell-command command nil command)))

...

+(make-process
+ :name "mailcap-view-file" :connection-type 'pipe :noquery t
+ :buffer nil ; "*Messages*" may be suitable for debugging
+ :sentinel (lambda (proc event)
+ (when (and (memq (process-status proc) '(exit signal))
+(/= (process-exit-status proc) 0))
+   (message
+"Command %s: %s."
+(mapconcat #'identity (process-command proc) " ")
+(substring event 0 -1
+ :command (list shell-file-name shell-command-switch command


First, you replace start-process-shell-command with make-process, and
I'm not sure I understand why.  If all you want is to use pipes, why
not simply bind process-connection-type around the call to
start-process-shell-command?



+;; An alternative is `process-connection-type' let-bound to nil for
+;; `start-process-shell-command' call (with no chance to report failure).

---


If main process of the handler fails then show a message with exit
reason.

---^^


And with other handlers, this could be an
incompatible behavior change if the handler behaves differently when
its standard handles are connected to a pipe rather than a terminal
device.


Example of such handler, please.

bug#12972: [PATCH] Avoid regression in mailcap-view-file similar to Bug#44824

[Maxim Nikulin]

On 01/06/2021 13:56, Lars Ingebrigtsen wrote:

So I've now added this to Emacs 28 under the name `mailcap-view-file'.


I am attaching a patch similar to proposed to Org mode that should help 
to avoid obscure failures of viewers due to unnecessary terminal sessions.
>From de55b623810736df04641a4d8f6027ccb04caa7f Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Thu, 1 Jul 2021 23:41:16 +0700
Subject: [PATCH] mailcap.el: Avoid xdg-open silent failure

* lisp/net/mailcap.el (mailcap-view-file): Use 'pipe :connection-type
instead of 'pty to prevent killing of background process on handler
exit.  Avoid regression similar to Bug#44824.

Problem happens only in some desktop environments where mailcap handler
launches actual viewer (as defined in .desktop files and obtained from
mimeapps.list) in background.  E.g. xdg-open invokes "gio open" or
kde-open5 for Gnome or KDE accordingly and these handlers launch e.g.
eog or okular in background.  As soon as main process exits, temporary
terminal session created by `start-process-shell-command' is terminated.
As a result background processes receive SIGHUP.

Previously command were executed with no buffer as well, so the change
does not affect "needsterminal" and "copiousoutput" mailcap features,
they are not supported as earlier.

If main process of the handler fails then show a message with exit
reason.  Output (including error messages) is ignored as before.
Gtk applications tend to report significant amount of failed asserts
hardly informative for majority of users.
---
 lisp/net/mailcap.el | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/lisp/net/mailcap.el b/lisp/net/mailcap.el
index 54f7f416ab..a53e385662 100644
--- a/lisp/net/mailcap.el
+++ b/lisp/net/mailcap.el
@@ -1177,7 +1177,23 @@ See \"~/.mailcap\", `mailcap-mime-data' and related files and variables."
 		   (shell-quote-argument (convert-standard-filename file))
 		   command
 		   nil t))
-(start-process-shell-command command nil command)))
+;; Handlers such as "gio open" and kde-open5 start viewer in background
+;; and exit immediately.  Avoid `start-process' since it assumes
+;; :connection-type 'pty and kills children processes with SIGHUP
+;; when temporary terminal session is finished (Bug#44824).
+;; An alternative is `process-connection-type' let-bound to nil for
+;; `start-process-shell-command' call (with no chance to report failure).
+(make-process
+ :name "mailcap-view-file" :connection-type 'pipe :noquery t
+ :buffer nil ; "*Messages*" may be suitable for debugging
+ :sentinel (lambda (proc event)
+ (when (and (memq (process-status proc) '(exit signal))
+(/= (process-exit-status proc) 0))
+   (message
+"Command %s: %s."
+(mapconcat #'identity (process-command proc) " ")
+(substring event 0 -1
+ :command (list shell-file-name shell-command-switch command
 
 (provide 'mailcap)
 
-- 
2.25.1

Re: Bug: ODT export of Chinese text inserts spaces for line breaks

[Maxim Nikulin]

On 29/06/2021 10:47, James Harkins wrote:

* Test
1本人不想亲自拿到学历学位证书、急于离校者,可书面委托他人代领学历学位证
书,29日起即可离校;2本人想亲自领取学历学位证书者,按学校规定的程序及有关
要求办理离校手续,领取相关证书后离校;



Exporting to ODT produces the following (body text, omitting titles,
headers and such).

1本人不想亲自拿到学历学位证书、急于离校者,可书面委托他人代领学历学位证 书,29日起即可离校;2本人想亲自领取学历学位证书者,按学校规定的程序及有关 
要求办理离校手续,领取相关证书后离校;


Confirmed: newlines are copied to ODT document as is and they appear as 
spaces in libreoffice. I did not tried HTML since I am unsure if 
browsers should glue paragraphs with newlines into continuous string 
without spaces. Maybe it is necessary to add some attributes for proper 
representation (e.g. "lang"), however "#+LANGUAGE: cn" does not help 
even though libreoffice considers paragraph as Chinese.


On 30/06/2021 01:19, Eric Abrahamsen wrote:

There are a few ways to approach this:

(aref char-script-table ?中) -> 'han
(string-match-p "\\cc" "中") -> 0
(aref (char-category-set ?中) ?|) -> t


Thank you. I have not noticed all features hidden behind \c. I believe,

(rx (category can-break))

is more readable and I am a bit surprised that there is no descriptive 
aliases char-categories such as ?|. Just to add another example:


(category-set-mnemonics (char-category-set ?ф)) -> ".LYchjy"

and `describe-categories' to decipher it.

As to splicing lines, I found `fill-delete-newlines' that uses 
`fill-nospace-between-words-table' besides ?| category to determine 
whether space should be suppressed while splicing lines. In addition 
there are some variables to tune behavior.

Re: Bug: ODT export of Chinese text inserts spaces for line breaks

[Maxim Nikulin]

On 29/06/2021 10:47, James Harkins wrote:

So, it would make sense to add a rule to the exporter: if one of the
characters before or after a source-text line break is a Chinese,
Japanese or Korean character, do not add a space.


On 29/06/2021 11:43, tumashu wrote:

You can try the below config :-)
     (let ((regexp "[[:multibyte:]]")
   (string text))
   (setq string
     (replace-regexp-in-string
  (format "\\(%s\\) *\n *\\(%s\\)" regexp regexp)
  "\\1\\2" string))


Notice that [[:multibyte:]] means almost any non-ASCII script, e.g. 
Cyrillic:


(let ((sample "abc абв def"))
  (and (string-match "[[:multibyte:]]\+" sample)
   (match-string 0 sample)))
"абв"

It seems, `org-fill-paragraph' M-q is smart enough to avoid a space 
before or after a CJK character, so it is possible to determine correct 
way to splice lines, despite e.g. "Script" Unicode property is not 
exposed to elisp: 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Character-Properties.html 
(Anyway maintaining explicit list of scripts is not a straightforward 
approach.)


P.S.
JavaScript in browsers allows to filter characters that belong to 
particular script:


"abc абв def".match(/\p{Script=Cyrillic}+/u)
Array [ "абв" ]

I have not found such feature in regular expressions available in Emacs.

Re: Adding a goto-heading protocol to org-protocol

[Maxim Nikulin]

On 23/06/2021 13:34, Allen Li wrote:


The idea is to add a new protocol that looks like 
"org-protocol://goto-heading?id=UUID-HERE" that jumps to the specified 
Org heading in Emacs.  The implementation is really simple:


It may be convenient in some cases. I prefer custom_id properties 
however since it provides some hint what is the link target. I am 
unsure, but maybe if you post complete recipe for customization and a 
bit more verbose description of use cases, more people will try it and 
will suggest possible improvements. In my opinion, currently conscious 
choices require more effort that actual code.



;;;###autoload
(defun goto-heading (arg)
   "Org protocol handler for navigating to Org headings.
ARG is a property list."
   (org-id-goto (plist-get arg :id))


- It seems `org-id-goto' does not call `org-mark-ring-push' (unlike 
`org-id-open' that is non-interactive). As a result it is impossible tu 
jump back using C-&.

- Should that heading be opened in current window or in other one?
- Likely other variants of specifying targets will be requested soon. Is 
it necessary to support file+custom_id, file+heading text, file+line 
number? If so, files should be filtered somehow to prevent security 
issues when attempt of opening of specific file (executable, image, 
etc.) can cause unexpected actions.

- Maybe even subprotocol name is not the best one.

Re: Large source block causes org-mode to be unusable

[Maxim Nikulin]

On 22/06/2021 19:32, Léo Ackermann wrote:


I suggest that special block in LateX export 
(https://orgmode.org/manual/Special-blocks-in-LaTeX-export.html) may be 
treated differently.


Generally I agree with you that #+begin_proof should be a thin wrapper 
unlike a source code block. However I am not familiar with 
fontification, so I can not reason if it is feasible. Maybe the code is 
watching if something like #+end_proof appeared, so block finished.


On the other hand my impression was that \begin{proof} environment is 
intended to emphasize a paragraph or two to separate it from regular 
text on the same page. My expectation is that 4-page long proof should 
be formatted as ordinary sections and subsections.


If proof environment is still required, maybe it is possible to 
customize export backend to have proofs as a headings with some property 
in the source document. E.g. beamer backend converts headings to frames. 
I have never tried such tricks, so I am unaware how much work is 
required for such approach.

Re: literate programming, development log -- ideas? (ominbus reply)

[Maxim Nikulin]

On 09/06/2021 00:21, Samuel Banya wrote:


I've noticed that some times after git merge events across a few 
machines (ex: I forgot I had already pushed notes for my private notes 
on one machine, and had to merge the results from another machine), I'll 
get weird "HEAD" and "END" statements inserted by Git.


Likely you are writing about merge conflicts. They are not weird, it is 
just necessary to resolve them manually and commit clean variant. There 
is no general way to guess what particular user mean when some file in 
different branches has been modified at the same places. See e.g.

https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging#_basic_merge_conflicts

I remember one case when *clean* merge resulted in JS file that was 
syntactically correct but actually had broken code. It is extremely rare 
accident.



On Tue, Jun 8, 2021, at 1:15 PM, Greg Minshall wrote:

Eric, when you use something RCS-like as your version control system, i
assume that makes grepping to find some old note easy enough.  but,
these days i tend to use git.  when (assuming) you use git, do you have
some easy way to say "well, i had this code that looked sort of like
this... where was it?"?  (sorry, that's really a git question, but ...)


There is "git log -Gpattern" option that allows to search for code 
changes. Do not confuse it with "--grep" that is for commit messages. 
"git gui blame file" can show file state before some line was changed. 
Probably magit has a similar command.

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 06/06/2021 00:50, Utkarsh Singh wrote:

On 2021-06-05, 19:40 +0700, Maxim Nikulin wrote:


This line should be added to org-table.el, but notice "rx ‘not’ syntax
error: (or ". This logical operators have a bit different meaning for
regular expression.


I just "tested" most of them by M-x eval-buffer on
the patched file and then checked the output of `with-temp-buffer'.


I get "Eager macro-expansion failure: (error "rx ‘not’ syntax error: (or 
10 32)")" even in response to M-x eval-buffer.



After reading FAQ about multiple lines in table field I don't think this
PATCH makes much sense as my main driving force for this patch was to
simplify regular expression in `org-table-convert-region' and somehow
add newlines support.


You may check whether csv-mode have got support of newline inside cell 
already and try to sell your parser to them.


There are regular discussions how wonderful it would be to extend org 
table syntax. In my opinion org spreadsheet is overloaded already. On 
the other hand, newline in table cell sometimes is really useful. I have 
seen an idea to put some macro to single-line cell that expands to 
format-specific representation during export ({{{nl}}} =>  in the 
case of HTML). Such macro does not require extension of syntax, so it 
may be viable at least in user configuration.



P.S. Have you read "Structure and Interpretation of Computer Programs"

I also tried Racket after reading:


SICP is dedicated to general concepts rather than to particular 
language, that is why I consider it as a "special case".

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 04/06/2021 11:04, Utkarsh Singh wrote:

On 2021-06-02, 23:44 +0700, Maxim Nikulin wrote:


On 02/06/2021 22:08, Utkarsh Singh wrote:

;;;###autoload
   (defun org-table-import (file separator)
@@ -955,12 +971,13 @@ lines.  It can have the following values:
   - integer When a number, use that many spaces, or a TAB, as field separator.
   - regexp  When a regular expression, use it to match the separator."
 (interactive (list (read-file-name "Import file: ")
- (prefix-numeric-value current-prefix-arg)))
+ current-prefix-arg))


It seems, prefix argument works now. Let me remind that file name
completion was working better before your change.


If you are still going to update this patch, I recommend you to check if 
`list' form of `interactive' declaration is supported by Emacs 24.3. 
Recently Kyle had to almost revert one my patch. Fortunately a simpler 
alternative supported by old Emacs versions were already discussed.



I have noticed a couple of error messages. Unsure what is going wrong, I
hope, command to launch emacs is correct (-Q -L ~/src/org-mode/lisp
test.org).

At startup:
Eager macro-expansion failure: (error "rx ‘not’ syntax error: (or 10 44)")

In response to M-x org-table-import:
rx-check-not: rx ‘not’ syntax error: (or 10 44)


Is `rx' library loading correctly? If no then you can try:

+(require 'rx)


This line should be added to org-table.el, but notice "rx ‘not’ syntax 
error: (or ". This logical operators have a bit different meaning for 
regular expression.


Nicolas suggested you to use `list' to avoid `eval'. Certainly `eval' 
should be avoided whenever possible, but there is alternative with "`" 
and "," instead of "'". Though I often prefer `list'.




Can you also share test.org so that I can test locally?


I can reproduce it with an empty org file. Emacs-26.3. So I am curious 
if your tested your patch at all.



Now on the topic of CSV parser, I have succesfully implemented a CSV
parser in less than 65 LOC which also preserves newline character but I
am facing a dilemma on how should I represent it as a Org table.  For
ex:

...

6,"Cell with new
Line",6.28


Due to
https://orgmode.org/worg/org-faq.html#Tables
"Will there ever be support for multiple lines in a table field? No."
(there are may be some tricks in Org manual) any variant is acceptable, 
even current one. Since during export split cell is considered as 
different rows, it is responsibility of the user to preprocess original 
file and import it in a form suitable for Org.


However independently of chosen variant, it would be great to get list 
of warnings with ambiguous lines similar to compiler errors. Such 
feature is not implemented in Org yet, the most close is `org-lint' 
result (its behavior differs from compiler errors).



2. Represent every newline as distinct cell.  I was trying to implement
this method but the algorithm that I came up with requires time
complexity of O(n^2) to just print each row which is too much for large
CSV files.

(defun org-table--print-row (row)
   (let ((maxlines 1)
 list1 list2 tmp)
 ;; get maxlines
 (dolist (i row)


I expect complexity O(number of lines in result). I do not like 
excessive usage of `dolist'. I would try slicing subrows using `mapcar' 
while there is at least one non-nil element.


P.S. Have you read "Structure and Interpretation of Computer Programs" 
by Abelson, Sussman, and Sussman?

https://sarabander.github.io/sicp/html/index.xhtml
or https://mitpress.mit.edu/sites/default/files/sicp/index.html
While reading, it is necessary to realize that elisp is not scheme, tail 
recursion should be avoided in elisp and there are over differences.

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 02/06/2021 22:08, Utkarsh Singh wrote:

;;;###autoload
  (defun org-table-import (file separator)
@@ -955,12 +971,13 @@ lines.  It can have the following values:
  - integer When a number, use that many spaces, or a TAB, as field separator.
  - regexp  When a regular expression, use it to match the separator."
(interactive (list (read-file-name "Import file: ")
- (prefix-numeric-value current-prefix-arg)))
+ current-prefix-arg))


It seems, prefix argument works now. Let me remind that file name 
completion was working better before your change.


I have noticed a couple of error messages. Unsure what is going wrong, I 
hope, command to launch emacs is correct (-Q -L ~/src/org-mode/lisp 
test.org).


At startup:
Eager macro-expansion failure: (error "rx ‘not’ syntax error: (or 10 44)")

In response to M-x org-table-import:
rx-check-not: rx ‘not’ syntax error: (or 10 44)


Currently I am trying to refactor CSV parsing by applying techniques
used in pcsv library (MELPA package) which I think you will also enjoy
to play with!


I do not know opinion of Org maintainers. Personally I believe that org 
can take advantage of Emacs core features or of other packages if they 
are available and fallback to minimal implementation otherwise. Unsure 
whether borrowing code from pcsv can cause license issues. Current CSV 
parser is not perfect but it works reasonably well.

Re: bug#12972: 24.3.50; Move `org-open-file' and associated code out of Org mode

[Maxim Nikulin]

On 01/06/2021 13:56, Lars Ingebrigtsen wrote:

So I've now added this to Emacs 28 under the name
`mailcap-view-file'.


I am sorry if it is a false alarm. Feel free to close the bug again if 
something changed recently in `start-process-shell-command' or if you 
prefer to discuss the issue as another bug.


It seems that implementation of `mailcap-view-file' is unreliable due to 
creation of unnecessary terminal session and it can cause obscure and

difficult to reproduce failures similar to
https://lists.gnu.org/archive/html/emacs-orgmode/2020-09/msg00195.html
https://lists.gnu.org/archive/html/emacs-orgmode/2020-06/msg00332.html
The thread is actually longer than it is shown in the archive interface.
Another lengthy discussion:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44824

In Org latest change was required for compatibility reason:
https://code.orgmode.org/bzg/org-mode/commit/869b7a21b94ed112f6640c8f2711c2a68b661dea

Let-bind (process-connection-type nil) is a minimal required change to 
avoid unnecessary terminal session. However it is not friendly to users 
in the case when troubleshooting is required. `make-process' with 
sentinel is hopefully better.


The following could be ignored since it likely requires significant 
amount of work with unclear benefits.


1. `org-open-file' besides Org-specific stuff allows to specify precise 
target inside the file. It can be quite useful, e.g.


okular --page 11 --find "some pattern" file.pdf

PDF files have internal anchors as well. I have no consistent vision how 
to express additional "locators" in general API.


2. There are at least two sources of truth for MIME-handlers on linux 
desktop that are not necessary synchronized. Info from extracted from 
.desktop files may be configurable from desktop UI unlike mailcap. 
Distros may have some instruments to mitigate discrepancies. Debian adds 
entries from .desktop handlers to system-wide mailcap DB. Another 
approach is to add to maicap greedy xdg-open handler that tries to guess 
currently running desktop and pass arguments to appropriate command.


Maybe mailcap should be secondary MIME database in Emacs, not the 
primary one.


3. Currently only file suffix is inspected to determine MIME type of a 
file. libmagic (or file command) usually provides more precise info, so 
it is possible to open an incorrectly named file.


4. Mailcap has more features that are not addressed in Emacs. They may 
be handy if Emacs is launched in terminal on remote server. It might 
allow e.g. to open PDF file using pdftotext handler.

- A buffer for command output should be created for "copiousoutput"  option.
- A buffer should be created and terminal session should be enabled if 
an entry "needsterminal".
- There are more substitutions than "%s". However I am unsure if it is 
possible to provide more info than application can obtain from the file.

I think, it is intended for mail multipart messages and additional headers.

On the other hand mailcap handlers might expect safe file names (minimal 
ASCII subset), users may have files with arbitrary names (national 
charset or some special characters). I hope, almost all handlers do not 
have such problem.


In summary, during launch of external command terminal session must be 
suppressed. There is enough room for MIME-related improvements in Emacs 
in general and in Org mode in particular.

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 02/06/2021 00:46, Utkarsh Singh wrote:

Org 9.4.5+patches M-x org-table-import

| 1,Word,66.3e-35 ||  |   |
| 2,Unquoted  | cell,2.7   |  |   |
| 3,"Quoted   | cell",3.14 |  |   |
| 4,"Cell | ""with | quotes""",2021-06-01 |   |
| 5,"Next | cell   | is   | empty","" |
| 6,"Cell | with   | new  |   |
| Line",6.28  ||  |   |

So my personal conclusion is that CSV file is imported incorrectly in
both cases: with guessed separator and with explicitly requested through
prefix argument. Completion works a bit worse too.


Currently `org-table-guess-separator' returns "," (COMMA as string) and
`org-table-covert-region' uses '(4) to represent COMMA as separator
which is the main cause of breakdown in importing.

To make importing work well we have to:

+ Guess right separator (`org-table-guess-separator')
+ Parse CSV with this separator (`org-table-covert-region')

As far as I can tell "guessing part" works well and now we just have to
make parser work well with new separators.


Notice that for "C-u M-x org-table-import" heuristics is not necessary, 
the separator is specified explicitly. I see that your intention was to 
improve user interface of org-table-import, but actually you broke it by 
the "org-table-import: add file prompt" patch. I have not tried it, but 
my expectation is that user prompt can be customized while keeping all 
other things working. Maybe the docstring should be updated to make it 
more friendly to novices (like me while reviewing your patch).

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 17/05/2021 12:29, Bastien wrote:

Utkarsh Singh  writes:

For now can you review the patches I proposed earlier in this
thread?


Not until both you and Maxim are confident this is useful, complete
and predictable.


I have too many points to object to consider my opinion as objective.

I am unsure if colon as a separator (passwd "db") is widely used case. I 
was surprised that it is impossible to implement locale-aware detection 
of semicolon as separator due to limitations of Emacs that is not 
friendly in respect to internationalization of number formatting.


Bastien, I do not know how much tests you prefer to have in Org. Nicolas 
asked for extensive tests 
https://orgmode.org/list/875z07jx6n@nicolasgoaziou.fr I think, 
before starting work on tests, it is necessary to decide if the patches 
are acceptable in general.


Personally, I have realized that I would prefer to have anything related 
to CSV (besides very basic features) in a dedicated package, e.g. in 
csv-mode. It might define a special yank handler for copy-paste to 
org-mode. Unsure if the author of csv-mode agrees with my point of view. 
Another option is to pass files through python code to take advantage of 
more advanced heuristics.


On 15/05/2021 18:09, https://orgmode.org/list/87im3kdzi5@gmail.com 
Utkarsh Singh wrote:

--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -954,7 +954,8 @@ lines.  It can have the following values:
 - (64)Prompt for a regular expression as field separator.
 - integer When a number, use that many spaces, or a TAB, as field separator.
 - regexp  When a regular expression, use it to match the separator."
-  (interactive "f\nP")
+  (interactive (list (read-file-name "Import file: ")
+ (prefix-numeric-value current-prefix-arg)))
   (when (and (called-interactively-p 'any)


Sending patches, I am afraid to break something I was not aware about.

I have read docstrings for the modified functions and tried to import 
the following file "tbl.csv" with some CSV features. LibreOffice imports 
it correctly, it even normalizes 66.3e-35 to 6.63e-34 (I can not say 
that I always appreciate such silent modifications).


1,Word,66.3e-35
2,Unquoted cell,2.7
3,"Quoted cell",3.14
4,"Cell ""with quotes""",2021-06-01
5,"Next cell is empty",""
6,"Cell with new
Line",6.28

My optimistic expectation was (OK, I did not believe I got such result):

| 1 | Word   |   66.3e-35 |
| 2 | Unquoted cell  |2.7 |
| 3 | Quoted cell|   3.14 |
| 4 | Cell "with quotes" | 2021-06-01 |
| 5 | Next cell is empty ||
| 6 | Cell with new  |   6.28 |
|   | Line   ||

Org 9.1.9 M-x org-table-import
and C-u M-x org-table-import
actual results are close enough to my expectations

| 1 | Word   |   66.3e-35 |
| 2 | Unquoted cell  |2.7 |
| 3 | Quoted cell|   3.14 |
| 4 | Cell "with quotes" | 2021-06-01 |
| 5 | Next cell is empty ||
| 6 | "Cell with new ||
| Line" | 6.28   ||

M-x org-table-import RET tbl RET
completes file name to tbl.csv

Org 9.4.5+patches M-x org-table-import

| 1,Word,66.3e-35 ||  |   |
| 2,Unquoted  | cell,2.7   |  |   |
| 3,"Quoted   | cell",3.14 |  |   |
| 4,"Cell | ""with | quotes""",2021-06-01 |   |
| 5,"Next | cell   | is   | empty","" |
| 6,"Cell | with   | new  |   |
| Line",6.28  ||  |   |

Org 9.4.5+patches C-u M-x org-table-import

| 1,Word,66.3e-35 |
| 2,Unquoted cell,2.7 |
| 3,"Quoted cell",3.14|
| 4,"Cell ""with quotes""",2021-06-01 |
| 5,"Next cell is empty",""   |
| 6,"Cell with new|
| Line",6.28  |

M-x org-table-import RET tbl RET
complains that file name extension is not txt, tsv or csv.

So my personal conclusion is that CSV file is imported incorrectly in 
both cases: with guessed separator and with explicitly requested through 
prefix argument. Completion works a bit worse too.


One more note concerning locale support.

On 18/05/2021 22:05, Utkarsh Singh wrote:
> On 2021-05-18, 19:31 +0700, Maxim Nikulin wrote:
The question may be risen in emacs-devel but I am unsure if I will 
participate in discussion.


Why?


I am aware of some problems related to localization but I do not have 
consistent vision what API emacs should have. I have no idea what 
information is available in Windows. That is why I expect that 
discussion may be time consuming while I am not sure that someone will 
be ready to implement new features.

Re: [PATCH] org-faq.org: Expand "What is the best setup for indenting?"

[Maxim Nikulin]

Hi Greg,

Bastien merged that patch just before I was going to modify it in 
response to your suggestions. I think, 3 commits is too much noise in 
git history for the single item in FAQ. However I can to prepare one 
more correction. Maybe you have got a better idea how to avoid confusion 
with Org versions in the tables.


On 21/05/2021 09:45, Greg Minshall wrote:



I have no idea how to express it in a better way. The tables are same
for Org >= 9.5 and for Org < 9.5 (modulo bugs in indentation fixed in
9.4.6).  Only default configuration evolves in time.


again, i apologize.  my confusion was in not having read the text to
understand that the "(Org >= 9.5)" was the default.  i assumed it meant,
"this is the behavior for Org >= 9.5", even though that meant the tables
really didn't make sense.  if you wanted to compensate for such a "user
feature" :), maybe remove "(Org >= 9.5)", and add that information as a
short "NB:" line after each table?

...

sorry, i should have caught *both* "relatively".  i'd suggest


a. "do not indent anything",
"always indent relative to the element above",
"indent just ~PROPERTIES/LOGBOOK~ drawers relative
to the current heading level"


which klings better to my ears.  (sadly, i can't explain any grammatical
reason why, though.)

[Patch] tests for org-remove-invisible

[Maxim Nikulin]

The main patch that fixes org-remove-invisible to improve list sorting is 
landed.

Let me remind that there were patches that added more test cases:
https://orgmode.org/list/s5p88r$go9$1...@ciao.gmane.io
Is there any interest in them?

In the following subthread Nicolas mentioned that some of them could fail
https://orgmode.org/list/87r1j6b6ku@nicolasgoaziou.fr
I do not see any reason for failure. I just have tried C.UTF-8, en_US.UTF-8,
es_ES.UTF-8, and ru_RU.UTF-8 locales (interactively) and do not see any
problem. This set of locales has 3 different collation rules,
however I do not think it matters for tests.

On 20/04/2021 19:37, Maxim Nikulin wrote:
> On 20/04/2021 00:50, Nicolas Goaziou wrote:
>> Maxim Nikulin writes:
>>
>>> On 19/04/2021 23:08, Nicolas Goaziou wrote:
>>>>> +  ;; Space role in sorting.
>>>>> +  ;; Test would fail for locales with ignored space, e.g. en_US, it 
>>>>> works
>>>>> +  ;; in C and currently rare locales having significant space 
>>>>> (es_ES, pl_PL)
>>>>> +  (should
>>>>> +   (equal "- Time stamp\n- Timer\n"
>>>>> +  (org-test-with-temp-text "- Timer\n- Time stamp\n"
>>>>> +(org-sort-list t ?a)
>>>>> +(buffer-string))
>>>> Since this test is bound to fail for some developers, I assume it
>>>> shouldn't be included.
>>>
>>> Locale "C" is forced for this group of tests.
>>
>> Sorry, I don't understand. There is no locale change around this test,
>> so it will fail, for example, for me. I wouldn't want to get a noisy
>> failure each time I run tests.
> 
> Certainly flaky tests must be avoided. However I can not identify the source 
> of confusion (yours or mine).
> 
> There is redefinition of `string-collate-lessp' to run tests with "C" locale:
> https://code.orgmode.org/bzg/org-mode/src/master/testing/lisp/test-org-list.el#L1207
> 
> And it works for me

Re: [PATCH] org-faq.org: Expand "What is the best setup for indenting?"

[Maxim Nikulin]

Thank you for your comments, Greg.

On 20/05/2021 01:06, Greg Minshall wrote:
> 
> patches to patches... :)  i think these are really just typos, rather
> than any useful substantial comment.
> 
> - s/is enable (the default) or not:/is enabled (the default) or not:/

The patch removed this phrase.

> - i would suggest separate tables for >= 9.5 and < 9.5.  just so the
>   differences between with/without =electric-indent-mode= are easier to
>   spot.

I have no idea how to express it in a better way. The tables are same
for Org >= 9.5 and for Org < 9.5 (modulo bugs in indentation fixed in 9.4.6).
Only default configuration evolves in time.

1. Before 9.4. (org-adapt-indentation t) electric-indent-mode disabled.
   Bugs remained unnoticed. Users were unaware that default configuration
   assumed indented text aligned to headline title.
2. 9.4. (org-adapt-indentation t) electric-indent-mode enabled.
   Hot discussions and recommendations to disable electric-indent-mode
   since even with (org-adapt-indentation headline-data) behavior was strange.
3. Recent changes. Bugs fixed, (org-adapt-indentation nil), so as earlier
   text is not indented by default. Now absence of indentation is consistent
   with settings.

> - s/C-j anywhere/C-j elsewhere/ (and, iiuc, maybe also the "unadorned"
>   [C-j] table entry should probably be [C-j elsewhere]?)
> 
> - s/lost formatting/lose formatting/
> 
> - s/and, that is/and,/

Agree.

> - s/always indent relatively/always indent relative/

Not being a native speaker, I would either leave it as is or would change
all 3 phrases attempting to make their style more uniform.

a. "do not indent anything",
   "always indent relatively to the element above",
   "indent just ~PROPERTIES/LOGBOOK~ drawers relatively
   to the current heading level"

b. "no special indentation",
   "indentation of all elements is relative to the element above",
   "only ~PROPERTIES/LOGBOOK~ drawers get indentation relative
   to the current heading level"


If you think that your variant is better, I can add it to the patch.

> - s/the the element/to the element/

Agree.

Intermediate patch is attached.
>From 1881aacf22fda650408434394a96f3aaea2c6928 Mon Sep 17 00:00:00 2001
From: Max Nikulin 
Date: Thu, 20 May 2021 22:48:58 +0700
Subject: [PATCH] org-faq.org: Fix typos and style in "setup for indenting"

* org-faq.org (What is the best setup for indenting?): Minor
corrections suggested by Greg Minshall.
---
 org-faq.org | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/org-faq.org b/org-faq.org
index 0f8934af..54d5e403 100644
--- a/org-faq.org
+++ b/org-faq.org
@@ -1039,7 +1039,7 @@ In more details, ~org-adapt-indentation~ controls indentation with
 regards to previous element.  Apparent effect is increased indentation
 for content of deeper nested headings.  The variable accepts three
 values: =nil= (no special indentation), =t= (always indent relatively
-the the element above) and =headline-data= (only indent the
+to the element above) and =headline-data= (only indent the
 ~PROPERTIES/LOGBOOK~ drawers relatively to the current level).  Value
 =t= is suitable for short entries especially if you plan to share your
 documents with someone who does not use Emacs.  If you just want to
@@ -1059,8 +1059,8 @@ With ~electric-indent-mode~ enabled:
 |---+++|
 | RET after a headline  | Do not indent  | Indent | Do not indent  |
 | C-j after a headline  | Do not indent  | Do not indent  | Do not indent  |
-| RET anywhere  | Indent wrt previous| Indent wrt previous| Indent wrt previous|
-| C-j anywhere  | Do not indent wrt prev | Do not indent wrt previous | Do not indent wrt previous |
+| RET elsewhere | Indent wrt previous| Indent wrt previous| Indent wrt previous|
+| C-j elsewhere | Do not indent wrt prev | Do not indent wrt previous | Do not indent wrt previous |
 | Insert PROPERTIES/LOGBOOK | Do not indent  | Indent wrt headline| Indent wrt headline|
 
 With ~electric-indent-mode~ disabled:
@@ -1069,21 +1069,21 @@ With ~electric-indent-mode~ disabled:
 |---+-++|
 | RET after a headline  | Do not indent   | Do not indent  | Do not indent  |
 | C-j after a headline  | Do not indent   | Indent | Do not indent  |
-| RET   | Do not indent   | Do not indent wrt previous | Do not indent wrt previous |
-| C-j   | Indent wrt previous | Indent wrt previous| Indent wrt previous|
+| RET elsewhere | Do not indent   | Do not indent wrt previous | Do not 

Re: [POLL] Setting `org-adapt-indentation' to nil by default?

[Maxim Nikulin]

On 16/05/2021 03:51, Bastien wrote:
> Maxim Nikulin writes:
> 
>> I think, the something like the following should be added to the
>> answer.
> 
> Patch welcome!  Either for Worg or etc/ORG-NEWS, if you think we
> should add something to the existing explanations.

I have tried to express my ideas as patches. Feel free to pick 
only a part of changes or discard them at all.

[PATCH] etc/ORG-NEWS: Suggest against disabling `electric-indent-mode'

[Maxim Nikulin]

---
 etc/ORG-NEWS | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index f49d2c043..8707222e0 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -55,8 +55,11 @@ If you want to automatically indent headlines' metadata, set 
it to
 If you want to automatically indent every line to the headline's
 current indentation, set it to =t=.
 
-Also beware that the behavior of =RET= and =C-j= also depends on the
-value of ~electric-indent-mode~. See 
[[https://orgmode.org/worg/org-faq.html#indentation][this FAQ]] for more 
details.
+Indent added by =RET= and =C-j= also depends on the value of
+~electric-indent-mode~.  Enabling this mode by default in 9.4 revealed
+some bugs caused confusing behavior.  If you disabled
+~electric-indent-mode~ for this reason, it is time to try it again.
+Hopefully problems have been fixed.  See 
[[https://orgmode.org/worg/org-faq.html#indentation][this FAQ]] for more 
details.
 
 *** ~org-speed-commands-user~ is obsolete, use ~org-speed-commands~
 
-- 
2.25.1

[PATCH] org-faq.org: Expand "What is the best setup for indenting?"

[Maxim Nikulin]

---
 org-faq.org | 61 ++---
 1 file changed, 44 insertions(+), 17 deletions(-)

diff --git a/org-faq.org b/org-faq.org
index 9cc193c4..0f8934af 100644
--- a/org-faq.org
+++ b/org-faq.org
@@ -1029,21 +1029,33 @@ something like this:
:CUSTOM_ID: indentation
:END:
 
-Indentation is a matter of personal preferences.  General indentation
-preferences are controlled in Emacs via ~electric-indent-mode~.  Org
-indentation behavior changes depending on ~org-adapt-indentation~, which
-accepts three values: =nil= (no special indentation), =t= (always indent
-relatively the the element above) and =headline-data= (only indent the
-~PROPERTIES/LOGBOOK~ drawers relatively to the current level).  On top
-of these two configuration areas, there the difference between =RET= and
-=C-j=.
-
-Here are two tables summarizing the behavior depending on whether
-~electric-indent-mode~ is enable (the default) or not:
-
-With `electric-indent-mode' enabled:
-
-| org-adapt-indentation =>  | nil| t   
   | headline-data  |
+Indentation, both appearance and behavior, is a matter of personal
+preferences.  You may try if the following adjustments suits better
+for you than the defaults.  Set ~org-adapt-indentation~ to have
+content aligned to headline titles.  Disable ~electric-indent-mode~ to
+avoid automatic indentation in response to =RET= key.
+
+In more details, ~org-adapt-indentation~ controls indentation with
+regards to previous element.  Apparent effect is increased indentation
+for content of deeper nested headings.  The variable accepts three
+values: =nil= (no special indentation), =t= (always indent relatively
+the the element above) and =headline-data= (only indent the
+~PROPERTIES/LOGBOOK~ drawers relatively to the current level).  Value
+=t= is suitable for short entries especially if you plan to share your
+documents with someone who does not use Emacs.  If you just want to
+make heading level more prominent then consider adding visual left
+margin using =#+STARTUP: indent= as described in the 
[[https://orgmode.org/manual/Clean-View.html#Clean-View][Clean View]]
+section of the manual.  The option works without adding extra spaces
+to the document text.
+
+Configured indentation may be applied in response to =RET= or to
+=C-j= depending on the state of ~electric-indent-mode~.  The following
+tables summarizes the difference.  Version number is added to column
+header if it describes default settings.
+
+With ~electric-indent-mode~ enabled:
+
+| org-adapt-indentation =>  | nil (Org >= 9.5)   | t (Org 9.4) 
   | headline-data  |
 
|---+++|
 | RET after a headline  | Do not indent  | Indent  
   | Do not indent  |
 | C-j after a headline  | Do not indent  | Do not indent   
   | Do not indent  |
@@ -1051,9 +1063,9 @@ With `electric-indent-mode' enabled:
 | C-j anywhere  | Do not indent wrt prev | Do not indent wrt 
previous | Do not indent wrt previous |
 | Insert PROPERTIES/LOGBOOK | Do not indent  | Indent wrt headline 
   | Indent wrt headline|
 
-With `electric-indent-mode' disabled:
+With ~electric-indent-mode~ disabled:
 
-| org-adapt-indentation =>  | nil | t  
| headline-data  |
+| org-adapt-indentation =>  | nil | t  
| headline-data (Org < 9.4)  |
 
|---+-++|
 | RET after a headline  | Do not indent   | Do not indent  
| Do not indent  |
 | C-j after a headline  | Do not indent   | Indent 
| Do not indent  |
@@ -1061,6 +1073,21 @@ With `electric-indent-mode' disabled:
 | C-j   | Indent wrt previous | Indent wrt previous
| Indent wrt previous|
 | Insert PROPERTIES/LOGBOOK | Do not indent   | Indent wrt headline
| Indent wrt headline|
 
+Do not try to avoid or to ignore indentation of heading body or
+properties drawer determined by current state of
+~org-adapt-indentation~ and ~electric-indent-mode~ by pressing =C-j=
+instead of =RET= (or vice versa). The result is transient and you will
+lost formatting when you refile heading or change its level (promote
+or demote it).
+
+You may have noticed recommendation to disable ~electric-indent-mode~
+to restore behavior prior to Org 9.4.  In Org 9.5
+~org-adapt-indentation~ default value changed to =nil= and, that is
+more important, a number of bugs related to indentation were fixed.
+Using =RET= with enabled ~electric-indent-mode~ should be convenient
+now.  Just customize ~org-adapt-indentation~ variable 

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

It seems, there is no reliable way to work with numbers in a 
locale-aware way in emacs.  I am still against hard-coded list of 
locales. Requirement to customize a variable is rather inconvenient. 
Considering such properties as a part of translation is a little better 
but I prefer to avoid it.


The question may be risen in emacs-devel but I am unsure if I will 
participate in discussion.


On 18/05/2021 17:24, Utkarsh Singh wrote:

On 2021-05-16, 23:24 +0700, Maxim Nikulin wrote:


On 14/05/2021 21:54, Utkarsh Singh wrote:

On 2021-05-13, 00:08 +0700, Maxim Nikulin wrote:


Comma is decimal separator for es_ES, de_DE, ru_RU, etc. The point is
that order in which separator candidates are tried should depend on
active locale.


I am willing to work on this problem but before this can you identify
any other locale with similar problem or a resource from where I can
information's about locale.


I do not think list of locales should be hard coded. I am not familiar
with elisp enough to tell you if locale properties such as decimal
separator are exposed through some function. I expect, it is quite
probable. As a last measure, some number, e.g. 0.5 or 1.5 could be
formatted using a locale-aware function and result string could be
checked if it contains ",".


Can you test this function:

(defun org-table--comma-as-decimal-sep ()
   "Return nil or 2 if separator is dot or comma respectively."
   (string-search "," (format "%f" 10)))


No, it does not work. `format' always uses dot. It is reasonable when 
e.g. during writing a config file or during data exchange when locales 
must be ignored.


I was too optimistic. I did not expect that support of locales are so 
poor in Emcacs. I do not see any traces of localeconv(3) in sources that 
would allow to get value of decimal_point directly.


Numbers are forced to use "C" locale and I have not noticed any way to 
override it. Initial settings:


http://git.savannah.gnu.org/cgit/emacs.git/tree/src/emacs.c#n1490

http://git.savannah.gnu.org/cgit/emacs.git/tree/src/emacs.c#n2861

setlocale (LC_NUMERIC, "C");


To test I am using:
$ LANG=de_DE.UTF-8 emacs -Q

But I am getting this as warning:
(process:1787): Gtk-WARNING **: 15:40:49.375: Locale not supported by C library.
Using the fallback 'C' locale.


You get this error due to you have not generated this locale. On debian 
& ubuntu


dpkg-reconfigure locales

allows to select desired locales and performs all necessary actions.

Re: [PATCH] Possibility of using alternative separators in macros

[Maxim Nikulin]

On 17/05/2021 02:21, Christian Moe wrote:

Maxim Nikulin writes:

On 03/05/2021 04:08, Christian Moe wrote:

[snip]

#+MACRO: allargshack (eval (format "- /%s/ :: %s" $1 (mapconcat
#'identity _ ",")))

{{{allargshack(one, two, three)}}}

I do not know if Eric can swap order of arguments of his credits
macro. Extracting namely last argument requires a bit more lisp code.


Yes, I didn't think that far. This would provide a comprehensive
backwards-compatible solution to the comma-escaping problem, though
perhaps not the most newbie-friendly one. It would also make macros more
flexible and powerful in the bargain (I'm sure people will think of
other uses for this than commas).


I agree that it would abuse arguments syntax, but I expect that namely 
newbies would not bother since it would "just work":


#+MACRO definition  - $1 ::$_

{{{definition(one, two, three}}}

It is more experienced users who may be confused why it works.

Re: [PATCH] Possibility of using alternative separators in macros

[Maxim Nikulin]

On 16/05/2021 19:17, Bastien wrote:


Juan Manuel Macías writes:


(On the other hand, maybe better than an alternate separator, some kind of
warning for unescaped commas might be more useful, as Maxim commented
here: https://orgmode.org/list/s7g...@ciao.gmane.io/)


Yes, probably -- feel free to propose a patch for this.  Thanks!


Such warnings should be property of particular macros. Sometimes 
ignoring of arguments may be handy. So no patch is required. The trick 
could be documented somewhere, but I am unsure concerning proper place. 
Actually, I do not think, fatal error is user-friendly behavior. I am 
not aware if export already have convenient facilities to report 
warnings. Currently I do not feel I am ready to implement such feature 
if it does not exist yet.


However the point of that message was that extra commas may be made 
"transparent" for macros by introducing additional substitution, e.g. 
"$_" that expands into "rest" arguments. I consider "$@" or "$*" as 
worse variant since it rather mean "all arguments", so it is less 
flexible. For "eval" lisp macros, it is just replacing "_" by "$_" in 
argument list. Simple text macros require a bit more work but it is 
still comparable with documenting the feature.


We need a decision if "rest arguments" feature should be introduced. 
Once added, it will be harder to discard it due to compatibility issues.

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 14/05/2021 21:54, Utkarsh Singh wrote:

On 2021-05-13, 00:08 +0700, Maxim Nikulin wrote:


Comma is decimal separator for es_ES, de_DE, ru_RU, etc. The point is
that order in which separator candidates are tried should depend on
active locale.


I am willing to work on this problem but before this can you identify
any other locale with similar problem or a resource from where I can
information's about locale.


I do not think list of locales should be hard coded. I am not familiar 
with elisp enough to tell you if locale properties such as decimal 
separator are exposed through some function. I expect, it is quite 
probable. As a last measure, some number, e.g. 0.5 or 1.5 could be 
formatted using a locale-aware function and result string could be 
checked if it contains ",".

Re: [Patch] to correctly sort the items with emphasis marks in a list

[Maxim Nikulin]

On 15/05/2021 20:32, Bastien wrote:

Bastien  writes:


Bastien  writes:


Thanks for the heads-up.  I reverted the commit.  Please go ahead with
whatever you see fit.


PS: Re-opening this work-in-progress patch for updates.orgmode.org.


I'm closing this now -- Juan, feel free to resubmit a patch if you
are still working on this.


Bastien, the following message in this thread contains alternative draft 
from Nicolas:

https://orgmode.org/list/87a6pt9hyd@nicolasgoaziou.fr/

It is better than initial patches from Juan Manuel. It fixes the problem 
with removing characters around emphasized text and prepending a space 
instead. As a result it fixes sorting items in es_ES locale.

Re: [PATCH] org-refile.el: Fix the case of emtpy buffer name

[Maxim Nikulin]

In my opinion, patch is the minimal change that fixes particular 
workflow and can be committed in the current form. Commit message may be 
improved a bit.


I have a question (mainly to maintainer) if another approach could lead 
to undesired effects, see below.


I have noticed a couple of old issues that can be improved later since 
they may require more changes than allowed for "TINYCHANGE".


On 14/05/2021 01:20, satotake wrote:

[PATCH] org-refile.el: Fix the case of emtpy buffer name


Buffer not associated with a file may be more precise since "*scratch*" 
has name.



* lisp/org-refile.el (org-refile-get-targets): Ensure
arg of `file-name-non' and `file-truename' is non-nil.


truncated function name

Comments below a loosely related to the suggested patch.


--- a/lisp/org-refile.el
+++ b/lisp/org-refile.el
@@ -310,11 +310,13 @@ converted to a headline before refiling."
 (setq f (buffer-file-name (buffer-base-buffer f
   (setq f (and f (expand-file-name f)))
   (when (eq org-refile-use-outline-path 'file)
-(push (list (file-name-nondirectory f) f nil nil) tgs))
+(push (list (and f (file-name-nondirectory f)) f nil nil) tgs))
   (when (eq org-refile-use-outline-path 'buffer-name)
 (push (list (buffer-name (buffer-base-buffer)) f nil nil) tgs))
   (when (eq org-refile-use-outline-path 'full-file-path)
-(push (list (file-truename (buffer-file-name 
(buffer-base-buffer))) f nil nil) tgs))


Notice `file-truename'


+(push (list (and (buffer-file-name (buffer-base-buffer))
+  (file-truename (buffer-file-name 
(buffer-base-buffer
+ f nil nil) tgs))
   (org-with-wide-buffer
(goto-char (point-min))
(setq org-outline-path-cache nil)
@@ -337,9 +339,10 @@ converted to a headline before refiling."
#'identity
(append
 (pcase org-refile-use-outline-path
-  (`file (list (file-name-nondirectory
-(buffer-file-name
- (buffer-base-buffer)
+  (`file (list
+   (and (buffer-file-name 
(buffer-base-buffer))


I hope, additional operation in the inner loop has minimal performance 
penalty. Additional variable may be introduced and initialized before 
the loop over headings. It may help to avoid discrepancy similar to 
`file-truename'.



+(file-name-nondirectory
+ (buffer-file-name 
(buffer-base-buffer))
   (`full-file-path
(list (buffer-file-name
   (buffer-base-buffer


Earlier patch, that added `file-truename' above, missed this point. At 
first I was surprised why this clause does not require modification.


To maintainers:
What are negative consequences of completely skipping of buffers that 
have no associated files? I mean not to add them to the list for 
iteration. I can anticipate some tests should be fixed but I do not 
think it is a problem. Anyway `org-goto' and `org-refile' do not work in 
such buffers (from my point of view, reaction to such issue should be 
saving content of buffer to some file since critical size of notes is 
achieved):


On 14/05/2021 01:20, satotake wrote:

In addition to your point, I found that we cannot refile internally even with

my patch. In other words, if we can cache and reuse it, error ("Please 
indicate a target file in the refile path") is raised

when we select it as refile target.
Probably, we need to some additional fixations.
It may be good to filter `files' which does not have `buffer-file-name'.


Have you tried org-capture? 

Yes. Mostly I use org-capture, especially, for creating TODO tasks.
I sometimes start with fundamental-mode new buffer.
After writing some texts, I switch to org-mode and try to call refile.
I do not know why I do it by myself clearly but I tend to do it when I
do not have any clear goal for the file.


Thank you for details. I think, it is a kind of valid workflow. 
Personally I would consider it unreliable however. I suppose, 
`org-default-notes-file' (default capture target) better suited for 
temporary notes since it minimizes risk of lost text in the case of 
failure. It requires to change habits a bit to call org-capture with 
simple template in advance.

Re: URLs with brackets not recognised

[Maxim Nikulin]

On 13/05/2021 03:06, Rudolf Adamkovič wrote:

Maxim Nikulin writes:


I do not think it is a bug. Plain text links detection is a kind of
heuristics. It will be always possible to win competition with regexp.
Consider it as a limitation requiring some hints from an intelligent
user.


I disagree.


Me too. I disagree with most of statements in this thread, even with 
some arguments supposed to support my opinion. Exception is Ihor's 
message. I hope, more liberal regexp will not interfere with parsing of 
other constructs.


Actually I think, you do not realize that detection of URLs in arbitrary 
text is tricky. Maybe you have not noticed corner cases before.


False positives may be even more annoying. At least in the past "smart" 
detection of smiles and emoji in skype transformed code snippets into 
unreadable mess of "glasses of wine" and other "funny" stuff.



URLs are well-specified. Per RFC 3986,


It describes isolated URI assuming some protocol that allows to 
determine begin and end of URI string. It is impossible to unambiguously 
extract URLs from text written in human languages. Tom pointed that some 
character sequences in URLs can interfere with org markup.



the characters
allowed in a URL are [A-Za-z0-9\-._~!$&'()*+,;=:@\/?].


1. Surrounded text may use the same characters. I do not think, you 
would be happy if you got


- <https://orgmode.org/,>
- <https://orgmode.org/worg/org-faq.html)>

from

"(see https://orgmode.org/, https://orgmode.org/worg/org-faq.html)"

just because of "," and ")" characters are allowed in URIs. There is 
just some heuristics that works more or less acceptable in common cases. 
Various implementation have their strong and weak sides.


2. Allowed characters are specified at protocol level. Fortunately in 
user interface most of unicode characters are allowed.


Certainly the following URLs are more portable and reliable
https://el.wikipedia.org/wiki/%CE%9B%CE%AC%CE%BC%CE%B4%CE%B1
https://ja.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC
https://ru.wikipedia.org/wiki/%D0%A1%D1%82%D0%BE%D0%BB%D0%BB%D0%BC%D0%B0%D0%BD,_%D0%A0%D0%B8%D1%87%D0%B0%D1%80%D0%B4_%D0%9C%D1%8D%D1%82%D1%82%D1%8C%D1%8E#%D0%9A%D1%80%D0%B0%D1%82%D0%BA%D0%B0%D1%8F_%D0%B1%D0%B8%D0%BE%D0%B3%D1%80%D0%B0%D1%84%D0%B8%D1%8F
However unicode variants are more informative and readable for humans
https://el.wikipedia.org/wiki/Λάμδα
https://ja.wikipedia.org/wiki/日本
https://ru.wikipedia.org/wiki/Столлман,_Ричард_Мэттью#Краткая_биография

The same is applicable for domain names. Extreme case: 
https://xn--i-7iq.ws/ - https://i❤️.ws/


Even space characters can be used in query part. Modern applications are 
able to convert them to "+" or to "%20" for communication with HTTP servers.



Org mode should
implement proper URL detection, not asking its users "to give it some
hints" and using "a kind of heuristics".


Some tools detect www.google.com as valid URL, others (including org) do 
not. Heuristics can evolve in time. Org render on github can differ from 
elisp original code. Explicit markup is a way to avoid problems.


More complicated regexp makes it harder to support it. (Explaining to 
user that technologies have limitations is a kind of maintenance cost as 
well). Long regexp will have performance penalty and still can be fooled.


Example of link that causes problems even with brackets:
https://lists.gnu.org/archive/html/emacs-orgmode/2020-12/msg00706.html
https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#metricsV2:graph=~(view~'timeSeries~stacked~false~metrics~(~(~'CWAgent~'backup_time~'host~'desktop~'metric_type~'timing))~region~'us-east-1);query=~'*7bCWAgent*2chost*2cmetric_type*7d

On 12/05/2021 23:44, Colin Baxter wrote:

It might be worthwhile to issue an warning each time a url is written in
an org file without enclosing brackets < > or [[ ]].


Simple links works well. I am afraid that detecting, whether a 
particular link is a corner case that needs brackets, may require more 
complicated logic than regexp detecting links.


On 13/05/2021 09:21, Tim Cross wrote:
> As this is defined and documented behaviour,

My impression that nuances of recognition of plain text links are not 
documented. Even unit tests exists only in the proposed patch. Actually 
I do not think that such details are necessary in the manual. 
Fontification provides feedback. As soon as problems noticed, explicit 
marks can be added.


On 13/05/2021 05:23, Tom Gillespie wrote:

A quick fix is to percent encode the troublesome characters


org-lint does not like percent encoding in links. It is heritage of a 
period when *extra* pass of percent encoding was used to escape square 
brackets and spaces. Current recommendation is to escape only brackets 
and backslashes leaving spaces as is (however org-fill-paragraph 
believes that it has full rights to do something with spaces).


Per

Re: bug#47885: [PATCH] org-table-import: Make it more smarter for interactive use

[Maxim Nikulin]

On 11/05/2021 01:36, Utkarsh Singh wrote:

What do we mean by interactive preview?  Does this mean that we should
present a user with a list of possible delimiters using minibuffer?


I mean something like the dialog that LibreOffice shows on opening of a 
csv file. There are various options and table preview that allows to 
check if options are selected correctly before dropping to usual 
spreadsheet interface. I have no idea what is the best way to implement 
something like this in emacs. Likely it is out of scope of the discussed 
patch. I do not know what is your use cases. My intention was to show 
that CSV import could be quite cumbersome.


Some users believe that CSV is a reliable portable format and expect 
support of most features from Excel. Actually there are plenty of 
dialects and no way to determine which one was used (e.g. no header that 
defines field or string separator).



For ex (please review my usage of alist):

#+begin_src elisp
(defvar my-separator-alist '(("comma" . ",")
 ("tab" . "\t")
 ("semicolon" . ";")
 ("colon" . ":")
 ("space" . " ")))

(defun my/table-import (file separator)
   (interactive (list (read-file-name "Import CSV file: " nil nil t)
 (cdr (assoc (completing-read "Separator: " 
my-separator-alist)
 my-separator-alist
   (org-table-import file separator))
#+end_src


With such function it is necessary to open file at first to see what 
separator is used inside.


My comments were related to the "guess" part of your patch. Comma is 
tried first. Consider the following file


A;1,2;3,4;5,6
B;7,8;9,1;11,12
C;13,14;15,16;17,18

Decimal separator is ",". Field separator is ";" but there are plenty of 
"," in each row.


LANG=fr_FR.UTF-8 python3 -c "import locale as l; l.setlocale(l.LC_ALL, 
''); print(l.format_string('%.2f', 123456.789))"

123456,79

Comma is decimal separator for es_ES, de_DE, ru_RU, etc. The point is 
that order in which separator candidates are tried should depend on 
active locale.


I do not insist that interactive preview or smarter approach to guess 
separator have to be implemented. Feel free to disregard my comments. I 
am just not sure whether you are aware of limitations for noticeable 
part of users.

Re: URLs with brackets not recognised

[Maxim Nikulin]

On 12/05/2021 17:38, Nick Savage wrote:

I can confirm this as a bug.


> On 5/12/21 3:32 AM, Colin Baxter wrote:
https://en.wikipedia.org/wiki/Cathedral_Basilica_of_St._John_the_Baptist_(Savannah,_Georgia) 


I do not think it is a bug. Plain text links detection is a kind of 
heuristics. It will be always possible to win competition with regexp. 
Consider it as a limitation requiring some hints from an intelligent user.


Do the following links have the same problem?

- 

- 
[[https://en.wikipedia.org/wiki/Cathedral_Basilica_of_St._John_the_Baptist_(Savannah,_Georgia)]]


Support of parenthesis exists, but it is not perfect. Punctuation often 
should be outside of URL.

Re: [PATCH] Possibility of using alternative separators in macros

[Maxim Nikulin]

On 03/05/2021 04:08, Christian Moe wrote:


I frequently need to escape commas in macros, which is a bit of a pain
and easy to forget.


Maybe it is not convenient, but if unescaped comma is a real pain, you 
could detect it and report an error


# single line may be wrapped by mailer
#+MACRO: extraerror (eval (if (not $2) (concat "*" $1 "*") (error 
(format "%s: unescaped comma %S" (line-number-at-pos) $2


{{{extraerror(valid)}}}
{{{extraerror(valid\, with escaped comma)}}}
{{{extraerror(missed, comma)}}}

Org gurus might suggest a recipe how to convert error into warning, that 
is easily noticeable but still not fatal, to get all problems after 
single export attempt. Preferably it should act similar to compiler 
errors allowing to jump between problem points.


Org 9.3 requires a bit different macro
#MACRO: extraerror (eval (if (equal "" $2) (concat "*" $1 "*") (error 
(format "%s: unescaped comma %S" (line-number-at-pos) $2



Another point: Something that would help, without adding new syntax, is
making macro expansion smart enough to *ignore* separators when the
macro definition contains only *one* argument anyway, as in the cases
above.


I think, this is an idea of the best approach. Unsure concerning precise 
form. Maybe e.g. "$_" could expand into all arguments greater than 
maximum referenced number. No promise of forward compatibility of the 
following hack since it relies on undocumented implementation details.


#+MACRO: allargshack (eval (format "- /%s/ :: %s" $1 (mapconcat 
#'identity _ ",")))


{{{allargshack(one, two, three)}}}

I do not know if Eric can swap order of arguments of his credits macro. 
Extracting namely last argument requires a bit more lisp code.

Re: [PATCH 2/2] org-refile.el: Fix test case

[Maxim Nikulin]

On 09/05/2021 23:05, satotake wrote:

* lisp/org-refile.el (org-refile-get-targets): Check
`buffer-file-name' return value instead of `buffer-base-buffer'.

To pass the related tests, we need to check `buffer-file-name'.


Thank you for the patches. Maybe I have missed something, but it seems 2 
patches are not logically independent, second one touches the same code 
to fix regression in tests. If so, could you, please, squash both 
patches into a single one (e.g. using git rebase --interactive)? I hope, 
it will help to review the change and to avoid confusion with several 
versions of second patch in 2 threads.


Code around has at least one more problem: questionable behavior in the 
case of indirect buffers (e.g. C-x 4 c). Refile targets cache, when 
enabled, is not reused for indirect buffer.


On 09/05/2021 21:47, satotake wrote:

I often use org-mode for note-taking and so on.
Sometimes, I would like to refile the headings in the scratch to other
org files while I am writing.


Have you tried org-capture? https://orgmode.org/manual/Capture.html
https://sachachua.com/blog/2015/02/learn-take-notes-efficiently-org-mode/#unnumbered-3
I think, existence of this feature is the reason why this bug was 
remaining unnoticed.

Re: [PATCH] Use cache in org-up-heading-safe

[Maxim Nikulin]

On 07/05/2021 09:08, Ihor Radchenko wrote:

Maxim Nikulin writes:

Did you just replace gethash by avl-tree?


Yes


Likely my idea is based on a
wrong assumption. I hoped that having positions of headers it is
possible to avoid jumps (goto-char ...) preceded or followed by regexp
matching almost completely. Previous header for arbitrary initial
position can be found using binary search through structure obtained
during scan.


Sorry, I cannot follow what you mean. The call to goto-char in
org-up-heading-safe is required by function docstring - we need to move
point to the parent heading.


I am trying to minimize number of regexp searches. Mostly it is applied 
when information concerning multiple headings is required (agenda, 
refile targets). It unlikely will get some benefits during interactive 
calls related to single heading.


For a file having 3000 headings, scanning though it takes ~0.1sec to get 
the following properties: position, level, list of parents (with same 
properties). Note that no expensive operations are performed like 
cleaning up of heading title.


Having list of headings (and its length), it is possible to build a tree 
for binary search in linear time. It takes ~0.01sec.


Having the tree, it is possible to instantly find heading for 
*arbitrary* position in the buffer. Likely the next operation is goto to 
the heading or to it parent and parsing the line for more detailed 
properties. The latter is cacheable, structure for heading properties 
can be expanded.


Hash works only for fixed set of positions, to use hash it is necessary 
to find position of the heading at first. On the other hand, to take 
advantage of binary tree, more substantial modification of code is required.


Since there is no operations as insertion or deletion of nodes from 
tree, no complex code is required to implement balancing rotations. That 
is why I think that avl-tree is an overkill.


See the attachment for experimental (thus almost untested) code. Likely 
you will find code style quite ugly. I am not happy with 0.1 sec for a 
moderately large file. It is close to the limit for comfortable 
interactive operations.



+   (re-search-backward
+ (format "^\\*\\{1,%d\\} " level-up) nil t)
+   (funcall outline-level


Unsure concerning the following optimization from the point of
readability and reliability in respect to future modifications. Outline
level can be derived from the length of matched string without the
funcall requiring extra regexp.


I am not sure here. outline-level also consults outline-heading-alist,
though I found no references to that variable in Org mode code.
Otherwise, outline-level already reuses match-data. There is no regexp
matching there.


Sorry. You are right. The function does what I proposed to write 
explicitly. For some reason I believed that outline-level calls 
something like looking-at. Maybe I checked it earlier and completely forgot.



#+begin_src elisp :results none

  ;; Heading properties
  (defun nm-heading-properties-new (position level parents)
"Heading properties: (position . (level . parent))"
(cons position (cons level parents)))

  (defun nm-heading-properties-level (props)
(cadr props))

  (defun nm-heading-properties-pos (props)
(car props))

  (defun nm-heading-properties-parents (props)
(cddr props))

  (defun nm-heading-pos-lessp (value props)
(< value (nm-heading-properties-pos props)))

  (defun nm-buffer-headings-reversed (buffer)
(with-current-buffer buffer
  (save-restriction
(save-excursion
  (widen)
  (goto-char (point-min))
  (let ((count 0)
(headings ())
(parents ()))
(while (re-search-forward org-outline-regexp-bol nil t)
  (let* ((pos (match-beginning 0))
 (level (- (match-end 0) pos 1)))
(while (and parents (>= (nm-heading-properties-level (car 
parents)) level))
  (pop parents))
(setq count (1+ count))
(let ((props (nm-heading-properties-new pos level parents)))
  (push props headings)
  (push props parents
(and headings (cons headings count)))


  ;; binary search tree
  (defun nm-btree-new-node ()
"((left right) . properties"
(cons (cons nil nil) nil))

  (defun nm-btree-node-left (node)
(caar node))

  (defun nm-btree-node-set-left (node child)
(setcar (car node) child))

  (defun nm-btree-node-set-right (node child)
(setcdr (car node) child))

  (defun nm-btree-node-right (node)
(cdar node))

  (defun nm-btree-node-properties (node)
(cdr node))

  (defun nm-btree-node-set-properties (node properties)
(setcdr node properties))

  (defun nm-btree-from-reversed (scan-result)
(and
 scan-result
 (let* ((key-properties-list (c

Re: [PATCH] Use cache in org-up-heading-safe

[Maxim Nikulin]

Though I still have not tested the patch, I think, it is an improvement 
and it is helpful in its current form. I am unable to tell if it follows 
code style.


Despite continuing discussion, I am unsure if it could be significantly 
better.


On 06/05/2021 21:34, Ihor Radchenko wrote:

Maxim Nikulin writes:

In org-agenda.el org-up-heading-safe function is called only from
org-find-top-headline.


That's probably not the slowest part. org-agenda also calls
org-up-heading-safe indirectly. In particular, org-get-tags is calling
org-up-heading-safe to get inherited tags. It is org-get-tags that is
taking >50% time of agenda generation in my agendas. And
org-up-heading-safe was the largest contributor to that >50% time.


My bad, you mentioned tags earlier, but I grepped org-agenda.el only.
My new idea is that org-get-tags may have its own cache as well. Unsure 
if it should be tried.



Scan through the whole buffer could be faster, but it is not always
desired. Most of Org code only need information for current headline.
Re-scanning the whole buffer would be costly.

Also, I tried to compare avl-tree with hash table. However, avl-tree
does not give much benefit for my test case of an agenda with ~12000
todo items. Since avl-tree requires much more custom code, hash table
should be better.


Did you just replace gethash by avl-tree? Likely my idea is based on a 
wrong assumption. I hoped that having positions of headers it is 
possible to avoid jumps (goto-char ...) preceded or followed by regexp 
matching almost completely. Previous header for arbitrary initial 
position can be found using binary search through structure obtained 
during scan.



+   (re-search-backward
+ (format "^\\*\\{1,%d\\} " level-up) nil t)
+   (funcall outline-level


Unsure concerning the following optimization from the point of 
readability and reliability in respect to future modifications. Outline 
level can be derived from the length of matched string without the 
funcall requiring extra regexp.


If originally this code path had 50% contribution and performance 
already becomes several times better, further optimization of this part 
does not matter.

Re: About multilingual documents

[Maxim Nikulin]

On 05/05/2021 01:55, Aleksandar Dimitrov wrote:

Yeah, I know the issue, which is why I rely on XCompose for Latin
scripts. For Cyrillic, alas, that is impossible. It means that I
basically can't control Emacs while using a Cyrillic layout, which is a
pity. I have no good workaround.


Generally, the idea is to enable layout (Xkb group) per window and to 
reset layout to English if active window is Emacs. I have not tried 
recipes with managing Xkb group from emacs itself, e.g.

https://github.com/lislon/emacs-switch-lang
https://github.com/Mihara/kbd-indicator.el

Another approach it to set global hotkey and if Emacs is focused, send 
some special key event that is bound to switching of input method. I 
have some links but the pages are not in English. Personally, I have not 
fully polished my setup, however it works with some limitations. I 
started from bash script calling xdotool, xvkbd, and xprop. Then I 
realized that C code is not dramatically longer but it allows to avoid 
struggling with limitations of such tools.


Tim Cross suggested me to raise the question concerning keymaps in 
emacs-devel once more, but I still do not feel that I am ready to 
discussion of technical aspects (e.g. hotkey handling in applications 
that fixed similar issues several years ago)

https://orgmode.org/list/87r1lnvjh0@gmail.com

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 06/05/2021 18:53, Bruce D'Arcus wrote:


Your example should just use the default org-cite citation, without
any style or sub-style.

The rest would be handled by latex/bibtex.

Right?


Yes, simple \cite{k1,k2} is mostly enough for numeric citations. E.g 
cases like separated list of references to author's works are handled by 
a customized \begin{thebibliography} and dedicated counter having some 
prefix.


There is might be some uncertainty concerning handling of prefixes and 
suffixes. However my impression is that even e.g. \cite[p.~7]{key} is 
quite rare. For completeness, keys without prefixes, suffixes could be 
combined into single \cite{key1,key2}, otherwise individual \cite{key1} 
may be required with additional text in between.

Re: [POLL] Setting `org-adapt-indentation' to nil by default?

[Maxim Nikulin]

Bastien, thank you for the fixes of electric-indent-mode, there is no 
feeling that it is necessary to choose between broken and inconvenient 
configuration options any more.


On 03/05/2021 15:06, Bastien wrote:


This might help: https://orgmode.org/worg/org-faq.html#indentation

"What is the best setup for indenting?"

I think, the something like the following should be added to the answer.
It was not obvious for me at first. Gustavo explained it in 
https://orgmode.org/list/87blfxv966@gmail.com/


Do not try to avoid or ignore indentation of heading body or properties 
drawer determined by current org-adapt-indentation and 
electric-indent-mode by pressing C-j instead of RET (or vice versa). It 
is unsure way. When you refile heading or change its level (promote or 
demote it), you may find that despite your efforts, elements are 
indented accordingly to Org mode current settings instead of your visual 
preferences. It is better to customize org-adapt-indentation.


I would suggest to mention =#+STARTUP: indent= as well as a visual 
alternative to (org-adapt-indentation t) that actually cancels its effect.


Maybe it should be stressed in the ORG-NEWS file that previously there 
were suggestion to set electric-indent-local-mode to -1 for Org buffers. 
Now it is hopefully not necessary due to bug fixes and changed defaults.


Finally, a case that might be fixed.

- item 1
  + item 2
  +← cursor is here

In this case TAB allows to cycle through indentation variants and it is 
great. Unfortunately just after RET (there are only some spaces on the 
new line) TAB fixes indentation as continuation of previous item and 
does not allow to shift left before marker is added. It is confusing at 
first since e.g. python mode is more liberal.


Is there equivalent of TAB for indentation cycle when some item text is 
added since TAB is busy for switching of item visibility?

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 06/05/2021 00:16, Bruce D'Arcus wrote:

On Wed, May 5, 2021 at 12:59 PM Maxim Nikulin wrote:

On 04/05/2021 23:59, Nicolas Goaziou wrote:


Is the default \cite{key} command (without any other package) used? I'm
not sure we should provide it since we are working towards more complete
solutions.


In some fields simple "[3,7]" citations are traditional.


How do you achieve such output, with which formats, packages, commands?


It is default behavior. Minimal example:

--- 8< ---
\documentclass{article}
\begin{document}
New threads related to citations in Org~\cite{ml_natbib,ml_cite_org3}.

\begin{thebibliography}{9}
\bibitem{ml_cite_org3} Nicolas Goaziou. Notes about citations in Org 
(part 3).
\bibitem{ml_cite_status} Bruce D'Arcus. wip-cite status question and 
feedback.

\bibitem{ml_natbib} Nicolas Goaziou. [wip-cite-new] New natbib processor.
\end{thebibliography}
\end{document}
--- 8< ---
New threads related to citations in Org [3, 1].

References
[1] Nicolas Goaziou. Notes about citations in Org (part 3).
[2] Bruce D’Arcus. wip-cite status question and feedback.
[3] Nicolas Goaziou. [wip-cite-new] New natbib processor.
--- 8< ---

Additional packages are required to sort citations or to replace 
sequence with range.



For example, do you get that with the default \cite command in latex,
assuming the right bst file?


Do you think something is wrong with such citations? Papers may be 
published without using of BibTeX at all. "Right" bst file depends on 
particular journal. Even between styles, that generate simple 
\bibitem{key} causing numerical citations, difference may be rather 
significant in respect to what parts of entry are formatted, in which 
order how they are emphasized, what is omitted completely and what is 
shortened.

Re: [PATCH] Use cache in org-up-heading-safe

[Maxim Nikulin]

On 04/05/2021 22:08, Ihor Radchenko wrote:


While testing another patch for agenda fontification, I noticed that
agenda can spend up to half!! time doing org-up-heading-safe. Mostly
inside queries for inherited tags and properties.

I managed to make org-up-heading-safe up to 50x faster using position
cache.


I have not tested the patch due to I do not use agenda. My interest was 
stimulated solely by my attempts to make org-refile-get-targets faster.


I consider it as an improvement. I have noticed the only thing that must 
be fixed: comments with old variant of the function. Reaction to my 
other comments is optional.


In org-agenda.el org-up-heading-safe function is called only from 
org-find-top-headline. So the purpose of the dance with backward 
searches is to get top level headings. My expectation is that scan 
through the whole buffer and storing result in a structure that allows 
binary search of position (array, red-black tree, etc) may be even faster.


Alternatively lazily obtained position of top heading could be stored in 
cache to reduce number of iterations on org-find-top-line.



+(let ((level-cache (gethash (point) org--up-heading-cache)))
+  (if (and level-cache
+   (eq (buffer-chars-modified-tick) org--up-heading-cache-tick))


If buffer-chars-modified-tick is faster than gethash then reordering 
might result in very slight improvement of performance.



+;; Parent is inside accessible part of the buffer.
+(progn (goto-char level-cache)
+   (funcall outline-level)))


I do not see any reason why outline level can not be cached in pair with 
position.



+  (let (result)
+(setq result


I am not a lisp guru, so from my point of view this can be reduced to
(let ((result ...


+(format "^\\*\\{1,%d\\} " level-up) nil t)


\t as an alternative to " " is used in org-refile.el,
e.g. "^\\*+[ \t]+". Unsure which variant is canonical one and I see that 
regexp is taken from original function, so no regression is expected.



+;; (defun org-up-heading-safe ()
+;;   "Move to the heading line of which the present line is a subheading.


Clean-up here is required, other notes may be ignored.

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 04/05/2021 23:59, Nicolas Goaziou wrote:


Is the default \cite{key} command (without any other package) used? I'm
not sure we should provide it since we are working towards more complete
solutions.


In some fields simple "[3,7]" citations are traditional. A couple of 
randomly picked examples:

https://www.osapublishing.org/oe/fulltext.cfm?uri=oe-29-10-15413=450715
https://arxiv.org/abs/2105.01093

Personally, I do not mind to have a hint (e.g. a tooltip) if format 
allows such features, e.g.
[13]

Re: About multilingual documents

[Maxim Nikulin]

On 04/05/2021 14:30, Aleksandar Dimitrov wrote:


I don't usually switch input methods. Instead I rely on the X-Server's
facilities, including group toggles and XCompose. For example I use
XCompose to write all languages with a Latin alphabet without having to
switch layouts/input methods.


You mentioned Cyrillic, and it is inconvenient to switch keyboard layout 
(Xkb group) for any command (C-c ...). Unfortunately keymaps in emacs 
are unaware of keysyms from "base" group when another group is active. 
On the other hand, emacs input method requires special tricks to keep 
emacs window (almost) always with latin keyboard layout while other 
applications rely on xkb.


On 04/05/2021 15:19, Eric S Fraga wrote:

So, on this note, without hopefully hijacking the thread, maybe somebody
can tell me: what is the "default" input method, i.e. the one I get when
I start Emacs and haven't changed input methods at all?


Default input method depend on locale. E.g. en_US.UTF-8 does not require 
anything special.