Re: [Pre-PATCH] Overhaul of the LaTeX preview system

[Visuwesh]

[திங்கள் ஆகஸ்ட் 21, 2023] Karthik Chikmagalur wrote:

>> I have lualatex installed as part of another texlive package but I am
>> not sure if org uses it however.
>
> If you haven't changed `org-latex-compiler' (globally or in the Org
> buffer using a keyword), it's using pdflatex.

I am indeed using pdflatex.

>>> 3. If no, do you mind sharing this file (or a stripped down version
>>> you're okay sharing that still has these issues) along with the LaTeX
>>> preamble?  That would be `org-latex-preview-preamble' and
>>> `org-latex-packages-alist'.  I would like to reproduce this bug.
>
> Please let us know if you can reproduce the precompilation failure.
> This shouldn't be happening.

Hmm, I tried to reproduce it again but failed to do it.  Perhaps, being
a bit more patient is the key.  Sorry for the false alarm.  I will turn
on precompilation and report if/when I can reproduce the failure.

>> No, AFAICT I don't see the issue with png previews.  
>
> Sizing issues with pdflatex+svg previews should be fixed soon.
>
>> However, all latex previews, \( \) and \[ \], seem to be of the same
>> size now.
>
> Should they not be the same size?

After using the svg previews, I was under the impression that inline
maths environments would be at the same size as the rest of the text
whereas the other environments would be at a slightly higher size.

>>> Previewing unadorned LaTeX macros is officially unsupported but
>>> partially supported in practice.  If you move the cursor over the \ch{}
>>> and call `org-latex-preview' it should be previewed.  From that point it
>>> should behave like a regular LaTeX fragment.  Editing it should cause
>>> the preview to be auto-updated (If you are using
>>> org-latex-preview-auto-mode).
>>
>> Is there no automagic way to do this?  Like changing a regexp variable
>> and hoping it would work till the end of time™.  (:
>
> You can change `org-latex-preview--tentative-math-re' to suit your
> needs.  This variable is used to identify LaTeX fragments when calling
> `org-latex-preview'.  All subsequent checks involve the org-element api,
> but for reasons I can't recall at the moment the initial search is via a
> regexp.

Thanks, that seems to work!

> Karthik

Re: [Pre-PATCH] Overhaul of the LaTeX preview system

[Karthik Chikmagalur]

> I have lualatex installed as part of another texlive package but I am
> not sure if org uses it however.

If you haven't changed `org-latex-compiler' (globally or in the Org
buffer using a keyword), it's using pdflatex.

>> 3. If no, do you mind sharing this file (or a stripped down version
>> you're okay sharing that still has these issues) along with the LaTeX
>> preamble?  That would be `org-latex-preview-preamble' and
>> `org-latex-packages-alist'.  I would like to reproduce this bug.

Please let us know if you can reproduce the precompilation failure.
This shouldn't be happening.

> No, AFAICT I don't see the issue with png previews.  

Sizing issues with pdflatex+svg previews should be fixed soon.

> However, all latex previews, \( \) and \[ \], seem to be of the same
> size now.

Should they not be the same size?

>> Previewing unadorned LaTeX macros is officially unsupported but
>> partially supported in practice.  If you move the cursor over the \ch{}
>> and call `org-latex-preview' it should be previewed.  From that point it
>> should behave like a regular LaTeX fragment.  Editing it should cause
>> the preview to be auto-updated (If you are using
>> org-latex-preview-auto-mode).
>
> Is there no automagic way to do this?  Like changing a regexp variable
> and hoping it would work till the end of time™.  (:

You can change `org-latex-preview--tentative-math-re' to suit your
needs.  This variable is used to identify LaTeX fragments when calling
`org-latex-preview'.  All subsequent checks involve the org-element api,
but for reasons I can't recall at the moment the initial search is via a
regexp.

Karthik

Re: [Pre-PATCH] Overhaul of the LaTeX preview system

[Visuwesh]

[திங்கள் ஆகஸ்ட் 21, 2023] Karthik Chikmagalur wrote:

>> It worked well all around except for three things:
>>
>> 1. I had to set org-latex-preview-precompile to nil to produce the
>>preview for ~2000 snippets.  I got errors in process filters such
>>as arg-out-of-range, and Emacs completely blocked itself.  I can
>>confirm that mylatexformat is installed
>
> 1. Are you using xelatex/lualatex?  

I have lualatex installed as part of another texlive package but I am
not sure if org uses it however.

> 2. If yes, precompilation should have turned itself off with a warning.
> 3. If no, do you mind sharing this file (or a stripped down version
> you're okay sharing that still has these issues) along with the LaTeX
> preamble?  That would be `org-latex-preview-preamble' and
> `org-latex-packages-alist'.  I would like to reproduce this bug.
>
> Previewing 2000 snippets is exactly where the new async/speedy system is
> expected to shine compared to the old one.
>
>> 2. Apparently, org-latex-default-packages-alist no longer contain
>>asmsymb and asmmath?  This change pulled the rug under me since
>>it took me quite a while to figure out why some of my formulas
>>were coloured red...
>
> `org-latex-default-packages-alist' was modified to omit amsmath and
> amssymb (among others) in preparation for a conditional export system
> where required packages will be determined automatically from the buffer
> text -- i.e. to avoid exactly the kind of headache you encountered.
> Unfortunately this export system is not part of the patch set yet.

Ah, OK.  I will add them in my config and call it a day.  ;-)

>> 3. The image sizes of the same latex environment (e.g., \[ \]) is
>>different for different formulas.  I can send a screenshot of
>>this if required.
>
> This is a recent regression.  Could you generate png previews by:
>
> 1. (setq org-latex-preview-default-process 'dvipng)
> 2. regenerating previews
>
> and checking if this still happens?  If these look fine, this is a
> dvisvgm issue and we will fix it soon.

No, AFAICT I don't see the issue with png previews.  However, all latex
previews, \( \) and \[ \], seem to be of the same size now.

>>> If you do come across any issues, please let me know either in a reply here 
>>> or
>>> the org-mode matrix room. If you could also run
>>>  and 
>>> share the
>>> diagnostic info, that would be quite helpful.
>>
>> If needed, I can do this after reproducing (1).
>>
>> BTW, would it possible to arrange other latex environments (\ch{} in my
>> case) to be previewed as well?  Currently, I do \(\ch{}\) but omitting
>> \( \) would be nice.
>
> Previewing unadorned LaTeX macros is officially unsupported but
> partially supported in practice.  If you move the cursor over the \ch{}
> and call `org-latex-preview' it should be previewed.  From that point it
> should behave like a regular LaTeX fragment.  Editing it should cause
> the preview to be auto-updated (If you are using
> org-latex-preview-auto-mode).

Is there no automagic way to do this?  Like changing a regexp variable
and hoping it would work till the end of time™.  (:

> Karthik

Re: [Pre-PATCH] Overhaul of the LaTeX preview system

[Karthik Chikmagalur]

> It worked well all around except for three things:
>
> 1. I had to set org-latex-preview-precompile to nil to produce the
>preview for ~2000 snippets.  I got errors in process filters such
>as arg-out-of-range, and Emacs completely blocked itself.  I can
>confirm that mylatexformat is installed

1. Are you using xelatex/lualatex?  
2. If yes, precompilation should have turned itself off with a warning.
3. If no, do you mind sharing this file (or a stripped down version
you're okay sharing that still has these issues) along with the LaTeX
preamble?  That would be `org-latex-preview-preamble' and
`org-latex-packages-alist'.  I would like to reproduce this bug.

Previewing 2000 snippets is exactly where the new async/speedy system is
expected to shine compared to the old one.

> 2. Apparently, org-latex-default-packages-alist no longer contain
>asmsymb and asmmath?  This change pulled the rug under me since
>it took me quite a while to figure out why some of my formulas
>were coloured red...

`org-latex-default-packages-alist' was modified to omit amsmath and
amssymb (among others) in preparation for a conditional export system
where required packages will be determined automatically from the buffer
text -- i.e. to avoid exactly the kind of headache you encountered.
Unfortunately this export system is not part of the patch set yet.

> 3. The image sizes of the same latex environment (e.g., \[ \]) is
>different for different formulas.  I can send a screenshot of
>this if required.

This is a recent regression.  Could you generate png previews by:

1. (setq org-latex-preview-default-process 'dvipng)
2. regenerating previews

and checking if this still happens?  If these look fine, this is a
dvisvgm issue and we will fix it soon.

>> If you do come across any issues, please let me know either in a reply here 
>> or
>> the org-mode matrix room. If you could also run
>>  and 
>> share the
>> diagnostic info, that would be quite helpful.
>
> If needed, I can do this after reproducing (1).
>
> BTW, would it possible to arrange other latex environments (\ch{} in my
> case) to be previewed as well?  Currently, I do \(\ch{}\) but omitting
> \( \) would be nice.

Previewing unadorned LaTeX macros is officially unsupported but
partially supported in practice.  If you move the cursor over the \ch{}
and call `org-latex-preview' it should be previewed.  From that point it
should behave like a regular LaTeX fragment.  Editing it should cause
the preview to be auto-updated (If you are using
org-latex-preview-auto-mode).

Karthik

C-RET and empty lines

[Max Nikulin]

Hi,

A couple of recent threads reminded me about quite peculiar behavior 
when a new heading is added by C-RET. I prefer to have an empty line 
before headings. Sometimes such empty lines disappear. Is such behavior 
(highly dependent of context) intentional? From my point of view it is 
confusing.


--- 8< ---
Try =C-RET= when cursor in the following heading titles
* No empty line added

Text
* Existing empty line consumed

Text

* Empty line added before new heading

Text
--- >8 ---

Variants of C-RET effect:

--- 8< ---
* No empty line added

Text
* |
* Existing empty line consumed
--- >8 ---

or

--- 8< ---
* Existing empty line consumed

Text
* |
* Empty line added before new heading
--- >8 ---

or

--- 8< ---
* Empty line added before new heading

Text

* |
--- >8 ---

Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Max Nikulin]

On 21/08/2023 14:04, Ihor Radchenko wrote:

+(defconst org-shell-arg-literal (gensym "literal")


(opinion) Perhaps a better name exists. Maybe 
org-shell-arg-tag-unescaped (or unquoted)



+  "Symbol to be used to mark shell arguments that should not be escaped.
+See `org-make-shell-command'.")
+(defun org-make-shell-command (command  args)
+  "Build safe shell command string to run COMMAND with ARGS.
+
+The resulting shell command is safe against malicious shell expansion.
+
+ARGS can be nil, strings, (LITERAL STRING), or a list of


(opinion) I would give an example

`(,ob-shell-arg-literal STRING)

to avoid "LITERAL" that is confusing from my point of view. Perhaps it 
is better to describe its purpose more clearly: prevent raw shell 
constructs in ob header arguments in Org documents unless they appear in 
evaluated expressions.



+such elements.  LITERAL must be the value of `org-shell-arg-literal'.
+
+Strings will be quoted with `shell-quote-argument' while \(literal
+STRING) will be used without quoting.  nil values will be ignored."
+  (concat
+   command (when command " ")
+   (mapconcat
+#'identity
+(delq
+ nil
+ (mapcar
+  (lambda (str-def)
+(pcase str-def
+  (`(or nil "") nil)


An empty string may be an important argument. E.g.

read -r -d "" var

allows to read values separated by null character (\0), e.g. from output 
of find -print0. That is why I would leave just nil.



+ (when separator (format "-separator %s" separator))


Sorry, I made a typo previous time. It would not work. -separator and 
the separator character must *not* be combined into single argument. 
Sqlite does not support it.


(and separator `("-separator" ,separator))

or (format "%s" separator) if it may have a type other than string.

I hope, this approach does not have unnoticed flaws.

Re: [POLL] ob-C: Should we allow saving compiled src block to specified binary excecutable? (was: [RFC/PATCH] naming src/bin files in ob-C.el)

[Ihor Radchenko]

Ihor Radchenko  writes:

> I feel slightly reluctant about adding side effects to babel evaluation.
> Not too reluctant though.
>
> I'd like to hear from other ob-C users, so I changed this thread into a poll.

No replies.
Closed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: org-element-cache-warning

[Ihor Radchenko]

Gregor Zattler  writes:

> Dear org-mode developers, Ihor, I cought an
> org-element-cache warning with a rather up-to-date
> emacs and org-mode ...

Thanks for reporting!

> The element is: "(headline (:standard-properties [159509 160713 159557 160712 
> 1..."
>  The real element is: "(headline (:standard-properties [159509 160713 159557 
> 160713 0..."

This was obsolete saved cache being loaded (wrongly).
Headlines no longer own blank lines and instead include them into their
contents.

Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=7219b006b

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Pre-PATCH] Overhaul of the LaTeX preview system

[Visuwesh]

[ஞாயிறு மார்ச் 12, 2023] Timothy wrote:

> Hi All,
>
> After months of work, Karthink and I have prepared a rather large patch-set
> completely overhauling the LaTeX preview system. I hope to have a patch set
> shortly, but in the mean time it would be good to get some more people testing
> this.
>
> To test this feature, please check out the `dev' branch of
>  (it’s the default branch). There 
> are
> also some other changes there currently, but I don’t think anything is broken.

I tested commit '6eb77305361a1f6a3f67c3229f837f1c4e95f546' of the dev
branch.

It worked well all around except for three things:

1. I had to set org-latex-preview-precompile to nil to produce the
   preview for ~2000 snippets.  I got errors in process filters such
   as arg-out-of-range, and Emacs completely blocked itself.  I can
   confirm that mylatexformat is installed

   % dpkg-query -S mylatexformat
   texlive-latex-extra: 
/usr/share/texlive/texmf-dist/tex/latex/mylatexformat
   texlive-latex-extra: 
/usr/share/texlive/texmf-dist/tex/latex/mylatexformat/mylatexformat.ltx

2. Apparently, org-latex-default-packages-alist no longer contain
   asmsymb and asmmath?  This change pulled the rug under me since
   it took me quite a while to figure out why some of my formulas
   were coloured red...

3. The image sizes of the same latex environment (e.g., \[ \]) is
   different for different formulas.  I can send a screenshot of
   this if required.

Except for these hiccups, it works without a hitch.  Thank you very much
for your work!

> If you do come across any issues, please let me know either in a reply here or
> the org-mode matrix room. If you could also run
>  and share 
> the
> diagnostic info, that would be quite helpful.

If needed, I can do this after reproducing (1).

BTW, would it possible to arrange other latex environments (\ch{} in my
case) to be previewed as well?  Currently, I do \(\ch{}\) but omitting
\( \) would be nice.

org-element-cache-warning

[Gregor Zattler]

Dear org-mode developers, Ihor, I cought an
org-element-cache warning with a rather up-to-date
emacs and org-mode while calling via a key binding this
simple function:

  (defun my/org-goto-agenda-heading ()
(interactive)
(org-refile '(4)))

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0) of 
2023-08-20
Org mode version 9.7-pre (release_9.6.7-709-gdd2f05 @ 
/home/grfz/src/org-mode/lisp/)

I hope this helps somehow.

Ciao; Gregor



 ■  Warning (org-element-cache): org-element--cache: Warning(grfz.org): 
(my/org-goto-agenda-heading) Cached element is incorrect in grfz.org. (Cache 
tic up to date: "yes") Resetting.
If this warning appears regularly, please report the warning text to Org mode 
mailing list (M-x org-submit-bug-report).
The element is: "(headline (:standard-properties [159509 160713 159557 160712 1 
159509 (:title) t nil 159559 160710 nil element 3 (headline 
(:standard-properties [2550 183136 2561 183133 3 2550 ... t nil 2669 183131 nil 
element 1 ... nil nil #] :pre-blank 0 :raw-value 
[org-element-deferred org-element--headline-raw-value ... nil] :title 
[org-element-deferred org-element-property-2 ... nil] :level 1 :priority nil 
:tags nil :todo-keyword nil :todo-type nil :footnote-section-p 
[org-element-deferred org-element--headline-footnote-section-p nil nil] 
:archivedp [org-element-deferred org-element--headline-archivedp nil nil] 
:commentedp nil :ARCHIVE [org-element-deferred org-element--substring ... nil] 
:ID [org-element-deferred org-element--substring ... nil])) nil nil #] :pre-blank 0 :raw-value [org-element-deferred 
org-element--headline-raw-value (4 47) nil] :title [org-element-deferred 
org-element-property-2 (:raw-value) nil] :level 2 :priority nil :tags nil 
:todo-keyword nil :todo-type nil :footnote-section-p [org-element-deferred 
org-element--headline-footnote-section-p nil nil] :archivedp 
[org-element-deferred org-element--headline-archivedp nil nil] :commentedp 
nil))"
 The real element is: "(headline (:standard-properties [159509 160713 159557 
160713 0 159509 (:title) nil nil 159559 160713 nil nil 3 [org-element-deferred 
org-element--headline-parent-deferred nil t] [org-element-deferred 
org-element--headline-deferred nil t] nil #] :pre-blank 0 
:raw-value [org-element-deferred org-element--headline-parse-title (t) nil] 
:title [org-element-deferred org-element--headline-parse-title (t) nil] :level 
[org-element-deferred org-element--headline-parse-title (t) nil] :priority 
[org-element-deferred org-element--headline-parse-title (t) nil] :tags 
[org-element-deferred org-element--headline-parse-title (t) nil] :todo-keyword 
[org-element-deferred org-element--headline-parse-title (t) nil] :todo-type 
[org-element-deferred org-element--headline-parse-title (t) nil] 
:footnote-section-p [org-element-deferred org-element--headline-parse-title (t) 
nil] :archivedp [org-element-deferred org-element--headline-parse-title (t) 
nil] :commentedp [org-element-deferred org-element--headline-parse-title (t) 
nil]))"
 Cache around :begin:
"(headline (:standard-properties [158019 159509 158064 159509 0 158019 (:title) 
t nil 158066 159507 nil element 3 (headline (:standard-properties [2550 183136 
2561 183133 3 2550 ... t nil 2669 183131 nil element 1 ... nil nil #] :pre-blank 0 :raw-value [org-element-deferred 
org-element--headline-raw-value ... nil] :title [org-element-deferred 
org-element-property-2 ... nil] :level 1 :priority nil :tags nil :todo-keyword 
nil :todo-type nil :footnote-section-p [org-element-deferred 
org-element--headline-footnote-section-p nil nil] :archivedp 
[org-element-deferred org-element--headline-archivedp nil nil] :commentedp nil 
:ARCHIVE [org-element-deferred org-element--substring ... nil] :ID 
[org-element-deferred org-element--substring ... nil])) nil nil #] :pre-blank 0 :raw-value [org-element-deferred 
org-element--headline-raw-value (4 44) nil] :title [org-element-deferred 
org-element-property-2 (:raw-value) nil] :level 2 :priority nil :tags nil 
:todo-keyword nil :todo-type nil :footnote-section-p [org-element-deferred 
org-element--headline-footnote-section-p nil nil] :archivedp 
[org-element-deferred org-element--headline-archivedp nil nil] :commentedp 
nil))"
"(headline (:standard-properties [159509 160713 159557 160712 1 159509 (:title) 
t nil 159559 160710 nil element 3 (headline (:standard-properties [2550 183136 
2561 183133 3 2550 ... t nil 2669 183131 nil element 1 ... nil nil #] :pre-blank 0 :raw-value [org-element-deferred 
org-element--headline-raw-value ... nil] :title [org-element-deferred 
org-element-property-2 ... nil] :level 1 :priority nil :tags nil :todo-keyword 
nil :todo-type nil :footnote-section-p [org-element-deferred 
org-element--headline-footnote-section-p nil nil] :archivedp 
[org-element-deferred org-element--headline-archivedp nil nil] :commentedp nil 
:ARCHIVE [org-element-deferred org-element--substring ... nil] :ID 
[org-element-deferred org-element--substring ... nil])) 

Re: Maintenance status of individual Org libraries

[Ihor Radchenko]

Matt  writes:

>   On Mon, 30 Jan 2023 14:41:18 +0100  Ihor Radchenko  wrote --- 
>
>  > The following libraries have no maintainer or author active on the list:
>  
>  > ob-comint, ob-core, ...
>
> I'm interested (and technically still maintainer of ob-shell). ob-comint and 
> ob-core are both used by ob-shell, so I've looked at good bits of both.  I'd 
> be willing to work on those.

Thanks!

> Until the disclaimer language gets resolved, I may be able to help in other 
> ways.  I have notes explaining the ob- API 
> (https://lists.gnu.org/archive/html/emacs-orgmode/2023-02/msg00531.html).  
> Maybe if that were completed, it would make being a maintainer more 
> approachable for people?  I'd also be happy to mentor new maintainers.  What 
> other ways might I help?

You may help by confirming bug reports, discussing new features, and
providing idea about how to fix things.

For example, check out
- Discussion about merging header arguments:  
https://list.orgmode.org/8735451u20@gmail.com/
- Discussion about :results header arg: 
https://list.orgmode.org/orgmode/87sf9x8n38.fsf@localhost/
- Some ob-shell Windows-only bug: 
https://list.orgmode.org/orgmode/87y1rbeohp.fsf@localhost/
- Some weird edge cases related to comint and command that read key:
  https://list.orgmode.org/orgmode/878re5g0ir.fsf@localhost/

You also do not need FSF assignment to contribute to WORG documentation.
Your example babel template may go there.

> I'd also be happy to mentor new maintainers.

It would also be nice to help people who submit patches to
ob-core/ob-comint, by suggesting improvements and answering their
questions. These people are the perspective maintainers - we should
better help them early :)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [Bug] org-update-dblock randomly broken for more than 10 columns and larger files

[Jakob Schöttl]

Am 20.08.23 um 10:57 schrieb Ihor Radchenko:

Thanks for the reproducer!
I committed a fix onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=58c5c5882
Nice, thank you very much! Now, spaces are only added to headings and 
the resulting table is reproducible and correct.

Re: [BUG] WORG example for ob-lilypond is no longer working as described (was: Moving some lisp/ob-*.el files to org-contrib - your advice?)

[Ihor Radchenko]

Jonathan Gregory  writes:

>>> -  b c d e
>>> +  b4 c d e
>>
>> Is there any specific reason for this change?
>
> This is to ensure that the notes use the correct duration in 
> arrange-mode. 4 is the default duration and is carried over until 
> a new value is added, in this case c1. 1 is then carried over 
> making all subsequent "e"s have a value of 1, which is incorrect.

I see.
For future, please make sure to document every change in the commit
message. Or future readers might be confused about the purpose.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Org orphanage?

[Ihor Radchenko]

Corwin Brust  writes:

> Hi all; I added worg co-curator krupalinbox to CC.

I think you did not. I added him now.

> I would be happy to help with administration here.   I have a GitHub
> account, as well as commit on bastian's sr.ht.
>
> In fact, I wonder if worg curators should have some general
> responsibilities to help with the orphanage.
>
> Speaking for myself, I can help:
> - send the various "let me know" emails discussed in this thread

I am not sure if I understand. May you elaborate?

> - suggest packages that need love to (prospective) volunteers

Do you have any specific packages in mind?

> - review the bonafides (experience/involvement and, via Eli/SM,
> assignment status ) for prospective adopting maintainers

Not sure about emacsorphanage side, but for things we decide to manage
on sr.ht, we can all discuss new maintainers on this mailing list. The
maintenance requests should also go here.

We may need to update https://orgmode.org/worg/org-contribute.html,
similar to:

Maintain an Org file: If a file in the git repository does not have a
maintainer 1 and you want to help by maintaining it, please read more on
how Org is maintained and let us know by sending an email to the mailing
list.

but for orphaned packages specifically.

The information about how to apply to be a maintainer may also be added
to org-orphanage WORG page.

> - assign permissions
> - move repos
> - apply well tested patches to orphaned packages

I think you can participate in issue discussion for the existing
orphaned packages first. (The problem is that you do not have a record
contributing to Emacs/Org mode. I can only see two trivial commits
64d211179db and a70a3449417 in Emacs)

> Irrespective of whether worg curators should help with this (or some
> other vols, or just the org maintainers, or...), I think it would make
> sense to document these "responsibilities" (and the detail of the
> plans made in this thread, in general), along with the whos and hows.

+1

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Bug: org-anniversary dates not exported to iCalendar [9.4.6 (9.4.6-3-g37f69b-elpaplus @ /home/marcus/.emacs.d/elpa/org-plus-contrib-20210531/)]

[Marcus Zibrowius]

Just to confirm:  It also works with Org 9.6 for me. Thanks for looking into 
this!

Marcus

On Tue, 2023-07-25 at 07:05 +, Ihor Radchenko wrote:
> Marcus Zibrowius 
> <[marcus.zibrow...@posteo.net](mailto:marcus.zibrow...@posteo.net)> writes:
> 
> 
> 
> > > Please provide mode details about what you tried.  
> > 
> > 
> > OK, I will try.  I start emacs with
> > ...
> >     Org mode version 9.5.5 (release_9.5.5 @ 
> > /snap/emacs/current/usr/share/emacs/28.2/lisp/org/)
> 
> 
> I can also reproduce with this Org version.  
> But not with Org 9.6 (the latest release).
> 

[SECURITY] Shell expansion of babel header args (was: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands)

[Ihor Radchenko]

Max Nikulin  writes:

> P.S. Babel backends should be consistent in respect to treating options 
> for header arguments:
> - use as is
> - expand ~user and $VAR
> - allow any shell expression

We cannot generally know which header arg values can or cannot be
shell-expanded. It is something only individual babel backends can know.

However, there are frequently used header arguments like :cmd, where it
does make sense to allow shell expansion. But we may need to safeguard
them behind user prompt for safety, similar to what has to be done for
Elisp evaluation.

We can allow backends to specify "safety" of the header argument value
similar to how we now define the allowed values in
`org-babel-common-header-args-w-values'. Then, babel can prompt for user
confirmation every time "unsafe" argument value is encountered.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Ihor Radchenko]

Max Nikulin  writes:

>> Do you have any ideas how to work around the deliberately constructed
>> header argument values like in your example?
>
> Perhaps `gensym' may be used to create a symbol that can not appear in a 
> document. I am unsure if the following `pcase' variant may be improved
> ...
> ;; or ob-shell-argument-literal-symbol
> (defconst ob-literal-symbol (gensym "literal"))

Good idea.
I am attaching tentative fix that uses the proposed approach.
Not yet merging - need to go through other babel backends and make them
use the new API.

>From dfc03c0330b96ff4fbe14df39ba895427b8fd004 Mon Sep 17 00:00:00 2001
Message-ID: 
From: Ihor Radchenko 
Date: Mon, 21 Aug 2023 09:57:50 +0300
Subject: [PATCH 1/2] org-macs: New common API function to quote shell
 arguments

* lisp/org-macs.el (org-shell-arg-literal): New auxiliary constant.
(org-make-shell-command): New function that returns shell command
built from individual shell arguments, escaping them to prevent
malicious code execution.

Link: https://orgmode.org/list/ub549k$q11$1...@ciao.gmane.io
---
 lisp/org-macs.el | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index 907e8bed7..95af9e45e 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -1593,6 +1593,37 @@ (defun org-sxhash-safe (obj  counter)
 	  (puthash hash obj org-sxhash-objects)
 	  (puthash obj hash org-sxhash-hashes)
 
+(defconst org-shell-arg-literal (gensym "literal")
+  "Symbol to be used to mark shell arguments that should not be escaped.
+See `org-make-shell-command'.")
+(defun org-make-shell-command (command  args)
+  "Build safe shell command string to run COMMAND with ARGS.
+
+The resulting shell command is safe against malicious shell expansion.
+
+ARGS can be nil, strings, (LITERAL STRING), or a list of
+such elements.  LITERAL must be the value of `org-shell-arg-literal'.
+
+Strings will be quoted with `shell-quote-argument' while \(literal
+STRING) will be used without quoting.  nil values will be ignored."
+  (concat
+   command (when command " ")
+   (mapconcat
+#'identity
+(delq
+ nil
+ (mapcar
+  (lambda (str-def)
+(pcase str-def
+  (`(or nil "") nil)
+  ((pred stringp) (shell-quote-argument str-def))
+  (`(,(pred (eq org-shell-arg-literal)) ,(and (pred stringp) str))
+   str)
+  ((pred listp) (apply #'org-make-shell-command nil str-def))
+  (_ (error "Unknown ARG specification: %S" str-def
+  args))
+" ")))
+
 (defun org-compile-file (source process ext  err-msg log-buf spec)
   "Compile a SOURCE file using PROCESS.
 
-- 
2.41.0

>From d7a8dd47aa06e715b6bb213914d43f973c6cb413 Mon Sep 17 00:00:00 2001
Message-ID: 
In-Reply-To: 
References: 
From: Ihor Radchenko 
Date: Mon, 21 Aug 2023 09:59:12 +0300
Subject: [PATCH 2/2] org-babel-execute:sqlite: Fix shell arg expansion
 vulnerability

* lisp/ob-sqlite.el (org-babel-execute:sqlite): Use
`org-make-shell-command' to escape the strings taken from Org file.
This will prevent abusing shell expansion.

Reported-by: Max Nikulin 
Link: https://orgmode.org/list/ub549k$q11$1...@ciao.gmane.io
---
 lisp/ob-sqlite.el | 34 ++
 1 file changed, 14 insertions(+), 20 deletions(-)

diff --git a/lisp/ob-sqlite.el b/lisp/ob-sqlite.el
index 7510e5158..27e495fce 100644
--- a/lisp/ob-sqlite.el
+++ b/lisp/ob-sqlite.el
@@ -77,26 +77,20 @@ (defun org-babel-execute:sqlite (body params)
 (with-temp-buffer
   (insert
(org-babel-eval
-	(org-fill-template
-	 "%cmd %header %separator %nullvalue %others %csv %db "
-	 (list
-	  (cons "cmd" org-babel-sqlite3-command)
-	  (cons "header" (if headers-p "-header" "-noheader"))
-	  (cons "separator"
-		(if separator (format "-separator %s" separator) ""))
-	  (cons "nullvalue"
-		(if nullvalue (format "-nullvalue %s" nullvalue) ""))
-	  (cons "others"
-		(mapconcat
-		 (lambda (arg) (format "-%s" (substring (symbol-name arg) 1)))
-		 others " "))
-	  ;; for easy table parsing, default header type should be -csv
-	  (cons "csv" (if (or (member :csv others) (member :column others)
-			  (member :line others) (member :list others)
-			  (member :html others) separator)
-			  ""
-			"-csv"))
-  (cons "db" (or db ""
+(org-make-shell-command
+ org-babel-sqlite3-command
+ (if headers-p "-header" "-noheader")
+ (when separator (format "-separator %s" separator))
+ (when nullvalue (format "-nullvalue %s" nullvalue))
+ (mapcar
+	  (lambda (arg) (format "-%s" (substring (symbol-name arg) 1)))
+	  others)
+ ;; for easy table parsing, default header type should be -csv
+ (unless (or (member :csv others) (member :column others)
+		 (member :line others) (member :list others)
+		 (member :html others) separator)
+	   "-csv")
+ db)
 	;; body of the code block
 	(org-babel-expand-body:sqlite body params)))
   

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[Liu Hui]

> Thanks for reporting these misbehaving examples. I think the root of the
> problem is `org-babel-script-escape', which is too aggressive in
> recursively converting strings to lists. We may need to rewrite our own
> implementation for ob-python.
>
> Also, I agree that moving the python code to an external file will be
> helpful in handling these more complex cases.
>
> I may leave these tasks for future patches. In the meantime, we may have
> to recommend ":results verbatim" for these more complex cases that
> ":results table" doesn't fully handle yet.

Understand. Thanks again for your work!