Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Max Nikulin]

On 18/08/2023 18:05, Ihor Radchenko wrote:

Max Nikulin writes:


Ihor, this is a list, not an expression to be evaluated. There are some
conditions to avoid user prompts for strings, lists, etc. They are
considered safe.

This particular case is handled namely by ob-sqlite and the proposed
function in org-macs.


Do you have any ideas how to work around the deliberately constructed
header argument values like in your example?


Perhaps `gensym' may be used to create a symbol that can not appear in a 
document. I am unsure if the following `pcase' variant may be improved


(`(,(and s (guard (eq s ob-literal-symbol))) ,(and (pred stringp) str))
 str)

for

;; or ob-shell-argument-literal-symbol
(defconst ob-literal-symbol (gensym "literal"))

I hope, list values can not be used to bypass escaping with such 
approach. It is still possible to use evaluated expressions, but user 
prompt for such cases should be fixed anyway.


P.S. Babel backends should be consistent in respect to treating options 
for header arguments:

- use as is
- expand ~user and $VAR
- allow any shell expression

Clarification on blank lines following list items

[Tom Alexander]

I am noticing the list items have some very context-sensitive specific behavior 
regarding ownership of the trailing blank lines. I was hoping to get some 
clarification on this (namely, are my observations correct, am I stumbling 
across a bug, or have I not dug deep enough to suss out the real rules?). The 
org-mode documentation states:

> With the exception of list items and footnote definitions blank lines belong 
> to the preceding element with the narrowest possible scope

but it does not state who ends up owning those blank lines.

In a previous email the incredibly helpful Ihor Radchenko expanded on this 
further with:

> Also, in addition to list items, footnote-definitions do not extend their 
> contents to the trailing blank lines.

which, I would interpret as the list items do not own their trailing blank 
lines but rather the list owns them. But that is not the behavior I am seeing. 
If I had to summarize the behavior I am seeing into words it would be:

> List items own their trailing blank lines unless they are both the final list 
> item and not the final element of a non-final list item.

Below I have how I reached this conclusion, but before diving into the weeds I 
want to point two things out:

1. I have hastily thrown together a tool to help rapidly visualize the 
ownership of nodes in org-mode's AST. Before this tool, I was manually running 
org-element-parse-buffer and using M-g c to jump around to the various indices 
to see where nodes started/ended. With this tool, I can paste in my org-mode 
source and get a tree showing the contents of each node and I can click on the 
nodes to highlight the relevant characters in the org-mode source. It is 
available at https://github.com/tomalexander/org_mode_ast_investigation_tool .

2. I have flattened my analysis for plain-text consumption over email below, 
but if you'd prefer the original org-mode version of this investigation it is 
available at 
https://github.com/tomalexander/org_mode_ast_investigation_tool/blob/cba1d1e988be230f3104f5f63dfaeaaf5cd0d280/notes/plain_list_ownership_notes.org
 .

And now, here is how I reached that conclusion:

*** Test case 1
```
1. foo

   1. bar

   2. baz

2. lorem

ipsum
```
| Plain List *Item*  | Owns trailing blank lines |
|+---|
| foo (includes bar baz) | Yes   |
| bar| Yes   |
| baz| Yes   |
| lorem  | No|

In this test case, we see that the only list item that doesn't own its trailing 
blank lines is "lorem", the final list item of the outer-most list.


*** Test case 2
We add "cat" as a paragraph at the end of foo which makes "baz" lose its 
trailing blank lines
```
1. foo

   1. bar

   2. baz

   cat

2. lorem

ipsum
```
| Plain List *Item* | Owns trailing blank lines |
|---+---|
| foo -> cat (includes bar baz) | Yes   |
| bar   | Yes   |
| baz   | No|
| lorem | No|

In isolation, this implies that the final plain list item does not own its 
trailing blank lines, which conflicts with "baz" from test 1.

New theory: List items own their trailing blank lines unless they are both the 
final list item and not the final element of a list item.

Adding why to the table:
| Plain List *Item* | Owns trailing blank lines | Why   
|
|---+---+---|
| foo -> cat (includes bar baz) | Yes   | Not the final 
list item   |
| bar   | Yes   | Not the final 
list item   |
| baz   | No| Final item of 
bar->baz and not the final element of "foo" |
| lorem | No| Final item of 
foo->lorem and not contained in a list item |


*** Test case 3
So if that theory is true, taking the entire (foo -> lorem) list from test 1 
and nesting it inside a list should coerce "lorem" to own its trailing blank 
lines since it would then be a final list item (of foo -> lorem) and the final 
element of the new list.
```
1. cat
   1. foo

  1. bar

  2. baz

   2. lorem

ipsum
```
| Plain List *Item*   | Owns trailing blank lines |
|-+---|
| cat (includes foo -> lorem) | No|
| foo (includes bar baz)  | Yes   |
| bar | Yes   |
| baz | Yes   

Re: Inconvenient end-of-file behavior in org-mode 9.6 (emacs 29.1)

[Max Nikulin]

On 16/08/2023 20:19, Kaiyu Zheng wrote:

The issue is: let's say I have a simple org-mode file, and let's say
the cursor starts at the top of the file (marked by '|'):

|* Sec1
** SubSec1
** SubSec2


A similar, but not so severe change in comparison to 9.5:

In response to C-RET Org-9.5 just adds first level heading (so M- 
puts it to the same level as SubSec2, it is my usual way to append a 
subheading), but Org>=9.6 adds ellipsis after cursor. When some content 
is pasted from clipboard, it appears folded.


Complete example:
 8< 
#+startup: content
* Sec1
** SubSec1

text

** SubSec2

text

Re: [BUG] Warning when creating preview

[Edgar Lux]

On Aug 18, 2023 at 6:04 PM, Ihor Radchenko  wrote:Edgar 
Lux  writes:

> The Org version is very strange. Emacs 28 ships with Org 9.5.5 and the
> latest stable version of Org available on ELPA is Org 9.6.7. The warning
> comes from Org >9.6.

Super weird. I don't know why I ended up in some 9.4 branch (a246a1318), and 
when I did checkout master, I would still be there. Now in 9.7-pre

> Then, let us know if the warning keeps appearing.

I saw you holding your breath :P .

THAAANKS

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

-- 
Sent with https://mailfence.com  
Secure and private email

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[Jack Kamm]

gerard.vermeu...@posteo.net writes:

> I do not know how much this "abuse" of defconst is frowned
> upon (elisp manual says defconst is advisory), but maybe it
> can be advertised as a feature.

org-babel-python--def-format-value is a "private" variable (it has
double dash "--" in its name).  Therefore it's not generally recommended
to modify it.

Of course, elisp doesn't have true private variables or functions, and
you are free to change things as you wish -- this is one of the perks of
Emacs :) But you've been warned, since this is a private variable, we
make no guarantees, and may break things in backward-incompatible ways
in the future.

As to the broader point, I agree there are many more features that would
be nice to add ob-python results handling. But making ob-python too
complex will be difficult to maintain, especially since the Python code
is all in quoted strings without proper linting.

So I am thinking now about how we could make this more extensible in
future. One idea is to create a Python package for interfacing with Org
Babel, and release it on PyPi. If we detect the package is installed,
then we can delegate to it for results formatting. And the community
could contribute results handling for all sorts of Python objects to
that package.

That is just one idea for improving extensibility -- I'm not sure it's
the best, and am open to other suggestions as well.

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[Jack Kamm]

Ihor Radchenko  writes:

> This is an ORG-NEWS entry for Version 9.4. Is it an intentional change?

Sorry, that was an accident. I've reverted it now:
https://github.com/jackkamm/org-mode/commit/f12a695d67bc5c06013d9fbe0af844c9739e347a

>> @@ -142,7 +144,9 @@ (defun org-babel-python-table-or-string (results)
>>"Convert RESULTS into an appropriate elisp value.
>>  If the results look like a list or tuple, then convert them into an
>>  Emacs-lisp table, otherwise return the results as a string."
>> -  (let ((res (org-babel-script-escape results)))
>> +  (let ((res (if (string-equal "{" (substring results 0 1))
>> + results ;don't covert dicts to elisp
>> +   (org-babel-script-escape results
>
> You may also need to update the docstring for
> `org-babel-python-table-or-string' after this change.

That change got reverted in subsequent update when I changed dict to
return as table by default instead of string. So there's no need to
update the docstring anymore.

>> -body)))
>> -   (`value (let ((tmp-file (org-babel-temp-file "python-")))
>> +(if graphics-file
>> +(format 
>> org-babel-python--output-graphics-wrapper
>> +body graphics-file)
>> +  body
>> +   (`value (let ((results-file (or graphics-file
>> +   (org-babel-temp-file "python-"
>
> What about :results graphics file ?

Not entirely sure what you mean here.

When ":results graphics file", then graphics-file will be non-nil --
org-babel-execute:python passes graphics-file onto
org-babel-python-evaluate and then
org-babel-python-evaluate-external-process. In case of ":results
graphics file output", org-babel-python--output-graphics-wrapper is used
to save pyplot.gcf(). Or if ":results graphics file value", then
org-babel-python--def-format-value saves the result with
Figure.savefig().

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[Jack Kamm]

Liu Hui  writes:

> Hi,
>
> Thank you for the patch!

Thanks for your feedback, I've incorporated it into
https://github.com/jackkamm/org-mode/tree/python-results-revisited-2023

More specifically, here:
https://github.com/jackkamm/org-mode/commit/af1d18314073446045395ff7a3d1de0303e92586

> Do we need to limit the table/list size by default, or handle them
> only with relevant result type (e.g. `table/list')? Dataframe/array
> are often large.

I've updated the patch so that Dataframe/Array are converted to table
only when ":results table" is explicitly set now. If ":results table" is
not set, they will be returned as string by default.

So code blocks that return large dataframes/arrays can continue to be
safely run.

Note I did make an additional change to Numpy array default behavior:
Previously, numpy arrays would be returned as table, but get mangled
when they were very large, e.g.:

  #+begin_src python
  import numpy as np
  return np.zeros((30,40))
  #+end_src
  
  #+RESULTS:
  | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | ... | (0 0 0 
... 0 0 0) | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) |

But now, Numpy array is returned in string form by default, in the same
format as in Jupyter:

  #+begin_src python
  import numpy as np
  return np.zeros((30,40))
  #+end_src
  
  #+RESULTS:
  : array([[0., 0., 0., ..., 0., 0., 0.],
  :[0., 0., 0., ..., 0., 0., 0.],
  :[0., 0., 0., ..., 0., 0., 0.],
  :...,
  :[0., 0., 0., ..., 0., 0., 0.],
  :[0., 0., 0., ..., 0., 0., 0.],
  :[0., 0., 0., ..., 0., 0., 0.]])


>> +if isinstance(result, pandas.DataFrame):
>> +result = [[''] + list(result.columns), None] + \
>
> Here we can use '{}'.format(df.index.name) to show the name of index

Patch has been updated to print the index name when it is non-None.

> Maybe `org-babel-python--def-format-value' can be evaluated only once
> in the session mode? It would shorten the string sent to the python
> shell, where temp files are used for long strings.

Patch has been updated to evaluate `org-babel-python--def-format-value'
once per session.

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[Jack Kamm]

Ihor Radchenko  writes:

>>>  #+begin_src python :results list
>>>return {"a": 1, "b": 2}
>>>  #+end_src
>>>
>>>  #+RESULTS:
>>>  - a :: 1
>>>  - b :: 2
>>
>> This seems harder, and may require more widespread changes beyond
>> ob-python. In particular, I think we'd need to change
>> `org-babel-insert-result' so that it can call `org-list-to-org' with a
>> list of type "descriptive" instead of "unordered" here:
>
> Actually, (org-list-to-org '(unordered ("a :: b") ("c :: d")))
> will just work.
>
> We do not support nested lists when transforming output anyway. So,
> unordered/descriptive does not matter in practice.

You're right, thanks for the suggestion.

I've added it now to
https://github.com/jackkamm/org-mode/tree/python-results-revisited-2023

More specifically, here:
https://github.com/jackkamm/org-mode/commit/0440caa3326b867a3a15d5f92a6f99cbf94c14d5

Re: desirability of boxquote-style snippets for helping new users

[Samuel Wales]

On 1/15/09, Kevin Rodgers  wrote:
>> However, most prefix every line.  For example, boxquote by default uses
>> "|".
>>
>> While an experienced user can figure out rectangle commands or write a
>> command to unpack the quote, new users and users who can't type much
>> might skip using the code to avoid having to do that.  And a very new
>> user could actually stick the whole thing in .emacs and wonder why it
>> doesn't work.  It adds to the burden of fixing a problem or meeting a
>> need in emacs without providing much benefit.  Even a few keystrokes
>> can do that.
>>
>> Perhaps quotes of code, in all packages like boxquote, could by
>> default get fancy only on the lines precediing and following.
>>
>> What do you think?
>
> ,[ C-h f boxquote-unbox RET ]
> | boxquote-unbox is an interactive compiled Lisp function in `boxquote.el'.
> | (boxquote-unbox)
> |
> | Remove the boxquote that contains `point'.
> |
> | [back]
> `

thanks for your suggestion.

as i said, i am suggesting that the prefixed lines should never be
prefixed in the first place.  i don't think it adds enough to outweigh
the extra keystrokes, rectangle, boxquote package, etc.  i don't think
all newcomers will know what to do.

imho, above and below quoting is a preferable default convention.
exceptions are few and can be optional.

[Tip] Popup-menu with several actions for a link

[Juan Manuel Macías]

Hi,

I’ve been experimenting for a while with the popup.el library
(), which offers an easy way
to create popup menus (even cascading menus), with auto-completion
functions. I’m sharing here a popup menu that I’ve defined to perform
various actions on an Org link, in case anyone finds it useful.

In this list I store the functions that I am writing to manipulate links
(open the link with eww, open the file with an external application,
attach it to an email, upload it to Imgur, copy it to another directory
or move it, visit the file directory, etc.). Something like this:

┌
│ (setq my-org-link-actions-list
│   '(("Action 1" . function1)
│ ("Action 2" . function2)
│ ("etc..." . etc)))
└

Then, I have defined this popup-menu that is displayed on a link:

┌
│ (defun my-org-actions-link-popup ()
│ (interactive)
│ (funcall
│  (popup-menu*
│   (mapcar
│(lambda (x)
│(popup-make-item (car x) :value (cdr x)))
│my-org-link-actions-list)
│   :isearch t)))
└

And a little addendum. Being a Hyperbole user, it occurred to me a while
ago that a ’secondary action key’ could be very useful to me in certain
contexts. So I defined this:

┌
│ (defvar my-hyp-alt-act nil)
│ 
│ (defun my-hyp-action-key-alt ()
│   (interactive)
│   (let ((my-hyp-alt-act t))
│ (action-key)))
└

If `C-c i' is for the ’primary’ action key, `C-c I' is for the
’secondary’ action:

┌
│ (global-set-key (kbd "C-c I") #'mi-hyp-action-key-alt)
└

And then I’ve modified hyperbole `org-link' a bit:

┌
│ (defact org-link ( link)
│   "Follows an optional Org mode LINK to its target.
│ If LINK is nil, follows any link at point.  Otherwise, triggers an error."
│   (if (not my-hyp-alt-act)
│   (if (stringp link)
│ (org-link-open-from-string link)
│   (org-open-at-point))
│ (hact #'my-org-actions-link-popup)))
└

In this way, if I have the point over the link and press `C-c i', the
link opens. If I press `C-c I' the popup-menu is displayed.

Best regards,

Juan Manuel 

-- 
--
--
Juan Manuel Macías 

https://juanmanuelmacias.com

https://lunotipia.juanmanuelmacias.com

https://gnutas.juanmanuelmacias.com

Wider margins on text, but no margins on tables?

[William Denton]

I have always run Emacs full screen and never set any margins, so the text goes 
from one side of my screen to the other.  (It's not a big screen.)  Today I'm 
trying out narrowing how text is displayed by setting left-margin-width and 
right-margin-width, so there's blank space on either side.  It looks good for 
most things, and helps readability.


But for tables it can be a problem because sometimes I have a table that i want 
as wide as possible.  Is there a way to have tables fit inside the margin-width 
settings until they get too wide, then expand out?  That way a small table isn't 
flush left, but a big one is.


Or perhaps there is a way to set Org tables to have margin-widths of 0, so it 
would be:  narrowed paragraph, narrowed paragraph, table flush left, narrowed 
paragraph.


Has anyone here configured something like this?  If there's a hook to use it's 
beyond what I can hack.  Any related tips or configurations would be welcome too.



Thanks,

Bill

--
William Denton
https://www.miskatonic.org/
Librarian, artist and licensed private investigator.
Toronto, Canada
CO₂: 419.19 ppm (Mauna Loa Observatory, 2023-08-17)

Re: [BUG] Warning when creating preview

[Ihor Radchenko]

Edgar Lux  writes:

> I don't really need help with this. Org has been telling me for days
> that I should report this, but I don't really want to share my
> configuration publicly, and I thought that it was only for a file on
> which I was working. It seems to be general for my configuration. I
> hope that it helps. If you have an encrypted personal account to which
> I can send my config file, let me know.

You do not have to. If you wish, you can disable the warning by clicking
on Disable showing or Disable logging.

However, the warning indicates a problem with Org parser cache. Org
recovers from it, but not necessarily immediately. You may get certain
edits bugging out if the parser cache is not correct. For example,
certain edits may "lose" first star of a heading behind. That's why we
are so into the face of users with this warning you are seeing.

> * What happens
>
> When I type some equations and get the preview, I get this warning
>
> #+begin_quote
> \begin{gather}
> t = 0
> \end{gather}
> #+end_quote
>
> * My "system"
>
> Emacs  : GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, 
> cairo version 1.17.6)
>  of 2023-01-03
> Package: Org mode version 9.4.6 (release_9.4.6-13-g4be129)
>
> * The message
>
> Warning (org-element-cache): org-element--cache: Org parser error in 
> t.org::35. Resetting.
>  The error was: (error "Invalid search bound (wrong side of point)")

The Org version is very strange. Emacs 28 ships with Org 9.5.5 and the
latest stable version of Org available on ELPA is Org 9.6.7. The warning
comes from Org >9.6.

It appears to me that you have some problem with Org installation.

May you try to install the latest stable Org first, following
https://orgmode.org/manual/Installation.html? You should get M-x
org-version printing Org 9.6.7.

Then, let us know if the warning keeps appearing.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Inconvenient end-of-file behavior in org-mode 9.6 (emacs 29.1)

[Ihor Radchenko]

Kaiyu Zheng  writes:

> In 28.3 and before, when I do `M->`, and hit 'Enter', I get to the end
> of file on a new line outside of SubSec2, like so:
>
> * Sec1
> ** SubSec1
> ** SubSec2
> |
>
> And now I could type `** SubSec3` to add a new section like below:
>
> * Sec1
> ** SubSec1
> ** SubSec2
> ** SubSec3|
>
> But in 29.1, when I do the same sequence, the cursor is "trapped" in
> SubSec2, and I had to expand SubSec2 first in order to see what I'm
> typing. I don't like this behavior -- I wonder why it is this way in
> the new version. How is this better? Could we revert back?

You can try some other key that is not enter. For example, .
'Enter' should work on main, the development branch.

I will see what can be done on stable Org version.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Inconvenient end-of-file behavior in org-mode 9.6 (emacs 29.1)

[Kaiyu Zheng]

Dear Org-mode maintainers,

Org-mode has been fantastic. However, in the most recent release as
part of emacs 29.1, I am experiencing an inconvenience that made me
switch back to 28.3.  (I learned from this page
https://irreal.org/blog/?p=10982 that emacs 29.1 released org-mode
9.6).

The issue is: let's say I have a simple org-mode file, and let's say
the cursor starts at the top of the file (marked by '|'):

|* Sec1
** SubSec1
** SubSec2

In 28.3 and before, when I do `M->`, and hit 'Enter', I get to the end
of file on a new line outside of SubSec2, like so:

* Sec1
** SubSec1
** SubSec2
|

And now I could type `** SubSec3` to add a new section like below:

* Sec1
** SubSec1
** SubSec2
** SubSec3|

But in 29.1, when I do the same sequence, the cursor is "trapped" in
SubSec2, and I had to expand SubSec2 first in order to see what I'm
typing. I don't like this behavior -- I wonder why it is this way in
the new version. How is this better? Could we revert back?

Now, you could argue that I should type in the keybinding for
inserting the subsection instead of doing this manually. But I simply
don't remember (and never tried remembering) the keybinding for adding
a new section -- I don't think I'm wrong by not remembering that
keybinding. I was able to do what I wanted before in a simple and
natural way.

I hope my concern is valid.

Best,
Kaiyu

Re: [MAINTENANCE] Org orphanage?

[Alexis]

Ihor Radchenko  writes:

https://github.com/flexibeast/org-vcard page explicitly says 
that it is
looking for a new maintainer at least since 9 months ago. (The 
author is

in CC for this thread)


Indeed, and i've been actively following this discussion. :-)

i've only had one bite, in private, and that was from someone who 
wasn't particularly familiar with either Lisps or vCard, but who 
was offering to pitch in alongside me. i explained that it's not 
that i need an assistant, it's that i need a replacement, as i can 
no longer be a maintainer. (i've got so much on my life plate that 
i'm trying to wind back my volunteer ICT commitments to a 
relatively small core, including my man page ports, and random 
straightforward contributions involving few to no hoops.) The 
person felt they were not suitable for such a role.


Also, as a general note, i'm actually no longer using Org for 
contacts management, so i'm not even dogfooding org-vcard anymore.



Alexis.

Re: org-assert-version considered harmful

[Ihor Radchenko]

Ihor Radchenko  writes:

> 1. emacs -Q
> 2. M-x org-version (built-in)
> 3. M-: (push "/path/to/git/version/of/org/lisp" load-path)
> 4. M-x org-mode
> 5. Observe recursive loading error

... which is also happening with the other approach using (provide
'org-foo-9.7-pre)

progn: Recursive load: "/home/yantar92/Git/org-mode/lisp/org-element.el", 
"/home/yantar92/Git/org-mode/lisp/org.el", 
"/home/yantar92/Git/org-mode/lisp/org-element.el", 
"/home/yantar92/Git/org-mode/lisp/org.el", 
"/home/yantar92/Git/org-mode/lisp/org-element.el", 
"/home/yantar92/Git/org-mode/lisp/org.el", 
"/home/yantar92/Git/org-mode/lisp/org-element.el", 
"/home/yantar92/Git/org-mode/lisp/org.el", 
"/home/yantar92/Git/org-mode/lisp/org-element.el

I should be missing something.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: org-assert-version considered harmful

[Ihor Radchenko]

Stefan Monnier  writes:

 But then it will not run during byte-compilation.
>>> Yeah, I was assuming that you had replaced all `require`s with
>>> `org-require-with-shadowcheck`, but that's probably not what you'd done.
>> That's exactly what I have done.
>
> Ah.  The issue comes from the fact that `require` is treated specially
> by the byte compiler, so if you define `org-require-with-shadowcheck` as
> a function it indeed won't do quite the same as `require`.
>
> The way `require` is treated specially by the byte-compiler only affects
> top-level uses (making them behave as if they're wrapped inside
> `eval-and-compile`).  So you want to use `eval-and-compile` only for
> those `require`s that are at top-level.

I see. Then, I can resolve the issue simply by
(put 'org-require-with-shadowcheck 'byte-hunk-handler 
'byte-compile-file-form-require)

> Personally, I'd only change those `require`s that load `org-macs`, which
> should be enough to cover almost all cases (and I wouldn't just
> silently reload the file, but I'd also emit a warning explaining that
> we detected a bad situation and using a workaround since that
> workaround is not 100% reliable anyway).

I first want to try the most generous way to reveal any potential
pitfalls.

Like recursive load that is still happening when I do

1. emacs -Q
2. M-x org-version (built-in)
3. M-: (push "/path/to/git/version/of/org/lisp" load-path)
4. M-x org-mode
5. Observe recursive loading error

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: org-assert-version considered harmful

[Stefan Monnier]

>>> But then it will not run during byte-compilation.
>> Yeah, I was assuming that you had replaced all `require`s with
>> `org-require-with-shadowcheck`, but that's probably not what you'd done.
> That's exactly what I have done.

Ah.  The issue comes from the fact that `require` is treated specially
by the byte compiler, so if you define `org-require-with-shadowcheck` as
a function it indeed won't do quite the same as `require`.

The way `require` is treated specially by the byte-compiler only affects
top-level uses (making them behave as if they're wrapped inside
`eval-and-compile`).  So you want to use `eval-and-compile` only for
those `require`s that are at top-level.

Personally, I'd only change those `require`s that load `org-macs`, which
should be enough to cover almost all cases (and I wouldn't just
silently reload the file, but I'd also emit a warning explaining that
we detected a bad situation and using a workaround since that
workaround is not 100% reliable anyway).


Stefan

Re: org-assert-version considered harmful

[Ihor Radchenko]

Stefan Monnier  writes:

>> But then it will not run during byte-compilation.
>
> Yeah, I was assuming that you had replaced all `require`s with
> `org-require-with-shadowcheck`, but that's probably not what you'd done.

That's exactly what I have done.

> Not knowing what you had done (beside define
> `org-require-with-shadowcheck`), it's hard to reproduce/investigate, tho.

https://git.sr.ht/~yantar92/org-mode/tree/feature/shadowcheck

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[BUG] Warning when creating preview

[Edgar Lux]

Hi,

I don't really need help with this. Org has been telling me for days that I 
should report this, but I don't really want to share my configuration publicly, 
and I thought that it was only for a file on which I was working. It seems to 
be general for my configuration. I hope that it helps. If you have an encrypted 
personal account to which I can send my config file, let me know.

* What happens

When I type some equations and get the preview, I get this warning

#+begin_quote
\begin{gather}
t = 0
\end{gather}
#+end_quote

* My "system"

Emacs  : GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, 
cairo version 1.17.6)
 of 2023-01-03
Package: Org mode version 9.4.6 (release_9.4.6-13-g4be129)

* The message

Warning (org-element-cache): org-element--cache: Org parser error in t.org::35. 
Resetting.
 The error was: (error "Invalid search bound (wrong side of point)")
 Backtrace:
"  backtrace-to-string(nil)
  (progn (backtrace-to-string (backtrace-get-frames 'backtrace)))
  (if (and (fboundp 'backtrace-get-frames) (fboundp 'backtrace-to-string)) 
(progn (backtrace-to-string (backtrace-get-frames 'backtrace
  (format \"Org parser error in %s::%S. Resetting.\\n The error ...\" 
(buffer-name (current-buffer)) epom err (if (and (fboundp 
'backtrace-get-frames) (fboundp 'backtrace-to-string)) (progn 
(backtrace-to-string (backtrace-get-frames 'backtrace)
  (let* ((format-string (format \"Org parser error in %s::%S. Resetting.\\n The 
error ...\" (buffer-name (current-buffer)) epom err (if (and (fboundp 
'backtrace-get-frames) (fboundp 'backtrace-to-string)) (progn 
(backtrace-to-string (backtrace-get-frames ...)) (format-string (if (or 
(not org-element--cache-diagnostics-ring) (not (eq 'backtrace 
org-element--cache-self-verify))) format-string (prog1 (concat (format 
\"Warning(%s): \" (buffer-name ...)) format-string \"\\nBacktrace:\\n  \" 
(mapconcat #'identity (ring-elements org-element--cache-diagnostics-ring) \"\\n 
 \")) (setq org-element--cache-diagnostics-ring nil) (if (and (boundp 
'org-batch-test) org-batch-test) (error \"%s\" (concat \"org-element--cache: \" 
format-string)) (display-warning 'org-element-cache (concat 
\"org-element--cache: \" format-string
  (condition-case err (org-element--parse-to epom) ((debug error) (let* 
((format-string (format \"Org parser error in %s::%S. Resetting.\\n The error 
...\" (buffer-name (current-buffer)) epom err (if (and ... ...) (progn ... 
(format-string (if (or (not org-element--cache-diagnostics-ring) (not ...)) 
format-string (prog1 (concat ... format-string \"\\nBacktrace:\\n  \" ...) 
(setq org-element--cache-diagnostics-ring nil) (if (and (boundp 
'org-batch-test) org-batch-test) (error \"%s\" (concat \"org-element--cache: \" 
format-string)) (display-warning 'org-element-cache (concat 
\"org-element--cache: \" format-string (org-element-cache-reset) 
(org-element--parse-to epom)))
  (if cached-only (if (and (org-element--cache-active-p) (or (not 
org-element--cache-sync-requests) (< epom (aref (car 
org-element--cache-sync-requests) 1 (progn (org-element--cache-find epom))) 
(condition-case err (org-element--parse-to epom) ((debug error) (let* 
((format-string (format \"Org parser error in %s::%S. Resetting.\\n The error 
...\" (buffer-name ...) epom err (if ... ...))) (format-string (if (or ... ...) 
format-string (prog1 ... ... (if (and (boundp 'org-batch-test) 
org-batch-test) (error \"%s\" (concat \"org-element--cache: \" format-string)) 
(display-warning 'org-element-cache (concat \"org-element--cache: \" 
format-string (org-element-cache-reset) (org-element--parse-to epom
  (setq element (if cached-only (if (and (org-element--cache-active-p) (or (not 
org-element--cache-sync-requests) (< epom (aref (car 
org-element--cache-sync-requests) 1 (progn (org-element--cache-find epom))) 
(condition-case err (org-element--parse-to epom) ((debug error) (let* 
((format-string (format \"Org parser error in %s::%S. Resetting.\\n The error 
...\" ... epom err ...)) (format-string (if ... format-string ...))) (if (and 
(boundp ...) org-batch-test) (error \"%s\" (concat \"org-element--cache: \" 
format-string)) (display-warning 'org-element-cache (concat 
\"org-element--cache: \" format-string (org-element-cache-reset) 
(org-element--parse-to epom)
  (let (element) (if (org-element--cache-active-p) (progn (if (not 
org-element--cache) (org-element-cache-reset) (if cached-only nil 
(org-element--cache-sync (current-buffer) epom) (setq element (if 
cached-only (if (and (org-element--cache-active-p) (or (not 
org-element--cache-sync-requests) (< epom (aref ... 1 (progn 
(org-element--cache-find epom))) (condition-case err (org-element--parse-to 
epom) ((debug error) (let* ((format-string ...) (format-string ...)) (if (and 
... org-batch-test) (error \"%s\" ...) (display-warning ... ...))) 
(org-element-cache-reset) (org-element--parse-to epom) (if (and 

Re: org-assert-version considered harmful

[Stefan Monnier]

>> For this one I can see the problem.  You define:
>>
>> (defmacro org-require-with-shadowcheck (feature)
>>   [...]
>>   `(eval-and-compile ...))
>>
>> so it will behave like a function, except that it's also
>> executed during macroexpansion, so the (require 'org-element) within
>> `org-set-modules` will be eagerly executed while loading `org.el` :-(
>>
>> You should define `org-require-with-shadowcheck` as a function (just
>> like `require`).
>
> But then it will not run during byte-compilation.

Yeah, I was assuming that you had replaced all `require`s with
`org-require-with-shadowcheck`, but that's probably not what you'd done.

Not knowing what you had done (beside define
`org-require-with-shadowcheck`), it's hard to reproduce/investigate, tho.


Stefan

Re: Emacs 29.1, org-agenda and SCHEDULED entries

[Ihor Radchenko]

Ihor Radchenko  writes:

> We should update the parser to treat such malformed SCHEDULED/DEADLINE
> lines as ordinary paragraphs and report them in org-lint.

I went another way, without changing the existing syntax.

org-agenda will now ignore scheduled/deadline with inactive timestamps,
as it did in the past.
org-lint now has a checker that will report such scenarios.
Fixed, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=7cc208af9
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3cbd9f423

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] ob-sqlite: can not override header argument

[Ihor Radchenko]

Max Nikulin  writes:

> On 18/08/2023 01:03, Ihor Radchenko wrote:
>> The source says
>> 
>>;; for easy table parsing, default header type should be -csv
>> 
>> So, it is at least intentional.
>
> With ":results verbatim" it is not convincing.

Maybe. But not a bug either. I am also not a user of ob-sqlite, so I am
not in position to judge what is better. My current stance is not to
touch the working code.

If there are any users of ob-sqlite, I invite them to chime in, if they
have an opinion.

>>  csv
>>  the default SQLite output format for Babel SQLite source code blocks.
>
> I do not try to dispute that it is default. The issue is that ":csv no" 
> is not supported. It is at least not intuitive that to disable :csv and 
> to activate :separator it is necessary to add :list. So the effect of 
> sqlite3 command line options differs from the effect of ob-sqlite header 
> arguments.

Sure. But it is not exactly a bug as well. I would accept patches that
will allow disabling these header arguments, if someone is bothered
enough to submit them.

Handled. We can continue the discussion though.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] ob-sqlite: can not override header argument

[Max Nikulin]

On 18/08/2023 01:03, Ihor Radchenko wrote:

The source says

  ;; for easy table parsing, default header type should be -csv

So, it is at least intentional.


With ":results verbatim" it is not convincing.


Note that (member :csv others) yielding "" is also intentional because
it is already added by that time in %others.

It is also documented to be the default:

 csv
 the default SQLite output format for Babel SQLite source code blocks.


I do not try to dispute that it is default. The issue is that ":csv no" 
is not supported. It is at least not intuitive that to disable :csv and 
to activate :separator it is necessary to add :list. So the effect of 
sqlite3 command line options differs from the effect of ob-sqlite header 
arguments.


I am not an active sqlite user and I am almost unfamiliar with its 
options, so I am unsure what is the best way to integrate sqlite into 
org-babel.

Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Ihor Radchenko]

Max Nikulin  writes:

> Ihor, this is a list, not an expression to be evaluated. There are some 
> conditions to avoid user prompts for strings, lists, etc. They are 
> considered safe.
>
> This particular case is handled namely by ob-sqlite and the proposed 
> function in org-macs.

Do you have any ideas how to work around the deliberately constructed
header argument values like in your example?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Max Nikulin]

On 18/08/2023 15:43, Ihor Radchenko wrote:

Max Nikulin writes:


#+begin_src sqlite :db '(literal "/tmp/ob.sqlite$(date
  >/tmp/ob-sqlite-vuln.log)")
select 1
#+end_src


Handling lisp values in header arguments is much more general issue not
tied to ob-sql or even to running shell commands.

It should be addressed alongside with 
https://orgmode.org/list/87edsd5o89.fsf@localhost


Ihor, this is a list, not an expression to be evaluated. There are some 
conditions to avoid user prompts for strings, lists, etc. They are 
considered safe.


This particular case is handled namely by ob-sqlite and the proposed 
function in org-macs.

Re: org-assert-version considered harmful

[Ihor Radchenko]

Stefan Monnier  writes:

>> My attempt to use shadowcheck idea you proposed failed with some very
>> strange errors and I gave up.
>> See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#311
>
> For this one I can see the problem.  You define:
>
> (defmacro org-require-with-shadowcheck (feature)
>   [...]
>   `(eval-and-compile ...))
>
> so it will behave like a function, except that it's also
> executed during macroexpansion, so the (require 'org-element) within
> `org-set-modules` will be eagerly executed while loading `org.el` :-(
>
> You should define `org-require-with-shadowcheck` as a function (just
> like `require`).

But then it will not run during byte-compilation.
And compilation will yield

[...]
org-datetree.el:278:14: Warning: the function ‘org-cut-subtree’ is not known to 
be defined.
org-datetree.el:264:22: Warning: the function ‘org-up-heading-safe’ is not 
known to be defined.
org-datetree.el:263:35: Warning: the function ‘org-back-to-heading’ is not 
known to be defined.
org-datetree.el:257:37: Warning: the function ‘org-time-string-to-time’ is not 
known to be defined.
org-datetree.el:237:6: Warning: the function ‘org-paste-subtree’ is not known 
to be defined.
[...]

and no single macro from other library will be expanded.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>IR > May you share your changes?
>Sure!
>Here they are: In these slices I take the upper part of the fork (where in 
> case, assuming a small-big usage convention ;)) as that is the value that 
> surely testify the effort estimation overrun. Being so, at the time of this 
> writing I just realized the use of pcase is likely replaceable by a simpler 
> (car (last (mapcar...))) call :).
>> (let* ((effort-in-minutes
>
>> (pcase (mapcar #'org-duration-to-minutes (split-string 
> org-clock-effort "-"))
>
>>   (`(,_ ,value) value)
>
>>   (`(,value) value)))

I see.
As I said, Org does not currently support EFFORT to be a range.
Ranges only appear in column est+ summaries.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [MAINTENANCE] Org orphanage?

[Ihor Radchenko]

Jonas Bernoulli  writes:

>> What about https://github.com/flexibeast/org-vcard and
>> https://github.com/nikclayton/ob-sql-mode ?
>
> Are you saying these packages are unmaintained and asking whether they
> should be moved to the orphanage?

https://github.com/flexibeast/org-vcard page explicitly says that it is
looking for a new maintainer at least since 9 months ago. (The author is
in CC for this thread)

The author (also CCed) of https://github.com/nikclayton/ob-sql-mode
explicitly asked for help 8 months ago in
https://list.orgmode.org/CAKJTzL5bdw=vcbk0s9o3dfh2fkasro3m++whqmhcp9obaph...@mail.gmail.com/T/#u
We got no volunteers.

>> Also, we might want to add org-json and org-redmine to
>> https://orgmode.org/worg/org-orphanage.html
>
> Yes.

Done.
https://git.sr.ht/~bzg/worg/commit/494157f6

> Unmaintained packages frequently come up on Melpa, so we have documented
> how we try to handle that.  A lot of what is being said there, also
> applies when packages in need are brought up elsewhere.
>
> https://github.com/melpa/melpa/wiki/Unmaintained-Packages-and-Forks.

It will be a good idea to document these things in WORG as well. Under a
separate section in org-orphanage.org like "I want to volunteer
maintaining a package from this list".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [patch] ox-latex.el: fix blank lines behavior in verse block

[Ihor Radchenko]

Juan Manuel Macías  writes:

> Subject: [PATCH] lisp/ox-latex.el: Add the `:literal' attribute to verse
>  block.

Thanks!
Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=2eb4fd890

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG][SECURITY] ob-sqlite header args allows execution of arbitrary shell commands

[Ihor Radchenko]

Max Nikulin  writes:

> On 13/08/2023 14:52, Ihor Radchenko wrote:
>> What do you think about creating a new API to built shell commands and
>> then using it across all the babel backends?
>
> I support the idea in general, but not its particular implementation as 
> `org-make-shell-command' in your patch.
>
> It does not address the issue I raised.
>
> #+begin_src sqlite :db '(literal "/tmp/ob.sqlite$(date 
>  >/tmp/ob-sqlite-vuln.log)")
>select 1
> #+end_src

Handling lisp values in header arguments is much more general issue not
tied to ob-sql or even to running shell commands.

It should be addressed alongside with 
https://orgmode.org/list/87edsd5o89.fsf@localhost

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   IR > May you share your changes?
   Sure!
   Here they are: In these slices I take the upper part of the fork (where in 
case, assuming a small-big usage convention ;)) as that is the value that 
surely testify the effort estimation overrun. Being so, at the time of this 
writing I just realized the use of pcase is likely replaceable by a simpler 
(car (last (mapcar...))) call :).
   Cheers,
 Andrea

   Job: diff -bwi org-clock.el{.old,}

   720c720,723

   < (let* ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   > (let* ((effort-in-minutes

   > (pcase (mapcar #'org-duration-to-minutes (split-string 
org-clock-effort "-"))

   >   (`(,_ ,value) value)

   >   (`(,value) value)))

   828c831,833

   < (let ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   > (let ((effort-in-minutes (pcase (mapcar #'org-duration-to-minutes 
(split-string org-clock-effort "-"))

   >(`(,_ ,value) value)

   >(`(,value) value)))

   Da emacs-orgmode-bounces+andrea=fedeli...@gnu.org
   A and...@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Wed, 16 Aug 2023 10:15:13 +
   Oggetto Re: [BUG] Error in data input and output format for 
org-columns--summary-estimate
   and...@fedeli.eu writes:

   > Howdy!
   > I'm back to a previous element partially discussed as I found other org 
places where the duration had to be adapted to be able to deal with ranges: 
org-clock-get-clock-string and
   > org-clock-notify-once-if-expired, both in og-clock.el; both get into 
action if you have a task you estimated and for which you're now tracking 
development time (quite handy, I have to say, as you're immediately warned 
you've running beyond estimations. For those two functions, I have introduced a 
similar change to the one I did originally to go from the basic string-to 
number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for 
the debug-on-entry feature :))

   May you share your changes? I am not sure if I fully understand what and
   why you did without seeing the diff.

   > Two considerations here:
   > 1. I understand the fact that est+ doesn't have to necessarily be 
associated with effort, but it is quite clear from the docs which is the intent 
with which it was introduced: the only provided example is on times, and there 
we have to consider that time is expressed in durations.
   > What I mean is that it does NOT make much sense to me to tell users the 
effort is to be written as 3d if given as a single value, and it has to be 
rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if 
somewhere else some other duration unit is used..

   It should not. The reason `org-columns--summary-estimate' uses
   split-string is because it may have to work with recursively calculated
   estimates from subtrees. EFFORT property itself does not officially
   support ranges.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at .
   Support Org development at ,
   or support my work at 

Re: Add a Chinese version to index.org of orgmode.org

[lux]

On Tue, 2023-08-08 at 22:53 +0800, Ruijie Yu wrote:
> Hello Ihor, Lux and all,
> 
> On Aug 8, 2023, at 14:55, Ihor Radchenko  wrote:
> > 
> > lux  writes:
> > 
> > > Hi
> > >  To facilitate Chinese users' understanding of Org Mode, I have
> > > translated index.org into Simplified Chinese. Please review it.
> > 
> > Thanks!
> > However, we already have another, more complete translation
> > pending.
> > See https://list.orgmode.org/orgmode/sdvzg71zzor@netyu.xyz/
> 
> Thank you, Ihor, for mentioning my translation. 
> 
> I had been busy for a few months (and will be for a while as far as I
> know), and don’t currently have the energy to push the translation
> further until probably early October.  However, I believe what I had
> done was pretty much complete (apart from the changelog etc that are
> pointless to translate), so, Lux, please try to make improvements to
> it first (by, I guess, comparing your translation with mine and
> determining what needs rewording).  Getting that patchset installed
> would help the lot of Chinese speaking users in understanding Org
> better.  
> 
> FWIW, all my translation work was done before my employment status
> changed (and the patchset consisted of almost no real code), so I
> have no concern with seeing it installed without having gotten the
> employer disclaimer. 
> 
> P.S.: I do still scan over the Emacs mailing lists daily, but I’m
> only on mobile where I am not fully equipped for email conversations
> nor sending patches. :)
> 
> > -- 
> > Ihor Radchenko // yantar92,
> > Org mode contributor,
> > Learn more about Org mode at .
> > Support Org development at ,
> > or support my work at 
> > 
> 

Hello Ihor, Ruijie, 

I recently set out to compare translations, thanks.

Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots

[gerard . vermeulen]

On 18.08.2023 06:37, gerard.vermeu...@posteo.net wrote:

On 17.08.2023 14:10, Ihor Radchenko wrote:

gerard.vermeu...@posteo.net writes:

Your patches allow anyone to change 
org-babel-python--def-format-value.

For instance, I want to use black to "pretty-print" certain tree-like
structures


May you simply add an extra code to transform output as needed?


Yes, it is a way to switch between Jack's first and second set of 
patches if
one would like.  Or to add code to transform other Python data 
structures.


I take back the switching between Jack's first and second set of 
patches,

but I stand by "to add code to transform other Python data structures".