Re: Incorrect quantity of en-spaces

2023-10-18 Thread Tom Alexander
> This appears to be a special case, not documented on org-syntax page.

Sounds good, thanks! 

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Comments following not-really-affiliated keywords are becoming paragraphs

2023-10-16 Thread Tom Alexander
Thanks!

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Incorrect quantity of en-spaces

2023-10-16 Thread Tom Alexander
The org-mode syntax document describes entities as:
> \NAME POST
> \NAME{}
> Where NAME and POST are not separated by a whitespace character.

and POST is defined as:
> Either the end of line or a non-alphabetic character.

So using the test document:
```
\_   Foo
```
(a backslash, underscore, three spaces, and then the word Foo)

I would expect to get only 2 en-spaces but I am getting 3. Looking at 
org-entities, an underscore with 2 spaces gets 2 en-spaces, whereas an 
underscore with 3 spaces gets 3 en-spaces, but if we match all 3 spaces as NAME 
then POST becomes invalid because "F" is neither the end of the line nor a 
non-alphabetic character, so we can only match the first two spaces as NAME.

emacs version: 29.1
org-mode version: 9bbc21df84d507e568a3ebd17e105cdb9e163784 (latest in git)

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Clock becomes a paragraph by prefixing with not-really-affiliated-keyword

2023-10-16 Thread Tom Alexander
Thanks!

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Clock becomes a paragraph by prefixing with not-really-affiliated-keyword

2023-10-12 Thread Tom Alexander
This test document correctly parses as a clock:
```
CLOCK: [2023-04-21 Fri 19:43]
```

This test document incorrectly parses as a paragraph:
```
#+NAME: foo
CLOCK: [2023-04-21 Fri 19:43]
```

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Keyword becoming a paragraph based on optval

2023-10-12 Thread Tom Alexander
> Note that _affiliated keyword_ has an optional form of

Ah, that was what I was missing, thanks!

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Keyword becoming a paragraph based on optval

2023-10-12 Thread Tom Alexander
Emacs version: 29.1
Org-mode version: f3de4c3e041e0ea825b5b512dc0db37c78b7909e (latest in git)

This test document parses as a keyword:
```
#+CAPTION[*foo*]: baz
```

but this test document parses as a paragraph:
```
#+CAPTION[*foo* bar]: baz
```

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Comments following not-really-affiliated keywords are becoming paragraphs

2023-10-11 Thread Tom Alexander
Emacs version: 29.1
Org-mode version: e1569918cc94253650781e83a09695739c93352f (latest in git)

Test document:
```
#+CAPTION: foo
# bar
```

This parses as a paragraph with the caption of foo and the body of "# bar" when 
it should parse as a regular keyword followed by a comment.

Relevant org-syntax[1] bit:

> a keyword with the same KEY as an affiliated keyword may occur so long as it 
> is not immediately preceding a valid element that can be affiliated. For 
> example, an instance of #+caption: hi followed by a blank line will be parsed 
> as a keyword, not an affiliated keyword.


[1] https://orgmode.org/worg/org-syntax.html#Keywords

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Org-mode starting with 37d6bde27 errors out parsing org-mode/testing/examples/pub/a.org

2023-10-11 Thread Tom Alexander
Steps to reproduce:
 1. Build emacs 29.1
 2. Build org-mode with revision 37d6bde27fe228cdadcb5cdaa09287872a508777
 3. Run the following:
```
emacs -q --no-site-file --no-splash --batch --eval "(progn
 (require 'org)
 (setq vc-handled-backends nil)
 (find-file-read-only \"org-mode/testing/examples/pub/a.org\")
 (org-mode)
 (message \"%s\" (pp-to-string (org-element-parse-buffer)))
)"
```

I've attached a Dockerfile that reproduces the issue. Just throw that in a 
directory and run `docker build -t temp .` to see it fail. Change the `ARG 
ORG_VERSION=` line to `ac108a3ac1b332bf27ff2984a9cf26af3744185d` to see it 
succeed.

Error message:
```
File mode specification error: (void-function org-export--list-bound-variables)

Error: void-function (org-export--list-bound-variables)
  mapbacktrace(#f(compiled-function (evald func args flags) #))
  debug-early-backtrace()
  debug-early(error (void-function org-export--list-bound-variables))
  org-export--list-bound-variables()
  org-element--generate-copy-script(# :copy-unreadable 
do-not-check :drop-visibility t :drop-narrowing t :drop-contents t :drop-locals 
nil)
  org-element-copy-buffer(:to-buffer # :drop-visibility t 
:drop-narrowing t :drop-contents t :drop-locals nil)
  org-element-parse-secondary-string("<2014-03-04 Tue>" (bold citation code 
entity export-snippet inline-babel-call inline-src-block italic line-break 
latex-fragment link macro radio-target statistics-cookie strike-through 
subscript superscript target timestamp underline verbatim))
  org-macro--find-date()
  org-macro--collect-macros()
  org-macro-initialize-templates()
  org-mode()
  (progn (require 'org) (setq vc-handled-backends nil) (find-file-read-only 
"/input/home/talexander/git/org-mode/testing/examples/pub/a.org") (org-mode) 
(message "%s" (pp-to-string (org-element-parse-buffer
  command-line-1(("--no-splash" "--eval" "(progn\n (require 'org)\n 
(setq vc-handled-backends nil)\n (find-file-read-only 
\"/input/home/talexander/git/org-mode/testing/examples/pub/a.org\")\n 
(org-mode)\n (message \"%s\" (pp-to-string 
(org-element-parse-buffer)))\n)"))
  command-line()
  normal-top-level()
Symbol’s function definition is void: org-export--list-bound-variables
```

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc


Dockerfile
Description: Binary data


Document-level properties incorrect and/or missing based on preceding blank lines and/or comments

2023-10-11 Thread Tom Alexander
Emacs version: Emacs 29.1
Org-mode version: e1569918cc94253650781e83a09695739c93352f  (latest in git)

The org-mode syntax document[1] says that property drawers can exist in the 
zeroth section with the format:
```
BEGINNING-OF-FILE
BLANK-LINES
COMMENT
PROPERTYDRAWER
```

Using this test document:
```
:PROPERTIES:
:FOO:bar
:END:
```

I correctly get the foo property in the top-level org-data
```
(org-data
(:standard-properties
  [1 1 1 33 33 0 nil org-data nil nil nil 32 33 nil # nil nil 
nil]
  :path nil :FOO "bar" :CATEGORY nil)
```

But now there are two separate issues:

### Issue 1

Putting a comment before it makes the value for the foo property incorrect 
(seems to be grabbing an earlier string slice):
```
# baz
:PROPERTIES:
:FOO:bar
:END:
```

```
(org-data
(:standard-properties
  [1 1 1 39 39 0 nil org-data nil nil nil 38 39 nil # nil nil 
nil]
  :path nil :FOO "O: " :CATEGORY nil)
```

Interestingly, looking farther down the AST, the value for foo is properly set 
in the node-property, just not the org-data:
```
(node-property
(:standard-properties
 [20 20 nil nil 33 0 nil node-property nil nil nil nil nil nil # nil nil #2]
 :key "FOO" :value "bar"))
```

### Issue 2

Putting any blank lines before it makes the foo property not appear in org-data 
at all
```

:PROPERTIES:
:FOO:bar
:END:
```

```
(org-data
(:standard-properties
  [1 1 2 34 34 0 nil org-data nil nil nil 4 34 nil # nil nil 
nil]
  :path nil :CATEGORY nil)
```

Looking farther down the AST it seems the property-drawer became a regular 
drawer


[1] https://orgmode.org/worg/org-syntax.html#Property_Drawers

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Inconsistent text markup handling when double-nesting markers

2023-10-11 Thread Tom Alexander
> Fixed, on main.

Thanks!

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

2023-10-11 Thread Tom Alexander
Thanks!

> We aim to reduce config-dependent Org syntax in the long term.

Thats wonderful news! Sometimes this stuff can really surprise you. For 
example, the structure of the document created by running `echo "1. foo\n 
1.bar\n1.baz\n\t1.lorem"` changes based on the user's **tab-width**!!

If tab-width is less than 8 then this is:
```text
1. foo
  1. bar
1. baz
  2. lorem
```

If tab-width is 8 then this is:
```text
1. foo
  1. bar
1. baz
2. lorem
```

and if tab-width is greater than 8 this is:
```text
1. foo
  1. bar
1. baz
  1. lorem
```

Absolute madness! I always considered tab-width to be a personal aesthetic 
choice and not something that would functionally change how documents other 
people wrote will be parsed.

Idk if its been discussed, but personally if I were given dictatorship over 
org-mode I would take all of these emacs variables that are defined outside of 
the document, and instead of having them influence org-mode directly, I would 
*only* use them to pre-populate values for in-buffer settings templates.

For example, if a user had set `org-odd-levels-only` then I wouldn't have that 
impact ANY existing document they open, but if they open a new document then I 
would have it auto-insert `#+STARTUP: odd` at the top of the fresh document.

Otherwise it seems like org-mode is unsuitable for multi-person collaboration 
without dictating the contents of everyone's `.emacs` file.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Inconsistent text markup handling when double-nesting markers

2023-10-09 Thread Tom Alexander
I used the following test document:
```
__foo__

**foo**
```

I'd expect the two to behave the same but the first one parses as:
```
(paragraph
  "_"
  (subscript "foo")
  "__"
  )
```

Whereas the second parses as:
```
(paragraph
  (bold
(bold
  "foo"
  )
)
  )
```

This pattern happens in worg at [2]

Looking at the description for text markup in the syntax document[1], I don't 
see any reason the first wouldn't be parsed as an underline:

1. PRE: valid because it is the beginning of a line
2. MARKER: valid underscore
3. CONTENTS: valid. Series of objects from standard set includes both subscript 
and text markup, so regardless of how we parse the interior, its valid. Also 
cannot begin or end with whitespace but there is no whitespace in the CONTENTS.
4. MARKER: valid underscore
5. POST: Only valid if we extend the underline to the 2nd underscore so it ends 
at the end of the line. But the 2nd line shows us that having copies of the 
marker inside the CONTENTS is fine so I see two possible expected parses of the 
CONTENTS:
4a. (underline "foo")
4b. ((subscript "foo") (plain-text "_"))

I also ran the following test document to further prove that having copies of 
the marker inside the CONTENTS is fine:
```
*foo*bar*
```
which parses as (bold "foo*bar")

So the only way the top line would fail to parse as an underline is if it 
matched the first closing underscore as closing the underline, but that would 
be invalid because underscore is not a valid POST character and invalid copies 
of the closing marker are ignored as proven by both "**foo**" and "*foo*bar*".


[1] https://orgmode.org/worg/org-syntax.html#Emphasis_Markers
[2] 
https://git.sr.ht/~bzg/worg/tree/ba6cda890f200d428a5d68e819eef15b5306055f/org-contrib/babel/intro.org#L117

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Clarify that REST is not supported on the start TIME in a time-range timestamp.

2023-10-06 Thread Tom Alexander
> As for the problem with REST you raised, I am inclined to remove it from
> syntax doc for the time being - it only creates more confusion,
> unfortunately.

Makes sense, thanks. Is there anything we do to mark patches as rejected? I 
removed [PATCH] from the subject line.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Lesser blocks allowing unescaped lines

2023-10-06 Thread Tom Alexander
Thank you! Makes sense.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: [PATCH] Clarify that REST is not supported on the start TIME in a time-range timestamp.

2023-10-02 Thread Tom Alexander
Potentially related, org-mode is accepting this malformed timestamp from[1]:
```
<2016-02-14 Sun ++y>
```

The org-mode documentation[2] only includes REST with TIME, defining TIME as 
"H:MMREST". The above does not have any TIME but it accepts the timestamp 
anyway:
```
(timestamp
  :type active
  :range-type nil
  :raw-value "<2016-02-14 Sun ++y>"
  :year-start 2016
  :month-start 2
  :day-start 14
  :hour-start nil
  :minute-start nil
  :year-end 2016
  :month-end 2
  :day-end 14
  :hour-end nil
  :minute-end nil
)
```

Perhaps that grammar is wrong and REST needs to be separated from TIME?

[1] 
https://github.com/howardabrams/pdx-emacs-hackers/blob/bfb7bd640fdf0ce3def21f9fc591ed35d776b26d/workshops/org-mode-gtd-feature-demo.org#L183
[2] https://orgmode.org/worg/org-syntax.html#Timestamps

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



[PATCH] Clarify that REST is not supported on the start TIME in a time-range timestamp.

2023-10-02 Thread Tom Alexander
If REST is included in the first TIME on a time-range timestamp then the entire 
timestamp becomes a single range-less timestamp. To test I used the following 
test document:
```
[1970-01-01 Thu 8:15-13:15foo]
[1970-01-01 Thu 8:15foo-13:15]
```

The first line parses as a timerange from 8:15-13:15.
The second line parses as a single timestamp at 8:15.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc
From b1114e983d961d48e1d837b8d2ad209a976a5417 Mon Sep 17 00:00:00 2001
From: Tom Alexander 
Date: Mon, 2 Oct 2023 17:35:28 -0400
Subject: [PATCH] * org-syntax.org (Timestamps): Clarify that REST is not
 supported on the start TIME in a time-range timestamp.

If REST is included in the first TIME on a time-range timestamp then the entire timestamp becomes a single range-less timestamp. To test I used the following test document:
```
[1970-01-01 Thu 8:15-13:15foo]
[1970-01-01 Thu 8:15foo-13:15]
```

The first line parses as a timerange from 8:15-13:15.
The second line parses as a single timestamp at 8:15.
---
 org-syntax.org | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/org-syntax.org b/org-syntax.org
index c2061431..0c326ba8 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -1686,9 +1686,10 @@ -MM-DD DAYNAME
   - DAYNAME (optional) :: A string consisting of non-whitespace
 characters except =+=, =-=, =]=, =>=, a digit, or =\n=.
 + TIME (optional) :: An instance of the pattern =H:MMREST= where =H=
-  represents a one to two digit number (and can start with =0=), and =M=
-  represents a single digit.  =REST= can contain anything but =\n= or
-  closing bracket.
+  represents a one to two digit number (and can start with =0=), and
+  =M= represents a single digit.  =REST= can contain anything but =\n=
+  or closing bracket. =REST= cannot exist on the start TIME in a
+  time-range timestamp (the patterns with =TIME-TIME=).
 + REPEATER-OR-DELAY (optional) :: An instance of the following pattern:
   #+begin_example
 MARK VALUE UNIT
-- 
2.42.0



Re: Extra paragraphs incorrectly spawning when ":end:" appears.

2023-10-02 Thread Tom Alexander
Hmm thanks, that makes sense. I guess a post-processing step to merge adjacent 
paragraphs wouldn't work either since that wouldn't stitch together objects 
like the bold in this test document without re-parsing the entire paragraph:
```
foo *bar
:end:
baz*
```

oh well 路

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Extra paragraphs incorrectly spawning when ":end:" appears.

2023-09-30 Thread Tom Alexander
Same problem occurs with this sample document:
```
foo
#+BEGIN: bar
baz
```

which parses as:
```
(section
  (paragraph "foo\n")
  (paragraph "#+BEGIN: bar\nbaz\n)
)
```

again, no blank lines and no non-paragraph elements but the single paragraph 
got split in two.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Extra paragraphs incorrectly spawning when ":end:" appears.

2023-09-30 Thread Tom Alexander
This test document has 1 paragraph:
```
foo
bar
baz
```
which parses as:
```
(section
  (paragraph "foo\nbar\nbaz\n")
)
```

This test document should have 1 paragraph but org-mode is parsing it as 2:
```
foo
:end:
baz
```

which parses as:
```
(section
  (paragraph "foo\n")
  (paragraph ":end:\nbaz\n")
)
```

The paragraph documentation[1] states that:
> Empty lines and other elements end paragraphs.

But the document contains no empty lines and we can see in the output that it 
only contains paragraphs.

[1] https://orgmode.org/worg/org-syntax.html#Paragraphs

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc


Lesser blocks allowing unescaped lines

2023-09-29 Thread Tom Alexander
This happens in worg at: 
https://git.sr.ht/~bzg/worg/tree/ba6cda890f200d428a5d68e819eef15b5306055f/exporters/ox-docstrings.org#L2490

The documentation for lesser blocks[1] states:
> Lines beginning with an asterisk or `#+` must be quoted by a comma (`,*`, 
> `,#+`).

However, the following test document parses as a lesser block despite 
containing a line starting with an unescaped #+:
```
#+CATEGORY: foo
#+begin_src text
#+CATEGORY: bar
#+end_src
```

which parses as:
```
(org-data
(:standard-properties
  [1 1 1 60 60 0 nil org-data nil nil nil 3 60 nil # nil nil 
nil]
  :path nil :CATEGORY "foo")
(section
  (:standard-properties
   [1 1 1 60 60 0 nil first-section nil nil nil 1 60 nil # 
nil nil #0])
  (keyword
   (:standard-properties
[1 1 nil nil 17 0 nil top-comment nil nil nil nil nil nil # nil nil #1]
:key "CATEGORY" :value "foo"))
  (src-block
   (:standard-properties
[17 17 nil nil 60 0 nil nil nil nil nil nil nil nil # nil 
nil #1]
:language "text" :switches nil :parameters nil :number-lines nil 
:preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value 
"#+CATEGORY: bar\n"
```

whereas I would expect this to be
```
(section
  (keyword :key "CATEGORY" :value "foo")
  (paragraph "#+begin_src text")
  (keyword :key "CATEGORY" :value "bar")
  (paragraph "#+end_src")
)
```

This test document shows that lines with an unescaped "*" do break up the 
lesser block:
```
* foo
#+begin_src text
* bar
#+end_src
```


[1] https://orgmode.org/worg/org-syntax.html#Blocks

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter

2023-09-29 Thread Tom Alexander
It seems that COUNTER-SET[1] is not being honored when exporting to utf-8 for 
alphabetical lists even though it is honored for numeric lists. When exporting 
to html, COUNTER-SET is honored for both.

Test document:
```
# An ordered list starting at 3
1. [@3] foo


# An ordered list starting at 1
m. bar


# An ordered list starting at 11
m. [@k] baz
```

Launching emacs with: (Setting org-list-allow-alphabetical is necessary or else 
the alphabetical lists will become paragraphs)
```
emacs -q --eval '(setq org-list-allow-alphabetical t)' /tmp/test.org
```

When exporting to html you get (edited to remove whitespace for clarity):
```
foo
bar
baz
```

But when exporting to utf-8 you get: (whitespace removed again)
```
3. foo
m. bar
m. baz
```

Whereas I would expect:  (whitespace removed again)
```
3. foo
m. bar
k. baz
```

On a slightly related note: it seems the COUNTER-SET[1] allows single-letter 
values even when org-list-allow-alphabetical is nil. I don't think that is 
going to hurt anyone but I figured I should mention it in case its a bug (test 
doc: `1. [@k] foo` is a plain list starting at 11 even when 
org-list-allow-alphabetical is nil).

[1] https://orgmode.org/worg/org-syntax.html#Items

Emacs 29.1, Org-mode version 9.7-pre (release_9.6.8-781-gc70354)

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Subscript with parenthesis

2023-09-29 Thread Tom Alexander
> Not true. I tried
>
> b^(*asd*) and bold inside superscript does get parsed.

Ah thanks for double-checking! You're right, that is getting parsed. Not sure 
what test document I was using to make me think objects didn't work inside the 
parenthesis.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: [PATCH] Add backslash to list of POST characters for text markup

2023-09-29 Thread Tom Alexander
Thanks!

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc

On Fri, Sep 22, 2023, at 5:29 AM, Ihor Radchenko wrote:
> "Tom Alexander"  writes:
>
>> Backslash appears to be supported. To test I used the following test 
>> document:
>> ```
>> foo ~bar~\& baz
>> ```
>
> Thanks!
> You are right.
> Applied, onto master, with minor amendments to the commit message.
> https://git.sr.ht/~bzg/worg/commit/ba6cda89
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>



[PATCH] Add backslash to list of POST characters for text markup

2023-09-21 Thread Tom Alexander
Backslash appears to be supported. To test I used the following test document:
```
foo ~bar~\& baz
```

This happens in a document in worg: 
https://git.sr.ht/~bzg/worg/tree/ae64e1a54185232d4ebdcab174d8d4319ffd564d/org-release-notes.org#L

The ampersand was chosen for the test document since that is not a supported 
POST character, to make sure backslash was not simply escaping the next 
character.

In the documentation I wrote out the word "backslash" in parenthesis to 
disambiguate between backslash and escaping the following comma.

Patch is attached.

--
Tom Alexander
pgp: https://fizz.buzz/pgp.ascFrom 098434680b5e3942acc00684a47389f2cdab6208 Mon Sep 17 00:00:00 2001
From: Tom Alexander 
Date: Thu, 21 Sep 2023 21:14:33 -0400
Subject: [PATCH] Add backslash to list of POST characters for text markup.

Backslash appears to be supported. To test I used the following test document:
```
foo ~bar~\& baz
```
The ampersand was chosen since that is not a supported POST character, to make sure backslash was not simply escaping the next character.

In the documentation I wrote out the word "backslash" in parenthesis to disambiguate between backslash and escaping the following comma.
---
 org-syntax.org | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/org-syntax.org b/org-syntax.org
index c5299741..8f0f9b0c 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -249,9 +249,9 @@ discarded.  This also applies to single-line elements.
 
 :This paragraph will not contain
 :a long sequence of spaces before "a".
-: 
+:
 :This paragraph does not have leading spaces according to the parser.
-: 
+:
 :#+begin_src emacs-lisp
 :  (+ 1 2)
 :#+end_src
@@ -1742,7 +1742,7 @@ whitespace characters.
   verbatim) or a series of objects from the standard set. In both
   cases, CONTENTS may not begin or end with whitespace.
 + [[#Special_Tokens][POST]] :: Either a whitespace character, =-=, =.=, =,=, =;=, =:=, =!=, =?=, ='=, =)=, =}=,
-  =[=, ="=, or the end of a line.
+  =[=, ="=, =\= (backslash), or the end of a line.
 
 *Examples*
 
-- 
2.42.0



Consecutive plain list items of different types

2023-09-21 Thread Tom Alexander
The org-mode documentation[1] states for plain lists that:
> List types are mutually exclusive at the same level of indentation, if both 
> types are present consecutively then they parse as separate lists.

first a minor nit-pick that "both" is probably not the correct word here since 
there are 3 types of lists, not two (unordered, ordered, and descriptive). I'd 
go with "multiple" instead IMO.

but more importantly, based on that description I would expect the following 
test document to parse into three separate plain lists, but it parses as a 
single plain list with 3 items:

```
1. foo
- bar
- lorem :: ipsum
```

[1] https://orgmode.org/worg/org-syntax.html#Plain_Lists

--
Tom Alexander
pgp: https://fizz.buzz/pgp.asc



Re: Subscript with parenthesis

2023-09-21 Thread Tom Alexander
Some additional things I'm noticing:

- when using parenthesis, :use-brackets-p is nil, so they're not equivalent to 
curly braces.
- it does not support objects inside the parenthesis, just plain text, which 
again means they're not equivalent to braces.
- it, however, seems to require that the parenthesis are balanced because this 
test document does NOT contain a subscript:
```
foo_(b(ar)
```
which is closer to the curly braces requirement since that seems to be the only 
part of the subscript/superscript documentation that mentions needing balance.

-- 
Tom Alexander



Subscript with parenthesis

2023-09-21 Thread Tom Alexander
The org-mode documentation[1] states that the SCRIPT portion of the 
subscript/superscript is either an asterisk, the standard set of objects 
wrapped in balanced curly braces, or an optional sign followed by "Either the 
empty string, or a string consisting of any number of alphanumeric characters, 
commas, backslashes, and dots"

But I'm seeing the following test document parse as containing a subscript 
despite using parenthesis which I do not think matches any of the above 
criteria:
```
foo_(bar)
```

[1] https://orgmode.org/worg/org-syntax.html#Subscript_and_Superscript

--
Tom Alexander




Re: Fixed width areas not allowing tab after leading colon.

2023-09-19 Thread Tom Alexander
Thanks!

-- 
Tom Alexander

On Sun, Sep 17, 2023, at 5:48 AM, Ihor Radchenko wrote:
> "Tom Alexander"  writes:
>
>> The documentation for fixed width areas states: A “fixed-width line” starts 
>> with a colon character (:) and either a whitespace character or the 
>> immediate end of the line. 
>> ...
>> Fixed-width area documentation: 
>> https://orgmode.org/worg/org-syntax.html#Fixed_Width_Areas
>
> org-syntax.html is not accurate here. The parser only allows ": " (colon
> followed by space) and no other variant.
>
> Fixed, on master.
> https://git.sr.ht/~bzg/worg/commit/a42f57ac
>
>> This happens in a document in worg: 
>> https://git.sr.ht/~bzg/worg/tree/74e80b0f7600801b1d1594542602394c085cc2f9/item/org-contrib/org-bom.org#L499
>
> Fixed, on master.
> https://git.sr.ht/~bzg/worg/commit/0c8d5679
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>



Re: [PATCH] Re: Description list with " :: " in the tag.

2023-09-19 Thread Tom Alexander
Sorry for the delay, I've been busy in the IRLs. I've updated the patch to 
reflect that the parser grabs the text before the last " :: " and then parses 
it as objects. The new patch is attached.

-- 
Tom Alexander

On Thu, Sep 14, 2023, at 7:24 AM, Ihor Radchenko wrote:
> "Tom Alexander"  writes:
>
>> I've written a patch (attached) with my proposed wording changes to
>> the documentation, should I be starting another thread or does
>> dropping it here work best?
>
> You can just modify subject with [PATCH], as I did.
>
>> ... I do not have commit access so I'd need
>> someone with such authority to do the last bit.
>
> Sure.
>
>> +  =TAG-TEXT= is one of more objects from the standard set so long as
>> +  they do not contain a newline character, until the last occurrence
>> +  of the substring =" :: "= (two colons surrounded by whitespace,
>> +  without the quotes).
>
> It does not fully represent what is going on - Org parser is top-down
> and does not parse objects before it is done parsing the descriptive
> list item. So,
>
> - *foo :: bar* does not actually contain bold markup
>
> Rather it is "* foo" tag + "bar* does not actually contain bold markup" 
> description.
>
> What happens is that the parser splits the first line of the item by the
> last " :: " and only then proceeds with parsing the tag and description
> using standard set of objects:
>
> - <> :: 
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>From c8812bf7d81dc824d8ecf2c03368f58884773ddf Mon Sep 17 00:00:00 2001
From: Tom Alexander 
Date: Wed, 13 Sep 2023 18:19:05 -0400
Subject: [PATCH] org-syntax.org: Fix definition of description list tags.

Description lists support objects in their tags and they support the substring " :: ".
---
 org-syntax.org | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/org-syntax.org b/org-syntax.org
index 123fc232..fc5e9a37 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -470,9 +470,10 @@ BULLET COUNTER-SET CHECK-BOX TAG CONTENTS
 + CHECK-BOX (optional) :: A single whitespace character, an =X=
   character, or a hyphen enclosed by square brackets (i.e. =[ ]=, =[X]=, or =[-]=).
 + TAG (optional) :: An instance of the pattern =TAG-TEXT ::= where
-  =TAG-TEXT= represents a string consisting of non-newline characters
-  that does not contain the substring =" :: "= (two colons surrounded by
-  whitespace, without the quotes).
+  =TAG-TEXT= is the text up until the last occurrence of of the
+  substring =" :: "= (two colons surrounded by whitespace, without the
+  quotes) on that line. =TAG-TEXT= is then parsed with the standard
+  set of objects.
 + CONTENTS (optional) :: A collection of zero or more elements, ending
   at the first instance of one of the following:
   - The next item.
-- 
2.42.0



Fixed width areas not allowing tab after leading colon.

2023-09-16 Thread Tom Alexander
The documentation for fixed width areas states: A “fixed-width line” starts 
with a colon character (:) and either a whitespace character or the immediate 
end of the line. 

Using the test document:
```
:foo
```

parses as a paragraph instead of a fixed-width area:
```
(org-data
(:standard-properties
  [1 1 1 7 7 0 nil org-data nil nil nil 3 7 nil # nil nil nil]
  :path nil :CATEGORY nil)
(section
  (:standard-properties
   [1 1 1 7 7 0 nil first-section nil nil nil 1 7 nil # nil 
nil #0])
  (paragraph
   (:standard-properties
[1 1 1 7 7 0 nil top-comment nil nil nil nil nil nil # 
nil nil #1])
   #(": foo\n" 0 6
 (:parent #2)
```

This happens in a document in worg: 
https://git.sr.ht/~bzg/worg/tree/74e80b0f7600801b1d1594542602394c085cc2f9/item/org-contrib/org-bom.org#L499

Emacs version: GNU Emacs 29.1 (build 1, x86_64-pc-linux-musl)
Org-mode version: c703541ffcc14965e3567f928de1683a1c1e33f6 (latest in git)

Fixed-width area documentation: 
https://orgmode.org/worg/org-syntax.html#Fixed_Width_Areas
--
Tom Alexander




Re: Description list with " :: " in the tag.

2023-09-13 Thread Tom Alexander
I've written a patch (attached) with my proposed wording changes to the 
documentation, should I be starting another thread or does dropping it here 
work best? I do not have commit access so I'd need someone with such authority 
to do the last bit.

-- 
Tom Alexander
From 20addaa5ab7d4e9420ade1125c2a337345ecdd31 Mon Sep 17 00:00:00 2001
From: Tom Alexander 
Date: Wed, 13 Sep 2023 18:19:05 -0400
Subject: [PATCH] org-syntax.org: Fix definition of description list tags.

Description lists support objects in their tags and they support the substring " :: ".
---
 org-syntax.org | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/org-syntax.org b/org-syntax.org
index 123fc232..3046e26c 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -470,9 +470,10 @@ BULLET COUNTER-SET CHECK-BOX TAG CONTENTS
 + CHECK-BOX (optional) :: A single whitespace character, an =X=
   character, or a hyphen enclosed by square brackets (i.e. =[ ]=, =[X]=, or =[-]=).
 + TAG (optional) :: An instance of the pattern =TAG-TEXT ::= where
-  =TAG-TEXT= represents a string consisting of non-newline characters
-  that does not contain the substring =" :: "= (two colons surrounded by
-  whitespace, without the quotes).
+  =TAG-TEXT= is one of more objects from the standard set so long as
+  they do not contain a newline character, until the last occurrence
+  of the substring =" :: "= (two colons surrounded by whitespace,
+  without the quotes).
 + CONTENTS (optional) :: A collection of zero or more elements, ending
   at the first instance of one of the following:
   - The next item.
-- 
2.42.0



Re: Description list with " :: " in the tag.

2023-09-11 Thread Tom Alexander
Thanks!

-- 
Tom Alexander

On Sat, Sep 9, 2023, at 5:06 AM, Ihor Radchenko wrote:
> "Tom Alexander"  writes:
>
>> Emacs version: 29.1
>> Org-mode version: 163bafb43dcc2bc94a2c7ccaa77d3d1dd488f1af
>>
>> Found a conflict between the documentation and the parser behavior. The 
>> org-mode documentation[1] for description list items says that TAG '[...] 
>> does not contain the substring " :: "'
>>
>> Using this sample document, I have created a plain list item with a tag that 
>> contains that substring by wrapping it in a verbatim block:
>> ```
>> - =foo :: bar= :: baz
>> ```
>>(item
>> ...
>> ((1 0 "- " nil nil "=foo :: bar=" 23))
>> ...
>> It seems that "TAG-TEXT" is not just text but it can include objects and 
>> those objects can include the substring " :: ".
>
> It is simpler.
> Everything after the bullet and before the last " :: " is considered as
> tag. Everything after the last " :: " is description.
> Then, tag and description are parsed, allowing objects inside.
>
> org-syntax document is inaccurate here - it says that the _first_ " :: "
> is used as tag:description delimiter, not the _last_.
>
> I do not see any benefit changing the current parser. So, we probably
> need to update org-syntax document instead.
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>



Description list with " :: " in the tag.

2023-09-08 Thread Tom Alexander
Emacs version: 29.1
Org-mode version: 163bafb43dcc2bc94a2c7ccaa77d3d1dd488f1af

Found a conflict between the documentation and the parser behavior. The 
org-mode documentation[1] for description list items says that TAG '[...] does 
not contain the substring " :: "'

Using this sample document, I have created a plain list item with a tag that 
contains that substring by wrapping it in a verbatim block:
```
- =foo :: bar= :: baz
```

Which parses to:
```
(org-data
 (:standard-properties
  [1 1 1 23 23 0 nil org-data nil nil nil 3 23 nil # nil nil 
nil]
  :path nil :CATEGORY nil)
 (section
  (:standard-properties
   [1 1 1 23 23 0 nil first-section nil nil nil 1 23 nil # 
nil nil #0])
  (plain-list
   (:standard-properties
[1 1 1 23 23 0 nil top-comment nil nil nil nil nil nil # 
nil
   ((1 0 "- " nil nil "=foo :: bar=" 23))
   #1]
:type descriptive)
   (item
(:standard-properties
 [1 1 19 23 23 0
(:tag)
item nil nil nil nil nil nil # nil
((1 0 "- " nil nil "=foo :: bar=" 23))
#2]
 :bullet "- " :checkbox nil :counter nil :pre-blank 0 :tag
 ((verbatim
   (:standard-properties
[3 nil nil nil 15 0 nil nil nil nil nil nil nil nil # 
nil nil #3]
:value "foo :: bar"
(paragraph
 (:standard-properties
  [19 19 19 23 23 0 nil nil nil nil nil nil nil nil # nil 
nil #3])
 #("baz\n" 0 4
   (:parent #4)))
```

It seems that "TAG-TEXT" is not just text but it can include objects and those 
objects can include the substring " :: ".

[1] https://orgmode.org/worg/org-syntax.html#Items

--
Tom Alexander




Re: [RFC] Quoting property names in tag/property matches [Was: [BUG?] Matching tags: & operator no more implicit between tags and special property]

2023-09-01 Thread Tom Gillespie
Ignore the previous message. I see that this was about matching
tags not about specifying them. Best,
Tom



Re: [RFC] Quoting property names in tag/property matches [Was: [BUG?] Matching tags: & operator no more implicit between tags and special property]

2023-09-01 Thread Tom Gillespie
Without wading too far into this, why do we need escape syntax for this?
The only character that might need an escape would be colon :, but
my reading of the syntax doc is that colo : will immediately terminate
the property, so we would update the doc to make it clear that property
names cannot contain a colon. As written, if there is an issue with the
minus sign in property names then that is a bug, but I feel like I might
be missing something?

Tom



Re: [DISCUSSION] May we recognize everything like [[protocol:uri]] as a non-fuzzy link? (was: [BUG] URI handling is overly complicated and nonstandard [9.6.7 (N/A @ /gnu/store/mg7223g8mw90lccp6mm5g6f3

2023-09-01 Thread Tom Gillespie
This is a timely discussion. I have been thinking about how
to deal with prefixes defined by the #+link: keyword which is
directly related to this question.

I think the following might be a solution that also avoids the
issue brought up by Arne.

The original "bug" cannot be resolved because bare URIs
have syntax that conflicts with Org syntax. However I think
we can do better than directing users to org-link-set-parameters.

My suggestion is as follows. Schemes/prefixes defined by the
#+link: keyword can be used without surrounding syntax markers
but may not contain spaces etc. To support this Org parsers
should always parse prefix:suffix as a _putative_ link which
must then be checked against a list of known schemes that
are either built in or have been declared by the user to indeed
be legitimate schemes.

In the tel: case, the way to solve the original bug is simply
to add the line #+link: tel tel: which would tell Org that e.g.
tel:555-555- is a real uri, and that it should expand to
itself.

At the same time this solution would avoid Arne's issue
(which I also have in some of my documents where I have
use fig: and tbl: as prefixes in names and reference them
via [[fig:figure-name]]) because the parser would only treat
prefix: in an internal link as a scheme if it is defined explicitly
by the user in a #+link: keyword or in their init.el.

Thoughts?
Tom



Text markup and TeX-style LaTeX fragments spanning more than three lines

2023-08-27 Thread Tom Alexander
The documentation for both text markup and TeX-style LaTeX fragments states 
that their contents (CONTENTS for text markup, BODY for TeX-style LaTeX 
fragments) cannot span more than three lines, but I'm not seeing such 
limitation for either in practice. I don't know if this is a documentation 
error or a parser error.

Documentation text markup: 
https://orgmode.org/worg/org-syntax.html#Emphasis_Markers
Documentation LaTeX fragments: 
https://orgmode.org/worg/org-syntax.html#LaTeX_Fragments

Test document text markup:
```
foo *bar
baz* lorem

text *markup
cannot
span* more

than *three
lines.
foo
bar* baz

foo *bar \\
baz \\
lorem \\
ipsum \\
dolar* cat
```

Test document TeX-style LaTeX fragments:
```
$foo
bar
baz
lorem
ipsum
dolar$
```

--
Tom Alexander




Re: Clarification on blank lines following list items

2023-08-24 Thread Tom Alexander
Thanks!

-- 
Tom Alexander

On Tue, Aug 22, 2023, at 4:26 AM, Ihor Radchenko wrote:
> Ihor Radchenko  writes:
>
>>> I am seeing dual-ownership of the post-blank in the examples below, but at 
>>> the same time if I put a plain-list inside a footnote definition, the 
>>> footnote definition ends up with sole custody of the post-blank.
>>
>> It is a bug in the list parser.
>
> Fixed, on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=53c9d91d3
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>



Re: Clarification on blank lines following list items

2023-08-20 Thread Tom Alexander
Thank you so much for explaining all of that! There is some good information 
there I was missing. I think the most important bit I was missing is the 
post-blank stuff. I was only looking at begin->end but I think digging into the 
post-blank is what makes this consistent.

I've got 2 separate questions:

1. Is the following statement true? "Two elements can count the same character 
in their post-blank?"
I am seeing dual-ownership of the post-blank in the examples below, but at the 
same time if I put a plain-list inside a footnote definition, the footnote 
definition ends up with sole custody of the post-blank.

2. I'm still not sure about some behavior I'm seeing. I think it would be 
easiest to see if we focus on exactly 1 blank line:

```
1. bar
2. baz
   < this blank line here
ipsum
```

In this example, the blank line gets counted in the post-blank for the 
plain-list but not for the item:
```
plain-list: post-blank 1 | begin 1 end 16 | contents-begin 1 contents-end 15
item: post-blank 0 | begin 1 end 8 | contents-begin 4 contents-end 8
paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 
8
item: post-blank 0 | begin 8 end 15 | contents-begin 11 contents-end 15
paragraph: post-blank 0 | begin 11 end 15 | contents-begin 11 
contents-end 15
paragraph: post-blank 0 | begin 16 end 22 | contents-begin 16 contents-end 22
```

but if we take that plain-list and nest it inside another plain-list:
```
1. foo
   1. bar
   2. baz
   < this blank line here
2. lorem
ipsum
```

The blank line gets counted as a post-blank for both the item "foo" and the 
item "baz":
```
plain-list: post-blank 0 | begin 1 end 38 | contents-begin 1 contents-end 38
item: post-blank 1 | begin 1 end 29 | contents-begin 4 contents-end 28
paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 
8
plain-list: post-blank 0 | begin 8 end 29 | contents-begin 8 
contents-end 29
item: post-blank 0 | begin 8 end 18 | contents-begin 14 
contents-end 18
paragraph: post-blank 0 | begin 14 end 18 | contents-begin 14 
contents-end 18
item: post-blank 1 | begin 18 end 29 | contents-begin 24 
contents-end 28
paragraph: post-blank 0 | begin 24 end 28 | contents-begin 24 
contents-end 28
item: post-blank 0 | begin 29 end 38 | contents-begin 32 contents-end 38
paragraph: post-blank 0 | begin 32 end 38 | contents-begin 32 
contents-end 38
paragraph: post-blank 0 | begin 38 end 44 | contents-begin 38 contents-end 44
```

Meaning the post-blank did this movement:
```
plain-list: post-blank 0
item: post-blank 1   <---<<<-\
paragraph: post-blank 0  |
plain-list: post-blank 0 >>--|
item: post-blank 0   |
paragraph: post-blank 0  |
item: post-blank 1   <---<---/
paragraph: post-blank 0
item: post-blank 0
paragraph: post-blank 0
paragraph: post-blank 0
```


Question ---> So why is the item "baz" gaining a post-blank instead of the 
inner plain-list (bar baz) keeping that post-blank?

I would expect it to instead be:
```
plain-list: post-blank 0
item: post-blank 1
paragraph: post-blank 0
here -> plain-list: post-blank 1
item: post-blank 0
paragraph: post-blank 0
not here -> item: post-blank 0
paragraph: post-blank 0
item: post-blank 0
paragraph: post-blank 0
paragraph: post-blank 0
```

I re-did both test cases using greater blocks and lesser blocks instead of 
paragraphs to make sure it wasn't that historical exception at the end of your 
email, and the post-blank behavior was exactly the same.


-- 
Tom Alexander



Clarification on blank lines following list items

2023-08-18 Thread Tom Alexander
  |
| bar | Yes   |
| baz | Yes   |
| lorem   | No|

Against expectations, we did not coerce lorem to consume its trailing blank 
lines. What is different between "baz" and "lorem"? Well, "baz" is contained 
within "foo" which has a "lorem" after it, whereas "lorem" is contained within 
"cat" which does not have any list items after it.

New theory: List items own their trailing blank lines unless they are both the 
final list item and not the final element of a non-final list item.
| Plain List *Item*   | Owns trailing blank lines | Why 
 |
|-+---+--|
| cat (includes foo -> lorem) | No| Final list item and 
not contained in a list item |
| foo (includes bar baz)  | Yes   | Not the final list 
item  |
| bar | Yes   | Not the final list 
item  |
| baz | Yes   | Final element of 
non-final list item |
| lorem   | No| Final list item and 
final element of final list item |


*** Test case 4
So if that theory is true, then we should be able to coerce lorem to consume 
its trailing blank lines by adding a second item to the cat list.
```
1. cat
   1. foo

  1. bar

  2. baz

   2. lorem

2. dog

ipsum
```
| Plain List *Item*   | Owns trailing blank lines |
|-+---|
| cat (includes foo -> lorem) | Yes   |
| foo (includes bar baz)  | Yes   |
| bar | Yes   |
| baz | Yes   |
| lorem   | Yes   |
| dog | No|

For the first time our expectations were met!

Enduring theory: List items own their trailing blank lines unless they are both 
the final list item and not the final element of a non-final list item.
| Plain List *Item*   | Owns trailing blank lines | Why 
 |
|-+---+--|
| cat (includes foo -> lorem) | Yes   | Not the final list 
item  |
| foo (includes bar baz)  | Yes   | Not the final list 
item  |
| bar | Yes   | Not the final list 
item  |
| baz | Yes   | Final element of 
non-final list item |
| lorem   | Yes   | Final element of 
non-final list item |
| dog | No| Final list item and 
not contained in a list item |


--
Tom Alexander



Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior

2023-08-16 Thread Tom Gillespie
On Wed, Aug 16, 2023 at 2:09 AM Ihor Radchenko  wrote:
>
> Tom Gillespie  writes:
>
> > Subject: [PATCH] ob-tangle.el: restore :tangle closure evaluation before 
> > eval
> >  info
> > This patch fixes a bug where header arguments like :tangle (or "no")
> > were treated as if they were tangling to a file named "(or \"no\")".
> > As a result, org-bable would call org-babel-get-src-block-info with
> > 'no-eval set to nil, causing parameters to be evaluated despite the
> > fact that when :tangle no or equivalent is set, the other parameters
> > should never be evaluated.
>
> What do you mean by "restore"? Were it evaluated in the past?
> May you please provide a reproducer?

Hrm. I think I may have mixed two commit lines. It is the case that
:tangle closures used to work, but you are right, the historical behavior
when tangling closures meant that all parameters were evaluated (tested
with the block below in 27 and 28).

#+begin_src elisp :var value=(error "oops") :tangle (or "no")
value
#+end_src

My use case is that I have blocks that I want to tangle that set :var
from e.g. the library of babel, which isn't always loaded, but which also
is not required if :tangle is no.

> > -(defun org-babel-tangle--unbracketed-link (params)
> > +(defun org-babel-tangle--unbracketed-link (params  
> > info-was-evaled)
>
> This is not acceptable. Taking care about evaluating INFO should be done
> in a single place instead of adding checks across the babel code. If we
> go the proposed way, I expect a number of bugs appearing when somebody
> forgets to change the eval check in some place.

I don't like the solution either. I see two potential alternatives.
1. change the structure of the info list to indicate whether it has
already been evaluated
2. always call org-babel-read on (cdr (assq :tangle params)) even
if it may already have been evaluated which can lead to some unexpected
and potentially nasty results.

I don't think we can consolidate evaluating parameters
into one place in the general case because there are
order dependencies where a setting in one param header
should mask others (as is the case here). In principle we
could consolidate them, but I think that would add significant
complexity because we would have to push all the logic for
handling whether a given ordering restriction applies inside
that location. e.g. if I have a block set :eval (if ev "yes" "no")
it would be bad form to evaluate the parameters before determining
whether the :eval closure evaluates to "yes" or "no". Should that
go inside org-process-params, or should it be handled locally
by e.g. org-babel-tangle and org-babel-execute-src-block separately?

Thoughts?



Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior

2023-08-16 Thread Tom Gillespie
> My confusion about you patch comes from the fact that
>
> #+begin_src emacs-lisp :tangle (if (= 1 1) "yes")
> 2
> #+end_src
>
> works just fine on main.

It appears to work fine on main, but that is because
what is actually happening behind the scenes is that in the test
(unless (or (string= src-tfile "no") ...) ...) the actual comparison is
(string= "(if (= 1 1) \"yes\")" "no") which appears to work, but is
not comparing the result of the closure, only its string value.

> I admit that I don't fully understand your use case.

I want to use a closure to conditionally control whether a block will tangle.
If I hardcode :tangle no, then :var x=(error "oops") will not evaluate. The
(error "oops") is a placeholder for a number of things that will result in
an error if the condition for :tangle (when condition "file-name") is not
satisfied.

> Something like (org-babel-get-heading-arg :tangle info/params)

I need to go to bed, because I definitely started on an implementation
of that I forgot about it as a potential solution. Yes, this seems like
a better and clearer way to go about it.

> May you please elaborate?

Disregard, your suggestion clarified what you meant, and in
that case, yes we can consolidate.



Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute

2023-08-16 Thread Tom Gillespie
> It was a slip when the patch was applied.
> See the table of :results params vs. expected output that Nicolas
> provided in
> https://list.orgmode.org/orgmode/87zjbqrapy@nicolasgoaziou.fr/
>
> Or maybe I miss something.
>
> May you please explain more about {{{results(=value=)}}} problem?
> Isn't it sufficient to do src_elisp[:results verbatim]{'value} 
> {{{results(=value=)}}}?

The issue is the opposite I think. Currently the default value (i.e. absent)
for :results does not produce {{{results(value)}}} as suggested, and instead
producers {{{results(=value=)}}}. This means that without :results drawer
there isn't an obvious way to get {{{results(value)}}} because you can't e.g.
use [:results default], or if a user overrides the default value for
inline header
args at file level then they have no way to reset to the default.

It looks like there used to be an option [:results wrap] which was deprecated
a _very_ long time ago. [:results drawer] replaced that, and while there is
some confusion about the name (because there is no actual drawer in an
inline result) the behavior was meant to replace the old :results wrap behavior
where the name does make sense since {{{results(value)}}} do "wrap" the value.

I think that covers it, but let me know if something doesn't make sense. Best,
Tom



Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute

2023-08-16 Thread Tom Gillespie
Confirming fixed. Thanks!

PS A new issue arises however caused by
487f39efa68fa2d857f8d446d1c4b3a3b3e3f482,
which is that it is now confusing  to get the {{{results(=value=)}}}
macro without verbatim which is what :results drawer meant in
that context. I expect that change will break things for a number
of people beyond myself. A bit of reading the code revealed that
setting :wrap t makes it possible to get {{{results(value)}}} again,
but line 2647 [1] seems to indicate that drawer is an expected
and valid value for inline :results.


1.
   ((or (member "drawer" result-params)
;; Stay backward compatible with <7.9.2
(member "wrap" result-params))
(goto-char beg) (when (org-at-table-p) (org-cycle))
(funcall wrap ":results:" ":end:" 'no-escape nil
 "{{{results(" ")}}}"))



Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute

2023-08-15 Thread Tom Gillespie
I have tested this again against bugfix, the issue is still present
when the cursor is on the s in #+caption: src_. Best,
Tom



Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior

2023-08-15 Thread Tom Gillespie
Here is a corrected patch that fixes the fact that closures passed to the
:tangle header were not being evaluated. Details in the commit message.
Best,
Tom

On Tue, Aug 15, 2023 at 6:41 PM Tom Gillespie  wrote:
>
> After a bit more investigation don't apply this patch because the change
> is insufficient to correct another issue.
>
> Specifically org-babel-tangle-collect-blocks must check for and resolve
> any closures that are passed to :tangle _before_ testing (string=
> src-tfile "no").
> As it stands blocks that are marked :tangle (and "no") with a closure
> incorrectly make it to a call to (org-babel-get-src-block-info) which causes
> a call to org-babel-process-params when :tangle would be "no".
>
> Working on a proper fix now.
From ce56fe5c1a24ba37c5c7c2f541c48684b0cb1e0b Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Tue, 15 Aug 2023 13:46:08 -0700
Subject: [PATCH] ob-tangle.el: restore :tangle closure evaluation before eval
 info

* lisp/ob-tangle.el (org-babel-tangle): check that :tangle is not no
before attempting to tangle a single block
(org-babel-tangle--unbracketed-link): new optional argument
info-was-evaled to control whether (cdr (assq :tangle params)) is
expanded via `org-babel-read'
(org-babel-tangle-collect-blocks): expand closures passed to :tangle
with `org-babel-read' so that blocks with :tangle closure that eval to
"no" will not incorrectly eval other parameters
(org-babel-tangle-single-block): src-tfile handle cases where output
of :tangle closure could be nil and conver to "no"

This patch fixes a bug where header arguments like :tangle (or "no")
were treated as if they were tangling to a file named "(or \"no\")".
As a result, org-bable would call org-babel-get-src-block-info with
'no-eval set to nil, causing parameters to be evaluated despite the
fact that when :tangle no or equivalent is set, the other parameters
should never be evaluated.

This patch also restores the original behavior of the :tangle header
argument when a closure returned nil, e.g. #+header: :tangle (or), by
using (or (cdr (assq :tangle params)) "no"). Without this the call to
`file-name-directory' in `org-babel-tangle--unbracketed-link' can fail
with a wrong-type-argument stringp nil error.
---
 lisp/ob-tangle.el | 32 +++-
 1 file changed, 23 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 4566e03ad..a9fb0e286 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -316,9 +316,13 @@ matching a regular expression."
 		 (write-region nil nil file-name)
 		 (mapc (lambda (mode) (set-file-modes file-name mode)) modes))
(push file-name path-collector))
-	 (if (equal arg '(4))
-	 (org-babel-tangle-single-block 1 t)
-	   (org-babel-tangle-collect-blocks lang-re tangle-file)))
+ (if (equal arg '(4))
+ (let* ((info (org-babel-get-src-block-info 'no-eval))
+(params (nth 2 info))
+(src-tfile (or (org-babel-read (cdr (assq :tangle params))) "no")))
+   (unless (string= src-tfile "no")
+ (org-babel-tangle-single-block 1 t)))
+   (org-babel-tangle-collect-blocks lang-re tangle-file)))
 	(message "Tangled %d code block%s from %s" block-counter
 		 (if (= block-counter 1) "" "s")
 		 (file-name-nondirectory
@@ -473,14 +477,14 @@ code blocks by target file."
 		  (org-in-archived-heading-p))
 	(let* ((info (org-babel-get-src-block-info 'no-eval))
 	   (src-lang (nth 0 info))
-	   (src-tfile (cdr (assq :tangle (nth 2 info)
+	   (src-tfile (or (org-babel-read (cdr (assq :tangle (nth 2 info "no")))
 	  (unless (or (string= src-tfile "no")
 		  (and tangle-file (not (equal tangle-file src-tfile)))
 		  (and lang-re (not (string-match-p lang-re src-lang
 	;; Add the spec for this block to blocks under its tangled
 	;; file name.
 	(let* ((block (org-babel-tangle-single-block counter))
-   (src-tfile (cdr (assq :tangle (nth 4 block
+   (src-tfile (or (cdr (assq :tangle (nth 4 block))) "no"))
 		   (file-name (org-babel-effective-tangled-filename
buffer-fn src-lang src-tfile))
 		   (by-fn (assoc file-name blocks)))
@@ -490,7 +494,7 @@ code blocks by target file."
 (mapcar (lambda (b) (cons (car b) (nreverse (cdr b
 	(nreverse blocks
 
-(defun org-babel-tangle--unbracketed-link (params)
+(defun org-babel-tangle--unbracketed-link (params  info-was-evaled)
   "Get a raw link to the src block at point, without brackets.
 
 The PARAMS are the 3rd element of the info for the same src block."
@@ -510,7 +514,10 @@ The PARAMS are the 3rd element of the info for the same src block.&q

Re: Export snippet matching without closing "@@"

2023-08-15 Thread Tom Alexander
> Thanks for reporting!
> Fixed, on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e340dde63

Thanks for fixing!

-- 
Tom Alexander



Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior

2023-08-15 Thread Tom Gillespie
After a bit more investigation don't apply this patch because the change
is insufficient to correct another issue.

Specifically org-babel-tangle-collect-blocks must check for and resolve
any closures that are passed to :tangle _before_ testing (string=
src-tfile "no").
As it stands blocks that are marked :tangle (and "no") with a closure
incorrectly make it to a call to (org-babel-get-src-block-info) which causes
a call to org-babel-process-params when :tangle would be "no".

Working on a proper fix now.



[PATCH] ob-tangle.el: restore :tangle closure nil behavior

2023-08-15 Thread Tom Gillespie
Hi,
   Here's a patch to fix the :tangle header behavior when it is
passed a closure that returns nil. Best,
Tom
From f1e15e0634fffed4648aa11628a14e0a68c3b18d Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Tue, 15 Aug 2023 13:46:08 -0700
Subject: [PATCH] ob-tangle.el: restore :tangle closure nil behavior

* lisp/ob-tangle.el (org-babel-tangle-collect-blocks):
* lisp/ob-tangle.el (org-babel-tangle--unbracketed-link):
* lisp/ob-tangle.el (org-babel-tangle-single-block): src-tfile fix
case where (cdr (assq :tangle params)) could be nil

This patch restores the original behavior of the :tangle header
argument when a closure returned nil, e.g. #+header: :tangle (or),
by using (or (cdr (assq :tangle params)) "no"). Without this the
call to `file-name-directory' in `org-babel-tangle--unbracketed-link'
can fail with a wrong-type-argument stringp nil error.
---
 lisp/ob-tangle.el | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 4566e03ad..0df649d7e 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -473,14 +473,14 @@ code blocks by target file."
 		  (org-in-archived-heading-p))
 	(let* ((info (org-babel-get-src-block-info 'no-eval))
 	   (src-lang (nth 0 info))
-	   (src-tfile (cdr (assq :tangle (nth 2 info)
+	   (src-tfile (or (cdr (assq :tangle (nth 2 info))) "no")))
 	  (unless (or (string= src-tfile "no")
 		  (and tangle-file (not (equal tangle-file src-tfile)))
 		  (and lang-re (not (string-match-p lang-re src-lang
 	;; Add the spec for this block to blocks under its tangled
 	;; file name.
 	(let* ((block (org-babel-tangle-single-block counter))
-   (src-tfile (cdr (assq :tangle (nth 4 block
+   (src-tfile (or (cdr (assq :tangle (nth 4 block))) "no"))
 		   (file-name (org-babel-effective-tangled-filename
buffer-fn src-lang src-tfile))
 		   (by-fn (assoc file-name blocks)))
@@ -510,7 +510,7 @@ The PARAMS are the 3rd element of the info for the same src block."
   (concat "file:"
   (file-relative-name (substring bare (match-end 0))
   (file-name-directory
-   (cdr (assq :tangle params)
+   (or (cdr (assq :tangle params)) "no"
 bare))
 
 (defvar org-outline-regexp) ; defined in lisp/org.el
@@ -580,7 +580,7 @@ non-nil, return the full association list to be used by
 			 (match-end 0)
 		   (point-min
 	  (point)
- (src-tfile (cdr (assq :tangle params)))
+ (src-tfile (or (cdr (assq :tangle params)) "no"))
 	 (result
 	  (list start-line
 		(if org-babel-tangle-use-relative-file-links
-- 
2.41.0



Re: [POLL] Should we accept breaking changes to get rid of Org libraries that perform side effects when loading?

2023-08-15 Thread Tom Gillespie
> Any other ideas? I'd be happy to see some brain-storming.

Maybe a #+keyword[options]: that doubles as a dynamic block or
something like that?
#+inline_task[options]: TODO some tiny task
#+inline_task[options]: TODO some small task
DEADLINE: <2023-11-11>
:PROPERTIES:
:SOMETHING: or other
:END:
#+inline_task_end:

Migration path should be straight forward, the exact implementation of
the behavior might need
a bit of work, and I'm not sure that the scheduling line will work in
that context, it is too fart
outside the usual behavior for keywords. However it might be possible
to move the deadline into [options]
the syntax would be a bit different than regular scheduling lines, but
it would at least be consistent
with keyword syntax.

The other question is whether you actually need an #+inline_task_end:
or whether there is another way.

The idea could use a few more iterations.



Export snippet matching without closing "@@"

2023-08-14 Thread Tom Alexander
I suspect this is a regression. I am using this test document:
```
@@latex:



\documentclass[margin,11pt]{res} % default is 10 pt



@@
```

and dumped the AST with M-x eval-expression (message "%s" (pp-to-string 
(org-element-parse-buffer)))

When using the latest in main (commit b89bc55867d7cb809c379d371d12d409db785154, 
Org mode version 9.7-pre (release_N/A-N/A-b89bc5 @ 
/usr/share/emacs/site-lisp/org/)) the "@@latex:" gets parsed as an export 
snippet despite not having a closing "@@" (because the "@@" is in a different 
paragraph).

When using the version of org-mode that shipped with emacs 29.1 (Org mode 
version 9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/)) the "@@latex:" 
gets parsed as plaintext.

The docs seem to show that a "@@" is required as the end of an export snippet 
based on the pattern "@@BACKEND:VALUE@@": 
https://orgmode.org/worg/org-syntax.html#Export_Snippets

AST from org-mode in main:
```
(org-data
 (:standard-properties
  [1 71 1 71 0 1 nil nil nil 3 71 org-data nil nil nil nil nil #]
  :path nil :CATEGORY nil)
 (section
  (:standard-properties
   [1 71 1 71 0 1 nil nil nil 1 71 first-section nil nil #0 nil nil #])
  (paragraph
   (:standard-properties
[1 13 1 10 3 1 nil nil nil nil nil top-comment nil nil #1 nil nil #])
   (export-snippet
(:standard-properties
 [1 9 nil nil 0 nil nil nil nil nil nil nil nil nil #2 nil nil #]
 :back-end "latex" :value nil))
   #("\n" 0 1
 (:parent #2)))
  (paragraph
   (:standard-properties
[13 68 13 65 3 13 nil nil nil nil nil nil nil nil #1 nil nil #])
   (latex-fragment
(:standard-properties
 [13 46 nil nil 1 nil nil nil nil nil nil nil nil nil #2 nil nil #]
 :value "\\documentclass[margin,11pt]{res}"))
   #("% default is 10 pt\n" 0 19
 (:parent #2)))
  (paragraph
   (:standard-properties
[68 71 68 71 0 68 nil nil nil nil nil nil nil nil #1 nil nil #])
   #("@@\n" 0 3
 (:parent #2)
```

AST from org-mode in emacs 29.1:
```
(org-data
 (:begin 1 :contents-begin 1 :contents-end 71 :end 71 :robust-begin 3 
:robust-end 69 :post-blank 0 :post-affiliated 1 :path nil :mode org-data 
:CATEGORY nil :granularity nil)
 (section
  (:begin 1 :end 71 :contents-begin 1 :contents-end 71 :robust-begin 1 
:robust-end 69 :post-blank 0 :post-affiliated 1 :mode first-section 
:granularity nil :parent #0)
  (paragraph
   (:begin 1 :end 13 :contents-begin 1 :contents-end 10 :post-blank 3 
:post-affiliated 1 :mode top-comment :granularity nil :parent #1)
   #("@@latex:\n" 0 9
 (:parent #2)))
  (paragraph
   (:begin 13 :end 68 :contents-begin 13 :contents-end 65 :post-blank 3 
:post-affiliated 13 :mode nil :granularity nil :parent #1)
   (latex-fragment
(:value "\\documentclass[margin,11pt]{res}" :begin 13 :end 46 :post-blank 1 
:parent #2))
   #("% default is 10 pt\n" 0 19
 (:parent #2)))
  (paragraph
   (:begin 68 :end 71 :contents-begin 68 :contents-end 71 :post-blank 0 
:post-affiliated 68 :mode nil :granularity nil :parent #1)
   #("@@\n" 0 3
 (:parent #2)
```

--
Tom Alexander



Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute

2023-08-14 Thread Tom Gillespie
> org-element-at-point is not supposed to return objects. So, there is no
> problem here.

Right. My original issue description is wrong for this part. It seems that
org-babel-get-src-block-info returns nil when immediately before the start
of the ala |src_ in #+caption: but will return non-nil in other contexts. In
the #+caption: it will only work if the cursor is past the s ala s|rc_. Or
if in non evil mode when the cursor is on the s in src_.

I can workaround the issue for now with forward-char, but it seems there
is an off-by-one error involving #+caption: followed immediately by src_,
if there is anything in between #+caption: and src_ then it works correctly.

> Also, C-c C-c works for me on bugfix in the above example.

Hrm, I can't get it to work, but I haven't done a proper control that
loads only bugfix and nothing else. Will report back if I get it to work.



[BUG] inline src blocks in caption of not-inline src blocks do not execute

2023-08-14 Thread Tom Gillespie
I've found a bug where inline source blocks cannot be executed and
org-element cannot find/parse them if they are in #+captions: where
the caption is affiliated with a not-inline source block. Example
below. Best,
Tom

Issue present in:
Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/)
GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version
1.17.8) of 2023-08-0

This inline src block works with C-c C-c.

#+caption: src_elisp[:results drawer]{"All good!"} {{{results(All good!)}}}
[[file:./images/happy-figure.png]]

This one fails to work with C-c C-c, and org-element-at-point returns
the bash src block below.

#+caption: src_elisp[:results drawer]{"I've made a horrible mistake"}
#+begin_src bash
echo OH NO
#+end_src



Re: [POLL] Should we accept breaking changes to get rid of Org libraries that perform side effects when loading?

2023-08-14 Thread Tom Gillespie
> Same here, I'd be tempted to deny Org citizenship to inline tasks: it
> always felt like a nice hack for a niche use-case, but a hack anyway.
>
> If it modifies Org syntax in surprising ways, this is another argument
> for removing org-inlinetask.el from Org's core.  Remember: this is not
> to say that inline tasks are forbidden, it's just a message for users
> that inline tasks are something not maintained by Org's core team.
>
> > And it is not clear how to fix this. We did not make inlinetasks into
> > standard Org syntax in the past and now it is in the weird state when we
> > have (featurep 'org-inlinetask) sprinkled across the code just to
> > accommodate for this conditional syntax.
>
> Yes, this is ugly.
>
> > Inlinetasks are too similar in syntax with headlines, so it is
> > impossible to make the change backwards-compatible.

Chiming in here to say that inline tasks make it exceptionally annoying
to specify a sane grammar for org because they require putting a special
case in the headline tokenizer, and/or making it possible to toggle the
behavior of the tokenizer.

As such I strongly support removing them from any official status in
the name of simplifying the syntax (among other things).



Re: Seemingly inconsistent whitespace ownership

2023-08-11 Thread Tom Alexander
On August 11, 2023 5:20:08 AM EDT, Ihor Radchenko  wrote:
>"Tom Alexander"  writes:
>
>> The org-mode documentation states: "With the exception of list items, blank 
>> lines belong to the preceding element with the narrowest possible scope. For 
>> example, if at the end of a section we have a paragraph and a blank line, 
>> that blank line is considered part of the paragraph."
>>
>> https://orgmode.org/worg/org-syntax.html#org9bc833d
>>
>> So I created a test org-mode document with exactly that scenario described:
>> ...
>> and then I ran a M-x eval-expression (message "%s" (pp-to-string 
>> (org-element-parse-buffer))) to see the parse tree. I'll paste the output at 
>> the end because its a wall of text, but analyzing it:
>> ...
>> Its only child is a paragraph which has a :begin at 1 and :end at 10, which 
>> only includes the line break at the end of "foo bar." but this is exactly 
>> the scenario described in the documentation (a paragraph at the end of a 
>> section) so I would expect the paragraph to end at 14, not 10.
>
>You are right and you revealed an existing inconsistency between our
>incremental and recursive parsers (org-element-parse-buffer is the
>recursive parser).
>
>Fixed, on main.
>https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bfe1179dc
>
>Also, in addition to list items, footnote-definitions do not extend
>their contents to the trailing blank lines.
>I now clarified it in org-syntax document.
>https://git.sr.ht/~bzg/worg/commit/3e91f5be
>
>-- 
>Ihor Radchenko // yantar92,
>Org mode contributor,
>Learn more about Org mode at <https://orgmode.org/>.
>Support Org development at <https://liberapay.com/org-mode>,
>or support my work at <https://liberapay.com/yantar92>

Oh neat, I didn't know there were two! I'll check that out. Thanks!

Seemingly inconsistent whitespace ownership

2023-08-10 Thread Tom Alexander
The org-mode documentation states: "With the exception of list items, blank 
lines belong to the preceding element with the narrowest possible scope. For 
example, if at the end of a section we have a paragraph and a blank line, that 
blank line is considered part of the paragraph."

https://orgmode.org/worg/org-syntax.html#org9bc833d

So I created a test org-mode document with exactly that scenario described:
```
foo bar.




* Lorem
baz




* Ipsum
alpha




beta
```

and then I ran a M-x eval-expression (message "%s" (pp-to-string 
(org-element-parse-buffer))) to see the parse tree. I'll paste the output at 
the end because its a wall of text, but analyzing it:

The zeroth section has :begin at 1 and :end at 14, which is expected.

Its only child is a paragraph which has a :begin at 1 and :end at 10, which 
only includes the line break at the end of "foo bar." but this is exactly the 
scenario described in the documentation (a paragraph at the end of a section) 
so I would expect the paragraph to end at 14, not 10.

The "Lorem" section is repeating the test to prove this isn't behavior unique 
to the zeroth section. It behaved the same, so there is no need to go over it.

The "Ipsum" section is where I put two separate paragraphs with a bunch of 
blank lines in between. This time the "alpha" paragraph goes all the way to 
character 48 which means it includes ALL of the blank lines between "alpha" and 
"beta".

So the "Ipsum" section proves that ":begin" to ":end" is supposed to contain 
the trailing blank lines of an element.

I am using org-mode version: Org mode version 9.6.6 (release_9.6.6 @ 
/usr/share/emacs/29.1/lisp/org/)

And finally, that parse tree I mentioned:

```
(org-data
 (:begin 1 :contents-begin 1 :contents-end 53 :end 53 :robust-begin 3 
:robust-end 51 :post-blank 0 :post-affiliated 1 :path nil :mode org-data 
:CATEGORY nil :granularity nil)
 (section
  (:begin 1 :end 14 :contents-begin 1 :contents-end 10 :robust-begin 1 
:robust-end 8 :post-blank 4 :post-affiliated 1 :mode first-section :granularity 
nil :parent #0)
  (paragraph
   (:begin 1 :end 10 :contents-begin 1 :contents-end 10 :post-blank 0 
:post-affiliated 1 :mode top-comment :granularity nil :parent #1)
   #("foo bar.\n" 0 9
 (:parent #2
 (headline
  (:raw-value "Lorem" :begin 14 :end 30 :pre-blank 0 :contents-begin 22 
:contents-end 26 :robust-begin nil :robust-end nil :level 1 :priority nil :tags 
nil :todo-keyword nil :todo-type nil :post-blank 4 :footnote-section-p nil 
:archivedp nil :commentedp nil :post-affiliated 14 :title
  (#("Lorem" 0 5
 (:parent #1)))
  :mode nil :granularity nil :parent #0)
  (section
   (:begin 22 :end 30 :contents-begin 22 :contents-end 26 :robust-begin 22 
:robust-end 24 :post-blank 4 :post-affiliated 22 :mode section :granularity nil 
:parent #1)
   (paragraph
(:begin 22 :end 26 :contents-begin 22 :contents-end 26 :post-blank 0 
:post-affiliated 22 :mode planning :granularity nil :parent #2)
#("baz\n" 0 4
  (:parent #3)
 (headline
  (:raw-value "Ipsum" :begin 30 :end 53 :pre-blank 0 :contents-begin 38 
:contents-end 53 :robust-begin 40 :robust-end 51 :level 1 :priority nil :tags 
nil :todo-keyword nil :todo-type nil :post-blank 0 :footnote-section-p nil 
:archivedp nil :commentedp nil :post-affiliated 30 :title
  (#("Ipsum" 0 5
 (:parent #1)))
  :mode nil :granularity nil :parent #0)
  (section
   (:begin 38 :end 53 :contents-begin 38 :contents-end 53 :robust-begin 38 
:robust-end 51 :post-blank 0 :post-affiliated 38 :mode section :granularity nil 
:parent #1)
   (paragraph
(:begin 38 :end 48 :contents-begin 38 :contents-end 44 :post-blank 4 
:post-affiliated 38 :mode planning :granularity nil :parent #2)
#("alpha\n" 0 6
  (:parent #3)))
   (paragraph
(:begin 48 :end 53 :contents-begin 48 :contents-end 53 :post-blank 0 
:post-affiliated 48 :mode nil :granularity nil :parent #2)
#("beta\n" 0 5
  (:parent #3))
```

--
Tom Alexander




Re: Documentation: Equal sign incorrectly listed as disallowed in plain links

2023-07-22 Thread Tom Alexander
Thanks for fixing all the documentation issues I submitted today!

On Sat, Jul 22, 2023, at 2:58 AM, Ihor Radchenko wrote:
> "Tom Alexander"  writes:
>
>> The documentation for plain links states that PATHPLAIN is:
>>
>>> A string containing non-whitespace non-bracket (()[]<>=) characters, 
>>> optionally containing parenthesis-wrapped non-whitespace non-bracket 
>>> substrings up to a depth of two. The string must end with either a 
>>> non-punctation non-whitespace character, a forwards slash, or a 
>>> parenthesis-wrapped substring.
>>
>> but if we create an org-mode document with the following test content:
>> ```
>> http://foo/bar=baz
>
> As you can see from the sentence, PATHPLAIN must be "non-whitespace"
> "non-bracket". "=" is just a messed up verbatim markup marker.
>
> Fixed, on master.
> https://git.sr.ht/~bzg/worg/commit/85ef10ed
>
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>



Documentation: incorrect character name in line breaks

2023-07-21 Thread Tom Alexander
The documentation for line breaks says that PRE is:

> Anything but backspace (`\`).

I suspect this is supposed to say "backslash" (ascii 92) not "backspace" (ascii 
8)

Link to the documentation: https://orgmode.org/worg/org-syntax.html#Line_Breaks



Documentation: Equal sign incorrectly listed as disallowed in plain links

2023-07-21 Thread Tom Alexander
The documentation for plain links states that PATHPLAIN is:

> A string containing non-whitespace non-bracket (()[]<>=) characters, 
> optionally containing parenthesis-wrapped non-whitespace non-bracket 
> substrings up to a depth of two. The string must end with either a 
> non-punctation non-whitespace character, a forwards slash, or a 
> parenthesis-wrapped substring.

but if we create an org-mode document with the following test content:
```
http://foo/bar=baz

http://foo/bar[baz
```

and run a M-x eval-expression (message "%s" (pp-to-string 
(org-element-parse-buffer)))

Then we can see that it allows the equals sign in the PATHPLAIN (output pasted 
below)

Link to the documentation: https://orgmode.org/worg/org-syntax.html#org6934900

Syntax tree:
```
(org-data
(:begin 1 :contents-begin 1 :contents-end 40 :end 40 :robust-begin 3 
:robust-end 38 :post-blank 0 :post-affiliated 1 :path "/tmp/test/test.org" 
:mode org-data :CATEGORY "test" :granularity nil)
(section
  (:begin 1 :end 40 :contents-begin 1 :contents-end 40 :robust-begin 1 
:robust-end 38 :post-blank 0 :post-affiliated 1 :mode first-section 
:granularity nil :parent #0)
  (paragraph
   (:begin 1 :end 21 :contents-begin 1 :contents-end 20 :post-blank 1 
:post-affiliated 1 :mode top-comment :granularity nil :parent #1)
   (link
(:type "http" :path "//foo/bar=baz" :format plain :raw-link 
"http://foo/bar=baz; :application nil :search-option nil :begin 1 :end 19 
:contents-begin nil :contents-end nil :post-blank 0 :parent #2))
   #("\n" 0 1
 (:parent #2)))
  (paragraph
   (:begin 21 :end 40 :contents-begin 21 :contents-end 40 :post-blank 0 
:post-affiliated 21 :mode nil :granularity nil :parent #1)
   (link
(:type "http" :path "//foo/bar" :format plain :raw-link "http://foo/bar; 
:application nil :search-option nil :begin 21 :end 35 :contents-begin nil 
:contents-end nil :post-blank 0 :parent #2))
   #("[baz\n" 0 5
 (:parent #2)
```




Documentation: Reference to BODY field that is not specified in the grammar

2023-07-21 Thread Tom Alexander
The documentation for Inline Babel Calls mentions that opening and closing 
square brackets must be balanced within BODY but the grammar above it only 
defines NAME, ARGUMENTS, HEADER1, and HEADER2.

Link to the documentation: 
https://orgmode.org/worg/org-syntax.html#Inline_Babel_Calls



Documentation: Reference to "KEYCITES" field that is not specified in grammar

2023-07-21 Thread Tom Alexander
The documentation for Citations when specifying which syntactic elements have 
spaces between them references a "KEYCITES" syntactic element that does not 
appear anywhere else on the page.

Link to the documentation issue: 
https://orgmode.org/worg/org-syntax.html#Citations



Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

2023-07-17 Thread Tom Gillespie
> We might probably generalize to
> PRE  = Zs Zl Pc Pd Ps Pi ' "
> POST = Zs Zl Pc Pd Pe Pf . ; : ! ? ' " \ [

If this works I think it is reasonable. We might want to
specify what to do in cases where an org implementation
might not fully support unicode, and might want to do a
review of related issues in syntax with respect to ascii
vs unicode, because iirc there is some ambiguity in
the current syntax doc.

For example, I'm pretty sure that I'm mixing and matching
unicode and ascii whitespace in the tokenizer I have in Racket.

> Though we need to take care excluding zero-width spaces.

Ya, I removed a comment to this effect in the paragraph about
the usual alternate solution.

> Emacs does not support them though (yet?).

Racket has full support for the latest unicode standards iirc,
so I will see if I can leverage that support for testing in laundry.

> At the end, it is the current ASCII limitation plus partially arbitrary
> choice of boundaries that keep some users confused (we are getting bug
> reports about confusing markup from time to time).

Ya, it would be good to try to generalize the affordance if possible since
users of text in non-ascii languages have certain valid expectations. Hopefully,
the unicode consortium has managed to cover the categories we need.



Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)

2023-07-17 Thread Tom Gillespie
Hi Ihor,
   Thank you for looping me in. Best,
Tom

The way I have implemented this is by maintaining an explicit list of
characters that are safe for pre markup and another for post markup.

It is not possible to use unicode punctuation for this because there
are a variety of punctuation marks that cannot appear in that position
and be considered markup, those include @, #, % to name just a few.

Therefore, if we want to do this we commit to extending and then
maintaining the lists of valid pre and post markup delimiters as
special cases.

Note also this could produce changes from current behavior because
things that previously tokenized as a series of words connected by
e.g. underscores could become markup.

The alternative would be (as usual in these cases) for the user to
add a zero width space or something like that between the end of the
markup marker and the symbol they want to follow the markup. This
solution is (trivially) backward compatible, and works for all chars
regardless of whether org-mode has blessed them as sanctioned marks.

My inclination would be not to make this change because there are a
potentially infinite number of future "left right neutral" marks
that we would have to maintain and would occasionally have to field
requests from users to add them, and those solutions would not work
with older versions of org.



[BUG] Contents of the *Warnings* buffer, which has begun popping up at random intervals

2023-05-13 Thread Tom Davey
I was prompted by a buffer named *Warnings* to send an email to
emacs-orgmode@gnu.org<mailto:emacs-orgmode@gnu.org>.

This buffer appears seemingly at random every few days or so (I use
Emacs and Org mode for about eight hours a day). The warnings began
after I upgraded to Org 9.6.3. Find the contents of that buffer below.

Thank you for the work on Org mode, which is magnificent and
without peer.

Tom Davey



■  Warning (org-element-cache): org-element--cache: Org parser error in 
journal.org::18359990. Resetting.
The error was: (error "Invalid search bound (wrong side of point)")

Emacs  : GNU Emacs 29.0.50 (build 1, x86_64-w64-mingw32)
of 2022-09-14
Package: Org mode version 9.6.3 ( @ 
c:/Users/tdavey/Documents/Dropbox/Emacs/Prod/.emacs.d/load-path-subdirs/td-esdf/td-startup-lisp/Elpa/org-9.6.3/)

current state:
==
(setq
org-archive-location "%s_archive::datetree/"
org-link-elisp-confirm-function 'yes-or-no-p
org-directory "~/org-files"
org-yank-adjusted-subtrees t
org-tag-faces '(("PROJ" :background "indianred3" :foreground "cornsilk2" 
:weight bold))
org-agenda-custom-commands '(("c" "Computers and Project Tags"
   ((agenda "") (tags-todo "project") (tags-todo 
"orgmode") (tags-todo "emacs")
(tags-todo "computers"))
   )
  ("e" "Email tag" ((tags-todo "email") (agenda 
"")))
  ("g" "Microscheduling tag" ((tags-todo 
"microscheduling") (tags-todo "today") (agenda "")))
  ("h" "Housing tags"
   ((tags "+Strivers") (tags "+SGCagenda") (tags 
"+SGCbudget") (tags "+SGCcuo")
(tags "+SGCdiscussion") (tags "+SGChouseRules") 
(tags "+SGCminutes") (tags "+SGCstaff")
(tags "+SGCunit") (tags "+SGCvote") (tags 
"+improvement") (tags "+household"))
   ((org-columns-default-format "%CATEGORY %75ITEM 
%TODO %TAGS %25CREATED %25TIMESTAMP")))
  ("n" "Agenda and all TODOs" ((agenda "") (alltodo 
"")))
  ("p" "People tags"
   ((agenda "") (tags-todo "atrajkov") (tags-todo 
"apotter") (tags-todo "dmcewan")
(tags-todo "espector") (tags-todo "jbousley") 
(tags-todo "kmandel") (tags-todo "rosoria")
(tags-todo "twasson") (tags-todo "cfrappier") 
(tags-todo "cmorales") (tags-todo "cwolan")
(tags-todo "dbolen") (tags-todo "jterranella") 
(tags-todo "lortiz") (tags-todo "malzuru")
(tags-todo "dhalsey") (tags-todo "jkendall") 
(tags-todo "kolson"))
   )
  ("b" "Tom's basic block agenda"
   ((agenda "") (tags-todo 
"+homeImprovement-household") (tags-todo "FAredesign")
(tags-todo "rhaass") (tags-todo "kolson") 
(tags-todo "dhalsey") (tags-todo "hr")
(tags-todo "lhammes") (tags-todo 
"StaffMeeting") (tags-todo "cfrappier") (tags-todo "espector")
(tags-todo "ahenry") (tags-todo "atrajkov")
(tags-todo
 
"+work-rhaass-kolson-cfrappier-jmcgrath-tkatavic-egosselin-aqualls-cshehadi-espector-pituk-ahalliday-metchison-nfreiberg-lhammes-emacs-lifelongLearning")
(tags-todo "computers") (tags-todo "desk") 
(tags-todo "errands") (tags-todo "household")
(tags-todo "lifelongLearning") (tags-todo 
"reading") (tags-todo "social"))
   )
  )
org-log-into-drawer t
org-use-fast-tag-selection t
org-agenda-files '("~/org-files/")
org-capture-templates '(("a" "Appointments & Meetings:appt: 
  timestamp" entry
  

Re: bug#59882: Multiple versions of Org in load-path problem

2023-04-18 Thread Tom Gillespie
> The state of our unloading support is not good enough to use it blindly

I have managed to use unload-feature to reload a new version of org
after the builtin version has been previously loaded, but the dance you
have to do to avoid nasty errors (and segfaults if using native comp!)
is about 100 lines of arcane footwork, and if any one step is off you
will encounter one of the many hidden landmines.

unload-feature cannot be used blindly, and there are a very large
number of side effects along the path that are not controlled which
makes any process that tries to use unload-feature brittle and hard
to debug.

https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1010-L1064
https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1073-L1177



Re: Inline markup: How does org identify nested code/verbatim?

2023-01-30 Thread Tom Gillespie
In short, you cannot nest code in verbatim or verbatim in code
because they are both terminal (end of the line for nesting).
In fact you can't nest anything inside them by their very nature.

Anything inside of them cannot have special functionality, even
escape codes don't play well in that part of the grammar.

There is no way around this because you cannot nest inside
things that are by definition terminal. However, from your
examples it seems that you can get the effect you are looking
for using ~is~ =verbatim= ~in code~.


Re: Org mode timestamps on the Moon ;] (was: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda)

2023-01-26 Thread Tom Gillespie
Oh boy. In short, I think we can only hope they come up with
LTC and we already have a syntactic space to say whether
our reference seconds are TAI/UTC/LTC/MTC/JTC etc. And
being the privileged squats that we are if the time system is
left out then it means UTC. The friendly thing to do would be
to always include the time system code in our timestamps,
but I don't think those have standard reference codes yet?

> would gain about 56 microseconds over 24 hours

And here I was naively hoping to avoid having to deal with
relativistic effects. That would seem to be a pretty serious
issue, but my assumption further down that comment is
hopefully what they will do. Specifically:

> let's just assume that clock synchonization will happen between earth and
> mars so that the unix epoch could be synchronized between the frames

The only generalized solution is to record the full location
(see intro to http://naggum.no/lugm-time.html which I'm surprised
hasn't been linked in this thread yet, but was linked on the same
topic back in 2011 it seems) and let people sort out how to translate
that local time from that reference frame in that particular gravity well.

I don't think we want to force users to start embedding their gps
coordinates, their current acceleration, etc. along with their
timestamp. But it turns out that the unix epoch is only standard
at 9.8m/s^2. So ... minutes should be ok right?

Joking aside. Clocks run faster on the moon, slower on the ISS,
so either clocks are synchronized back to earth seconds, or
they use some alternate standard reference time.

My expectation is that there will be some time keeping standard
that will ensure uniformity and that what we will want to capture
is whose seconds they are using to keep track of time, because
unix epoch on the moon has a different integer value than on
earth, and is likely off by multiple seconds.

I guess as long as the moon can participate in earth NTP it will
be ok? Otherwise I suspect that the NTP network standard for
the celestial body will be what we want to go with/record. I think
that will fit in the opening you have created for this in the syntax.

So instead of UTC it would be LTC (lunar coordinated time).
If you're getting logs from servers on the moon knowing that the
system clock was running on LTC and not UTC will be critical.

Some future engineer: "Leaving earth was a mistake."



[BUG] org-clock-in opens buffers for all agenda files

2023-01-21 Thread Tom Gillespie
I just tried to clock in to a task for the first time
in ... a while, and was immediately accosted with
hundreds of buffers being opened.

This seems to be because org-clock-in somehow
calls org-agenda or org-agenda-prepare or something
similar, however I cannot find the call site.

Is there a way to disable this so that clocking does not
destroy my emacs buffer state by starting org-agenda?

Thanks!
Tom



Re: [BUG] org-clock-in opens buffers for all agenda files

2023-01-21 Thread Tom Gillespie
> This is because Org is checking if you have any dangling clocks present.

Seems like a reasonable default.

> See org-clock-auto-clock-resolution

Perfect solution for my case. Thanks!



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-16 Thread Tom Gillespie
> Getting the rules and explanation clear is the issue.  It's a mistake
> that a great many people make with scheduling meetings.  Those two
> behaviors need different encodings because they behave differently.

This is related to why I suggested splitting timezones and offsets into
two separate categories. I think we have to assume that the written
content of timestamps in an org file cannot/will-not be changed
automatically.

Therefore if the timezone is specified then the numbers will never
change, but the actual time might if the timezone spec changes.

If an offset is used then it will not account for changes due
to DST, but it will always remain stable, shuffling a meeting an hour
one way or the other at some points in the year, which is usually
undesirable compared to say, shuffling a meeting 1 hour in one
direction for people who are not in the defining timezone for the
duration of the mismatch between DST changes in different regions.



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-16 Thread Tom Gillespie
> > As for years BC, <-0001-...> will be a breaking change. But I do not
> > think that we need to really worry about this. Not unless we actually
> > get feature request. What is the practical application?

Using org as a format for writing about history and being able to
reference dates in the past accurately and have the dates be
first class entities that can be parsed and checked etc.

The example in my head is a history professor who wants to
write about e.g. the collapse of the Roman republic and not
have to come up with their own time keeping system or force
any one who wants to work with referenced dates to do a bunch
of math to translate from a roman time system to a modern one.

> Given that the stated approach is to leverage off OS facilities in this
> area, it probably should also be noted that some OSs don't handle
> historical dates, especially BC ones, at all well. For example, some OS
> use a 32 bit number to represent the date+time and can really only
> handle dates between approx 1900 and 2038 (or around there - cannot
> remember specific range). So with respect to timestamps and time related
> calculations, we are limited by the capabilities of the least capable
> supported OS.

I'm mostly concerned about the syntactic features where org already
supports dates well outside the facilities of various operating systems.
I don't think that it is wise or practical for org-mode code to use
anything other than the os provided time keeping facilities right now,
but it is important to enable people who might want to do so. Org as
a format for documents has a wider range of use cases for dates and
times than Org as a life organizer and planner. At the same time those
wider use cases don't always need as much precision or ux considerations
because I don't think anyone using org right now is going to be early or
late to their meeting at [3023-01-16 Thu 12:00] (regardless of the timezone).
But org does tell me that it will be a Thursday! Best,
Tom



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-16 Thread Tom Gillespie
> I strongly disagree. I'd prefer to allow only internationally recognized
> time zone format. Let's not make life harder for Org file parsers.

So offsets and tz database names but no time zone abbreviations?

That seems reasonable since there isn't a sane way to handle the
timezone with dst vs abbreviation for an offset, so better to force
only US/Central aka America/Chicago and then -06:00 and -05:00
if users want CST/CDT to avoid any ambiguity?



[PATCH] fix compat with emacs 25 due using temporary-file-directory

2023-01-15 Thread Tom Gillespie
Small bugfix for emacs 25 compat. Best!
Tom
From f04a44e5cc8143fc6cdbbc92b9d2afb76de4a459 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Mon, 16 Jan 2023 00:25:47 -0500
Subject: [PATCH] fix compat with emacs 25 due using temporary-file-directory
 function

* lisp/ob-core.el: org-babel-temporary-stable-directory use the symbol
version of temporary-file-directory since the function form is not
available on emacs 25.
---
 lisp/ob-core.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 5d5edadd2..255a767bb 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -3276,7 +3276,7 @@ Emacs shutdown.")
   (while (or (not dir) (file-exists-p dir))
 (setq dir (expand-file-name
(format "babel-stable-%d" (random 1000))
-   (temporary-file-directory
+   temporary-file-directory)))
   (make-directory dir)
   dir))
   "Directory to hold temporary files created to execute code blocks.
-- 
2.38.2



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-15 Thread Tom Gillespie
> So I guess the timestamp format and the code which manages them will
> need the ability to use the full TZ name and not just the abbreviated
> form (and I guess an option to allow the user to select). In fact, we
> probably need a way to select between abbreviated/full dynamically as
> well as you might use the different TZ types as a way to flag which
> timestamps you want to adjust due to TZ changes (either in TZ db or in
> user location) and those you don't want changed.

I think the only sane way to do this is to require timezone abbreviations
/expansions to be defined in the file itself and never in an init.el there is
simply too much ambiguity and if the information is lost we are out of luck.

Of course then the question comes whether we can use something like
#+tzinfo: US/Central CT
#+tzoff: CST -06:00
#+tzoff: CDT -05:00
I don't think it is wise to allow users to map #+tzinfo: US/Central CST
because that will produce surprising and results when US/Central
changes to DST. Ugh, what a mess.



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-15 Thread Tom Gillespie
> In anticipation to add time zones in future, I have added the following
> to the Org timestamp spec (see
> https://orgmode.org/worg/org-syntax.html#Timestamps):
>
> DATE TIME REPEATER-OR-DELAY
>
> TIME (optional)
> An instance of the pattern H:MMREST where H represents a one to two digit 
> number (and can start with 0), and M represents a single digit. REST can 
> contain anything but \n or closing bracket.
>
> Note that REST imply that almost arbitrary suffix can be in TIME without
> braking the existing Org timestamp parsing code.

I'm not sure how I feel about the REST in the grammar, I think it is a
reasonable approach but need to double check. I'm worried that there
can be some nasty interactions with REPEATER-OR-DELAY syntax, but that
may not actually be an issue.

I will note that this doesn't address the issue of syntax for
historical and future dates. For historical dates those almost always
require significant additional metadata to compensate for things like
the julian/gregorian calendar switchover etc. for future dates we may
want to go ahead and specify something beyond -.

I'm less concerned about the rest of the issues beyond the fact that
adding syntactic support for timezones seems to have opened up
countless usability and bad assumption issues that have been mentioned
elsewhere in the thread.

So, I think the syntax may be ok, but as written I think there can be
quite nasty interactions with REPEATER-OR-DELAY (not to mention what
happens if a delay tries to track across a change in timezones).

Best,
Tom



Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

2023-01-13 Thread Tom Gillespie
Without wading too far into this at the moment,
timezones are an extremely tricky problem with
a whole bunch of design considerations. I am
reproducing the heading comment from laundry's
timestamp.rkt in its entirety here. Best!
Tom

https://github.com/tgbugs/laundry/blob/master/laundry/grammar/timestamp.rkt

; the org spec is currently silent on how org will handle dates beyond
;  and , the relevant standards for expanded representations
; are ISO 8601-1:2019,5.2.2.3 and ISO 8601-2:2019,8.4.3. The problem
; with expanded representations is that they require all parties to
; agree ahead of time how wide the year is and prefix with + and
; leading zeros, I am not sure that this is as robust as we would like
; -01-01 is january first of the year zero, -0001-01-01 is jan
; first of the year 1BCE, and +1-01-01 for jan 1 of 1CE 5
; digits must be prefixed by the plus sign, the standard is not
; entirely clear, but I think that we could do PLUS digits HYPHEN
; or DIGIT-4 HYPHEN and everything would work out

; on a separate but related note, org timestamps are always in local
; time and do not currently support timezones, which is a problem

; further absurdities that are related to the timezone issue: the
; date/time implementation as a text format completely fails if
; authoring on another planet

; given my objective to ensure that org documents can be interpreted
; without having to stick stupid things like #+planet: mars in the
; header or risk your earthling readers getting incorrect dates --- I
; suggest that org switch to storing all dates and times in earth zulu
; time or with an offset as it stands, the timezone setting of the
; computer drafting the document must be known to determine what day
; it was etc. users could still draft using <2020-12-16> but C-c C-c
; would add the correct offset? actually this is tricky, because there
; are two cases, one where the location is clear, "napoleon on [1812-01-01]"
; and the other where it is not, the issue is that a single date refers
; to as many different time intervals as there are timezones and without
; knowing the timezone you don't know the date, similar issue if we were
; to try to convert to sols
; https://www.eecis.udel.edu/~mills/missions.html
; https://www.eecis.udel.edu/~mills/ipin.html
; https://en.wikipedia.org/wiki/Timekeeping_on_Mars
; HOWEVER having applied a bit more brain power to the problem, let's
; just assume that clock synchonization will happen between earth and
; mars so that the unix epoch could be synchronized between the frames
; probably with multiple reference stations in free fall and/or in deep
; space to account for gravitaitonal differences etc. so actually the
; only issue would be a practical notational one, or a UI one, since we
; can use timestamp with timezone or just straight epoc for this to work
; earth vs martian year/day is a much harder issue
; the org spec is currently silent on how org will handle dates beyond



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-31 Thread Tom Gillespie
> P.S. Considering intense discussion around the topic, what about
> reverting my commit from the release? We can then re-consider the whole
> design and apply something more elaborate later.

I was actually going to mention that in my previous
message but forgot. I think that given the potential
for disruption and our desire to do a good job on the
design of the new system I think it would make sense
to revert the commit from the release so that we don't
rush in trying to fix something under pressure without
taking the time we need to consider all the angles.

Best!
Tom



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-30 Thread Tom Gillespie
Hi Bastien,
   In short, we need two variables due to the case where
a user wants to set nil for all vars and a function for code.
More replies in line. Best!
Tom

> I see nothing in etc/ORG-NEWS that describes this change: am I missing
> something?

Looks like any mention of the change is missing to me as well.

> >> Whether it changed or not, what is the problem in 9.6?
> >
> > The problem is that Org now displays more queries.
> >
> >> How does the patch solves this problem?
> >
> > It allows disabling these new queries about lisp evaluation outside
> > code blocks without disabling code block eval confirmation completely.

> Is there a real use-case for this?

Yes. To give one example. I have many files that need to run hundreds
of cells at tangling/export time. I have reviewed all the code in the cells
and know patterns that are safe and wont' burn me. I do not want to
allow execution of code blocks blindly during tangling and confirm eval
is a secondary line of defense against running those blocks during
export. I want to allow the cells to run uninhibited as they did before
this change, so that I don't have to fight with org during the tangling
process.

As it is right now the change has completely broken various CI
pipelines for me, and if these changes do not get in to emacs 29
in time I will be unable to use emacs 29 for CI until they are in the
base image without having to resort to insane hacks and massive
additional configuration changes to work around the issues discussed
in the loading multiple org versions thread.

> It looks like the patch fixes the "too many queries" problem by
> providing a setup that is similar to what was the previous default
> (i.e. only asking about code block evaluation when
> org-confirm-babel-evaluate-cell is set to nil.)

This interpretation is not quite right. org-confirm-babel-evaluate-cell has
no interaction with evaluating code blocks, only with evaluating cells.
When org-confirm-babel-evaluate-cell is nil or evaluates to nil then the
org-babel-confirm-evaluate code runs on a fake block that is created
out of the cell. (This is where there is still a bug that I mentioned up
thread.)

> But are we sure that users need to confirm header args evaluation
> separately from code blocks evaluation?

I am sure that in my workflow I want them split, especially for
global nil/t behavior. I need to think a bit more about your
suggestion to add more values to org-confirm-babel-evaluate
to see whether it might work and what the relative complexity
would be.

It seems to me that duplicating the variable is certainly easier
to maintain, if not to explain to those who do not use babel
regularly. I would be interested to hear from other babel users
their thoughts on the design, because it seems obvious to me,
but again I wrote the code so am the last person whose view
on the clarity of the should be considered. It seems that it
was not clear to Max initially.

> IOW I have difficulties understanding when these would be relevant:

> (setq org-confirm-babel-evaluate-cell nil)

Anywhere that a parenthesized elisp expression occurs
that is not in #+begin_src block.

> (setq org-confirm-babel-evaluate t)

Only inside #+begin_src blocks.

> I think that's the best route: providing, optionnally, more, while not
> annoying users who don't want more.

I think this is probably the right default as well.

> > Yes. Ideally, we need to improve the code evaluation query. It should
> > allow confirming evaluation in bulk and add some code blocks/files to
> > whitelist. Similar to `org--confirm-resource-safe'.
>
> I see, indeed.

Improving general code auditing prior to evaluation of blocks would
be greatly appreciated. The fighting between the minibuffer and
the primary buffer for priority and the fact that you often cannot
see that code that will be run is a significant issue. However, this is
orthogonal to the issue at hand.

> > A concern have been expressed that more queries may annoy users and
> > drive them towards setting `org-confirm-babel-evaluate' to nil globally.
> > Upon doing this, the future more flexible security queries may be not
> > used by such users.
>
> The future more flexible security queries will be made visibile enough
> so that users currently setting `org-confirm-babel-evaluate' to nil
> will *want* to have a look at it.

Yep. We need to get the fundamentals in place to enable those
workflows. I'm too paranoid to simply set that variable to nil, even
when I'm only running code that I wrote, but I suppose that is not
the case for everyone.

> I find "cell" confusing here (as Max said earlier in the thread).  It
> either refers to a table cell or, in Elisp jargon, a "cons cell" (or a
> "function cell").

I also dislike the name cell, but i'm not sure what to cal

[PATCH] Fix missing null check for tangle links during export

2022-12-19 Thread Tom Gillespie
Hi,
   Here is a patch that fixes broken tangling during export. I don't have a
simple reproduction, but was able to run git bisect to find the originating
commit. The commit text explains the issue in detail. Best,
Tom
From 0b4e35453874db0acf046e46c571cdef5f39f59a Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Mon, 19 Dec 2022 21:25:02 -0500
Subject: [PATCH] Fix missing null check for tangle links during export

* lisp/ob-tangle.el (org-babel-tangle--unbracketed-link): Add the let
bound variable l to the and statement to prevent l from being passed
to `string-match' in the event that it is nil.

When tangling nested noweb blocks during export it is possible for the
results of `org-store-link' to return nil.  This commit ensures that
the value returned for l is only passed to `string-match' when it is
non-nil, avoiding a `wrong-type-argument' (stringp nil) error.

Handling of comments and nesting of babel blocks is known to have
issues.  The bug is from 8a781d35dc68f20fa2a5546c98ba3d9b77ee3cda
where new code was introduced to obtain the value for bare which was
not present in early code.  I'm guessing that the bug appears now
because `string-match' is called on l (aka link) at a point in time
when it was never previously called and it was thus masked because in
the old version it was impossible to call `string-match' when `link'
was nil because another variable was always nil, masking the issue.
---
 lisp/ob-tangle.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index bd17bda32..fd6b6f3b9 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -500,7 +500,8 @@ The PARAMS are the 3rd element of the info for the same src block."
  (cl-letf (((symbol-function 'org-store-link-functions)
 (lambda () nil)))
(org-store-link nil
- (bare (and (string-match org-link-bracket-re l)
+ (bare (and l
+(string-match org-link-bracket-re l)
 (match-string 1 l
 (when bare
   (if (and org-babel-tangle-use-relative-file-links
-- 
2.38.2



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-19 Thread Tom Gillespie
One more correction. The source of the issue is that the two values in the
list need to be different, one for the message and one for the actual test.
Best,
Tom

(list "emacs-lisp" cell
  '((:eval . yes)) nil (format "%S" cell)
  nil nil)


Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-19 Thread Tom Gillespie
By the way, while we're on bugfixes. I just noticed that (format "%S" cell)
is being used to create the fake body for info. This is incorrect because
cell is already a string at this point and this makes the behavior
inconsistent with the rest of org babel as a result. Fix below. Best,
Tom


The fix is to pass

(list "emacs-lisp" cell
  '((:eval . yes)) nil cell
  nil nil)


Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)

2022-12-16 Thread Tom Gillespie
Hi Ihor,
   Chiming in here with a slight variant on what
others have said. Best!
Tom

I don't think this should be handled at the syntactic
layer at all. The empty string block language should
be syntactically valid with any special behavior
needed being handled later.

Linters could treat it as a warning/error though,
but the parsing is made significantly easier if
the empty string is present allowing the grammar
to be fully closed and regular.

Thus, I don't think we need to make this a syntactic
error or pun a src block without a lang to another type.

I think we can add an implementation for when the
block language is the empty string. This keeps
the grammar regular by removing a special case.

I assume that internally the empty string block lang
would mostly call the example block codepaths,
except that it should probably issue a warning or fail
if someone tries to org-src-edit the block so that we
can alert them that they are missing the lang.

Treating src blocks missing a lang as paragraphs is
incorrect because according to the syntax spec they
are syntactically still blocks (greater or lesser depending
on your inclinations).

I think the general principle we want to follow here is
that a block (or any entity in general) should not lose
its type because some part of its syntax is malformed
(I have made similar arguments about property drawers).

That is, if something starts with #+begin_NAME stuff
and there is a corresponding #+end_NAME, then it
is a block.

The choice of how a src block without a lang should
behave is a bit more complex as there are multiple
consumers of src blocks that make different assumptions.

As mentioned above. I think that if a block is missing the lang
we could think of it instead as the null language. If we have the
:var language because someone has other contents on the line
they have a well formed src block, but will get a different error
because there is currently no known language ":var".


Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-15 Thread Tom Gillespie
> It's purpose is also different.
> I am generally skeptical about our ability to provide universal way to
> judge if a given sexp is safe or not. It should be left to the user.

I am in complete agreement.



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-14 Thread Tom Gillespie
Interestingly, as I was looking through my notes in
orgstrap, I see my past self had found a macro
org-babel-one-header-arg-safe-p which pointed to
defconst org-babel-safe-header-args, but that is
a const and not really user configurable. Of course
the user could cl-setf on it, but it also only checks
strings and has no ability to e.g. see if the value of
(symbol-function #'identity) has changed since the
check function was defined. Two examples.

(let ((old-identity (symbol-function #'identity)))
  (cl-letf* (((symbol-function #'identity)
  (lambda (x) (message "there") x)))
(identity 'hello)
(equal (symbol-function #'identity) old-identity)))

(let ((old-and (symbol-function #'and)))
  (cl-letf* (((symbol-function #'old-andf) old-and)
 ((symbol-function #'and)
  (lambda ( args)
(message "oops %s" args)
(old-andf args
(list
 (and)
 (and 1 2 3)
 (equal (symbol-function #'and) old-and

> Tom, does not the following allow to achieve the same without your patch?

It works if I have a closed set of things I want to allow but not if I
want to set it to nil to e.g. restore the old behavior (worse for
security but not as bad as setting ocbe to nil), e.g. if I'm
under duress and need to get something that used to work to
work again without the risk of automatically running dangerous
code, (e.g. blocks that might rm something).

> I know, it does not work, but I think it is due to (format "%S" cell)
> instead of passing cell directly in
>
> -'((:eval . yes)) nil (format "%S" cell)
>
> My point is that if some expression is safe for a variable value then it
> is safe for the source block body.

There is another use case here, which I alluded to in the previous
comment, which is that sometimes ocbe is the last line of defense
against running dangerous code. Ideally users would have set
:eval never on blocks like that to be sure, but if they don't you
don't want someone already trying to get something to work
to get too much to work.

Again, this is focused on the ocbec -> nil case.

> Have you ever seen the prompt for a table?

Err ... maybe? So the answer is probably no.

> I suppose, tables are the most prominent security issue related to
> unsolicited code execution:

For me it would be arbitrary expressions in the headers
of source blocks. Hard to know which one is more prevalent
across the population of org users.

> Max Nikulin to emacs-orgmode. Re: [BUG][Security] begin_src :var
> evaluated before the prompt to confirm execution. Fri, 28 Oct 2022
> 11:11:18 +0700. https://list.orgmode.org/tjfkp7$ggm$1...@ciao.gmane.io
>
> I am still in doubts if
>
> 10e857d42 2022-10-28 11:09:50 +0800 Ihor Radchenko: org-babel-read: Obey
> `org-confirm-babel-evaluate'
>
> was an unambiguous improvement. Perhaps it just forces more users to set
> `org-confirm-babel-evaluate' to nil compromising their security to more
> severe degree.

Heh. It is always a hard balance to strike. In the context of that
thread having a variable that would find-file-literally for untrusted
org files by default might be useful.

Again, it is a pain. I can tell you from experience having written
the system that does something similar for orgstrap. There is
no safe way other than a user-maintained whitelist based on
file hashes, everything else can be spoofed in one way or another.

I suspect that once we have the machinery in this patch in
place we can look for some sane defaults. Note that the example
function we keep passing around isn't quite good enough because
someone could probably figure out how to rewrite the identity
function so we would need to make sure that it had not changed
since emacs was loaded (unlikely, but if I can image it someone
could surely do it).


Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-13 Thread Tom Gillespie
> Will it be clear to users what "cell" means in this context?

I assume the language was originally chosen
with tables in mind, but I think it is clear? The
one issue is that using org-babel-confirm-evaluate
doesn't use the word "cell" in the yes-or-no-p prompt.

> Am I wrong that, roughly speaking, the effective value is (and
> org-confirm-babel-evaluate-cell org-confirm-babel-evaluate)?

You are correct. Only in the case that
org-confirm-babel-evaluate-cell is or evaluates to t will
the yes-or-no-p be displayed _IF_ org-confirm-babel-evaluate
is or evaluates to t. org-babel-confirm-evaluate (NOT
org-confirm-babel-evaluate) is the function that issues
the yes-or-no-p.

> Should it be described in the docstring?

Maybe? In the current implementation the value of
org-confirm-babel-evaluate takes precedence, which
I think is the correct behavior. If someone has told
us that they blindly want to execute all code, but
not cells, we should not change the default behavior,
and thus we should execute all cells and blocks

> It seems e.g. :eval query affects the latter, but not the former.

:eval has no impact on cells. In fact in some files I use
:eval (and this-file-done-loading "never") on some blocks that
(setq this-file-done-loading) at the end to achieve only-once
behavior.

> If it is so then it is too complicated for me to feel firm ground
> while reasoning on security.

:eval has never, and cannot impact evaluation of cells, because
it is specifically used to control evaluation of blocks. Similarly
:eval has no interaction with any of the confirm-babel-evaluate
machinery.

I see one possible point of confusion which is that he :eval that
you see in the code of org-babel-read is a hack to create fake
info for a cell.

Are you maybe looking for a file level :eval-cell option/property
that would allow you to declare that you should NEVER run
cells at all and also never prompt? Basically this is an option
that says "treat all cells as if they and their header property
are the default value".

I'm fairly certain that implementing such a thing is a bad
idea, for example if someone sets :dir using a cell and
it is overwritten they could execute code in the wrong
environment, which can lead to nasty security issues.
Thus the only global option for cells that we could safely
implement would have the semantics of "treat all cells
as if they are (error)".

> I have not decided if it is feasible to add some property to the INFO
> argument passed to `org-babel-confirm-evaluate' and to put there
> condition which variable should determine result.

Having done so before, I would rather not touch
org-babel-confirm-evaluate if at all possible. The only
disadvantage is that for now when users are prompted
they will not be informed that it is a cell, however from
the short nature of cells the will hopefully be able to
figure it out.

The fact that org already jumps to the location of
the cell in the file helps a lot with this. Obviously
this doesn't help in a batch context, but if a user
is hitting this in batch mode they are in trouble
anyway.



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-12 Thread Tom Gillespie
Hi Ihor,
   Here's the updated patch using :safe, and an additional
patch for the news entry to make it easier to apply the
core change to bugfix if needed. Best!
Tom

> I am also wondering if we should include this into bugfix.

I can vouch for the fact that trying to work around this in
any other way is going to be a massive pain. For example,
it will be hard to use vanilla 29 for various CI types of things
where someone might need to execute a cell but not
want to allow arbitrary codeblocks.
From 4a78e1b5ea98dee569ff690037c661ab5c300194 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sat, 10 Dec 2022 12:11:17 -0800
Subject: [PATCH 1/2] ob-core: add org-confirm-babel-evaluate-cell custom
 variable

* lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control
execution of cells separate from execution of src blocks, it works in
exactly the same way as `org-confirm-babel-evaluate'.
* lisp/ob-core.el (org-babel-read): `org-confirm-babel-evaluate-cell'
is used to check cells independent of `org-confirm-babel-evaluate'.

Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it
became extremely annoying to tangle files that make extensive use of
elisp expression in src block #+header: statements.

This commit resolves the issue by making it possible to ignore checks
on cells (the old behavior) without compromising general security for
running src blocks.

This is necessary because there is no easy way to hop swap
`org-confirm-babel-evaluate' between `org-get-src-block-info' where
`org-babel-read' is called and the execution of that src block.  It
could probably be done using advice around `org-babel-read', but that
is a level of hackery that should be avoided.
---
 lisp/ob-core.el | 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 2fa9d8978..d56e47de5 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -128,6 +128,14 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding."
 ;; don't allow this variable to be changed through file settings
 (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t)))
 
+(defcustom org-confirm-babel-evaluate-cell t
+  "Confirm before evaluating a cell.
+This follows the same conventions as `org-confirm-babel-evaluate'."
+  :group 'org-babel
+  :package-version '(Org . "9.6")
+  :type '(choice boolean function)
+  :safe (lambda (x) (eq x t)))
+
 (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil
   "\\\
 Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding."
@@ -3180,11 +3188,14 @@ situations in which is it not appropriate."
 		  (string= cell "*this*")))
  ;; Prevent arbitrary function calls.
  (if (and (memq (string-to-char cell) '(?\( ?`))
+  (if (functionp org-confirm-babel-evaluate-cell)
+  (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell)
+org-confirm-babel-evaluate-cell)
   (not (org-babel-confirm-evaluate
-  ;; See `org-babel-get-src-block-info'.
-  (list "emacs-lisp" (format "%S" cell)
-'((:eval . yes)) nil (format "%S" cell)
-nil nil
+;; See `org-babel-get-src-block-info'.
+(list "emacs-lisp" (format "%S" cell)
+  '((:eval . yes)) nil (format "%S" cell)
+  nil nil
  ;; Not allowed.
  (user-error "Evaluation of elisp code %S aborted." cell)
 	   (eval (read cell) t)))
-- 
2.37.4

From 03aad0a73acfca05245a01e83bae3609e6d3ed04 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Mon, 12 Dec 2022 17:45:14 -0800
Subject: [PATCH 2/2] * etc/ORG-NEWS: Add entry for
 `org-confirm-babel-evaluate-cell'.

---
 etc/ORG-NEWS | 36 
 1 file changed, 36 insertions(+)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 5d5e726e0..16ff5ba67 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -12,6 +12,42 @@ See the end of the file for license conditions.
 Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
+
+** New options
+*** A new custom setting ~org-confirm-babel-evaluate-cell~ to control confirming execution of cells
+
+Following recent changes to ~org-babel-read~ it became annoying to
+tangle files that make extensive use of elisp expression in src
+block #+header: statements.
+
+~org-confirm-babel-evaluate-cell~ resolves the issue by making it
+possible to ignore checks on cells (the old behavior) without
+compromising general security for running src blocks.
+
+It works in the same way as ~org-confirm-babel-evaluate~, accepting a
+boolean or a function of two arguments (lang body).
+
+Here is a

Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats

2022-12-12 Thread Tom Gillespie
Here's the rebased patch. It should apply without issue. Best!
Tom
From 5247d2459800f82434f1bc3aeea136c18af7923c Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Dec 2022 01:02:35 -0800
Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
 refactor

* lisp/org-expiry.el (org-expiry-insert-created)
(org-expiry-insert-expiry): timestamp formats dropped delimiters so a
slight modification is required following org commit
e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward
compatible by removing < and > from the old timestamp format
---
 lisp/org-expiry.el | 32 +---
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el
index b359610..c3dad28 100644
--- a/lisp/org-expiry.el
+++ b/lisp/org-expiry.el
@@ -301,12 +301,17 @@ update the date."
 		 (current-time)))
   (setq d-hour (format-time-string "%H:%M" d-time))
   (setq timestr
-(org-expiry-format-timestamp
- ;; two C-u prefixes will call org-read-date
-	 (if (equal arg '(16))
-		 (org-read-date nil nil nil nil d-time d-hour)
-	   (format-time-string (cdr org-time-stamp-formats)))
- org-expiry-inactive-timestamps))
+	;; two C-u prefixes will call org-read-date
+(concat "<"
+(if (equal arg '(16))
+(org-read-date nil nil nil nil d-time d-hour)
+  (format-time-string
+   (replace-regexp-in-string "\\(^<\\|>$\\)" ""
+   (cdr org-time-stamp-formats
+">"))
+  ;; maybe transform to inactive timestamp
+  (if org-expiry-inactive-timestamps
+	  (setq timestr (concat "[" (substring timestr 1 -1) "]")))
   (save-excursion
 	(org-entry-put
 	 (point) org-expiry-created-property-name timestr)
@@ -321,11 +326,16 @@ and insert today's date."
 (setq d-time (if d (org-time-string-to-time d)
 		   (current-time)))
 (setq d-hour (format-time-string "%H:%M" d-time))
-(setq timestr (org-expiry-format-timestamp
-   (if today
-		   (format-time-string (cdr org-time-stamp-formats))
-		 (org-read-date nil nil nil nil d-time d-hour))
-   org-expiry-inactive-timestamps))
+(setq timestr (concat "<"
+  (if today
+  (format-time-string
+   (replace-regexp-in-string "\\(^<\\|>$\\)" ""
+   (cdr org-time-stamp-formats)))
+(org-read-date nil nil nil nil d-time d-hour))
+  ">"))
+;; maybe transform to inactive timestamp
+(if org-expiry-inactive-timestamps
+	(setq timestr (concat "[" (substring timestr 1 -1) "]")))
 
 (save-excursion
   (org-entry-put
-- 
2.37.4



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-11 Thread Tom Gillespie
> Please use :package-version and let the mapping be handled by
> customize-package-emacs-version-alist.

Got it. Here's the updated patch (I think I did it correctly?).
From 47a47aa9453a54a4f5f2e9188e2ad072a77c50cb Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sat, 10 Dec 2022 12:11:17 -0800
Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

* lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control
execution of cells separate from execution of src blocks, it works in
exactly the same way as org-confirm-babel-evaluate.
* lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is
now used to check cells independent of org-confirm-babel-evaluate.

Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it
became extremely annoying to tangle files that make extensive use of
elisp expression in src block #+header: statements.

This commit resolves the issue by making it possible to ignore checks
on cells (the old behavior) without compromising general security for
running src blocks.

This is necessary because there is no easy way to hop swap
org-confirm-babel-evaluate between org-get-src-block-info where
org-babel-read is called and the execution of that src block. It could
probably be done using advice around org-babel-read, but that is a
level of hackery that should be avoided.
---
 lisp/ob-core.el | 20 
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 62b0d3612..aa73a9cbc 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -128,6 +128,15 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding."
 ;; don't allow this variable to be changed through file settings
 (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t)))
 
+(defcustom org-confirm-babel-evaluate-cell t
+  "Confirm before evaluating a cell.
+This follows the same conventions as `org-confirm-babel-evaluate'."
+  :group 'org-babel
+  :package-version '(Org . "9.6")
+  :type '(choice boolean function))
+;; don't allow this variable to be changed through file settings
+(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t)))
+
 (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil
   "\\\
 Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding."
@@ -3180,11 +3189,14 @@ situations in which is it not appropriate."
 		  (string= cell "*this*")))
  ;; Prevent arbitrary function calls.
  (if (and (memq (string-to-char cell) '(?\( ?`))
+  (if (functionp org-confirm-babel-evaluate-cell)
+  (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell)
+org-confirm-babel-evaluate-cell)
   (not (org-babel-confirm-evaluate
-  ;; See `org-babel-get-src-block-info'.
-  (list "emacs-lisp" (format "%S" cell)
-'((:eval . yes)) nil (format "%S" cell)
-nil nil
+;; See `org-babel-get-src-block-info'.
+(list "emacs-lisp" (format "%S" cell)
+  '((:eval . yes)) nil (format "%S" cell)
+  nil nil
  ;; Not allowed.
  (user-error "Evaluation of elisp code %S aborted." cell)
 	   (eval (read cell) t)))
-- 
2.37.4



Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats

2022-12-11 Thread Tom Gillespie
Looks like string-replace doesn't support that notation,
so both too aggressive and without sufficient control.
I've switched to replace-regexp-in-string which does
what we want.

Here's the updated patch.
From 12ca29965e867acd64fecaecd14f2f74e90d7e99 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Dec 2022 01:02:35 -0800
Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
 refactor

* lisp/org-expiry.el (org-expiry-insert-created)
(org-expiry-insert-expiry): timestamp formats dropped delimiters so a
slight modification is required following org commit
e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward
compatible by removing < and > from the old timestamp format
---
 lisp/org-expiry.el | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el
index 98ad58a..0462735 100644
--- a/lisp/org-expiry.el
+++ b/lisp/org-expiry.el
@@ -299,10 +299,13 @@ update the date."
   (setq d-hour (format-time-string "%H:%M" d-time))
   (setq timestr
 	;; two C-u prefixes will call org-read-date
-	(if (equal arg '(16))
-		(concat "<" (org-read-date
-			 nil nil nil nil d-time d-hour) ">")
-	  (format-time-string (cdr org-time-stamp-formats
+(concat "<"
+(if (equal arg '(16))
+(org-read-date nil nil nil nil d-time d-hour)
+  (format-time-string
+   (replace-regexp-in-string "\\(^<\\|>$\\)" ""
+   (cdr org-time-stamp-formats
+">"))
   ;; maybe transform to inactive timestamp
   (if org-expiry-inactive-timestamps
 	  (setq timestr (concat "[" (substring timestr 1 -1) "]")))
@@ -320,10 +323,13 @@ and insert today's date."
 (setq d-time (if d (org-time-string-to-time d)
 		   (current-time)))
 (setq d-hour (format-time-string "%H:%M" d-time))
-(setq timestr (if today
-		  (format-time-string (cdr org-time-stamp-formats))
-		(concat "<" (org-read-date
- nil nil nil nil d-time d-hour) ">")))
+(setq timestr (concat "<"
+  (if today
+  (format-time-string
+   (replace-regexp-in-string "\\(^<\\|>$\\)" ""
+   (cdr org-time-stamp-formats)))
+(org-read-date nil nil nil nil d-time d-hour))
+  ">"))
 ;; maybe transform to inactive timestamp
 (if org-expiry-inactive-timestamps
 	(setq timestr (concat "[" (substring timestr 1 -1) "]")))
-- 
2.37.4



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-11 Thread Tom Gillespie
Here is an updated version of the patch with the convention
corrected and the docstring updated for clarity. It would be
great to try to get this into any 9.6 patches before the 29
release, but I'm not sure if that is possible. If it is missing
we are likely to get a lot of messages from unhappy users.
Best!
Tom
From 54e468b60f17b54d81edafd8ee9e22311e519793 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sat, 10 Dec 2022 12:11:17 -0800
Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

* lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control
execution of cells separate from execution of src blocks, it works in
exactly the same way as org-confirm-babel-evaluate.
* lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is
now used to check cells independent of org-confirm-babel-evaluate.

Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it
became extremely annoying to tangle files that make extensive use of
elisp expression in src block #+header: statements.

This commit resolves the issue by making it possible to ignore checks
on cells (the old behavior) without compromising general security for
running src blocks.

This is necessary because there is no easy way to hop swap
org-confirm-babel-evaluate between org-get-src-block-info where
org-babel-read is called and the execution of that src block. It could
probably be done using advice around org-babel-read, but that is a
level of hackery that should be avoided.
---
 lisp/ob-core.el | 20 
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 62b0d3612..60dabab0a 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -128,6 +128,15 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding."
 ;; don't allow this variable to be changed through file settings
 (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t)))
 
+(defcustom org-confirm-babel-evaluate-cell t
+  "Confirm before evaluating a cell.
+This follows the same conventions as `org-confirm-babel-evaluate'."
+  :group 'org-babel
+  :version "29.1"
+  :type '(choice boolean function))
+;; don't allow this variable to be changed through file settings
+(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t)))
+
 (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil
   "\\\
 Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding."
@@ -3180,11 +3189,14 @@ situations in which is it not appropriate."
 		  (string= cell "*this*")))
  ;; Prevent arbitrary function calls.
  (if (and (memq (string-to-char cell) '(?\( ?`))
+  (if (functionp org-confirm-babel-evaluate-cell)
+  (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell)
+org-confirm-babel-evaluate-cell)
   (not (org-babel-confirm-evaluate
-  ;; See `org-babel-get-src-block-info'.
-  (list "emacs-lisp" (format "%S" cell)
-'((:eval . yes)) nil (format "%S" cell)
-nil nil
+;; See `org-babel-get-src-block-info'.
+(list "emacs-lisp" (format "%S" cell)
+  '((:eval . yes)) nil (format "%S" cell)
+  nil nil
  ;; Not allowed.
  (user-error "Evaluation of elisp code %S aborted." cell)
 	   (eval (read cell) t)))
-- 
2.37.4



Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-11 Thread Tom Gillespie
Hi Max,
   Thank you for the feedback. More replies in lines. Best!
Tom

> I am not sure concerning "exactly".
>
> lisp/ob-core.el:248
> `org-confirm-babel-evaluate' is called with 2 arguments. In your patch
> `org-confirm-babel-evaluate-cell' has a single argument.

You're right, and in point of fact I should have retained the structure
exactly because in other contexts I have thought about ways to use
other languages in contexts like that. At the moment everything is
elisp so I dropped the argument, but that is clearly a mistake.

> It seems, you do not change defaults. Could you, please, provide an
> example of configuration that is less annoying, but still safe?

#+begin_src elisp :results none
(setq-local
 org-confirm-babel-evaluate-cell
 (lambda (lang body)
   (ignore lang)
   (let ((rb (read body)))
 (not ; aka (unless condition t)
  (or
   (member rb
   '((or)
 (and)
 ;; add more forms that are known safe here
 ))
   (and
(eq (car rb) 'identity)
(let ((v (cadr rb)))
  (or
   (symbolp v)
   (stringp v)
   (numberp v)
   
#+end_src

#+header: :var v1=(or) v2=(and) v3=(identity nil)
#+header: :var v4=(identity default-directory) v5=(identity #o0755)
#+header: :var v6=(identity "not sure why you would want to do this")
#+header: :var v7=(identity (concat "this" "will" "fail"))
#+header: :var v8="reminder that strings are ok"
#+begin_src elisp
(mapcar
 #'list
 (list v1 v2 v3 v4 v5 v6 v7 v8))
#+end_src

> I was thinking if it is possible to collect requests to confirm and to
> allow the user to decide for the whole bunch of expressions and code
> blocks. Besides implementation issues, there is a question concerning UI
> that will allow to inspect code to be evaluated.

Yes, in the example above I thought about including something
with a yes-or-no-p where users could quickly add forms to a
safe list some (defcustom org-known-safe-cells '()) or something
like that. A user could do that with the new machinery, and we
could do the same for the default implementation. I think that
is the next step once we get the basics in place.

> Calling convention for the case of function value is not described. If
> it is really the same as for `org-confirm-babel-evaluate' then this user
> option should be mentioned in the docstring.

When I correct the function signature to actually match
I will make a note in the docstring.

> :package-version instead of :version?

I think because org is part of emacs core we use the emacs version?
I see "24.1" included with other org defcustoms.

> Is there any reason to not use the :safe property of `defcustom'? I see
> that you take definition of `org-confirm-babel-evaluate' as a template
> so I wonder if there is some particular reason or the original code was
> just written before introducing of :safe.

I'm guessing that it was written before :safe, but don't
know for sure. A systematic cleanup of stuff like that
could come after this maybe?



Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats

2022-12-10 Thread Tom Gillespie
Here is an updated patch. Best!
Tom

On Thu, Dec 8, 2022 at 3:42 AM Ihor Radchenko  wrote:
>
> Tom Gillespie  writes:
>
> > Hi,
> >Here is a patch for org-contrib/lisp/org-expiry.el to account for
> > recent changes to org-time-stamp-formats. Best,
> > Tom
> >
> > PS is this list still the best place to send org-contrib patches?
>
> Yes, this list is suitable as long as org-expiry is orphaned.
>
> For the patch, note that it is not a good idea to bump the minimal
> required Org version of Org 9.6. I'd rather suggest keeping backwards
> compatibility by stripping "<" and ">" from org-time-stamp-formats, if
> any. That will not affect existing users of older Emacs versions and
> built-in Org.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
From 248104955707a7195a594712d1c7ad9d64dee1c1 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Dec 2022 01:02:35 -0800
Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
 refactor

* lisp/org-expiry.el (org-expiry-insert-created)
(org-expiry-insert-expiry): timestamp formats dropped delimiters so a
slight modification is required following org commit
e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward
compatible by removing < and > from the old timestamp format
---
 lisp/org-expiry.el | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el
index 98ad58a..0909aaf 100644
--- a/lisp/org-expiry.el
+++ b/lisp/org-expiry.el
@@ -299,10 +299,13 @@ update the date."
   (setq d-hour (format-time-string "%H:%M" d-time))
   (setq timestr
 	;; two C-u prefixes will call org-read-date
-	(if (equal arg '(16))
-		(concat "<" (org-read-date
-			 nil nil nil nil d-time d-hour) ">")
-	  (format-time-string (cdr org-time-stamp-formats
+(concat "<"
+(if (equal arg '(16))
+(org-read-date nil nil nil nil d-time d-hour)
+  (format-time-string
+   (string-replace "<" ""
+   (string-replace ">" "" (cdr org-time-stamp-formats)
+">"))
   ;; maybe transform to inactive timestamp
   (if org-expiry-inactive-timestamps
 	  (setq timestr (concat "[" (substring timestr 1 -1) "]")))
@@ -320,10 +323,13 @@ and insert today's date."
 (setq d-time (if d (org-time-string-to-time d)
 		   (current-time)))
 (setq d-hour (format-time-string "%H:%M" d-time))
-(setq timestr (if today
-		  (format-time-string (cdr org-time-stamp-formats))
-		(concat "<" (org-read-date
- nil nil nil nil d-time d-hour) ">")))
+(setq timestr (concat "<"
+  (if today
+  (format-time-string
+   (string-replace "<" ""
+   (string-replace ">" "" (cdr org-time-stamp-formats
+(org-read-date nil nil nil nil d-time d-hour))
+  ">"))
 ;; maybe transform to inactive timestamp
 (if org-expiry-inactive-timestamps
 	(setq timestr (concat "[" (substring timestr 1 -1) "]")))
-- 
2.37.4



Re: Multiple versions of Org in load-path problem

2022-12-10 Thread Tom Gillespie
> I don't think there is any safe way to install an updated version of
> org-mode other than

I don't know whether it counts as safe, but since I don't see it
mentioned anywhere else in the thread you can do the following.

(require 'package)
(unless package--initialized
  (package-initialize))
(assq-delete-all 'org package--builtins)
(assq-delete-all 'org package--builtin-versions)
(package-install 'org)



[PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

2022-12-10 Thread Tom Gillespie
Hi,
Here is a patch that improves the ergonomics and thus hopefully
the security for the recent changes to check evaluation for cells. Full
details in the commit message on the patch. Best!
Tom
From da3a88e2919ec5bac6328d55eb99b3352f25b9b4 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sat, 10 Dec 2022 12:11:17 -0800
Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable

* lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control
execution of cells separate from execution of src blocks, it works in
exactly the same way as org-confirm-babel-evaluate.
* lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is
now used to check cells independent of org-confirm-babel-evaluate.

Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it
became extremely annoying to tangle files that make extensive use of
elisp expression in src block #+header: statements.

This commit resolves the issue by making it possible to ignore checks
on cells (the old behavior) without compromising general security for
running src blocks.

This is necessary because there is no easy way to hop swap
org-confirm-babel-evaluate between org-get-src-block-info where
org-babel-read is called and the execution of that src block. It could
probably be done using advice around org-babel-read, but that is a
level of hackery that should be avoided.
---
 lisp/ob-core.el | 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 62b0d3612..14669d940 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -128,6 +128,14 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding."
 ;; don't allow this variable to be changed through file settings
 (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t)))
 
+(defcustom org-confirm-babel-evaluate-cell t
+  "Confirm before evaluating a cell."
+  :group 'org-babel
+  :version "29.1"
+  :type '(choice boolean function))
+;; don't allow this variable to be changed through file settings
+(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t)))
+
 (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil
   "\\\
 Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding."
@@ -3180,11 +3188,14 @@ situations in which is it not appropriate."
 		  (string= cell "*this*")))
  ;; Prevent arbitrary function calls.
  (if (and (memq (string-to-char cell) '(?\( ?`))
+  (if (functionp org-confirm-babel-evaluate-cell)
+  (funcall org-confirm-babel-evaluate-cell cell)
+org-confirm-babel-evaluate-cell)
   (not (org-babel-confirm-evaluate
-  ;; See `org-babel-get-src-block-info'.
-  (list "emacs-lisp" (format "%S" cell)
-'((:eval . yes)) nil (format "%S" cell)
-nil nil
+;; See `org-babel-get-src-block-info'.
+(list "emacs-lisp" (format "%S" cell)
+  '((:eval . yes)) nil (format "%S" cell)
+  nil nil
  ;; Not allowed.
  (user-error "Evaluation of elisp code %S aborted." cell)
 	   (eval (read cell) t)))
-- 
2.37.4



[PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats

2022-12-04 Thread Tom Gillespie
Hi,
   Here is a patch for org-contrib/lisp/org-expiry.el to account for
recent changes to org-time-stamp-formats. Best,
Tom

PS is this list still the best place to send org-contrib patches?
From 2408e92a9c5e155b55a374462d1314aabbe50fe0 Mon Sep 17 00:00:00 2001
From: Tom Gillespie 
Date: Sun, 4 Dec 2022 01:02:35 -0800
Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
 refactor

* lisp/org-expiry.el (org-expiry-insert-created)
(org-expiry-insert-expiry): timestamp formats dropped delimiters so a
slight modification is required following org commit
e3a7c01874c9bb80e04ffa58c578619faf09e7f0, also bump version to 0.3 and
add a dependency on org 9.6 to ensure that the new version of
org-time-stamp-formats is present and users of older versions of org
will not accidentally load the new version
---
 lisp/org-expiry.el | 21 -
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el
index 98ad58a..d8d604b 100644
--- a/lisp/org-expiry.el
+++ b/lisp/org-expiry.el
@@ -3,9 +3,10 @@
 ;; Copyright 2007-2021 Free Software Foundation, Inc.
 ;;
 ;; Author: Bastien Guerry 
-;; Version: 0.2
+;; Version: 0.3
 ;; Keywords: org, expiry
 ;; Homepage: https://git.sr.ht/~bzg/org-contrib
+;; Package-Requires: ((org "9.6"))
 
 ;; This file is not part of GNU Emacs.
 
@@ -299,10 +300,11 @@ update the date."
   (setq d-hour (format-time-string "%H:%M" d-time))
   (setq timestr
 	;; two C-u prefixes will call org-read-date
-	(if (equal arg '(16))
-		(concat "<" (org-read-date
-			 nil nil nil nil d-time d-hour) ">")
-	  (format-time-string (cdr org-time-stamp-formats
+(concat "<"
+(if (equal arg '(16))
+(org-read-date nil nil nil nil d-time d-hour)
+  (format-time-string (cdr org-time-stamp-formats)))
+">"))
   ;; maybe transform to inactive timestamp
   (if org-expiry-inactive-timestamps
 	  (setq timestr (concat "[" (substring timestr 1 -1) "]")))
@@ -320,10 +322,11 @@ and insert today's date."
 (setq d-time (if d (org-time-string-to-time d)
 		   (current-time)))
 (setq d-hour (format-time-string "%H:%M" d-time))
-(setq timestr (if today
-		  (format-time-string (cdr org-time-stamp-formats))
-		(concat "<" (org-read-date
- nil nil nil nil d-time d-hour) ">")))
+(setq timestr (concat "<"
+  (if today
+  (format-time-string (cdr org-time-stamp-formats))
+(org-read-date nil nil nil nil d-time d-hour))
+  ">"))
 ;; maybe transform to inactive timestamp
 (if org-expiry-inactive-timestamps
 	(setq timestr (concat "[" (substring timestr 1 -1) "]")))
-- 
2.37.4



Re: org-assert-version considered harmful

2022-12-03 Thread Tom Gillespie
Hi Ihor,
I have been able to use `unload-feature' (with some additional
hackery) to get things working. It is not a pretty sight (there are a
bunch of pitfalls and unexpected side-effects that are likely bugs
when using unload-feature), so yes, ideally Emacs would make it
possible to have multiple versions of the same package. Many thanks!
Tom



Re: Feedback on Org syntax document

2022-12-02 Thread Tom Gillespie
Hi Ihor,
   Thank you for resurfacing this. I'll keep an eye out for the other
threads. I've been preoccupied with trying to wrap up my PhD work so
have been (and will continue to be for a bit longer) focused on other
things. I'll be back to focus on many of these questions some time in
the late winter/early spring if I had to guess. Best!
Tom



Re: org-assert-version considered harmful

2022-12-02 Thread Tom Gillespie
Sorry to be so late chiming in here. I've only now encountered this
due to the 9.6 release. In short, org-assert-version is an absolute
disaster for me.

At the very least org-assert-version should be non-fatal by default.

Without going into too much detail, an orgstrap shebang block is
forced to use the system installed version of org because it is
intended to work in the absence of an init.el file, or before an
init.el file can ever be loaded.

This means that if a newer version of org is installed then no code
can ever run again after that package is visible on the load path
because loading the newer version of org will immediately cause an
error when something (e.g. ob-python) tries to require org-macs,
terminating the execution of the orgstrap block prematurely. There is
no simple workaround, and there is no guaranteed workaround aside from
going to great lengths to only ever use the builtin version of org.

I'm not going to write anything else at the moment because I've just
spent the last 3+ hours trying to deal with this and am in an
extremely uncharitable mood.

Tom



Re: [PATCH] Delete some Emacs 24 compat code

2022-08-09 Thread Tom Gillespie
> The manual actually says
>
>   "If this exists, it names packages on which the current package
>   depends for proper operation."
>
> so I think it is reasonable to only list the minimum supported Emacs
> version, not the minimum version where it partially or fully works, but
> is not supported.

The weasel words here are "proper operation" because it covers everything
from "will byte compile without errors (but maybe with warnings)" to "has
zero bugs and will never fail under any circumstances." My interpretation
is that Package-Requires means "will byte compile without errors" because
all software has bugs. Unfortunately package metadata doesn't seem to
have another field for something like Package-Supported-Version.

> Problem I see with your approach is there will be an expectation that if
> it lists Emacs 25.x that it works under that version and anything which
> doesn't work is a bug. People will not check this list, README or NEWS
> files to verify what version of Emacs is compatible with - if they can
> use package.el to install it, they will expect that it works without any
> issues and any encountered are either a configuration error or a bug.

I agree that this is an issue. I think the easiest solution would be to add
something to org-submit-bug-report which would inform the user that they
are running a version of org that is too new for their version of emacs and
is thus unsupported.

Another solution would be for package metadata to deconflate "will
immediately fail if you try to run this on old versions" from "only make
this update available to users running emacs at or above this version."
I think this is a variant of your suggestion to make upgrading to unsupported
versions more difficult but not impossible.

> Even worse, once a problem with (for example) Emacs 25.x is found, what
> do we do? Would we have to push out a new version just to now update the
> requires line and forcing an update for all users? Which commit do we
> use to push out that update (given there will have been changes since
> the last release and we may not be ready to push them out in a new
> version yet).

I don't see how a bug that is only encountered on 25 is different from
any other bug in this case. We aren't going to continue to support
25 by continuing to maintain a 9.5.x branch when we go on to 9.6,
but as long as we don't e.g. start using e.g. functions that are not
present in 25 that cause immediate runtime failures or byte compile
failures, then hard blocking users on 25 from installing from elpa at
all seems like artificially depriving users of the ability to choose at
their own risk (albeit to make the maintainer's lives easier).

> An alternative approach is to deliberately make it harder to upgrade org
> if your running an unsupported version of Emacs. This would prevent
> automatic updates to a version which is not supported and (possibly) doe
> sot work, either partially or fully.  Manage user expectations by making
> it very explicit to the end user they are running a older version of
> emacs which may not be compatible with latest version of org.They can
> either decide to continue with the existing version they have installed
> or they can upgrade to a more recent Emacs or they can install org
> manually if they really want to accept the risk and run in an
> unsupported configuration.

As mentioned above, I also like this approach. We could create a hack
to work around the missing package metadata field, which would cause
a failure when trying to build on emacs < 26 unless org-i-know-what-i-am-doing
or some such is non-nil. The error message would say something along
the lines of "this version of org {org-version} will run on {emacs-version}"
but it is not supported. If you still want to install it, please run
(setq org-i-know-what-i-am-doing t) and then install the package again"
or something like that.

Best!

Tom



  1   2   3   4   5   6   >