Re: [PATCH] initialize org-babel-tangle-lang-exts to nil
> More accurate approach would be using (eval-after-load 'ob-tangle > ...). This is what is being done in WIP branch that addresses many > similar problems across Org mode: > https://git.sr.ht/~yantar92/org-mode/log/feature/refactor-deps-v2 > https://git.sr.ht/~yantar92/org-mode/commit/6bcb99413e18324367fa27f8572955d5a92092ce Got it. Will keep an eye on this. Thanks! Tom
[PATCH] initialize org-babel-tangle-lang-exts to nil
Hi, Here is a fix for bad init values for org-babel-tangle-lang-exts. Details in the patch commit message. Best, Tom From 71558cc80181b5d7a1d720785165b358984284e3 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Fri, 28 Jun 2024 14:08:08 -0700 Subject: [PATCH] initialize org-babel-tangle-lang-exts to nil * lisp/ob-C.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-awk.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-clojure.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-fortran.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-groovy.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-haskell.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-java.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-latex.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-lisp.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-lua.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-maxima.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-ocaml.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-perl.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-processing.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-python.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-ruby.el (org-babel-tangle-lang-exts): init to nil * lisp/ob-sed.el (org-babel-tangle-lang-exts): init to nil org-bable-tangle-lang-exts should be initialized to nil and not as a void variable, if it is not already initialized then this will cause a void-variable error immediately when it is used in add-to-list this corrects the original addition in 4a0e5cf88f684db775ccf43dc5edc2e8754a2d92 as well as other files that followed the pattern --- lisp/ob-C.el | 2 +- lisp/ob-awk.el| 2 +- lisp/ob-clojure.el| 2 +- lisp/ob-fortran.el| 2 +- lisp/ob-groovy.el | 2 +- lisp/ob-haskell.el| 2 +- lisp/ob-java.el | 2 +- lisp/ob-latex.el | 2 +- lisp/ob-lisp.el | 2 +- lisp/ob-lua.el| 2 +- lisp/ob-maxima.el | 2 +- lisp/ob-ocaml.el | 2 +- lisp/ob-perl.el | 2 +- lisp/ob-processing.el | 2 +- lisp/ob-python.el | 2 +- lisp/ob-ruby.el | 2 +- lisp/ob-sed.el| 2 +- 17 files changed, 17 insertions(+), 17 deletions(-) diff --git a/lisp/ob-C.el b/lisp/ob-C.el index cecaa60e7..0e5235d62 100644 --- a/lisp/ob-C.el +++ b/lisp/ob-C.el @@ -42,7 +42,7 @@ (declare-function org-entry-get "org" (pom property &optional inherit literal-nil)) -(defvar org-babel-tangle-lang-exts) +(defvar org-babel-tangle-lang-exts nil) (add-to-list 'org-babel-tangle-lang-exts '("C++" . "cpp")) (add-to-list 'org-babel-tangle-lang-exts '("D" . "d")) diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el index 0c07aec82..203fd0041 100644 --- a/lisp/ob-awk.el +++ b/lisp/ob-awk.el @@ -42,7 +42,7 @@ (declare-function org-babel-ref-resolve "ob-ref" (ref)) (declare-function orgtbl-to-generic "org-table" (table params)) -(defvar org-babel-tangle-lang-exts) +(defvar org-babel-tangle-lang-exts nil) (add-to-list 'org-babel-tangle-lang-exts '("awk" . "awk")) (defvar org-babel-awk-command "awk" diff --git a/lisp/ob-clojure.el b/lisp/ob-clojure.el index c7e95..f760bf698 100644 --- a/lisp/ob-clojure.el +++ b/lisp/ob-clojure.el @@ -65,7 +65,7 @@ (defvar cider-buffer-ns) -(defvar org-babel-tangle-lang-exts) +(defvar org-babel-tangle-lang-exts nil) (add-to-list 'org-babel-tangle-lang-exts '("clojure" . "clj")) (add-to-list 'org-babel-tangle-lang-exts '("clojurescript" . "cljs")) diff --git a/lisp/ob-fortran.el b/lisp/ob-fortran.el index f2047d767..de654866a 100644 --- a/lisp/ob-fortran.el +++ b/lisp/ob-fortran.el @@ -39,7 +39,7 @@ (declare-function org-entry-get "org" (pom property &optional inherit literal-nil)) -(defvar org-babel-tangle-lang-exts) +(defvar org-babel-tangle-lang-exts nil) (add-to-list 'org-babel-tangle-lang-exts '("fortran" . "F90")) (defvar org-babel-default-header-args:fortran '()) diff --git a/lisp/ob-groovy.el b/lisp/ob-groovy.el index 73b0d33fc..5624fdf0b 100644 --- a/lisp/ob-groovy.el +++ b/lisp/ob-groovy.el @@ -37,7 +37,7 @@ (require 'ob) -(defvar org-babel-tangle-lang-exts) ;; Autoloaded +(defvar org-babel-tangle-lang-exts nil) ;; Autoloaded (add-to-list 'org-babel-tangle-lang-exts '("groovy" . "groovy")) (defvar org-babel-default-header-args:groovy '()) (defcustom org-babel-groovy-command "groovy" diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el index 05f340fa0..74a595914 100644 --- a/lisp/ob-haskell.el +++ b/lisp/ob-haskell.el @@ -53,7 +53,7 @@ "ext:inf-haskell" (&optional reload)) (declare-function org-entry-get "org" (pom property &optional inherit literal-nil)) -(
Re: Possible to set block switches "globally"?
> I think that we can take names from the parser internals: I had a similar thought immediately after I sent my previous email. > :number-lines > :preserve-indent > :retain-labels > :use-labels > :label-fmt I thought that we might be able to combine :retain-labels and :use-labels, but now I understand that :use-labels is as opposed to line numbers. So think think that using them as is is probably the best approach. Defaults are effectively :number-lines no :preserve-indent (identity org-src-preserve-indentation) :retain-labels yes :use-labels yes :label-fmt (or)
Re: Possible to set block switches "globally"?
Hi, Thanks for getting things going on this again João. Based on what is in org-element-example-block-parser and org-element-src-block-parser I think :number-lines (yes|no|continue) as João proposes :indent (preserve|align|???) not sure about naming :labels (link|keep|remove|remove-whitespace) :label-format regexp-string Defaults would be :number-lines no, :indent align, :labels remove-whitespace and :label-format would inherit from the default. The remove-whitespace option would remove the label itself along with any whitespace leading up to it, this avoids the user needing to specify the leading whitespace in :label-format. The remove option by itself is retained in the event that someone has aligned their labels and wants to retain the whitespace. This may be too complex though and remove-whitespace is not needed because :label-format can be modified as needed. Header arguments need to have a single value, so I think we should split :labels and :label-format, that way users can also specify :label-format without having to specify :labels first (otherwise there is ambiguity about what a single argument :labels means. Best, Tom
Re: Consecutive plain list items of different types
IIRC I wrote the part about there being two types of lists at the syntax level, which are bulleted and numbered, descriptive are effectively bulleted in the context of this behavior, but I think it is fine to say multiple. With regard to the stated behavior about consecutive being parsed separately I don't remember writing that part unless as was suggested I mean that they were fully separate lists, but that wouldn't make sense. The behavior is that everything at a level is converted to the type of the first occurring list item at that level. One wrinkle is that when applied to individual list items org-element will report the type as written (which is expected) however on export they are converted to the type of the first occurring item. So the now-removed sentence might have referred to the org-element behavior. Regardless, the change is an improvement. Thanks! Tom
Re: Incorrect quantity of en-spaces
> This appears to be a special case, not documented on org-syntax page. Sounds good, thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Comments following not-really-affiliated keywords are becoming paragraphs
Thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Incorrect quantity of en-spaces
The org-mode syntax document describes entities as: > \NAME POST > \NAME{} > Where NAME and POST are not separated by a whitespace character. and POST is defined as: > Either the end of line or a non-alphabetic character. So using the test document: ``` \_ Foo ``` (a backslash, underscore, three spaces, and then the word Foo) I would expect to get only 2 en-spaces but I am getting 3. Looking at org-entities, an underscore with 2 spaces gets 2 en-spaces, whereas an underscore with 3 spaces gets 3 en-spaces, but if we match all 3 spaces as NAME then POST becomes invalid because "F" is neither the end of the line nor a non-alphabetic character, so we can only match the first two spaces as NAME. emacs version: 29.1 org-mode version: 9bbc21df84d507e568a3ebd17e105cdb9e163784 (latest in git) -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Clock becomes a paragraph by prefixing with not-really-affiliated-keyword
Thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Clock becomes a paragraph by prefixing with not-really-affiliated-keyword
This test document correctly parses as a clock: ``` CLOCK: [2023-04-21 Fri 19:43] ``` This test document incorrectly parses as a paragraph: ``` #+NAME: foo CLOCK: [2023-04-21 Fri 19:43] ``` -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Keyword becoming a paragraph based on optval
> Note that _affiliated keyword_ has an optional form of Ah, that was what I was missing, thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Keyword becoming a paragraph based on optval
Emacs version: 29.1 Org-mode version: f3de4c3e041e0ea825b5b512dc0db37c78b7909e (latest in git) This test document parses as a keyword: ``` #+CAPTION[*foo*]: baz ``` but this test document parses as a paragraph: ``` #+CAPTION[*foo* bar]: baz ``` -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Comments following not-really-affiliated keywords are becoming paragraphs
Emacs version: 29.1 Org-mode version: e1569918cc94253650781e83a09695739c93352f (latest in git) Test document: ``` #+CAPTION: foo # bar ``` This parses as a paragraph with the caption of foo and the body of "# bar" when it should parse as a regular keyword followed by a comment. Relevant org-syntax[1] bit: > a keyword with the same KEY as an affiliated keyword may occur so long as it > is not immediately preceding a valid element that can be affiliated. For > example, an instance of #+caption: hi followed by a blank line will be parsed > as a keyword, not an affiliated keyword. [1] https://orgmode.org/worg/org-syntax.html#Keywords -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Org-mode starting with 37d6bde27 errors out parsing org-mode/testing/examples/pub/a.org
Steps to reproduce: 1. Build emacs 29.1 2. Build org-mode with revision 37d6bde27fe228cdadcb5cdaa09287872a508777 3. Run the following: ``` emacs -q --no-site-file --no-splash --batch --eval "(progn (require 'org) (setq vc-handled-backends nil) (find-file-read-only \"org-mode/testing/examples/pub/a.org\") (org-mode) (message \"%s\" (pp-to-string (org-element-parse-buffer))) )" ``` I've attached a Dockerfile that reproduces the issue. Just throw that in a directory and run `docker build -t temp .` to see it fail. Change the `ARG ORG_VERSION=` line to `ac108a3ac1b332bf27ff2984a9cf26af3744185d` to see it succeed. Error message: ``` File mode specification error: (void-function org-export--list-bound-variables) Error: void-function (org-export--list-bound-variables) mapbacktrace(#f(compiled-function (evald func args flags) #)) debug-early-backtrace() debug-early(error (void-function org-export--list-bound-variables)) org-export--list-bound-variables() org-element--generate-copy-script(# :copy-unreadable do-not-check :drop-visibility t :drop-narrowing t :drop-contents t :drop-locals nil) org-element-copy-buffer(:to-buffer # :drop-visibility t :drop-narrowing t :drop-contents t :drop-locals nil) org-element-parse-secondary-string("<2014-03-04 Tue>" (bold citation code entity export-snippet inline-babel-call inline-src-block italic line-break latex-fragment link macro radio-target statistics-cookie strike-through subscript superscript target timestamp underline verbatim)) org-macro--find-date() org-macro--collect-macros() org-macro-initialize-templates() org-mode() (progn (require 'org) (setq vc-handled-backends nil) (find-file-read-only "/input/home/talexander/git/org-mode/testing/examples/pub/a.org") (org-mode) (message "%s" (pp-to-string (org-element-parse-buffer command-line-1(("--no-splash" "--eval" "(progn\n (require 'org)\n (setq vc-handled-backends nil)\n (find-file-read-only \"/input/home/talexander/git/org-mode/testing/examples/pub/a.org\")\n (org-mode)\n (message \"%s\" (pp-to-string (org-element-parse-buffer)))\n)")) command-line() normal-top-level() Symbol’s function definition is void: org-export--list-bound-variables ``` -- Tom Alexander pgp: https://fizz.buzz/pgp.asc Dockerfile Description: Binary data
Document-level properties incorrect and/or missing based on preceding blank lines and/or comments
Emacs version: Emacs 29.1 Org-mode version: e1569918cc94253650781e83a09695739c93352f (latest in git) The org-mode syntax document[1] says that property drawers can exist in the zeroth section with the format: ``` BEGINNING-OF-FILE BLANK-LINES COMMENT PROPERTYDRAWER ``` Using this test document: ``` :PROPERTIES: :FOO:bar :END: ``` I correctly get the foo property in the top-level org-data ``` (org-data (:standard-properties [1 1 1 33 33 0 nil org-data nil nil nil 32 33 nil # nil nil nil] :path nil :FOO "bar" :CATEGORY nil) ``` But now there are two separate issues: ### Issue 1 Putting a comment before it makes the value for the foo property incorrect (seems to be grabbing an earlier string slice): ``` # baz :PROPERTIES: :FOO:bar :END: ``` ``` (org-data (:standard-properties [1 1 1 39 39 0 nil org-data nil nil nil 38 39 nil # nil nil nil] :path nil :FOO "O: " :CATEGORY nil) ``` Interestingly, looking farther down the AST, the value for foo is properly set in the node-property, just not the org-data: ``` (node-property (:standard-properties [20 20 nil nil 33 0 nil node-property nil nil nil nil nil nil # nil nil #2] :key "FOO" :value "bar")) ``` ### Issue 2 Putting any blank lines before it makes the foo property not appear in org-data at all ``` :PROPERTIES: :FOO:bar :END: ``` ``` (org-data (:standard-properties [1 1 2 34 34 0 nil org-data nil nil nil 4 34 nil # nil nil nil] :path nil :CATEGORY nil) ``` Looking farther down the AST it seems the property-drawer became a regular drawer [1] https://orgmode.org/worg/org-syntax.html#Property_Drawers -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Inconsistent text markup handling when double-nesting markers
> Fixed, on main. Thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter
Thanks! > We aim to reduce config-dependent Org syntax in the long term. Thats wonderful news! Sometimes this stuff can really surprise you. For example, the structure of the document created by running `echo "1. foo\n 1.bar\n1.baz\n\t1.lorem"` changes based on the user's **tab-width**!! If tab-width is less than 8 then this is: ```text 1. foo 1. bar 1. baz 2. lorem ``` If tab-width is 8 then this is: ```text 1. foo 1. bar 1. baz 2. lorem ``` and if tab-width is greater than 8 this is: ```text 1. foo 1. bar 1. baz 1. lorem ``` Absolute madness! I always considered tab-width to be a personal aesthetic choice and not something that would functionally change how documents other people wrote will be parsed. Idk if its been discussed, but personally if I were given dictatorship over org-mode I would take all of these emacs variables that are defined outside of the document, and instead of having them influence org-mode directly, I would *only* use them to pre-populate values for in-buffer settings templates. For example, if a user had set `org-odd-levels-only` then I wouldn't have that impact ANY existing document they open, but if they open a new document then I would have it auto-insert `#+STARTUP: odd` at the top of the fresh document. Otherwise it seems like org-mode is unsuitable for multi-person collaboration without dictating the contents of everyone's `.emacs` file. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Inconsistent text markup handling when double-nesting markers
I used the following test document: ``` __foo__ **foo** ``` I'd expect the two to behave the same but the first one parses as: ``` (paragraph "_" (subscript "foo") "__" ) ``` Whereas the second parses as: ``` (paragraph (bold (bold "foo" ) ) ) ``` This pattern happens in worg at [2] Looking at the description for text markup in the syntax document[1], I don't see any reason the first wouldn't be parsed as an underline: 1. PRE: valid because it is the beginning of a line 2. MARKER: valid underscore 3. CONTENTS: valid. Series of objects from standard set includes both subscript and text markup, so regardless of how we parse the interior, its valid. Also cannot begin or end with whitespace but there is no whitespace in the CONTENTS. 4. MARKER: valid underscore 5. POST: Only valid if we extend the underline to the 2nd underscore so it ends at the end of the line. But the 2nd line shows us that having copies of the marker inside the CONTENTS is fine so I see two possible expected parses of the CONTENTS: 4a. (underline "foo") 4b. ((subscript "foo") (plain-text "_")) I also ran the following test document to further prove that having copies of the marker inside the CONTENTS is fine: ``` *foo*bar* ``` which parses as (bold "foo*bar") So the only way the top line would fail to parse as an underline is if it matched the first closing underscore as closing the underline, but that would be invalid because underscore is not a valid POST character and invalid copies of the closing marker are ignored as proven by both "**foo**" and "*foo*bar*". [1] https://orgmode.org/worg/org-syntax.html#Emphasis_Markers [2] https://git.sr.ht/~bzg/worg/tree/ba6cda890f200d428a5d68e819eef15b5306055f/org-contrib/babel/intro.org#L117 -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Clarify that REST is not supported on the start TIME in a time-range timestamp.
> As for the problem with REST you raised, I am inclined to remove it from > syntax doc for the time being - it only creates more confusion, > unfortunately. Makes sense, thanks. Is there anything we do to mark patches as rejected? I removed [PATCH] from the subject line. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Lesser blocks allowing unescaped lines
Thank you! Makes sense. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: [PATCH] Clarify that REST is not supported on the start TIME in a time-range timestamp.
Potentially related, org-mode is accepting this malformed timestamp from[1]: ``` <2016-02-14 Sun ++y> ``` The org-mode documentation[2] only includes REST with TIME, defining TIME as "H:MMREST". The above does not have any TIME but it accepts the timestamp anyway: ``` (timestamp :type active :range-type nil :raw-value "<2016-02-14 Sun ++y>" :year-start 2016 :month-start 2 :day-start 14 :hour-start nil :minute-start nil :year-end 2016 :month-end 2 :day-end 14 :hour-end nil :minute-end nil ) ``` Perhaps that grammar is wrong and REST needs to be separated from TIME? [1] https://github.com/howardabrams/pdx-emacs-hackers/blob/bfb7bd640fdf0ce3def21f9fc591ed35d776b26d/workshops/org-mode-gtd-feature-demo.org#L183 [2] https://orgmode.org/worg/org-syntax.html#Timestamps -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
[PATCH] Clarify that REST is not supported on the start TIME in a time-range timestamp.
If REST is included in the first TIME on a time-range timestamp then the entire timestamp becomes a single range-less timestamp. To test I used the following test document: ``` [1970-01-01 Thu 8:15-13:15foo] [1970-01-01 Thu 8:15foo-13:15] ``` The first line parses as a timerange from 8:15-13:15. The second line parses as a single timestamp at 8:15. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc From b1114e983d961d48e1d837b8d2ad209a976a5417 Mon Sep 17 00:00:00 2001 From: Tom Alexander Date: Mon, 2 Oct 2023 17:35:28 -0400 Subject: [PATCH] * org-syntax.org (Timestamps): Clarify that REST is not supported on the start TIME in a time-range timestamp. If REST is included in the first TIME on a time-range timestamp then the entire timestamp becomes a single range-less timestamp. To test I used the following test document: ``` [1970-01-01 Thu 8:15-13:15foo] [1970-01-01 Thu 8:15foo-13:15] ``` The first line parses as a timerange from 8:15-13:15. The second line parses as a single timestamp at 8:15. --- org-syntax.org | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/org-syntax.org b/org-syntax.org index c2061431..0c326ba8 100644 --- a/org-syntax.org +++ b/org-syntax.org @@ -1686,9 +1686,10 @@ -MM-DD DAYNAME - DAYNAME (optional) :: A string consisting of non-whitespace characters except =+=, =-=, =]=, =>=, a digit, or =\n=. + TIME (optional) :: An instance of the pattern =H:MMREST= where =H= - represents a one to two digit number (and can start with =0=), and =M= - represents a single digit. =REST= can contain anything but =\n= or - closing bracket. + represents a one to two digit number (and can start with =0=), and + =M= represents a single digit. =REST= can contain anything but =\n= + or closing bracket. =REST= cannot exist on the start TIME in a + time-range timestamp (the patterns with =TIME-TIME=). + REPEATER-OR-DELAY (optional) :: An instance of the following pattern: #+begin_example MARK VALUE UNIT -- 2.42.0
Re: Extra paragraphs incorrectly spawning when ":end:" appears.
Hmm thanks, that makes sense. I guess a post-processing step to merge adjacent paragraphs wouldn't work either since that wouldn't stitch together objects like the bold in this test document without re-parsing the entire paragraph: ``` foo *bar :end: baz* ``` oh well 🤷 -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Extra paragraphs incorrectly spawning when ":end:" appears.
Same problem occurs with this sample document: ``` foo #+BEGIN: bar baz ``` which parses as: ``` (section (paragraph "foo\n") (paragraph "#+BEGIN: bar\nbaz\n) ) ``` again, no blank lines and no non-paragraph elements but the single paragraph got split in two. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Extra paragraphs incorrectly spawning when ":end:" appears.
This test document has 1 paragraph: ``` foo bar baz ``` which parses as: ``` (section (paragraph "foo\nbar\nbaz\n") ) ``` This test document should have 1 paragraph but org-mode is parsing it as 2: ``` foo :end: baz ``` which parses as: ``` (section (paragraph "foo\n") (paragraph ":end:\nbaz\n") ) ``` The paragraph documentation[1] states that: > Empty lines and other elements end paragraphs. But the document contains no empty lines and we can see in the output that it only contains paragraphs. [1] https://orgmode.org/worg/org-syntax.html#Paragraphs -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Lesser blocks allowing unescaped lines
This happens in worg at: https://git.sr.ht/~bzg/worg/tree/ba6cda890f200d428a5d68e819eef15b5306055f/exporters/ox-docstrings.org#L2490 The documentation for lesser blocks[1] states: > Lines beginning with an asterisk or `#+` must be quoted by a comma (`,*`, > `,#+`). However, the following test document parses as a lesser block despite containing a line starting with an unescaped #+: ``` #+CATEGORY: foo #+begin_src text #+CATEGORY: bar #+end_src ``` which parses as: ``` (org-data (:standard-properties [1 1 1 60 60 0 nil org-data nil nil nil 3 60 nil # nil nil nil] :path nil :CATEGORY "foo") (section (:standard-properties [1 1 1 60 60 0 nil first-section nil nil nil 1 60 nil # nil nil #0]) (keyword (:standard-properties [1 1 nil nil 17 0 nil top-comment nil nil nil nil nil nil # nil nil #1] :key "CATEGORY" :value "foo")) (src-block (:standard-properties [17 17 nil nil 60 0 nil nil nil nil nil nil nil nil # nil nil #1] :language "text" :switches nil :parameters nil :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "#+CATEGORY: bar\n" ``` whereas I would expect this to be ``` (section (keyword :key "CATEGORY" :value "foo") (paragraph "#+begin_src text") (keyword :key "CATEGORY" :value "bar") (paragraph "#+end_src") ) ``` This test document shows that lines with an unescaped "*" do break up the lesser block: ``` * foo #+begin_src text * bar #+end_src ``` [1] https://orgmode.org/worg/org-syntax.html#Blocks -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
COUNTER-SET for alphabetical ordered lists ignored for utf-8 exporter
It seems that COUNTER-SET[1] is not being honored when exporting to utf-8 for alphabetical lists even though it is honored for numeric lists. When exporting to html, COUNTER-SET is honored for both. Test document: ``` # An ordered list starting at 3 1. [@3] foo # An ordered list starting at 1 m. bar # An ordered list starting at 11 m. [@k] baz ``` Launching emacs with: (Setting org-list-allow-alphabetical is necessary or else the alphabetical lists will become paragraphs) ``` emacs -q --eval '(setq org-list-allow-alphabetical t)' /tmp/test.org ``` When exporting to html you get (edited to remove whitespace for clarity): ``` foo bar baz ``` But when exporting to utf-8 you get: (whitespace removed again) ``` 3. foo m. bar m. baz ``` Whereas I would expect: (whitespace removed again) ``` 3. foo m. bar k. baz ``` On a slightly related note: it seems the COUNTER-SET[1] allows single-letter values even when org-list-allow-alphabetical is nil. I don't think that is going to hurt anyone but I figured I should mention it in case its a bug (test doc: `1. [@k] foo` is a plain list starting at 11 even when org-list-allow-alphabetical is nil). [1] https://orgmode.org/worg/org-syntax.html#Items Emacs 29.1, Org-mode version 9.7-pre (release_9.6.8-781-gc70354) -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Subscript with parenthesis
> Not true. I tried > > b^(*asd*) and bold inside superscript does get parsed. Ah thanks for double-checking! You're right, that is getting parsed. Not sure what test document I was using to make me think objects didn't work inside the parenthesis. -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: [PATCH] Add backslash to list of POST characters for text markup
Thanks! -- Tom Alexander pgp: https://fizz.buzz/pgp.asc On Fri, Sep 22, 2023, at 5:29 AM, Ihor Radchenko wrote: > "Tom Alexander" writes: > >> Backslash appears to be supported. To test I used the following test >> document: >> ``` >> foo ~bar~\& baz >> ``` > > Thanks! > You are right. > Applied, onto master, with minor amendments to the commit message. > https://git.sr.ht/~bzg/worg/commit/ba6cda89 > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
[PATCH] Add backslash to list of POST characters for text markup
Backslash appears to be supported. To test I used the following test document: ``` foo ~bar~\& baz ``` This happens in a document in worg: https://git.sr.ht/~bzg/worg/tree/ae64e1a54185232d4ebdcab174d8d4319ffd564d/org-release-notes.org#L The ampersand was chosen for the test document since that is not a supported POST character, to make sure backslash was not simply escaping the next character. In the documentation I wrote out the word "backslash" in parenthesis to disambiguate between backslash and escaping the following comma. Patch is attached. -- Tom Alexander pgp: https://fizz.buzz/pgp.ascFrom 098434680b5e3942acc00684a47389f2cdab6208 Mon Sep 17 00:00:00 2001 From: Tom Alexander Date: Thu, 21 Sep 2023 21:14:33 -0400 Subject: [PATCH] Add backslash to list of POST characters for text markup. Backslash appears to be supported. To test I used the following test document: ``` foo ~bar~\& baz ``` The ampersand was chosen since that is not a supported POST character, to make sure backslash was not simply escaping the next character. In the documentation I wrote out the word "backslash" in parenthesis to disambiguate between backslash and escaping the following comma. --- org-syntax.org | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/org-syntax.org b/org-syntax.org index c5299741..8f0f9b0c 100644 --- a/org-syntax.org +++ b/org-syntax.org @@ -249,9 +249,9 @@ discarded. This also applies to single-line elements. :This paragraph will not contain :a long sequence of spaces before "a". -: +: :This paragraph does not have leading spaces according to the parser. -: +: :#+begin_src emacs-lisp : (+ 1 2) :#+end_src @@ -1742,7 +1742,7 @@ whitespace characters. verbatim) or a series of objects from the standard set. In both cases, CONTENTS may not begin or end with whitespace. + [[#Special_Tokens][POST]] :: Either a whitespace character, =-=, =.=, =,=, =;=, =:=, =!=, =?=, ='=, =)=, =}=, - =[=, ="=, or the end of a line. + =[=, ="=, =\= (backslash), or the end of a line. *Examples* -- 2.42.0
Consecutive plain list items of different types
The org-mode documentation[1] states for plain lists that: > List types are mutually exclusive at the same level of indentation, if both > types are present consecutively then they parse as separate lists. first a minor nit-pick that "both" is probably not the correct word here since there are 3 types of lists, not two (unordered, ordered, and descriptive). I'd go with "multiple" instead IMO. but more importantly, based on that description I would expect the following test document to parse into three separate plain lists, but it parses as a single plain list with 3 items: ``` 1. foo - bar - lorem :: ipsum ``` [1] https://orgmode.org/worg/org-syntax.html#Plain_Lists -- Tom Alexander pgp: https://fizz.buzz/pgp.asc
Re: Subscript with parenthesis
Some additional things I'm noticing: - when using parenthesis, :use-brackets-p is nil, so they're not equivalent to curly braces. - it does not support objects inside the parenthesis, just plain text, which again means they're not equivalent to braces. - it, however, seems to require that the parenthesis are balanced because this test document does NOT contain a subscript: ``` foo_(b(ar) ``` which is closer to the curly braces requirement since that seems to be the only part of the subscript/superscript documentation that mentions needing balance. -- Tom Alexander
Subscript with parenthesis
The org-mode documentation[1] states that the SCRIPT portion of the subscript/superscript is either an asterisk, the standard set of objects wrapped in balanced curly braces, or an optional sign followed by "Either the empty string, or a string consisting of any number of alphanumeric characters, commas, backslashes, and dots" But I'm seeing the following test document parse as containing a subscript despite using parenthesis which I do not think matches any of the above criteria: ``` foo_(bar) ``` [1] https://orgmode.org/worg/org-syntax.html#Subscript_and_Superscript -- Tom Alexander
Re: Fixed width areas not allowing tab after leading colon.
Thanks! -- Tom Alexander On Sun, Sep 17, 2023, at 5:48 AM, Ihor Radchenko wrote: > "Tom Alexander" writes: > >> The documentation for fixed width areas states: A “fixed-width line” starts >> with a colon character (:) and either a whitespace character or the >> immediate end of the line. >> ... >> Fixed-width area documentation: >> https://orgmode.org/worg/org-syntax.html#Fixed_Width_Areas > > org-syntax.html is not accurate here. The parser only allows ": " (colon > followed by space) and no other variant. > > Fixed, on master. > https://git.sr.ht/~bzg/worg/commit/a42f57ac > >> This happens in a document in worg: >> https://git.sr.ht/~bzg/worg/tree/74e80b0f7600801b1d1594542602394c085cc2f9/item/org-contrib/org-bom.org#L499 > > Fixed, on master. > https://git.sr.ht/~bzg/worg/commit/0c8d5679 > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
Re: [PATCH] Re: Description list with " :: " in the tag.
Sorry for the delay, I've been busy in the IRLs. I've updated the patch to reflect that the parser grabs the text before the last " :: " and then parses it as objects. The new patch is attached. -- Tom Alexander On Thu, Sep 14, 2023, at 7:24 AM, Ihor Radchenko wrote: > "Tom Alexander" writes: > >> I've written a patch (attached) with my proposed wording changes to >> the documentation, should I be starting another thread or does >> dropping it here work best? > > You can just modify subject with [PATCH], as I did. > >> ... I do not have commit access so I'd need >> someone with such authority to do the last bit. > > Sure. > >> + =TAG-TEXT= is one of more objects from the standard set so long as >> + they do not contain a newline character, until the last occurrence >> + of the substring =" :: "= (two colons surrounded by whitespace, >> + without the quotes). > > It does not fully represent what is going on - Org parser is top-down > and does not parse objects before it is done parsing the descriptive > list item. So, > > - *foo :: bar* does not actually contain bold markup > > Rather it is "* foo" tag + "bar* does not actually contain bold markup" > description. > > What happens is that the parser splits the first line of the item by the > last " :: " and only then proceeds with parsing the tag and description > using standard set of objects: > > - <> :: > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>From c8812bf7d81dc824d8ecf2c03368f58884773ddf Mon Sep 17 00:00:00 2001 From: Tom Alexander Date: Wed, 13 Sep 2023 18:19:05 -0400 Subject: [PATCH] org-syntax.org: Fix definition of description list tags. Description lists support objects in their tags and they support the substring " :: ". --- org-syntax.org | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/org-syntax.org b/org-syntax.org index 123fc232..fc5e9a37 100644 --- a/org-syntax.org +++ b/org-syntax.org @@ -470,9 +470,10 @@ BULLET COUNTER-SET CHECK-BOX TAG CONTENTS + CHECK-BOX (optional) :: A single whitespace character, an =X= character, or a hyphen enclosed by square brackets (i.e. =[ ]=, =[X]=, or =[-]=). + TAG (optional) :: An instance of the pattern =TAG-TEXT ::= where - =TAG-TEXT= represents a string consisting of non-newline characters - that does not contain the substring ="Â ::Â "= (two colons surrounded by - whitespace, without the quotes). + =TAG-TEXT= is the text up until the last occurrence of of the + substring ="Â ::Â "= (two colons surrounded by whitespace, without the + quotes) on that line. =TAG-TEXT= is then parsed with the standard + set of objects. + CONTENTS (optional) :: A collection of zero or more elements, ending at the first instance of one of the following: - The next item. -- 2.42.0
Fixed width areas not allowing tab after leading colon.
The documentation for fixed width areas states: A “fixed-width line” starts with a colon character (:) and either a whitespace character or the immediate end of the line. Using the test document: ``` :foo ``` parses as a paragraph instead of a fixed-width area: ``` (org-data (:standard-properties [1 1 1 7 7 0 nil org-data nil nil nil 3 7 nil # nil nil nil] :path nil :CATEGORY nil) (section (:standard-properties [1 1 1 7 7 0 nil first-section nil nil nil 1 7 nil # nil nil #0]) (paragraph (:standard-properties [1 1 1 7 7 0 nil top-comment nil nil nil nil nil nil # nil nil #1]) #(": foo\n" 0 6 (:parent #2) ``` This happens in a document in worg: https://git.sr.ht/~bzg/worg/tree/74e80b0f7600801b1d1594542602394c085cc2f9/item/org-contrib/org-bom.org#L499 Emacs version: GNU Emacs 29.1 (build 1, x86_64-pc-linux-musl) Org-mode version: c703541ffcc14965e3567f928de1683a1c1e33f6 (latest in git) Fixed-width area documentation: https://orgmode.org/worg/org-syntax.html#Fixed_Width_Areas -- Tom Alexander
Re: Description list with " :: " in the tag.
I've written a patch (attached) with my proposed wording changes to the documentation, should I be starting another thread or does dropping it here work best? I do not have commit access so I'd need someone with such authority to do the last bit. -- Tom Alexander From 20addaa5ab7d4e9420ade1125c2a337345ecdd31 Mon Sep 17 00:00:00 2001 From: Tom Alexander Date: Wed, 13 Sep 2023 18:19:05 -0400 Subject: [PATCH] org-syntax.org: Fix definition of description list tags. Description lists support objects in their tags and they support the substring " :: ". --- org-syntax.org | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/org-syntax.org b/org-syntax.org index 123fc232..3046e26c 100644 --- a/org-syntax.org +++ b/org-syntax.org @@ -470,9 +470,10 @@ BULLET COUNTER-SET CHECK-BOX TAG CONTENTS + CHECK-BOX (optional) :: A single whitespace character, an =X= character, or a hyphen enclosed by square brackets (i.e. =[ ]=, =[X]=, or =[-]=). + TAG (optional) :: An instance of the pattern =TAG-TEXT ::= where - =TAG-TEXT= represents a string consisting of non-newline characters - that does not contain the substring ="Â ::Â "= (two colons surrounded by - whitespace, without the quotes). + =TAG-TEXT= is one of more objects from the standard set so long as + they do not contain a newline character, until the last occurrence + of the substring ="Â ::Â "= (two colons surrounded by whitespace, + without the quotes). + CONTENTS (optional) :: A collection of zero or more elements, ending at the first instance of one of the following: - The next item. -- 2.42.0
Re: Description list with " :: " in the tag.
Thanks! -- Tom Alexander On Sat, Sep 9, 2023, at 5:06 AM, Ihor Radchenko wrote: > "Tom Alexander" writes: > >> Emacs version: 29.1 >> Org-mode version: 163bafb43dcc2bc94a2c7ccaa77d3d1dd488f1af >> >> Found a conflict between the documentation and the parser behavior. The >> org-mode documentation[1] for description list items says that TAG '[...] >> does not contain the substring " :: "' >> >> Using this sample document, I have created a plain list item with a tag that >> contains that substring by wrapping it in a verbatim block: >> ``` >> - =foo :: bar= :: baz >> ``` >>(item >> ... >> ((1 0 "- " nil nil "=foo :: bar=" 23)) >> ... >> It seems that "TAG-TEXT" is not just text but it can include objects and >> those objects can include the substring " :: ". > > It is simpler. > Everything after the bullet and before the last " :: " is considered as > tag. Everything after the last " :: " is description. > Then, tag and description are parsed, allowing objects inside. > > org-syntax document is inaccurate here - it says that the _first_ " :: " > is used as tag:description delimiter, not the _last_. > > I do not see any benefit changing the current parser. So, we probably > need to update org-syntax document instead. > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
Description list with " :: " in the tag.
Emacs version: 29.1 Org-mode version: 163bafb43dcc2bc94a2c7ccaa77d3d1dd488f1af Found a conflict between the documentation and the parser behavior. The org-mode documentation[1] for description list items says that TAG '[...] does not contain the substring " :: "' Using this sample document, I have created a plain list item with a tag that contains that substring by wrapping it in a verbatim block: ``` - =foo :: bar= :: baz ``` Which parses to: ``` (org-data (:standard-properties [1 1 1 23 23 0 nil org-data nil nil nil 3 23 nil # nil nil nil] :path nil :CATEGORY nil) (section (:standard-properties [1 1 1 23 23 0 nil first-section nil nil nil 1 23 nil # nil nil #0]) (plain-list (:standard-properties [1 1 1 23 23 0 nil top-comment nil nil nil nil nil nil # nil ((1 0 "- " nil nil "=foo :: bar=" 23)) #1] :type descriptive) (item (:standard-properties [1 1 19 23 23 0 (:tag) item nil nil nil nil nil nil # nil ((1 0 "- " nil nil "=foo :: bar=" 23)) #2] :bullet "- " :checkbox nil :counter nil :pre-blank 0 :tag ((verbatim (:standard-properties [3 nil nil nil 15 0 nil nil nil nil nil nil nil nil # nil nil #3] :value "foo :: bar" (paragraph (:standard-properties [19 19 19 23 23 0 nil nil nil nil nil nil nil nil # nil nil #3]) #("baz\n" 0 4 (:parent #4))) ``` It seems that "TAG-TEXT" is not just text but it can include objects and those objects can include the substring " :: ". [1] https://orgmode.org/worg/org-syntax.html#Items -- Tom Alexander
Re: [RFC] Quoting property names in tag/property matches [Was: [BUG?] Matching tags: & operator no more implicit between tags and special property]
Ignore the previous message. I see that this was about matching tags not about specifying them. Best, Tom
Re: [RFC] Quoting property names in tag/property matches [Was: [BUG?] Matching tags: & operator no more implicit between tags and special property]
Without wading too far into this, why do we need escape syntax for this? The only character that might need an escape would be colon :, but my reading of the syntax doc is that colo : will immediately terminate the property, so we would update the doc to make it clear that property names cannot contain a colon. As written, if there is an issue with the minus sign in property names then that is a bug, but I feel like I might be missing something? Tom
Re: [DISCUSSION] May we recognize everything like [[protocol:uri]] as a non-fuzzy link? (was: [BUG] URI handling is overly complicated and nonstandard [9.6.7 (N/A @ /gnu/store/mg7223g8mw90lccp6mm5g6f3
This is a timely discussion. I have been thinking about how to deal with prefixes defined by the #+link: keyword which is directly related to this question. I think the following might be a solution that also avoids the issue brought up by Arne. The original "bug" cannot be resolved because bare URIs have syntax that conflicts with Org syntax. However I think we can do better than directing users to org-link-set-parameters. My suggestion is as follows. Schemes/prefixes defined by the #+link: keyword can be used without surrounding syntax markers but may not contain spaces etc. To support this Org parsers should always parse prefix:suffix as a _putative_ link which must then be checked against a list of known schemes that are either built in or have been declared by the user to indeed be legitimate schemes. In the tel: case, the way to solve the original bug is simply to add the line #+link: tel tel: which would tell Org that e.g. tel:555-555- is a real uri, and that it should expand to itself. At the same time this solution would avoid Arne's issue (which I also have in some of my documents where I have use fig: and tbl: as prefixes in names and reference them via [[fig:figure-name]]) because the parser would only treat prefix: in an internal link as a scheme if it is defined explicitly by the user in a #+link: keyword or in their init.el. Thoughts? Tom
Text markup and TeX-style LaTeX fragments spanning more than three lines
The documentation for both text markup and TeX-style LaTeX fragments states that their contents (CONTENTS for text markup, BODY for TeX-style LaTeX fragments) cannot span more than three lines, but I'm not seeing such limitation for either in practice. I don't know if this is a documentation error or a parser error. Documentation text markup: https://orgmode.org/worg/org-syntax.html#Emphasis_Markers Documentation LaTeX fragments: https://orgmode.org/worg/org-syntax.html#LaTeX_Fragments Test document text markup: ``` foo *bar baz* lorem text *markup cannot span* more than *three lines. foo bar* baz foo *bar \\ baz \\ lorem \\ ipsum \\ dolar* cat ``` Test document TeX-style LaTeX fragments: ``` $foo bar baz lorem ipsum dolar$ ``` -- Tom Alexander
Re: Clarification on blank lines following list items
Thanks! -- Tom Alexander On Tue, Aug 22, 2023, at 4:26 AM, Ihor Radchenko wrote: > Ihor Radchenko writes: > >>> I am seeing dual-ownership of the post-blank in the examples below, but at >>> the same time if I put a plain-list inside a footnote definition, the >>> footnote definition ends up with sole custody of the post-blank. >> >> It is a bug in the list parser. > > Fixed, on main. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=53c9d91d3 > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
Re: Clarification on blank lines following list items
Thank you so much for explaining all of that! There is some good information there I was missing. I think the most important bit I was missing is the post-blank stuff. I was only looking at begin->end but I think digging into the post-blank is what makes this consistent. I've got 2 separate questions: 1. Is the following statement true? "Two elements can count the same character in their post-blank?" I am seeing dual-ownership of the post-blank in the examples below, but at the same time if I put a plain-list inside a footnote definition, the footnote definition ends up with sole custody of the post-blank. 2. I'm still not sure about some behavior I'm seeing. I think it would be easiest to see if we focus on exactly 1 blank line: ``` 1. bar 2. baz < this blank line here ipsum ``` In this example, the blank line gets counted in the post-blank for the plain-list but not for the item: ``` plain-list: post-blank 1 | begin 1 end 16 | contents-begin 1 contents-end 15 item: post-blank 0 | begin 1 end 8 | contents-begin 4 contents-end 8 paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 8 item: post-blank 0 | begin 8 end 15 | contents-begin 11 contents-end 15 paragraph: post-blank 0 | begin 11 end 15 | contents-begin 11 contents-end 15 paragraph: post-blank 0 | begin 16 end 22 | contents-begin 16 contents-end 22 ``` but if we take that plain-list and nest it inside another plain-list: ``` 1. foo 1. bar 2. baz < this blank line here 2. lorem ipsum ``` The blank line gets counted as a post-blank for both the item "foo" and the item "baz": ``` plain-list: post-blank 0 | begin 1 end 38 | contents-begin 1 contents-end 38 item: post-blank 1 | begin 1 end 29 | contents-begin 4 contents-end 28 paragraph: post-blank 0 | begin 4 end 8 | contents-begin 4 contents-end 8 plain-list: post-blank 0 | begin 8 end 29 | contents-begin 8 contents-end 29 item: post-blank 0 | begin 8 end 18 | contents-begin 14 contents-end 18 paragraph: post-blank 0 | begin 14 end 18 | contents-begin 14 contents-end 18 item: post-blank 1 | begin 18 end 29 | contents-begin 24 contents-end 28 paragraph: post-blank 0 | begin 24 end 28 | contents-begin 24 contents-end 28 item: post-blank 0 | begin 29 end 38 | contents-begin 32 contents-end 38 paragraph: post-blank 0 | begin 32 end 38 | contents-begin 32 contents-end 38 paragraph: post-blank 0 | begin 38 end 44 | contents-begin 38 contents-end 44 ``` Meaning the post-blank did this movement: ``` plain-list: post-blank 0 item: post-blank 1 <---<<<-\ paragraph: post-blank 0 | plain-list: post-blank 0 >>--| item: post-blank 0 | paragraph: post-blank 0 | item: post-blank 1 <---<---/ paragraph: post-blank 0 item: post-blank 0 paragraph: post-blank 0 paragraph: post-blank 0 ``` Question ---> So why is the item "baz" gaining a post-blank instead of the inner plain-list (bar baz) keeping that post-blank? I would expect it to instead be: ``` plain-list: post-blank 0 item: post-blank 1 paragraph: post-blank 0 here -> plain-list: post-blank 1 item: post-blank 0 paragraph: post-blank 0 not here -> item: post-blank 0 paragraph: post-blank 0 item: post-blank 0 paragraph: post-blank 0 paragraph: post-blank 0 ``` I re-did both test cases using greater blocks and lesser blocks instead of paragraphs to make sure it wasn't that historical exception at the end of your email, and the post-blank behavior was exactly the same. -- Tom Alexander
Clarification on blank lines following list items
| | bar | Yes | | baz | Yes | | lorem | No| Against expectations, we did not coerce lorem to consume its trailing blank lines. What is different between "baz" and "lorem"? Well, "baz" is contained within "foo" which has a "lorem" after it, whereas "lorem" is contained within "cat" which does not have any list items after it. New theory: List items own their trailing blank lines unless they are both the final list item and not the final element of a non-final list item. | Plain List *Item* | Owns trailing blank lines | Why | |-+---+--| | cat (includes foo -> lorem) | No| Final list item and not contained in a list item | | foo (includes bar baz) | Yes | Not the final list item | | bar | Yes | Not the final list item | | baz | Yes | Final element of non-final list item | | lorem | No| Final list item and final element of final list item | *** Test case 4 So if that theory is true, then we should be able to coerce lorem to consume its trailing blank lines by adding a second item to the cat list. ``` 1. cat 1. foo 1. bar 2. baz 2. lorem 2. dog ipsum ``` | Plain List *Item* | Owns trailing blank lines | |-+---| | cat (includes foo -> lorem) | Yes | | foo (includes bar baz) | Yes | | bar | Yes | | baz | Yes | | lorem | Yes | | dog | No| For the first time our expectations were met! Enduring theory: List items own their trailing blank lines unless they are both the final list item and not the final element of a non-final list item. | Plain List *Item* | Owns trailing blank lines | Why | |-+---+--| | cat (includes foo -> lorem) | Yes | Not the final list item | | foo (includes bar baz) | Yes | Not the final list item | | bar | Yes | Not the final list item | | baz | Yes | Final element of non-final list item | | lorem | Yes | Final element of non-final list item | | dog | No| Final list item and not contained in a list item | -- Tom Alexander
Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior
On Wed, Aug 16, 2023 at 2:09 AM Ihor Radchenko wrote: > > Tom Gillespie writes: > > > Subject: [PATCH] ob-tangle.el: restore :tangle closure evaluation before > > eval > > info > > This patch fixes a bug where header arguments like :tangle (or "no") > > were treated as if they were tangling to a file named "(or \"no\")". > > As a result, org-bable would call org-babel-get-src-block-info with > > 'no-eval set to nil, causing parameters to be evaluated despite the > > fact that when :tangle no or equivalent is set, the other parameters > > should never be evaluated. > > What do you mean by "restore"? Were it evaluated in the past? > May you please provide a reproducer? Hrm. I think I may have mixed two commit lines. It is the case that :tangle closures used to work, but you are right, the historical behavior when tangling closures meant that all parameters were evaluated (tested with the block below in 27 and 28). #+begin_src elisp :var value=(error "oops") :tangle (or "no") value #+end_src My use case is that I have blocks that I want to tangle that set :var from e.g. the library of babel, which isn't always loaded, but which also is not required if :tangle is no. > > -(defun org-babel-tangle--unbracketed-link (params) > > +(defun org-babel-tangle--unbracketed-link (params &optional > > info-was-evaled) > > This is not acceptable. Taking care about evaluating INFO should be done > in a single place instead of adding checks across the babel code. If we > go the proposed way, I expect a number of bugs appearing when somebody > forgets to change the eval check in some place. I don't like the solution either. I see two potential alternatives. 1. change the structure of the info list to indicate whether it has already been evaluated 2. always call org-babel-read on (cdr (assq :tangle params)) even if it may already have been evaluated which can lead to some unexpected and potentially nasty results. I don't think we can consolidate evaluating parameters into one place in the general case because there are order dependencies where a setting in one param header should mask others (as is the case here). In principle we could consolidate them, but I think that would add significant complexity because we would have to push all the logic for handling whether a given ordering restriction applies inside that location. e.g. if I have a block set :eval (if ev "yes" "no") it would be bad form to evaluate the parameters before determining whether the :eval closure evaluates to "yes" or "no". Should that go inside org-process-params, or should it be handled locally by e.g. org-babel-tangle and org-babel-execute-src-block separately? Thoughts?
Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior
> My confusion about you patch comes from the fact that > > #+begin_src emacs-lisp :tangle (if (= 1 1) "yes") > 2 > #+end_src > > works just fine on main. It appears to work fine on main, but that is because what is actually happening behind the scenes is that in the test (unless (or (string= src-tfile "no") ...) ...) the actual comparison is (string= "(if (= 1 1) \"yes\")" "no") which appears to work, but is not comparing the result of the closure, only its string value. > I admit that I don't fully understand your use case. I want to use a closure to conditionally control whether a block will tangle. If I hardcode :tangle no, then :var x=(error "oops") will not evaluate. The (error "oops") is a placeholder for a number of things that will result in an error if the condition for :tangle (when condition "file-name") is not satisfied. > Something like (org-babel-get-heading-arg :tangle info/params) I need to go to bed, because I definitely started on an implementation of that I forgot about it as a potential solution. Yes, this seems like a better and clearer way to go about it. > May you please elaborate? Disregard, your suggestion clarified what you meant, and in that case, yes we can consolidate.
Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute
> It was a slip when the patch was applied. > See the table of :results params vs. expected output that Nicolas > provided in > https://list.orgmode.org/orgmode/87zjbqrapy@nicolasgoaziou.fr/ > > Or maybe I miss something. > > May you please explain more about {{{results(=value=)}}} problem? > Isn't it sufficient to do src_elisp[:results verbatim]{'value} > {{{results(=value=)}}}? The issue is the opposite I think. Currently the default value (i.e. absent) for :results does not produce {{{results(value)}}} as suggested, and instead producers {{{results(=value=)}}}. This means that without :results drawer there isn't an obvious way to get {{{results(value)}}} because you can't e.g. use [:results default], or if a user overrides the default value for inline header args at file level then they have no way to reset to the default. It looks like there used to be an option [:results wrap] which was deprecated a _very_ long time ago. [:results drawer] replaced that, and while there is some confusion about the name (because there is no actual drawer in an inline result) the behavior was meant to replace the old :results wrap behavior where the name does make sense since {{{results(value)}}} do "wrap" the value. I think that covers it, but let me know if something doesn't make sense. Best, Tom
Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute
Confirming fixed. Thanks! PS A new issue arises however caused by 487f39efa68fa2d857f8d446d1c4b3a3b3e3f482, which is that it is now confusing to get the {{{results(=value=)}}} macro without verbatim which is what :results drawer meant in that context. I expect that change will break things for a number of people beyond myself. A bit of reading the code revealed that setting :wrap t makes it possible to get {{{results(value)}}} again, but line 2647 [1] seems to indicate that drawer is an expected and valid value for inline :results. 1. ((or (member "drawer" result-params) ;; Stay backward compatible with <7.9.2 (member "wrap" result-params)) (goto-char beg) (when (org-at-table-p) (org-cycle)) (funcall wrap ":results:" ":end:" 'no-escape nil "{{{results(" ")}}}"))
Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute
I have tested this again against bugfix, the issue is still present when the cursor is on the s in #+caption: src_. Best, Tom
Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior
Here is a corrected patch that fixes the fact that closures passed to the :tangle header were not being evaluated. Details in the commit message. Best, Tom On Tue, Aug 15, 2023 at 6:41 PM Tom Gillespie wrote: > > After a bit more investigation don't apply this patch because the change > is insufficient to correct another issue. > > Specifically org-babel-tangle-collect-blocks must check for and resolve > any closures that are passed to :tangle _before_ testing (string= > src-tfile "no"). > As it stands blocks that are marked :tangle (and "no") with a closure > incorrectly make it to a call to (org-babel-get-src-block-info) which causes > a call to org-babel-process-params when :tangle would be "no". > > Working on a proper fix now. From ce56fe5c1a24ba37c5c7c2f541c48684b0cb1e0b Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Tue, 15 Aug 2023 13:46:08 -0700 Subject: [PATCH] ob-tangle.el: restore :tangle closure evaluation before eval info * lisp/ob-tangle.el (org-babel-tangle): check that :tangle is not no before attempting to tangle a single block (org-babel-tangle--unbracketed-link): new optional argument info-was-evaled to control whether (cdr (assq :tangle params)) is expanded via `org-babel-read' (org-babel-tangle-collect-blocks): expand closures passed to :tangle with `org-babel-read' so that blocks with :tangle closure that eval to "no" will not incorrectly eval other parameters (org-babel-tangle-single-block): src-tfile handle cases where output of :tangle closure could be nil and conver to "no" This patch fixes a bug where header arguments like :tangle (or "no") were treated as if they were tangling to a file named "(or \"no\")". As a result, org-bable would call org-babel-get-src-block-info with 'no-eval set to nil, causing parameters to be evaluated despite the fact that when :tangle no or equivalent is set, the other parameters should never be evaluated. This patch also restores the original behavior of the :tangle header argument when a closure returned nil, e.g. #+header: :tangle (or), by using (or (cdr (assq :tangle params)) "no"). Without this the call to `file-name-directory' in `org-babel-tangle--unbracketed-link' can fail with a wrong-type-argument stringp nil error. --- lisp/ob-tangle.el | 32 +++- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el index 4566e03ad..a9fb0e286 100644 --- a/lisp/ob-tangle.el +++ b/lisp/ob-tangle.el @@ -316,9 +316,13 @@ matching a regular expression." (write-region nil nil file-name) (mapc (lambda (mode) (set-file-modes file-name mode)) modes)) (push file-name path-collector)) - (if (equal arg '(4)) - (org-babel-tangle-single-block 1 t) - (org-babel-tangle-collect-blocks lang-re tangle-file))) + (if (equal arg '(4)) + (let* ((info (org-babel-get-src-block-info 'no-eval)) +(params (nth 2 info)) +(src-tfile (or (org-babel-read (cdr (assq :tangle params))) "no"))) + (unless (string= src-tfile "no") + (org-babel-tangle-single-block 1 t))) + (org-babel-tangle-collect-blocks lang-re tangle-file))) (message "Tangled %d code block%s from %s" block-counter (if (= block-counter 1) "" "s") (file-name-nondirectory @@ -473,14 +477,14 @@ code blocks by target file." (org-in-archived-heading-p)) (let* ((info (org-babel-get-src-block-info 'no-eval)) (src-lang (nth 0 info)) - (src-tfile (cdr (assq :tangle (nth 2 info) + (src-tfile (or (org-babel-read (cdr (assq :tangle (nth 2 info "no"))) (unless (or (string= src-tfile "no") (and tangle-file (not (equal tangle-file src-tfile))) (and lang-re (not (string-match-p lang-re src-lang ;; Add the spec for this block to blocks under its tangled ;; file name. (let* ((block (org-babel-tangle-single-block counter)) - (src-tfile (cdr (assq :tangle (nth 4 block + (src-tfile (or (cdr (assq :tangle (nth 4 block))) "no")) (file-name (org-babel-effective-tangled-filename buffer-fn src-lang src-tfile)) (by-fn (assoc file-name blocks))) @@ -490,7 +494,7 @@ code blocks by target file." (mapcar (lambda (b) (cons (car b) (nreverse (cdr b (nreverse blocks -(defun org-babel-tangle--unbracketed-link (params) +(defun org-babel-tangle--unbracketed-link (params &optional info-was-evaled) "Get a raw link to the src block at point, without brackets. The PARAMS are the 3rd element of the info for the same src block." @@ -510,7 +514,10 @@ The PA
Re: Export snippet matching without closing "@@"
> Thanks for reporting! > Fixed, on main. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e340dde63 Thanks for fixing! -- Tom Alexander
Re: [PATCH] ob-tangle.el: restore :tangle closure nil behavior
After a bit more investigation don't apply this patch because the change is insufficient to correct another issue. Specifically org-babel-tangle-collect-blocks must check for and resolve any closures that are passed to :tangle _before_ testing (string= src-tfile "no"). As it stands blocks that are marked :tangle (and "no") with a closure incorrectly make it to a call to (org-babel-get-src-block-info) which causes a call to org-babel-process-params when :tangle would be "no". Working on a proper fix now.
[PATCH] ob-tangle.el: restore :tangle closure nil behavior
Hi, Here's a patch to fix the :tangle header behavior when it is passed a closure that returns nil. Best, Tom From f1e15e0634fffed4648aa11628a14e0a68c3b18d Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Tue, 15 Aug 2023 13:46:08 -0700 Subject: [PATCH] ob-tangle.el: restore :tangle closure nil behavior * lisp/ob-tangle.el (org-babel-tangle-collect-blocks): * lisp/ob-tangle.el (org-babel-tangle--unbracketed-link): * lisp/ob-tangle.el (org-babel-tangle-single-block): src-tfile fix case where (cdr (assq :tangle params)) could be nil This patch restores the original behavior of the :tangle header argument when a closure returned nil, e.g. #+header: :tangle (or), by using (or (cdr (assq :tangle params)) "no"). Without this the call to `file-name-directory' in `org-babel-tangle--unbracketed-link' can fail with a wrong-type-argument stringp nil error. --- lisp/ob-tangle.el | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el index 4566e03ad..0df649d7e 100644 --- a/lisp/ob-tangle.el +++ b/lisp/ob-tangle.el @@ -473,14 +473,14 @@ code blocks by target file." (org-in-archived-heading-p)) (let* ((info (org-babel-get-src-block-info 'no-eval)) (src-lang (nth 0 info)) - (src-tfile (cdr (assq :tangle (nth 2 info) + (src-tfile (or (cdr (assq :tangle (nth 2 info))) "no"))) (unless (or (string= src-tfile "no") (and tangle-file (not (equal tangle-file src-tfile))) (and lang-re (not (string-match-p lang-re src-lang ;; Add the spec for this block to blocks under its tangled ;; file name. (let* ((block (org-babel-tangle-single-block counter)) - (src-tfile (cdr (assq :tangle (nth 4 block + (src-tfile (or (cdr (assq :tangle (nth 4 block))) "no")) (file-name (org-babel-effective-tangled-filename buffer-fn src-lang src-tfile)) (by-fn (assoc file-name blocks))) @@ -510,7 +510,7 @@ The PARAMS are the 3rd element of the info for the same src block." (concat "file:" (file-relative-name (substring bare (match-end 0)) (file-name-directory - (cdr (assq :tangle params) + (or (cdr (assq :tangle params)) "no" bare)) (defvar org-outline-regexp) ; defined in lisp/org.el @@ -580,7 +580,7 @@ non-nil, return the full association list to be used by (match-end 0) (point-min (point) - (src-tfile (cdr (assq :tangle params))) + (src-tfile (or (cdr (assq :tangle params)) "no")) (result (list start-line (if org-babel-tangle-use-relative-file-links -- 2.41.0
Re: [POLL] Should we accept breaking changes to get rid of Org libraries that perform side effects when loading?
> Any other ideas? I'd be happy to see some brain-storming. Maybe a #+keyword[options]: that doubles as a dynamic block or something like that? #+inline_task[options]: TODO some tiny task #+inline_task[options]: TODO some small task DEADLINE: <2023-11-11> :PROPERTIES: :SOMETHING: or other :END: #+inline_task_end: Migration path should be straight forward, the exact implementation of the behavior might need a bit of work, and I'm not sure that the scheduling line will work in that context, it is too fart outside the usual behavior for keywords. However it might be possible to move the deadline into [options] the syntax would be a bit different than regular scheduling lines, but it would at least be consistent with keyword syntax. The other question is whether you actually need an #+inline_task_end: or whether there is another way. The idea could use a few more iterations.
Export snippet matching without closing "@@"
I suspect this is a regression. I am using this test document: ``` @@latex: \documentclass[margin,11pt]{res} % default is 10 pt @@ ``` and dumped the AST with M-x eval-expression (message "%s" (pp-to-string (org-element-parse-buffer))) When using the latest in main (commit b89bc55867d7cb809c379d371d12d409db785154, Org mode version 9.7-pre (release_N/A-N/A-b89bc5 @ /usr/share/emacs/site-lisp/org/)) the "@@latex:" gets parsed as an export snippet despite not having a closing "@@" (because the "@@" is in a different paragraph). When using the version of org-mode that shipped with emacs 29.1 (Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/)) the "@@latex:" gets parsed as plaintext. The docs seem to show that a "@@" is required as the end of an export snippet based on the pattern "@@BACKEND:VALUE@@": https://orgmode.org/worg/org-syntax.html#Export_Snippets AST from org-mode in main: ``` (org-data (:standard-properties [1 71 1 71 0 1 nil nil nil 3 71 org-data nil nil nil nil nil #] :path nil :CATEGORY nil) (section (:standard-properties [1 71 1 71 0 1 nil nil nil 1 71 first-section nil nil #0 nil nil #]) (paragraph (:standard-properties [1 13 1 10 3 1 nil nil nil nil nil top-comment nil nil #1 nil nil #]) (export-snippet (:standard-properties [1 9 nil nil 0 nil nil nil nil nil nil nil nil nil #2 nil nil #] :back-end "latex" :value nil)) #("\n" 0 1 (:parent #2))) (paragraph (:standard-properties [13 68 13 65 3 13 nil nil nil nil nil nil nil nil #1 nil nil #]) (latex-fragment (:standard-properties [13 46 nil nil 1 nil nil nil nil nil nil nil nil nil #2 nil nil #] :value "\\documentclass[margin,11pt]{res}")) #("% default is 10 pt\n" 0 19 (:parent #2))) (paragraph (:standard-properties [68 71 68 71 0 68 nil nil nil nil nil nil nil nil #1 nil nil #]) #("@@\n" 0 3 (:parent #2) ``` AST from org-mode in emacs 29.1: ``` (org-data (:begin 1 :contents-begin 1 :contents-end 71 :end 71 :robust-begin 3 :robust-end 69 :post-blank 0 :post-affiliated 1 :path nil :mode org-data :CATEGORY nil :granularity nil) (section (:begin 1 :end 71 :contents-begin 1 :contents-end 71 :robust-begin 1 :robust-end 69 :post-blank 0 :post-affiliated 1 :mode first-section :granularity nil :parent #0) (paragraph (:begin 1 :end 13 :contents-begin 1 :contents-end 10 :post-blank 3 :post-affiliated 1 :mode top-comment :granularity nil :parent #1) #("@@latex:\n" 0 9 (:parent #2))) (paragraph (:begin 13 :end 68 :contents-begin 13 :contents-end 65 :post-blank 3 :post-affiliated 13 :mode nil :granularity nil :parent #1) (latex-fragment (:value "\\documentclass[margin,11pt]{res}" :begin 13 :end 46 :post-blank 1 :parent #2)) #("% default is 10 pt\n" 0 19 (:parent #2))) (paragraph (:begin 68 :end 71 :contents-begin 68 :contents-end 71 :post-blank 0 :post-affiliated 68 :mode nil :granularity nil :parent #1) #("@@\n" 0 3 (:parent #2) ``` -- Tom Alexander
Re: [BUG] inline src blocks in caption of not-inline src blocks do not execute
> org-element-at-point is not supposed to return objects. So, there is no > problem here. Right. My original issue description is wrong for this part. It seems that org-babel-get-src-block-info returns nil when immediately before the start of the ala |src_ in #+caption: but will return non-nil in other contexts. In the #+caption: it will only work if the cursor is past the s ala s|rc_. Or if in non evil mode when the cursor is on the s in src_. I can workaround the issue for now with forward-char, but it seems there is an off-by-one error involving #+caption: followed immediately by src_, if there is anything in between #+caption: and src_ then it works correctly. > Also, C-c C-c works for me on bugfix in the above example. Hrm, I can't get it to work, but I haven't done a proper control that loads only bugfix and nothing else. Will report back if I get it to work.
[BUG] inline src blocks in caption of not-inline src blocks do not execute
I've found a bug where inline source blocks cannot be executed and org-element cannot find/parse them if they are in #+captions: where the caption is affiliated with a not-inline source block. Example below. Best, Tom Issue present in: Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/) GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.17.8) of 2023-08-0 This inline src block works with C-c C-c. #+caption: src_elisp[:results drawer]{"All good!"} {{{results(All good!)}}} [[file:./images/happy-figure.png]] This one fails to work with C-c C-c, and org-element-at-point returns the bash src block below. #+caption: src_elisp[:results drawer]{"I've made a horrible mistake"} #+begin_src bash echo OH NO #+end_src
Re: [POLL] Should we accept breaking changes to get rid of Org libraries that perform side effects when loading?
> Same here, I'd be tempted to deny Org citizenship to inline tasks: it > always felt like a nice hack for a niche use-case, but a hack anyway. > > If it modifies Org syntax in surprising ways, this is another argument > for removing org-inlinetask.el from Org's core. Remember: this is not > to say that inline tasks are forbidden, it's just a message for users > that inline tasks are something not maintained by Org's core team. > > > And it is not clear how to fix this. We did not make inlinetasks into > > standard Org syntax in the past and now it is in the weird state when we > > have (featurep 'org-inlinetask) sprinkled across the code just to > > accommodate for this conditional syntax. > > Yes, this is ugly. > > > Inlinetasks are too similar in syntax with headlines, so it is > > impossible to make the change backwards-compatible. Chiming in here to say that inline tasks make it exceptionally annoying to specify a sane grammar for org because they require putting a special case in the headline tokenizer, and/or making it possible to toggle the behavior of the tokenizer. As such I strongly support removing them from any official status in the name of simplifying the syntax (among other things).
Re: Seemingly inconsistent whitespace ownership
On August 11, 2023 5:20:08 AM EDT, Ihor Radchenko wrote: >"Tom Alexander" writes: > >> The org-mode documentation states: "With the exception of list items, blank >> lines belong to the preceding element with the narrowest possible scope. For >> example, if at the end of a section we have a paragraph and a blank line, >> that blank line is considered part of the paragraph." >> >> https://orgmode.org/worg/org-syntax.html#org9bc833d >> >> So I created a test org-mode document with exactly that scenario described: >> ... >> and then I ran a M-x eval-expression (message "%s" (pp-to-string >> (org-element-parse-buffer))) to see the parse tree. I'll paste the output at >> the end because its a wall of text, but analyzing it: >> ... >> Its only child is a paragraph which has a :begin at 1 and :end at 10, which >> only includes the line break at the end of "foo bar." but this is exactly >> the scenario described in the documentation (a paragraph at the end of a >> section) so I would expect the paragraph to end at 14, not 10. > >You are right and you revealed an existing inconsistency between our >incremental and recursive parsers (org-element-parse-buffer is the >recursive parser). > >Fixed, on main. >https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bfe1179dc > >Also, in addition to list items, footnote-definitions do not extend >their contents to the trailing blank lines. >I now clarified it in org-syntax document. >https://git.sr.ht/~bzg/worg/commit/3e91f5be > >-- >Ihor Radchenko // yantar92, >Org mode contributor, >Learn more about Org mode at <https://orgmode.org/>. >Support Org development at <https://liberapay.com/org-mode>, >or support my work at <https://liberapay.com/yantar92> Oh neat, I didn't know there were two! I'll check that out. Thanks!
Seemingly inconsistent whitespace ownership
The org-mode documentation states: "With the exception of list items, blank lines belong to the preceding element with the narrowest possible scope. For example, if at the end of a section we have a paragraph and a blank line, that blank line is considered part of the paragraph." https://orgmode.org/worg/org-syntax.html#org9bc833d So I created a test org-mode document with exactly that scenario described: ``` foo bar. * Lorem baz * Ipsum alpha beta ``` and then I ran a M-x eval-expression (message "%s" (pp-to-string (org-element-parse-buffer))) to see the parse tree. I'll paste the output at the end because its a wall of text, but analyzing it: The zeroth section has :begin at 1 and :end at 14, which is expected. Its only child is a paragraph which has a :begin at 1 and :end at 10, which only includes the line break at the end of "foo bar." but this is exactly the scenario described in the documentation (a paragraph at the end of a section) so I would expect the paragraph to end at 14, not 10. The "Lorem" section is repeating the test to prove this isn't behavior unique to the zeroth section. It behaved the same, so there is no need to go over it. The "Ipsum" section is where I put two separate paragraphs with a bunch of blank lines in between. This time the "alpha" paragraph goes all the way to character 48 which means it includes ALL of the blank lines between "alpha" and "beta". So the "Ipsum" section proves that ":begin" to ":end" is supposed to contain the trailing blank lines of an element. I am using org-mode version: Org mode version 9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/) And finally, that parse tree I mentioned: ``` (org-data (:begin 1 :contents-begin 1 :contents-end 53 :end 53 :robust-begin 3 :robust-end 51 :post-blank 0 :post-affiliated 1 :path nil :mode org-data :CATEGORY nil :granularity nil) (section (:begin 1 :end 14 :contents-begin 1 :contents-end 10 :robust-begin 1 :robust-end 8 :post-blank 4 :post-affiliated 1 :mode first-section :granularity nil :parent #0) (paragraph (:begin 1 :end 10 :contents-begin 1 :contents-end 10 :post-blank 0 :post-affiliated 1 :mode top-comment :granularity nil :parent #1) #("foo bar.\n" 0 9 (:parent #2 (headline (:raw-value "Lorem" :begin 14 :end 30 :pre-blank 0 :contents-begin 22 :contents-end 26 :robust-begin nil :robust-end nil :level 1 :priority nil :tags nil :todo-keyword nil :todo-type nil :post-blank 4 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 14 :title (#("Lorem" 0 5 (:parent #1))) :mode nil :granularity nil :parent #0) (section (:begin 22 :end 30 :contents-begin 22 :contents-end 26 :robust-begin 22 :robust-end 24 :post-blank 4 :post-affiliated 22 :mode section :granularity nil :parent #1) (paragraph (:begin 22 :end 26 :contents-begin 22 :contents-end 26 :post-blank 0 :post-affiliated 22 :mode planning :granularity nil :parent #2) #("baz\n" 0 4 (:parent #3) (headline (:raw-value "Ipsum" :begin 30 :end 53 :pre-blank 0 :contents-begin 38 :contents-end 53 :robust-begin 40 :robust-end 51 :level 1 :priority nil :tags nil :todo-keyword nil :todo-type nil :post-blank 0 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 30 :title (#("Ipsum" 0 5 (:parent #1))) :mode nil :granularity nil :parent #0) (section (:begin 38 :end 53 :contents-begin 38 :contents-end 53 :robust-begin 38 :robust-end 51 :post-blank 0 :post-affiliated 38 :mode section :granularity nil :parent #1) (paragraph (:begin 38 :end 48 :contents-begin 38 :contents-end 44 :post-blank 4 :post-affiliated 38 :mode planning :granularity nil :parent #2) #("alpha\n" 0 6 (:parent #3))) (paragraph (:begin 48 :end 53 :contents-begin 48 :contents-end 53 :post-blank 0 :post-affiliated 48 :mode nil :granularity nil :parent #2) #("beta\n" 0 5 (:parent #3)) ``` -- Tom Alexander
Re: Documentation: Equal sign incorrectly listed as disallowed in plain links
Thanks for fixing all the documentation issues I submitted today! On Sat, Jul 22, 2023, at 2:58 AM, Ihor Radchenko wrote: > "Tom Alexander" writes: > >> The documentation for plain links states that PATHPLAIN is: >> >>> A string containing non-whitespace non-bracket (()[]<>=) characters, >>> optionally containing parenthesis-wrapped non-whitespace non-bracket >>> substrings up to a depth of two. The string must end with either a >>> non-punctation non-whitespace character, a forwards slash, or a >>> parenthesis-wrapped substring. >> >> but if we create an org-mode document with the following test content: >> ``` >> http://foo/bar=baz > > As you can see from the sentence, PATHPLAIN must be "non-whitespace" > "non-bracket". "=" is just a messed up verbatim markup marker. > > Fixed, on master. > https://git.sr.ht/~bzg/worg/commit/85ef10ed > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92>
Documentation: incorrect character name in line breaks
The documentation for line breaks says that PRE is: > Anything but backspace (`\`). I suspect this is supposed to say "backslash" (ascii 92) not "backspace" (ascii 8) Link to the documentation: https://orgmode.org/worg/org-syntax.html#Line_Breaks
Documentation: Equal sign incorrectly listed as disallowed in plain links
The documentation for plain links states that PATHPLAIN is: > A string containing non-whitespace non-bracket (()[]<>=) characters, > optionally containing parenthesis-wrapped non-whitespace non-bracket > substrings up to a depth of two. The string must end with either a > non-punctation non-whitespace character, a forwards slash, or a > parenthesis-wrapped substring. but if we create an org-mode document with the following test content: ``` http://foo/bar=baz http://foo/bar[baz ``` and run a M-x eval-expression (message "%s" (pp-to-string (org-element-parse-buffer))) Then we can see that it allows the equals sign in the PATHPLAIN (output pasted below) Link to the documentation: https://orgmode.org/worg/org-syntax.html#org6934900 Syntax tree: ``` (org-data (:begin 1 :contents-begin 1 :contents-end 40 :end 40 :robust-begin 3 :robust-end 38 :post-blank 0 :post-affiliated 1 :path "/tmp/test/test.org" :mode org-data :CATEGORY "test" :granularity nil) (section (:begin 1 :end 40 :contents-begin 1 :contents-end 40 :robust-begin 1 :robust-end 38 :post-blank 0 :post-affiliated 1 :mode first-section :granularity nil :parent #0) (paragraph (:begin 1 :end 21 :contents-begin 1 :contents-end 20 :post-blank 1 :post-affiliated 1 :mode top-comment :granularity nil :parent #1) (link (:type "http" :path "//foo/bar=baz" :format plain :raw-link "http://foo/bar=baz"; :application nil :search-option nil :begin 1 :end 19 :contents-begin nil :contents-end nil :post-blank 0 :parent #2)) #("\n" 0 1 (:parent #2))) (paragraph (:begin 21 :end 40 :contents-begin 21 :contents-end 40 :post-blank 0 :post-affiliated 21 :mode nil :granularity nil :parent #1) (link (:type "http" :path "//foo/bar" :format plain :raw-link "http://foo/bar"; :application nil :search-option nil :begin 21 :end 35 :contents-begin nil :contents-end nil :post-blank 0 :parent #2)) #("[baz\n" 0 5 (:parent #2) ```
Documentation: Reference to BODY field that is not specified in the grammar
The documentation for Inline Babel Calls mentions that opening and closing square brackets must be balanced within BODY but the grammar above it only defines NAME, ARGUMENTS, HEADER1, and HEADER2. Link to the documentation: https://orgmode.org/worg/org-syntax.html#Inline_Babel_Calls
Documentation: Reference to "KEYCITES" field that is not specified in grammar
The documentation for Citations when specifying which syntactic elements have spaces between them references a "KEYCITES" syntactic element that does not appear anywhere else on the page. Link to the documentation issue: https://orgmode.org/worg/org-syntax.html#Citations
Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)
> We might probably generalize to > PRE = Zs Zl Pc Pd Ps Pi ' " > POST = Zs Zl Pc Pd Pe Pf . ; : ! ? ' " \ [ If this works I think it is reasonable. We might want to specify what to do in cases where an org implementation might not fully support unicode, and might want to do a review of related issues in syntax with respect to ascii vs unicode, because iirc there is some ambiguity in the current syntax doc. For example, I'm pretty sure that I'm mixing and matching unicode and ascii whitespace in the tokenizer I have in Racket. > Though we need to take care excluding zero-width spaces. Ya, I removed a comment to this effect in the paragraph about the usual alternate solution. > Emacs does not support them though (yet?). Racket has full support for the latest unicode standards iirc, so I will see if I can leverage that support for testing in laundry. > At the end, it is the current ASCII limitation plus partially arbitrary > choice of boundaries that keep some users confused (we are getting bug > reports about confusing markup from time to time). Ya, it would be good to try to generalize the affordance if possible since users of text in non-ascii languages have certain valid expectations. Hopefully, the unicode consortium has managed to cover the categories we need.
Re: Org markup and non-ASCII punctuation (was: org parser and priorities of inline elements)
Hi Ihor, Thank you for looping me in. Best, Tom The way I have implemented this is by maintaining an explicit list of characters that are safe for pre markup and another for post markup. It is not possible to use unicode punctuation for this because there are a variety of punctuation marks that cannot appear in that position and be considered markup, those include @, #, % to name just a few. Therefore, if we want to do this we commit to extending and then maintaining the lists of valid pre and post markup delimiters as special cases. Note also this could produce changes from current behavior because things that previously tokenized as a series of words connected by e.g. underscores could become markup. The alternative would be (as usual in these cases) for the user to add a zero width space or something like that between the end of the markup marker and the symbol they want to follow the markup. This solution is (trivially) backward compatible, and works for all chars regardless of whether org-mode has blessed them as sanctioned marks. My inclination would be not to make this change because there are a potentially infinite number of future "left right neutral" marks that we would have to maintain and would occasionally have to field requests from users to add them, and those solutions would not work with older versions of org.
[BUG] Contents of the *Warnings* buffer, which has begun popping up at random intervals
I was prompted by a buffer named *Warnings* to send an email to emacs-orgmode@gnu.org<mailto:emacs-orgmode@gnu.org>. This buffer appears seemingly at random every few days or so (I use Emacs and Org mode for about eight hours a day). The warnings began after I upgraded to Org 9.6.3. Find the contents of that buffer below. Thank you for the work on Org mode, which is magnificent and without peer. Tom Davey ■ Warning (org-element-cache): org-element--cache: Org parser error in journal.org::18359990. Resetting. The error was: (error "Invalid search bound (wrong side of point)") Emacs : GNU Emacs 29.0.50 (build 1, x86_64-w64-mingw32) of 2022-09-14 Package: Org mode version 9.6.3 ( @ c:/Users/tdavey/Documents/Dropbox/Emacs/Prod/.emacs.d/load-path-subdirs/td-esdf/td-startup-lisp/Elpa/org-9.6.3/) current state: == (setq org-archive-location "%s_archive::datetree/" org-link-elisp-confirm-function 'yes-or-no-p org-directory "~/org-files" org-yank-adjusted-subtrees t org-tag-faces '(("PROJ" :background "indianred3" :foreground "cornsilk2" :weight bold)) org-agenda-custom-commands '(("c" "Computers and Project Tags" ((agenda "") (tags-todo "project") (tags-todo "orgmode") (tags-todo "emacs") (tags-todo "computers")) ) ("e" "Email tag" ((tags-todo "email") (agenda ""))) ("g" "Microscheduling tag" ((tags-todo "microscheduling") (tags-todo "today") (agenda ""))) ("h" "Housing tags" ((tags "+Strivers") (tags "+SGCagenda") (tags "+SGCbudget") (tags "+SGCcuo") (tags "+SGCdiscussion") (tags "+SGChouseRules") (tags "+SGCminutes") (tags "+SGCstaff") (tags "+SGCunit") (tags "+SGCvote") (tags "+improvement") (tags "+household")) ((org-columns-default-format "%CATEGORY %75ITEM %TODO %TAGS %25CREATED %25TIMESTAMP"))) ("n" "Agenda and all TODOs" ((agenda "") (alltodo ""))) ("p" "People tags" ((agenda "") (tags-todo "atrajkov") (tags-todo "apotter") (tags-todo "dmcewan") (tags-todo "espector") (tags-todo "jbousley") (tags-todo "kmandel") (tags-todo "rosoria") (tags-todo "twasson") (tags-todo "cfrappier") (tags-todo "cmorales") (tags-todo "cwolan") (tags-todo "dbolen") (tags-todo "jterranella") (tags-todo "lortiz") (tags-todo "malzuru") (tags-todo "dhalsey") (tags-todo "jkendall") (tags-todo "kolson")) ) ("b" "Tom's basic block agenda" ((agenda "") (tags-todo "+homeImprovement-household") (tags-todo "FAredesign") (tags-todo "rhaass") (tags-todo "kolson") (tags-todo "dhalsey") (tags-todo "hr") (tags-todo "lhammes") (tags-todo "StaffMeeting") (tags-todo "cfrappier") (tags-todo "espector") (tags-todo "ahenry") (tags-todo "atrajkov") (tags-todo "+work-rhaass-kolson-cfrappier-jmcgrath-tkatavic-egosselin-aqualls-cshehadi-espector-pituk-ahalliday-metchison-nfreiberg-lhammes-emacs-lifelongLearning") (tags-todo "computers") (tags-todo "desk") (tags-todo "errands") (tags-todo "household") (tags-todo "lifelongLearning") (tags-todo "reading") (tags-todo "social")) ) ) org-log-into-drawer t org-use-fast-tag-selection t org-agenda-files '("~/org-files/") org-capture-templates '(("a" "Appointments & Meetings:appt:
Re: bug#59882: Multiple versions of Org in load-path problem
> The state of our unloading support is not good enough to use it blindly I have managed to use unload-feature to reload a new version of org after the builtin version has been previously loaded, but the dance you have to do to avoid nasty errors (and segfaults if using native comp!) is about 100 lines of arcane footwork, and if any one step is off you will encounter one of the many hidden landmines. unload-feature cannot be used blindly, and there are a very large number of side effects along the path that are not controlled which makes any process that tries to use unload-feature brittle and hard to debug. https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1010-L1064 https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1073-L1177
Re: Inline markup: How does org identify nested code/verbatim?
In short, you cannot nest code in verbatim or verbatim in code because they are both terminal (end of the line for nesting). In fact you can't nest anything inside them by their very nature. Anything inside of them cannot have special functionality, even escape codes don't play well in that part of the grammar. There is no way around this because you cannot nest inside things that are by definition terminal. However, from your examples it seems that you can get the effect you are looking for using ~is~ =verbatim= ~in code~.
Re: Org mode timestamps on the Moon ;] (was: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda)
Oh boy. In short, I think we can only hope they come up with LTC and we already have a syntactic space to say whether our reference seconds are TAI/UTC/LTC/MTC/JTC etc. And being the privileged squats that we are if the time system is left out then it means UTC. The friendly thing to do would be to always include the time system code in our timestamps, but I don't think those have standard reference codes yet? > would gain about 56 microseconds over 24 hours And here I was naively hoping to avoid having to deal with relativistic effects. That would seem to be a pretty serious issue, but my assumption further down that comment is hopefully what they will do. Specifically: > let's just assume that clock synchonization will happen between earth and > mars so that the unix epoch could be synchronized between the frames The only generalized solution is to record the full location (see intro to http://naggum.no/lugm-time.html which I'm surprised hasn't been linked in this thread yet, but was linked on the same topic back in 2011 it seems) and let people sort out how to translate that local time from that reference frame in that particular gravity well. I don't think we want to force users to start embedding their gps coordinates, their current acceleration, etc. along with their timestamp. But it turns out that the unix epoch is only standard at 9.8m/s^2. So ... minutes should be ok right? Joking aside. Clocks run faster on the moon, slower on the ISS, so either clocks are synchronized back to earth seconds, or they use some alternate standard reference time. My expectation is that there will be some time keeping standard that will ensure uniformity and that what we will want to capture is whose seconds they are using to keep track of time, because unix epoch on the moon has a different integer value than on earth, and is likely off by multiple seconds. I guess as long as the moon can participate in earth NTP it will be ok? Otherwise I suspect that the NTP network standard for the celestial body will be what we want to go with/record. I think that will fit in the opening you have created for this in the syntax. So instead of UTC it would be LTC (lunar coordinated time). If you're getting logs from servers on the moon knowing that the system clock was running on LTC and not UTC will be critical. Some future engineer: "Leaving earth was a mistake."
[BUG] org-clock-in opens buffers for all agenda files
I just tried to clock in to a task for the first time in ... a while, and was immediately accosted with hundreds of buffers being opened. This seems to be because org-clock-in somehow calls org-agenda or org-agenda-prepare or something similar, however I cannot find the call site. Is there a way to disable this so that clocking does not destroy my emacs buffer state by starting org-agenda? Thanks! Tom
Re: [BUG] org-clock-in opens buffers for all agenda files
> This is because Org is checking if you have any dangling clocks present. Seems like a reasonable default. > See org-clock-auto-clock-resolution Perfect solution for my case. Thanks!
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
> Getting the rules and explanation clear is the issue. It's a mistake > that a great many people make with scheduling meetings. Those two > behaviors need different encodings because they behave differently. This is related to why I suggested splitting timezones and offsets into two separate categories. I think we have to assume that the written content of timestamps in an org file cannot/will-not be changed automatically. Therefore if the timezone is specified then the numbers will never change, but the actual time might if the timezone spec changes. If an offset is used then it will not account for changes due to DST, but it will always remain stable, shuffling a meeting an hour one way or the other at some points in the year, which is usually undesirable compared to say, shuffling a meeting 1 hour in one direction for people who are not in the defining timezone for the duration of the mismatch between DST changes in different regions.
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
> > As for years BC, <-0001-...> will be a breaking change. But I do not > > think that we need to really worry about this. Not unless we actually > > get feature request. What is the practical application? Using org as a format for writing about history and being able to reference dates in the past accurately and have the dates be first class entities that can be parsed and checked etc. The example in my head is a history professor who wants to write about e.g. the collapse of the Roman republic and not have to come up with their own time keeping system or force any one who wants to work with referenced dates to do a bunch of math to translate from a roman time system to a modern one. > Given that the stated approach is to leverage off OS facilities in this > area, it probably should also be noted that some OSs don't handle > historical dates, especially BC ones, at all well. For example, some OS > use a 32 bit number to represent the date+time and can really only > handle dates between approx 1900 and 2038 (or around there - cannot > remember specific range). So with respect to timestamps and time related > calculations, we are limited by the capabilities of the least capable > supported OS. I'm mostly concerned about the syntactic features where org already supports dates well outside the facilities of various operating systems. I don't think that it is wise or practical for org-mode code to use anything other than the os provided time keeping facilities right now, but it is important to enable people who might want to do so. Org as a format for documents has a wider range of use cases for dates and times than Org as a life organizer and planner. At the same time those wider use cases don't always need as much precision or ux considerations because I don't think anyone using org right now is going to be early or late to their meeting at [3023-01-16 Thu 12:00] (regardless of the timezone). But org does tell me that it will be a Thursday! Best, Tom
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
> I strongly disagree. I'd prefer to allow only internationally recognized > time zone format. Let's not make life harder for Org file parsers. So offsets and tz database names but no time zone abbreviations? That seems reasonable since there isn't a sane way to handle the timezone with dst vs abbreviation for an offset, so better to force only US/Central aka America/Chicago and then -06:00 and -05:00 if users want CST/CDT to avoid any ambiguity?
[PATCH] fix compat with emacs 25 due using temporary-file-directory
Small bugfix for emacs 25 compat. Best! Tom From f04a44e5cc8143fc6cdbbc92b9d2afb76de4a459 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Mon, 16 Jan 2023 00:25:47 -0500 Subject: [PATCH] fix compat with emacs 25 due using temporary-file-directory function * lisp/ob-core.el: org-babel-temporary-stable-directory use the symbol version of temporary-file-directory since the function form is not available on emacs 25. --- lisp/ob-core.el | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 5d5edadd2..255a767bb 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -3276,7 +3276,7 @@ Emacs shutdown.") (while (or (not dir) (file-exists-p dir)) (setq dir (expand-file-name (format "babel-stable-%d" (random 1000)) - (temporary-file-directory + temporary-file-directory))) (make-directory dir) dir)) "Directory to hold temporary files created to execute code blocks. -- 2.38.2
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
> So I guess the timestamp format and the code which manages them will > need the ability to use the full TZ name and not just the abbreviated > form (and I guess an option to allow the user to select). In fact, we > probably need a way to select between abbreviated/full dynamically as > well as you might use the different TZ types as a way to flag which > timestamps you want to adjust due to TZ changes (either in TZ db or in > user location) and those you don't want changed. I think the only sane way to do this is to require timezone abbreviations /expansions to be defined in the file itself and never in an init.el there is simply too much ambiguity and if the information is lost we are out of luck. Of course then the question comes whether we can use something like #+tzinfo: US/Central CT #+tzoff: CST -06:00 #+tzoff: CDT -05:00 I don't think it is wise to allow users to map #+tzinfo: US/Central CST because that will produce surprising and results when US/Central changes to DST. Ugh, what a mess.
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
> In anticipation to add time zones in future, I have added the following > to the Org timestamp spec (see > https://orgmode.org/worg/org-syntax.html#Timestamps): > > DATE TIME REPEATER-OR-DELAY > > TIME (optional) > An instance of the pattern H:MMREST where H represents a one to two digit > number (and can start with 0), and M represents a single digit. REST can > contain anything but \n or closing bracket. > > Note that REST imply that almost arbitrary suffix can be in TIME without > braking the existing Org timestamp parsing code. I'm not sure how I feel about the REST in the grammar, I think it is a reasonable approach but need to double check. I'm worried that there can be some nasty interactions with REPEATER-OR-DELAY syntax, but that may not actually be an issue. I will note that this doesn't address the issue of syntax for historical and future dates. For historical dates those almost always require significant additional metadata to compensate for things like the julian/gregorian calendar switchover etc. for future dates we may want to go ahead and specify something beyond -. I'm less concerned about the rest of the issues beyond the fact that adding syntactic support for timezones seems to have opened up countless usability and bad assumption issues that have been mentioned elsewhere in the thread. So, I think the syntax may be ok, but as written I think there can be quite nasty interactions with REPEATER-OR-DELAY (not to mention what happens if a delay tries to track across a change in timezones). Best, Tom
Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda
Without wading too far into this at the moment, timezones are an extremely tricky problem with a whole bunch of design considerations. I am reproducing the heading comment from laundry's timestamp.rkt in its entirety here. Best! Tom https://github.com/tgbugs/laundry/blob/master/laundry/grammar/timestamp.rkt ; the org spec is currently silent on how org will handle dates beyond ; and , the relevant standards for expanded representations ; are ISO 8601-1:2019,5.2.2.3 and ISO 8601-2:2019,8.4.3. The problem ; with expanded representations is that they require all parties to ; agree ahead of time how wide the year is and prefix with + and ; leading zeros, I am not sure that this is as robust as we would like ; -01-01 is january first of the year zero, -0001-01-01 is jan ; first of the year 1BCE, and +1-01-01 for jan 1 of 1CE 5 ; digits must be prefixed by the plus sign, the standard is not ; entirely clear, but I think that we could do PLUS digits HYPHEN ; or DIGIT-4 HYPHEN and everything would work out ; on a separate but related note, org timestamps are always in local ; time and do not currently support timezones, which is a problem ; further absurdities that are related to the timezone issue: the ; date/time implementation as a text format completely fails if ; authoring on another planet ; given my objective to ensure that org documents can be interpreted ; without having to stick stupid things like #+planet: mars in the ; header or risk your earthling readers getting incorrect dates --- I ; suggest that org switch to storing all dates and times in earth zulu ; time or with an offset as it stands, the timezone setting of the ; computer drafting the document must be known to determine what day ; it was etc. users could still draft using <2020-12-16> but C-c C-c ; would add the correct offset? actually this is tricky, because there ; are two cases, one where the location is clear, "napoleon on [1812-01-01]" ; and the other where it is not, the issue is that a single date refers ; to as many different time intervals as there are timezones and without ; knowing the timezone you don't know the date, similar issue if we were ; to try to convert to sols ; https://www.eecis.udel.edu/~mills/missions.html ; https://www.eecis.udel.edu/~mills/ipin.html ; https://en.wikipedia.org/wiki/Timekeeping_on_Mars ; HOWEVER having applied a bit more brain power to the problem, let's ; just assume that clock synchonization will happen between earth and ; mars so that the unix epoch could be synchronized between the frames ; probably with multiple reference stations in free fall and/or in deep ; space to account for gravitaitonal differences etc. so actually the ; only issue would be a practical notational one, or a UI one, since we ; can use timestamp with timezone or just straight epoc for this to work ; earth vs martian year/day is a much harder issue ; the org spec is currently silent on how org will handle dates beyond
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
> P.S. Considering intense discussion around the topic, what about > reverting my commit from the release? We can then re-consider the whole > design and apply something more elaborate later. I was actually going to mention that in my previous message but forgot. I think that given the potential for disruption and our desire to do a good job on the design of the new system I think it would make sense to revert the commit from the release so that we don't rush in trying to fix something under pressure without taking the time we need to consider all the angles. Best! Tom
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Hi Bastien, In short, we need two variables due to the case where a user wants to set nil for all vars and a function for code. More replies in line. Best! Tom > I see nothing in etc/ORG-NEWS that describes this change: am I missing > something? Looks like any mention of the change is missing to me as well. > >> Whether it changed or not, what is the problem in 9.6? > > > > The problem is that Org now displays more queries. > > > >> How does the patch solves this problem? > > > > It allows disabling these new queries about lisp evaluation outside > > code blocks without disabling code block eval confirmation completely. > Is there a real use-case for this? Yes. To give one example. I have many files that need to run hundreds of cells at tangling/export time. I have reviewed all the code in the cells and know patterns that are safe and wont' burn me. I do not want to allow execution of code blocks blindly during tangling and confirm eval is a secondary line of defense against running those blocks during export. I want to allow the cells to run uninhibited as they did before this change, so that I don't have to fight with org during the tangling process. As it is right now the change has completely broken various CI pipelines for me, and if these changes do not get in to emacs 29 in time I will be unable to use emacs 29 for CI until they are in the base image without having to resort to insane hacks and massive additional configuration changes to work around the issues discussed in the loading multiple org versions thread. > It looks like the patch fixes the "too many queries" problem by > providing a setup that is similar to what was the previous default > (i.e. only asking about code block evaluation when > org-confirm-babel-evaluate-cell is set to nil.) This interpretation is not quite right. org-confirm-babel-evaluate-cell has no interaction with evaluating code blocks, only with evaluating cells. When org-confirm-babel-evaluate-cell is nil or evaluates to nil then the org-babel-confirm-evaluate code runs on a fake block that is created out of the cell. (This is where there is still a bug that I mentioned up thread.) > But are we sure that users need to confirm header args evaluation > separately from code blocks evaluation? I am sure that in my workflow I want them split, especially for global nil/t behavior. I need to think a bit more about your suggestion to add more values to org-confirm-babel-evaluate to see whether it might work and what the relative complexity would be. It seems to me that duplicating the variable is certainly easier to maintain, if not to explain to those who do not use babel regularly. I would be interested to hear from other babel users their thoughts on the design, because it seems obvious to me, but again I wrote the code so am the last person whose view on the clarity of the should be considered. It seems that it was not clear to Max initially. > IOW I have difficulties understanding when these would be relevant: > (setq org-confirm-babel-evaluate-cell nil) Anywhere that a parenthesized elisp expression occurs that is not in #+begin_src block. > (setq org-confirm-babel-evaluate t) Only inside #+begin_src blocks. > I think that's the best route: providing, optionnally, more, while not > annoying users who don't want more. I think this is probably the right default as well. > > Yes. Ideally, we need to improve the code evaluation query. It should > > allow confirming evaluation in bulk and add some code blocks/files to > > whitelist. Similar to `org--confirm-resource-safe'. > > I see, indeed. Improving general code auditing prior to evaluation of blocks would be greatly appreciated. The fighting between the minibuffer and the primary buffer for priority and the fact that you often cannot see that code that will be run is a significant issue. However, this is orthogonal to the issue at hand. > > A concern have been expressed that more queries may annoy users and > > drive them towards setting `org-confirm-babel-evaluate' to nil globally. > > Upon doing this, the future more flexible security queries may be not > > used by such users. > > The future more flexible security queries will be made visibile enough > so that users currently setting `org-confirm-babel-evaluate' to nil > will *want* to have a look at it. Yep. We need to get the fundamentals in place to enable those workflows. I'm too paranoid to simply set that variable to nil, even when I'm only running code that I wrote, but I suppose that is not the case for everyone. > I find "cell" confusing here (as Max said earlier in the thread). It > either refers to a table cell or, in Elisp jargon, a "cons cell" (or a > "function cell"). I also dislik
[PATCH] Fix missing null check for tangle links during export
Hi, Here is a patch that fixes broken tangling during export. I don't have a simple reproduction, but was able to run git bisect to find the originating commit. The commit text explains the issue in detail. Best, Tom From 0b4e35453874db0acf046e46c571cdef5f39f59a Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Mon, 19 Dec 2022 21:25:02 -0500 Subject: [PATCH] Fix missing null check for tangle links during export * lisp/ob-tangle.el (org-babel-tangle--unbracketed-link): Add the let bound variable l to the and statement to prevent l from being passed to `string-match' in the event that it is nil. When tangling nested noweb blocks during export it is possible for the results of `org-store-link' to return nil. This commit ensures that the value returned for l is only passed to `string-match' when it is non-nil, avoiding a `wrong-type-argument' (stringp nil) error. Handling of comments and nesting of babel blocks is known to have issues. The bug is from 8a781d35dc68f20fa2a5546c98ba3d9b77ee3cda where new code was introduced to obtain the value for bare which was not present in early code. I'm guessing that the bug appears now because `string-match' is called on l (aka link) at a point in time when it was never previously called and it was thus masked because in the old version it was impossible to call `string-match' when `link' was nil because another variable was always nil, masking the issue. --- lisp/ob-tangle.el | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el index bd17bda32..fd6b6f3b9 100644 --- a/lisp/ob-tangle.el +++ b/lisp/ob-tangle.el @@ -500,7 +500,8 @@ The PARAMS are the 3rd element of the info for the same src block." (cl-letf (((symbol-function 'org-store-link-functions) (lambda () nil))) (org-store-link nil - (bare (and (string-match org-link-bracket-re l) + (bare (and l +(string-match org-link-bracket-re l) (match-string 1 l (when bare (if (and org-babel-tangle-use-relative-file-links -- 2.38.2
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
One more correction. The source of the issue is that the two values in the list need to be different, one for the message and one for the actual test. Best, Tom (list "emacs-lisp" cell '((:eval . yes)) nil (format "%S" cell) nil nil)
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
By the way, while we're on bugfixes. I just noticed that (format "%S" cell) is being used to create the fake body for info. This is incorrect because cell is already a string at this point and this makes the behavior inconsistent with the rest of org babel as a result. Fix below. Best, Tom The fix is to pass (list "emacs-lisp" cell '((:eval . yes)) nil cell nil nil)
Re: [Syntax discussion] Should we treat src blocks without LANG as paragraphs? (was: [BUG] ox-html does not export captions of source blocks without language)
Hi Ihor, Chiming in here with a slight variant on what others have said. Best! Tom I don't think this should be handled at the syntactic layer at all. The empty string block language should be syntactically valid with any special behavior needed being handled later. Linters could treat it as a warning/error though, but the parsing is made significantly easier if the empty string is present allowing the grammar to be fully closed and regular. Thus, I don't think we need to make this a syntactic error or pun a src block without a lang to another type. I think we can add an implementation for when the block language is the empty string. This keeps the grammar regular by removing a special case. I assume that internally the empty string block lang would mostly call the example block codepaths, except that it should probably issue a warning or fail if someone tries to org-src-edit the block so that we can alert them that they are missing the lang. Treating src blocks missing a lang as paragraphs is incorrect because according to the syntax spec they are syntactically still blocks (greater or lesser depending on your inclinations). I think the general principle we want to follow here is that a block (or any entity in general) should not lose its type because some part of its syntax is malformed (I have made similar arguments about property drawers). That is, if something starts with #+begin_NAME stuff and there is a corresponding #+end_NAME, then it is a block. The choice of how a src block without a lang should behave is a bit more complex as there are multiple consumers of src blocks that make different assumptions. As mentioned above. I think that if a block is missing the lang we could think of it instead as the null language. If we have the :var language because someone has other contents on the line they have a well formed src block, but will get a different error because there is currently no known language ":var".
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
> It's purpose is also different. > I am generally skeptical about our ability to provide universal way to > judge if a given sexp is safe or not. It should be left to the user. I am in complete agreement.
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Interestingly, as I was looking through my notes in orgstrap, I see my past self had found a macro org-babel-one-header-arg-safe-p which pointed to defconst org-babel-safe-header-args, but that is a const and not really user configurable. Of course the user could cl-setf on it, but it also only checks strings and has no ability to e.g. see if the value of (symbol-function #'identity) has changed since the check function was defined. Two examples. (let ((old-identity (symbol-function #'identity))) (cl-letf* (((symbol-function #'identity) (lambda (x) (message "there") x))) (identity 'hello) (equal (symbol-function #'identity) old-identity))) (let ((old-and (symbol-function #'and))) (cl-letf* (((symbol-function #'old-andf) old-and) ((symbol-function #'and) (lambda (&rest args) (message "oops %s" args) (old-andf args (list (and) (and 1 2 3) (equal (symbol-function #'and) old-and > Tom, does not the following allow to achieve the same without your patch? It works if I have a closed set of things I want to allow but not if I want to set it to nil to e.g. restore the old behavior (worse for security but not as bad as setting ocbe to nil), e.g. if I'm under duress and need to get something that used to work to work again without the risk of automatically running dangerous code, (e.g. blocks that might rm something). > I know, it does not work, but I think it is due to (format "%S" cell) > instead of passing cell directly in > > -'((:eval . yes)) nil (format "%S" cell) > > My point is that if some expression is safe for a variable value then it > is safe for the source block body. There is another use case here, which I alluded to in the previous comment, which is that sometimes ocbe is the last line of defense against running dangerous code. Ideally users would have set :eval never on blocks like that to be sure, but if they don't you don't want someone already trying to get something to work to get too much to work. Again, this is focused on the ocbec -> nil case. > Have you ever seen the prompt for a table? Err ... maybe? So the answer is probably no. > I suppose, tables are the most prominent security issue related to > unsolicited code execution: For me it would be arbitrary expressions in the headers of source blocks. Hard to know which one is more prevalent across the population of org users. > Max Nikulin to emacs-orgmode. Re: [BUG][Security] begin_src :var > evaluated before the prompt to confirm execution. Fri, 28 Oct 2022 > 11:11:18 +0700. https://list.orgmode.org/tjfkp7$ggm$1...@ciao.gmane.io > > I am still in doubts if > > 10e857d42 2022-10-28 11:09:50 +0800 Ihor Radchenko: org-babel-read: Obey > `org-confirm-babel-evaluate' > > was an unambiguous improvement. Perhaps it just forces more users to set > `org-confirm-babel-evaluate' to nil compromising their security to more > severe degree. Heh. It is always a hard balance to strike. In the context of that thread having a variable that would find-file-literally for untrusted org files by default might be useful. Again, it is a pain. I can tell you from experience having written the system that does something similar for orgstrap. There is no safe way other than a user-maintained whitelist based on file hashes, everything else can be spoofed in one way or another. I suspect that once we have the machinery in this patch in place we can look for some sane defaults. Note that the example function we keep passing around isn't quite good enough because someone could probably figure out how to rewrite the identity function so we would need to make sure that it had not changed since emacs was loaded (unlikely, but if I can image it someone could surely do it).
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
> Will it be clear to users what "cell" means in this context? I assume the language was originally chosen with tables in mind, but I think it is clear? The one issue is that using org-babel-confirm-evaluate doesn't use the word "cell" in the yes-or-no-p prompt. > Am I wrong that, roughly speaking, the effective value is (and > org-confirm-babel-evaluate-cell org-confirm-babel-evaluate)? You are correct. Only in the case that org-confirm-babel-evaluate-cell is or evaluates to t will the yes-or-no-p be displayed _IF_ org-confirm-babel-evaluate is or evaluates to t. org-babel-confirm-evaluate (NOT org-confirm-babel-evaluate) is the function that issues the yes-or-no-p. > Should it be described in the docstring? Maybe? In the current implementation the value of org-confirm-babel-evaluate takes precedence, which I think is the correct behavior. If someone has told us that they blindly want to execute all code, but not cells, we should not change the default behavior, and thus we should execute all cells and blocks > It seems e.g. :eval query affects the latter, but not the former. :eval has no impact on cells. In fact in some files I use :eval (and this-file-done-loading "never") on some blocks that (setq this-file-done-loading) at the end to achieve only-once behavior. > If it is so then it is too complicated for me to feel firm ground > while reasoning on security. :eval has never, and cannot impact evaluation of cells, because it is specifically used to control evaluation of blocks. Similarly :eval has no interaction with any of the confirm-babel-evaluate machinery. I see one possible point of confusion which is that he :eval that you see in the code of org-babel-read is a hack to create fake info for a cell. Are you maybe looking for a file level :eval-cell option/property that would allow you to declare that you should NEVER run cells at all and also never prompt? Basically this is an option that says "treat all cells as if they and their header property are the default value". I'm fairly certain that implementing such a thing is a bad idea, for example if someone sets :dir using a cell and it is overwritten they could execute code in the wrong environment, which can lead to nasty security issues. Thus the only global option for cells that we could safely implement would have the semantics of "treat all cells as if they are (error)". > I have not decided if it is feasible to add some property to the INFO > argument passed to `org-babel-confirm-evaluate' and to put there > condition which variable should determine result. Having done so before, I would rather not touch org-babel-confirm-evaluate if at all possible. The only disadvantage is that for now when users are prompted they will not be informed that it is a cell, however from the short nature of cells the will hopefully be able to figure it out. The fact that org already jumps to the location of the cell in the file helps a lot with this. Obviously this doesn't help in a batch context, but if a user is hitting this in batch mode they are in trouble anyway.
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Hi Ihor, Here's the updated patch using :safe, and an additional patch for the news entry to make it easier to apply the core change to bugfix if needed. Best! Tom > I am also wondering if we should include this into bugfix. I can vouch for the fact that trying to work around this in any other way is going to be a massive pain. For example, it will be hard to use vanilla 29 for various CI types of things where someone might need to execute a cell but not want to allow arbitrary codeblocks. From 4a78e1b5ea98dee569ff690037c661ab5c300194 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sat, 10 Dec 2022 12:11:17 -0800 Subject: [PATCH 1/2] ob-core: add org-confirm-babel-evaluate-cell custom variable * lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control execution of cells separate from execution of src blocks, it works in exactly the same way as `org-confirm-babel-evaluate'. * lisp/ob-core.el (org-babel-read): `org-confirm-babel-evaluate-cell' is used to check cells independent of `org-confirm-babel-evaluate'. Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it became extremely annoying to tangle files that make extensive use of elisp expression in src block #+header: statements. This commit resolves the issue by making it possible to ignore checks on cells (the old behavior) without compromising general security for running src blocks. This is necessary because there is no easy way to hop swap `org-confirm-babel-evaluate' between `org-get-src-block-info' where `org-babel-read' is called and the execution of that src block. It could probably be done using advice around `org-babel-read', but that is a level of hackery that should be avoided. --- lisp/ob-core.el | 19 +++ 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 2fa9d8978..d56e47de5 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -128,6 +128,14 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding." ;; don't allow this variable to be changed through file settings (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t))) +(defcustom org-confirm-babel-evaluate-cell t + "Confirm before evaluating a cell. +This follows the same conventions as `org-confirm-babel-evaluate'." + :group 'org-babel + :package-version '(Org . "9.6") + :type '(choice boolean function) + :safe (lambda (x) (eq x t))) + (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil "\\\ Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding." @@ -3180,11 +3188,14 @@ situations in which is it not appropriate." (string= cell "*this*"))) ;; Prevent arbitrary function calls. (if (and (memq (string-to-char cell) '(?\( ?`)) + (if (functionp org-confirm-babel-evaluate-cell) + (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell) +org-confirm-babel-evaluate-cell) (not (org-babel-confirm-evaluate - ;; See `org-babel-get-src-block-info'. - (list "emacs-lisp" (format "%S" cell) -'((:eval . yes)) nil (format "%S" cell) -nil nil +;; See `org-babel-get-src-block-info'. +(list "emacs-lisp" (format "%S" cell) + '((:eval . yes)) nil (format "%S" cell) + nil nil ;; Not allowed. (user-error "Evaluation of elisp code %S aborted." cell) (eval (read cell) t))) -- 2.37.4 From 03aad0a73acfca05245a01e83bae3609e6d3ed04 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Mon, 12 Dec 2022 17:45:14 -0800 Subject: [PATCH 2/2] * etc/ORG-NEWS: Add entry for `org-confirm-babel-evaluate-cell'. --- etc/ORG-NEWS | 36 1 file changed, 36 insertions(+) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 5d5e726e0..16ff5ba67 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -12,6 +12,42 @@ See the end of the file for license conditions. Please send Org bug reports to mailto:emacs-orgmode@gnu.org. * Version 9.7 (not released yet) + +** New options +*** A new custom setting ~org-confirm-babel-evaluate-cell~ to control confirming execution of cells + +Following recent changes to ~org-babel-read~ it became annoying to +tangle files that make extensive use of elisp expression in src +block #+header: statements. + +~org-confirm-babel-evaluate-cell~ resolves the issue by making it +possible to ignore checks on cells (the old behavior) without +compromising general security for running src blocks. + +It works in the same
Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
Here's the rebased patch. It should apply without issue. Best! Tom From 5247d2459800f82434f1bc3aeea136c18af7923c Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sun, 4 Dec 2022 01:02:35 -0800 Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats refactor * lisp/org-expiry.el (org-expiry-insert-created) (org-expiry-insert-expiry): timestamp formats dropped delimiters so a slight modification is required following org commit e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward compatible by removing < and > from the old timestamp format --- lisp/org-expiry.el | 32 +--- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el index b359610..c3dad28 100644 --- a/lisp/org-expiry.el +++ b/lisp/org-expiry.el @@ -301,12 +301,17 @@ update the date." (current-time))) (setq d-hour (format-time-string "%H:%M" d-time)) (setq timestr -(org-expiry-format-timestamp - ;; two C-u prefixes will call org-read-date - (if (equal arg '(16)) - (org-read-date nil nil nil nil d-time d-hour) - (format-time-string (cdr org-time-stamp-formats))) - org-expiry-inactive-timestamps)) + ;; two C-u prefixes will call org-read-date +(concat "<" +(if (equal arg '(16)) +(org-read-date nil nil nil nil d-time d-hour) + (format-time-string + (replace-regexp-in-string "\\(^<\\|>$\\)" "" + (cdr org-time-stamp-formats +">")) + ;; maybe transform to inactive timestamp + (if org-expiry-inactive-timestamps + (setq timestr (concat "[" (substring timestr 1 -1) "]"))) (save-excursion (org-entry-put (point) org-expiry-created-property-name timestr) @@ -321,11 +326,16 @@ and insert today's date." (setq d-time (if d (org-time-string-to-time d) (current-time))) (setq d-hour (format-time-string "%H:%M" d-time)) -(setq timestr (org-expiry-format-timestamp - (if today - (format-time-string (cdr org-time-stamp-formats)) - (org-read-date nil nil nil nil d-time d-hour)) - org-expiry-inactive-timestamps)) +(setq timestr (concat "<" + (if today + (format-time-string + (replace-regexp-in-string "\\(^<\\|>$\\)" "" + (cdr org-time-stamp-formats))) +(org-read-date nil nil nil nil d-time d-hour)) + ">")) +;; maybe transform to inactive timestamp +(if org-expiry-inactive-timestamps + (setq timestr (concat "[" (substring timestr 1 -1) "]"))) (save-excursion (org-entry-put -- 2.37.4
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
> Please use :package-version and let the mapping be handled by > customize-package-emacs-version-alist. Got it. Here's the updated patch (I think I did it correctly?). From 47a47aa9453a54a4f5f2e9188e2ad072a77c50cb Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sat, 10 Dec 2022 12:11:17 -0800 Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable * lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control execution of cells separate from execution of src blocks, it works in exactly the same way as org-confirm-babel-evaluate. * lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is now used to check cells independent of org-confirm-babel-evaluate. Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it became extremely annoying to tangle files that make extensive use of elisp expression in src block #+header: statements. This commit resolves the issue by making it possible to ignore checks on cells (the old behavior) without compromising general security for running src blocks. This is necessary because there is no easy way to hop swap org-confirm-babel-evaluate between org-get-src-block-info where org-babel-read is called and the execution of that src block. It could probably be done using advice around org-babel-read, but that is a level of hackery that should be avoided. --- lisp/ob-core.el | 20 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 62b0d3612..aa73a9cbc 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -128,6 +128,15 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding." ;; don't allow this variable to be changed through file settings (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t))) +(defcustom org-confirm-babel-evaluate-cell t + "Confirm before evaluating a cell. +This follows the same conventions as `org-confirm-babel-evaluate'." + :group 'org-babel + :package-version '(Org . "9.6") + :type '(choice boolean function)) +;; don't allow this variable to be changed through file settings +(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t))) + (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil "\\\ Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding." @@ -3180,11 +3189,14 @@ situations in which is it not appropriate." (string= cell "*this*"))) ;; Prevent arbitrary function calls. (if (and (memq (string-to-char cell) '(?\( ?`)) + (if (functionp org-confirm-babel-evaluate-cell) + (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell) +org-confirm-babel-evaluate-cell) (not (org-babel-confirm-evaluate - ;; See `org-babel-get-src-block-info'. - (list "emacs-lisp" (format "%S" cell) -'((:eval . yes)) nil (format "%S" cell) -nil nil +;; See `org-babel-get-src-block-info'. +(list "emacs-lisp" (format "%S" cell) + '((:eval . yes)) nil (format "%S" cell) + nil nil ;; Not allowed. (user-error "Evaluation of elisp code %S aborted." cell) (eval (read cell) t))) -- 2.37.4
Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
Looks like string-replace doesn't support that notation, so both too aggressive and without sufficient control. I've switched to replace-regexp-in-string which does what we want. Here's the updated patch. From 12ca29965e867acd64fecaecd14f2f74e90d7e99 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sun, 4 Dec 2022 01:02:35 -0800 Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats refactor * lisp/org-expiry.el (org-expiry-insert-created) (org-expiry-insert-expiry): timestamp formats dropped delimiters so a slight modification is required following org commit e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward compatible by removing < and > from the old timestamp format --- lisp/org-expiry.el | 22 ++ 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el index 98ad58a..0462735 100644 --- a/lisp/org-expiry.el +++ b/lisp/org-expiry.el @@ -299,10 +299,13 @@ update the date." (setq d-hour (format-time-string "%H:%M" d-time)) (setq timestr ;; two C-u prefixes will call org-read-date - (if (equal arg '(16)) - (concat "<" (org-read-date - nil nil nil nil d-time d-hour) ">") - (format-time-string (cdr org-time-stamp-formats +(concat "<" +(if (equal arg '(16)) +(org-read-date nil nil nil nil d-time d-hour) + (format-time-string + (replace-regexp-in-string "\\(^<\\|>$\\)" "" + (cdr org-time-stamp-formats +">")) ;; maybe transform to inactive timestamp (if org-expiry-inactive-timestamps (setq timestr (concat "[" (substring timestr 1 -1) "]"))) @@ -320,10 +323,13 @@ and insert today's date." (setq d-time (if d (org-time-string-to-time d) (current-time))) (setq d-hour (format-time-string "%H:%M" d-time)) -(setq timestr (if today - (format-time-string (cdr org-time-stamp-formats)) - (concat "<" (org-read-date - nil nil nil nil d-time d-hour) ">"))) +(setq timestr (concat "<" + (if today + (format-time-string + (replace-regexp-in-string "\\(^<\\|>$\\)" "" + (cdr org-time-stamp-formats))) +(org-read-date nil nil nil nil d-time d-hour)) + ">")) ;; maybe transform to inactive timestamp (if org-expiry-inactive-timestamps (setq timestr (concat "[" (substring timestr 1 -1) "]"))) -- 2.37.4
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Here is an updated version of the patch with the convention corrected and the docstring updated for clarity. It would be great to try to get this into any 9.6 patches before the 29 release, but I'm not sure if that is possible. If it is missing we are likely to get a lot of messages from unhappy users. Best! Tom From 54e468b60f17b54d81edafd8ee9e22311e519793 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sat, 10 Dec 2022 12:11:17 -0800 Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable * lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control execution of cells separate from execution of src blocks, it works in exactly the same way as org-confirm-babel-evaluate. * lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is now used to check cells independent of org-confirm-babel-evaluate. Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it became extremely annoying to tangle files that make extensive use of elisp expression in src block #+header: statements. This commit resolves the issue by making it possible to ignore checks on cells (the old behavior) without compromising general security for running src blocks. This is necessary because there is no easy way to hop swap org-confirm-babel-evaluate between org-get-src-block-info where org-babel-read is called and the execution of that src block. It could probably be done using advice around org-babel-read, but that is a level of hackery that should be avoided. --- lisp/ob-core.el | 20 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 62b0d3612..60dabab0a 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -128,6 +128,15 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding." ;; don't allow this variable to be changed through file settings (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t))) +(defcustom org-confirm-babel-evaluate-cell t + "Confirm before evaluating a cell. +This follows the same conventions as `org-confirm-babel-evaluate'." + :group 'org-babel + :version "29.1" + :type '(choice boolean function)) +;; don't allow this variable to be changed through file settings +(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t))) + (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil "\\\ Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding." @@ -3180,11 +3189,14 @@ situations in which is it not appropriate." (string= cell "*this*"))) ;; Prevent arbitrary function calls. (if (and (memq (string-to-char cell) '(?\( ?`)) + (if (functionp org-confirm-babel-evaluate-cell) + (funcall org-confirm-babel-evaluate-cell "emacs-lisp" cell) +org-confirm-babel-evaluate-cell) (not (org-babel-confirm-evaluate - ;; See `org-babel-get-src-block-info'. - (list "emacs-lisp" (format "%S" cell) -'((:eval . yes)) nil (format "%S" cell) -nil nil +;; See `org-babel-get-src-block-info'. +(list "emacs-lisp" (format "%S" cell) + '((:eval . yes)) nil (format "%S" cell) + nil nil ;; Not allowed. (user-error "Evaluation of elisp code %S aborted." cell) (eval (read cell) t))) -- 2.37.4
Re: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Hi Max, Thank you for the feedback. More replies in lines. Best! Tom > I am not sure concerning "exactly". > > lisp/ob-core.el:248 > `org-confirm-babel-evaluate' is called with 2 arguments. In your patch > `org-confirm-babel-evaluate-cell' has a single argument. You're right, and in point of fact I should have retained the structure exactly because in other contexts I have thought about ways to use other languages in contexts like that. At the moment everything is elisp so I dropped the argument, but that is clearly a mistake. > It seems, you do not change defaults. Could you, please, provide an > example of configuration that is less annoying, but still safe? #+begin_src elisp :results none (setq-local org-confirm-babel-evaluate-cell (lambda (lang body) (ignore lang) (let ((rb (read body))) (not ; aka (unless condition t) (or (member rb '((or) (and) ;; add more forms that are known safe here )) (and (eq (car rb) 'identity) (let ((v (cadr rb))) (or (symbolp v) (stringp v) (numberp v) #+end_src #+header: :var v1=(or) v2=(and) v3=(identity nil) #+header: :var v4=(identity default-directory) v5=(identity #o0755) #+header: :var v6=(identity "not sure why you would want to do this") #+header: :var v7=(identity (concat "this" "will" "fail")) #+header: :var v8="reminder that strings are ok" #+begin_src elisp (mapcar #'list (list v1 v2 v3 v4 v5 v6 v7 v8)) #+end_src > I was thinking if it is possible to collect requests to confirm and to > allow the user to decide for the whole bunch of expressions and code > blocks. Besides implementation issues, there is a question concerning UI > that will allow to inspect code to be evaluated. Yes, in the example above I thought about including something with a yes-or-no-p where users could quickly add forms to a safe list some (defcustom org-known-safe-cells '()) or something like that. A user could do that with the new machinery, and we could do the same for the default implementation. I think that is the next step once we get the basics in place. > Calling convention for the case of function value is not described. If > it is really the same as for `org-confirm-babel-evaluate' then this user > option should be mentioned in the docstring. When I correct the function signature to actually match I will make a note in the docstring. > :package-version instead of :version? I think because org is part of emacs core we use the emacs version? I see "24.1" included with other org defcustoms. > Is there any reason to not use the :safe property of `defcustom'? I see > that you take definition of `org-confirm-babel-evaluate' as a template > so I wonder if there is some particular reason or the original code was > just written before introducing of :safe. I'm guessing that it was written before :safe, but don't know for sure. A systematic cleanup of stuff like that could come after this maybe?
Re: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats
Here is an updated patch. Best! Tom On Thu, Dec 8, 2022 at 3:42 AM Ihor Radchenko wrote: > > Tom Gillespie writes: > > > Hi, > >Here is a patch for org-contrib/lisp/org-expiry.el to account for > > recent changes to org-time-stamp-formats. Best, > > Tom > > > > PS is this list still the best place to send org-contrib patches? > > Yes, this list is suitable as long as org-expiry is orphaned. > > For the patch, note that it is not a good idea to bump the minimal > required Org version of Org 9.6. I'd rather suggest keeping backwards > compatibility by stripping "<" and ">" from org-time-stamp-formats, if > any. That will not affect existing users of older Emacs versions and > built-in Org. > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> From 248104955707a7195a594712d1c7ad9d64dee1c1 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sun, 4 Dec 2022 01:02:35 -0800 Subject: [PATCH] lisp/org-expiry.el: Account for org-time-stamp-formats refactor * lisp/org-expiry.el (org-expiry-insert-created) (org-expiry-insert-expiry): timestamp formats dropped delimiters so a slight modification is required following org commit e3a7c01874c9bb80e04ffa58c578619faf09e7f0, the change is made backward compatible by removing < and > from the old timestamp format --- lisp/org-expiry.el | 22 ++ 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/lisp/org-expiry.el b/lisp/org-expiry.el index 98ad58a..0909aaf 100644 --- a/lisp/org-expiry.el +++ b/lisp/org-expiry.el @@ -299,10 +299,13 @@ update the date." (setq d-hour (format-time-string "%H:%M" d-time)) (setq timestr ;; two C-u prefixes will call org-read-date - (if (equal arg '(16)) - (concat "<" (org-read-date - nil nil nil nil d-time d-hour) ">") - (format-time-string (cdr org-time-stamp-formats +(concat "<" +(if (equal arg '(16)) +(org-read-date nil nil nil nil d-time d-hour) + (format-time-string + (string-replace "<" "" + (string-replace ">" "" (cdr org-time-stamp-formats) +">")) ;; maybe transform to inactive timestamp (if org-expiry-inactive-timestamps (setq timestr (concat "[" (substring timestr 1 -1) "]"))) @@ -320,10 +323,13 @@ and insert today's date." (setq d-time (if d (org-time-string-to-time d) (current-time))) (setq d-hour (format-time-string "%H:%M" d-time)) -(setq timestr (if today - (format-time-string (cdr org-time-stamp-formats)) - (concat "<" (org-read-date - nil nil nil nil d-time d-hour) ">"))) +(setq timestr (concat "<" + (if today + (format-time-string + (string-replace "<" "" + (string-replace ">" "" (cdr org-time-stamp-formats +(org-read-date nil nil nil nil d-time d-hour)) + ">")) ;; maybe transform to inactive timestamp (if org-expiry-inactive-timestamps (setq timestr (concat "[" (substring timestr 1 -1) "]"))) -- 2.37.4
Re: Multiple versions of Org in load-path problem
> I don't think there is any safe way to install an updated version of > org-mode other than I don't know whether it counts as safe, but since I don't see it mentioned anywhere else in the thread you can do the following. (require 'package) (unless package--initialized (package-initialize)) (assq-delete-all 'org package--builtins) (assq-delete-all 'org package--builtin-versions) (package-install 'org)
[PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable
Hi, Here is a patch that improves the ergonomics and thus hopefully the security for the recent changes to check evaluation for cells. Full details in the commit message on the patch. Best! Tom From da3a88e2919ec5bac6328d55eb99b3352f25b9b4 Mon Sep 17 00:00:00 2001 From: Tom Gillespie Date: Sat, 10 Dec 2022 12:11:17 -0800 Subject: [PATCH] ob-core: add org-confirm-babel-evaluate-cell custom variable * lisp/ob-core.el (org-confirm-babel-evaluate-cell): Added to control execution of cells separate from execution of src blocks, it works in exactly the same way as org-confirm-babel-evaluate. * lisp/ob-core.el (org-babel-read): org-confirm-babel-evaluate-cell is now used to check cells independent of org-confirm-babel-evaluate. Following the change in 10e857d42859a55b23cd4206ffce3ebd0f678583 it became extremely annoying to tangle files that make extensive use of elisp expression in src block #+header: statements. This commit resolves the issue by making it possible to ignore checks on cells (the old behavior) without compromising general security for running src blocks. This is necessary because there is no easy way to hop swap org-confirm-babel-evaluate between org-get-src-block-info where org-babel-read is called and the execution of that src block. It could probably be done using advice around org-babel-read, but that is a level of hackery that should be avoided. --- lisp/ob-core.el | 19 +++ 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/lisp/ob-core.el b/lisp/ob-core.el index 62b0d3612..14669d940 100644 --- a/lisp/ob-core.el +++ b/lisp/ob-core.el @@ -128,6 +128,14 @@ remove code block execution from the `\\[org-ctrl-c-ctrl-c]' keybinding." ;; don't allow this variable to be changed through file settings (put 'org-confirm-babel-evaluate 'safe-local-variable (lambda (x) (eq x t))) +(defcustom org-confirm-babel-evaluate-cell t + "Confirm before evaluating a cell." + :group 'org-babel + :version "29.1" + :type '(choice boolean function)) +;; don't allow this variable to be changed through file settings +(put 'org-confirm-babel-evaluate-cell 'safe-local-variable (lambda (x) (eq x t))) + (defcustom org-babel-no-eval-on-ctrl-c-ctrl-c nil "\\\ Remove code block evaluation from the `\\[org-ctrl-c-ctrl-c]' key binding." @@ -3180,11 +3188,14 @@ situations in which is it not appropriate." (string= cell "*this*"))) ;; Prevent arbitrary function calls. (if (and (memq (string-to-char cell) '(?\( ?`)) + (if (functionp org-confirm-babel-evaluate-cell) + (funcall org-confirm-babel-evaluate-cell cell) +org-confirm-babel-evaluate-cell) (not (org-babel-confirm-evaluate - ;; See `org-babel-get-src-block-info'. - (list "emacs-lisp" (format "%S" cell) -'((:eval . yes)) nil (format "%S" cell) -nil nil +;; See `org-babel-get-src-block-info'. +(list "emacs-lisp" (format "%S" cell) + '((:eval . yes)) nil (format "%S" cell) + nil nil ;; Not allowed. (user-error "Evaluation of elisp code %S aborted." cell) (eval (read cell) t))) -- 2.37.4