Re: [DISCUSSION] The meaning of :cmdline header argument across babel backends
On Fri, 03 May 2024 14:11:56 +0200 Ihor Radchenko wrote --- > Maybe instead use two different extra arguments: > :interpreter-args / :compiler-args ? > > Then, also :script-args / :program-args I'm okay with these. I can start on a patch for :script-args and :program-args within ob-shell. I anticipate :interpreter-args will be more involved since it will interact with 'shell-command-switch' and it'd be nice to have it also work with sessions. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [DISCUSSION] The meaning of :cmdline header argument across babel backends
On Wed, 01 May 2024 20:01:03 +0200 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I disagree with one aspect: we shouldn't use Worg as a source of > > truth. The argument holds based on historical behavior of :cmdline. > > AFAIU, Worg is a wiki which is open, more or less, to anyone. Worg > > contents, AFAIU, have not always undergone review. The manual should > > be the final authority. Fortunately, there's nothing in the manual > > about :cmdline. > > For babel backends specifically, WORG is _the_ documentation for the > built-in backends. It is what we will eventually move to the official > manual and it is what we point users to from the manual for now. Okay, I didn't know we made an explicit reference. > (2) nobody got around to actually move things to the manual. I'd be happy to help with this. Discussion for another thread, though. > > Are we thinking of implementing these for other languages, beyond > > ob-shell? > > Yes. The title of this thread has "across babel backends" :) :) > > If we're looking at these as general headers, then I don't think "arg" > > is the correct term here since a switch may not take a value. For > > example, the "-r" option for Bash (IIUC). > > > Quick name ideas that aren't good yet may inspire better ones by > > inspiring disgust--:switches, :flags, :options, (using an "i" prefix > > for "interpreter") :iswitches, :iflags, :ioptions > > Emm... but "command line arguments". No? I was thinking it'd be strange to have an "argument to an argument." However, the Bash man says things like "non-option arguments". It wasn't my intent to have an argument argument argument. Since we're considering this for all babel backends, "interpreter-args" wouldn't describe gcc or other compilers. What about :command-args, :command-args, :cmd-args? "Command" is the only thing I can think of that describes all of the languages, other than maybe "binary" or "executable". Maybe we could simply use ":args"? Maybe ":meta-args" since gcc or bash is meta to the script/artifact? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [DISCUSSION] The meaning of :cmdline header argument across babel backends
On Sat, 27 Apr 2024 12:53:25 +0200 Max Nikulin wrote --- > On 26/04/2024 20:09, Ihor Radchenko wrote: > > Max Nikulin writes: > > > >> However looking wider, I do not like that :cmdline for ob-shell has > >> different meaning than for other languages, see e.g. ob-sql. Only for > >> shell this parameter is treated as arguments of a *script*. In other > >> cases :cmdline is used to specify arguments of *interpreter* and I think > >> ob-shell should follow this convention. > > > > Alas, we already have the current state of affairs documented in > > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html#orge70bc7b > > > > So, no breaking changes. Both points are valid. I disagree with one aspect: we shouldn't use Worg as a source of truth. The argument holds based on historical behavior of :cmdline. AFAIU, Worg is a wiki which is open, more or less, to anyone. Worg contents, AFAIU, have not always undergone review. The manual should be the final authority. Fortunately, there's nothing in the manual about :cmdline. On Mon, 29 Apr 2024 15:33:38 +0200 Ihor Radchenko wrote --- > I like :script-args. + 1 > The counterpart should then be :interpreter-args? Are we thinking of implementing these for other languages, beyond ob-shell? If not, I believe we should consider it. That may be a path to resolve the issues of consistency and backwards compatibility. If we're looking at these as general headers, then I don't think "arg" is the correct term here since a switch may not take a value. For example, the "-r" option for Bash (IIUC). Quick name ideas that aren't good yet may inspire better ones by inspiring disgust--:switches, :flags, :options, (using an "i" prefix for "interpreter") :iswitches, :iflags, :ioptions -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
[BUG] ob-shell: internal representation of cmdline arguments the same
While investigating "[BUG] ob-shell: :shebang changes interpretation of :cmdline" (https://list.orgmode.org/orgmode/18f01342a2f.124ad27612732529.8693431365849276...@excalamus.com/), it was observed that :cmdline 1 2 3 behaves like :cmdline "1 2 3" The first argument is the first space delimited character following ":cmdline": #+begin_src bash :cmdline 1 2 3 echo "$1" #+end_src #+RESULTS: : 1 The following makes it seem like command-line arguments may be grouped using quotes: #+begin_src bash :cmdline "1 2" 3 echo "$1" #+end_src #+RESULTS: : 1 2 The following demonstrates that quotes do not always indicate separate arguments: #+begin_src bash :cmdline "1 2 3" echo "$1" #+end_src #+RESULTS: : 1 It was stated that, #+begin_quote AFAICT, it's due to how headers are parsed by 'org-babel-parse-header-arguments' using 'org-babel-read'. The cell "\"1 2 3\"" (corresponding to :cmdline "1 2 3") is reduced through 'string-match' to "1 2 3". The cell "1 2 3" (corresponding to :cmdline 1 2 3), on the other hand, passes through. The result is that :cmdline "1 2 3" and :cmdline 1 2 3 become indistinguishable. #+end_quote Org mode version 9.7-pre (release_9.6.27-1393-ge0f24a @ /home/ahab/Projects/org-mode/lisp/), commit e0f24a3f6 -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
[BUG] ob-shell: results missing leading quotes
While investigating "[BUG] ob-shell: :shebang changes interpretation of :cmdline" (https://list.orgmode.org/orgmode/18f01342a2f.124ad27612732529.8693431365849276...@excalamus.com/), behavior like the following was observed in which leading quote marks are sometimes removed from results: #+begin_src bash :results raw echo \"\"1\"\" #+end_src #+RESULTS: ""1"" #+begin_src bash echo \"\"1\"\" #+end_src #+RESULTS: : 1"" #+begin_src bash :results output echo \"\"1\"\" #+end_src #+RESULTS: : ""1"" Org mode version 9.7-pre (release_9.6.27-1393-ge0f24a @ /home/ahab/Projects/org-mode/lisp/), commit e0f24a3f6 -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline
Thank you both for your thoughtful replies. There's a lot to process in this bug. We've also uncovered, at least, related four bugs. I think it would help to submit separate bug reports for each of the related issues so that we may discuss them separately, if possible. Is Woof! the way to do that? I recall there being some issues with Woof!. Is there something not in https://tracker.orgmode.org/howto that I should know? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: keeping subtree heading on export
thanks for the response, Ihor -- reply inline below: On Fri, Apr 26, 2024 at 9:45 AM Ihor Radchenko wrote: > Matt Price writes: > > > ... I need to export each individual page > > of the "book" to its own markdown page. However, jupyter-book expects to > > find the title of the page in the initial first-level heading. So I'd > like > > to retain the subtree "title" as a first-level heading, and demote the > > remaining headings to their original state within the larger org > document. > > > > Does anyone know of an existing exporter that already does this, from > which > > I can steal? Or if not, how would you suggest I go about doing this? > > May you provide an example demonstrating initial Org mode document and > how the exported md documents should look like? > > I don't think my request was very clear. Let's say I'm writing a "book" (really a documentation set of some kind) with several "chapters" (really, each chapter is an indiviual html page, though more complex nesting is permitted by jupyter-book). I write in org-mode: - * Chapter 1 text ** Chapter 1.1 text ** Chapter 1.2 * Chapter 2 text ** Chapter 2.1 text - And I want to produce two markdown files: chapter-1.md: - # Chapter 1 text ## Chapter 1.1 text ## Chapter 1.2 -- chapter-2.md # Chapter 2 text ## Chapter 2.1 text - I tried to learn a little more on my own after posting. I can set `org-md-toplevel-hlevel` to `2`, and then in the template function add the title property "manually" by extracting it from the info communication channel: -- (defun org-myst-inner-template (contents info) "Return body of document after converting it to Markdown syntax. CONTENTS is the transcoded contents string. INFO is a plist holding export options." (let* ((depth (plist-get info :with-toc)) (headlines (and depth (org-export-collect-headlines info depth))) (toc-string (or (mapconcat (lambda (headline) (org-myst-format-toc headline info)) headlines "\n") "")) (toc-tail (if headlines "\n\n" "")) (front-matter (org-myst-front-matter)) (title (org-export-data (plist-get info :title) info))) (org-trim (concat front-matter toc-string toc-tail "\n" "# " title "\n\n" contents (org-myst-footnote-section info) -- This works ok! But the problem ocmes because I would like to be able to *also* sometimes export a whole file (rather than just a subtree) using the same exporter.That's because I have inherited osme projects where files take hte form: chapter-1.org -- * Chapter 1 text ** Chapter 1.1 text ** Chapter 1.2 -- etc. The real problem I have is that, when exporting a subtree, I want to set org-md-toplevel-hlevel to "2" and add the title; when exporting a whole file, I want to set org-md-toplevel-hlevel to "1" and ignore the title. I don't think this is how org exporters are supposed to work, but I'm trying to interface to an established system that has chosen osme un-org-like conventions. Is this clearer, and do you see a way to do this? I am just looking at the source code and I wonder if I could add some (let) bindings inside org-myst-export-to-markdown and -as-markdown before calling org-export-to-[buffer|file]. hmm. Just tried it and it seems to work. Something seems wrong about this kind of code, though, in which I let-bind a variable in one function solely so that I can use it in another. Is there a better way? Un that case ,I would want to set > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at <https://orgmode.org/>. > Support Org development at <https://liberapay.com/org-mode>, > or support my work at <https://liberapay.com/yantar92> >
keeping subtree heading on export
I'm writing a primitive exporter for jupyter-book <https://github.com/executablebooks/jupyter-book>, which is a neat way to maintain static documentation and executable jupyter notebooks from the same source files. The source files are written in Myst flavored markdown <https://mystmd.org/>, so I started with a simpleminded fork of the gfm exporter <https://github.com/titaniumbones/ox-myst>. What I'd really like, though is something more like ox-hugo <https://github.com/kaushalmodi/ox-hugo>, which maintains a whole blog in a single org file. For this to work, I need to export each individual page of the "book" to its own markdown page. However, jupyter-book expects to find the title of the page in the initial first-level heading. So I'd like to retain the subtree "title" as a first-level heading, and demote the remaining headings to their original state within the larger org document. Does anyone know of an existing exporter that already does this, from which I can steal? Or if not, how would you suggest I go about doing this? Thanks, Matt
[PATCH] Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline
ot;1\" \"2\" \"3\"" | ("\"1\" \"2\" \"3\"") |
| header | 'org-babel-sh-evaluate' | process-file ARGS |
| | cmdline variable value | cmdline |
|--+-+---|
| :cmdline 1 2 3 | "1 2 3" | ("1 2 3") |
| :cmdline "1 2" 3"| "\"1 2\" 3" | ("\"1 2\" 3") |
| :cmdline "1" "2" "3" | "\"1\" \"2\" \"3\"" | ("\"1\" \"2\" \"3\"") |
Notice that the ARGS passed to 'process-file' are the same for both cases. The
problem is that the "block equivalent shell calls" are *not* the same. If we
arrange the equivalent shell calls from the blocks given above into a table, we
see that the forms are different:
| header | cmdline variable value | shebang+cmdline call
|
|--++|
| :cmdline 1 2 3 | "1 2 3"|
/tmp/babel-Xd6rGS/sh-script-61jvMa "1 2 3" |
| :cmdline "1 2" 3"| "\"1 2\" 3"|
/tmp/babel-Xd6rGS/sh-script-61jvMa "\"1 2\" 3" |
| :cmdline "1" "2" "3" | "\"1\" \"2\" \"3\""|
/tmp/babel-Xd6rGS/sh-script-61jvMa "\"1\" \"2\" \"3\"" |
| header | cmdline variable value | cmdline call
|
|--++|
| :cmdline 1 2 3 | "1 2 3"| bash -c "./sh-script-8GJzdG 1
2 3" |
| :cmdline "1 2" 3"| "\"1 2\" 3"| bash -c "./sh-script-8GJzdG
\"1 2\" 3" |
| :cmdline "1" "2" "3" | "\"1\" \"2\" \"3\""| bash -c "./sh-script-8GJzdG
\"1\" \"2\" \"3\"" |
The reported bug exists because shebang+cmdline interprets the characters
following :cmdline as a *single* string. Without :shebang, a lone :cmdline
interprets them as space delimited.
One possible solution is to reformat the 'process-file' ARGS for the
shebang+cmdline case so that characters following :cmdline are interpreted as
space delimited. This is possible using 'split-string-and-unquote':
(split-string-and-unquote "1 2 3") -> ("1" "2" "3")
(split-string-and-unquote "\"1 2\" 3") -> ("1 2" "3")
(split-string-and-unquote "\"1\" \"2\" \"3\"") -> ("1" "2" "3")
Whether this is a solution, in part, depends on the perennial problem of shell
blocks: knowing what's wrong means knowing what's right.
The proposed solution assumes we intend to parse the characters following
:cmdline as space delimited and grouped by quotes. However, AFAICT, the
parsing issue makes this solution ambiguous.
Thoughts?
--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
[fn:1] AFAICT, it's due to how headers are parsed by
'org-babel-parse-header-arguments' using 'org-babel-read'. The cell "\"1 2
3\"" (corresponding to :cmdline "1 2 3") is reduced through 'string-match' to
"1 2 3". The cell "1 2 3" (corresponding to :cmdline 1 2 3), on the other
hand, passes through. The result is that :cmdline "1 2 3" and :cmdline 1 2 3
become indistinguishable. I mention this because it's easy to get confused by
this issue which, AFAICT, is independent of the one we're trying to fix. The
reported issue appears only to be related to how the result of :cmdline header
parsing is passed to the subprocess.
0001-testing-lisp-test-ob-shell.el-Test-shebang-cmdline-r.patch
Description: Binary data
0002-lisp-ob-shell.el-Add-comments-to-apply-call.patch
Description: Binary data
0003-lisp-ob-shell.el-Fix-cmdline-shebang-inconsistencies.patch
Description: Binary data
Re: #7 [[bbb:OrgMeetup]] on Wed, Apr 24, 19:00 UTC+3
On Wed, 10 Apr 2024 10:33:15 +0200 Ihor Radchenko wrote --- > Another OrgMeetup will be scheduled on the last Wednesday of February, > in two weeks. (Sorry, I cannot make it second Wednesday this time, and > even third.) Thanks for running this. I look forward to it :) -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] Prompt appears in async shell results
On Wed, 27 Mar 2024 11:59:06 +0100 Ihor Radchenko wrote --- > Over a week have passed. No objections have been raised. > Feel free to apply the patches. Done. See: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e9c288dfaccc2960e5b6889e6aabea700ad4e05a and parents. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] Prompt appears in async shell results
On Sat, 23 Mar 2024 15:16:37 +0100 Ihor Radchenko wrote --- > Yup. Tests are passing on my side as well, and I had no comments. > So, we are waiting for other interested users to test. > For at least a week, so that people who check the mailing list on > weekend have a chance to see the request. Cool. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] Prompt appears in async shell results
Friendly bump. I saw no comments on the patch part of my previous message and fear it may have been missed due to my inclusion of the lengthy and separate topic, "bug? org-babel-comint-with-output return value type". The patches fix the prompt appearing in async shell results and are ready to go, barring the concern I have about being unable to test how the fix may affect non-shell languages. > On Mon, 19 Feb 2024 12:07:55 +0100 Ihor Radchenko wrote --- > > Matt m...@excalamus.com> writes: > > > > > - Split prompt filtering and input echoing into two filters > > > + this seems to imply a =-hook= or =-functions= type implementation > > > > Not necessarily. You may just split the filter into (1) prompt > > remover; (2) body remover. The filter does not need to interact with > > comint buffer and may simply be provided a regexp/string to be removed. > > Then, the caller will be responsible to supply prompt regexp/body. > > Attached are patches which split the prompt and echo filtering into separate > functions. These are then used to refactor =org-babel-comint-with-output= > and to remove the prompt from async results. I also wrote tests, one for the > reported bug and two for the filter functions. Tests for the filter > functions are in a new file, 'test-ob-comint.el'. > > Two thoughts: > > 1. Can people please test that the changes to =org-babel-comint-with-output= > haven't broken other languages? > > The filter, which I extracted from =org-babel-comint-with-output=, works > according to the test I wrote. The test is based on ob-shell output. > However, =org-babel-comint-with-output= is used by other languages. I wasn't > sure if echoes showed up for other languages and I had no examples from other > languages to work from. The filter is untested for anything but shell, aside > from running 'make test' (which returns no unexpected errors). If no echo > argument is given, the filter simply returns the string it would otherwise > try to remove the echo from. So, I suspect the worst outcome would be that > echoes might start showing up in output. But when have my predictions ever > been right? :) -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode 0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch Description: Binary data 0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch Description: Binary data 0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch Description: Binary data 0004-lisp-ob-comint.el-Create-comint-echo-filter.patch Description: Binary data 0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch Description: Binary data 0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch Description: Binary data 0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch Description: Binary data
Re: check.sh setup.sh, nohup.out
On Tue, 19 Mar 2024 21:41:05 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > Should I do as before and use 'git revert'? Or, do we want to remove them > > completely from the history? > > Yes, git revert. Reverted https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/ Thank you for your patience and guidance :) -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: check.sh setup.sh, nohup.out
On Tue, 19 Mar 2024 18:11:50 +0100 William Denton wrote --- > Some unusual files sneaked in with commit 37cd00bb120. Thank you, good catch. My bad. The files are garbage that existed in my repo when I did 'git add .'. They should not have been added to the commit. Should I do as before and use 'git revert'? Or, do we want to remove them completely from the history? I would be happy to correct my mistake. However, I'm heading to bed now and unsure if I'll be able to work on this tomorrow or Thursday. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
bug? org-babel-comint-with-output return value type
| yes | as returned value (for :results value) | |---+--+--+--+--| | ob-haskell.el | 169 | yes, (cdr (reverse ...)) | yes | as returned value? (for :results output) hard to tell, see lines 194 and 201 | |---+--+--+--+--| | ob-octave.el | 239 | yes, (cdr (reverse ...)) | yes | as returned value (for :results output) | |---+--+--+--+--| | ob-ocaml.el | 72 | yes, whoa... | yes | as returned value (for at least :results output) | |---+--+--+--+--| | ob-R | 460 | yes, mapconcat | yes | as returned value (for :results output) | |---+--+--+--+--| It looks like the original implementation aggregated process buffer output, collecting it into a list. It then passed this list on to a function that did, as predicted, (cdr (reverse ...)) to get a string. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=dd0392a4f23b40fae7491c55aa44a2324248c103 It seems like the return type of =org-babel-comint-with-output= is not what it should be. It looks like it should be a string. Am I missing something? Obviously, changing the return type would be a big job, requiring updates to at least 10 files, and has the potential to introduce bugs. However, as far as I can tell, the return type isn't appropriate and so requires some kind of workaround, of which there are currently three. I guess the best argument I can come up with at the moment for taking on the task of changing the return type is that, assuming my understanding is correct, the current implementation is simply an inappropriate, or outdated, design choice that only adds complexity. My hope is that Babel continues to support new languages. As that happens, they will need to apply work arounds, further entrenching the design and making it harder to correct. What are people's thoughts on changing the return type of =org-babel-comint-with-output=? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] Prompt appears in async shell results
On Mon, 19 Feb 2024 12:07:55 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > - Split prompt filtering and input echoing into two filters > > + this seems to imply a =-hook= or =-functions= type implementation > > Not necessarily. You may just split the filter into (1) prompt > remover; (2) body remover. The filter does not need to interact with > comint buffer and may simply be provided a regexp/string to be removed. > Then, the caller will be responsible to supply prompt regexp/body. Attached are patches which split the prompt and echo filtering into separate functions. These are then used to refactor =org-babel-comint-with-output= and to remove the prompt from async results. I also wrote tests, one for the reported bug and two for the filter functions. Tests for the filter functions are in a new file, 'test-ob-comint.el'. Two thoughts: 1. Can people please test that the changes to =org-babel-comint-with-output= haven't broken other languages? The filter, which I extracted from =org-babel-comint-with-output=, works according to the test I wrote. The test is based on ob-shell output. However, =org-babel-comint-with-output= is used by other languages. I wasn't sure if echoes showed up for other languages and I had no examples from other languages to work from. The filter is untested for anything but shell, aside from running 'make test' (which returns no unexpected errors). If no echo argument is given, the filter simply returns the string it would otherwise try to remove the echo from. So, I suspect the worst outcome would be that echoes might start showing up in output. But when have my predictions ever been right? :) 2. Legacy code woes I'll start another thread to rant, er, discuss. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode 0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch Description: Binary data 0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch Description: Binary data 0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch Description: Binary data 0004-lisp-ob-comint.el-Create-comint-echo-filter.patch Description: Binary data 0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch Description: Binary data 0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch Description: Binary data 0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch Description: Binary data
Re: How to properly attribute authorship with Git (was Re: [PATCH] lisp/ob-shell.el: Also override explicit-shell-file-name)
On Sun, 17 Mar 2024 17:26:48 +0100 Max Nikulin wrote --- > On 17/03/2024 20:42, Matt wrote: > > On Sun, 17 Mar 2024 11:31:00 +0100 Ihor Radchenko wrote --- > > > > > > (1) revert the commit; (2) re-apply the > > > commit version with the correct author attribution. > > > > Done. > > > > 1. git revert > > 2. make changes (e.g. emacs followed by *type-type-type* or some > > incantation of 'git apply' or 'git am') > > 3. git commit --author "Arthur Author " > > In this particular case "git am" works fine and no explicit "git commit" > is necessary. Thank you for taking the time to verify the patch formatting. Sounds like a PEBKAC. I'll look into fixing it :) -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: How to properly attribute authorship with Git (was Re: [PATCH] lisp/ob-shell.el: Also override explicit-shell-file-name)
On Sun, 17 Mar 2024 11:31:00 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > First, would you like me to update the commit? If so, I will need > > guidance. The correct procedure to change the author after committing to > > remote is unclear to me. I would think it's something like sync my local > > copy with the latest remote version, update the author locally, and force > > push the change. I would then expect that the next time someone pulls, it > > would update their local with the author change. It would, however, cause > > a conflict, I think, for someone in the middle of making a change who has > > not synced with the forced push version and is trying to push their change. > > We should avoid force pushing unless something is terribly broken. > What you may do instead is (1) revert the commit; (2) re-apply the > commit version with the correct author attribution. Done. For the benefit of future me or anyone else who cares, I did: 1. git revert 2. make changes (e.g. emacs followed by *type-type-type* or some incantation of 'git apply' or 'git am') 3. git commit --author "Arthur Author " 4. git push 'git revert', in this case, basically swaps the plus and minus signs in the diff for the specified commit and submits that as a new set of changes. After applying those changes, it's possible, in this case, to proceed with "what you should have done in the first place". > > Second, I can update Worg with an explanation that it's important to > > credit authors using git's author field and how to do this. Unless I > > missed it, worg/org-contribute makes no mention of the author field. The > > version of git packaged by my distro is 2.41.0 and, AFAICT, has no -A flag > > for 'git' or 'git commit'. However, the following works on my machine > > and, I guess, is the long option form: > > > > git commit --author "Arthur Override " > > You are right. Looks like -A is just Magit shortcut. > > As for crediting authors, we may document it in > https://orgmode.org/worg/org-maintenance.html#copyright > Although, it is under "core maintainer" section. Maybe we can make a > dedicated section for maintainers on how to deal with patch submissions. I added a little section within copyright: https://git.sr.ht/~bzg/worg/commit/80152bee771b755aedfbe488497c5e4d0e7457c2 -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
How to properly attribute authorship with Git (was Re: [PATCH] lisp/ob-shell.el: Also override explicit-shell-file-name)
On Sat, 16 Mar 2024 11:11:38 +0100 Ihor Radchenko wrote --- > I notice that you changed the commit author and only referenced Aaron in > Submitted By: metadata. > > For future, we prefer keeping the original commit author in the "author" > field of the commits - this is important to keep track of the number of > changed lines for contributors without FSF copyright assignment. Thank you for letting me know this is an issue. First, would you like me to update the commit? If so, I will need guidance. The correct procedure to change the author after committing to remote is unclear to me. I would think it's something like sync my local copy with the latest remote version, update the author locally, and force push the change. I would then expect that the next time someone pulls, it would update their local with the author change. It would, however, cause a conflict, I think, for someone in the middle of making a change who has not synced with the forced push version and is trying to push their change. Second, I can update Worg with an explanation that it's important to credit authors using git's author field and how to do this. Unless I missed it, worg/org-contribute makes no mention of the author field. The version of git packaged by my distro is 2.41.0 and, AFAICT, has no -A flag for 'git' or 'git commit'. However, the following works on my machine and, I guess, is the long option form: git commit --author "Arthur Override " Third, this is at least the second time I've had issues working with a diff/patch. The reason I submitted the change the way I did is that I could not get 'git apply ' to work. I only got a useless error like "error: corrupt patch at line 10". It's not clear to me if this is an error on my end or if the patch is indeed ill-formatted. Can you confirm that the submitted patch is well-formatted? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] lisp/ob-shell.el: Also override explicit-shell-file-name
Finally had time to commit it. https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=0e2a9524dc6da8b4d60672e85aba74076baac211 Nice catch and thanks for your contribution! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]
On Tue, 12 Mar 2024 10:27:46 +0100 Ilya Konovalov wrote --- > > After some usage I found that this exact patch breaks > `org-babel-import-elisp-from-file's own separator detection. It looks > like it's better to leave it as `nil' unless specified. Thanks for taking the time to look into it. According to my notes, :sep and :separator don't work for anything beyond basic table layout. I'm not familiar with org-table and there isn't much documentation on those keywords in the manual. The :sep keyword shows up for Texinfo and LaTeX. It looks like :sep is used for export? It's not clear to me how they're supposed to work with shell block output, or if that was even a consideration at all. They were added in https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a60a83fbb65a368bb96ea6aaf1db20706b3e99c5 claiming "better handling of table in shell scripts" So, it looks like their inclusion was intentional. Maybe someone has deeper insights? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] lisp/ob-shell.el: Also override explicit-shell-file-name
On Mon, 11 Mar 2024 06:12:47 +0100 Aaron L. Zeng wrote --- > * lisp/ob-shell.el (org-babel-shell-initialize): Override > explicit-shell-file-name in addition to shell-file-name. > > When a session with shell source blocks, execution calls `shell', > which checks `explicit-shell-file-name' variable before > `shell-file-name', to determine what shell to run. If the user has > customized this variable to affect the behavior of M-x shell, > `org-babel-shell-initialize' should still run the shell specified by > the org source block's language name. > > TINYCHANGE > --- > lisp/ob-shell.el | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el > index 551c3785d..35d9e9376 100644 > --- a/lisp/ob-shell.el > +++ b/lisp/ob-shell.el > @@ -81,7 +81,8 @@ is modified outside the Customize interface." > (lambda (body params) > (:documentation > (format "Execute a block of %s commands with Babel." name)) > - (let ((shell-file-name name)) > + (let ((explicit-shell-file-name name) > +(shell-file-name name)) > (org-babel-execute:shell body params >(put fname 'definition-name 'org-babel-shell-initialize)) > (defalias (intern (concat "org-babel-variable-assignments:" name)) > -- > 2.42.0 Thank you for your report and the patch. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] ob-shell async incorrect table and value results
On Fri, 08 Mar 2024 10:25:09 +0100 Ihor Radchenko wrote --- > Confirmed. > This is because `ob-shell-async-chunk-callback' never returns a list. > > In `org-babel-sh-evaluate', there is a part that is responsible for > detecting the result type: > > (when (and results value-is-exit-status) > (setq results (car (reverse (split-string results "\n" t) > (when results > (let ((result-params (cdr (assq :result-params params > (org-babel-result-cond result-params > results > (let ((tmp-file (org-babel-temp-file "sh-"))) > (with-temp-file tmp-file (insert results)) > (org-babel-import-elisp-from-file tmp-file) > > This part is skipped for :async results. Thanks for looking at it. I'm still working on "[BUG] Prompt appears in async shell results" (https://list.orgmode.org/18dbc1f273c.11687295c1395973.3345700621594100...@excalamus.com/T/#u) and will circle back to this when that's done. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
[BUG] ob-shell async incorrect table and value results
#+name: sync table #+begin_src sh :session *test* :results table echo "hello world" #+end_src #+RESULTS: | hello world | #+name: async table #+begin_src sh :session *test* :results table :async t echo "hello world" #+end_src #+RESULTS: : hello world #+name: sync value #+begin_src sh :session *test* :results value echo "hello world" #+end_src #+RESULTS: : 0 #+name: async value #+begin_src sh :session *test* :results value :async t echo "hello world" #+end_src #+RESULTS: : hello world : 0 -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] doc/org-manual.org (Checkboxes): move section 'Checkboxes' from 'TODO Items' to 'Plain Lists'
On Wed, 28 Feb 2024 13:17:54 +0100 Ihor Radchenko wrote ---
> Matt m...@excalamus.com> writes:
>
> > I had responded in favor here:
> > https://list.orgmode.org/18d4cf138a6.10fb9c6702382826.5023996590743168...@excalamus.com/
>
> Did I miss something...
Yes, it appears there's a little bit of a mix up because of a bad subject line.
There's also some nitpicky logic. The tl;dr is, we shouldn't use the current
patch.
>...or did you not provided arguments /why/ the
> section should be moved? I need to understand what kind of improvement
> it would provide to the manual.
I didn't know that's what you were looking for. When I said, "I had responded
in favor..." it was in response to your prior message which said,
> No comments arrived within one month.
This is incorrect albeit understandable. I had responded and, therefore, there
were not "no comments." However, it looks like I responded in the wrong
thread! ("Re: [PATCH] doc/org-manual.org: Checkboxes, add checkbox states
examples") That's my bad!
Regarding reasoning, I'm in favor of the move for the reasons Sławomir gave:
> Because checkbox can only exist in a plain list, as a plain list feature.
> So the section should be under 'Plain Lists' heading not under 'TODO Items'.
The issue is checkbox usage is split between different sections of the manual.
You had responded to this by saying,
> Both arrangements are logical. Checkboxes are useful as a complement to
> TODO items. And they are also indeed a plain list feature.
It seems we're all agreed the proposed arrangement is logical and that the
issue is valid. I don't think it needs extra justification.
Conceding this point, which we all appear to, the issue becomes which
arrangement we should use?
Originally, we were reluctant to move the Checkboxes section only because
Carston had moved it previously. Unfortunately, we don't know *why* Carston
moved it. This isn't a very contestable justification.
My latest reply gives a specific reason to *not* apply the current patch. That
is, to *not* move the Checkbox section as-is. The reason is:
> > The Checkboxes section is written assuming the reader knows what
> > Properties are. The GNU documentation guidelines suggest writing as
> > though readers have read from the beginning [fn:1] [fn:2]. That is,
> > unless introducing a concept, only use concepts that have already been
> > explained. Properties are introduced in Section 2.7 and Checkboxes is
> > currently 5.6. The proposal is to move Checkboxes to 2.6.1 *before*
> > properties are introduced. This is a problem.
I suspect this is the reason Carston moved the section. Regardless, it's a
valid reason to have moved it and gives us clear criteria for why we can't
apply the patch. It also gives us a precise target for what would need to be
fixed in order to resolve the issue of checkbox usage being split between
sections by moving the Checkbox section.
> We start talking about properties as early as in 2.2.2 Initial
> visibility and in many other places. Re-ordering the manual to avoid
> referring to future concepts would entail a major rewrite.
I believe that arranging documentation in conceptual order is always a
worthwhile goal. It's obviously better to have concepts introduced in order.
It's also completely reasonable to not want to do that work right now. I'm not
willing to at the moment and it sounds like you aren't either. That's okay.
If Sławomir or someone else wants to, I still think the original point is
valid. However, the proposed patch, moving the section as-is, won't work
because it (re)introduces problems with conceptual ordering.
If someone wants to suggest a patch which resolves the issue of checkbox usage
being split between sections which preserves, or improves, the conceptual
order, I'd be happy to assist.
--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
Re: [PATCH] doc/org-manual.org (Checkboxes): move section 'Checkboxes' from 'TODO Items' to 'Plain Lists'
On Mon, 26 Feb 2024 09:56:44 +0100 Ihor Radchenko wrote --- > Ihor Radchenko yanta...@posteo.net> writes: > > > I do not have a strong opinion here though. If more people chime in here > > and support re-arrangement, I will not oppose. > > No comments arrived within one month. Does not look like people care > much about this. So, let's preserve the existing structure of the > manual. > > Canceled. I had responded in favor here: https://list.orgmode.org/18d4cf138a6.10fb9c6702382826.5023996590743168...@excalamus.com/ However, before moving the section we'd need to find a way to introduce properties before the checkboxes section. The Checkboxes section is written assuming the reader knows what Properties are. The GNU documentation guidelines suggest writing as though readers have read from the beginning [fn:1] [fn:2]. That is, unless introducing a concept, only use concepts that have already been explained. Properties are introduced in Section 2.7 and Checkboxes is currently 5.6. The proposal is to move Checkboxes to 2.6.1 *before* properties are introduced. This is a problem. [fn:1] https://www.gnu.org/prep/standards/standards.html#Documentation [fn:2] https://www.fsf.org/licensing/gnu-press/GNU-Press-styleguide.pdf -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b
On Sat, 24 Feb 2024 17:42:54 +0100 Bruno Barbier > I'll publish a branch soon; it will be a major rewrite of my current > proposal. It should be less confusing and, I hope, address some of your > comments. > > Thanks again for your questions and feedbacks, You're welcome and thanks for sharing your ideas. Any lack of comments from me recently is just limited time and trying to focus on maintenance. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b
On Mon, 19 Feb 2024 01:31:42 +0100 Jack Kamm wrote --- > I don't remember the details, but my past self [1] thought it would be > good to find a way to replace `process-file' with `make-process' in > `org-babel--shell-command-on-region' or `org-babel-eval', and it seems > you are thinking along those lines with `my-org-babel-eval-async'. Hope > you're able to make progress on this and get the improvements into > ob-eval.el eventually. > > [1] https://list.orgmode.org/871rczg7bi@gmail.com/#t AFAIK, aside from appending "&", =make-process= is the only way. It would probably make sense to continue using =shell= though. If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer? On Mon, 19 Feb 2024 10:03:36 +0100 Ihor Radchenko wrote --- > Note that we already have a WIP an asynchronous API in the works. > Check out `org-async-call' in > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > If possible, we should reuse it. :O - What's the status? - Anything I could help with? - Are there any notes or supplements that go along with it? > I have several general questions: > > - what if append/prepend to result asynchronously? > - what if we replace the result, call async evaluation two times, and they > arrive in opposite order (first called is calculated after the second)? > - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How > will it interact with asynchronous evaluation? What if we have multiple > simultaneously arriving error notifications? > Example: try running > #+begin_src bash > idontexist > #+end_src Are these open questions for the `org-async-call' implementation? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b
another way persistence may be achieved? Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. All that is mainly academic. The idea I think most interesting is using a single primitive to handle all evaluation. It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached). It works as follows. I defined a new Babel "language" called "blub". Blub could be a shell, python, ruby, whatever. I wanted to test that the implementation could work with different interpreters or compilers. Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out. Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. Curious to hear other people's thoughts! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode ob-blub-test.org Description: Binary data ob-blub.el Description: Binary data
Re: [BUG] Prompt appears in async shell results
On Mon, 05 Feb 2024 15:00:18 +0100 Ihor Radchenko wrote --- > Yes. The right fix would be extracting the filter from > `org-babel-comint-with-output' and re-using it. This message documents my progress so that I may switch focus to Bruno's patches in "Asynchronous blocks for everything" attention (https://list.orgmode.org/65cfa0d8.050a0220.cb569.c...@mx.google.com/T/#u). Attached is a failed attempt at extracting the filter in =org-babel-comint-with-output=. It tries to extract the filter more-or-less directly: 1. take the filter code from =org-babel-comint-with-output= and put it into a separate function, =org-babel-comint-process-output-filter= 2. call =org-babel-comint-process-output-filter= from =org-babel-comint-with-output= and =org-babel-comint-async-filter= The unmodified =org-babel-comint-with-output= has a comment that says, "Filter out prompts". This is misleading. The filter code does two things: removes prompts *and* removes echoed input. The problem is the filter which removes echoes uses the body of the source block. It's unclear how to give =org-babel-comint-async-filter= the block body. =org-babel-comint-async-filter= is a =comint-output-filter-function= which receives a single input, "a string containing the text as originally inserted" in the comint buffer. Thoughts: - Split prompt filtering and input echoing into two filters + this seems to imply a =-hook= or =-functions= type implementation + where could input echo filter go? Where has access to the block body? - creating a generic prompt filter duplicates =ob-shell-async-chunk-callback= or, more fundamentally, =org-babel-comint-async-chunk-callback= - What would it take to consolidate output filtering? In addition to prompt filtering and input echo filtering, ob-shell filters the =org-babel-sh-eoe-indicator=. I'm sure there's other filtering that happens on block output. Wouldn't it be nice if that were in a single place, like right before results are inserted? Please feel free to provide feedback and suggestions. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode v01-refactor-filter.diff Description: Binary data
Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build
If I followed correctly, the topic switched to discussing async generally within Babel. I've started a new thread accordingly. On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier > FWIW, I've been trying to use asynchronous blocks for everything, not only > the source blocks that are based on the comint mode. I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with. > I think it would be good if ob-core itself could provide an asynchronous API. Fortunately or unfortunately, depending on how you look at it, Babel already does. The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default. For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API: - "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. - "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. - "Synchronous" means that execution prevents the user from editing the document while results are obtained. - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained. > I've modified my Org so that it does have such an API. This is work in > progress; let me describe it. > > I've modified ob-core itself to allow asynchronicity. In the asynchrosous > case, instead of calling: > > (org-babel-execute:LANG body params) > > I'm calling: > > (org-babel-schedule:LANG body params handle-result) > > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with > the result (or the error) when it is known; `handle-result' takes care to > call `org-babel-insert-result' at the correct place (and > `org-babel-insert-result' is only called with a real result). > > While the execution is pending, I'm using the same technique that Org is > using when a source block is being edited: the result is left untouched, but > below an overlay. The overlay is used to know where to insert the result > and to display the status/progress of the execution. If the file is closed > and the execution fails, nothing is lost, the old result is still available. The use of the overlay is a really cool idea! I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise. > If that technique looks safe enough and interesting, I can prepare a set of > patches so that we can discuss it further and, maybe, add it in Org. Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
[BUG] Prompt appears in async shell results
* [BUG] Prompt appears in async shell results ** Minimal reproducible example #+begin_src emacs-lisp (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src #+begin_src sh :results output :session *test* :async t cd /tmp echo "hello world" #+end_src #+RESULTS: : org_babel_sh_prompt> hello world or #+begin_src sh :results output :session *test* :async t # comment # comment #+end_src #+RESULTS: : org_babel_sh_prompt> org_babel_sh_prompt> or #+begin_src sh :results output :session *test* :async t # print message echo \"hello world\" #+end_src #+RESULTS: : org_babel_sh_prompt> "hello world" Interestingly, this returns without the prompt: ,#+begin_src sh :results output :session *test* :async t echo "hello" echo "world" #+end_src #+RESULTS: : hello : world ** Test Here's a test that checks one of the MREs: #+begin_src emacs-lisp (ert-deftest test-ob-shell/session-async-removes-prompt-from-results () "Test that async evaluation removes prompt from results." (let* ((session-name "test-ob-shell/session-async-removes-prompt-from-results") (kill-buffer-query-functions nil) (start-time (current-time)) (wait-time (time-add start-time 3)) uuid-placeholder) (org-test-with-temp-text (concat "#+begin_src sh :session " session-name " :async t # print message echo \"hello world\" ,#+end_src") (setq uuid-placeholder (org-trim (org-babel-execute-src-block))) (catch 'too-long (while (string-match uuid-placeholder (buffer-string)) (progn (sleep-for 0.01) (when (time-less-p wait-time (current-time)) (throw 'too-long (ert-fail "Took too long to get result from callback")) (search-forward "#+results") (beginning-of-line 2) (if (should (string= ": hello world\n" (buffer-substring-no-properties (point) (point-max (kill-buffer session-name) #+end_src ** Thoughts A quick fix is: #+begin_src diff modified lisp/ob-shell.el @@ -289,7 +289,7 @@ See `org-babel-comint-async-indicator'.") (defun ob-shell-async-chunk-callback (string) "Filter applied to results before insertion. See `org-babel-comint-async-chunk-callback'." - (replace-regexp-in-string comint-prompt-regexp "" string)) + (replace-regexp-in-string org-babel-sh-prompt "" string)) (defun org-babel-sh-evaluate (session body params stdin cmdline) "Pass BODY to the Shell process in BUFFER. #+end_src I'm not sure this is the best way. There are two ways I can think to look at it: how text is passed to the process and what's done with it afterward. Regarding how text is passed to the process: Blocks without :session and :async are sent via =process-file=. Blocks with :session, including those with :async, are inserted into a process buffer and sent to the process with =comint-send-input=. The details differ, but I think the general concept holds: a chunk of text is inserted into the process buffer and that chunk is then sent to the process. This is in contrast to successive insert-send pairs. AFAICT, there's no major difference in how text is passed to the process between :session only and :session with :async. Regarding how process results are handled: AFAIU, =process-file= interfaces directly with the process and no terminal emulation is involved. So, there's no prompt to worry about. Blocks with only :session go through =org-babel-comint-with-output= which filters out the prompt (https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ob-comint.el#n110). Async block results go through =org-babel-comint-async-filter= which doesn't filter results (https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/lisp/ob-comint.el#n212). It seems to me that we should extract the filter from =org-babel-comint-with-output= and use it in both =org-babel-comint-with-output= and =org-babel-comint-async-filter=. Thoughts? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Suggested manual changes - section 7.2 Special Properties
Ihor, I think this would be a good topic to go over in the monthly meet up, especially if no one is able to follow up further on this by then. On Mon, 29 Jan 2024 04:37:33 +0100 Tommy Kelly wrote --- > It’s here: https://www.reddit.com/r/orgmode/s/3viPfF78Lr > In a nutshell, I was trying to understand how to have one of the columns in > column mode be the creation date of each headline. I thought that because I > almost always insert a clock entry when I create (using org-capture) a new > item then I already had the required timestamp — the first one in the clock > entry. However, it turns out that timestamp isn’t usable, and so the thread > was me looking for help in figuring out the best way to do it. Thank you very much. Unfortunately, my knowledge of this part of Org isn't enough to give meaningful feedback at the moment. I should also say that I appreciate the effort and thoughtfulness you've put into this. Definite bonus points :) If I'm able to find the time to read up on this part of Org, I'd be happy to work with you further on this. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Suggested manual changes - section 7.2 Special Properties
Thank you for your suggestions. Unfortunately, I'm not yet familiar with this part of Org. Can you give a link to the Reddit thread where this was discussed? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] doc/org-manual.org: Checkboxes, add checkbox states examples
On Thu, 25 Jan 2024 23:33:53 +0100 Ihor Radchenko wrote --- > I do not have a strong opinion here though. If more people chime in here > and support re-arrangement, I will not oppose. I think Sławomir makes a good case for moving it. What concepts are used and where are they introduced in the manual? - [X] TODO + Mentioned in Section 1.5 [[https://www.gnu.org/software/emacs/manual/html_node/org/Conventions.html][Conventions]] - [ ] Properties + Mentioned in Section 2.7 [[https://www.gnu.org/software/emacs/manual/html_node/org/Drawers.html][Drawers]] + Explained in Section 7.1 [[https://www.gnu.org/software/emacs/manual/html_node/org/Property-Syntax.html][Property Syntax]] - [X] Headlines + Explained in Section 2.1 [[https://www.gnu.org/software/emacs/manual/html_node/org/Headlines.html][Headlines]] TODOs, Properties, and headlines are talked about in the Checkboxes section. Section 2.7 introduces properties. We're proposing inserting Checkboxes into position 2.6.1. This would mean a concept used wouldn't be defined yet. Out of time tonight so I don't have any suggestions for how to address it yet. > In fact, Checkboxes section of the manual was previously under Document > structure. This was changed in release_4.45. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6a9c670d I checked the mailing list around that time for any issues that might mention checkboxes and didn't find anything. On 2008-01-27, they were discussing results of a recent survey. I was able to find a copy from the week after the change. I didn't see anything that would have motivated the change. Here's the archived survey. It's interesting. https://web.archive.org/web/20080202231951/https://orgmode.org/survey.html -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Appending results from babel block
On Sat, 27 Jan 2024 03:20:37 +0100 Ken Mankoff wrote --- > I'd like to append results from a babel block, but without the leading ":" > that sometimes appears. Is this possible. I'm using v9.7-pre. AFAIK, the only ways are to use 'raw' or something that wraps the result, like :wrap or 'drawer'. As you observed, using 'raw' will "prepend" the latest output because Org doesn't consider the previous output a result. I'm curious, what broader goal are you trying to achieve? How does the leading ":" stop you? > Weirdly, > > :results append drawer > > Appends result #2, but then inserts all results after the first. I'm not sure what you mean. However, trying it, I see bunches of "results-end" groups. I assume this is what you see, too? #+BEGIN_SRC bash :results append drawer echo "$(date)" #+END_SRC #+RESULTS: :results: Sat Jan 27 09:11:21 PM CET 2024 :end: :results: Sat Jan 27 09:11:21 PM CET 2024 :end: :results: Sat Jan 27 09:11:21 PM CET 2024 :end: > Is this last behavior a bug? That's a good question. Skimming the manual, I didn't see mention of it, although I may have missed it. The behavior I see doesn't match my expectation. I would expect to see: #+RESULTS: :results: Sat Jan 27 09:11:21 PM CET 2024 Sat Jan 27 09:11:21 PM CET 2024 Sat Jan 27 09:11:21 PM CET 2024 :end: -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [BUG] conda doesn't work in ob-shell sessions
On Fri, 26 Jan 2024 01:42:59 +0100 Jack Kamm wrote --- > Second version of the patch works on my test example now. The initial > block hangs for a few seconds but then finishes. Subsequent blocks seem > to work without any noticeable hanging. I was able to confirm this as well. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Patch to allow adjusting latex fragment display scale factor
Timothy writes: > Hi Matt and Ihor, > I must admit that with the new system, I don’t see much value in accepting a > function: we now scale the previews based on a combination of the :scale > parameter and their actual LaTeX display size. Yep, we can close/cancel this patch. I had been using that feature to permit rescaling latex fragments for font size changes, but that's already handled by the current implementation. Matt
Re: [PATCH] ob-shell: consistent prefix
I'm going to defer these changes for now. It seems like it would be easier to handle renaming one function/variable at a time. Renaming requires creating an alias mapping between the old name and the new one. Any subsequent refactoring would require updating the mapping. It seems better to do any non-naming refactoring and then handle aliasing and obsoleting in order to minimize the number of changes that need to happen. Also, renaming now invalidates any references, like in my notes or documentation. I have questions regarding development that I'll ask in a different subject thread. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Backwards compatibility (was: Re: [PATCH] ob-shell: consistent prefix)
On Tue, 16 Jan 2024 14:10:00 +0100 Ihor Radchenko wrote --- > I do not mind changing the names, except that we must not break > backwards compatibility. In particular, the non-private function > and variable names that were present in the latest Org stable > release must be either supplied with an alias or declared > obsolete. Otherwise, the third-party code using the old names will > be broken. How far does backwards compatibility extend with regard to Org itself? For the next version, for all time, or something else? The rule I infer from your comment is, "renamed symbols must be aliased." This implies that any arguments associated with the symbol are the same after the name change. How do we handle changes in function APIs? I'm thinking of something like =org-babel-execute:shell= and =org-babel-sh-evaluate= where it would make sense to refactor the =stdin= and =cmdline= parameters. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [Pre-PATCH] Overhaul of the LaTeX preview system
Matt Huszagh writes: > Another minor issue I've encountered. Using different delimiters results > in images of slightly different sizes. > > For example, in a file containing: > > ``` > Using first delimiters: \(F=ma\). > > Using other delimiters: $F=ma$. > ``` > > The snippet with dollar sign delimiters is slightly larger than the one > using \(\). The following code, where you compute the ascent value, seems relevant to this observation: ;; The baseline seems to tend to sit slightly ;; lower than it should be, and a very mild ;; bias seems to improve the visual result. ;; From testing with a collecting of LaTeX ;; maths fonts (cm, cmbright, arev, pxfonts, ;; notomath, nextxsf, eulervm) decreacing the ;; depth measurement by 0.02pt in the baseline ;; calculation seems to work well. ;; I have yet to come across any situation ;; where this results in a negative depth, ;; however we may as well ensure that never ;; occurs. (round (* 100 (- 1 (/ (max 0.0 (- depth 0.02)) height When using \(\) delimiters, using a depth adjustment of 0 (instead of 0.02) looks correct to me. I checked this by blowing up the fragment with a very large scale factor (eg 10) and then baseline misalignments become more obvious. This is how I ensured my baseline computation was correct when I wrote that patch aligning the baseline several years ago. I /think/ that's a valid method, and I've been using my code for the last couple years and the baseline has always looked correct. Anyway, can you explain more why you came to the conclusion of that slight depth adjustment? Are you using $$ delimiters? That also appears to produce other visual imperfections. For $F=ma$, I see the bottom of the "m" and "a" cut off slightly. I wonder why different delimiters produce different results. I used slightly different settings for dvisvgm in my implementation (including --exact-bbox). I wonder if that has any relevance... I also used a different document class - standalone in preview mode. Now that I think about it, IIRC that was to address another corner case I ran into, which is that for large images, article will crop it before it gets to dvisvgm. It's been a while since I did this and my memory is hazy, but I think that's why I used standalone. I can try to investigate that with a minimal issue. Matt
Re: [Pre-PATCH] Overhaul of the LaTeX preview system
Timothy, Another minor issue I've encountered. Using different delimiters results in images of slightly different sizes. For example, in a file containing: ``` Using first delimiters: \(F=ma\). Using other delimiters: $F=ma$. ``` The snippet with dollar sign delimiters is slightly larger than the one using \(\). Matt
Re: [Pre-PATCH] Overhaul of the LaTeX preview system
Hi Timothy,
Thanks for your work on this.
I've run into an issue related to the change you made in
209e5f5f4047a34db27c3b5dff4077bb1da9ceed that makes org-latex-compile
asynchronous.
This breaks the org-babel-latex-pdf-svg-process functionality I
introduced in commit ae35a345903c640397a8d29812112d72a9f8494a.
To reproduce the issue, you can create an org file with the following
single latex source block:
#+header: :file "out.svg"
#+begin_src latex :hidden
\begin{equation}\tag{1.6.3.1}
\mathbf{F} = q \mathbf{E},
\end{equation}
#+end_src
If you execute this with C-c C-c, it complains about the PDF file not
being present. The problem can be found starting in line 182 of
ob-latex.el of your current (as of this writing) feature branch commit
8384289762b41b26e75d2e80c37ec84bcc552d32. The issue is that
org-babel-latex-tex-to-pdf (which delegates to org-latex-compile)
creates the PDF after org-compile-file (in line 185) tries to generate
an SVG file from it. In fact, I was initially confused because I
couldn't reproduce the issue with edebug (obviously, because that gave a
long enough delay for the file to be created before attempting the svg
creation).
org-latex-compile seems like it could be a dangerous function to change
to be asynchronous, as it's used in a lot of places and its very
possible that other people (like myself) were relying on it being
synchronous when they made some change that used it. I haven't looked at
its other invocations, but I wouldn't be surprised if this breaks other
things too. I hesitate to suggest alternatives because I don't know
enough about your use case. But, maybe you could achieve what you want
through a new org-latex-compile-async function, or add a nondefault
option to org-latex-compile?
Matt
Re: Patch to allow adjusting latex fragment display scale factor
Hi, Bringing this back up. Ihor, I've added you to the thread directly (hope that's ok). Any thoughts? I have not yet updated it for the most recent changes to main, but I can do that. Matt On Sun, Oct 10, 2021 at 9:40 PM Matt Huszagh wrote: > > Matt Huszagh writes: > > > I've created a patch to allow adjusting the scale factor used for inline > > latex image fragments. This involves a customizable variable that can > > either be set to a scale factor (defaults to 1.0) or a function that > > evaluates to a scale factor. > > > > This feature is in addition to the existing scale factor adjustment > > capability provided by `org-preview-latex-process-alist' through > > `:image-size-adjust'. Wherease image-size-adjust performs scaling at the > > time of image generation, the new change performs it during > > display. This can lead to significant time saving and suffers no loss of > > quality for vector graphics. > > > > As an example of use, I have several functions for changing frame > > scaling. I've added > > > > (if (eq major-mode 'org-mode) > > (progn > > (org-clear-latex-preview) > > ;; 16 corresponds to the C-u C-u arg prefix. > > (org-latex-preview 16))) > > > > to these functions so that changing the frame scaling also > > correspondingly changes the latex preview/fragment scaling to match the > > new size of the surrounding text. Because of this new feature, this > > change is effectively instantaneous for reasonably numbers of > > overlays. Obviously, something similar could be done for > > `text-scale-adjust' (e.g., through `advice-add'). > > > > Feedback appreciated. > > Apologies, the patch I sent is slightly wrong. The line numbers also > reflect an earlier patch I made. Here is a corrected version. >
Re: [Patch] Align baseline of latex fragments and surrounding text
Ihor Radchenko writes: > Matt Huszagh writes: > >> These are nice ideas, and ones I hadn't considered. Thanks for the >> suggestions! I'll think about it a bit and may modify the patch >> accordingly. > > May I ask if you further considered changing the patch? Hey Ihor, I'm sorry for the late reply. I'm just now looking at several patches I submitted a few years ago and left hanging... The answer to this question is yes I did, and IIRC got most of the way through implementing them but ran into a largely unrelated issue that I didn't, at the time, find a solution for. It looks like I still have the branch with those changes and I can take a look back at it and try to resolve it. However, I noticed that there appears to be another (large) patch in the works that also does this (email subject line "Overhaul of the LaTeX preview system"). That's a long thread and I haven't followed it. Is there still scope for my change? Is it worth getting this in? Matt
[PATCH] ob-shell: consistent prefix
The attached patch makes names in ob-shell.el use the same prefix "org-babel-shell-" (along with a few similar minor consistency updates). Without the patch, three naming conventions exist: - org-babel-shell - org-babel-sh - ob-shell <- that's my fault :) The Org Babel shell library was previously called "ob-sh.el" and later changed to "ob-shell.el". When this happened, the meaning of the "sh" changed. When the library was called "ob-sh.el", "sh" meant a generic shell. When the library was renamed "ob-shell.el", "shell" took on the role of a generic shell and "sh" became a reference to /bin/sh. The current "sh" names are generic and not specific to "/bin/sh" or something like the Bourne Shell. The patch updates all "sh" names to "shell". When I submitted my async changes, I introduced an "ob-shell" prefix. Emacs names typically begin with the library name and I just continued in that style. Org is different, however, because it's the "org-" library. For sub-libraries, like Babel, the convention is "org-babel--". ob-shell.el is unusual in that it supports multiple languages. However, ob-C.el also supports multiple languages and uses "org-babel--". I find there's no confusion using this style. FWIW, I found it confusing when I first started learning the Org code base that the naming convention is at the "org-" level and not at the library level (for example, "ob-shell-"). A large part of my confusion, I think, was simply inconsistencies within ob-shell. It's late for me, so rather than commit in a rush, I'm posting this for review/comment. The tests pass and I think I preserved all the API function names (like org-babel-execute:template). Thoughts? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode 0001-make-name-prefixes-consistent.diff Description: Binary data
Re: ob-shell: proposal to remove "posh"
On Sat, 13 Jan 2024 19:00:16 +0100 Ihor Radchenko wrote --- > Since posh is POSIX-compliant, generic PS1/PS2 variables should work, and we > do not need a special entry in `org-babel-shell-set-prompt-commands' - > we can remove "posh" record from there. > > I'd rather keep the value in `org-babel-shell-names' though. It should > do no harm. Done https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/ -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: ob-shell: proposal to remove "posh"
On Thu, 11 Jan 2024 21:30:59 +0100 Matt wrote ---
> The change on August 26, 2022 should have caused a breaking error for
> someone using the "Policy-compliant Ordinary SHell." The prompt for "posh"
> in "org-babel-shell-set-prompt-commands" is valid PowerShell syntax (AFAIKT)
> and invalid bash/dash syntax:
>
> function prompt { "org_babel_sh_prompt> " }
>
> It's not clear to me what this would do in the "Policy-compliant Ordinary
> SHell."
It would cause an error.
I was able to build the "Policy-compliant Ordinary SHell" on Guix using:
#+begin_src guile
(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(gnu packages autotools)
(gnu packages perl)
((guix licenses) #:prefix license:))
(define-public posh
(package
(name "posh")
(version "0.14.1")
(source (origin
(method url-fetch)
(uri (string-append
"https://salsa.debian.org/clint/posh/-/archive/debian/;
version "/posh-debian-" version ".tar.gz"))
(sha256
(base32
"070xnn996cjnc5yzp5819y36sgfikkrplhri4kx5r36h1fmp641d"
(native-inputs (list autoconf automake perl))
(build-system gnu-build-system)
(home-page "https://salsa.debian.org/clint/posh;)
(synopsis "Policy-compliant Ordinary SHell")
(description
"Policy-compliant Ordinary SHell
posh is a stripped-down version of pdksh that aims for compliance with
Debian's policy, and few extra features.")
(license (list license:gpl2+
posh
#+end_src
The result is:
ahab@pequod /gnu/store/64wiqdp9lqjgsz0jg1v1sq2b3afincrb-posh-0.14.1/bin$ ./posh
$ function prompt { "org_babel_sh_prompt> " }
./posh: function: not found
This is expected because "function" is not a keyword in the "Policy-compliant
Ordinary SHell." According to the man page:
#+begin_quote
name () command
Defines the function name. See Functions below. Note that redirections
specified after a function definition are performed whenever the
function is executed, not when the function definition is executed.
-- https://manpages.debian.org/bookworm/posh/posh.1.en.html
#+end_quote
So, yes, indeed, commit a35d1636 introduced a breaking change for the
"Policy-compliant Ordinary SHell."
--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode
Re: ob-shell: proposal to remove "posh"
On Fri, 12 Jan 2024 11:04:33 +0100 Morgan Willcock wrote --- > As I understand it, the code for the cmd.exe shell is not released > anywhere. Ah, okay. I clearly didn't search too hard for it. > That said, I imagine Wine does contain its own implementation of > cmd.exe, although I wouldn't expect it to be 100% compatible with the > real thing. That's an interesting idea. Might be a "good enough" work around. The VM image I mentioned says, "The VM will require a minimum of 8GB of RAM and at least 70GB of disk space". XD -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
ob-shell: proposal to remove "posh"
ell" and add languages for "pwsh" or "powershell". This would need to be communicated PowerShell users. I don't know what other keyword we could use for the "Policy-compliant Ordinary SHell". We could keep "posh" for PowerShell, drop all "support" for the "Policy-compliant Ordinary SHell", and commit to PowerShell only. This would need to be communicated to the "Policy-compliant Ordinary SHell" users. In any case, we need to communicate our decision, in ORG-NEWS, through our mailing list communications, and definitely by updating the comment. The cost of keeping PowerShell means maintaining it and whatever weirdo stuff comes with it. I suggest we make the "t" in =org-babel-shell-set-prompt-commands= a catch-all for anything with PS1/PS2 (anything POSIX-y?) and remove the "posh" language header. The "t" case should handle bash, dash, and (I suspect) the "Policy-compliant Ordinary SHell". This has the potential to break someone using "posh" for PowerShell. However, I see no indication of anyone seriously using it. We've said for years we don't support it. The only places I'm aware of saying we support PowerShell is the code comment and a few people on the mailing list. Moreover, a "breakage" happens anyway, for either "Policy-compliant Ordinary SHell" users (as currently likely exists) or PowerShell users (if we give a new keyword). If we're going to have to put a work-around or message out there anyway, I figure let's see about removing any semblance of supporting PowerShell until we talk it over. Fortunately, when I wrote the Worg page, I put "the "Policy-compliant Ordinary SHell" for "posh." So, the only place someone might think that PowerShell is supported is from the code comment or from the mailing list. ** PowerShell or cmd.exe? I'm not claiming that people haven't wanted PowerShell support. It's been discussed a few times on the list. It always reduces to "it would be nice, however we lack a developer or the equipment, so sadly no." As current ob-shell maintainer, here's how I see it: I like the idea of supporting PowerShell. I like the idea of supporting cmd.exe much, much more. Both are associated with a non-free system. Providing support (even partially) for non-free systems is good because it provides an opportunity to teach people about software freedom. AFAIKT, both PowerShell and cmd are MIT licensed: - https://github.com/microsoft/terminal - https://github.com/PowerShell/PowerShell The thought of compiling either for a GNU system is...ugh. But maybe someone else has gotten them working? Otherwise, it looks like Microsoft distributes a developer VM image of Windows. All together, this means there's no *technical* barrier preventing us from running (and hence developing for) PowerShell or cmd. It's important to note that Emacs defaults to cmdproxy.exe on Windows. Using "shell" in Babel on Windows returns from cmd.exe (through cmdproxy.exe). I believe it's best to support the defaults first. Because it's the default, I've always used cmd when using Emacs on Windows. I would personally much prefer support for cmd.exe over PowerShell. That is, I'd be happy to work on PowerShell stuff after cmd stuff and both after cleaning up ob-shell, ob-comint, and ob-eval. Thoughts overall? -- Matt Trzcinski Emacs Org maintainer (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v2] doc/org-manual.org (Summary): Clarify the Org markup is human-readable
On Sat, 06 Jan 2024 13:53:28 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > As a non-native speaker, I have some difficulties understanding > > > "legible" meaning in this context. > > > > How does it come across to you? > > Mostly a confusion between "legible" and "eligible" (I know they have > completely different meanings). > > Also, "legible" is not used as frequently as simple "readable", for > example. See "Trends" data in > https://www.collinsdictionary.com/us/dictionary/english/legible vs. > https://www.collinsdictionary.com/us/dictionary/english/readable > > So, there are chances that non-native speakers are simply not familiar > with the meaning of "legible", making it more likely to confuse with > similarly-looking "eligible". Thanks, I understand. That makes sense. > > What do you mean by "human-readable"? > > This\_ > > > Readability also means that what's written can be inferred without > > referencing other sources. It's readable like "apparent, easily > > understood". I think this is probably what you're thinking of. Gotcha. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v2] doc/org-manual.org (Summary): Clarify the Org markup is human-readable
On Fri, 05 Jan 2024 14:04:17 +0100 Ihor Radchenko wrote --- > Thanks for the comments! > I am attaching the next iteration of the patch. Looks good > Matt m...@excalamus.com> writes: > > > +Org Mode is an authoring tool and a to-do lists manager for GNU Emacs. > > +It uses a legible plain-text notation to show formatting, structure, > > +relationships. Anyone able to edit text can write using Org. Anyone > > +able to read text can view it. > > As a non-native speaker, I have some difficulties understanding > "legible" meaning in this context. How does it come across to you? > Also, "anyone able to read text can > view it" would be a bit of a bold claim, I think. So, I went with more > moderate wording. Good call. That whole sentence is a mess. When I wrote "anyone able to read text", I intended to say something like "if a computer can render the text". However, that reduces to "if a computer can render it, then you can view it" which isn't very impressive. When you say Org markup is "human-readable", I see that coming from it being plain-text. It requires no special tools. It's readable like "accessible, easily obtained". I'm contrasting that with Jupyter Notebooks. I think this aspect of readability is something we should emphasize. Readability also means that what's written can be inferred without referencing other sources. It's readable like "apparent, easily understood". I think this is probably what you're thinking of. What do you mean by "human-readable"? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Comments on Summary section of Org manual (was: [PATCH] doc/org-manual.org (Summary): Clarify the Org markup is human-readable)
On Wed, 03 Jan 2024 14:41:29 +0100 Ihor Radchenko wrote --- > I do not see much benefit changing "todo list" to "to-do list". Both are > clear and both are grammatically correct. I said "TODO" to "to-do". I was reacting to it being all caps. I'm sorry I didn't say that. When it's all caps, TODO is a keyword. Otherwise, it's prose. The following are different (AFAIK): * TODO Finish this headline * todo Finish this headline * to-do Finish this headline Agreed, it's a minor detail. > > Regarding "markup language," that reads to me like programmer jargon. > > What does it mean and why should someone care? Again, who are we > > writing to? A markup language is a notation for formatting, > > structure, and relationships. I think it would be best to directly > > say that. > > What about "plain text file format"? I like that it's less technical. I worry that it's less precise. I still think a word like "notation," that doesn't derive from computer culture, has a greater chance of being meaningful to people from other domains, like authors, scientists, or engineers. Of course, these same people can easily understand the idea when they see it. It's more important to explain why it matters and that's something we're already doing. "Markup language" is fine. It's correct. Org syntax is a markup language. I'm happy to leave it rather than risk bike-shedding it more than I have. If we learn that it caused friction for a newcomer, we can change it easily enough. > > I would also soften that Org "relies" on its markup. It doesn't. I > > used Org only for lists for a long time. I believe lists to be a > > fundamental feature of Org (and hence a great item for the first > > sentence). Lists are as simple as dashes. It's hard to say that > > dashes before list items is a markup language. > > Yet, it is. You cannot, for example, use "." as bullets in Org mode. > Also, indentation matters in Org lists, while it does not matter in more > free-style writing. Fair points. I concede. :) > > Finally, I don't think the file extension is relevant for the first > > paragraph. Technically, an extension isn't necessary. A person can > > call M-x org-mode or use a file local variable. Worse, I think the > > extension contradicts the point that any text editor can view an Org > > file. Ever try to open a .org file in Windows? It asks for the > > program. Yes, *technically* Windows could open a .org file *if* the > > person opening it knew which program to use (or to change the > > extension to something like .txt). Again, who are we writing to? If > > it's someone who believes file extensions matter, then this would > > introduce unnecessary friction. It seems best to avoid it. Better to > > do as you've done and say Org is readable (which it is) rather than > > specify the extension (which doesn't really matter). > > I am mostly neutral here, but I can see an argument why mentioning .org > extension may be useful - unlike Windows, GitHub does expect .org file > extension specifically to render Org mode files. The same goes for > non-Emacs editors that support Org markup. For example, Vim/Neovim. That's a good point. Knowing about the .org extension is useful. I don't think it hurts anything other than taking up valuable space. If we need to bump something from the first paragraph, this gets my vote. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] doc/org-manual.org (Summary): Clarify the Org markup is human-readable
On Sun, 31 Dec 2023 16:22:00 +0100 Ihor Radchenko wrote --- > > I'd like to amend the Org manual introduction as in the attached patch. > The idea is to highlight that Org markup is designed to be > human-readable first and foremost rather than just computer-readable. > > This is the first section of the manual. Most of the new users will see > it. So, I'd like to hear any objections before installing the patch. I like the points about using any text editor and understanding Org without an editor. The main point, it seems, is Org mode avoids lock-in. Two perspectives, the author and the reader, are addressed. An important question is, "Who are we writing to?" Addressing authors and readers covers pretty much everyone, I think :) I believe that makes these good points for the intro. The first paragraph of the current introduction reads, "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a lightweight plain-text markup language used in files with the =.org= extension." Since the first paragraph already mentions "lightweight plain-text markup language," I think making a similar point in the next paragraph is unnecessary. It may be better to say what you want in the first paragraph. I like "human-readable." It's clear and direct. However, trying to incorporate it into the first paragraph becomes heavy: "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a human-readable, plain-text markup language used in files with the =.org= extension." That's a lot of hyphenated words and syllables! Maybe "legible" would be better? It's more precise. But maybe precise isn't good. I'm wondering if there's a simpler word that accommodates non-native speakers better (and not just for this word). "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a legible, plain-text markup language used in files with the =.org= extension." Last comment on your changes. I understand what you mean by "with a naked eye" and I think it's a valid point. However, not everyone who uses Emacs can see. I wonder, do the benefits of Org syntax only apply visually? Several things in the first paragraph unrelated to your changes stick out to me. I can't help but make some other remarks. "TODO" should probably be "to-do". Every dictionary I looked in has an entry for "to-do". I think that's the common spelling. Regarding "markup language," that reads to me like programmer jargon. What does it mean and why should someone care? Again, who are we writing to? A markup language is a notation for formatting, structure, and relationships. I think it would be best to directly say that. I would also soften that Org "relies" on its markup. It doesn't. I used Org only for lists for a long time. I believe lists to be a fundamental feature of Org (and hence a great item for the first sentence). Lists are as simple as dashes. It's hard to say that dashes before list items is a markup language. Finally, I don't think the file extension is relevant for the first paragraph. Technically, an extension isn't necessary. A person can call M-x org-mode or use a file local variable. Worse, I think the extension contradicts the point that any text editor can view an Org file. Ever try to open a .org file in Windows? It asks for the program. Yes, *technically* Windows could open a .org file *if* the person opening it knew which program to use (or to change the extension to something like .txt). Again, who are we writing to? If it's someone who believes file extensions matter, then this would introduce unnecessary friction. It seems best to avoid it. Better to do as you've done and say Org is readable (which it is) rather than specify the extension (which doesn't really matter). Text encodings notwithstanding (plain-text is only as portable as far as you can guess the encoding), what do you think of something like this? modified doc/org-manual.org @@ -18,9 +18,10 @@ :END: #+cindex: summary -Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. -It relies on a lightweight plain-text markup language used in files -with the =.org= extension. +Org Mode is an authoring tool and a to-do lists manager for GNU Emacs. +It uses a legible plain-text notation to show formatting, structure, +relationships. Anyone able to edit text can write using Org. Anyone +able to read text can view it. As an authoring tool, Org helps you write structured documents and provides exporting facilities. Org files can also be used for literate -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v4] org-manual: Describe export process flow
On Thu, 28 Dec 2023 13:05:27 +0100 Ihor Radchenko wrote --- > See the attached. All looks good to me. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v3] org-manual: Describe export process flow
On Wed, 27 Dec 2023 18:05:05 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > Checking ox.el, I see that the commentary says, > > > > ";; See https://orgmode.org/worg/dev/org-export-reference.html> for > > ;; more information." > > > > Maybe also let maintainers know about this manual section: > > > > "See: > > https://orgmode.org/worg/dev/org-export-reference.html > > some-kind-of-reference-to-the-new-manual-section > > " > > AFAIU, you are asking to add it to WORG page. However, it is too early. > The online Org manual that we can link to is only for stable Org > version. So, only after the next release. I was making a different point. I was trying to say that it may be a good idea for the ox.el commentary section to specifically reference the new manual section (next to where it references Worg). The new manual section is unique in that it gives detailed steps about the order of execution. Without knowing the code (so I may be mistaken), it seems like a risk that the ox.el code and manual could get out of sync. This is why I wouldn't suggest documenting the execution order for something like source blocks. However, since end-users have a lot of flexiblity in customizing the export process, it's helpful for them to understand it. It makes sense to add the new manual section. Mentioning the new manual section in the ox.el commentary would help someone new to the ox.el code. It may also help keep the new manual section and ox.el aligned (should anything in the execution order/process change in ox.el) by reminding someone changing the code that there's a manual section specifically about *how* the code works. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v2] org-manual: Describe export process flow
On Wed, 27 Dec 2023 14:41:32 +0100 Ihor Radchenko wrote --- > Thank you all for the feedback! Thanks for taking the time to write and revise the patches. You didn't explicitly ask for more feedback, so I hope the following feedback on the new patches is in order. Both patches look good. I see no issues with v2-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch For v2-0002-doc-org-manual.org-Describe-export-flow.patch here are some quibbles: > On high-level, the export process consists of 4 major steps: I don't think "On high level" is necessary (it's also missing an 'a'; "On *a* high level"). "Major" implies high-level. "The export process consists of 4 major steps:" If you wanted to keep "high-level", you could replace major with it: "The export process consists of 4 high-level steps:" > 2. Parse the buffer, converting plain Org markup into abstract syntax >tree (AST); Missing "an": "...converting plain Org markup into *an* abstract syntax..." > 5. Parse the accessible portion of the temporary buffer to generate >AST. The AST is a nested list of lists representing Org syntax >elements "...of the temporary buffer to generate *the* AST." or "...of the temporary buffer to generate *an* AST." >Past this point, modifications in the temporary buffer copy no >longer affect export; Org export works only with the AST; Replace "in the" with "to the". "no longer affect export" -> "no longer affect *the* export" Replace "temporary buffer copy" with "temporary buffer" or "temporary copy". "Past this point, modifications to the temporary buffer no longer affect the export; Org export works only with the AST;" > 6. Remove elements that will not be exported from the AST: "Remove elements that are not exported from the AST:" or "Remove elements not exported from the AST:" >- Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export > keywords, and =task=, =inline=, =arch= export options (see > [[*Export Settings]]); Remove comma after "keywords": "Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export keywords and =task=, =inline=, =arch= export options (see [[*Export Settings]]);" >- Clocks, drawers, fixed-width environments, footnotes, LaTeX > environments and fragments, node properties, planning lines, > property drawers, statistics cookies, timestamps, timestamps, > etc according to =#+OPTIONS= keyword (see [[*Export Settings]]); "timestamps" is written twice. > - Table columns containing recalc marks (see [[*Advanced features]]). This is indented two spaces more than the previous item. Is that what you want? > 7. Expand environment variables in file link AST nodes, according to >the =expand-links= export option (see [[*Export Settings]]); Comma after "nodes" not necessary. "Expand environment variables in file link AST nodes according to the =expand-links= export option (see [[*Export Settings]]);" > 8. Execute ~org-export-filter-parse-tree-functions~. These >functions can modify AST by side effect; I would say "*the* AST" and make "side effect" plural: "...can modify *the* AST by side effect*s*;" > 4. Convert the nodes with children to text, passing the nodes >themselves and their contents to the corresponding transcoders >and then to export filters (see [[*Filters]]). Add a "the": "...and then to *the* export filters (see [[*Filters]])." - Checking ox.el, I see that the commentary says, ";; See <https://orgmode.org/worg/dev/org-export-reference.html> for ;; more information." Maybe also let maintainers know about this manual section: "See: <https://orgmode.org/worg/dev/org-export-reference.html> " - Again, good work. AFAIU, you used "affect" and "effect" correctly. Breaking the big list into smaller ones makes it much easier to read. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] org-manual: Describe export process flow
On Tue, 26 Dec 2023 22:56:00 +0100 Matt wrote --- > I tend leave the period of the last sentence of a list. Typo. I meant to write "I tend TO leave the period OFF the last sentence of a list." That is, the item ends without a period. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] org-manual: Describe export process flow
e-cell-functions | X | | | org-export-filter-table-functions | X | | | org-export-filter-table-row-functions | X | | | org-export-filter-target-functions | X | | | org-export-filter-timestamp-functions | X | | | org-export-filter-underline-functions | X | | | org-export-filter-verbatim-functions| X | | | org-export-filter-verse-block-functions | X | | |-++---| * Feedback 1: How are the functions not present in the patch handled? I have no clue if they're relevant here or if someone would be confused by their absence. I only noticed that some of them were used in the patch and that the purpose of the patch also deals with filters. * Feedback 2: If I read the patch correctly, the it places the summary *before* the "Export hooks" section. This means that the summary will refer to the hooks before they're introduced (AFAICT, they're only ever mentioned in the "Export hooks" section). The summary should come after the hooks. Maybe before "Extending an existing back-end" or before the next Chapter (Publishing)? * Feedback 3: These are nitpicky things. > 1. The source Org mode buffer is copied into temporary throwaway > buffer that can be edited by export hooks "throwaway" is unnecessary because "temporary" implies it. > 6. When ~org-export-use-babel~ is non-nil (default), all the src > blocks and babel calls that are not inside archived headings are > processed I would write out "src" as "source". Do we have an official way to refer to source blocks? For example, we standardize on "Org": https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/Documentation_Standards.org#n47 Capitalize "babel" since it's a name. > 8. Export option values are calculated, according to in-buffer >keywords, =#+BIND= keywords, buffer-local and global >customizations. The first comma is not necessary. The option values are calculated according to the list that follows (in-buffer keywords, BIND, etc.). Remove the trailing period or add periods to all the others. I tend leave the period of the last sentence of a list. I'm not sure of a style guide that recommends one or the other. Maybe someone knows what's "right"? > - Heading are removed according to =SELECT_TAGS= and > =EXCLUDE_TAGS= export keywords; =task=, =inline=, =arch= export > options "Heading" should be "Headings" > 17. AST is transcoded according to the chosen export backend > - The export happens recursively, depth-first > - Each transcoded AST node, as a string, is passed to the > corresponding export filter (see [[*Filters]]) > 18. The transcoded AST body is formatted according to backend's > "inner" template Maybe use "converted" instead of "transcoded"? I'm a native speaker but I wonder if "converted" is a simpler word for people who aren't. The sentence is simpler by removing the commas and rearranging, "Each transcoded AST node is passed as a string to the corresponding export filter (see [[*Filters]])"" What's an "inner" template and why does that matter? > 20. The filtered body is formatted according to backend's outer > template What's an "outer" template and why does it matter? Should "outer" be quoted like "inner" was? Or, should "inner" be unquoted? > 21. The resulting output is processed by citation backend finalizer "Finalizer" is vague. What's its significance? Does it correspond to a function the end-user might look up? Overall, great work! For each "negative" bit of feedback, there were at least two things that deserve "positive" comments. I saw no spelling errors, you wrote out abbreviations (AST) before using them, and there are no voice issues. Everything was to the point. At first I was confused why this is being added to the manual and not to Worg. However, after reading it more thoroughly, I believe it will be a welcome and helpful addition to the manual. The export process is complex! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Documentation Results of Evaluation (was Re: Documentation of hline symbol in source blocks results)
ng" is inaccurate > "Code /should/"? I agree, this can improve. "Should" in this context reads as "probably" and is non-committal. Documentation has a duty to be authoritative. PROBLEM: "code should execute like a function" is non-committal > "Like a function"? I agree that this is unclear. I believe what it's trying to address is that it's possible to have multiple return values from a source block. For example, Unix shell commands return an exit code indicating success, failure, and failure type. They may also write text to stdout and stderr. The header arguments ":results value" and ":results output" distinguish these two types of return values. PROBLEM: terms are not well-defined > Why is the Python requirement stated here? I see several reasons. The simplest is the piecemeal development style already mentioned. PROBLEM: non-specific requirement reference. What other languages are "like Python"? > Why "using ‘:results value’" when the paragraph should (only) > document this? Do I understand correctly that you're saying "using ':results value'" is redundant? PROBLEM: redundant words > "Result is the value"? What's the problem you see with this? PROBLEM: change of voice PROBLEM: singular form used to reference a plural (should we say "result" or "results") > What kind of value? What do you mean by "kind"? > Why are there references to Ruby and R here? At risk of being pedantic, because of the previous sentence: "Org has to use language-specific methods to obtain the value." I'm having difficulty seeing your perspective. Can you please share more of your thoughts about the confusion for this sentence? > All this confuses me and does not provide the information I > searched for (emphasis on me). I'm sorry for your confusion. I've been there. Few things make my blood boil quite as much as bad documentation. My understanding is that the information you searched for is the answer to these questions: - "How to make a named column table result for a source block?" - "How to insert a horizontal line between two rows of a table result for a source block?" Is that correct? If so, I think Ihor addressed this already by saying that, unfortunately, this functionality isn't currently available (although contributions are welcome). > I would probably prefer a clean-slate approach that starts > with something along the lines of: "Source blocks produce > results that can be integrated into an Org document and used > as input for other source blocks. This is covered in the "Feature Overview": https://www.gnu.org/software/emacs/manual/html_node/org/Features-Overview.html That's not to say that the manual is clear. Maybe including it at the very start (in the "Working with Source Code" section) would have helped? https://www.gnu.org/software/emacs/manual/html_node/org/Working-with-Source-Code.html - A specific section of manual was referenced, "Results of Evaluation": - https://www.gnu.org/software/emacs/manual/html_node/org/Results-of-Evaluation.html - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org#n18460 The specific problems I've identified: PROBLEM: "wrapping" is inaccurate PROBLEM: "code should execute like a function" is non-committal PROBLEM: terms are not well-defined Different language has been used over the past 10+ years to describe behavior of source blocks. The "Results of Evaluation" section mentions "functional mode" and "scripting mode". Collaboration notes (org-babel.org) were kept in the early days of Babel. Eric (the original author of Org Babel) made a distinction between (what he calls) "functional" and "imperative." It's not clear to me what's meant by these terms. Some snapshots of org-babel.org: - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/contrib/babel/org-babel.org?id=c0554c775344207e6adaf191258312fb8d1b6a15 - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/org-babel.org?id=b1c103890c1523f99e380d88ed684454d902414e Here's how I think of it: "Session" means an environment is "persistent." Each call is executed in the same environment. State exists between calls. In the early-history of Babel, this was called the "imperative" style. "Non-session" means an environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. In the early-history of Babel, this was called the "functional" style. All of this relates to the ":results value" and ":results output" header arguments. This is not the first time this has been discussed: https
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
On Thu, 14 Dec 2023 15:35:13 +0100 Ihor Radchenko wrote --- > Matthew, this thread might be of interest for you as the new feature is > largely aiming at the shell block output. > Feel free to jump in if you have comments on the design of the > ASCII fontification for complex shell block output. Thank you for bringing this to my attention and thank you Nathaniel for your work on this. I have no comments on the design presently (my knowledge of Emacs fontification is currently limited) and my current priorities prevent me from dedicating the time this topic deserves. I think the topic is interesting and important. I've had issues with ANSI escape codes (in particular progress bars) in source block results. I made a note to return to this thread in case the escape codes don't bring me back :) Nathaniel, if you and I happen to cross paths in one of Ihor's "office hours," I would enjoy learning more about what you're doing.
Re: [PATCH] [WORG] Maintenance: on the frequency of bugfix releases
On Sun, 10 Dec 2023 13:24:30 +0100 Ihor Radchenko wrote --- > Doing the releases too frequently will increase a chance of such bugs > crawling into ELPA releases. So, there should be at least a minimal > waiting period to get a chance to handle such problems. Thank you for explaining. I had missed that ELPA gets a combination of major, minor, and bugfix releases.
Re: [PATCH] [WORG] Maintenance: on the frequency of bugfix releases
On Sun, 10 Dec 2023 12:27:01 +0100 Ihor Radchenko wrote --- > What do you think about the attached patch? Not part of the patch, but part of the context: "you shoud *NOT*" -> "you should *NOT*" (missing the 'l') Already fixed and pushed (https://git.sr.ht/~bzg/worg/commit/a11b256086d567d0894d337b548ec13049a8731b) Regarding the patch, it seems reasonable. Potentially stupid question, asked without knowing precisely what's involved with a bugfix release (it seems like a matter of tagging and pushing?)...what's the problem with making a new release for every fix? Frequency of fixes? Time required to make the release? Storage?
Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline
On Mon, 04 Dec 2023 14:55:58 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > #+begin_src bash :cmdline 1 2 3 > > >printf "%s\n" "$1" > > > #+end_src > > > > > > #+RESULTS: > > > : 1 > > > > > > #+begin_src bash :cmdline 1 2 3 :shebang #!/bin/bash > > >printf "%s\n" "$1" > > > #+end_src > > > > > > #+RESULTS: > > > : 1 2 3 > > > > Thank you! This makes a good test case. > > I am confused. Isn't it a bug? Yes, it's a bug. And the test was wrong. Here is a corrected version. When the bug is fixed, the following should pass. It currently fails because the first block returns 1 and the second block returns 1 2 3 (as seen in the quote). Both blocks should return 1. (ert-deftest test-cmdline-alone-and-with-shebang-have-same-result () "Pass arguments to a block. Don't use shebang. Then use shebang set to the same language as the block. The result should be the same." (should (equal (org-test-with-temp-text "#+begin_src bash :cmdline 1 2 3 echo \"$1\" #+end_src" (org-babel-execute-src-block)) (org-test-with-temp-text "#+begin_src bash :cmdline 1 2 3 :shebang #!/bin/bash echo \"$1\" #+end_src" (org-babel-execute-src-block)
Re: [BUG] ob-shell: :cmdline fails with single argument (was Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline)
On Sun, 19 Nov 2023 07:57:26 +0100 Max Nikulin wrote --- > I would say that :cmdline is treated in a different way in comparison to > :var: It most definitely is. It's a completely separate process from :var.
[BUG] ob-shell: :cmdline fails with single argument (was Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline)
On Sat, 18 Nov 2023 16:54:39 +0100 Max Nikulin wrote --- > I have faced an inconsistency with :cmdline treatment in ob-shell.el. These are sadly easy to find. If you run: #+begin_src bash :cmdline 1 echo "$1" #+end_src Then it fails with list: Wrong type argument: sequencep, 1 However, running this works: #+begin_src bash :cmdline "1" echo "$1" #+end_src I didn't dig too much into it, but it looks like :cmdline expects a sequence. When multiple arguments are passed, such as :cmdline 1 2 3, then "1 2 3" is passed into process-file. (sequencep "1 2 3") is t. (sequencep 1) is nil. So, to work around this a single :cmdline argument must be surrounded by quotes to make it a sequence. (sequencep "1") is t. Obviously, this should be fixed. Attached is a patch to test for this whenever we're ready to tackle making execution mutually consistent. I'm still reviewing the library and am not quite ready for that yet. test-cmdline-with-single-argument-shouldnt-require-quotes.patch Description: Binary data
Re: [BUG] ob-shell: :shebang changes interpretation of :cmdline
On Sat, 18 Nov 2023 16:54:39 +0100 Max Nikulin wrote --- > I have faced an inconsistency with :cmdline treatment in ob-shell.el. I > expect same results in the following cases: > > #+begin_src bash :cmdline 1 2 3 >printf "%s\n" "$1" > #+end_src > > #+RESULTS: > : 1 > > #+begin_src bash :cmdline 1 2 3 :shebang #!/bin/bash >printf "%s\n" "$1" > #+end_src > > #+RESULTS: > : 1 2 3 Thank you! This makes a good test case. Ihor or Bastion, Craig and my employer reached agreement on the disclaimer language. I've sent a signed copy to Craig and haven't heard back yet with the go-ahead to continue contributing. I sent the paperwork only a week or so ago, so I'm sure he'll get back to me in time. Because of this, I'm including patches rather than pushing (which assumes I still have access (I haven't checked)). test-cmdline-alone-and-with-shebang-have-same-result.patch Description: Binary data
Re: bash source code block: problem after ssh commands
On Fri, 17 Nov 2023 23:07:57 +0100 Matt wrote --- > The second claim has nothing to do with Org Babel. I was able to confirm it > and provide the steps to reproduce. I think it would make sense to report > it upstream and let them decide if it's expected behavior. I'm still happy > to do that, but I need to step away from the keyboard :) Submitted. https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 09:54:46 +0100 Bruno Barbier > But, you're right. To be safe, from now on, I'll use: > > cat /tmp/test.sh | bash -c bash It's still not clear to me if this is "what Emacs does". However, that's the best I could come up with. Evaluating the following #+name: /tmp/test.sh #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt #+end_src does exactly what cat /tmp/test.sh | bash -c bash does. Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line. So, I would say that what happens in Org is the "expected" behavior.
Re: bash source code block: problem after ssh commands
To clarify a previous typo I made. Everything I've written was also done using: 1. emacs -q 2. C-x b "*scratch*" 3. M-x org-mode 4. Execute #+begin_src emacs-lisp :results none (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src This corresponds to Org mode version 9.6.6 (release_9.6.6 @ /gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/) I had previously, and incorrectly, given my version used with my full init (which uses a different Org version). On Sat, 18 Nov 2023 09:19:13 +0100 Bruno Barbier > It seems your SSH failed to connect. In that case, I cannot swallow the > second command; thus the command "echo bar" is executed. > > I can reproduce what you see on my side if I force the connection to fail: > > #+begin_src bash :results output > ssh WRONG_REMOTE "echo foo>foo_file" > echo "bar" > #+end_src > > #+RESULTS: > : bar Thank you! That makes sense. I forgot that without set -e Bash keeps evaluating when errors happen. Yes, once it connects, I get the same behavior as reported. Here is how I connected without a password prompt: 1. ssh-keygen with id_babel, , 2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys 3. chmod og-wx ~/.ssh/authorized_keys 4. ssh-add .ssh/id_babel
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 09:29:56 +0100 Bruno Barbier
> IIUC, what Max is saying is that you should not concentrate on
> *that specific command* because that command doesn't do what you think
> it does.
Cool, it sounds like we're agreed (albeit for different reasons).
> To reproduce, I'm personally still using:
>
> cat /tmp/test.sh | bash
>
> which is, IIUC, what:
>
> (process-file "bash" "/tmp/test.sh")
Yes, agreed. I think that's more like what's happening.
What about the ("-c" "bash") passed into process-file?
The whole call looks like this:
(process-file
"bash"
"/tmp/two-lines.sh"
'(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
nil
"-c" "bash")
Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)
On Sat, 18 Nov 2023 04:11:03 +0100 Max Nikulin wrote --- > > bash -c bash /tmp/two-lines.sh > > From my point of view it was a plain mistake in attempts to simulate > the issue outside of Emacs. There is no point to concentrate on this > command. I tried to explain that it is incorrect usage of "-c" shell > option and what is the actual effect of this call, but I seems I failed. As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs. This is why I've focused on the command. It acts as a guide for what is "correct." Maybe this is misguided for a reason I don't yet see? I concluded that the command is not "what Emacs does" and therefore isn't a valid source of "truth". I agree it's not worth concentrating on it further.
Re: bash source code block: problem after ssh commands
On Fri, 17 Nov 2023 10:20:28 +0100 Ihor Radchenko wrote ---
> This has nothing to do with Emacs comint and this is also not a bug in
> Emacs
Ihor, there were two claims made in the original report. I was referring to
Claim 2. That deals with M-x shell and therefore comint-mode.
Regarding Claim 1:
- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
+ M-x org-version
+ M-x emacs-version
I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @
/home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1,
x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).
* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with
contents "foo" as well as give "bar" as the result.
#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src
The reported behavior is that "foo_file" is created with "foo" (with "foo" is
not stated, but implied) and "bar" is *not* given as the result.
** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like
xfce4-terminal executes the ssh line as expected and outputs the result of the
second line. It was noted that this does not happen with M-x shell.
* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result. The result
is "bar" for me.
The exact "expected behavior" for a shell block is a little fuzzy. According
to my analysis (given below), what Alain reports (remote file and no "bar") is
the "expected" behavior. What I see (no remote file and "bar") is actually
"unexpected".
I used the following to test the claim:
#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar"
#+end_src
I am unable to reproduce the reported behavior (of "bar" not returning).
Instead, I get an ssh-askpass permission denied error, foo_file is not created,
and "bar" is given as the result. I do not see anywhere in the thread that the
original claim was reproduced.
The thread preceded something like follows.
Leo Butler suggested two work arounds:
- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.
Russell Adams suggested another work around:
- add -n to the ssh command
Ihor identified that a non-session call does something like the following
command:
bash -c bash /tmp/temp-file-with-source-block-code.sh
where
- /tmp/temp-file-with-source-block-code -
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
-
The second line (significantly different from the original report) pipes the
echo result to stdout and to a file, bar.txt. Writing to a file allows us to
confirm if that line was executed.
Ihor suggested that
bash -c bash /tmp/temp-file-with-source-block-code.sh
does not run the second line because an interactive password prompt is
displayed by ssh. The reasoning is that the prompt hangs the process while
waiting for input and the second line never runs. Indeed, running the command
does not produce /tmp/bar.txt.
Ihor is correct about prompts messing with shell blocks (this is not the first
time he's seen this). However, the way it's stated does not demonstrate it.
This is because Emacs does *not* make a call like
bash -c bash /tmp/temp-file-with-source-block-code.sh
Alain responded by pointing out that
bash -c bash /tmp/temp-file-with-source-block-code.sh
does not execute the first line. This is true. Consider calling
bash -c bash /tmp/two-lines.sh
where
-- /tmp/two-lines.sh --
echo "first" > /tmp/first.txt
echo "second" > /tmm/second.txt
---
Neither first.txt or second.txt are created.
Max Nikulin interjected with a helpful reminder that Bash scripting is a
snakepit of footguns. (What Max said is more than that and interesting. I
skip it here because it depends on the form of the call.)
Before trying to untangle what a given Bash command does, we need to be sure
what command is actually called. Unfortunately, there's not a clear Bash
command corresponding to how Emacs makes the call.
What happens goes something like this:
1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path
to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object
implemented in C
3. DEFUN ("call-process") forwards this information to the C function
call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess
A lot of cleaning and setup happens which is dependent on the system (GNU,
Window, Darwin, etc.). There's a call to openp which looks for the executable.
An emacs_pipe is set up
Re: bash source code block: problem after ssh commands
alaincochard alain.coch...@unistra.fr writes: > Yes. (So I think that explains the 'no' above.) Thank you for clarifying. > Anyway, this gives me the opportunity to come back to the question of > whether or not there is a problem with emacs itself (like some people > here thought), and if some message should be sent to some emacs list. > Again my argument was that the 2 commands copy/yank'ed in an emacs > terminal don't work as expected, while they do if the same is done in > an X terminal. Okay, I follow you now. Yes, I agree with what others have said, it's related to Emacs (probably comint-mode). It happens with M-x shell and *not* with M-x eshell. It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste. M-x with term-line-mode doesn' t print "bar". > I guess it is not important for Org, because workarounds have be > proposed, but I think if there is a problem with emacs it should be > reported, right? I think it's worth reporting. I'm headed to bed and can submit it tomorrow for you if you're not comfortable with doing it or don't have the time. Here are the steps I followed to reproduce with M-x shell: 1. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 2. Paste them into a shell (tested with xterm or xfce4-terminal) and press return 3. After entering the correct password, "bar" is echoed 4. Paste them into the buffer associated with M-x shell 5. After entering the correct password, "bar" is NOT echoed Here are the steps I followed to reproduce with M-x term: 1. M-x term 2. C-c C-j to switch to term-line-mode 3. Copy the following two lines: ssh $USER@localhost "echo foo>foo_file" echo "bar" 4. Paste them into the buffer associated with M-x term 5. After entering the correct password, "bar" is NOT echoed
Re: bash source code block: problem after ssh commands
On Thu, 16 Nov 2023 10:30:59 +0100 Ihor Radchenko wrote --- > Or we can make use of TRAMP as we usually do to access remote > environment. IMHO, it is more reliable as TRAMP takes care about > arranging all the Elisp FS interaction to work on remote system. Agreed. Also, I wasn't aware TRAMP worked with shell block evaluation. That's nice. Sure enough, these both work: #+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote* echo foo>foo_file echo "bar" #+end_src #+begin_src bash :results output :dir /ssh:user@localhost:/home/user echo foo>foo_file echo "bar" #+end_src Of course, both calls happen on the remote. I wonder, would this have helped Alain? I'm admittedly a little confused about the original report, #+name: original-report #+begin_src bash :results output ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src Clearly, the ssh line is intended to happen on the remote server. Was the echo "bar" intended to happen locally? PS: I've updated the WORG page for ob-shell with a :dir section (https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54)
Re: bash source code block: problem after ssh commands
On Wed, 15 Nov 2023 17:32:19 +0100 Matt wrote --- > (let ((buff "*localhost-process-buffer*")) > (switch-to-buffer >(make-comint-in-buffer > "localhost-process" > buff > "ssh" > nil > (format "%s@localhost" (getenv "USER") It looks like the way this is called actually uses `start-file-process` rather than `open-network-stream`. It's not clear to me how to call it so that `open-network-stream` is used. However, that's more a question for the Emacs mailing list and, honestly, not one that I think I care about right now. Posting this just to correct what I said before.
Re: bash source code block: problem after ssh commands
I was poking around, learning how sessions are started. Basically, `shell` creates the comint buffer using `make-comint-in-buffer`. What I find interesting is that `make-comint-in-buffer` can also create a comint buffer from a network stream: (let ((buff "*localhost-process-buffer*")) (switch-to-buffer (make-comint-in-buffer "localhost-process" buff "ssh" nil (format "%s@localhost" (getenv "USER") So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that. Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`. `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used. Therefore, we could provide header arguments to pass parameters and the destination to ssh. I haven't made a judgment yet about whether any of this is good or bad. I thought it was interesting and figured I'd share.
Re: Replace org link with footnote
On Sat, 11 Nov 2023 04:54:15 +0100 Max Nikulin wrote --- > https://list.orgmode.org/ugucve$12l4$1...@ciao.gmane.io Thanks for the reference. It's always good to relate associated information :)
Replace org link with footnote
Here's a quick hack to replace an Org link with a footnote. Maybe a fun first commit for someone would be to update `org-footnote-new` to accept optional LABEL and DEFINITION arguments that allow a footnote to be created programmatically? (defun my-replace-link-with-footnote () "Replace an org link with a footnote. Place the link target in the footnote and the description in the location of the original Org link." (interactive) (if (org-in-regexp org-link-bracket-re 1) (save-excursion (let ((remove (list (match-beginning 0) (match-end 0))) (link (org-match-string-no-properties 1)) (description ; link is description (if (match-end 2) (org-match-string-no-properties 2) (org-match-string-no-properties 1 (apply 'delete-region remove) ;; replaces link with footnote and jumps to footnote (call-interactively 'org-footnote-new nil) (insert " " link) (beginning-of-line) ;; follow link back to original point (org-open-at-point) (insert description)
Re: bash source code block: problem after ssh commands
On Thu, 09 Nov 2023 13:13:36 +0100 Ihor Radchenko wrote --- > A bigger problem is that errors *Org Babel Error > Output* are not clickable: Interesting. > So, it looks like we may need an alternative `org-babel-eval' function > that works with script files rather than input and arranges > stdout/stderr separation. Yes, it seems like a separate handler may be in order. > We should reuse `process-file' in this scenario as well, > but utilizing its ARGS argument + arrange to generate the temporary > script file in the appropriate context - see `make-nearby-temp-file'. I was unaware of `make-nearby-temp-file'. Thank you. It's helpful for this situation. I'm not in a position to address these points right now. My main focus is re-familiarizing myself with the =ob-shell= and =ob-comint= source. When I was implementing async shell evaluatation earlier this year, I remember having the impression that execution paths, especially `org-babel-comint-with-output', could probably be simplified. It seems to me that everything we're talking about in this thread aligns with that objective. I've made a note of your points and have every intention to return to them when I get to writing code. However, don't let me stop you if this is something you were interested in doing. I'll do what I can to assist.
Re: bash source code block: problem after ssh commands
On Tue, 07 Nov 2023 09:53:46 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > > > I am wondering about the possible downsides of using script approach > > > instead of stdin redirection. > > > > I'm curious to hear more about what you're thinking. > > I am thinking to change the > (t (org-babel-eval shell-file-name (org-trim body))) > clause in `org-babel-sh-evaluate' to something that uses a script file. > > It will clearly solve the discussed problem, possibly at the cost of > small overhead to write the script file to disk. Thanks for clarifying. I've been away from the codebase for a bit and, now that the FSF paperwork is signed (still need to get Craig a copy), I'm reviewing =ob-shell= with an eye for what could be cleaned or improved. I feel like =org-babel-sh-evaluate= could use some attention. I'm open to your suggestion. The good is that using a script is how :shebang and :cmdline are processed currently (like you pointed out) so there's precedence and experience with it. Also, it would make all non-session execution use the same model (script versus comint). I like how that would create a clean separation. For bad, nothing jumps out to me as obviously a problem. Let me "think out loud" for a moment. We'd need to write to disk. Like you say, this incurs overhead opening, writing, and closing the file. It's not like we'd forget to close it, though. Nor is running out of space or inodes our problem. Writing requires permission. That's not an issue with /tmp. Then, it needs to execute. Aside from permission, any code we insert needs to be correct. For example, a shebang would need to point to the correct application and any arguments would need to correspond to the implementation being called. I doubt we'd need anything beyond /bin/. FWIW, it looks like there's been at least one instance where :shebang's formatting was questioned (https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/). We'd also need to control for what environment the script runs in. That was another issue I saw raised in the list (https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/). Of course, we'd need to read the stdout and stderr. This is handled by =process-file=. Any step I missed or some kind of failure I didn't consider?
Re: bash source code block: problem after ssh commands
On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > I am wondering about the possible downsides of using script approach > instead of stdin redirection. I'm curious to hear more about what you're thinking.
Re: bash source code block: problem after ssh commands
On Fri, 27 Oct 2023 13:47:24 +0200 alain.coch...@unistra.fr wrote --- > Russell Adams writes on Thu 26 Oct 2023 16:44: > > [...] are you using an SSH key to connect, or entering a password? > > If you use an SSH key for passwordless access, try adding -n (ie: > > > "ssh -n derp@host mycommand"). > > Yes, I use an SSH key, and yes, '-n' saves the day. Thanks. Hi Alain, I'm the supposed maintainer of ob-shell who's been missing in action for this whole discussion. I want to confirm, are you able to accomplish your task by using '-n'? Matt
Re: Maintenance status of individual Org libraries
On Mon, 30 Jan 2023 14:41:18 +0100 Ihor Radchenko wrote --- > The following libraries have no maintainer or author active on the list: > ob-comint, ob-core, ... I'm interested (and technically still maintainer of ob-shell). ob-comint and ob-core are both used by ob-shell, so I've looked at good bits of both. I'd be willing to work on those. Unfortunately, my new job has yet to sign the FSF disclaimer, so I'm unable to contribute at the moment (I'm doing what I can to facilitate communication between parties). Feel free to reach out off-list with advice for how I might help that process. Until the disclaimer language gets resolved, I may be able to help in other ways. I have notes explaining the ob- API (https://lists.gnu.org/archive/html/emacs-orgmode/2023-02/msg00531.html). Maybe if that were completed, it would make being a maintainer more approachable for people? I'd also be happy to mentor new maintainers. What other ways might I help?
Re: org babel output
On Sat, 01 Jul 2023 05:52:42 +0200 Robert Synofzik wrote ---
> Hello,
> the exact same error occurs for me:
> https://emacs.stackexchange.com/questions/72243/macos-org-babel-python-with-session-output-give-python-el-eval
>
> do you possibly have any idea on how to fix this?
Just to check, because you didn't explicitly say it in your message, does the
solution given in the StackExchange not work for you? That is, do you get a
different result when you name the session? What happens when you run the
following?
#+begin_src python :session my-named-session :results output
print("yes")
#+end_src
Re: Export Documentation Confusions
Changes have been pushed
Re: Export Documentation Confusions
On Thu, 29 Jun 2023 12:24:12 +0200 Ihor Radchenko wrote --- > > - Section 13.18: Export in Foreign Buffers > > > > The section uses the term "foreign buffer" without providing a definition. > > > > Proposed solution: Change wording from "foreign buffers" to "Convert > > Region" and "non-Org" buffer. > > "Convert region" may be still confusing. Maybe something like > "Convert Org region to other markup syntax"? How about renaming the section to "Export Region"? I was getting ahead of myself. The confusion was about "foreign buffers" and not the meaning of "export". > > - Section 13.18: Export in Foreign Buffers > > > > There was confusion caused by the function names. > > We can also create more intuitive function aliases. How about `org-export-region-to-...`? This would correspond nicely with options given by `org-export-dispatch`.
Export Documentation Confusions
In the Emacs Berlin meeting today, we noticed several items that might be improved in the manual. - Section 13.18. 1: Bare HTML may be better placed under Section 13.9 HTML Export. The information in this section appears unrelated to Exporting in non-Org buffers. In fact, the settings given for a minimal HTML export do not apply when converting a region. For example, when converting a region containing an Org table to HTML, a preamble is not inserted even when the variable `org-html-preamble` is t. https://orgmode.org/manual/Bare-HTML.html Proposed solution: Move Section 13.18.1 nearer the sections on HTML export. - Section 13.18: Export in Foreign Buffers The section uses the term "foreign buffer" without providing a definition. It appears to mean a buffer without "org-mode" enabled. The section describes functions which allow users to write Org mode syntax in non-Org buffers and use various functions to convert the Org syntax to other syntaxes, like HTML, Markdown, etc. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Change wording from "foreign buffers" to "Convert Region" and "non-Org" buffer. The functionality is really about conversion rather than anything to do with non-Org buffers. If fact, Org respects users' freedom to run the program as they wish–it does not prevent them from converting Org formatted text within an Org buffer to a non-Org format, no matter how absurd it may appear :) - Section 13.18: Export in Foreign Buffers There was confusion caused by the function names. Specifically, the "namespace" (such as "org-ascii-" or "org-texinfo-") is confusing to users who are not familiar with Org function naming conventions. For example, one participant thought `org-ascii-convert-region-to-utf8` might convert non-utf8 ascii to utf8 and was confused why `org-texinfo-convert-region-to-texinfo` would convert texinfo to texinfo. Of course, this is not a correct understanding of the function behavior. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Clarify the overall purpose of the functions. Another solution could be to make an aside and clarify what is meant by "Org export back end". For example, change "The export back-ends in Org often include..." to "The export back-ends in Org (for example, Org ASCII or Org HTML) often include..." However, my hope is that the changes I made with regard to "foreign buffers" also clarifies the purpose and use of functions listed in this section. These changes are given in the attached patches. If there are no suggested improvements, I could merge them in. Please let me know your thoughts. move-bare-html-section-to-html-export-section.patch Description: Binary data change-foreign-to-non-org.patch Description: Binary data
Re: Reintroduction and Questions
On Sun, 25 Jun 2023 18:07:34 +0200 Ihor Radchenko wrote --- > Be warned though that Woof! is not yet stable (see > https://git.sr.ht/~bzg/woof and https://lists.sr.ht/~bzg/woof). Thanks for the warning. To verify my understanding, the mailing list is the "source of truth" for issues and Woof! "scrapes" that? > You may also try to ask on Emacs Berlin mailing list. > https://emacs-berlin.org/ Good suggestion!
Reintroduction and Questions
Hello again! I'm slowly settling in after moving to Germany and hoping to get back into maintenance. As the FSF disclaimer is still being worked out with my employer, I'm hesitant to begin non-trivial code contributions. Until then, I'm hoping to do things that don't require attribution, such as organizing tasks, issues, and updating Worg. 1. What do the FR, FP, RFC, and RFE Woof! prefixes stand for? These are listed in https://tracker.orgmode.org/howto for the "request" type under the "Adding a report" section. I assume FR is "Feature Request" and RFC is "Request for Comment". 2. Is anyone in the Hamburg, Deutschland area interested in meeting in person? They say it's hard to make friends here and I'm taking applications :) I think it'd be cool to talk Emacs and work on contributions together. Maybe we could coordinate this with an online Meetup (https://lists.gnu.org/archive/html/emacs-orgmode/2023-06/msg00196.html).
Re: Interest in an Org video meetup?
Thoughts on dates and times?
Re: ob-shell sessions will send final echo '...' as input key sequence when previous command reads input interactively (was: [Need Help] Error to evaluate "mpv" command in inline src block)
On Tue, 02 May 2023 00:17:51 -0400 Christopher M. Miles wrote --- > Indeed, this option solved the problem real neat. > I'm surprised by your knowledge and digging problem skill. > I have to say a big TANKS to you. THANKS, THANKS, THANKS. Hahaha My thanks as well to everyone who helped. I'm tied up with personal matters currently and am only seeing this thread now. On Mon, 01 May 2023 07:36:23 -0400 Ihor Radchenko wrote --- > The only way I know how to work around this is a giant one-liner like > echo ''; ; echo '...' > > However, (1) ";" may not work in some shells; (2) may > contain multiple lines, leading to the same issue. > > Matt, maybe you have some ideas about this edge case? I have no other ideas within the current ob-shell implementation. As for modifications, I have the following thoughts. First, how might we state the problem for this edge case? To me, it looks like, "How can the ob-shell :async option manage interactive input?" Do you agree with this formulation? If not, how do you see it differently? One thought is to update :async to work with the :stdin option so that the block is run as a script. Currently, :stdin runs synchronously in a separate shell. We might be able to grab the script's output and put it into the session buffer. See how the following runs in a temporary shell, regardless of the :session/:async options. #+name: answers Matt yes #+begin_src sh :stdin answers :results output :session *test* :async echo -n "What's your name?" read -r name echo echo "Hello, $name!" echo -n "Would you like to continue?" read -r continue echo if [ $continue == 'yes' ]; then echo "Continuing..." sleep 3 echo "Process complete" else echo "Aborted" fi #+end_src #+RESULTS: : What's your name? : Hello, Matt! : Would you like to continue? : Continuing... : Process complete
Re: Interest in an Org video meetup?
On Wed, 26 Apr 2023 17:13:47 -0400 Corwin Brust wrote --- > On Wed, Apr 26, 2023 at 9:31 AM Matt m...@excalamus.com> wrote: > > > > FSF associate members have access to a FSF hosted Jitsi instance. I'm an > > FSF associate member and could host it using that if I'm able to make the > > time. > > > > We can make EmacsConf instance of BigBlueButton (BBB) available, as > well. FWIW I had more success with BBB than Jitsi for larger > meetings. That's generally been my experience, too. Glad to hear that's an available option. Hopefully we can bring this to fruition.
Re: Interest in an Org video meetup?
> > On Oct 6, 2022, at 9:20 AM, Russell Adams rlad...@adamsinfoserv.com> wrote: > > Would there be any interest in a monthly 1-2 hour long ad-hoc screen > > sharing and video discussion for Org-mode? > > > > I'm offering to schedule and moderate the first few events. I'd > > propose a Saturday meeting in the afternoon European time to cover EU > > and NA. > > > > I'm considering using Jitsi, or maybe GNU Jami. > > > > Topics could include Q, demonstrations of features, interactive > > troubleshooting, etc. I hadn't considered presentations, but > > that could be an option too. > > > > Comments? I'm interested. I've thought of proposing something similar, yet hesitated to because topics and decisions made in a video chat wouldn't be captured on list. To me, that record is important. FSF associate members have access to a FSF hosted Jitsi instance. I'm an FSF associate member and could host it using that if I'm able to make the time.
Re: [PATCH] Async evaluation in ob-shell
On Mon, 17 Apr 2023 14:53:18 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > A small note on the WORG page: it may be more natural to use :async yes > > > rather than :async t. Both are viable - in fact, anything other than > > > :async no and :async none will be treated as "t". > > > > Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated > > ORG-NEWS with 6c9104f59. > > Thanks, but I am not seeing 6c9104f59. I'm sorry, I was unclear about which repo these commits were in. ORG-NEWS was updated here: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6c9104f59ca8085abe477a81857548461bf88f23
Re: [PATCH] Async evaluation in ob-shell
On Fri, 24 Mar 2023 05:11:38 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Thu, 23 Mar 2023 07:48:44 -0400 Ihor Radchenko wrote --- > > > May you also document this new feature in ORG-NEWS and in > > > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html ? > > > > Done. > > A small note on the WORG page: it may be more natural to use :async yes > rather than :async t. Both are viable - in fact, anything other than > :async no and :async none will be treated as "t". Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated ORG-NEWS with 6c9104f59.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Fri, 07 Apr 2023 11:29:59 -0400 Matt wrote --- > > On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > > > See the attached tentative patch. > > After applying the patch, I get the following error when trying to load > Emacs: > > Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's > value as variable is void sh-ancestor-alist > > I wasn't able to resolve it. I suspect the issue is on my end, such as a > mixed install or the need to re-byte-compile `sh-script.el'. I was able to resolve it by simply requiring `sh-script' in `org-src'. Committed and pushed.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > See the attached tentative patch. After applying the patch, I get the following error when trying to load Emacs: Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's value as variable is void sh-ancestor-alist I wasn't able to resolve it. I suspect the issue is on my end, such as a mixed install or the need to re-byte-compile `sh-script.el'. To run Org from source I do one of the following: ;; When using my init (use-package org :straight (:local-repo "/home/ahab/Projects/org-mode")) ;; When running emacs -q (add-to-list 'load-path "/home/ahab/Projects/org-mode/lisp") (require 'org-loaddefs) If I need to recompile Emacs byte code, I'm not sure how I'd do that since I'm running Guix and those files live in the write protected /gnu/store.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Thu, 30 Mar 2023 04:55:32 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I think this approach will work fine. I tried examples for each shell > > type and keywords like if/then/else and function names are highlighted. > > Even for posh (powershell)? Yes. It's not great since sh-mode looks for Korn-based keywords. It does string highlighting and common keywords like 'if', 'exit', and 'param'.
