Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Contributions to Worg aren't similarly restricted. Feel free to > push material there in the meantime. Looks like the email finally got sent to the right person at my company. Who knows how long it will take for them to get the FSF disclaimer back to me... Now that I'm on holiday, I've got a little more time to contribute to Worg. Here's a few patches showing what I've come up with. I'll finish it up in the next few days, hopefully. Not sure if the patches are formed correctly, sorry about that if not. It's late here and I wanted to show progress. I'll worry about those kinds of details later. :) 0001-Draft-introduction-and-setup.patch Description: Binary data 0001-org-contrib-babel-languages-ob-doc-shell.org-Create-.patch Description: Binary data
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Wow. Nice work! Thanks. I pushed things to Worg, if you haven't seen already (https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html). If you read it and find anything missing or unclear, please let me know. I'm still waiting on work to sign the FSF disclaimer in order to work on ob-shell.el. I might just implement something on the side and use that to inform any future contributions.
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
Apologies for the book. I've been sitting on this stuff for two months and am wondering how to proceed. IANAL but AFAIK/CT, my contract contains an exception for making contributions to projects like Org. I've gotten confirmation from my manager and by HR. However, until the CEO signs the FSF disclaimer, I can't officially contribute. I'm confident that I can publish changes (e.g. to a personal website); the FSF just can't accept my changes (yet). I could start working on ob-shell.el separately now and publish that myself. It's not clear how I would then contribute those changes back once I'm cleared by the FSF. I'm inclined towards some refactoring and I'm not sure how I could break that down in such a way that if it took two more months to get the copyright stuff worked out that anyone could make sense of the changes. I would much prefer to gradually submit patches, discuss them, and then eventually have them merged in turn. If I have a heap of changes elsewhere, I feel like it would be harder to have a conversion about them. Regardless, I think I should define test cases first. If those are considered valid, then any refactoring would be moot if they pass, right? With (agreed upon) test cases, it shouldn't matter if I refactor as long as functionality remains the same? Overall, what advice do you have? It looks to me like ob-shell.el could use some love in other respects besides async evaluation. I've detailed where below, partly for my own organization and partly for posterity, but mainly because this isn't my house, so to speak, and I don't want to barge in and start rearranging the furniture and eating whatever's in the fridge. Or, is it like Worg in that once I have the keys I can drive where I like, so long as there're no crashes? I'm interested in people's thoughts on these notes on ob-shell.el: * Tests There are several code paths, like shebang, cmdline, and basic execution, which don't always have something to do with one another in the code. Having tests would be really helpful to make sure everything jives. Before doing anything with the code base, I intend to create tests for all the functionality. * 2D-array I documented two options for the =:var= header[fn:1]. The ob-shell.el code actually defines a third option for 2D-arrays. I couldn't get it to work. It was one of several things not documented anywhere, at least as far as I could find, and which I had to figure out straight from the code. Between not being great at shell scripting and having a hard time deciphering that ob-shell.el code, I'm not sure 2D-arrays are actually fully or correctly implemented. * M-up behavior <> The =org-babel-load-session:shell= code path only works when M-up is used on a code block[fn:2]. Is M-up actually documented anywhere? Furthermore, the =org-babel-load-session:shell= only works for the "shell" language, which is not actually a "proper" shell (i.e. it's not listed in =org-babel-shell-names=). The M-up defaults to using $ESHELL or shell-file-name through the =shell= command. For example, try calling M-up on these: #+comment: (opaquely) calls the system shell #+begin_src shell :session my-shell echo "hello, world!" #+end_src #+comment: fails #+begin_src sh :session my-sh echo "hello, world!" #+end_src #+comment: fails #+begin_src bash :session my-bash echo "hello, world!" #+end_src To fix this, there needs to be an =org-babel-load-session:= for each language in =org-babel-shell-names=. This would probably make the most sense in =org-babel-shell-initialize=. However, that function [[org-babel-shell-initialize][needs attention]]. * Refactoring <> The ob-shell.el code appears inconsistent to me. Clearly, this is somewhat subjective. I've tried to give a rationale for each point to make it less so. My goal is to be maintainer of ob-shell.el, but that's not forever. These things were an obstacle for me and my aim is to remove them for the next person. ** =org-babel-shell-initialize= <> As alluded to elsewhere, =org-babel-shell-initialize= doesn't appear to adhere to the (elisp) Coding Conventions, #+begin_quote • Constructs that define a function or variable should be macros, not functions, and their names should start with ‘define-’. The macro should receive the name to be defined as the first argument. That will help various tools find the definition automatically. Avoid constructing the names in the macro itself, since that would confuse these tools. #+end_quote The =org-babel-shell-initialize= function defines =org-babel-execute:=, =org-babel-variable-assignments:=, and =org-babel-default-header-args:= for the "languages" in =org-babel-shell-names=. As it stands, that =org-babel-shell-initialize= is a function does no harm (aside from being confusing by way of straying from convention). However, if the [[M-up][M-up]] issue is to be resolved, it seems to me that =org-babel-shell-initialize= should be
[PATCH] ob-doc-shell.org (was [PATCH] ob-shell-test, test-ob-shell and introduction)
> Unfortunately, there is no ob-doc-shell.org at > worg/org-contrib/babel/languages/. > > There is an ob-doc-template.org in case you'd like to contribute > ob-doc-shell.org :) I created one, along with a sourcehut account. However, it looks like I need to be granted some permissions to write to the worg website. Is Bastien the contact for that? I've attached a patch in the mean time. 0001-org-contrib-babel-languages-ob-doc-shell.org-Create-.patch Description: Binary data
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> [FSF copyright assignment]. Have you done that yet? I just verified with my employer that my contract grants an exception for this project. Just emailed the request to ass...@gnu.org. Also, got access from Bastien for worg. I figure it's probably best to reserve any more changes 'til the paper work is done?
[PATCH] ob-shell-test, test-ob-shell and introduction
Hi, I'm interested in getting async into ob-shell.el. Since I've never contributed before, I figure it'd be good to start with a few easy tasks. It looks to me like the stdin and cmdline header args aren't documented anywhere (at least I couldn't find anything). To make sure I'm using them correctly before making a patch for the manual, here are some tests. Please let me know if things look okay. It wasn't clear to me how to send along a message with git send-email, so I formatted these patches and included them as an attachment. Is that fine? Thanks, Matt 0001-ob-shell-test.org-Add-example-for-stdin.patch Description: Binary data 0002-test-ob-shell.el-Add-tests-for-stdin-and-cmdline-hea.patch Description: Binary data
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> > I just verified with my employer that my contract grants an exception for > > this > > project. Just emailed the request to ass...@gnu.org. Not surprisingly the FSF hasn't resources to verify my contract's exception and needs a written disclaimer from my employer. I'm waiting on this now. > Feel free to develop and > share patches before the assignment is complete, we’ll just wait till it’s > gone > through before merging Given the contract exception, I'll start moving forward again with the assumption that I'll eventually get that written disclaimer. > Hope that helps. It does, thank you. It's nice to know that my words aren't going into the void. :)
Re: [PATCH] ob-shell-test, test-ob-shell and introduction
> Thank you, I was not aware of ":cmdline" argument and of ":shebang" as a > means to avoid stray prompts (I have seen mentions of similar problem > for other languages in this list). You're welcome! I was surprised to find those in the source. Glad to have documented them and even happier to hear that it's news to others. > - https://www.mirbsd.org/pics/logo-grey.png works only with http: > protocol for me. I updated (downgraded?) it to http. I'm also seeing that the kornshell logo is intermittently broken. Not quite sure the best way to go about fixing that. Maybe commit copies? Use something like archive.org? Sounds like a problem for someone else (e.g. future me). I'm out of time for tonight. > - I am unsure concerning example after "…or as a standlone script.", > maybe it should be wrapped into #+begin_example. Good call. That example was wonky. I've given a different one. Not sure it's better, though. I find the shebang behavior tricky to demo in a simple way. > - In some cases "sh" is used despite bashisms in the code like "declare" > or "echo -ne". Actually "printf" may be more convenient instead of "echo -n" > - "export" in session example is unnecessary. It is matter of taste > though. Bash and dash is not the case, but some shells require that > assignment and export should be separate commands. Good catch! Since arrays are only defined in ob-shell using bash, I've updated that example to use bash. Otherwise, I've removed the exports. > - In your examples variable values are simple. Often it is safer to add > double quotes around variable or command expansion. I would consider > adding quotes just to encourage people to do the same by default with > hope that less scripts will give strange results in response to a file > name containing a space. Actually my first impression was that > backslashes before quotes in some cases were added by mistake. Another > unescaped pair of quotes may make your intention more clear. However it > is related to code style where everybody has opinion, so I do not insist. I appreciate these clarifications. Admittedly, I'm not great with shell scripting. Your recommendations appear sound to me and I've tried to incorporate them. The updates have been made and are pushed. Thanks for your feedback!
Re: How do you manage complex project with Org-mode
On Mon, 28 Feb 2022 20:43:47 -0500 Sébastien Gendre wrote > And I don't know how to manage this kind of projects with Org-mode. How > to do it, without failing a 6 days project because I spent to much time > on something else and I have only 3 days left with 3 half-day important > appointment I cannot cancel. I can't risk failing a single one of these > project by trying. So, when I am in a period with a lot of these > projects, I stop using Org-mode and concentrate on doing these project > as fast as I can. And because I often have this kind of project, I spend > most of the year without being able to use Org-mode. It sounds like you have a lot going on! If none of what you need to do explicitly requires Org, you may have to scale back what you learn about Org to fit the time you've got. Give yourself permission to accept that the time you have right now for Org isn't what you want. (I hope that's because you're learning lots of other cool things in school.) In situations like these, I like to do just a little each day. Maybe that means reading one paragraph a night before bed. It sounds like you're really excited about Org. (If you are, you've come to the right place. The people here love Org :) Reading about Org would be something fun to look forward to each night. You might be surprised at how motivating that one little paragraph can be! Working through the manual in this way will give you a good overview of how Org can be used and what you personally might use Org it for. > So, if you have any suggestion on how to manage, in Org-mode, projects > with: > * Lot of work to do (many days) > * Short deadline (not enough time) > * High importance (disastrous consequences in my future in case of fail) > * Many of them in the same time > * Progression need to be followed to chose where to sacrifice time to > limit the damages > > I will be happy to read them. :) My suggestion is to not try something new on anything that has a tight deadline. Org is new to you and learning things takes time. This is all normal. Life will throw a lot at you. Some times all I get is 20 minutes at the end of the day. Some days, I get nothing at all. This is because I've filled my life with other cool things, like a partner, a house, friends, etc. If I get to spend some time doing something I think is worthwhile (like trying to help a fellow Org enthusiast), that's time well spent in my book. Isn't there some saying that goes like, "every avalance starts as a snowflake?" Read a little, experiment a little, and over time, you'll be surprised at how much you've learned. You'll get there!
Re: Shell SRC blocks under Windows
On Sat, 05 Feb 2022 10:48:02 -0500 H. Dieter Wilhelm
wrote
> >> Is is it possible to extend org-mode's src blocks with
> >> org-babel-shell-names to use either MSYS2 or MinGW shells (instead of
> >> cmd.exe)? Unfortunately ob-shell.el doesn't guide me further and I
> >> can't find examples or the respective lisp files on how to define the
> >> execution functions.
>
> Yes, you are right, when using
>
> > #+begin_src sh
>
> It seems that I'm also running some bash shell! :
Yes, it's likely possible to run those shells. Whether or not it will be
straight forward or not depends. It's not clear to me what you're trying to do.
First off, you're looking in the right place. Unfortunately, ob-shell.el can
be a little tricky to read. All the shells are initialized through the
=org-babel-shell-initialize= function. This creates a
"org-babel-execute:" for each of the shells given in the
=org-babel-shell-names= variable (where is given by #+begin_src )
. The default shells are: ("sh" "bash" "zsh" "fish" "csh" "ash" "dash" "ksh"
"mksh" "posh"). All execute calls make their way through the God function
=org-babel-sh-evaluate= (note: the "sh" in the function name appears to be
cruft from when the ob-shell.el module was called "ob-sh.el"; it is not tied
specifically to the sh shell). Technically, which shell is used to evaluate
the block depends on the source block parameters used. There are several
possible paths. Unless something weird is being done, however, the block
should be evaluated by . It looks like you're using a simple one-off
shell, not a session or a shebang, so I'll speak to that.
For #+begin_src sh, the block will we evaluated by whatever "sh" corresponds
to in the environment Emacs is running in. Under the hood, a call is made to
the =shell= command and the =shell-file-name= variable gets set to whatever
the block specifies. The an binding happens in
=org-babel-shell-initialize=. So, open a command line environment that would
be used to start Emacs and see what "sh" corresponds to:
which sh
That's the shell used by #+begin_src sh. Similarly, whatever "which bash"
shows is what #+begin_src bash uses.
This is why it would be surprising that cmd.exe would be used.
I know that was a lot. Depending, it may be as simple as updating the
=org-babel-shell-names= and rerunning =org-babel-shell-initialize=. Check the
doc strings on those. Otherwise, I hope this helps ob-shell.el guide you
better.
> But I'm not sure if I'm halfway there with "sh"? I need to run the
> following MSYS2 command AND switch between two arguments (for building
> Emacs).
>
> \MSYS2\msys2_shell.cmd -mingw64
>
> and
>
> \MSYS2\msys2_shell.cmd -msys.
>
> How could I achieve this within a src block?
I'm not sure I follow you here. Can you please describe what you're trying to
do and the system you're using (e.g. Windows, GNU/Linux, Haiku) as well the
shell used to run the Emacs instance you're doing this in?
Re: Shell SRC blocks under Windows
On Sat, 05 Feb 2022 17:22:29 -0500 Matt wrote > > But I'm not sure if I'm halfway there with "sh"? I need to run the > > following MSYS2 command AND switch between two arguments (for building > > Emacs). > > > > \MSYS2\msys2_shell.cmd -mingw64 > > > > and > > > > \MSYS2\msys2_shell.cmd -msys. > > > > How could I achieve this within a src block? > > I'm not sure I follow you here. Can you please describe what you're trying > to do and the system you're using (e.g. Windows, GNU/Linux, Haiku) as well > the shell used to run the Emacs instance you're doing this in? > Excuse me. Obviously you're using Windows. :)
Re: Shell SRC blocks under Windows
On Sun, 06 Feb 2022 12:51:49 -0500 H. Dieter Wilhelm wrote > Matt writes: > > > On Sat, 05 Feb 2022 17:22:29 -0500 Matt wrote > > > > > > > > But I'm not sure if I'm halfway there with "sh"? I need to run the > > > > following MSYS2 command AND switch between two arguments (for > > building > > > > Emacs). > > > > > > > > \MSYS2\msys2_shell.cmd -mingw64 > > > > > > > > and > > > > > > > > \MSYS2\msys2_shell.cmd -msys. > > > > > > > > How could I achieve this within a src block? > > > > > > I'm not sure I follow you here. Can you please describe what > > > you're trying to do and the system you're using (e.g. Windows, > > > GNU/Linux, Haiku) as well the shell used to run the Emacs instance > > > you're doing this in? > > > > > Excuse me. Obviously you're using Windows. :) > > Yes I have to, but please don't mind, thank you for your interest. :-) > I don't mind. I had asked about the system to help troubleshoot yet that info was clearly given in the thread title. :) You said you weren't sure if you're halfway there with "sh". Is that feeling resolved for you?
Re: Shell SRC blocks under Windows
On Tue, 08 Feb 2022 04:47:04 -0500 H. Dieter Wilhelm wrote > Not yet, I'm still grappling with your message > <17ecbfb46e9.e16df906236239.6523271905911672...@excalamus.com> and > ob-shell.el. I'll come back to you when I'm through. Okay, feel free to let me know if something I wrote is unclear or to ask any questions.
Re: bash source code block: problem after ssh commands
On Fri, 27 Oct 2023 13:47:24 +0200 alain.coch...@unistra.fr wrote --- > Russell Adams writes on Thu 26 Oct 2023 16:44: > > [...] are you using an SSH key to connect, or entering a password? > > If you use an SSH key for passwordless access, try adding -n (ie: > > > "ssh -n derp@host mycommand"). > > Yes, I use an SSH key, and yes, '-n' saves the day. Thanks. Hi Alain, I'm the supposed maintainer of ob-shell who's been missing in action for this whole discussion. I want to confirm, are you able to accomplish your task by using '-n'? Matt
Re: bash source code block: problem after ssh commands
On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > I am wondering about the possible downsides of using script approach > instead of stdin redirection. I'm curious to hear more about what you're thinking.
Re: Maintenance status of individual Org libraries
On Mon, 30 Jan 2023 14:41:18 +0100 Ihor Radchenko wrote --- > The following libraries have no maintainer or author active on the list: > ob-comint, ob-core, ... I'm interested (and technically still maintainer of ob-shell). ob-comint and ob-core are both used by ob-shell, so I've looked at good bits of both. I'd be willing to work on those. Unfortunately, my new job has yet to sign the FSF disclaimer, so I'm unable to contribute at the moment (I'm doing what I can to facilitate communication between parties). Feel free to reach out off-list with advice for how I might help that process. Until the disclaimer language gets resolved, I may be able to help in other ways. I have notes explaining the ob- API (https://lists.gnu.org/archive/html/emacs-orgmode/2023-02/msg00531.html). Maybe if that were completed, it would make being a maintainer more approachable for people? I'd also be happy to mentor new maintainers. What other ways might I help?
Thanks for good docs
Hi, I'm spending more time writing this email than took me to look up what the source block switch is for line numbering. I had an idea of what I wanted and didn't know where to look. Jump into info, go to the org manual, do a few searches across menu and node titles. Then I check the index and there's something promising...takes me right to what I was looking for. Thanks for indexing. Thanks for including examples. So much better than internet search engines. Y'all are doing good things. :) Thanks, Matt
Re: bash source code block: problem after ssh commands
On Tue, 07 Nov 2023 09:53:46 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Mon, 06 Nov 2023 14:32:16 +0100 Ihor Radchenko wrote --- > > > I am wondering about the possible downsides of using script approach > > > instead of stdin redirection. > > > > I'm curious to hear more about what you're thinking. > > I am thinking to change the > (t (org-babel-eval shell-file-name (org-trim body))) > clause in `org-babel-sh-evaluate' to something that uses a script file. > > It will clearly solve the discussed problem, possibly at the cost of > small overhead to write the script file to disk. Thanks for clarifying. I've been away from the codebase for a bit and, now that the FSF paperwork is signed (still need to get Craig a copy), I'm reviewing =ob-shell= with an eye for what could be cleaned or improved. I feel like =org-babel-sh-evaluate= could use some attention. I'm open to your suggestion. The good is that using a script is how :shebang and :cmdline are processed currently (like you pointed out) so there's precedence and experience with it. Also, it would make all non-session execution use the same model (script versus comint). I like how that would create a clean separation. For bad, nothing jumps out to me as obviously a problem. Let me "think out loud" for a moment. We'd need to write to disk. Like you say, this incurs overhead opening, writing, and closing the file. It's not like we'd forget to close it, though. Nor is running out of space or inodes our problem. Writing requires permission. That's not an issue with /tmp. Then, it needs to execute. Aside from permission, any code we insert needs to be correct. For example, a shebang would need to point to the correct application and any arguments would need to correspond to the implementation being called. I doubt we'd need anything beyond /bin/. FWIW, it looks like there's been at least one instance where :shebang's formatting was questioned (https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/). We'd also need to control for what environment the script runs in. That was another issue I saw raised in the list (https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/). Of course, we'd need to read the stdout and stderr. This is handled by =process-file=. Any step I missed or some kind of failure I didn't consider?
Replace org link with footnote
Here's a quick hack to replace an Org link with a footnote. Maybe a fun first commit for someone would be to update `org-footnote-new` to accept optional LABEL and DEFINITION arguments that allow a footnote to be created programmatically? (defun my-replace-link-with-footnote () "Replace an org link with a footnote. Place the link target in the footnote and the description in the location of the original Org link." (interactive) (if (org-in-regexp org-link-bracket-re 1) (save-excursion (let ((remove (list (match-beginning 0) (match-end 0))) (link (org-match-string-no-properties 1)) (description ; link is description (if (match-end 2) (org-match-string-no-properties 2) (org-match-string-no-properties 1 (apply 'delete-region remove) ;; replaces link with footnote and jumps to footnote (call-interactively 'org-footnote-new nil) (insert " " link) (beginning-of-line) ;; follow link back to original point (org-open-at-point) (insert description)
Re: Org as a workspace (an impromptu reflection)
On Fri, 29 Apr 2022 11:03:55 -0400 Juan Manuel Macías wrote > I don't know if anyone has had a similar experience... I tell people that Emacs changed my life. I feel it's that profound. My story is different from yours, yet similar in that it started with Org. I love that it has this affect on people. It brings a smile to my face. This really is a great community. Thanks for sharing your experience. :)
Re: C-c $ missing?
On Thu, 18 Aug 2022 12:46:23 -0400 Summer Emacs wrote --- > > Hi everyone, first time posting. I hope it's okay to ask this here (I'm > totally new): Welcome! > Since this morning (I updated a few packages but not sure which), C-c $ > is missing in org mode? It's how I used to archive tasks when they were > done. I know that I can use C-c C-x C-s but I used to just use C-c $ and > it worked. Did something change with an update? > > (I think it was C-c $ or maybe C-x $? Sometimes I get the two confused) I don't know if anything changed with Org, I'll let others speak to that. However, in the mean time, check C-h k C-c C-x C-s to see what command is bound to that sequence (is it org-archive-subtree?). Then you can bind it with something like: (eval-after-load 'org-mode (define-key org-mode-map (kbd "C-c C-x C-s") #'org-archive-subtree)) This says, "after loading org-mode for the first time, define the key sequence within the variable telling Emacs what keys do what for org-mode to call the function org-archive-subtree when it sees C-c C-x C-s." The C-h f and C-h v commands describe-function and describe-variable can probably explain that better. > -- > Summer, > Sent from MU4E in Emacs, > Because I'm becoming a NEWBIE at this! We all start somewhere! I hope this helps. Just saw your message as I was passing by. If it's unclear, that's totally my fault!
Re: Bash results broken?
On Sun, 25 Dec 2022 06:23:53 -0500 Ihor Radchenko wrote --- > As you see, "%S" have been used previously for non-string results. > I cannot find explanation in git log. > > That said, I think that it will be more consistent to leave strings > specifically as is. See the attached patch. That makes sense and works for me.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 30 Dec 2022 00:34:38 -0500 Matt wrote --- > > On Thu, 29 Dec 2022 06:08:59 -0500 Ihor Radchenko wrote --- > > > From: Matt Trzcinski m...@excalamus.com> > > > +(require 'org-test (expand-file-name "../org-test.el")) > > > > I am unsure here. What will happen if you run this file from > > default-directory not the same with file location? I was mistaken. Loading a new Emacs instance and running `(load "/path/to/test-ob-shell.el")' fails with file-missing error. I'm content to leave that alone for now. I've backed out the `require' change and adjusted everything else based on your feedback. There is a separate patch for each refactor that created a new test. The remaining refactors are in a single patch. I was also able to resolve the issue I had with inserting the test name for the session (the "yes" string in `test-ob-shell/session'). The issue with `org-babel-shell-initialize' still stands but is outside the scope of these patches and isn't dealt with here. 0001-test-ob-shell.el-Split-test-ob-shell-dont-error-on-e.patch Description: Binary data 0002-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0003-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0004-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data 0005-test-ob-shell.el-Refactor-test-names-and-kill-test-b.patch Description: Binary data 0006-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0007-test-ob-shell.el-Organize-tests.patch Description: Binary data
Refactor org-babel-shell-initialize? (was Re: ob-shell intentions and paperwork (was Bash results broken?))
On Sat, 31 Dec 2022 07:56:10 -0500 Ihor Radchenko wrote --- > As for being a macro, there will be not much gain - the convention is > mostly designed for things like `cl-defun' aimed to be used in the code. > `org-babel-shell-initialize' is only used by `org-babel-shell-names'. I'm not sure what you mean by "to be used in code". Do you mean called within the global scope? > I do not have objections if it were a macro though. (But I do not see > how it would help debugging). > > > Because `org-babel-shell-initialize' is a function factory, you can't > > easily examine or modify their definitions. `C-h f org-babel-execute:sh' > > jumps to the top of lisp/ob-shell.el. Changing the definition requires > > reevaluating the definition for all the execute functions (or first > > changing `org-babel-shell-names'). > > This is indeed a downside. Any better ideas? > ob-core dictates that we must have org-babel-execute:lang functions to > make things work. My apologies, I feel there are too many separate issues I've brought up. Since I've already brought them up, let me try to be more clear about them. I observe four issues with the current form of `org-babel-shell-initialize': 1. The source is not explicit for a given `org-babel-execute:some-shell', making it difficult to debug 2. Source navigation gets confused when looking up help for a generated function. For example, `C-h f org-babel-execute:sh' goes to the top of ob-shell.el 3. It does not adhere to the coding convention 4. Except for using the customize interface, changing `org-babel-shell-names' requires reevaluating the function generator, currently `org-babel-shell-initialize' Here's my perspective on each point. 1. The source is not explicit for a given `org-babel-execute:some-shell', making it difficult to debug The benefit of using a macro is clarity of the defined functions. Here's the core `org-babel-shell-initialize' functionality as a macro. Note that it doesn't loop through `org-babel-shell-names'. (defmacro define-babel-shell-1 (shell-name) (declare (indent nil) (debug t)) (let ((function-name (intern (concat "my-org-babel-execute:" shell-name `(progn (defun ,function-name (body params) ,(format "Execute a block of %s commands with Babel." shell-name) (let ((shell-file-name ,shell-name) (org-babel-prompt-command (or (alist-get ,shell-name org-babel-shell-set-prompt-commands) (alist-get t org-babel-shell-set-prompt-commands (org-babel-execute:shell body params))) (defalias ',(intern (concat "org-babel-variable-assignments:" shell-name)) 'org-babel-variable-assignments:shell ,(format "Return list of %s statements assigning to the block's variables." shell-name)) (defvar ,(intern (concat "org-babel-default-header-args:" shell-name)) nil You can expand to see the definitions: (pp-macroexpand-expression '(define-babel-shell-1 "csh")) Is there a way to see the definition of`org-babel-execute:csh' using the current `org-babel-shell-initialize', that is, when generated by a function? Looking at the expansion, I see what appears to be an error: (alist-get "csh" org-babel-shell-set-prompt-commands) `org-babel-shell-set-prompt-commands' is an alist keyed by string shell names and whose values are shell commands to set the prompt. `alist-get' uses `eq' by default which always returns nil for string comparisons. That is, (alist-get "csh" org-babel-shell-set-prompt-commands) returns nil, not because the corresponding alist value is nil. Rather, because the (eq "csh" "csh") comparison evaluates as nil. The TESTFN probably should be `equal' or `string=': (alist-get "csh" org-babel-shell-set-prompt-commands nil nil 'equal) Then, it will return the prompt associated with "csh". All this is visible from the function version of `org-babel-shell-initialize', of course. However, it requires mentally tracking that ,name resolves to a string. Using a macro along with `pp-macroexpand-expression' makes the defined function explicit. The forms generated by the macro expansion are readily available to eval whereas the function version makes the definitions inaccessible (AFAICT). 2. Source navigation gets confused when looking up help for a generated function. For example, `C-h f org-babel-execute:sh' goes to the top of ob-shell.el Generating the execute functions with a macro also has this problem. I'm not sure there's a way around it other than defining each function with `defun'. Doing that would be a bad idea because of the massive code duplication/maintenance it would create. 3. It does not adhere to the coding convention. I'll requote the convention here for convenience. > (elisp) Coding Conventions says, > > "• Constructs that define a function or variable should be macros, not > functions, and their names should start
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Sat, 31 Dec 2022 09:31:16 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I've backed out the `require' change and adjusted everything else based on > > your feedback. There is a separate patch for each refactor that created a > > new test. The remaining refactors are in a single patch. I was also > > able to resolve the issue I had with inserting the test name for the > > session (the "yes" string in `test-ob-shell/session'). > > Thanks! > > I can see that you still prefer to use `string-join' for Org body. > Why not a bare string? Because I often program in Python where double quoted strings typically don't span lines. One of the original tests had the string "#+BEGIN_SRC sh :results list\necho 1\necho 2\necho 3\n#+END_SRC" in it (that is, used newline characters instead of inserting new lines), so my brain assumed strings Emacs Lisp behaved similarly to Python. Silly brain :) Thanks for continuing to follow up on that. I've updated the patches to use bare strings. If this set of patches look good, I can push them to main. 0001-test-ob-shell.el-Split-test-ob-shell-dont-error-on-e.patch Description: Binary data 0002-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0003-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0004-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data 0005-test-ob-shell.el-Refactor-test-names-and-kill-test-b.patch Description: Binary data 0006-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0007-test-ob-shell.el-Organize-tests.patch Description: Binary data
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Mon, 02 Jan 2023 04:47:10 -0500 Ihor Radchenko wrote --- > They will not be reliable when tests are executed interactively. > If the `should' condition fails, `kill-buffer' will never be executed > leaving dirty state, especially for sessions. >From my perspective, that's the point and a good thing. That "dirty state" >information may be crucial to understanding why the failure happened. Can you explain why you would want a failed test to remove failure related information?
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Thu, 05 Jan 2023 06:21:16 -0500 Bastien Guerry wrote --- > My bad: I did not warn Emacs maintainers in time. Now it is done, > I will let you know when they grand you access to the Emacs project. I got an email from Eli on Thursday saying I was added. I've still been getting an error when trying to push: fatal: remote error: access denied or repository not exported: /emacs/org-mode.git
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Tue, 03 Jan 2023 05:50:17 -0500 Ihor Radchenko wrote --- > I was mostly worried about session states affecting subsequent test > invocations. But I do agree that it may be better to keep them. That makes sense. I tend to run tests one at a time unless I'm about to submit patches or push... > Feel free to push upstream. I'm not able to push to git://git.sv.gnu.org/emacs/org-mode.git. I've read through the following and, as far as I can tell, I've followed the directions. - https://orgmode.org/worg/org-contribute.html - https://orgmode.org/worg/org-maintenance.html - https://www.emacswiki.org/emacs/GitForEmacsDevs
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 13 Jan 2023 04:36:40 -0500 Ihor Radchenko wrote --- > Cgit displays our bugfix merges with all the required commits. > So, what happened what not ideal either way. Would you like me to correct how I've incorporated my changes?
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 11 Jan 2023 12:02:42 -0500 Ihor Radchenko wrote --- > It looks like you lost all the individual commits and commit messages in > the process. > > Could you please revert 4f319088ba5 and re-push in such a way that > individual commits do appear on main? Goodness! Sorry! Thanks for catching that. I reverted the merge and then applied the patches.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 11 Jan 2023 06:53:41 -0500 Ihor Radchenko wrote --- > I have the following Git configuration: > > u remote.origin.url > yanta...@git.savannah.gnu.org:/srv/git/emacs/org-mode.git > > Are you using the same? I was not, thank you. I've since pushed.
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Thu, 29 Dec 2022 06:08:59 -0500 Ihor Radchenko wrote --- > Does it mean that you are willing to maintain lisp/ob-shell.el? > We usually give write access to the maintainers and regular > contributors. AFAIR, you previously contributed to WORG but not to Org > core. You're correct, I've not contributed to core. I would love to maintain lisp/ob-shell.el. I'm expecting life changes in the coming months and can't anticipate how that will affect my time. Would it be a problem if I need to step down as maintainer for a period? > > From: Matt Trzcinski m...@excalamus.com> > > > > [fn:1] Session name is a string and not a variable (such as > > `session-name'). This is because `org-babel-execute:sh' produces a > > type error when the session name is defined with a variable. The > > source of the error appears to be the `params' symbol in > > `org-babel-execute:sh'. `params' does not evaluate as a variable as > > expected–it evaluates as a symbol. However, `org-babel-execute:sh' is > > defined within a function factory which makes it difficult to debug. > > Hard code the test name for now and refactor it later once > > `org-babel-execute:sh` is refactored. > > There is nothing wrong here. `org-babel-execute-src-block' takes care > about parameter processing making sure that :session value is always a > string. Originally the test used "yes" for a comparison string and the shell it ran in. I changed "yes" to be the test name because when it was "yes", I didn't know what test had produced it. However, the problem with the test name being hard coded as a string is that if the function name changes, the test string may get out of sync. More on this below. > > +;; TODO refactor into macro. Currently violates (elisp) Coding > > +;; Conventions and is hard to debug. > > (defun org-babel-shell-initialize () > >"Define execution functions associated to shell names. > > Could you please elaborate? Which particular convention does it violate? > What is hard to debug? (elisp) Coding Conventions says, "• Constructs that define a function or variable should be macros, not functions, and their names should start with ‘define-’. The macro should receive the name to be defined as the first argument. That will help various tools find the definition automatically. Avoid constructing the names in the macro itself, since that would confuse these tools." The `org-babel-shell-initialize' function defines *all* the `org-babel-execute:XXX' functions given by `org-babel-shell-names' (sh, bash, zsh, etc.). Because `org-babel-shell-initialize' is a function factory, you can't easily examine or modify their definitions. `C-h f org-babel-execute:sh' jumps to the top of lisp/ob-shell.el. Changing the definition requires reevaluating the definition for all the execute functions (or first changing `org-babel-shell-names'). This was a problem for me when I wanted to make the session name string for `test-ob-shell/session' the test name (mentioned above). In the test, when I replaced the session name string with a variable containing the string, `org-babel-execute:sh' failed with a type error. I couldn't get the variable to evaluate (with backquote and comma or otherwise). Without an explicit function definition or a macro to expand, I found it hard to debug/experiment with (and so left the test name as a hard coded string). > > (ert-deftest test-ob-shell/dont-insert-spaces-on-expanded-bodies () > >"Expanded shell bodies should not start with a blank line unless > > -the body of the tangled block does." > > +the strings of the tangled block does." > > What does "strings of the tangled block" refer to? The previous variant > is a lot more clear for me. I believe this is a query-replace error. Good catch! > > +(ert-deftest test-ob-shell/generic-uses-no-arrays () > > + "Test generic serialization of array into a single string." > > + (org-test-with-temp-text > > + (test-ob-shell-multiline-string > > + "#+NAME: sample_array" > > + "| one |" > > + "| two |" > > + "| three |" > > Why do you need `test-ob-shell-multiline-string' here? > Can simply type-in the string directly, as the rest of tests do. I probably don't need it and am happy to remove it.An older version of the function was more complex and made sense as a separate function (or so I thought). My aim was to make the test strings easy to read so that it's clearer what's being tested (i.e. not write multi-line strings on a single line). I could use concat and add "\n
Re: ob-shell intentions and paperwork (was Bash results broken?)
On Wed, 21 Dec 2022 01:17:50 -0500 Matt wrote --- > Currently, though, I'm refactoring the ob-shell tests to remove dependency > on ob-shell-test.org and to stop the suite from littering. Done. Branched off bugfix, 12e10eb0d, and refactored test-ob-shell.el. See attached patches. The main changes spanning the patches are: - tests are now standalone; they don't rely on ob-shell-test.org - tests now share a common prefix, "test-ob-shell". The entire suite runs with (ert "test-ob-shell") - buffers and processes created during a test are killed when the test passes > After that, I intend to incorporate the worg examples as tests. This is probably a good place for me to pause and get feedback before writing more code. I just got the signed paperwork back from Craig and am waiting to be admitted to the Emacs group on Savannah. 0001-test-ob-shell.el-Remove-mixed-tabs-and-spaces.patch Description: Binary data 0002-test-ob-shell.el-Split-cases-into-two-tests.patch Description: Binary data 0003-test-ob-shell.el-Refactor-test-ob-shell-session.patch Description: Binary data 0004-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0005-test-ob-shell.el-Refactor-ob-shell-bash-uses-arrays.patch Description: Binary data 0006-test-ob-shell.el-Refactor-ob-shell-generic-uses-no-a.patch Description: Binary data 0007-test-ob-shell.el-Refactor-ob-shell-bash-uses-assoc-a.patch Description: Binary data 0008-test-ob-shell.el-Refactor-ob-shell-simple-list.patch Description: Binary data 0009-test-ob-shell.el-Refactor-ob-shell-remote-with-stdin.patch Description: Binary data 0010-test-ob-shell.el-Refactor-ob-shell-results-table.patch Description: Binary data 0011-test-ob-shell.el-Refactor-ob-shell-results-list.patch Description: Binary data 0012-test-ob-shell.el-Refactor-ob-shell-standard-output-a.patch Description: Binary data 0013-test-ob-shell.el-Refactor-ob-shell-standard-output-a.patch Description: Binary data 0014-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0015-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0016-test-ob-shell.el-Refactor-ob-shell-error-output-afte.patch Description: Binary data 0017-test-ob-shell.el-Refactor-ob-shell-exit-code.patch Description: Binary data 0018-test-ob-shell.el-Refactor-ob-shell-exit-code-multipl.patch Description: Binary data 0019-ob-shell-test.org-Remove-ob-shell-test.org.patch Description: Binary data 0020-test-ob-shell.el-Organize-tests.patch Description: Binary data
Re: Bash results broken?
On Mon, 19 Dec 2022 19:44:12 -0500 Rudolf Adamkovič wrote ---
> Ihor Radchenko yanta...@posteo.net> writes:
>
> > Fixed on bugfix.
>
> I can confirm that the fix works. Thank you!
I can confirm it works, too, for the 1 2 3 example. However, it puts double
quotes around text:
#+BEGIN_SRC sh :results list
echo hello
echo world
echo !
#+END_SRC
#+RESULTS:
- "hello"
- "world"
- "!"
I was working on other ob-shell related stuff and my Emacs got into a mixed
state where =ob-shell/results-list= was defined, but the fix wasn't (or wasn't
loaded). I fixed the failing test before I saw this thread and Ihor's fix.
The way I solved it was to wrap either =org-babel-import-elisp-from-file= or
=org-babel-sh-evaluate= like,
(mapcar #'car (org-babel-import-elisp-from-file tmp-file))
or
(mapcar #'car (org-babel-sh-evaluate session full-body params stdin cmdline))
Doing this (without Ihor's fix in place) returns:
#+BEGIN_SRC sh :results list
echo "hello"
echo "world"
echo "!"
#+END_SRC
#+RESULTS:
- hello
- world
- !
The underlying issue, so far as I understand it, is that
=org-babel-import-elisp-from-file= returns a list-of-lists. For example, ((1)
(2) (3)) or (("hello") ("world") ("!")). I thought it appropriate to flatten
it out to (1 2 3) or ("hello" "world" "!").
Independently from how I addressed the bug, I feel it raises an important
question: how should :results list handle text/strings?
I'm not sure which approach is more "appropriate". If I echo a string without
quotes, as I did in the first example, should it return a list that's got
quotes? The hello world example is a little contrived–it's good practice to
use double-quotes. I think a more realistic use case would be listing a
directory. In that case, I'd probably want the list of items to be unquoted:
Original:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- ("test-ob-awk.el")
- ("test-ob-C.el")
- ("test-ob-clojure.el")
Ihor solution:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- "test-ob-awk.el"
- "test-ob-C.el"
- "test-ob-clojure.el"
MAPCAR solution:
#+BEGIN_SRC sh :results list
ls /home/ahab/.emacs.d/straight/repos/org/testing/lisp/ | head -n 3
#+END_SRC
#+RESULTS:
- test-ob-awk.el
- test-ob-C.el
- test-ob-clojure.el
For :results list, the manual says, "Interpret the results as an Org list. If
the result is a single value, create a list of one element". I don't find that
clarifies what would be best.
Thoughts?
ob-shell intentions and paperwork (was Bash results broken?)
On Fri, 16 Dec 2022 12:41:45 -0500 Ihor Radchenko wrote --- > We really need more tests. I'm working on giving ob-shell a little bit of love. I wrote the worg documentation for it earlier this year. I tried to include examples of all coded functionality, including previously undocumented features. I think the examples would make for good tests since they document the present and expected behavior. Currently, though, I'm refactoring the ob-shell tests to remove dependency on ob-shell-test.org and to stop the suite from littering. After that, I intend to incorporate the worg examples as tests. I'd like to adopt ob-shell, life permitting. Sadly, I'm not confident I'm able to commit to that right now. So, if I can get the FSF paperwork cleared, hopefully, I can make little contributions here and there. I've included patches of what I've been doing. They're not ready for actual submission yet, both in quality and legality. However, I'm including them for feedback, to make sure I'm not getting way off base. Overall, I'd *really* like to implement async for ob-shell similar to ob-python. However, I feel like it'd be more responsible and respectful to clean up the tests and code a little first before proposing such a big change (looking at you, org-babel-shell-initialize). Can someone please assist me with the FSF paperwork off list? 0001-Remove-mixed-tabs-and-spaces.patch Description: Binary data 0002-Split-test-ob-shell-dont-error-on-empty-results.patch Description: Binary data 0003-Refactor-test-ob-shell-session.patch Description: Binary data 0004-Refactor-ob-shell-generic-uses-no-arrays.patch Description: Binary data 0005-Refactor-ob-shell-bash-uses-arrays.patch Description: Binary data 0006-Refactor-ob-shell-generic-uses-no-assoc-arrays.patch Description: Binary data 0007-Refactor-ob-shell-bash-uses-assoc-arrays.patch Description: Binary data
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Mon, 16 Jan 2023 16:40:27 -0500 Matt wrote --- > That is, go through the steps to reproduce and before executing the block, > run `M-: (setq shell-command-switch "/k")'. Whoa, just re-read this after stepping away and it sounds super demanding! Please excuse me, I've been trying to use active voice while writing documentation. Let me try that again: What happens if you go through the steps to reproduce and run `M-: (setq shell-command-switch "/k") before executing the block?
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
Thank you for your report, Osher! Windows shells aren't currently supported by ob-shell, AFAIK. I'm open to including them. Unfortunately, I don't have a Windows machine to test against. On Mon, 16 Jan 2023 11:27:52 -0500 Osher Jacob wrote --- > Expected behaviour: > On Windows, all lines of the babel shell block should be evaluated, with > full output printed. What's does `C-h v shell-file-name' say? That should tell us what shell is being used. > #+begin_src shell > echo 1 > echo 2 > > > #+end_src > > #+RESULTS: > | Microsoft | Windows | [Version | 10.0.19044.2364] | > | | > | (c) | Microsoft | Corporation. | All | > rights | reserved. | > | | | | | > | | > | c:\Users\osherj>echo | 1 | | | > | | > | 1 | | | | > | | > | | | | | > | | > | c:\Users\osherj>More? | | | | > | | > > The hacky way I solved it was to change this line: > (t (org-babel-eval shell-file-name (org-trim body)) > to this: > (t (org-babel-eval shell-file-name (concat (org-trim body) "\n")) I don't think org-trim is the issue. Running the block (eventually) calls the default shell command, `org-babel--shell-command-on-region'. This calls `org-babel--get-shell-file-name' on a temp file containing the block source using the "-c" flag. I assume the shell used is cmdproxy.exe (https://git.savannah.gnu.org/cgit/emacs.git/tree/nt/cmdproxy.c). It looks like this converts a "-c" to a "/c", among other things. MSDN says for cmd.exe, Parameter Description /c Carries out the command specified by string and then stops. /k Carries out the command specified by string and continues. So, the way I reason it, ob-shell tries calling cmdproxy.exe using /c which basically calls cmd.exe /c, the block temp file is executed, and the shell stops. I see that `org-babel--shell-command-on-region' calls `process-file' using the `shell-command-switch'. It doesn't appear to be set anywhere else in `ob-eval' (see ob-eval:112). > Let me know if there's any other information you need, or if I can help in > any other way. I wonder if changing `shell-command-switch' to "/k" would make a difference? That is, go through the steps to reproduce and before executing the block, run `M-: (setq shell-command-switch "/k")'.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Tue, 17 Jan 2023 14:53:39 -0500 Osher Jacob wrote ---
> changing shell-command-switch to "/k" or "-k", I get a similar output:
Thanks for checking that.
> You also mentioned the source code block is being passed through the "-c"
> flag as a command-line argument.I might be misunderstanding something here,
> but it seems like it is being passed through the stdin of the shell process
> when the calls process-file -> call-process are being made.
That the block source is passed through "-c" is an educated guess. I keep
finding myself in C land where it's less clear to me what precisely happens.
You're correct that `call-process' acts on the block source. It's not clear to
me whether that's done through stdin or whether being passed through stdin
would even help us. Here's what I'm seeing.
Evaluating the block goes through 4 lisp commands and then into C:
1. Calls `(org-babel-eval shell-file-name (org-trim body))'
This evaluates as...
(org-babel-eval
"cmdproxy.exe" ; command, whatever `shell-file-name' is
(org-trim body) ; query, the block body
)
The query parameter (the block body) gets inserted into a temp buffer on which
`org-babel--shell-command-on-region' is called.
2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp
buffer containing block body.
This evaluates as...
(org-babel--shell-command-on-region
command ; "cmdproxy.exe"
error-buffer ; #
)
This in turn calls `process-file' where INPUT-FILE is a temp file containing
the contents of the buffer `org-babel--shell-command-on-region' was called on
(that is, the temp file contains the block body).
3. Calls `(process-file shell-file-name input-file (if error-file (list t
error-file) t) nil shell-command-switch command)'
This evaluates as...
(process-file
"cmdproxy.exe" ; shell-file-name
"/path/to/temp/containing/block/source" ; input-file
(t "/tmp/babel jTCHe/ob-error-yHOivA"); output destination (like
call-process's DESTINATION)
nil ;
display
"-c"; args
"cmdproxy.exe" ; args
)
Of course, I imagine the paths would be Windows paths when run on Windows.
According to the documentation, the args are passed to the process
(cmdproxy.exe) "verbatim without filename handling, as `call-process' does."
The `call-process' documentation says, "remaining arguments ARGS are strings
passed as command arguments to PROGRAM."
To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe. This
would explain the nested shell calls. If all this were written out, I imagine
it would look something like:
cmdproxy.exe -c cmdproxy.exe
What's not clear to me is how the INPUT-FILE is handled.
The `process-file' documentation says "normally". I assume it's how
`call-process' handles INPUT-FILE, but the documentation doesn't really address
it.
Finally, `process-file' finally calls `call-process'.
4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list
(car buffer) stderr-file) buffer) display args)'
This evaluates as...
(apply 'call-process
program
; cmdproxy
(or lc infile)
; local-copy of the INFILE, "/path/to/temp/containing/block/source"
(if stderr-file (list (car buffer) stderr-file) buffer) ; (t
"/tmp/babel jTCHe/ob-error-yHOivA" )
display
; nil
args)
; ("-c" "cmdproxy.exe")
How this actually works is non-trivial
(https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not
something I understand at the moment. We can see here that indeed a call like
`cmdproxy.exe -c cmdproxy.exe' is being made. Still, I'm not sure how the
INPUT-FILE gets processed. For example, is it passed in the second cmdproxy
call? Or, maybe it gets called first and then the second call to cmdproxy
happens and hangs? I don't know.
> Any ideas on how to proceed from here?
I have two ideas.
1. Another naive work around attempt. Again, I'm going from memory,
documentation, and what I have previously written.
I have in my init a command to open a terminal when working on Windows that
looks like:
(start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir)
This starts a cmd process with /c which terminates the prompt after the command
that follows is run. The command That follows starts another cmd process with
/k which changes the directory and leaves the terminal open.
I have another command to open QGIS that does the same thing:
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Thu, 19 Jan 2023 11:28:09 -0500 Osher Jacob wrote --- > If we're insistent on passing the input through the command line arguments, > I can think of two ways to go about this, but both seem undesirable They're good ideas and, I agree, aren't ideal. > I think it could be enough to make sure the input ends with a newline, in > which case it could be done the way I proposed in the first message, that is > changing:(t (org-babel-eval shell-file-name (org-trim body)) > to(t (org-babel-eval shell-file-name (concat (org-trim body) "\n")) Your original proposal was reasonable and looks even more so now. > I think as long as this change doesn't break the code in non-Windows > operating systems (How would we go about verifying this?), it would be > preferable due to its simplicity and minimality. Agreed, I like its simplicity. Tests are a first step and they all pass with your suggested change. > If for any other reason you would rather not change the > org-babel-sh-evaluate, and would like to implement a separate function for > Windows, that's also a viable option.I am willing to try and go down that > path, although I'm not a very experienced lisper. > Do any of these options sound like they could work well? I agree, putting in the newline is simple and it would be easy enough to create a separate path for Windows/cmd if it were a problem. I wonder if it might make sense to move where the trimming happens so that the body passed into `org-babel-sh-evaluate' is ready to go and we could handle the cmd case separately and earlier (by inserting the newline there)? I appreciate you taking the time to explore and for being open to what feels like me over-thinking it :) I'll be away from the keyboard for a few days and I look forward to getting this wrapped up for you. Meanwhile, I'm curious if Ihor or others have advice.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Fri, 20 Jan 2023 04:27:18 -0500 Ihor Radchenko wrote --- > I think `org-babel--shell-command-on-region' will be more appropriate. > Because similar issues might appear when attempting to evaluate other > code blocks on Windows, where `shell-file-name' is set to cmdproxy.exe. > > It will also be useful to leave a comment in the code explaining why we > need this newline. Works for me. How should I update bugfix to match main? Since we're considering this a bug, I figured I'd do this on bugfix. I notice, however, that bugfix doesn't have the latest updates to test-ob-shell. When I rebase main onto bugfix I get a ton of conflicts. If I instead merge main into bugfix, that shows many merge commits with bugfix. Since I don't see any "Merge with 'bugfix'" commits on bugfix currently, I assume that's not the way to do it.
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
> On Fri, 20 Jan 2023 04:27:18 -0500 Ihor Radchenko wrote --- > I think `org-babel--shell-command-on-region' will be more appropriate. > Because similar issues might appear when attempting to evaluate other > code blocks on Windows, where `shell-file-name' is set to cmdproxy.exe. Is something like this what you're thinking? Or, do we want to check explicitly for "cmdproxy.exe"? Also, this bug isn't currently tracked in Woof!. I'm not able to confirm it, but since we're putting in a fix, should be add it anyway? ob-eval-handle-windows-shell.patch Description: Binary data
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Thu, 26 Jan 2023 04:51:19 -0500 Ihor Radchenko wrote --- > This will add a newline to "cmdproxy.exe" command -> "cmdproxy.exe\n". > You should instead look into `org-babel--write-temp-buffer-input-file'. I made the change in `org-babel--shell-command-on-region' based on your recommendation in https://list.orgmode.org/orgmode/87bkmttv2h.fsf@localhost/ I'd be happy to put it in `org-babel--write-temp-buffer-input-file'. Unfortunately, I don't understand why you suggest the change be made there. Wouldn't it make more sense to insert the newline into the temp buffer that `org-babel--write-temp-buffer-input-file' writes from? I'm confused because I don't see a good way write a newline from `org-babel--write-temp-buffer-input-file'. All the file writing functions[fn:1], with the exception of `with-temp-file', require a region's start and end and `with-temp-file' appears to overwrite the file. ;; 99.999% sure this isn't what you meant (defun org-babel--write-temp-buffer-input-file (input-file) "Write the contents of the current temp buffer into INPUT-FILE." (let ((start (point-min)) (end (point-max))) (goto-char start) (push-mark (point) 'nomsg) (write-region start end input-file) +(with-temp-buffer + (insert "\n") + (write-region (point-min) (point-max) input-file)) (delete-region start end) (exchange-point-and-mark))) AFAICT, `org-babel--write-temp-buffer-input-file' is only called by `org-babel--shell-command-on-region' and that's only ever called by `org-babel-eval'. That's where the query (the block source) is inserted into a temp buffer. Maybe something like this? modified lisp/ob-eval.el @@ -64,7 +64,7 @@ Writes QUERY into a temp-buffer that is processed with (let ((error-buffer (get-buffer-create " *Org-Babel Error*")) exit-code) (with-current-buffer error-buffer (erase-buffer)) (with-temp-buffer - (insert query "\n") + (insert (concat query "\n")) (setq exit-code (org-babel--shell-command-on-region command error-buffer)) [fn:1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Writing-to-Files.html#Writing-to-Files
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
Pushed the change to `org-babel-eval'.
Re: [BUG] Org-Babel shell problems and crashes with images [9.6.1 (9.6.1-ga6c882 @ /home/joe/.emacs.d/straight/build/org/)]
On Wed, 25 Jan 2023 14:34:19 -0500 Josep Jesus Bigorra Algaba wrote --- > This is my first time posting to the mailing list Welcome and thanks for taking the time to send us a message! I love hearing how much you've enjoyed your experience with Org. We love it, too! > 2- Org babel tangle for shell scripts seems to exhibit strange behaviour > in latest build, > and it adds a strange BOM character at beggining of the script. This > makes it that the script is not runnable anymore. I can > fix this by manually going to exported file and doing a > =set-buffer-file-coding-system RET utf-8= . That seems to remove that > BOM character. > > My literate scripts use these options, thought sometimes shebang as > =#!/usr/bin/zsh= : > > #+begin_example > > #+title: Joe's spin on openSUSE Tumbleweed > #+property: header-args :tangle install :comments org :shebang > #!/usr/bin/bash > #+auto_tangle: t > > #+end_example I'm the ob-shell maintainer, so I'm most interested in your second point, the one related to shell things. 1. Your first point mentions that you've reproduced the bug in Emacs 28 through 30. Is the same true for this second, shell/tangle related issue? 2. Does the problem happen with both bash and zsh or just zsh? 3. Can you please give steps to reproduce the problem? I'm wondering if something like this is enough to reproduce it or if you're doing something different. (Note that I'm using #!/usr/bin/env zsh instead of #!/usr/bin/zsh because of how my system is set up.) #+property: header-args :tangle install :comments org :shebang #!/usr/bin/env zsh #+begin_src emacs-lisp :export no :tangle no (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src #+begin_src sh readlink /proc/$$/exe #+end_src 4. Also, are you running into the issue when starting Emacs using `emacs -Q'?
Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]
On Wed, 01 Feb 2023 07:05:45 -0500 Ihor Radchenko wrote --- > This and another commit are rather trivial - they are eligible for > bugfix. See https://orgmode.org/worg/org-maintenance.html#branches > I cherry-picked the commits for bugfix. Noted for the future. > I also added a comment clarifying the purpose of the newline. > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=b4a1133a1 My apology for not carrying over the note. Thank you for correcting it.
Re: [PATCH] Async evaluation in ob-shell
Hi Jack and Jeremie! I'm curious your thoughts about what Ihor and I are discussing at the end of this message about `md5'. On Tue, 07 Mar 2023 07:45:02 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > The actual prompt is "org_babel_sh_prompt> ". > > > > Agreed. > > > > > And we want to skip leading spaces in addition. > > > > What do you mean by this? > > I was following the pattern described in the docstring of > `comint-prompt-regexp', where it is suggested that prompts should follow > the pattern "^ *". > > In the case of ob-shell.el, `org-babel-sh-prompt' is a string to be used > as a prompt and the corresponding regexp patter will be > "^ *". Hence > > (concat "^" (regexp-quote org-babel-sh-prompt) " *") > > > > Adding " *" does not make the prompt match 2 spaces, but 1+. > > > > Agreed. > > > > It's not clear to me what pattern you're looking to match. > > I hope the above clarified things. I'm confused because when I run the Org from HEAD, I get: (concat "^" (regexp-quote org-babel-sh-prompt) " *") -> "^org_babel_sh_prompt> *" That's *two* spaces between '>' and '*', not one. The second space comes from either 1) the definition of `org-babel-sh-prompt', which is "org_babel_sh_prompt> " (with a single space) or 2) the " *" in the (concat "^" (regexp-quote org-babel-sh-prompt) " *") expression. Currently, the two combine via the concat to give *two* spaces in the regexp. If I understand you correctly, you're trying to match the pattern given in the `comint-prompt-regexp' which is *one* space. That's what I'm trying to do, too. Way back in https://list.orgmode.org/87sfeyc7qr.fsf@localhost/, we had this exchange: On Mon, 20 Feb 2023 11:24:52 + Ihor Radchenko wrote --- > Matt writes: > > > +(defun ob-shell-async-chunk-callback (string) > > + "Filter applied to results before insertion. > > +See `org-babel-comint-async-chunk-callback'." > > + (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string)) > > Why not using `comint-prompt-regexp'? I switched out `org-babel-sh-prompt' with `comint-prompt-regexp' so that the expression looks like: +(defun ob-shell-async-chunk-callback (string) + "Filter applied to results before insertion. +See `org-babel-comint-async-chunk-callback'." + (replace-regexp-in-string comint-prompt-regexp "" string)) This causes the new test `test-ob-shell/session-async-evaluation' to fail, as you pointed out: https://list.orgmode.org/87bkl96g6e.fsf@localhost/ The test fails when we switch out the prompt in the callback because `comint-prompt-regexp' has two spaces in it. The second space causes a prompt to not be filtered (by the callback). The output becomes ": 1\n: 2\n: org_babel_sh_prompt>\n" instead of ": 1\n: 2\n" . This looks like a bug in the `comint-prompt-regexp''. It could be that `test-ob-shell/session-async-evaluation' doesn't test correctly, but it looks right to me (I could certainly be mistaken). Therefore, I see only two options to fix it: remove a space from the concat expression (which I did in my latest patch) or remove a space from `org-babel-sh-prompt'. Am I missing something? > > > `md5' will be slightly faster compared to `org-id-uuid'. But it should > > > not matter. > > > > > > If we want use `org-id-uuid', lets move it to org-macs.el. Requiring the > > > whole org-id.el must not be done. It has side effects of defining id: > > > links. > > > > In the next revision (once we figure out the regex), I can create a > > separate commit moving `org-id-uuid' to org-macs.el and updating ob-R and > > ob-python from `md5' to `org-id-uuid' (assuming that's not an issue for > > the maintainers of those). If you think speed is a concern, however, I > > can switch ob-shell.el to use plain `md5'. > > I am in favour of using `org-id-uuid'. It might also be a useful generic > function for other purposes. > > A slight concern is that some third-party code might depend on the > current pattern used for UUID string in ob-python. But we made no > promises here. > > To be a bit safer, we can also refactor `org-uuidgen-p' exposing the > regexp used to match UUID. Also, it will make sense to move > `org-uuidgen-p' to org-macs.el. I'm okay with all that. I wonder, do Jack Kamm (of ob-python fame) and Jeremie Juste (of ob-R fame) have any thoughts on the matter. I ask out of courtesy since they're the maintainers of those packages and I don't want to cross any boundaries by changing their implementations beneath them.
Re: [SUGGESTION] ob-shell async result output should not contains shell prompt
On Wed, 22 Mar 2023 23:25:50 -0400 Christopher M. Miles wrote --- > > The ob-shell async result output contains the shell prompt. I think it > should not be captured. > > #+begin_src shell :session "test2" :async t > sleep 30 > echo "hello, world" > #+end_src > > #+RESULTS[(2023-03-23 11:19:22) 461ed5de684f6e619890709175ec73e80b67b2d6]: > : bash-5.2$ hello, world Thanks for reporting this. Try using for the babel language whatever shell the variable `shell-file-name' gives. For example, if `shell-file-name' is /bin/bash, do this: #+begin_src bash :session "test2" :async t sleep 1 echo "hello, world" #+end_src Is there a reason you're using "shell" instead of one of the shells listed in `org-babel-shell-names'?
Re: [SUGGESTION] ob-shell async result output should not contains shell prompt
On Thu, 23 Mar 2023 07:12:29 -0400 Christopher M. Miles wrote --- > #+begin_src bash :session "*ob-shell-bash*" :async t > sleep 30 > echo "hello, world" > #+end_src > > #+RESULTS[(2023-03-23 19:14:15) 23f9ad130f7a1268e21821c6baaea2b057c70d3e]: > : org_babel_sh_prompt> hello, world > > Still got a prompt. Is this intended? I think the output should be kept > clean because the result > output might be used as input for other source blocks as data. I absolutely agree. This is a bug. It should be an easy fix on my end but, in case there are details I'm overlooking, this specific example can use the following workaround: #+begin_src bash :session "*ob-shell-bash*" :async t sleep 30 && echo "hello, world" #+end_src To explain what's going on... #+begin_longwinded_explanation Shell output is filtered for prompts which should be removed before inserting the results. The issue is that the filter assumes the prompt starts at the beginning of the line. When sleep is called, it returns nothing and the next prompt appears on the same line: sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2= org_babel_sh_prompt> echo 'ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d' echo "1" sleep 3 echo "2" echo 'ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d' ob_comint_async_shell_start_770d9c8f-deda-4359-aee9-a433a75a5e0d org_babel_sh_prompt> 1 org_babel_sh_prompt> org_babel_sh_prompt> 2 org_babel_sh_prompt> ob_comint_async_shell_end_770d9c8f-deda-4359-aee9-a433a75a5e0d Changing the `ob-shell-async-chunk-callback' like this will fix it: @@ -276,7 +276,7 @@ See `org-babel-comint-async-indicator'.") (defun ob-shell-async-chunk-callback (string) "Filter applied to results before insertion. See `org-babel-comint-async-chunk-callback'." - (replace-regexp-in-string comint-prompt-regexp "" string)) + (replace-regexp-in-string (concat (regexp-quote org-babel-sh-prompt) " *") "" string)) (defun org-babel-sh-evaluate (session body params stdin cmdline) "Pass BODY to the Shell process in BUFFER. #+end_longwinded_explanation
Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)
> Matt m...@excalamus.com> writes: > > > Is there a reason you're using "shell" instead of one of the shells listed > > in `org-babel-shell-names'? I'm still curious why you're using "shell". I want to know if it's something you're using for a specific reason. There's no wrong answer! I ask because I have an agenda: as far as I can tell, "shell" as a Babel language is a historical accident. #+begin_longwinded_explanation Originally, ob-shell.el was called ob-sh.el. The main function was called `org-babel-execute:sh' and only /usr/bin/env sh was supported. Over time it became clear that to support other shells, the "sh" name shouldn't be used for the package or the main function. That is, 'sh' refers to a specific binary and, if other binaries such as bash, dash, csh, etc. were to be supported, it would be misleading for the Babel language to refer to a specific shell, 'sh'. So, the terminology was changed to something more general, "shell". The package was renamed to "ob-shell.el", the "namespace" updated to "shell" (for example, `org-babel-execute:shell'), and the package load call changed from (sh . t) to (shell . t). This officially happened with Org 8.2 (ORG-NEWS noted the change in commit 1a9adcb6d34bcbdff45445c827ed99dbf0b8, Tue Jan 21 09:52:31 2014). I think this gave people the (understandable) impression that "shell" was a valid Babel language, in addition to those listed in `org-babel-shell-names'. And this is where the accident happened: "shell" as a Babel language only **happens**to work. The Babel framework looks for a function prototype like "org-babel-execute:". When ob-sh.el was changed to ob-shell.el, the function `org-babel-execute:sh' became `org-babel-execute:shell'. A call like follows is perfectly legal as far as the Babel framework is concerned: #+begin_src shell echo "hello, world" #+end_src When such a block is run, Babel looks for a function called `org-babel-execute:shell'. Running the block prior to Org 8.2 should have failed because no `org-babel-execute:shell' function existed. The name change happened to source Fri Dec 13 09:52:05 2013 in commit 7a6c0e35415c4a173d101336029262f3a09abb91. After the name change, the function existed and a block using "shell" would execute! The "shell" language specifier, as far as I can tell, was never really intentionally supported. Instead, it just happened to work. It happened to work because, as far back as the first org-babel-sh.el commit, the process buffer is created using the `shell' function. I don't know the history of `shell', but presently the documentation says, Program used comes from variable ‘explicit-shell-file-name’, or (if that is nil) from the ESHELL environment variable, or (if that is nil) from ‘shell-file-name’. That is, the `shell' command falls back to `shell-file-name'. I assume that `shell' has always had that, or a similar, fallback. The `shell-file-name' is a direct path to an executable. This means that when "shell" is used for the language, `shell-file-name' is called and **any** startup script, such as .bash_profile or .bashrc, is called. The prompt could be set to **anything** and Emacs will never know, and can never know, what the prompt is without the user explicitly informing Emacs. Aside from the code change which allowed "shell" to work, "official" support of "shell" comes from Org manual commit 9d072ad67517875069f913315d762c9bb1e9c3ec, Sun Dec 17 11:06:05 2017 (for example, https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org?id=f7aa8c19f5170dbf09538686fb569f9b60acbd6c#n18410). This appears unconnected with the code change. The addition to the manual happened 4 years after the code name change and none of the commit messages around the time of code change suggest that "shell" was intended to work as a language. In fact, I found this email from Eric Schulte (creator of Babel and maintainer at the time of the code change) which suggests that "shell" is in fact not supported or intented as a language (https://lists.gnu.org/archive/html/emacs-orgmode/2013-12/msg00294.html): In response to the statement, "a coworker used "#+BEGIN_SRC shell" where he should have written "#+BEGIN_SRC sh" Eric says, "[The suggested work around] would protect against this particular error" #+end_longwinded_explanation Regardless of whether "shell" was intended to work as a Babel language, the fact remains that it does work and that it's been advertised in the manual (at least) for 6 years. What are the pros and cons of "shell"? What benefit does "shell" provide? - The "shell" language allows an arbitrary
Re: [PATCH] Async evaluation in ob-shell
On Thu, 23 Mar 2023 07:48:44 -0400 Ihor Radchenko wrote --- > May you also document this new feature in ORG-NEWS and in > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html ? Done.
Re: [PATCH] Async evaluation in ob-shell
On Fri, 17 Feb 2023 05:44:20 -0500 Ihor Radchenko wrote --- > What I mean is... I follow you now. Thank you. I've attached a patch (with commit message) for adding async to ob-shell. If it looks good, I can apply it to main. 0001-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Mon, 20 Feb 2023 06:24:52 -0500 Ihor Radchenko wrote --- > Why not simply doing the `should' test when the > `test-ob-shell/uuid-regex' match fails? Instead of returning `t'. Then, > we will not need to use advise. Great point. I had originally used advice to avoid a loop. However, when it became apparent that the loop was necessary, I overlooked that the advice was no longer needed. I've rewritten the test and updated the patch. 0001-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Fri, 03 Mar 2023 09:52:09 -0500 Ihor Radchenko wrote --- > I tried the patch, and I am getting failed tests: > > 1 unexpected results: >FAILED test-ob-shell/session-async-evaluation ((should (string= ": 1\n: > 2\n" (buffer-substring-no-properties (point) (point-max :form (string= > ": 1\n: 2\n" ": 1\n: 2\n: org_babel_sh_prompt>\n") :value nil :explanation > (arrays-of-different-length 8 31 ": 1\n: 2\n" ": 1\n: 2\n: > org_babel_sh_prompt>\n" first-mismatch-at 8)) Sorry for missing that. The issue is that when I replaced `org-babel-sh-prompt' with `comint-prompt-regexp', the regexp no longer matches the output and grabs the next prompt. It looks like the reason is `comint-prompt-regexp' is set to "^org_babel_sh_prompt> *" (with two spaces between the '>' and '*'). Attached is a revised patch which removes one of the spaces by changing how `org-babel-sh-initiate-session' sets the `comint-prompt-regexp'. Another place this could be done is in the defvar for `org-babel-sh-prompt' itself (which ends with a space). However, I think it's customary to leave a trailing space for prompts? > > + (let ((uuid (org-id-uuid))) > > Do you need to use `org-id-uuid' here? ob-python directly uses `md5'. > If you still prefer org-id-uuid, we probably need to move it to > org-macs.el I just need a random string. `md5' would work for that. However, might it be better to update ob-R and ob-python to use `org-id-uuid'? Both of those manually declare the randomness. It's conceivable that someone may delete or mistype the number (1), resulting in a lower entropy. An md5 is also not a uuid, strictly speaking. Of course, the chance of collision is still basically zero and the cost of collision about the same. Using `org-id-uuid' would only provide a consistent way to do things. 0003-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Sun, 05 Mar 2023 07:14:21 -0500 Ihor Radchenko wrote --- > > Matt m...@excalamus.com> writes: > > > > Sorry for missing that. The issue is that when I replaced > > `org-babel-sh-prompt' with `comint-prompt-regexp', the regexp no > > longer matches the output and grabs the next prompt. It looks like > > the reason is `comint-prompt-regexp' is set to "^org_babel_sh_prompt> > > *" (with two spaces between the '>' and '*'). Attached is a revised > > patch which removes one of the spaces by changing how > > `org-babel-sh-initiate-session' sets the `comint-prompt-regexp'. > > Another place this could be done is in the defvar for > > `org-babel-sh-prompt' itself (which ends with a space). However, I > > think it's customary to leave a trailing space for prompts? > > The actual prompt is "org_babel_sh_prompt> ". Agreed. > And we want to skip leading spaces in addition. What do you mean by this? > Adding " *" does not make the prompt match 2 spaces, but 1+. Agreed. It's not clear to me what pattern you're looking to match. > > > > + (let ((uuid (org-id-uuid))) > > > > > > Do you need to use `org-id-uuid' here? ob-python directly uses `md5'. > > > If you still prefer org-id-uuid, we probably need to move it to > > > org-macs.el > > > > I just need a random string. `md5' would work for that. However, > > might it be better to update ob-R and ob-python to use `org-id-uuid'? > > Both of those manually declare the randomness. It's conceivable that > > someone may delete or mistype the number (1), resulting in a > > lower entropy. An md5 is also not a uuid, strictly speaking. Of > > course, the chance of collision is still basically zero and the cost > > of collision about the same. Using `org-id-uuid' would only provide a > > consistent way to do things. > > `md5' will be slightly faster compared to `org-id-uuid'. But it should > not matter. > > If we want use `org-id-uuid', lets move it to org-macs.el. Requiring the > whole org-id.el must not be done. It has side effects of defining id: > links. In the next revision (once we figure out the regex), I can create a separate commit moving `org-id-uuid' to org-macs.el and updating ob-R and ob-python from `md5' to `org-id-uuid' (assuming that's not an issue for the maintainers of those). If you think speed is a concern, however, I can switch ob-shell.el to use plain `md5'. > > > (concat "^" (regexp-quote org-babel-sh-prompt) > > -" *")) > > +"*")) > > This is wrong. It unconditionally makes the last char in > `org-babel-sh-prompt' 0+. (Imagine it is changed to non-space in > future). When you say "imagine it is changed to non-space...", do you refer to `org-babel-sh-prompt'? Honestly, it's not clear to me what pattern(s) we need to match.
Re: Template for ob- packages?
On Thu, 16 Feb 2023 16:10:51 -0500 Galaxy Being wrote --- > Is there a generic starter template for writing an ob- package, some sort of > example code? Yes: https://git.sr.ht/~bzg/worg/tree/master/item/org-contrib/babel/ob-template.el When I started learned Babel, I took notes with the hope of expanding it into a Worg article. Here are the notes: https://excalamus.com/2021-11-03-org_babel.html If you're interested in fleshing out the notes into proper documentation, I'd be happy to help.
Re: [PATCH] Async evaluation in ob-shell
On Wed, 22 Feb 2023 05:29:59 -0500 Ihor Radchenko wrote --- > > +(defun ob-shell-async-chunk-callback (string) > > + "Filter applied to results before insertion. > > +See `org-babel-comint-async-chunk-callback'." > > + (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string)) > > Why not using `comint-prompt-regexp'? > > > +(ert-deftest test-ob-shell/session-async-valid-header-arg-values () > > + "Test that session runs asynchronously for certain :async values." > > + (let ((session-name > > "test-ob-shell/session-async-valid-header-arg-values") > > +(kill-buffer-query-functions nil)) > > +(cl-loop > > A simple `dolist' would do here. There is no reason to use `cl-loop'. Great points! Changed. 0002-ob-shell-Add-async-evaluation.patch Description: Binary data
Re: Template for ob- packages?
On Fri, 17 Feb 2023 17:32:18 -0500 Leo Butler wrote --- > Matt, thanks for sharing those notes. I would suggest that they be added > to worg in their current state. Thanks for your vote of confidence. I've attached the source, in case someone wants to do that. Otherwise, I'll get to it when I can. 2021-11-03-org_babel.org Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Fri, 24 Mar 2023 05:11:38 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > On Thu, 23 Mar 2023 07:48:44 -0400 Ihor Radchenko wrote --- > > > May you also document this new feature in ORG-NEWS and in > > > https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html ? > > > > Done. > > A small note on the WORG page: it may be more natural to use :async yes > rather than :async t. Both are viable - in fact, anything other than > :async no and :async none will be treated as "t". Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated ORG-NEWS with 6c9104f59.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Fri, 07 Apr 2023 11:29:59 -0400 Matt wrote --- > > On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > > > See the attached tentative patch. > > After applying the patch, I get the following error when trying to load > Emacs: > > Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's > value as variable is void sh-ancestor-alist > > I wasn't able to resolve it. I suspect the issue is on my end, such as a > mixed install or the need to re-byte-compile `sh-script.el'. I was able to resolve it by simply requiring `sh-script' in `org-src'. Committed and pushed.
Re: [PATCH] Async evaluation in ob-shell
On Mon, 17 Apr 2023 14:53:18 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > > A small note on the WORG page: it may be more natural to use :async yes > > > rather than :async t. Both are viable - in fact, anything other than > > > :async no and :async none will be treated as "t". > > > > Updated WORG with commits 9d153ea4, 754c72cb, and 18333299. Updated > > ORG-NEWS with 6c9104f59. > > Thanks, but I am not seeing 6c9104f59. I'm sorry, I was unclear about which repo these commits were in. ORG-NEWS was updated here: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6c9104f59ca8085abe477a81857548461bf88f23
Re: Interest in an Org video meetup?
On Wed, 26 Apr 2023 17:13:47 -0400 Corwin Brust wrote --- > On Wed, Apr 26, 2023 at 9:31 AM Matt m...@excalamus.com> wrote: > > > > FSF associate members have access to a FSF hosted Jitsi instance. I'm an > > FSF associate member and could host it using that if I'm able to make the > > time. > > > > We can make EmacsConf instance of BigBlueButton (BBB) available, as > well. FWIW I had more success with BBB than Jitsi for larger > meetings. That's generally been my experience, too. Glad to hear that's an available option. Hopefully we can bring this to fruition.
Re: Interest in an Org video meetup?
> > On Oct 6, 2022, at 9:20 AM, Russell Adams rlad...@adamsinfoserv.com> wrote: > > Would there be any interest in a monthly 1-2 hour long ad-hoc screen > > sharing and video discussion for Org-mode? > > > > I'm offering to schedule and moderate the first few events. I'd > > propose a Saturday meeting in the afternoon European time to cover EU > > and NA. > > > > I'm considering using Jitsi, or maybe GNU Jami. > > > > Topics could include Q, demonstrations of features, interactive > > troubleshooting, etc. I hadn't considered presentations, but > > that could be an option too. > > > > Comments? I'm interested. I've thought of proposing something similar, yet hesitated to because topics and decisions made in a video chat wouldn't be captured on list. To me, that record is important. FSF associate members have access to a FSF hosted Jitsi instance. I'm an FSF associate member and could host it using that if I'm able to make the time.
[BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
Of the shells given in `org-babel-shell-names' (that is, "sh" "bash" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh" "posh"), only "sh" and "bash" have font locking in source blocks. For example, #+begin_src sh if [ -z $TEST ]; then echo Fontified fi #+end_src #+begin_src bash if [ -z $TEST ]; then echo Fontified fi #+end_src #+begin_src zsh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src fish if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src csh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src ash if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src dash if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src ksh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src mksh if [ -z $TEST ]; then echo No fontification fi #+end_src #+begin_src posh if [ -z $TEST ]; then echo No fontification fi #+end_src Does anyone know which function is responsible for re-fontifing source blocks?
Re: Font lock for org-babel shell scripts?
On Tue, 28 Mar 2023 13:24:39 -0400 Derek Chen-Becker wrote --- > I'm trying to figure out whether there's a bug or just a misconfiguration on > my end with font lock for org-babel shell source blocks. If I run emacs 28.2 > (with -q) and open the following org file, I can evaluate both source blocks > but only the "bash" block has syntax highlighting. I've confirmed that > opening a zsh script (e.g. with a zsh shebang) highlights correctly. If this > list isn't the right place to ask about this issue, does someone know where > I could look for help? This is the right place to ask. And thanks for asking! I'm able to reproduce it. I'm not familiar with how Org handles the font lock. However, I notice that calling C-c ' (`org-edit-special') results in the error: No such language mode: zsh-mode I suspect that without a language mode, Org has no way to know how to color the block. As a quick work around, you can define zsh-mode as shell-script-mode to get coloring: (defalias 'zsh-mode 'shell-script-mode)
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Thu, 30 Mar 2023 04:55:32 -0400 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > I think this approach will work fine. I tried examples for each shell > > type and keywords like if/then/else and function names are highlighted. > > Even for posh (powershell)? Yes. It's not great since sh-mode looks for Korn-based keywords. It does string highlighting and common keywords like 'if', 'exit', and 'param'.
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Tue, 04 Apr 2023 08:30:34 -0400 Ihor Radchenko wrote --- > See the attached tentative patch. After applying the patch, I get the following error when trying to load Emacs: Warning (comp): /home/ahab/Projects/org-mode/lisp/org.el: Error: Symbol's value as variable is void sh-ancestor-alist I wasn't able to resolve it. I suspect the issue is on my end, such as a mixed install or the need to re-byte-compile `sh-script.el'. To run Org from source I do one of the following: ;; When using my init (use-package org :straight (:local-repo "/home/ahab/Projects/org-mode")) ;; When running emacs -q (add-to-list 'load-path "/home/ahab/Projects/org-mode/lisp") (require 'org-loaddefs) If I need to recompile Emacs byte code, I'm not sure how I'd do that since I'm running Guix and those files live in the write protected /gnu/store.
Re: [PATCH] Async evaluation in ob-shell
On Fri, 24 Mar 2023 05:13:34 -0400 Ihor Radchenko wrote ---
> A small note on the WORG page: it may be more natural to use :async yes
> rather than :async t. Both are viable - in fact, anything other than
> :async no and :async none will be treated as "t".
Ah, okay. I'll make that more clear.
Somewhat related, I had this left over from when I was working out the async
code. It prevents the user from running :async without the (required) :session
header.The :async header runs the default (blocking) process when :session
is missing.
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 86c2bf7a7..384bfcda8 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -206,11 +206,12 @@ comint outputs due to buffering.")
PARAMS are the header arguments as passed to
`org-babel-execute:lang'."
(let ((async (assq :async params))
-(session (assq :session params)))
+(sessionp (not (member (cdr (assq :session params)) '("no" "none")
+(if (and async (not sessionp))
+ (error (org-babel-eval-error-notify 1 "ERROR: must use 'async' with
'session'")))
(and async
-(not org-babel-exp-reference-buffer)
- (not (equal (cdr async) "no"))
- (not (equal (cdr session) "none")
+ sessionp
+ (not org-babel-exp-reference-buffer
(defun org-babel-comint-async-filter (string)
"Captures Babel async output from comint buffer back to Org mode buffers.
It's really just a nicety. The user can cancel the accidental blocking process
with C-g. However, the block is run in a different shell than expected and
it's jarring to have Emacs freeze when you expect async.
Thoughts on including it or something similar?
error-on-async-session-mismatch.diff
Description: Binary data
Re: Remove "shell" as a supported Babel language within ob-shell.el (was Re: [SUGGESTION] ob-shell async result output should not contains shell prompt)
On Fri, 24 Mar 2023 07:38:58 -0400 Ihor Radchenko wrote --- > I suggest the following: > 1. Introduce a new customization `org-babel-default-shell', defaulting >to (or (executable-find "sh") (executable-find "cmd.exe")). > 2. Use the value as default shell in "shell" code blocks. > 3. Document and announce the change. > 4. Create org-lint checker that will mark "shell" code blocks as not >desired. > > The above steps will ensure minimal breakage for existing uses of > "shell" blocks. Only users who wrote shell blocks for non-standard shell > will have to adapt. These are good suggestions. Thank you!
Re: [BUG] No font lock in src blocks for shells in org-babel-shell-names (was Re: Font lock for org-babel shell scripts?)
On Wed, 29 Mar 2023 13:04:31 -0400 Derek Chen-Becker wrote --- > Cool, I would be happy to submit a patch! Sure, if that's something you'd enjoy. I'm happy to assist, if needed. > On Wed, Mar 29, 2023 at 3:33 AM Ihor Radchenko yanta...@posteo.net> wrote: > > Org provides some defaults for bash in `org-src-lang-modes' but not for > other shell flavours. I guess we can add more defaults to > `org-src-lang-modes', if we know for sure that e.g. zsh can be fontified > sh-mode. I think this approach will work fine. I tried examples for each shell type and keywords like if/then/else and function names are highlighted.
Re: [PATCH] Async evaluation in ob-shell
> Matt m...@excalamus.com> writes: > > I see only two options to fix it: remove a space from the concat expression > (which I did in my latest patch) or remove a space from > `org-babel-sh-prompt'. Unfortunately, I was mistaken and the second option (removing the space from `org-babel-sh-prompt') doesn't fix the issue. The TLDR is that the code in `org-babel-comint-async-filter' which grabs the region between the indicators (incorrectly) fails to include the prompt's trailing space. #+begin_longwinded_explanation I'll first explain why removing the space from `org-babel-sh-prompt' doesn't fix the issue because it well also highlight the underlying problem. If we remove the space from the `org-babel-sh-prompt', then `comint-prompt-regexp' becomes "^org_babel_sh_prompt> *" (with one space). This would work if the string passed to the `ob-shell-async-chunk-callback' stayed the same. It doesn't (this is where my reasoning and testing failed). Changing the `org-babel-sh-prompt' to "org_babel_sh_prompt>" (without a space) causes the following string to be passed to the callback: "org_babel_sh_prompt>1 org_babel_sh_prompt>2 org_babel_sh_prompt" Note that the final prompt doesn't have a ">" and therefore the `comint-prompt-regexp' (which becomes "^org_babel_sh_prompt> * (with one space)) used in the callback fails to match it. When we remove the space from the `org-babel-sh-prompt', the session buffer looks like this: "sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt>";PS2= org_babel_sh_prompt>echo 'ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8' echo 1 echo 2 echo 'ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8' ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8 org_babel_sh_prompt>1 org_babel_sh_prompt>2 org_babel_sh_prompt>ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8 org_babel_sh_prompt>" The `org-babel-comint-async-filter' is what calls the `ob-shell-async-chunk-callback' (ob-comint.el:284). It monitors for the end indicator. When that appears, it passes the region between the beginning of the end indicator **less 1** and the character after the end of the start indicator to the callback. For a clean run of `test-ob-shell/session-async-evaluation', the beginning of the end indicator is at 361 and the character after the end of the start indicator is at 298. This is the string I gave above which is missing the ">". In order to make the second option work, we'd need to change the "less 1" part of `org-babel-comint-async-filter' from (- (match-beginning 0) 1) to (match-beginning 0). It turns out that's actually all we need to do. When `org-babel-sh-prompt' is "org_babel_sh_prompt> " (with one space), then the session buffer looks like: "sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2= org_babel_sh_prompt> echo 'ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26' echo 1 echo 2 echo 'ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26' ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26 org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt> ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26 org_babel_sh_prompt> " The region passed to the callback is then defined as 366 to 300, or "org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt>" (<-- no space) This looks okay at first glance. However, **the last line is not a valid prompt**. A prompt must end in a space! When the `org-babel-sh-prompt' is set to "org_babel_sh_prompt> " (with one space), the `comint-prompt-regexp' is "^org_babel_sh_prompt> *" (with two spaces). This means that the `comint-prompt-regexp' matches on a trailing space which the **region passed to the callback doesn't have**. Therefore, the match fails. Instead, if we modify the `org-babel-comint-async-filter' like modified lisp/ob-comint.el @@ -273,7 +273,7 @@ STRING contains the output originally inserted into the comint buffer." (res-str-raw (buffer-substring ;; move point to beginning of indicator - (- (match-beginning 0) 1) + (match-beginning 0) ;; find the matching start indicator (cl-loop do (re-search-backward indicator) then the region passed to the callback will be from 367 to 300, or "org_babel_sh_prompt> 1 org_babel_sh_prompt> 2 org_babel_sh_prompt> " (<-- with one space) The `comint-prompt-regexp' will now match the last prompt in the region. With this change, the `org-babel-sh-prompt' keeps the trailing space (like
[PATCH] Async evaluation in ob-shell
I'm excited to share that I've got async evaluation working (crudely) with ob-shell. A rough implementation is attached. It has clear issues, such as the prompt being present in the output: #+begin_src sh :session tester :async t echo "By sending delimiters separately..." sleep 3 slep 1 echo "typos don't cause problems--^" #+end_src #+RESULTS: : org_babel_sh_prompt> By sending delimiters separately... : org_babel_sh_prompt> org_babel_sh_prompt> sh: slep: command not found : org_babel_sh_prompt> typos don't cause problems--^ : org_babel_sh_prompt> It's not clear to me if that's something that a better regexp would handle or if it should be cleaned up in the callback function. I'm still figuring out how it's done in ob-python and ob-R. Any feedback or advice is welcome. ob-shell-async-separate-calls.patch Description: Binary data
Re: [PATCH] Async evaluation in ob-shell
On Sat, 11 Feb 2023 15:56:00 -0500wrote ---
> org-babel-comint-async-filter is capable of taking a similar approach,
> and reading/writing to tempfile.
There is some precedence in ob-shell for this. Currently, the cmdline, stdin,
and shebang headers use temp files. It may be that implementing async versions
of these could use this part of the async API. However, cmdline, stdin, and
shebang each use a temporary shell process rather than a dedicated comint, even
when the :session header is present. The async implementation requires a
comint buffer.
> I believe this approach would be
> generally more robust, but the major weakness is that it would break if
> you SSH'd to a remote host in the middle of the session.
Good point.
> There was an interesting patch to ob-shell that was never applied, that
> took the approach of wrapping the code block in a bash function before
> executing; I think it might be a promising approach:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2020-02/msg00923.html
This is interesting. Thanks for sharing! Instead of writing to a temp file,
it could use a here document. Here documents, AFAIK, are POSIX portable,
although function definitions aren't. Of course, if we're considering Windows
cmd (as we did in a recent bug report), there are all sorts of problems: batch
syntax (executed from a file) is different than CLI syntax, functions will
implemented using labels (I'm not sure how robust that is), and no here
documents. A comint would probably be best for the Windows use-case.
ahab@pequod ~$ guix shell dash
ahab@pequod ~ [env]$ dash <<- EOF
my_function() { echo "hello, world!"; echo "the end"; }
my_function;
EOF
hello, world!
the end
ahab@pequod ~$ guix shell fish
ahab@pequod ~ [env]$ fish <<- EOF
> function my_function
> echo "hello, world!"
> echo "the end"
> end
> my_function
> EOF
hello, world!
the end
> By the way, I took a look at ob-shell for the first time in awhile, and
> noticed that ob-shell now forces the prompt to be like:
>
> org_babel_sh_prompt>
>
> Which I think makes cleaning up the prompt markers a lot more
> robust. But it is a bit ugly, and also seems to break working with shell
> sessions started outside of Babel with M-x shell.
Is this something you consider a bug?
Re: [PATCH] Async evaluation in ob-shell
On Sat, 11 Feb 2023 06:44:56 -0500 Ihor Radchenko wrote ---
> 1. You should provide all the docstrings.
> 2. I generally feel that separate async and separate session code are
>either duplicating the same code or edge cases considered by session
>code may popup in async code unexpectedly.
Excellent points. Thank you.
As part of the commit, I want to include tests.
How to test an async block is non-obvious. The initial evaluation returns a
uuid which returns immediately and can be checked using a regex:
(defconst test-ob-shell/uuid-regex
"[0-9a-fA-F]\\{8\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{4\\}\\b-[0-9a-fA-F]\\{12\\}")
Technically, this is a ob-comint aspect and may be more rightly included in
(the currently non-existant) tests for that module.
Checking the final result from the callback is trickier. The following works,
but requires advice (which could potentially persist beyond the test) and a
delay (which slows testing overall and whose duration likely depends on the
hardware the test runs on). Without the delay, I could not get the callback to
execute within the test. It would execute when running manually in an Org
buffer, however. I'm not sure why.
(ert-deftest test-ob-shell/session-async-evaluation ()
(let ((session-name "test-ob-shell/session-async-evaluation")
(kill-buffer-query-functions nil)
result)
;; perform check after the callback executes which looks for the
;; expected result
(advice-add
'ob-shell-async-chunk-callback
:filter-return
(lambda ( r)
(let ((result (car r)))
(if (not (string= result "1\n2\n"))
(ert-fail (format "Expected 1\n2\n: %s" result)))
result))
`((name . ,session-name)))
;; always remove the advice, regardless of test outcome
(unwind-protect
(org-test-with-temp-text
(concat "#+begin_src sh :session " session-name " :async t
echo 1
echo 2
#+end_src")
;; execute the block; delay momentarily so that the callback
;; executes
(setq result (org-trim (org-babel-execute-src-block)))
(if (should
(and
;; if the block runs...
(string-match
test-ob-shell/uuid-regex
result)
;; ...and the callback executes without fail
(not (sleep-for 0.1
;; clean up buffer on success
(kill-buffer session-name)))
(advice-remove 'ob-shell-async-chunk-callback session-name
This works for me using the last patch.
Thoughts?
Re: [PATCH] Async evaluation in ob-shell
On Thu, 09 Feb 2023 06:23:42 -0500 Ihor Radchenko wrote --- > Could you please elaborate? What was the problem with > `buffer-substring'? I am asking because one of the previous versions of > `org-babel-comint-wait-for-output' relied upon 'face text property. See > a35d16368. The problem is I got mixed up about the printed string representation and thought having properties changed how functions operated on the string. The 02-ob-shell-async-non-file.patch works fine with `buffer-substring'. No need for the other patch. It seems to me like 02-ob-shell-async-non-file.patch is all that's needed for basic async in ob-shell. I'm able to run long processes like `guix pull' and `guix package -u' calls without issue and the results look like I expect. Similarly for running a web server, such as `python3 -m http.server' and killing it. Unless there's something you or others think needs to be done, I can get it committed (and try to write a test or two for it).
Re: [PATCH] Async evaluation in ob-shell
I've attached two patches which replace the previous. I found cleaning the output was dramatically helped by calling `buffer-substring-no-properties' instead of `buffer-substring' in `org-babel-comint-async-filter'. I'm not sure why `buffer-substring' was originally used. `make test' shows no failures, so I assume it doesn't make a difference...? On Tue, 07 Feb 2023 06:40:51 -0500 Ihor Radchenko wrote --- > That's likely because of the same reasons why prompt did not get cleaned > in synchronous blocks earlier. See `org-babel-comint-with-output'. That, my friend, was a wild ride. I'm curious about people's feelings on `org-babel-comint-with-output'. Personally, I get the heebie-jeebies. I can't shake feeling that there's a better way, especially since `org-babel-comint-async-filter' achieves similar ends. My hunch is that other Babel languages may want async and that now would be a good time to consolidate the common functionalities of `org-babel-comint-with-output' and `org-babel-comint-async-filter' . Maybe even unify the API. So far, `org-babel-comint-with-output' touches 9 languages and `org-babel-comint-async-filter' appears to touch 2 (soon to be 3). I suspect those numbers will only grow. I also can't shake the feeling that I might become ob-comint maintainer at some point...(not yet!). I'm curious what people's thoughts are. 01-ob-shell-remove-properties-from-callback-string.patch Description: Binary data 02-ob-shell-async-non-file.patch Description: Binary data
[BUG] shell sessions started outside of Babel broken
On Sat, 11 Feb 2023 15:56:00 -0500 Jack Kamm wrote --- > By the way, I took a look at ob-shell for the first time in awhile, and > noticed that ob-shell now forces the prompt to be like: > > org_babel_sh_prompt> > > Which I think makes cleaning up the prompt markers a lot more > robust. But it is a bit ugly, and also seems to break working with shell > sessions started outside of Babel with M-x shell. Can you please elaborate? The following does what I expect it to: 1. M-x shell 2. Call `org-ctrl-c-ctrl-c' on the following block in a separate Org buffer: #+begin_src sh :session *shell* echo "hello, world!" #+end_src #+RESULTS: | hello | world! | Are you doing something different?
[BUG] conda doesn't work in ob-shell sessions
On Sun, 12 Feb 2023 22:16:16 -0500 Jack Kamm wrote --- > But I also noticed another prompt-related issue: conda doesn't seem to > work in ob-shell sessions anymore. That is a bigger problem for my use > case. Could you elaborate? It looks like conda has a new init (from what I remember) which requires users to run a `conda init ' command in order to create a new environment. For example, I needed to run `conda init bash.' This added garbage to my .bashrc which automatically enabled the base environment. I was able to do this from ob-shell, however it required me to reload the shell (I reloaded Emacs to be sure I got the latest environment variables). After this, I was able to create and activate a new conda environment. 1. M-x shell 2. Executing the following in sequence: #+begin_src emacs-lisp (org-version nil t) #+end_src #+RESULTS: : Org mode version 9.6.1 (release_9.6.1-250-ge6353d @ /home/user/Projects/org-mode/lisp/) #+begin_src emacs-lisp (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) #+end_src #+begin_src sh :session *shell* :results output conda create --yes --name myenv python=3.9 #+end_src #+RESULTS: #+begin_example Downloading and Extracting Packages Preparing transaction: done Verifying transaction: done Executing transaction: done To activate this environment, use conda activate myenv To deactivate an active environment, use conda deactivate #+end_example #+begin_src sh :session *shell* :results output conda activate myenv #+end_src #+RESULTS: #+begin_src sh :session *shell* :results output which python #+end_src #+RESULTS: : /home/user/miniconda3/envs/myenv/bin/python The *shell* buffer's prompt also showed the expected [myenv] prefix.
Re: [PATCH] Async evaluation in ob-shell
On Wed, 15 Feb 2023 10:08:47 -0500 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > > Checking the final result from the callback is trickier. The following > > works, but requires advice (which could potentially persist beyond the > > test) and a delay (which slows testing overall and whose duration likely > > depends on the hardware the test runs on). Without the delay, I could not > > get the callback to execute within the test. It would execute when > > running manually in an Org buffer, however. I'm not sure why. > > Doesn't (while ... (sleep-for ...)) work? I'm afraid I don't follow what you mean. My biggest concern is the sleep-for itself. Aside from the concerns above, the test doesn't work without it. For example, the check made by the advice looks for "1\n2\n". I've changed the source block from "echo 1" to "echo 2" to induce a failure (it will now receive "2\n2\n"). I re-evaluate the test and call M-: (ert "test-ob-shell/session-async-evaluation"). The test passes! I can put a message or debug in the advice and see that it's never called. However, if I uncomment the sleep-for, the advice runs, and the test fails as expected. (ert-deftest test-ob-shell/session-async-evaluation () (let ((session-name "test-ob-shell/session-async-evaluation") (kill-buffer-query-functions nil) result) ;; check callback return for correct results (advice-add 'ob-shell-async-chunk-callback :filter-return (lambda ( r) (let ((result (car r))) (should (string= result "1\n2\n")) ; this was previously an if statement result)) `((name . ,session-name))) ;; always remove advice, regardless of test outcome (unwind-protect (org-test-with-temp-text (concat "#+begin_src sh :session " session-name " :async t echo 2 echo 2 #+end_src") ;; execute block; callback only executes when delay (setq result (org-trim (org-babel-execute-src-block))) (if (should (and (string-match test-ob-shell/uuid-regex result) ; block runs (not (sleep-for 0.1)) ; callback doesn't fail )) (kill-buffer session-name))) (advice-remove 'ob-shell-async-chunk-callback session-name
Re: ob-shell sessions will send final echo '...' as input key sequence when previous command reads input interactively (was: [Need Help] Error to evaluate "mpv" command in inline src block)
On Tue, 02 May 2023 00:17:51 -0400 Christopher M. Miles wrote --- > Indeed, this option solved the problem real neat. > I'm surprised by your knowledge and digging problem skill. > I have to say a big TANKS to you. THANKS, THANKS, THANKS. Hahaha My thanks as well to everyone who helped. I'm tied up with personal matters currently and am only seeing this thread now. On Mon, 01 May 2023 07:36:23 -0400 Ihor Radchenko wrote --- > The only way I know how to work around this is a giant one-liner like > echo ''; ; echo '...' > > However, (1) ";" may not work in some shells; (2) may > contain multiple lines, leading to the same issue. > > Matt, maybe you have some ideas about this edge case? I have no other ideas within the current ob-shell implementation. As for modifications, I have the following thoughts. First, how might we state the problem for this edge case? To me, it looks like, "How can the ob-shell :async option manage interactive input?" Do you agree with this formulation? If not, how do you see it differently? One thought is to update :async to work with the :stdin option so that the block is run as a script. Currently, :stdin runs synchronously in a separate shell. We might be able to grab the script's output and put it into the session buffer. See how the following runs in a temporary shell, regardless of the :session/:async options. #+name: answers Matt yes #+begin_src sh :stdin answers :results output :session *test* :async echo -n "What's your name?" read -r name echo echo "Hello, $name!" echo -n "Would you like to continue?" read -r continue echo if [ $continue == 'yes' ]; then echo "Continuing..." sleep 3 echo "Process complete" else echo "Aborted" fi #+end_src #+RESULTS: : What's your name? : Hello, Matt! : Would you like to continue? : Continuing... : Process complete
Re: org babel output
On Sat, 01 Jul 2023 05:52:42 +0200 Robert Synofzik wrote ---
> Hello,
> the exact same error occurs for me:
> https://emacs.stackexchange.com/questions/72243/macos-org-babel-python-with-session-output-give-python-el-eval
>
> do you possibly have any idea on how to fix this?
Just to check, because you didn't explicitly say it in your message, does the
solution given in the StackExchange not work for you? That is, do you get a
different result when you name the session? What happens when you run the
following?
#+begin_src python :session my-named-session :results output
print("yes")
#+end_src
Reintroduction and Questions
Hello again! I'm slowly settling in after moving to Germany and hoping to get back into maintenance. As the FSF disclaimer is still being worked out with my employer, I'm hesitant to begin non-trivial code contributions. Until then, I'm hoping to do things that don't require attribution, such as organizing tasks, issues, and updating Worg. 1. What do the FR, FP, RFC, and RFE Woof! prefixes stand for? These are listed in https://tracker.orgmode.org/howto for the "request" type under the "Adding a report" section. I assume FR is "Feature Request" and RFC is "Request for Comment". 2. Is anyone in the Hamburg, Deutschland area interested in meeting in person? They say it's hard to make friends here and I'm taking applications :) I think it'd be cool to talk Emacs and work on contributions together. Maybe we could coordinate this with an online Meetup (https://lists.gnu.org/archive/html/emacs-orgmode/2023-06/msg00196.html).
Export Documentation Confusions
In the Emacs Berlin meeting today, we noticed several items that might be improved in the manual. - Section 13.18. 1: Bare HTML may be better placed under Section 13.9 HTML Export. The information in this section appears unrelated to Exporting in non-Org buffers. In fact, the settings given for a minimal HTML export do not apply when converting a region. For example, when converting a region containing an Org table to HTML, a preamble is not inserted even when the variable `org-html-preamble` is t. https://orgmode.org/manual/Bare-HTML.html Proposed solution: Move Section 13.18.1 nearer the sections on HTML export. - Section 13.18: Export in Foreign Buffers The section uses the term "foreign buffer" without providing a definition. It appears to mean a buffer without "org-mode" enabled. The section describes functions which allow users to write Org mode syntax in non-Org buffers and use various functions to convert the Org syntax to other syntaxes, like HTML, Markdown, etc. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Change wording from "foreign buffers" to "Convert Region" and "non-Org" buffer. The functionality is really about conversion rather than anything to do with non-Org buffers. If fact, Org respects users' freedom to run the program as they wish–it does not prevent them from converting Org formatted text within an Org buffer to a non-Org format, no matter how absurd it may appear :) - Section 13.18: Export in Foreign Buffers There was confusion caused by the function names. Specifically, the "namespace" (such as "org-ascii-" or "org-texinfo-") is confusing to users who are not familiar with Org function naming conventions. For example, one participant thought `org-ascii-convert-region-to-utf8` might convert non-utf8 ascii to utf8 and was confused why `org-texinfo-convert-region-to-texinfo` would convert texinfo to texinfo. Of course, this is not a correct understanding of the function behavior. https://orgmode.org/manual/Export-in-Foreign-Buffers.html Proposed solution: Clarify the overall purpose of the functions. Another solution could be to make an aside and clarify what is meant by "Org export back end". For example, change "The export back-ends in Org often include..." to "The export back-ends in Org (for example, Org ASCII or Org HTML) often include..." However, my hope is that the changes I made with regard to "foreign buffers" also clarifies the purpose and use of functions listed in this section. These changes are given in the attached patches. If there are no suggested improvements, I could merge them in. Please let me know your thoughts. move-bare-html-section-to-html-export-section.patch Description: Binary data change-foreign-to-non-org.patch Description: Binary data
Re: Export Documentation Confusions
On Thu, 29 Jun 2023 12:24:12 +0200 Ihor Radchenko wrote --- > > - Section 13.18: Export in Foreign Buffers > > > > The section uses the term "foreign buffer" without providing a definition. > > > > Proposed solution: Change wording from "foreign buffers" to "Convert > > Region" and "non-Org" buffer. > > "Convert region" may be still confusing. Maybe something like > "Convert Org region to other markup syntax"? How about renaming the section to "Export Region"? I was getting ahead of myself. The confusion was about "foreign buffers" and not the meaning of "export". > > - Section 13.18: Export in Foreign Buffers > > > > There was confusion caused by the function names. > > We can also create more intuitive function aliases. How about `org-export-region-to-...`? This would correspond nicely with options given by `org-export-dispatch`.
Re: Export Documentation Confusions
Changes have been pushed
Re: Reintroduction and Questions
On Sun, 25 Jun 2023 18:07:34 +0200 Ihor Radchenko wrote --- > Be warned though that Woof! is not yet stable (see > https://git.sr.ht/~bzg/woof and https://lists.sr.ht/~bzg/woof). Thanks for the warning. To verify my understanding, the mailing list is the "source of truth" for issues and Woof! "scrapes" that? > You may also try to ask on Emacs Berlin mailing list. > https://emacs-berlin.org/ Good suggestion!
Re: Interest in an Org video meetup?
Thoughts on dates and times?
Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build
If I followed correctly, the topic switched to discussing async generally within Babel. I've started a new thread accordingly. On Tue, 06 Feb 2024 20:24:13 +0100 Bruno Barbier > FWIW, I've been trying to use asynchronous blocks for everything, not only > the source blocks that are based on the comint mode. I've been trying to figure out how to make everything async by default. I'm super interested to see what you've come up with. > I think it would be good if ob-core itself could provide an asynchronous API. Fortunately or unfortunately, depending on how you look at it, Babel already does. The challenge is that the Org Babel API has grown piecemeal over 14 years. It's been written by several authors with limited time and knowledge. The result, while powerful and useful, is a bit of a hodgepodge. A prime example is that the concepts of "persistence" and "synchronicity" are conflated. "Session" is often used to mean "asynchronous" even though the two ideas are orthogonal. Emacs provides primitives that could make non-session blocks asynchronous. It's historical accident that blocks aren't async by default. For me, the issue is that the Babel API needs some high level perspective in order to make it consistent. I see the following terms as guides. If we can separate these concepts within the API, then Babel to *feel* like an API: - "Session" means a shell environment is "persistent." Each call is executed in the same environment. State exists between calls. - "Non-session" means a shell environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. - "Synchronous" means that execution prevents the user from editing the document while results are obtained. - "Asynchronous" means that execution does not prevent the user from editing the document while results are obtained. > I've modified my Org so that it does have such an API. This is work in > progress; let me describe it. > > I've modified ob-core itself to allow asynchronicity. In the asynchrosous > case, instead of calling: > > (org-babel-execute:LANG body params) > > I'm calling: > > (org-babel-schedule:LANG body params handle-result) > > where `org-babel-schedule:LANG' is in charge of calling `handle-result' with > the result (or the error) when it is known; `handle-result' takes care to > call `org-babel-insert-result' at the correct place (and > `org-babel-insert-result' is only called with a real result). > > While the execution is pending, I'm using the same technique that Org is > using when a source block is being edited: the result is left untouched, but > below an overlay. The overlay is used to know where to insert the result > and to display the status/progress of the execution. If the file is closed > and the execution fails, nothing is lost, the old result is still available. The use of the overlay is a really cool idea! I hesitate to say that's a good way to convey success or failure. If a process failed, I want to see the output which tells me why so that I can correct it. Or, I might actually want the failure output. Maybe I want to literally demonstrate what a code failure looks like. Maybe I want to use that output in another block. For example, shell blocks have multiple output types. A shell process may return standard output/error or a failure code. The result of the failure may trigger something else. However, using an overlay to communicate "the process is still running" could be good. We'd need to be careful about accessibility, though, and make sure the overlay is apparent, visually and otherwise. > If that technique looks safe enough and interesting, I can prepare a set of > patches so that we can discuss it further and, maybe, add it in Org. Please do! I'm super interested. I've put a lot of thought into how we might make Babel async by default. I'm excited to see your interest in the topic and look forward to seeing what you've come up with. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b
On Mon, 19 Feb 2024 01:31:42 +0100 Jack Kamm wrote --- > I don't remember the details, but my past self [1] thought it would be > good to find a way to replace `process-file' with `make-process' in > `org-babel--shell-command-on-region' or `org-babel-eval', and it seems > you are thinking along those lines with `my-org-babel-eval-async'. Hope > you're able to make progress on this and get the improvements into > ob-eval.el eventually. > > [1] https://list.orgmode.org/871rczg7bi@gmail.com/#t AFAIK, aside from appending "&", =make-process= is the only way. It would probably make sense to continue using =shell= though. If I understand correctly, you (and others) jump between the Org buffer block and the comint buffer? On Mon, 19 Feb 2024 10:03:36 +0100 Ihor Radchenko wrote --- > Note that we already have a WIP an asynchronous API in the works. > Check out `org-async-call' in > https://code.tecosaur.net/tec/org-mode/src/branch/dev/lisp/org-macs.el#L377 > If possible, we should reuse it. :O - What's the status? - Anything I could help with? - Are there any notes or supplements that go along with it? > I have several general questions: > > - what if append/prepend to result asynchronously? > - what if we replace the result, call async evaluation two times, and they > arrive in opposite order (first called is calculated after the second)? > - on failure, Org babel sometimes uses ~org-babel-eval-error-notify~. How > will it interact with asynchronous evaluation? What if we have multiple > simultaneously arriving error notifications? > Example: try running > #+begin_src bash > idontexist > #+end_src Are these open questions for the `org-async-call' implementation? -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/b
another way persistence may be achieved? Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. All that is mainly academic. The idea I think most interesting is using a single primitive to handle all evaluation. It idea reminded me of exploration code I wrote a while back which uses =make-process= to run all code blocks asynchronously (attached). It works as follows. I defined a new Babel "language" called "blub". Blub could be a shell, python, ruby, whatever. I wanted to test that the implementation could work with different interpreters or compilers. Note that "blub" doesn't have any relationship to Paul Graham's blub; I just needed a name for a generic language that could be swapped out. Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. Curious to hear other people's thoughts! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode ob-blub-test.org Description: Binary data ob-blub.el Description: Binary data
Re: [BUG] Prompt appears in async shell results
On Mon, 05 Feb 2024 15:00:18 +0100 Ihor Radchenko wrote --- > Yes. The right fix would be extracting the filter from > `org-babel-comint-with-output' and re-using it. This message documents my progress so that I may switch focus to Bruno's patches in "Asynchronous blocks for everything" attention (https://list.orgmode.org/65cfa0d8.050a0220.cb569.c...@mx.google.com/T/#u). Attached is a failed attempt at extracting the filter in =org-babel-comint-with-output=. It tries to extract the filter more-or-less directly: 1. take the filter code from =org-babel-comint-with-output= and put it into a separate function, =org-babel-comint-process-output-filter= 2. call =org-babel-comint-process-output-filter= from =org-babel-comint-with-output= and =org-babel-comint-async-filter= The unmodified =org-babel-comint-with-output= has a comment that says, "Filter out prompts". This is misleading. The filter code does two things: removes prompts *and* removes echoed input. The problem is the filter which removes echoes uses the body of the source block. It's unclear how to give =org-babel-comint-async-filter= the block body. =org-babel-comint-async-filter= is a =comint-output-filter-function= which receives a single input, "a string containing the text as originally inserted" in the comint buffer. Thoughts: - Split prompt filtering and input echoing into two filters + this seems to imply a =-hook= or =-functions= type implementation + where could input echo filter go? Where has access to the block body? - creating a generic prompt filter duplicates =ob-shell-async-chunk-callback= or, more fundamentally, =org-babel-comint-async-chunk-callback= - What would it take to consolidate output filtering? In addition to prompt filtering and input echo filtering, ob-shell filters the =org-babel-sh-eoe-indicator=. I'm sure there's other filtering that happens on block output. Wouldn't it be nice if that were in a single place, like right before results are inserted? Please feel free to provide feedback and suggestions. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode v01-refactor-filter.diff Description: Binary data
Re: [PATCH] [WORG] Maintenance: on the frequency of bugfix releases
On Sun, 10 Dec 2023 12:27:01 +0100 Ihor Radchenko wrote --- > What do you think about the attached patch? Not part of the patch, but part of the context: "you shoud *NOT*" -> "you should *NOT*" (missing the 'l') Already fixed and pushed (https://git.sr.ht/~bzg/worg/commit/a11b256086d567d0894d337b548ec13049a8731b) Regarding the patch, it seems reasonable. Potentially stupid question, asked without knowing precisely what's involved with a bugfix release (it seems like a matter of tagging and pushing?)...what's the problem with making a new release for every fix? Frequency of fixes? Time required to make the release? Storage?
Documentation Results of Evaluation (was Re: Documentation of hline symbol in source blocks results)
ng" is inaccurate > "Code /should/"? I agree, this can improve. "Should" in this context reads as "probably" and is non-committal. Documentation has a duty to be authoritative. PROBLEM: "code should execute like a function" is non-committal > "Like a function"? I agree that this is unclear. I believe what it's trying to address is that it's possible to have multiple return values from a source block. For example, Unix shell commands return an exit code indicating success, failure, and failure type. They may also write text to stdout and stderr. The header arguments ":results value" and ":results output" distinguish these two types of return values. PROBLEM: terms are not well-defined > Why is the Python requirement stated here? I see several reasons. The simplest is the piecemeal development style already mentioned. PROBLEM: non-specific requirement reference. What other languages are "like Python"? > Why "using ‘:results value’" when the paragraph should (only) > document this? Do I understand correctly that you're saying "using ':results value'" is redundant? PROBLEM: redundant words > "Result is the value"? What's the problem you see with this? PROBLEM: change of voice PROBLEM: singular form used to reference a plural (should we say "result" or "results") > What kind of value? What do you mean by "kind"? > Why are there references to Ruby and R here? At risk of being pedantic, because of the previous sentence: "Org has to use language-specific methods to obtain the value." I'm having difficulty seeing your perspective. Can you please share more of your thoughts about the confusion for this sentence? > All this confuses me and does not provide the information I > searched for (emphasis on me). I'm sorry for your confusion. I've been there. Few things make my blood boil quite as much as bad documentation. My understanding is that the information you searched for is the answer to these questions: - "How to make a named column table result for a source block?" - "How to insert a horizontal line between two rows of a table result for a source block?" Is that correct? If so, I think Ihor addressed this already by saying that, unfortunately, this functionality isn't currently available (although contributions are welcome). > I would probably prefer a clean-slate approach that starts > with something along the lines of: "Source blocks produce > results that can be integrated into an Org document and used > as input for other source blocks. This is covered in the "Feature Overview": https://www.gnu.org/software/emacs/manual/html_node/org/Features-Overview.html That's not to say that the manual is clear. Maybe including it at the very start (in the "Working with Source Code" section) would have helped? https://www.gnu.org/software/emacs/manual/html_node/org/Working-with-Source-Code.html - A specific section of manual was referenced, "Results of Evaluation": - https://www.gnu.org/software/emacs/manual/html_node/org/Results-of-Evaluation.html - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org#n18460 The specific problems I've identified: PROBLEM: "wrapping" is inaccurate PROBLEM: "code should execute like a function" is non-committal PROBLEM: terms are not well-defined Different language has been used over the past 10+ years to describe behavior of source blocks. The "Results of Evaluation" section mentions "functional mode" and "scripting mode". Collaboration notes (org-babel.org) were kept in the early days of Babel. Eric (the original author of Org Babel) made a distinction between (what he calls) "functional" and "imperative." It's not clear to me what's meant by these terms. Some snapshots of org-babel.org: - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/contrib/babel/org-babel.org?id=c0554c775344207e6adaf191258312fb8d1b6a15 - https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/org-babel.org?id=b1c103890c1523f99e380d88ed684454d902414e Here's how I think of it: "Session" means an environment is "persistent." Each call is executed in the same environment. State exists between calls. In the early-history of Babel, this was called the "imperative" style. "Non-session" means an environment is "temporary." Each call is executed in an independent environment. State does not exist between calls. In the early-history of Babel, this was called the "functional" style. All of this relates to the ":results value" and ":results output" header arguments. This is not the first time this has been discussed: https
Re: [PATCH] org-manual: Describe export process flow
e-cell-functions | X | | | org-export-filter-table-functions | X | | | org-export-filter-table-row-functions | X | | | org-export-filter-target-functions | X | | | org-export-filter-timestamp-functions | X | | | org-export-filter-underline-functions | X | | | org-export-filter-verbatim-functions| X | | | org-export-filter-verse-block-functions | X | | |-++---| * Feedback 1: How are the functions not present in the patch handled? I have no clue if they're relevant here or if someone would be confused by their absence. I only noticed that some of them were used in the patch and that the purpose of the patch also deals with filters. * Feedback 2: If I read the patch correctly, the it places the summary *before* the "Export hooks" section. This means that the summary will refer to the hooks before they're introduced (AFAICT, they're only ever mentioned in the "Export hooks" section). The summary should come after the hooks. Maybe before "Extending an existing back-end" or before the next Chapter (Publishing)? * Feedback 3: These are nitpicky things. > 1. The source Org mode buffer is copied into temporary throwaway > buffer that can be edited by export hooks "throwaway" is unnecessary because "temporary" implies it. > 6. When ~org-export-use-babel~ is non-nil (default), all the src > blocks and babel calls that are not inside archived headings are > processed I would write out "src" as "source". Do we have an official way to refer to source blocks? For example, we standardize on "Org": https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/Documentation_Standards.org#n47 Capitalize "babel" since it's a name. > 8. Export option values are calculated, according to in-buffer >keywords, =#+BIND= keywords, buffer-local and global >customizations. The first comma is not necessary. The option values are calculated according to the list that follows (in-buffer keywords, BIND, etc.). Remove the trailing period or add periods to all the others. I tend leave the period of the last sentence of a list. I'm not sure of a style guide that recommends one or the other. Maybe someone knows what's "right"? > - Heading are removed according to =SELECT_TAGS= and > =EXCLUDE_TAGS= export keywords; =task=, =inline=, =arch= export > options "Heading" should be "Headings" > 17. AST is transcoded according to the chosen export backend > - The export happens recursively, depth-first > - Each transcoded AST node, as a string, is passed to the > corresponding export filter (see [[*Filters]]) > 18. The transcoded AST body is formatted according to backend's > "inner" template Maybe use "converted" instead of "transcoded"? I'm a native speaker but I wonder if "converted" is a simpler word for people who aren't. The sentence is simpler by removing the commas and rearranging, "Each transcoded AST node is passed as a string to the corresponding export filter (see [[*Filters]])"" What's an "inner" template and why does that matter? > 20. The filtered body is formatted according to backend's outer > template What's an "outer" template and why does it matter? Should "outer" be quoted like "inner" was? Or, should "inner" be unquoted? > 21. The resulting output is processed by citation backend finalizer "Finalizer" is vague. What's its significance? Does it correspond to a function the end-user might look up? Overall, great work! For each "negative" bit of feedback, there were at least two things that deserve "positive" comments. I saw no spelling errors, you wrote out abbreviations (AST) before using them, and there are no voice issues. Everything was to the point. At first I was confused why this is being added to the manual and not to Worg. However, after reading it more thoroughly, I believe it will be a welcome and helpful addition to the manual. The export process is complex! -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] org-manual: Describe export process flow
On Tue, 26 Dec 2023 22:56:00 +0100 Matt wrote --- > I tend leave the period of the last sentence of a list. Typo. I meant to write "I tend TO leave the period OFF the last sentence of a list." That is, the item ends without a period. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v2] org-manual: Describe export process flow
On Wed, 27 Dec 2023 14:41:32 +0100 Ihor Radchenko wrote --- > Thank you all for the feedback! Thanks for taking the time to write and revise the patches. You didn't explicitly ask for more feedback, so I hope the following feedback on the new patches is in order. Both patches look good. I see no issues with v2-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch For v2-0002-doc-org-manual.org-Describe-export-flow.patch here are some quibbles: > On high-level, the export process consists of 4 major steps: I don't think "On high level" is necessary (it's also missing an 'a'; "On *a* high level"). "Major" implies high-level. "The export process consists of 4 major steps:" If you wanted to keep "high-level", you could replace major with it: "The export process consists of 4 high-level steps:" > 2. Parse the buffer, converting plain Org markup into abstract syntax >tree (AST); Missing "an": "...converting plain Org markup into *an* abstract syntax..." > 5. Parse the accessible portion of the temporary buffer to generate >AST. The AST is a nested list of lists representing Org syntax >elements "...of the temporary buffer to generate *the* AST." or "...of the temporary buffer to generate *an* AST." >Past this point, modifications in the temporary buffer copy no >longer affect export; Org export works only with the AST; Replace "in the" with "to the". "no longer affect export" -> "no longer affect *the* export" Replace "temporary buffer copy" with "temporary buffer" or "temporary copy". "Past this point, modifications to the temporary buffer no longer affect the export; Org export works only with the AST;" > 6. Remove elements that will not be exported from the AST: "Remove elements that are not exported from the AST:" or "Remove elements not exported from the AST:" >- Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export > keywords, and =task=, =inline=, =arch= export options (see > [[*Export Settings]]); Remove comma after "keywords": "Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export keywords and =task=, =inline=, =arch= export options (see [[*Export Settings]]);" >- Clocks, drawers, fixed-width environments, footnotes, LaTeX > environments and fragments, node properties, planning lines, > property drawers, statistics cookies, timestamps, timestamps, > etc according to =#+OPTIONS= keyword (see [[*Export Settings]]); "timestamps" is written twice. > - Table columns containing recalc marks (see [[*Advanced features]]). This is indented two spaces more than the previous item. Is that what you want? > 7. Expand environment variables in file link AST nodes, according to >the =expand-links= export option (see [[*Export Settings]]); Comma after "nodes" not necessary. "Expand environment variables in file link AST nodes according to the =expand-links= export option (see [[*Export Settings]]);" > 8. Execute ~org-export-filter-parse-tree-functions~. These >functions can modify AST by side effect; I would say "*the* AST" and make "side effect" plural: "...can modify *the* AST by side effect*s*;" > 4. Convert the nodes with children to text, passing the nodes >themselves and their contents to the corresponding transcoders >and then to export filters (see [[*Filters]]). Add a "the": "...and then to *the* export filters (see [[*Filters]])." - Checking ox.el, I see that the commentary says, ";; See <https://orgmode.org/worg/dev/org-export-reference.html> for ;; more information." Maybe also let maintainers know about this manual section: "See: <https://orgmode.org/worg/dev/org-export-reference.html> " - Again, good work. AFAIU, you used "affect" and "effect" correctly. Breaking the big list into smaller ones makes it much easier to read. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v3] org-manual: Describe export process flow
On Wed, 27 Dec 2023 18:05:05 +0100 Ihor Radchenko wrote --- > Matt m...@excalamus.com> writes: > > Checking ox.el, I see that the commentary says, > > > > ";; See https://orgmode.org/worg/dev/org-export-reference.html> for > > ;; more information." > > > > Maybe also let maintainers know about this manual section: > > > > "See: > > https://orgmode.org/worg/dev/org-export-reference.html > > some-kind-of-reference-to-the-new-manual-section > > " > > AFAIU, you are asking to add it to WORG page. However, it is too early. > The online Org manual that we can link to is only for stable Org > version. So, only after the next release. I was making a different point. I was trying to say that it may be a good idea for the ox.el commentary section to specifically reference the new manual section (next to where it references Worg). The new manual section is unique in that it gives detailed steps about the order of execution. Without knowing the code (so I may be mistaken), it seems like a risk that the ox.el code and manual could get out of sync. This is why I wouldn't suggest documenting the execution order for something like source blocks. However, since end-users have a lot of flexiblity in customizing the export process, it's helpful for them to understand it. It makes sense to add the new manual section. Mentioning the new manual section in the ox.el commentary would help someone new to the ox.el code. It may also help keep the new manual section and ox.el aligned (should anything in the execution order/process change in ox.el) by reminding someone changing the code that there's a manual section specifically about *how* the code works. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH v4] org-manual: Describe export process flow
On Thu, 28 Dec 2023 13:05:27 +0100 Ihor Radchenko wrote --- > See the attached. All looks good to me. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] doc/org-manual.org (Summary): Clarify the Org markup is human-readable
On Sun, 31 Dec 2023 16:22:00 +0100 Ihor Radchenko wrote --- > > I'd like to amend the Org manual introduction as in the attached patch. > The idea is to highlight that Org markup is designed to be > human-readable first and foremost rather than just computer-readable. > > This is the first section of the manual. Most of the new users will see > it. So, I'd like to hear any objections before installing the patch. I like the points about using any text editor and understanding Org without an editor. The main point, it seems, is Org mode avoids lock-in. Two perspectives, the author and the reader, are addressed. An important question is, "Who are we writing to?" Addressing authors and readers covers pretty much everyone, I think :) I believe that makes these good points for the intro. The first paragraph of the current introduction reads, "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a lightweight plain-text markup language used in files with the =.org= extension." Since the first paragraph already mentions "lightweight plain-text markup language," I think making a similar point in the next paragraph is unnecessary. It may be better to say what you want in the first paragraph. I like "human-readable." It's clear and direct. However, trying to incorporate it into the first paragraph becomes heavy: "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a human-readable, plain-text markup language used in files with the =.org= extension." That's a lot of hyphenated words and syllables! Maybe "legible" would be better? It's more precise. But maybe precise isn't good. I'm wondering if there's a simpler word that accommodates non-native speakers better (and not just for this word). "Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. It relies on a legible, plain-text markup language used in files with the =.org= extension." Last comment on your changes. I understand what you mean by "with a naked eye" and I think it's a valid point. However, not everyone who uses Emacs can see. I wonder, do the benefits of Org syntax only apply visually? Several things in the first paragraph unrelated to your changes stick out to me. I can't help but make some other remarks. "TODO" should probably be "to-do". Every dictionary I looked in has an entry for "to-do". I think that's the common spelling. Regarding "markup language," that reads to me like programmer jargon. What does it mean and why should someone care? Again, who are we writing to? A markup language is a notation for formatting, structure, and relationships. I think it would be best to directly say that. I would also soften that Org "relies" on its markup. It doesn't. I used Org only for lists for a long time. I believe lists to be a fundamental feature of Org (and hence a great item for the first sentence). Lists are as simple as dashes. It's hard to say that dashes before list items is a markup language. Finally, I don't think the file extension is relevant for the first paragraph. Technically, an extension isn't necessary. A person can call M-x org-mode or use a file local variable. Worse, I think the extension contradicts the point that any text editor can view an Org file. Ever try to open a .org file in Windows? It asks for the program. Yes, *technically* Windows could open a .org file *if* the person opening it knew which program to use (or to change the extension to something like .txt). Again, who are we writing to? If it's someone who believes file extensions matter, then this would introduce unnecessary friction. It seems best to avoid it. Better to do as you've done and say Org is readable (which it is) rather than specify the extension (which doesn't really matter). Text encodings notwithstanding (plain-text is only as portable as far as you can guess the encoding), what do you think of something like this? modified doc/org-manual.org @@ -18,9 +18,10 @@ :END: #+cindex: summary -Org Mode is an authoring tool and a TODO lists manager for GNU Emacs. -It relies on a lightweight plain-text markup language used in files -with the =.org= extension. +Org Mode is an authoring tool and a to-do lists manager for GNU Emacs. +It uses a legible plain-text notation to show formatting, structure, +relationships. Anyone able to edit text can write using Org. Anyone +able to read text can view it. As an authoring tool, Org helps you write structured documents and provides exporting facilities. Org files can also be used for literate -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at https://liberapay.com/org-mode
Re: [PATCH] [WORG] Maintenance: on the frequency of bugfix releases
On Sun, 10 Dec 2023 13:24:30 +0100 Ihor Radchenko wrote --- > Doing the releases too frequently will increase a chance of such bugs > crawling into ELPA releases. So, there should be at least a minimal > waiting period to get a chance to handle such problems. Thank you for explaining. I had missed that ELPA gets a combination of major, minor, and bugfix releases.
Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)
On Thu, 14 Dec 2023 15:35:13 +0100 Ihor Radchenko wrote --- > Matthew, this thread might be of interest for you as the new feature is > largely aiming at the shell block output. > Feel free to jump in if you have comments on the design of the > ASCII fontification for complex shell block output. Thank you for bringing this to my attention and thank you Nathaniel for your work on this. I have no comments on the design presently (my knowledge of Emacs fontification is currently limited) and my current priorities prevent me from dedicating the time this topic deserves. I think the topic is interesting and important. I've had issues with ANSI escape codes (in particular progress bars) in source block results. I made a note to return to this thread in case the escape codes don't bring me back :) Nathaniel, if you and I happen to cross paths in one of Ihor's "office hours," I would enjoy learning more about what you're doing.
