Re: strange export problem with a file: link

[Fraga, Eric]

On Thursday, 18 Apr 2024 at 21:46, Max Nikulin wrote:
> Sorry that I was not clear enough. Org code should be fixed. For a
> while you may revert that commit in your local repository.

Ah, okay!  Thank you.  I'll leave my documents as they are then. :-)

-- 
: Eric S Fraga, with org release_9.6.23-1320-g1be2f9 in Emacs 30.0.50

Re: strange export problem with a file: link

[Max Nikulin]

On 18/04/2024 21:37, Fraga, Eric wrote:

On Thursday, 18 Apr 2024 at 19:08, Max Nikulin wrote:

It is a side effect of

72b0e9ff0 2024-02-05 16:39:05 +0100 Ihor Radchenko: org-export: Do not
strip link type by default during export


I have now checked the ORG-NEWS file, which I probably should have
looked at before posting (sorry!).  The implication is that I should use
org-link-set-parameters to define the :export field for the file link.


Sorry that I was not clear enough. Org code should be fixed. For a while 
you may revert that commit in your local repository.

Re: strange export problem with a file: link

[Fraga, Eric]

On Thursday, 18 Apr 2024 at 19:08, Max Nikulin wrote:
> It is a side effect of
>
> 72b0e9ff0 2024-02-05 16:39:05 +0100 Ihor Radchenko: org-export: Do not
> strip link type by default during export

Thank you Max.

I have now checked the ORG-NEWS file, which I probably should have
looked at before posting (sorry!).  The implication is that I should use
org-link-set-parameters to define the :export field for the file link.
I'll explore in due course.  It might be nice if the news entry gave a
simple example to get back to the previous behaviour.  At this point,
I've moved to using MathML instead for what I need doing right now so
the problem is no longer an issue for me (at the moment).

However, although I'm sure there was a good reason for the change, it
has broken quite a few of my documents!  Not complaining; just noting.
Living at the edge does carry a price...

Thanks again,
eric

-- 
: Eric S Fraga, with org release_9.6.23-1320-g1be2f9 in Emacs 30.0.50

Re: strange export problem with a file: link

[Max Nikulin]

On 18/04/2024 17:50, Fraga, Eric wrote:

#+begin_src latex :results file raw :exports results :file function.png
\[ y = f(x) \]
#+end_src

org-odt-export-to-odt: OpenDocument export failed: Opening input file: No such 
file or directory, /tmp/file:function.png


It is a side effect of

72b0e9ff0 2024-02-05 16:39:05 +0100 Ihor Radchenko: org-export: Do not 
strip link type by default during export

strange export problem with a file: link

[Fraga, Eric]

Hello all,

I have a file that consists of these three lines:

--8<---cut here---start->8---
#+begin_src latex :results file raw :exports results :file function.png
\[ y = f(x) \]
#+end_src
--8<---cut here---end--->8---

With "emacs -Q" and loading in org from the git repository as of maybe
an hour or so ago, I get this error:

org-odt-export-to-odt: OpenDocument export failed: Opening input file: No such 
file or directory, /tmp/file:function.png

when I try to export the file to ODT.  Exporting to LaTeX (PDF) works
just fine.  For the ODT export, the "file:" link specification is not
being removed.

I don't think I'm doing anything wrong; is this a bug in the exporter?

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.6.23-1320-g1be2f9 in Emacs 30.0.50

Re: problem with https://orgmode.org/manual/Results-of-Evaluation.html

[Ihor Radchenko]

Uwe Brauer via "General discussions about Org-mode."
 writes:

> Ok understood, but then, who can I add hlines to table that does not
> possess any?

#+name: test
| 1 | 2 |
| 3 | 4 |

#+name: add-hlines
#+begin_src emacs-lisp :var table='(1 2) :results value
(let (result)
  (when table
(unless (eq 'hline (car table)) (push 'hline result))
(while table
  (push (pop table) result)
  (unless (eq 'hline (car result))
(push 'hline result)))
(nreverse result)))
#+end_src


#+begin_src python :var table=test :post add-hlines(table=*this*)
return [[1+x for x in row] for row in table]
#+end_src

#+RESULTS[63f00119cafb9b4a9e762f7f19db5d5c0b215d2d]:
|---+---|
| 2 | 3 |
|---+---|
| 4 | 5 |
|---+---|


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: problem with https://orgmode.org/manual/Results-of-Evaluation.html

[General discussions about Org-mode.]

>>> "IR" == Ihor Radchenko  writes:

> Uwe Brauer via "General discussions about Org-mode."
>  writes:

>> #+NAME: many-cols
>> | a | b | c |
>> | d | e | f |
>> | g | h | i |
>> 
>> #+NAME: hline-please
>> #+BEGIN_SRC python :var tab=many-cols :hlines yes
>> return tab
>> #+END_SRC
>> 
>> #+RESULTS: hline-please
>> | a | b | c |
>> | d | e | f |
>> | g | h | i |
>> 
>> Works, but I also hoped that I could use python to add hlines to table
>> without hline, but alas it does not work!

> :hlines controls filtering  from input. It does nothing about code
> block output.

> More specifically, ":hlines no" takes care about removing hlines from
> the input. ":hlines yes" does not perform any filtering.

Ok understood, but then, who can I add hlines to table that does not
possess any?
-- 
I strongly condemn Hamas heinous despicable pogroms/atrocities on Israel
I strongly condemn Putin's war of aggression against Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the EU and NATO membership of Ukraine. 



smime.p7s
Description: S/MIME cryptographic signature

Re: problem with https://orgmode.org/manual/Results-of-Evaluation.html

[Ihor Radchenko]

Uwe Brauer via "General discussions about Org-mode."
 writes:

> #+NAME: many-cols
> | a | b | c |
> | d | e | f |
> | g | h | i |
>
> #+NAME: hline-please
> #+BEGIN_SRC python :var tab=many-cols :hlines yes
> return tab
> #+END_SRC
>
> #+RESULTS: hline-please
> | a | b | c |
> | d | e | f |
> | g | h | i |
>
> Works, but I also hoped that I could use python to add hlines to table
> without hline, but alas it does not work!

:hlines controls filtering  from input. It does nothing about code
block output.

More specifically, ":hlines no" takes care about removing hlines from
the input. ":hlines yes" does not perform any filtering.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: problem with https://orgmode.org/manual/Results-of-Evaluation.html

[General discussions about Org-mode.]

> Uwe Brauer via "General discussions about Org-mode."
>  writes:


> No, this is because you customized `org-src-preserve-indentation' or
> `org-edit-src-content-indentation', while the provided example only
> works with default indentation settings. Python is sensitive to
> whitespace and your config prevents Org mode from removing the two
> spaces in the example block.

> I adjusted the manual to not indent src blocks.  Then, they should work
> across more wide range of user settings.

Thanks, I confirm that 

#+NAME: many-cols
| a | b | c |
| d | e | f |
| g | h | i |

#+NAME: hline-please
#+BEGIN_SRC python :var tab=many-cols :hlines yes
return tab
#+END_SRC

#+RESULTS: hline-please
| a | b | c |
| d | e | f |
| g | h | i |

Works, but I also hoped that I could use python to add hlines to table
without hline, but alas it does not work!


-- 
I strongly condemn Hamas heinous despicable pogroms/atrocities on Israel
I strongly condemn Putin's war of aggression against Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the EU and NATO membership of Ukraine. 



smime.p7s
Description: S/MIME cryptographic signature

Re: problem with https://orgmode.org/manual/Results-of-Evaluation.html

[Ihor Radchenko]

Uwe Brauer via "General discussions about Org-mode."
 writes:

> I tried to use the examples of the above link
> ...
> #+NAME: no-hline
> #+BEGIN_SRC python :var tab=many-cols :hlines no
>   return tab
> #+END_SRC
>
> Now already the first failed:
> ,
> | 
> |   File "", line 4
> | return tab
> | ^
> | IndentationError: unexpected indent
> | [ Babel evaluation exited with code 1 ]
> `
> ...
> I presume the code is for python 3.X. How can I tell org to use that
> one?

No, this is because you customized `org-src-preserve-indentation' or
`org-edit-src-content-indentation', while the provided example only
works with default indentation settings. Python is sensitive to
whitespace and your config prevents Org mode from removing the two
spaces in the example block.

I adjusted the manual to not indent src blocks.  Then, they should work
across more wide range of user settings.

Handled.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=78754cdcf

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

problem with https://orgmode.org/manual/Results-of-Evaluation.html

[General discussions about Org-mode.]

Hi 

I tried to use the examples of the above link

#+NAME: many-cols
| a | b | c |
|---+---+---|
| d | e | f |
|---+---+---|
| g | h | i |

#+NAME: no-hline
#+BEGIN_SRC python :var tab=many-cols :hlines no
  return tab
#+END_SRC

Now already the first failed:
,
| 
|   File "", line 4
| return tab
| ^
| IndentationError: unexpected indent
| [ Babel evaluation exited with code 1 ]
`----

Problem could be that which python on my Ubuntu system returns
Python 2.7.12

I presume the code is for python 3.X. How can I tell org to use that
one?

Regards

Uwe Brauer 

-- 
I strongly condemn Hamas heinous despicable pogroms/atrocities on Israel
I strongly condemn Putin's war of aggression against Ukraine.
I support to deliver weapons to Ukraine's military. 
I support the EU and NATO membership of Ukraine. 

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 22/11/2023 02:01, Bruno Barbier wrote:

Ihor Radchenko writes:

__By default__, Org should produce more expected behavior - what users
would get from running a script file rather than from redirecting stdin.
We can optionally leave the stdin redirection as an option to be used by
the users who understand the peculiarities.


I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).


I do not like recommendations

curl https://... | sudo bash

but I regularly see them. So some users might expect behavior like 
namely stdin, not like interactive prompt commands.


Perhaps even sessions may be switched to creation of a temporary file 
and executing "source tmpfile.sh". However it may be shell-specific (posh?).


Likely it is possible to implement a header argument to explicitly 
control whether `process-file' should be used or the block should be 
executed as a temporary script file. Currently it is implicitly 
determined from other header arguments.


P.S.

https://mywiki.wooledge.org/BashFAQ/089
"I'm reading a file line by line and running ssh or ffmpeg, only the 
first line gets processed!"


A similar case, but only data is read from stdin, script is a regular 
file, no difference bash vs. dash.

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> The behavior of M-x shell can indeed be made use of.
> However, this particular difference with xterm, AFAIU, is not
> documented - unaware users may be surprised.
> The situation is worse with Org shell blocks - users naturally expect
> script-like behavior (even for :session), but run into edge cases like
> this and get confused.
>
> We should either document the caveats, or, preferably, make the behavior
> more consistent with expectations. At least, by default.

> That's why I think that filing a bug report makes sense from Org mode
> project point of view.

Thanks for the explanation.  I see you got my point.  We'll see what
Emacs maintainers will say about the current behavior of M-x shell;
filling the bug report was definitely a good idea anyway.



Ihor Radchenko  writes:
> __By default__, Org should produce more expected behavior - what users
> would get from running a script file rather than from redirecting stdin.
> We can optionally leave the stdin redirection as an option to be used by
> the users who understand the peculiarities.

I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).

Bruno

Re: Non-emacs shell (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Hi Max,

Max Nikulin  writes:

> On 17/11/2023 22:47, Bruno Barbier wrote:
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> I am unsure what do you expect from xterm, but perhaps it is not 
> responsibility of a terminal application.

It has been said in this thread that 'M-x shell' should be fixed to
match the behavior that we see in a plain terminal, when we copy
multiple lines.  I just wanted to point out that I do prefer the way
'M-x shell' handles the copy of multiple lines.


> Multiple lines can be copied to regular BASH prompt (bracketed paste is 
> enabled by default nowadays), however it may be inconvenient to edit.
> You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
> binding) to start an editor and to paste multiple commands there.
> See also "fc" BASH built-in for editing and re-executing last commands.

Thanks Max! I didn't know that.  I should definitely start using this
when I'm stuck in a console, to safely copy/edit my commands using
Emacs.

Thanks,


Bruno

Non-emacs shell (Re: bash source code block: problem after ssh commands)

[Max Nikulin]

On 17/11/2023 22:47, Bruno Barbier wrote:

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


I am unsure what do you expect from xterm, but perhaps it is not 
responsibility of a terminal application.


Multiple lines can be copied to regular BASH prompt (bracketed paste is 
enabled by default nowadays), however it may be inconvenient to edit.


You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
binding) to start an editor and to paste multiple commands there.


See also "fc" BASH built-in for editing and re-executing last commands.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 17:43, Ihor Radchenko wrote:


I still see it as a bug - what Org mode does to run the shell blocks is
not what users expect. _By default_, we _should_ produce more expected
behavior.


A shell without unexpected behavior is neither POSIX nor a descendant 
shell like BASH

https://mywiki.wooledge.org/BashPitfalls

Behavior if interactive and non-interactive shells is not the same anyway.


I cannot find any clear motivation behind using `process-file' + INFILE
in the git logs. I assume that it was used simply because it is easier
compared to trying to create and run temporary script file on remote
hosts.


From my point of view it is rather natural to use `process-file' since 
this function is available out of the box. I suspect some subtle issues 
may appear even if script file is forced.


On the other hand ob-shell already has a (bit buggy) implementation 
relying on script files. It is used when :stdin or :cmdline is specified.

Re: bash source code block: problem after ssh commands

[Matt]

  On Fri, 17 Nov 2023 23:07:57 +0100  Matt  wrote --- 
 
 > The second claim has nothing to do with Org Babel.  I was able to confirm it 
 > and provide the steps to reproduce.  I think it would make sense to report 
 > it upstream and let them decide if it's expected behavior.  I'm still happy 
 > to do that, but I need to step away from the keyboard :)

Submitted. 

https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Ihor Radchenko]

Matt  writes:

> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
> cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine 
> and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Yup, it is expected. But only in a sense that "Emacs does what we asked
for when calling `process-file'". From the user point of view, it is not
expected at all. And, what is worse, may depend on (1) shell used; (2)
ssh config (see StdinNull in man 5 ssh_config).

I believe that we do need to change how we execute shell blocks _by
default_ - to something more predictable, like creating and running a
temporary script file.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin  writes:

> ...
> I have not expected this difference.
>
> dash reads a block from stdin (whole file in this case) and interprets 
> commands.
>
> BASH reads just the ssh command and executes it. SSH reads "echo done" 
> from stdin, so when control is returned to bash, stdin is exhausted and 
> no commands remain to execute by BASH. SSH can not "unread" part of 
> input not consumed by the remote command despite it might be possible in 
> the case of the regular file as stdin.
>
> Actually bash reads the whole script file as well when called as it is 
> shown above, but it calls lseek before executing ssh. To make difference 
> more apparent (e.g. for strace), force creation of pipe(7) for which 
> lseek is not supported
> ...
> I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
> vs. dash issue.

I still see it as a bug - what Org mode does to run the shell blocks is
not what users expect. _By default_, we _should_ produce more expected
behavior.

The observed inconsistency between different shells just indicates that
our approach with `process-file' should not be used as it leads to
potentially confusing results (they are not confusing only to people who
dig deeply into ssh, bash/dash/etc, and Org mode internals).

I cannot find any clear motivation behind using `process-file' + INFILE
in the git logs. I assume that it was used simply because it is easier
compared to trying to create and run temporary script file on remote
hosts.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Bruno Barbier  writes:

>> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
>> the problem with M-x shell is not the problem you originally ran to. It
>> is a different problem (also, we ran into it in the past).
>
> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
> multiple lines, getting the same results as when I type them manually,
> or copy them line by line. My xterm doesn't seem to allow me to do that.

The behavior of M-x shell can indeed be made use of.
However, this particular difference with xterm, AFAIU, is not
documented - unaware users may be surprised.
The situation is worse with Org shell blocks - users naturally expect
script-like behavior (even for :session), but run into edge cases like
this and get confused.

We should either document the caveats, or, preferably, make the behavior
more consistent with expectations. At least, by default.

That's why I think that filing a bug report makes sense from Org mode
project point of view.

> It looks like it is a known SSH "feature" (see
> https://unix.stackexchange.com/a/688024):
>
> #+begin_src bash :results output
> seq 100 | (ssh phone sleep 1; wc -l)
> #+end_src
>
> #+RESULTS:
> : 675173
> ...

> IMHO, what ob-shell is doing today seems a valid way of evaluating
> source blocks (and it seems to have been like that for a long time).  It
> should probably be documented somewhere, so that users know how to write
> their source blocks, or switch to another way, like adding a :cmdline
> parameter as mentionned in this thread.

And, as Max demonstrated, this ssh feature only works with bash, but not
necessarily other shells. Moreover, "-n" option that disables the above
may depend on ssh configuration (see StdinNull in man 5 ssh_config). So,
I believe that it only leads to confusion, even if we try to document
it.

__By default__, Org should produce more expected behavior - what users
would get from running a script file rather than from redirecting stdin.
We can optionally leave the stdin redirection as an option to be used by
the users who understand the peculiarities.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
>  
> It's still not clear to me if this is "what Emacs does".  However, that's the 
> best I could come up with.  
>
> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
> cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine 
> and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Agreed.

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

  On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
 
 > But, you're right.  To be safe, from now on, I'll use:
 > 
 > cat /tmp/test.sh | bash -c bash

It's still not clear to me if this is "what Emacs does".  However, that's the 
best I could come up with.  

Evaluating the following 

#+name: /tmp/test.sh
#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
#+end_src

does exactly what

cat /tmp/test.sh | bash -c bash

does.  Both create a file "foo_file" containing "foo" on the remote machine and 
neither execute the second line.

So, I would say that what happens in Org is the "expected" behavior.

Re: bash source code block: problem after ssh commands

[Matt]

To clarify a previous typo I made.  Everything I've written was also done using:

1. emacs -q
2. C-x b "*scratch*"
3. M-x org-mode
4. Execute 

#+begin_src emacs-lisp :results none
(org-babel-do-load-languages
   'org-babel-load-languages
   '((shell . t)))
#+end_src

This corresponds to Org mode version 9.6.6 (release_9.6.6 @ 
/gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/)

I had previously, and incorrectly, given my version used with my full init 
(which uses a different Org version).

  On Sat, 18 Nov 2023 09:19:13 +0100  Bruno Barbier 
 
 > It seems your SSH failed to connect.  In that case, I cannot swallow the
 > second command; thus the command "echo bar" is executed.
 > 
 > I can reproduce what you see on my side if I force the connection to fail:
 > 
 > #+begin_src bash :results output
 >   ssh WRONG_REMOTE "echo foo>foo_file"
 >   echo "bar"
 > #+end_src
 > 
 > #+RESULTS:
 > : bar
 
Thank you!  That makes sense.   I forgot that without set -e Bash keeps 
evaluating when errors happen.

Yes, once it connects, I get the same behavior as reported.

Here is how I connected without a password prompt:

1. ssh-keygen with id_babel, , 
2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys
3. chmod og-wx ~/.ssh/authorized_keys
4. ssh-add .ssh/id_babel

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 
>
>  > IIUC, what Max is saying is that you should not concentrate on
>  > *that specific command* because that command doesn't do what you think
>  > it does.
>
> Cool, it sounds like we're agreed (albeit for different reasons).

Great :-)


>  > To reproduce, I'm personally still using:
>  > 
>  > cat /tmp/test.sh | bash
>  > 
>  > which is, IIUC, what:
>  > 
>  > (process-file "bash" "/tmp/test.sh")
>
> Yes, agreed.  I think that's more like what's happening.  
>
> What about the ("-c" "bash") passed into process-file?  

Useless indirection when the command is "bash" is the same as
'shell-file-name', like in our case ?

Maybe ...

But, you're right.  To be safe, from now on, I'll use:

cat /tmp/test.sh | bash -c bash


Thanks.

Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

  On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 

 > IIUC, what Max is saying is that you should not concentrate on
 > *that specific command* because that command doesn't do what you think
 > it does.

Cool, it sounds like we're agreed (albeit for different reasons).  

 > To reproduce, I'm personally still using:
 > 
 > cat /tmp/test.sh | bash
 > 
 > which is, IIUC, what:
 > 
 > (process-file "bash" "/tmp/test.sh")

Yes, agreed.  I think that's more like what's happening.  

What about the ("-c" "bash") passed into process-file?  

The whole call looks like this:

(process-file
   "bash"
   "/tmp/two-lines.sh"
   '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
   nil
   "-c" "bash")

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Max Nikulin  writes:

> On 25/10/2023 18:17, alain.coch...@unistra.fr wrote:
>> By contrast, it works with this one:
>> 
>> #+begin_src bash :results output
>> sshcoch...@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
>> #+end_src
>
> What about
>
> #+begin_src bash :results output
> ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
> echo more
> #+end_src
>
> ?

For me:

 #+begin_src bash :results output
 ssh phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

 #+RESULTS:
 : bar

And, telling SSH to not swallow the remaining commands (option '-n'):

 #+begin_src bash :results output
 ssh -n phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

#+RESULTS:
: bar
: more


Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt  writes:

>   On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 
>
>  > >  bash -c bash /tmp/two-lines.sh
>  > 
>  >  From my point of view it was a plain mistake in attempts to simulate 
>  > the issue outside of Emacs. There is no point to concentrate on this 
>  > command. I tried to explain that it is incorrect usage of "-c" shell 
>  > option and what is the actual effect of this call, but I seems I failed.
>
> As an ob-shell user, my expectation is that execution within Org produces the 
> same behavior as outside of Emacs.  This is why I've focused on the command.  
> It acts as a guide for what is "correct."  Maybe this is misguided for a 
> reason I don't yet see?

IIUC, what Max is saying is that you should not concentrate on
*that specific command* because that command doesn't do what you think
it does.


To reproduce, I'm personally still using:

cat /tmp/test.sh | bash

which is, IIUC, what:

(process-file "bash" "/tmp/test.sh")

is doing, that is:

   The program’s input comes from file INFILE


Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt,

Thanks this summary and for working on this!

Just a few comments/corrections about some specific points, hoping it
might help.

Matt  writes:

>   On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 
>
>  > This has nothing to do with Emacs comint and this is also not a bug in
>  > Emacs 
>
> Ihor, there were two claims made in the original report.  I was referring to 
> Claim 2.  That deals with M-x shell and therefore comint-mode.
>
> Regarding Claim 1:
>
> - Can anyone verify Claim 1?

I do: the file is created and the command "echo bar" is NOT executed.

Here is my code block and its results:

#+begin_src bash :results output
  ssh phone "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:

No results (the echo command is NOT executed).

The file "foo_file" is created on the remote; its content is "foo".

#+begin_src bash :results output
  date
  ssh -n phone "ls -alh foo_file"
  ssh -n phone "cat foo_file"
#+end_src

#+RESULTS:
: Sat Nov 18 08:33:59 CET 2023
: -rw--- 1 u0_a256 u0_a256 4 Nov 18 08:26 foo_file
: foo



> - What versions are people using?
>   + M-x org-version
>   + M-x emacs-version

#+begin_src elisp
  (list emacs-version org-version)
#+end_src

#+RESULTS:
| 30.0.50 | 9.7-pre |

GNU/Linux gentoo

> ...

> * Comments about the claims:

> ** Comment 1.
> ...
> I am unable to reproduce the reported behavior (of
> "bar" not returning).  Instead, I get an ssh-askpass permission denied
> error, foo_file is not created, and "bar" is given as the result.  I
> do not see anywhere in the thread that the original claim was
> reproduced.

It seems your SSH failed to connect.  In that case, I cannot swallow the
second command; thus the command "echo bar" is executed.

I can reproduce what you see on my side if I force the connection to fail:

#+begin_src bash :results output
  ssh WRONG_REMOTE "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:
: bar


>
> The thread preceded something like follows.
>
> Leo Butler suggested two work arounds:
>
> - add the -f to the ssh command


> - add a semi-colon and line continuation to the first line.
>
> Russell Adams suggested another work around:
>
> - add -n to the ssh command

That's the one I use; the option -n is enough for me ('-n' = Redirects
stdin from /dev/null). The option '-f' means SSH will go to background;
I'm not sure I want that.

> ...

> ... 
> He then proposes an experiment to close stdin.  To do this, he calls
>
> #+begin_src shell :results output
> exec 0>&-
> echo OK
> #+end_src
>
> He claims that "exec 0<&-" closes stdin.  I believe there is a typo.
> ...

You're right. Good catch, thanks!

Although it seems to work either way on my side.

#+begin_src shell :results output
  exec 0<&-
  echo OK
#+end_src

#+RESULTS:

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

#+RESULTS:


> What Bruno writes corresponds to "closing output file descriptor 0".  I 
> honestly don't know what the difference is between an "output file 
> descriptor" and an "input file descriptor".  I had no luck finding this 
> information in man bash or info bash.
>

My point was: the commands are read the standard input, thus, any
command that modifies that standard input will modify what gets
executed.


> ...
> This is what we see in Org.  I'll be honest, though, I don't
> really know what to expect with exec 0>&- and exec 0<&-.  When I call
> them in the terminal, it kills the terminal.

Let's forget about 'exec 0<&-' (closing the standard input/outputs):
this is bringing other corner cases.  But, yes, I would expect a
terminal to close itself automatically if its input is closed.

> ...
> As far as I can tell, though, that's not what prevents "bar" from being 
> returned.  As far as I can reproduce, calling
>
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar"
> #+end_src
>
> *does* give "bar" for results even though it shouldn't.

Does it echo bar when the SSH connection succeeds too ?


Thanks again for working on this.


Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

  On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 

 > >  bash -c bash /tmp/two-lines.sh
 > 
 >  From my point of view it was a plain mistake in attempts to simulate 
 > the issue outside of Emacs. There is no point to concentrate on this 
 > command. I tried to explain that it is incorrect usage of "-c" shell 
 > option and what is the actual effect of this call, but I seems I failed.

As an ob-shell user, my expectation is that execution within Org produces the 
same behavior as outside of Emacs.  This is why I've focused on the command.  
It acts as a guide for what is "correct."  Maybe this is misguided for a reason 
I don't yet see?

I concluded that the command is not "what Emacs does" and therefore isn't a 
valid source of "truth".  I agree it's not worth concentrating on it further.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 25/10/2023 18:17, alain.coch...@unistra.fr wrote:

By contrast, it works with this one:

#+begin_src bash :results output
sshcoch...@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
#+end_src


What about

   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
   echo more
   #+end_src

?

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 17/11/2023 17:17, Ihor Radchenko wrote:

I was only able to reproduce your problem with ssh asking a password.
We are discussing the reproduced case.


I see bash vs. dash difference with public key authorization, so no need 
for password prompts. I have not figured out how to construct an example 
without ssh since this command *may* read stdin, but does not do it in a 
same way as e.g. cat(1). Perhaps a small program performing single 
non-blocking read will allow it. The following behavior observed for a 
regular shell prompt, Emacs is not involved. Debian 12 bookworm.


cat ssh-script.sh
ssh -p  127.0.0.1 'echo foo>/tmp/foo'
echo done

Read commands from a script file:

dash ssh-script.sh
done

bash ssh-script.sh
done

Read commands from stdin

dash dash reads a block from stdin (whole file in this case) and interprets 
commands.


BASH reads just the ssh command and executes it. SSH reads "echo done" 
from stdin, so when control is returned to bash, stdin is exhausted and 
no commands remain to execute by BASH. SSH can not "unread" part of 
input not consumed by the remote command despite it might be possible in 
the case of the regular file as stdin.


Actually bash reads the whole script file as well when called as it is 
shown above, but it calls lseek before executing ssh. To make difference 
more apparent (e.g. for strace), force creation of pipe(7) for which 
lseek is not supported


cat ssh-script.sh | strace -o /tmp/bash.strace bash

I am unsure if POSIX specifies exact behavior of shell when commands are 
read from stdin. I think, the suggested earlier "-n" ssh option (or 
stdin. There is too much room for heuristics: interactive vs. 
non-interactive shell, a terminal vs. a regular file vs. a pipe as 
standard input. Be explicit to get reliable behavior.


ssh -n user@host 'command'

or

tar cvf - . | ssh user@host 'tar xvf -'

without "-n" when ssh needs stdin and it is explicitly specified.

I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
vs. dash issue.

Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Max Nikulin]

On 18/11/2023 05:07, Matt wrote:

Consider calling

 bash -c bash /tmp/two-lines.sh


Matt, could you, please, describe what this command should do 
accordingly to your expectations?


From my point of view it was a plain mistake in attempts to simulate 
the issue outside of Emacs. There is no point to concentrate on this 
command. I tried to explain that it is incorrect usage of "-c" shell 
option and what is the actual effect of this call, but I seems I failed.


Hints:
- Try this command in a regular shell prompt outside of Emacs
- Try "echo $$" before and after this command
- Try it with a script having a clear side effect and check whether it 
is produced, e.g. "date >>/tmp/touch-tmp-file.txt"

Re: bash source code block: problem after ssh commands

[Matt]

  On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 

 > This has nothing to do with Emacs comint and this is also not a bug in
 > Emacs 

Ihor, there were two claims made in the original report.  I was referring to 
Claim 2.  That deals with M-x shell and therefore comint-mode.

Regarding Claim 1:

- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
  + M-x org-version
  + M-x emacs-version

I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ 
/home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, 
x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).

* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with 
contents "foo" as well as give "bar" as the result.

#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src

The reported behavior is that "foo_file" is created with "foo" (with "foo" is 
not stated, but implied) and "bar" is *not* given as the result.

** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like 
xfce4-terminal executes the ssh line as expected and outputs the result of the 
second line.  It was noted that this does not happen with M-x shell.

* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result.  The result 
is "bar" for me.

The exact "expected behavior" for a shell block is a little fuzzy.  According 
to my analysis (given below), what Alain reports (remote file and no "bar") is 
the "expected" behavior.  What I see (no remote file and "bar") is actually 
"unexpected".

I used the following to test the claim:

#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar"
#+end_src

I am unable to reproduce the reported behavior (of "bar" not returning).  
Instead, I get an ssh-askpass permission denied error, foo_file is not created, 
and "bar" is given as the result.  I do not see anywhere in the thread that the 
original claim was reproduced.

The thread preceded something like follows.

Leo Butler suggested two work arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Russell Adams suggested another work around:

- add -n to the ssh command

Ihor identified that a non-session call does something like the following 
command:

bash -c bash /tmp/temp-file-with-source-block-code.sh

where

- /tmp/temp-file-with-source-block-code -
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
-

The second line (significantly different from the original report) pipes the 
echo result to stdout and to a file, bar.txt.  Writing to a file allows us to 
confirm if that line was executed.

Ihor suggested that

bash -c bash /tmp/temp-file-with-source-block-code.sh

does not run the second line because an interactive password prompt is 
displayed by ssh.  The reasoning is that the prompt hangs the process while 
waiting for input and the second line never runs.  Indeed, running the command 
does not produce /tmp/bar.txt.

Ihor is correct about prompts messing with shell blocks (this is not the first 
time he's seen this).  However, the way it's stated does not demonstrate it. 
This is because Emacs does *not* make a call like

bash -c bash /tmp/temp-file-with-source-block-code.sh

Alain responded by pointing out that

bash -c bash /tmp/temp-file-with-source-block-code.sh

does not execute the first line.  This is true.  Consider calling

bash -c bash /tmp/two-lines.sh

where

-- /tmp/two-lines.sh --
echo "first" > /tmp/first.txt
echo "second" > /tmm/second.txt
---

Neither first.txt or second.txt are created.

Max Nikulin interjected with a helpful reminder that Bash scripting is a 
snakepit of footguns.  (What Max said is more than that and interesting.  I 
skip it here because it depends on the form of the call.)

Before trying to untangle what a given Bash command does, we need to be sure 
what command is actually called.  Unfortunately, there's not a clear Bash 
command corresponding to how Emacs makes the call.

What happens goes something like this:

1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path 
to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object 
implemented in C
3. DEFUN ("call-process") forwards this information to the C function 
call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess

A lot of cleaning and setup happens which is dependent on the system (GNU, 
Window, Darwin, etc.).  There's a call to openp which looks for the executable. 
 An emacs_pipe is set up 

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt, Ihor, Alain,

Ihor Radchenko  writes:

> alain.coch...@unistra.fr writes:
>
>> At the most basic user level (i.e., non lisp aware), why is it not
>> necessarily a bug if "something" does the expected in an X terminal
>> but not in an emacs terminal?  I think Matt and I (and others) are on
>> the same page here, and he already essentially drafted the bug report
>> I would have sent (only better).
>
> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
> the problem with M-x shell is not the problem you originally ran to. It
> is a different problem (also, we ran into it in the past).

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


>> What also confuses me is that it seems to me that, in the minimum
>> working example, you consider the ssh command and the read command as
>> equivalent.  But I don't even enter the password when using ssh...

I've introduced the 'read' example, as a simpler way to modify the
standard input and get the same kind of "unexpected" results, without
relying on SSH.  It seems that I have only caused confusion, sorry about
that.

IIUC, the OP example is not working because SSH is modifying the
standard input (with or without passwords).

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.
>

I'm not the OP, but, my SSH is configured to work without passwords and
SSH is still consuming lines from standard input:

#+begin_src bash :results output
ssh phone echo "remote"
echo "local"
#+end_src

#+RESULTS:
: remote

It looks like it is a known SSH "feature" (see
https://unix.stackexchange.com/a/688024):

#+begin_src bash :results output
seq 100 | (ssh phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 675173

Same block, but asking SSH to not use stdin (using '-n' as mentionned in
this thread):

#+begin_src bash :results output
seq 100 | (ssh -n phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 100

IMHO, what ob-shell is doing today seems a valid way of evaluating
source blocks (and it seems to have been like that for a long time).  It
should probably be documented somewhere, so that users know how to write
their source blocks, or switch to another way, like adding a :cmdline
parameter as mentionned in this thread.


Hoping I didn't increase the confusion again,
:-)


Bruno




> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Fri, Nov 17 2023, Ihor Radchenko  wrote:

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.

I think you are confused, Ihor. The bug was confirmed here:

https://list.orgmode.org/orgmode/87bkcmlor9@t14.reltub.ca/

I mis-identified the culprit; you corrected me by pointing out the bug
is in `process-file'. And you confirmed the bug:

https://list.orgmode.org/orgmode/87fs1xbis1.fsf@localhost/

Russell Adams and I suggested ssh-specific work-arounds that prevent ssh
from grabbing stdin (either -n or -f).

Leo

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Fri 17 Nov 2023 09:22:
>
>  > I think that I need to clarify here.
>
> Thank you.
>
> Well, thank you for _trying_: most of the discussion in this thread is
> way beyond my pay grade.

My reply was to Matthew, the current ob-shell maintainer.
If you want, you need to examine ob-shell (`org-babel-sh-evaluate') and
ob-eval (`org-babel-eval') code, and try to step through the code using
debugger to understand technical details in thread better.

> At the most basic user level (i.e., non lisp aware), why is it not
> necessarily a bug if "something" does the expected in an X terminal
> but not in an emacs terminal?  I think Matt and I (and others) are on
> the same page here, and he already essentially drafted the bug report
> I would have sent (only better).

WRT M-x shell, feel free to submit a bug report. I mostly pointed that
the problem with M-x shell is not the problem you originally ran to. It
is a different problem (also, we ran into it in the past).

> What also confuses me is that it seems to me that, in the minimum
> working example, you consider the ssh command and the read command as
> equivalent.  But I don't even enter the password when using ssh...

I was only able to reproduce your problem with ssh asking a password.
We are discussing the reproduced case.

If you see problems with
   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src
even when ssh does not ask for a password, please provide more detailed
reproducer that we can replicate locally without guessing your ssh config.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Ihor Radchenko writes on Fri 17 Nov 2023 09:22:

 > I think that I need to clarify here.

Thank you.

Well, thank you for _trying_: most of the discussion in this thread is
way beyond my pay grade.

At the most basic user level (i.e., non lisp aware), why is it not
necessarily a bug if "something" does the expected in an X terminal
but not in an emacs terminal?  I think Matt and I (and others) are on
the same page here, and he already essentially drafted the bug report
I would have sent (only better).

What also confuses me is that it seems to me that, in the minimum
working example, you consider the ssh command and the read command as
equivalent.  But I don't even enter the password when using ssh...


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt  writes:

> Thank you for clarifying.
>
>  > Anyway, this gives me the opportunity to come back to the question of
>  > whether or not there is a problem with emacs itself (like some people
>  > here thought), and if some message should be sent to some emacs list.
>  > Again my argument was that the 2 commands copy/yank'ed in an emacs
>  > terminal don't work as expected, while they do if the same is done in
>  > an X terminal.
>
> Okay, I follow you now.
>
> Yes, I agree with what others have said, it's related to Emacs (probably 
> comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's 
> hard to reproduce with M-x term because term-char-mode doesn't allow 
> copy-paste.  M-x with term-line-mode doesn' t print "bar".

I think that I need to clarify here.
We are talking about two different things:
1. Bash src block without session
2. Bash src block with session

The original report used bash src blocks _without_ session.
In such scenario, comint (and M-x shell) is not relevant.
Org simply uses `process-file' with INFILE argument.
This is equivalent of someone opening __X shell__, and literally typing

  ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
  echo "bar"

In the scenario described in the report, when ssh asks for password, it
is equivalent to

$ ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
Password: 

This has nothing to do with Emacs comint and this is also not a bug in
Emacs - we use INFILE argument, that is equivalent to the above as _per
docstring_. So, Org mode is simply not using `process-file' function as
users expect - instead of executing a bash script, we emulate
interactive user input to bash.

(Side note: for someone aware about these details, it is possible to do
something like

#+begin_src bash
ssh remote-server;
<>
#+end_src

or, as a demo

#+begin_src bash
read x;
value2
echo "We just read \"$x\"";
#+end_src

#+RESULTS:
: We just read "value2"

)

Another case is (2), when we do use comint is session is what we
discussed in
https://list.orgmode.org/orgmode/cal1eyujntgbxy6a794qm7ptbxh3diu1af6oays7clf3kobs...@mail.gmail.com/
I guess we can try to report this as a bug, especially since it also
manifests itself when comint is used interactively.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

alaincochard alain.coch...@unistra.fr writes:

 > Yes.  (So I think that explains the 'no' above.) 

Thank you for clarifying.

 > Anyway, this gives me the opportunity to come back to the question of
 > whether or not there is a problem with emacs itself (like some people
 > here thought), and if some message should be sent to some emacs list.
 > Again my argument was that the 2 commands copy/yank'ed in an emacs
 > terminal don't work as expected, while they do if the same is done in
 > an X terminal.

Okay, I follow you now.

Yes, I agree with what others have said, it's related to Emacs (probably 
comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's hard 
to reproduce with M-x term because term-char-mode doesn't allow copy-paste.  
M-x with term-line-mode doesn' t print "bar".

 > I guess it is not important for Org, because workarounds have be
 > proposed, but I think if there is a problem with emacs it should be
 > reported, right?

I think it's worth reporting.  I'm headed to bed and can submit it tomorrow for 
you if you're not comfortable with doing it or don't have the time.

Here are the steps I followed to reproduce with M-x shell:

1. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

2. Paste them into a shell (tested with xterm or xfce4-terminal) and press 
return
3. After entering the correct password, "bar" is echoed
4. Paste them into the buffer associated with M-x shell
5. After entering the correct password, "bar" is NOT echoed

Here are the steps I followed to reproduce with M-x term:

1. M-x term
2. C-c C-j to switch to term-line-mode
3. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

4. Paste them into the buffer associated with M-x term
5. After entering the correct password, "bar" is NOT echoed

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Matt writes on Thu 16 Nov 2023 20:03:
 > 
 >   On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 
 > 
 >  > Or we can make use of TRAMP as we usually do to access remote
 >  > environment. IMHO, it is more reliable as TRAMP takes care about
 >  > arranging all the Elisp FS interaction to work on remote system.
 >  
 > Agreed.  Also, I wasn't aware TRAMP worked with shell block
 > evaluation.  That's nice.
 > 
 > Sure enough, these both work:
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user 
 > :session *remote*
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > Of course, both calls happen on the remote.
 > 
 > I wonder, would this have helped Alain?

No (see below).

 > I'm admittedly a little confused about the original report,  
 > 
 > #+name: original-report
 > #+begin_src bash :results output
 > ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
 > echo "bar"
 > #+end_src
 > 
 > Clearly, the ssh line is intended to happen on the remote server.
 > Was the echo "bar" intended to happen locally?

Yes.  (So I think that explains the 'no' above.)  Then I'm confused
about your confusion...

Anyway, this gives me the opportunity to come back to the question of
whether or not there is a problem with emacs itself (like some people
here thought), and if some message should be sent to some emacs list.
Again my argument was that the 2 commands copy/yank'ed in an emacs
terminal don't work as expected, while they do if the same is done in
an X terminal.

I guess it is not important for Org, because workarounds have be
proposed, but I think if there is a problem with emacs it should be
reported, right?

Thank you.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Matt]

  On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 

 > Or we can make use of TRAMP as we usually do to access remote
 > environment. IMHO, it is more reliable as TRAMP takes care about
 > arranging all the Elisp FS interaction to work on remote system.
 
Agreed.  Also, I wasn't aware TRAMP worked with shell block evaluation.  That's 
nice.

Sure enough, these both work:

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session 
*remote*
echo foo>foo_file
echo "bar"
#+end_src

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user
echo foo>foo_file
echo "bar"
#+end_src

Of course, both calls happen on the remote.

I wonder, would this have helped Alain?

I'm admittedly a little confused about the original report,  

#+name: original-report
#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src

Clearly, the ssh line is intended to happen on the remote server.  Was the echo 
"bar" intended to happen locally?

PS:  I've updated the WORG page for ob-shell with a :dir section 
(https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54)

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt  writes:

> So, rather than start a comint and then ssh from there, it's possible to let 
> Emacs start the subprocess, manage the ssh connection, and just read that.  
> Emacs makes a buffer from a network stream by calling start-process` using 
> `shell-file-name`.  `org-babel-shell-initialize` closes around 
> `shell-file-name` with whatever shell language is used.  Therefore, we could 
> provide header arguments to pass parameters and the destination to ssh.  

Or we can make use of TRAMP as we usually do to access remote
environment. IMHO, it is more reliable as TRAMP takes care about
arranging all the Elisp FS interaction to work on remote system.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Matt]

  On Wed, 15 Nov 2023 17:32:19 +0100  Matt  wrote --- 

 > (let ((buff "*localhost-process-buffer*"))
 >   (switch-to-buffer
 >(make-comint-in-buffer
 > "localhost-process"
 > buff
 > "ssh"
 > nil
 > (format "%s@localhost" (getenv "USER")

It looks like the way this is called actually uses `start-file-process` rather 
than `open-network-stream`.  It's not clear to me how to call it so that 
`open-network-stream` is used.  However, that's more a question for the Emacs 
mailing list and, honestly, not one that I think I care about right now.  
Posting this just to correct what I said before.

Re: bash source code block: problem after ssh commands

[Matt]

I was poking around, learning how sessions are started.  Basically, `shell` 
creates the comint buffer using `make-comint-in-buffer`.  What I find 
interesting is that `make-comint-in-buffer` can also create a comint buffer 
from a network stream:

(let ((buff "*localhost-process-buffer*"))
  (switch-to-buffer
   (make-comint-in-buffer
"localhost-process"
buff
"ssh"
nil
(format "%s@localhost" (getenv "USER")

So, rather than start a comint and then ssh from there, it's possible to let 
Emacs start the subprocess, manage the ssh connection, and just read that.  
Emacs makes a buffer from a network stream by calling start-process` using 
`shell-file-name`.  `org-babel-shell-initialize` closes around 
`shell-file-name` with whatever shell language is used.  Therefore, we could 
provide header arguments to pass parameters and the destination to ssh.  

I haven't made a judgment yet about whether any of this is good or bad.   I 
thought it was interesting and figured I'd share.

Re: bash source code block: problem after ssh commands

[Matt]

  On Thu, 09 Nov 2023 13:13:36 +0100  Ihor Radchenko  wrote --- 

 > A bigger problem is that errors *Org Babel Error
 > Output* are not clickable:
 
Interesting.

 > So, it looks like we may need an alternative `org-babel-eval' function
 > that works with script files rather than input and arranges
 > stdout/stderr separation.

Yes, it seems like a separate handler may be in order.

 > We should reuse `process-file' in this scenario as well,
 > but utilizing its ARGS argument + arrange to generate the temporary
 > script file in the appropriate context - see `make-nearby-temp-file'.

I was unaware of `make-nearby-temp-file'.  Thank you.  It's helpful for this 
situation.

I'm not in a position to address these points right now.  My main focus is 
re-familiarizing myself with the =ob-shell= and =ob-comint= source.  When I was 
implementing async shell evaluatation earlier this year, I remember having the 
impression that execution paths, especially `org-babel-comint-with-output', 
could probably be simplified.  It seems to me that everything we're talking 
about in this thread aligns with that objective.  I've made a note of your 
points and have every intention to return to them when I get to writing code.  
However, don't let me stop you if this is something you were interested in 
doing.  I'll do what I can to assist.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt  writes:

> ... Aside from permission, any code we
> insert needs to be correct. For example, a shebang would need to point
> to the correct application and any arguments would need to correspond
> to the implementation being called. I doubt we'd need anything beyond
> /bin/. FWIW, it looks like there's been at least one instance
> where :shebang's formatting was questioned
> (https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/).

It is a problem, but I am not sure how useful it is to solve the line
numbering there. A bigger problem is that errors *Org Babel Error
Output* are not clickable:

Try
#+begin_src bash
cdf
#+end_src
and you will get
-- *Org Babel Error Output* -
bash: line 1: cdf: command not found
[ Babel evaluation exited with code 127 ]
-

Clicking on "line 1" will yield file opening dialogue.

However, there is yet another problem revealed when we try

#+begin_src bash :cmdline bash
cdf
#+end_src

#+RESULTS:
: /tmp/babel-47DarV/sh-script-vkbRfJ: line 2: cdf: command not found

There is no *Org Babel Error Output* buffer at all.

So, it looks like we may need an alternative `org-babel-eval' function
that works with script files rather than input and arranges
stdout/stderr separation.

> We'd also need to control for what environment the script runs in.
> That was another issue I saw raised in the list
> (https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/). Of
> course, we'd need to read the stdout and stderr. This is handled by
> =process-file=. Any step I missed or some kind of failure I didn't
> consider?

Valid concern. We should reuse `process-file' in this scenario as well,
but utilizing its ARGS argument + arrange to generate the temporary
script file in the appropriate context - see `make-nearby-temp-file'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

  On Tue, 07 Nov 2023 09:53:46 +0100  Ihor Radchenko  wrote --- 
 > Matt m...@excalamus.com> writes:
 > 
 > >   On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > >  > I am wondering about the possible downsides of using script approach
 > >  > instead of stdin redirection.
 > >  
 > > I'm curious to hear more about what you're thinking.
 > 
 > I am thinking to change the
 >   (t (org-babel-eval shell-file-name (org-trim body)))
 > clause in `org-babel-sh-evaluate' to something that uses a script file.
 > 
 > It will clearly solve the discussed problem, possibly at the cost of
 > small overhead to write the script file to disk.
 
Thanks for clarifying.  I've been away from the codebase for a bit and, now 
that the FSF paperwork is signed (still need to get Craig a copy), I'm 
reviewing =ob-shell= with an eye for what could be cleaned or improved.   I 
feel like =org-babel-sh-evaluate= could use some attention.

I'm open to your suggestion.  The good is that using a script is how :shebang 
and :cmdline are processed currently (like you pointed out) so there's 
precedence and experience with it.  Also, it would make all non-session 
execution use the same model (script versus comint).  I like how that would 
create a clean separation.

For bad, nothing jumps out to me as obviously a problem.  Let me "think out 
loud" for a moment.  We'd need to write to disk.  Like you say, this incurs 
overhead opening, writing, and closing the file.  It's not like we'd forget to 
close it, though.  Nor is running out of space or inodes our problem.  Writing 
requires permission.  That's not an issue with /tmp.  Then, it needs to 
execute.  Aside from permission, any code we insert needs to be correct.  For 
example, a shebang would need to point to the correct application and any 
arguments would need to correspond to the implementation being called.  I doubt 
we'd need anything beyond /bin/.   FWIW, it looks like there's been at 
least one instance where :shebang's formatting was questioned 
(https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/).
  We'd also need to control for what environment the script runs in.  That was 
another issue I saw raised in the list 
(https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/).  Of course, 
we'd need to read the stdout and stderr.  This is handled by =process-file=.  
Any step I missed or some kind of failure I didn't consider?

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt  writes:

>   On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
>  > I am wondering about the possible downsides of using script approach
>  > instead of stdin redirection.
>  
> I'm curious to hear more about what you're thinking.

I am thinking to change the
  (t (org-babel-eval shell-file-name (org-trim body)))
clause in `org-babel-sh-evaluate' to something that uses a script file.

It will clearly solve the discussed problem, possibly at the cost of
small overhead to write the script file to disk.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Matt writes on Mon  6 Nov 2023 19:01:

 > I want to confirm, are you able to accomplish your task by using
 > '-n'?

Yes.  Evaluating this block (with C-c C-c):

   #+begin_src bash :results output
   ssh -n coch...@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src

gives

   #+RESULTS:
   : bar

(without the '-n', #+RESULTS: is empty).  Same conclusion if I
copy/yank the 2 bash instructions in an emacs terminal (M-x shell). 

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Matt]

  On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > I am wondering about the possible downsides of using script approach
 > instead of stdin redirection.
 
I'm curious to hear more about what you're thinking.

Re: bash source code block: problem after ssh commands

[Matt]

  On Fri, 27 Oct 2023 13:47:24 +0200  alain.coch...@unistra.fr wrote --- 
 > Russell Adams writes on Thu 26 Oct 2023 16:44:
 >  > [...] are you using an SSH key to connect, or entering a password?
 >  > If you use an SSH key for passwordless access, try adding -n (ie:
 > 
 >  > "ssh -n derp@host mycommand"). 
 > 
 > Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.
 
Hi Alain,

I'm the supposed maintainer of ob-shell who's been missing in action for this 
whole discussion.

I want to confirm, are you able to accomplish your task by using '-n'?

Matt

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Bruno Barbier  writes:

> IIUC, the elisp expression:
>
> (process-file "bash" "/tmp/test.sh")
>
> is more equivalent to:
>
> cat /tmp/test.sh | bash
>
> i.e. the shell is getting the commands from stdin.  Thus, any command
> that uses stdin might change what gets executed or not.

Looking at `org-babel-sh-evaluate', it should be enough to specify
:shebang, :cmdline, or :stdin header argument to force using script
rather than channel source block as input to bash:

#+begin_src shell :results output :cmdline bash
exec 0>&-
echo OK
#+end_src

#+RESULTS:
: OK


#+begin_src shell :results output :cmdline bash
echo 1
read -p "Next command? " NEXT_COMMAND
echo 2
echo 3
#+end_src

#+RESULTS:
: 1
: 2
: 3

I am wondering about the possible downsides of using script approach
instead of stdin redirection.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi,

alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 13:44:
>
>  > I can now reproduce the problem locally.
>  > 
>  > It boils down to
>  > 
>  > (setq exit-status
>  >  (process-file shell-file-name input-file
>  >(if error-file
>  >(list t error-file)
>  >  t)
>  >nil shell-command-switch command))
>  > 
>  > that is an equivalent of
>  > 
>  > bash -c bash /path/to/file-containing-the-source-code.sh
>  > 
...
> I am confused about what you specifically do to "evaluate the above".
> To start with, I have to use quotes to make your command be performed:
>
>bash -c "bash /path/to/file-containing-the-source-code.sh"
>
...
>
> But most importantly, the second line *is* produced, either if I use
> an SSH key for passwordless access or if I enter the password
> manually.

IIUC, the elisp expression:

(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash

i.e. the shell is getting the commands from stdin.  Thus, any command
that uses stdin might change what gets executed or not.

I'm able to reproduce using the following minimal script, without
passwords nor SSH (where the cryptic first line closes stdin).

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

The result is "OK" only when commenting out the first line; else, the
echo command is not executed (because stdin has been closed).

Here is an other example, where the second echo is eaten by the script
itself:

#+begin_src shell :results output
  echo 1
  read -p "Next command? " NEXT_COMMAND
  echo 2
  echo 3
#+end_src

#+RESULTS:
: 1
: 3

Bruno

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 28/10/2023 01:26, alain.coch...@unistra.fr wrote:

bash -c "bash /path/to/file-containing-the-source-code.sh"

Without the quotes, the 1st line is not executed either.  NB: I get
the same behavior with simply


Without quotes it means: execute shell script (just "bash" this case) 
with "$0" set to "/path/to/file-containing-the-source-code.sh". 
"/path/..." is ignored because shell script does not contain "$0". It is 
a way to safely pass arguments to shell, however "$0" is confusing:


sh -c 'echo "arg0: $0"; echo  args: "$@"' arg0 arg1 arg2
arg0: arg0
args: arg1 arg2

In this example "arg0: $0"; echo  args: "$@"'  has the same role as 
"bash" in the cited command.


If you need to pass another variable into "sh -c" the use

var="/bin/sh"
sh -c 'ls -l "$1"' argv0-for-sh "$var"

Unsafe variant ("var" contain quotes): sh -c "ls -l '$var'". The same 
warning is applicable to "emacs --eval "(func \"$var\")".



or, after making the script executable with 'chmod +x':

/path/to/file-containing-the-source-code.sh


Always add a shebang ("#!/bin/sh", "#!/bin/bash", etc.) to executable 
scripts. A chance to face a bug is a shell is higher when it is left up 
to the shell and the kernel what to do with files without shebangs.


Sorry, I have not tried to debug the original issue.

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Ihor Radchenko writes on Thu 26 Oct 2023 13:44:

 > I can now reproduce the problem locally.
 > 
 > It boils down to
 > 
 > (setq exit-status
 >(process-file shell-file-name input-file
 >  (if error-file
 >  (list t error-file)
 >t)
 >  nil shell-command-switch command))
 > 
 > that is an equivalent of
 > 
 > bash -c bash /path/to/file-containing-the-source-code.sh
 > 
 > - file-containing-the-source-code.sh 
 >  ssh localhost 'echo foo>/tmp/foo_file'
 >   echo $(uname -a) |tee /tmp/uname1.txt
 > -

 > If one tries to evaluate the above, the second line is not produced. It
 > has nothing to do with Emacs itself.

I am confused about what you specifically do to "evaluate the above".
To start with, I have to use quotes to make your command be performed:

   bash -c "bash /path/to/file-containing-the-source-code.sh"

Without the quotes, the 1st line is not executed either.  NB: I get
the same behavior with simply

   bash /path/to/file-containing-the-source-code.sh

or, after making the script executable with 'chmod +x':

   /path/to/file-containing-the-source-code.sh

But most importantly, the second line *is* produced, either if I use
an SSH key for passwordless access or if I enter the password
manually.

 > However, if I try
 > 
 > (process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
 > produced.

Here too, it is not clear to me how you "try".  The way I know is to
highlight

   (process-file "bash" "/path/to/file-containing-the-source-code.sh")

and do

   M-x eval-region

In this case, indeed, the 1st line is executed but not the 2nd one
(again, whether or not I use an SSH key).


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Russell Adams writes on Thu 26 Oct 2023 16:44:

 > [...] are you using an SSH key to connect, or entering a password?

 > If you use an SSH key for passwordless access, try adding -n (ie:
 > "ssh -n derp@host mycommand"). 

Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Russell Adams]

On Wed, Oct 25, 2023 at 01:17:42PM +0200, alain.coch...@unistra.fr wrote:
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src

I know that Ihor has already reproduced, but are you using an SSH key
to connect, or entering a password?

If entering a password, I'd expect there is some mixup. That I defer
to Ihor.

If you use an SSH key for passwordless access, try adding -n (ie: "ssh
-n derp@host mycommand"). The "-n" flag helps prevent the remote
command from interfering with your local terminal by redirecting
/dev/null as the remote stdin. I have to use this commonly in systems
administration, scripts, and tools like Ansible.

Thanks.

--
Russell Adamsrlad...@adamsinfoserv.com
https://www.adamsinfoserv.com/

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 08:44:
>
>  > I currently do not have a setup to test ssh commands,
>
> Sorry if that's irrelevant: I realized that one can ssh to the *same*
> machine.

You are indeed right.
I can now reproduce the problem locally.

It boils down to

(setq exit-status
  (process-file shell-file-name input-file
(if error-file
(list t error-file)
  t)
nil shell-command-switch command))

that is an equivalent of

bash -c bash /path/to/file-containing-the-source-code.sh

- file-containing-the-source-code.sh 
 ssh localhost 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
-

If one tries to evaluate the above, the second line is not produced. It
has nothing to do with Emacs itself.

However, if I try

(process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
produced.

This time, it is Emacs problem.
The problem is caused by interactive password prompt displayed by ssh.

I guess we may try to report this upstream, though I am not sure if this
is something that is supported to start with.

>From Org perspective, it is not the first time when interactive shell
programs are causing problems. I'd say that we have no technical ability
to support them reliably and one has to use ob-screen or similar
libraries to properly interact with such code blocks.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Ihor Radchenko writes on Thu 26 Oct 2023 08:44:

 > I currently do not have a setup to test ssh commands,

Sorry if that's irrelevant: I realized that one can ssh to the *same*
machine.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Leo Butler  writes:

> It looks like an issue in comint-mode and the way that it is handling
> the temporary ssh session. This is not an issue with org-mode.
> ...
> Not working:
>
> #+begin_src bash :results output
>   ssh SSH-HOST 'echo foo>/tmp/foo_file'
>   echo $(uname -a) |tee /tmp/uname1.txt
> #+end_src

By default (no session), the above command should not use comint-mode.
So, something else might be going on here.

I currently do not have a setup to test ssh commands, so I would
appreciate someone trying to dig why `org-babel-eval' apparently not
returning the expected result.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bash source code block: problem after ssh commands

[yaxp]

So it is an issue with emacs, right?


comint-mode

--
(yaxp me) => t

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Wed, Oct 25 2023, alain.coch...@unistra.fr wrote:

> Thanks much for the detailed explanations.
>
>  > It looks like an issue in comint-mode and the way that it is
>  > handling the temporary ssh session. This is not an issue with
>  > org-mode.
>
> So it is an issue with emacs, right?

Yes, that is my understanding. Perhaps you can report it to the
emacs-devel mailing list.

Leo

Re: bash source code block: problem after ssh commands

[Alain . Cochard]

Thanks much for the detailed explanations.

 > It looks like an issue in comint-mode and the way that it is
 > handling the temporary ssh session. This is not an issue with
 > org-mode.

So it is an issue with emacs, right?

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Wed, Oct 25 2023, alain.coch...@unistra.fr wrote:

> Hello.
>
> For me, 'C-c C-c' on the following group
>
>#+begin_src bash :results output
>echo "foo"
>echo "bar"
>#+end_src
>
>
> works as I expect: 'foo' and 'bar' are echo'ed.  But it fails on this
> one:
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
>
> The file 'foo_file' is created on the remote machine, but 'bar' is not
> echo'ed.  I have tried to insert other commands between the ssh and
> echo commands, or append semicolons at the end of the commands,
> without success.
>
> By contrast, it works with this one:
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
>#+end_src
>
> Perhaps it is worth noting that I observe the same behavior if I
> simply copy/yank the commands (I mean: the two commands yank'ed at
> once) in an emacs terminal (obtained with 'M-x shell').  However,
> everything works as I expect if I copy/yank the commands (all at once)
> in a "regular" X terminal (xfce in my case).

It looks like an issue in comint-mode and the way that it is handling
the temporary ssh session. This is not an issue with org-mode.

Here are a couple work-arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Working examples:

#+begin_src bash :results output
  ssh -f SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) 
x86_64 GNU/Linux

#+begin_src bash :results output
  ssh pearce-120 'echo foo>/tmp/foo_file' &&\
  echo $(uname -a) | tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) 
x86_64 GNU/Linux

Not working:

#+begin_src bash :results output
  ssh SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
#+end_src

If you look at /tmp after running the third example, you see that that
second command line has not been executed.

Leo

bash source code block: problem after ssh commands

[Alain . Cochard]

Hello.

For me, 'C-c C-c' on the following group

   #+begin_src bash :results output
   echo "foo"
   echo "bar"
   #+end_src

works as I expect: 'foo' and 'bar' are echo'ed.  But it fails on this
one:

   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src

The file 'foo_file' is created on the remote machine, but 'bar' is not
echo'ed.  I have tried to insert other commands between the ssh and
echo commands, or append semicolons at the end of the commands,
without success.

By contrast, it works with this one:

   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
   #+end_src

Perhaps it is worth noting that I observe the same behavior if I
simply copy/yank the commands (I mean: the two commands yank'ed at
once) in an emacs terminal (obtained with 'M-x shell').  However,
everything works as I expect if I copy/yank the commands (all at once)
in a "regular" X terminal (xfce in my case).

I wish I could understand what is going on.

Thanks

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]

Re: [BUG] HTML export locale problem [9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/)]

[Ihor Radchenko]

em...@supporter.mailer.me writes:

>  Today I was exporting my org file to HTML and I have run into an issue 
>  with #+LANGUAGE. 
>  I have set it to to "pl" and the Table of Contents name has changed, but 
>  the figure names did not. 
>  
>  Can somebody guide me on how could I fix it? 

This is because we do not have all the terms translated to Polish.
I invite you to look into `org-export-dictionary' variable and submit a
patch with the missing translations. Or just an email, listing Polish
names for untranslated terms.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[BUG] HTML export locale problem [9.6.6 (release_9.6.6 @ /usr/share/emacs/29.1/lisp/org/)]

[emacs]

Hello everyone, 
 
 Today I was exporting my org file to HTML and I have run into an issue 
 with #+LANGUAGE. 
 I have set it to to "pl" and the Table of Contents name has changed, but 
 the figure names did not. 
 
 Can somebody guide me on how could I fix it? 
 
 Kind regards 
Emacs User

 Info: 
 Emacs  : GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38, 
cairo version 1.17.8)
 Package: Org mode version 9.6.6 (release_9.6.6 @ 
/usr/share/emacs/29.1/lisp/org/)

Re: problem with org-map-entries and the doc for export hooks

[Victor A. Stoichita]

Le 08 Jun 2023, Ihor Radchenko  a écrit :

> "Victor A. Stoichita"  writes:
>
>> I still see the problem with org version 9.6.6 and your latest example
>> code. 
>>
>> I also checked with emacs -Q which has org version 9.6.1.
>>
>> I’m testing this on the attached file.
>
> Thanks for providing an example!
> You encountered a bug.
> Fixed, on bugfix.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=84e5a4da5

Great! Thank you!

Re: problem with org-map-entries and the doc for export hooks

[Ihor Radchenko]

"Victor A. Stoichita"  writes:

> I still see the problem with org version 9.6.6 and your latest example
> code. 
>
> I also checked with emacs -Q which has org version 9.6.1.
>
> I’m testing this on the attached file.

Thanks for providing an example!
You encountered a bug.
Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=84e5a4da5

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: problem with org-map-entries and the doc for export hooks

[Victor A. Stoichita]

Le 07 Jun 2023, Ihor Radchenko  a écrit :

> 1. Update Org mode. I do not see any hangs with the latest stable
> Org release.
> 2. Use alternative example. The current example may not work if you have
>headings not separated by contents or empty lines.
>See 
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=16f15f966

Thanks for the hints Ihor !

I still see the problem with org version 9.6.6 and your latest example
code. 

I also checked with emacs -Q which has org version 9.6.1.

I’m testing this on the attached file.

I also see that if I set the export scope to "subtree" instead of
"buffer" the process runs as expected.

Cheers,
Victor


simple-org-file.org
Description: application/vnd

Re: problem with org-map-entries and the doc for export hooks

[Ihor Radchenko]

"Victor A. Stoichita"  writes:

> I’m trying to use the example given here in the manual for "Export
> hooks": https://orgmode.org/manual/Advanced-Export-Configuration.html
> ...
> However the export process hangs indefinitely emacs after evaluating this.
> I have to C-g to quit. 
> The same happens with the newer org-export-before-parsing-functions
> instead of org-export-before-parsing-hook.
>
> Is there something else that needs to be updated for the example to work?

1. Update Org mode. I do not see any hangs with the latest stable Org release.
2. Use alternative example. The current example may not work if you have
   headings not separated by contents or empty lines.
   See https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=16f15f966

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

problem with org-map-entries and the doc for export hooks

[Victor A. Stoichita]

Greetings all!

I’m trying to use the example given here in the manual for "Export
hooks": https://orgmode.org/manual/Advanced-Export-Configuration.html

--8<---cut here---start->8---
(defun my-headline-removal (backend)
  "Remove all headlines in the current buffer.
BACKEND is the export back-end being used, as a symbol."
  (org-map-entries
   (lambda () (delete-region (point) (line-beginning-position 2)

(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
--8<---cut here---end--->8---

However the export process hangs indefinitely emacs after evaluating this.
I have to C-g to quit. 
The same happens with the newer org-export-before-parsing-functions
instead of org-export-before-parsing-hook.

Is there something else that needs to be updated for the example to work?

Regards,
Victor

Re: bug#59882: Multiple versions of Org in load-path problem

[Tom Gillespie]

> The state of our unloading support is not good enough to use it blindly

I have managed to use unload-feature to reload a new version of org
after the builtin version has been previously loaded, but the dance you
have to do to avoid nasty errors (and segfaults if using native comp!)
is about 100 lines of arcane footwork, and if any one step is off you
will encounter one of the many hidden landmines.

unload-feature cannot be used blindly, and there are a very large
number of side effects along the path that are not controlled which
makes any process that tries to use unload-feature brittle and hard
to debug.

https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1010-L1064
https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1073-L1177

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> I have noticed that neither `package--reload-previously-loaded` nor code
> loading already compiled package unload the loaddefs file. E.g. I added

The state of our unloading support is not good enough to use it blindly,
indeed: it's too easy/common to have packages where `unload` leaves you
with an Emacs that has serious problems (including being unusable).

So, instead we limit ourselves to force-reloading files which tends to
be much more harmless (tho in theory of course it's just as bad).
It also tends to be sufficient.


Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 18/12/2022 22:40, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:

- "emacs -l org" to simulate a regular user session with some opened files.
- Update to Org-9.6 from ELPA.
- Close emacs and start it again "emacs -l org"

Result:
byte-code: Invalid function: org-assert-version


Hmm... there's code in `package.el` intended to avoid that problem, but
clearly it's failing here.  Could Someone™ investigate why that code
doesn't do its job.

The code in question is in `package--reload-previously-loaded`


Since I managed to reproduce the issue with emacs-28.2 built from source 
tree, I tried current master (30.0.50) as well. I used the following 
trick to shadow org-assert-version:


sed -i -e 's/org-assert-version/org-assert-version-old/g' lisp/org/*.el

I am unable to reproduce the "Invalid function: org-assert-version" 
issue. Org compilation finishes without warnings.


I have noticed that neither `package--reload-previously-loaded` nor code 
loading already compiled package unload the loaddefs file. E.g. I added 
org-stale-function marked as autoload. It is possible to get help for 
this function, but attempts to call this function causes "Autoloading 
file" error. Ideally emacs should completely forget about such functions 
when newer elpa package is installed.

Re: problem interaction between jinx-mode and org mode

[Fraga, Eric]

Hello Ihor,

On Thursday,  6 Apr 2023 at 12:04, Ihor Radchenko wrote:
> This is one long paragraph containing ~..., which is matching a possible
> ~code~ markup. org-element-code-parser tries to search the matching the
> closing "~", running regexp search until the end of the paragraph.

Thank you for confirming my guess.  Knowing this, I can work around this
in a number of ways but it did catch me by surprise!

Thanks for getting back quickly.

eric

-- 
: Eric S Fraga, with org release_9.6.1-341-g614b96 in Emacs 30.0.50

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 03/02/2023 18:02, Ihor Radchenko wrote:


The problem seems to be real and appears to be some combination of
Debian/Ubuntu + Emacs.


It is not a Debian packaging issue. I have reproduced the "invalid 
function `org-assert-version'" bug using emacs built from emacs-28.2 git 
tag. It is important to execute "make install" and to launch emacs from 
from that directory.


I have no idea why behavior is different when emacs is launched from the 
build tree.


Environment: Debian-12 bookworm (frozen testing).

There is no problem when emacs is built by just "make" from debian 
sources with all patches applied. I admit that this experiment was not 
really pure since I did not use configure options from the 
debian/control file.

Re: problem interaction between jinx-mode and org mode

[Ihor Radchenko]

"Fraga, Eric"  writes:

> --8<---cut here---start->8---
> ┌ Debug: Placing hot node
> │   node = n29
> └ @ HeatExchangerNetworkDesign
> ~/s/research/julia/HeatExchangerNetworkDesign/src/HeatExchangerNetworkDesign.jl:1102
> ┌ Debug: Layout for node
> │   node = n18
> │   y = 6
> └ @ HeatExchangerNetworkDesign 
> --8<---cut here---end--->8---

This is one long paragraph containing ~..., which is matching a possible
~code~ markup. org-element-code-parser tries to search the matching the
closing "~", running regexp search until the end of the paragraph.

> The profiler output for CPU:
>
> --8<---cut here---start->8---
>47871  96%- jinx--check-pending
>47869  96% - jinx--check-region
>47861  96%  - jinx--flyspell-ignored-p
>47861  96%   - org-mode-flyspell-verify
>47837  96%- org--flyspell-object-check-p
>47837  96% - org-element-context
>46973  94%  - org-element--object-lex
>46022  92%   - org-element-code-parser

So, jinx--check-region is triggering org-element-context on every single
word. This will run regexp search N_words times, making Emacs do regexp
search until the end of the paragraph each time.

In the absence of object cache, this is indeed slow.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

problem interaction between jinx-mode and org mode

[Fraga, Eric]

Hello all,

I have just posted an issue on the jinx spell checking package's github
site:

https://github.com/minad/jinx/issues/35

There seems to be a strange interaction between jinx and orgmode when
the buffer has lines of the form:

--8<---cut here---start->8---
┌ Debug: Placing hot node
│   node = n29
└ @ HeatExchangerNetworkDesign 
~/s/research/julia/HeatExchangerNetworkDesign/src/HeatExchangerNetworkDesign.jl:1102
┌ Debug: Layout for node
│   node = n18
│   y = 6
└ @ HeatExchangerNetworkDesign 
--8<---cut here---end--->8---

lines which are the output of Julia's @debug macro, whose output uses
non-ASCII Unicode characters at the start of each line.  Emacs
essentially freezes for some seconds (10-20) every time I scroll (which
makes sense because jinx only spell-checks the visible part of the
buffer).

The profiler output for CPU:

--8<---cut here---start->8---
   47871  96%- jinx--check-pending
   47869  96% - jinx--check-region
   47861  96%  - jinx--flyspell-ignored-p
   47861  96%   - org-mode-flyspell-verify
   47837  96%- org--flyspell-object-check-p
   47837  96% - org-element-context
   46973  94%  - org-element--object-lex
   46022  92%   - org-element-code-parser
 340   0%- org-element--parse-generic-emphasis
 328   0% + rx-to-string
 859   1%   + org-element-italic-parser
--8<---cut here---end--->8---

and MEM:

--8<---cut here---start->8---
326,775,960  96% - timer-event-handler
326,674,976  96%  - apply
325,996,129  96%   - jinx--timer-handler
325,678,345  95%- jinx--check-pending
325,657,281  95% - jinx--check-region
325,636,601  95%  - jinx--flyspell-ignored-p
325,636,601  95%   - org-mode-flyspell-verify
325,072,246  95%- org--flyspell-object-check-p
325,072,246  95% - org-element-context
325,026,076  95%  - org-element--object-lex
215,616,248  63%   - org-element-italic-parser
211,670,968  62%- org-element--parse-generic-emphasis
211,670,968  62% - rx-to-string
186,054,064  54%  - rx--translate
186,054,064  54%   - rx--translate-form
181,758,312  53%- rx--translate-seq
155,393,544  45% - rx--translate
155,393,544  45%  - rx--translate-form
107,261,576  31%   - rx--translate-or
100,408,048  29%- mapcan
 99,194,744  29% + #
  2,524,984   0%  rx--translate
 45,311,856  13%   + rx--translate-not
  8,427,544   2% + rx--sequence
 10,559,024   3%  + rx--atomic-regexp
 83,089,056  24%   + org-element-code-parser
  4,458,332   1%   + org-element-verbatim-parser
342,616   0%+ org-log-into-drawer
--8<---cut here---end--->8---

Turning off jinx-mode in the org buffer restores normal behaviour.

I've not yet tried with emacs -Q, however, so it could be something in
my configuration... However, I do wonder whether it is the file name use
of "~" that may be causing problems with the rx?

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.6.1-341-g614b96 in Emacs 30.0.50

Re: [DISCUSS] "same text-property competing" problem in external packages

[Ihor Radchenko]

stardiviner  writes:

> I got a problem when more than one Emacs package competing on setting
> text-property on same target from different packages.
>
> Here is the original problem and discussion link:
> https://github.com/nobiot/org-transclusion/issues/166
>
> Does anybody have a good idea to solve this problem?

I am not sure which property you are referring to.

There is a number of possible solutions you may use depending on what is
more appropriate for your specific use case:

1. Use overlays with 'priority property. That way, you can "overlay" the
   property above/below existing and Emacs will automatically restore
   the existing value upon removing your overlay. This is the most
   common approach.

2. You can utilize `char-property-alias-alist' and actually store your
   property value in a different property. This will make the priority
   of the existing property _strictly higher_ than yours. org-fold-core
   uses this approach.

3. You can store a "backup" of the existing property and restore it when
   your minor mode is disabled. isearch.el uses this approach.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[DISCUSS] "same text-property competing" problem in external packages

[stardiviner]

I got a problem when more than one Emacs package competing on setting
text-property on same target from different packages.

Here is the original problem and discussion link:
https://github.com/nobiot/org-transclusion/issues/166

Does anybody have a good idea to solve this problem?

Thanks for your idea and suggestions.

[stardiviner] GPG key ID: 47C32433
IRC(freeenode): stardiviner Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi  writes:

> These are init files specific to other packages installed via OS package
> manager. But those are not loaded, it seems in the above steps. I tried to
> run (featurep 'clang-format) and it returned nil, while this returns t
> in my normal Emacs instance.
> ...
> None of these load org-mode.

They just define auto-loading, but I see not how they would affect Org
loading.

Very confusing...

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Wed, 22 Feb 2023 at 16:17, Ihor Radchenko  wrote:
>
> Bhavin Gandhi  writes:
>
> > I'm able to reproduce this on Fedora 37, Emacs 28.2.
> >
> > In Fedora we have:
> > - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> > from /usr/share/emacs/site-lisp/site-start.d).
> > - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).
>
> May I know what is the contents of
> /usr/share/emacs/site-lisp/site-start.d ?

These are init files specific to other packages installed via OS package
manager. But those are not loaded, it seems in the above steps. I tried to
run (featurep 'clang-format) and it returned nil, while this returns t
in my normal Emacs instance.

$ ls -1 /usr/share/emacs/site-lisp/site-start.d/
autoconf-init.el
  https://src.fedoraproject.org/rpms/autoconf/blob/rawhide/f/autoconf-init.el
clang-format.el
clang-include-fixer.el
clang-rename.el
cmake-init.el
  https://src.fedoraproject.org/rpms/cmake/blob/rawhide/f/cmake-init.el
desktop-entry-mode-init.el
  
https://src.fedoraproject.org/rpms/desktop-file-utils/blob/rawhide/f/desktop-entry-mode-init.el
gtypist-init.el
  https://src.fedoraproject.org/rpms/gtypist/blob/rawhide/f/gtypist.spec#_43-46
mercurial-site-start.el
  
https://src.fedoraproject.org/rpms/mercurial/blob/rawhide/f/mercurial-site-start.el

None of these load org-mode.

--
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi  writes:

> I'm able to reproduce this on Fedora 37, Emacs 28.2.
>
> In Fedora we have:
> - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> from /usr/share/emacs/site-lisp/site-start.d).
> - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

May I know what is the contents of
/usr/share/emacs/site-lisp/site-start.d ?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Fri, 3 Feb 2023 at 16:35, Ihor Radchenko  wrote:
>
> Max Nikulin  writes:
>
> > On 27/12/2022 16:47, Ihor Radchenko wrote:
> >> Can you then try to test using Emacs 28?
> >> The main question if whether this has been fixed in newer Emacs releases
> >> or it is also something to do with OS environment.
> >
> > I see quite the same issue with Emacs-28.2 in Debian testing. Compile
> > buffer displays a bit more warnings and usual `org-assert-version'
> > warnings and errors are present as well. It might have another level of
> > complexity due to .eln files. I am unsure what happens with calls to
> > undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.

I'm able to reproduce this on Fedora 37, Emacs 28.2.

In Fedora we have:
- /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
from /usr/share/emacs/site-lisp/site-start.d).
- /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

There are other patches but most of them are backports and seem to be
unrelated to package.el:
https://src.fedoraproject.org/rpms/emacs/tree/rawhide

Steps using with-emacs.sh (https://github.com/alphapapa/with-emacs.sh):

1. mkdir /tmp/emacs-59882
2. with-emacs.sh -d /tmp/emacs-59882 -- -l org
3. M-x package-list-packages
4. Install Org 9.6.1 from ELPA
5. When the package is compiled you get this error at the end (it is
there multiple times)
   ox.el:82:1: Error: Symbol’s function definition is void: org-assert-version
6. Close Emacs and start it again with
   with-emacs.sh -d /tmp/emacs-59882 -- -l org
7. I get byte-code: Invalid function: org-assert-version

-- 
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> Cc: emacs-orgmode@gnu.org, 59...@debbugs.gnu.org
> From: Richard Stallman 
> Date: Tue, 21 Feb 2023 00:22:04 -0500
> 
>   > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
>   > >> does use a modified Emacs startup which is not part of the standard
>   > >> Emacs distribution. They do this to enable the ability to have multiple
>   > >> versions of Emacs installed at the same time.
> 
> Would we like to add a feature like that to Emacs?
> I don't know how much work it would be, but I think
> it would be a desirable change if it is easy.

I'm not sure I understand: feature like what?  If you are talking
about having several Emacs versions installed on the same system, then
I think we already support that: I use that all the time.  That's why
we install various files in version-specific directories.

If you mean something else, please elaborate.  (The bug discussion is
very long, and the above quote is from a tangent, so I'm not sure it's
worth my while reading all the discussion from the beginning trying to
understand what is being alluded to here.)

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

On 2023-02-20, Bruno Barbier wrote:

> Jens Lechtenboerger  writes:
>
>> On 2023-02-20, Bruno Barbier wrote:
>>
>> However, if I use insert-file-contents-literally with a unicode
>> file, I do *not* have to set the coding-system-for-write.  This just
>> works:
>>
>>(with-temp-buffer
>>   (insert-file-contents-literally "~/unicode.org")
>>   (secure-hash 'md5 (current-buffer)))
>
> Humm. Emacs is amazing: it managed to guess the right encoding, from the
> buffer context, probably...
>
> But, what you are giving to 'org-export-string-as' is not the buffer,
> it's a string. So, let's try the same without using an org function:
>
>  (with-temp-buffer
>(insert (with-temp-buffer
>  (insert-file-contents-literally "~/unicode.org")
>  (buffer-string)))
>(secure-hash 'md5 (current-buffer)))
>
> And, that fails, requesting an encoding.

Thank you for this example.

>> In the context of Org export, secure-hash seems to require a coding
>> system.  Why?
>
> I'm not an expert, so, you'll need to confirm with other sources.  But
> secure-hash requires an encoding in all cases, to compute the hash of
> some text, because it needs the array of bytes that represents that text
> to compute its hash.
>
> I don't see any bug in org, and, I don't see any bug in secure-hash either.
>
> You literally shoud stop using "literally" ;-)

Indeed.  

> And, you might want to read:
>(info "(elisp) Non-ASCII Characters")

The first section was already helpful, thanks!  (I still need to
read more of this...)

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: bug#59882: Multiple versions of Org in load-path problem

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
  > >> does use a modified Emacs startup which is not part of the standard
  > >> Emacs distribution. They do this to enable the ability to have multiple
  > >> versions of Emacs installed at the same time.

Would we like to add a feature like that to Emacs?
I don't know how much work it would be, but I think
it would be a desirable change if it is easy.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-20, Bruno Barbier wrote:
>
> However, if I use insert-file-contents-literally with a unicode
> file, I do *not* have to set the coding-system-for-write.  This just
> works:
>
>(with-temp-buffer
>   (insert-file-contents-literally "~/unicode.org")
>   (secure-hash 'md5 (current-buffer)))

Humm. Emacs is amazing: it managed to guess the right encoding, from the
buffer context, probably...

But, what you are giving to 'org-export-string-as' is not the buffer,
it's a string. So, let's try the same without using an org function:

 (with-temp-buffer
   (insert (with-temp-buffer
 (insert-file-contents-literally "~/unicode.org")
 (buffer-string)))
   (secure-hash 'md5 (current-buffer)))
   
And, that fails, requesting an encoding.


> In the context of Org export, secure-hash seems to require a coding
> system.  Why?

I'm not an expert, so, you'll need to confirm with other sources.  But
secure-hash requires an encoding in all cases, to compute the hash of
some text, because it needs the array of bytes that represents that text
to compute its hash.

I don't see any bug in org, and, I don't see any bug in secure-hash either.

You literally shoud stop using "literally" ;-)

And, you might want to read:
   (info "(elisp) Non-ASCII Characters")



Bruno


>
> Best wishes
> Jens

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

On 2023-02-20, Bruno Barbier wrote:

> If you're always using utf-8, here is a way to force it so that
> secure-hash can compute the hash. This should work:
>
>(with-temp-buffer
>   (let ((coding-system-for-write 'utf-8))
> (insert "Lechtenb\303\266rger")
> (secure-hash 'md5 (current-buffer

Yes, that works.

However, if I use insert-file-contents-literally with a unicode
file, I do *not* have to set the coding-system-for-write.  This just
works:

   (with-temp-buffer
  (insert-file-contents-literally "~/unicode.org")
  (secure-hash 'md5 (current-buffer)))

In the context of Org export, secure-hash seems to require a coding
system.  Why?

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-17, Bruno Barbier wrote:
>
>> Here is a way to reproduce that doesn't use org, in case it might help
>> to manully fix your encoding issue:
>>
>>(with-temp-buffer
>>   (insert "Lechtenb\303\266rger")
>>   (let ((buffer-file-name (make-temp-file "mailtest")))
>> (save-buffer)))
>>
>> Does it work with your old config (with your old org) ?
>
> This also asks for an encoding.

If you're always using utf-8, here is a way to force it so that
secure-hash can compute the hash. This should work:

   (with-temp-buffer
  (let ((coding-system-for-write 'utf-8))
(insert "Lechtenb\303\266rger")
(secure-hash 'md5 (current-buffer

Without setting coding-system-for-write to utf-8, it asks for an
encoding:

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (secure-hash 'md5 (current-buffer)))


I'm still no getting your use case, but, let's hope that this naive hack
is enough for you :-)


Bruno


> Best wishes
> Jens

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

On 2023-02-17, Ihor Radchenko wrote:

> Jens Lechtenboerger  writes:

>> Also, when I call secure-hash on the literal buffer-string, no
>> problem arises.
>
> Org is calling secure-hash on buffer. Calling on buffer-string would
> require unnecessary memory allocation to create the string.

I can call secure-hash on the buffer with literally inserted
contents without problems.

>> It is not obvious that Org tries to write something here and why
>> that fails now
>
> Org is not trying to write something. In you example, Org is just trying
> to calculate buffer string hash - nothing wrong on Org side. "Something
> wrong with encoding" way my guess. If you think that your case should be
> perfectly fine, I recommend asking Emacs devs by filing a bug report to
> them.

Thank you for the clarifications.  Probably I have to do that.

For the record, if I insert "Lechtenb\303\266rger" as string into a
buffer, secure-hash asks for a decoding.  If I insert that literally
via an UTF-8 encoded file, secure-hash works.

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

On 2023-02-17, Bruno Barbier wrote:

> Here is a way to reproduce that doesn't use org, in case it might help
> to manully fix your encoding issue:
>
>(with-temp-buffer
>   (insert "Lechtenb\303\266rger")
>   (let ((buffer-file-name (make-temp-file "mailtest")))
> (save-buffer)))
>
> Does it work with your old config (with your old org) ?

This also asks for an encoding.

> What kind of failure do you get elsewhere if you let Emacs use the
> correct encoding (i.e. if you use `insert-file-contents') ?

I want to be sure to use the file contents in unchanged form, as
promised by insert-file-contents-literally.  For now, I copied part
of the code from insert-file-contents-literally to avoid
after-insert processing and file handlers.  I still do not
understand what is happening differently in my case, though...

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: Unicode problem with export of literal contents

[Ihor Radchenko]

Jens Lechtenboerger  writes:

> I was afraid you would say so.  To me, this is a breaking change.

It is not a breaking change. It is Org's change revealing issues with
your files. If you need to edit or act upon that part of the file, you
could see the same problem.

> Also, when I call secure-hash on the literal buffer-string, no
> problem arises.

Org is calling secure-hash on buffer. Calling on buffer-string would
require unnecessary memory allocation to create the string.

> It is not obvious that Org tries to write something here and why
> that fails now

Org is not trying to write something. In you example, Org is just trying
to calculate buffer string hash - nothing wrong on Org side. "Something
wrong with encoding" way my guess. If you think that your case should be
perfectly fine, I recommend asking Emacs devs by filing a bug report to
them.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> So, maybe my question is: Must text be decoded for Org mode from now on?

Yes. Since forever.  Emacs must know how to read/write from/to files and
what text to display to you. Org is just relying on Emacs for that part.

Bruno

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Jens Lechtenboerger  writes:

> On 2023-02-17, Ihor Radchenko wrote:
>
>> Jens Lechtenboerger  writes:
>>
>
>> Not a bug. You need to fix your files with improper encoding.
>
> The file has the proper encoding.  I insert literally on purpose as
> stated above.

IIUC, the file has the proper encoding. But, when loading it with
`insert-file-contents-literally', it doesn't: that's part of the
"literally" feature I guess.

When loading it with `insert-file-contents', it should work (it does in
my case).


Here is a way to reproduce that doesn't use org, in case it might help
to manully fix your encoding issue:

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (let ((buffer-file-name (make-temp-file "mailtest")))
(save-buffer)))

Does it work with your old config (with your old org) ?

What kind of failure do you get elsewhere if you let Emacs use the
correct encoding (i.e. if you use `insert-file-contents') ?



Bruno

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

On 2023-02-17, Ihor Radchenko wrote:

> Jens Lechtenboerger  writes:
>
>> With Org 9.6.1 from Emacs master, I get the following warning, and I
>> am asked to select a coding system:
>>
>>> These default coding systems were tried to encode the following
>>> problematic characters in the buffer ‘ *temp*’:
>>> ...
>>
>> With previous Org versions, this did not happen, export would just
>> work.  Note that I insert contents literally because I do not want
>> ‘find-file-hook’, automatic uncompression, etc. (which are avoided
>> according to the doc string of insert-file-contents-literally).
>
> This warning appears upon Org calling `secure-hash'.
> Org is doing nothing wrong here - your file does not have proper encoding.
> You did not see this error in the past by chance.

I was afraid you would say so.  To me, this is a breaking change.

Also, when I call secure-hash on the literal buffer-string, no
problem arises.

> Not a bug. You need to fix your files with improper encoding.

The file has the proper encoding.  I insert literally on purpose as
stated above.

It is not obvious that Org tries to write something here and why
that fails now (I could use the results in exporters writing to
files just fine previously).

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: Unicode problem with export of literal contents

[Ihor Radchenko]

Jens Lechtenboerger  writes:

> With Org 9.6.1 from Emacs master, I get the following warning, and I
> am asked to select a coding system:
>
>> These default coding systems were tried to encode the following
>> problematic characters in the buffer ‘ *temp*’:
>> ...
>
> With previous Org versions, this did not happen, export would just
> work.  Note that I insert contents literally because I do not want
> ‘find-file-hook’, automatic uncompression, etc. (which are avoided
> according to the doc string of insert-file-contents-literally).

This warning appears upon Org calling `secure-hash'.
Org is doing nothing wrong here - your file does not have proper encoding.
You did not see this error in the past by chance.

Not a bug. You need to fix your files with improper encoding.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: Unicode problem with export of literal contents

[Jens Lechtenboerger]

Hi Bruno,

On 2023-02-17, Bruno Barbier wrote:

> Hi Jens,
>
> Jens Lechtenboerger  writes:
>
>> ...
>> Note that I insert contents literally because I do not want
>> ‘find-file-hook’, automatic uncompression, etc. (which are avoided
>> according to the doc string of insert-file-contents-literally).
>>
>> Could the old behavior be restored?
>
> By using `insert-file-contents-literally' (as opposed to
> `insert-file-contents'), you're also forbidding Emacs to decode the
> binary content of your file into text.
>
> My guess is that it was working by chance in previous versions.

in any case, this will introduce failures elsewhere.

> In case somebody might help you, here is a simple way to trigger the
> encoding question with a recent version of org (mine is Org mode version 
> 9.6.1).
>
>(with-temp-buffer
>   (insert "Lechtenb\303\266rger")
>   (org-mode))

Thank you for the simpler recipe.  This indeed fails now.

So, maybe my question is: Must text be decoded for Org mode from now on?

Best wishes
Jens


smime.p7s
Description: S/MIME cryptographic signature

Re: Unicode problem with export of literal contents

[Bruno Barbier]

Hi Jens,

Jens Lechtenboerger  writes:

> ...
> Note that I insert contents literally because I do not want
> ‘find-file-hook’, automatic uncompression, etc. (which are avoided
> according to the doc string of insert-file-contents-literally).
>
> Could the old behavior be restored?

By using `insert-file-contents-literally' (as opposed to
`insert-file-contents'), you're also forbidding Emacs to decode the
binary content of your file into text.

My guess is that it was working by chance in previous versions.

In case somebody might help you, here is a simple way to trigger the
encoding question with a recent version of org (mine is Org mode version 9.6.1).

   (with-temp-buffer
  (insert "Lechtenb\303\266rger")
  (org-mode))



Bruno