Re: bash source code block: problem after ssh commands

2023-11-22 Thread Max Nikulin

On 22/11/2023 02:01, Bruno Barbier wrote:

Ihor Radchenko writes:

__By default__, Org should produce more expected behavior - what users
would get from running a script file rather than from redirecting stdin.
We can optionally leave the stdin redirection as an option to be used by
the users who understand the peculiarities.


I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).


I do not like recommendations

curl https://... | sudo bash

but I regularly see them. So some users might expect behavior like 
namely stdin, not like interactive prompt commands.


Perhaps even sessions may be switched to creation of a temporary file 
and executing "source tmpfile.sh". However it may be shell-specific (posh?).


Likely it is possible to implement a header argument to explicitly 
control whether `process-file' should be used or the block should be 
executed as a temporary script file. Currently it is implicitly 
determined from other header arguments.


P.S.

https://mywiki.wooledge.org/BashFAQ/089
"I'm reading a file line by line and running ssh or ffmpeg, only the 
first line gets processed!"


A similar case, but only data is read from stdin, script is a regular 
file, no difference bash vs. dash.





Re: bash source code block: problem after ssh commands

2023-11-21 Thread Bruno Barbier


Ihor Radchenko  writes:

> Bruno Barbier  writes:
>
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> The behavior of M-x shell can indeed be made use of.
> However, this particular difference with xterm, AFAIU, is not
> documented - unaware users may be surprised.
> The situation is worse with Org shell blocks - users naturally expect
> script-like behavior (even for :session), but run into edge cases like
> this and get confused.
>
> We should either document the caveats, or, preferably, make the behavior
> more consistent with expectations. At least, by default.

> That's why I think that filing a bug report makes sense from Org mode
> project point of view.

Thanks for the explanation.  I see you got my point.  We'll see what
Emacs maintainers will say about the current behavior of M-x shell;
filling the bug report was definitely a good idea anyway.



Ihor Radchenko  writes:
> __By default__, Org should produce more expected behavior - what users
> would get from running a script file rather than from redirecting stdin.
> We can optionally leave the stdin redirection as an option to be used by
> the users who understand the peculiarities.

I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).

Bruno



Re: Non-emacs shell (Re: bash source code block: problem after ssh commands)

2023-11-21 Thread Bruno Barbier


Hi Max,

Max Nikulin  writes:

> On 17/11/2023 22:47, Bruno Barbier wrote:
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> I am unsure what do you expect from xterm, but perhaps it is not 
> responsibility of a terminal application.

It has been said in this thread that 'M-x shell' should be fixed to
match the behavior that we see in a plain terminal, when we copy
multiple lines.  I just wanted to point out that I do prefer the way
'M-x shell' handles the copy of multiple lines.


> Multiple lines can be copied to regular BASH prompt (bracketed paste is 
> enabled by default nowadays), however it may be inconvenient to edit.
> You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
> binding) to start an editor and to paste multiple commands there.
> See also "fc" BASH built-in for editing and re-executing last commands.

Thanks Max! I didn't know that.  I should definitely start using this
when I'm stuck in a console, to safely copy/edit my commands using
Emacs.

Thanks,


Bruno



Non-emacs shell (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Max Nikulin

On 17/11/2023 22:47, Bruno Barbier wrote:

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


I am unsure what do you expect from xterm, but perhaps it is not 
responsibility of a terminal application.


Multiple lines can be copied to regular BASH prompt (bracketed paste is 
enabled by default nowadays), however it may be inconvenient to edit.


You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
binding) to start an editor and to paste multiple commands there.


See also "fc" BASH built-in for editing and re-executing last commands.




Re: bash source code block: problem after ssh commands

2023-11-18 Thread Max Nikulin

On 18/11/2023 17:43, Ihor Radchenko wrote:


I still see it as a bug - what Org mode does to run the shell blocks is
not what users expect. _By default_, we _should_ produce more expected
behavior.


A shell without unexpected behavior is neither POSIX nor a descendant 
shell like BASH

https://mywiki.wooledge.org/BashPitfalls

Behavior if interactive and non-interactive shells is not the same anyway.


I cannot find any clear motivation behind using `process-file' + INFILE
in the git logs. I assume that it was used simply because it is easier
compared to trying to create and run temporary script file on remote
hosts.


From my point of view it is rather natural to use `process-file' since 
this function is available out of the box. I suspect some subtle issues 
may appear even if script file is forced.


On the other hand ob-shell already has a (bit buggy) implementation 
relying on script files. It is used when :stdin or :cmdline is specified.






Re: bash source code block: problem after ssh commands

2023-11-18 Thread Matt


  On Fri, 17 Nov 2023 23:07:57 +0100  Matt  wrote --- 
 
 > The second claim has nothing to do with Org Babel.  I was able to confirm it 
 > and provide the steps to reproduce.  I think it would make sense to report 
 > it upstream and let them decide if it's expected behavior.  I'm still happy 
 > to do that, but I need to step away from the keyboard :)

Submitted. 

https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Ihor Radchenko
Matt  writes:

> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
> cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine 
> and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Yup, it is expected. But only in a sense that "Emacs does what we asked
for when calling `process-file'". From the user point of view, it is not
expected at all. And, what is worse, may depend on (1) shell used; (2)
ssh config (see StdinNull in man 5 ssh_config).

I believe that we do need to change how we execute shell blocks _by
default_ - to something more predictable, like creating and running a
temporary script file.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-18 Thread Ihor Radchenko
Max Nikulin  writes:

> ...
> I have not expected this difference.
>
> dash reads a block from stdin (whole file in this case) and interprets 
> commands.
>
> BASH reads just the ssh command and executes it. SSH reads "echo done" 
> from stdin, so when control is returned to bash, stdin is exhausted and 
> no commands remain to execute by BASH. SSH can not "unread" part of 
> input not consumed by the remote command despite it might be possible in 
> the case of the regular file as stdin.
>
> Actually bash reads the whole script file as well when called as it is 
> shown above, but it calls lseek before executing ssh. To make difference 
> more apparent (e.g. for strace), force creation of pipe(7) for which 
> lseek is not supported
> ...
> I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
> vs. dash issue.

I still see it as a bug - what Org mode does to run the shell blocks is
not what users expect. _By default_, we _should_ produce more expected
behavior.

The observed inconsistency between different shells just indicates that
our approach with `process-file' should not be used as it leads to
potentially confusing results (they are not confusing only to people who
dig deeply into ssh, bash/dash/etc, and Org mode internals).

I cannot find any clear motivation behind using `process-file' + INFILE
in the git logs. I assume that it was used simply because it is easier
compared to trying to create and run temporary script file on remote
hosts.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-18 Thread Ihor Radchenko
Bruno Barbier  writes:

>> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
>> the problem with M-x shell is not the problem you originally ran to. It
>> is a different problem (also, we ran into it in the past).
>
> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
> multiple lines, getting the same results as when I type them manually,
> or copy them line by line. My xterm doesn't seem to allow me to do that.

The behavior of M-x shell can indeed be made use of.
However, this particular difference with xterm, AFAIU, is not
documented - unaware users may be surprised.
The situation is worse with Org shell blocks - users naturally expect
script-like behavior (even for :session), but run into edge cases like
this and get confused.

We should either document the caveats, or, preferably, make the behavior
more consistent with expectations. At least, by default.

That's why I think that filing a bug report makes sense from Org mode
project point of view.

> It looks like it is a known SSH "feature" (see
> https://unix.stackexchange.com/a/688024):
>
> #+begin_src bash :results output
> seq 100 | (ssh phone sleep 1; wc -l)
> #+end_src
>
> #+RESULTS:
> : 675173
> ...

> IMHO, what ob-shell is doing today seems a valid way of evaluating
> source blocks (and it seems to have been like that for a long time).  It
> should probably be documented somewhere, so that users know how to write
> their source blocks, or switch to another way, like adding a :cmdline
> parameter as mentionned in this thread.

And, as Max demonstrated, this ssh feature only works with bash, but not
necessarily other shells. Moreover, "-n" option that disables the above
may depend on ssh configuration (see StdinNull in man 5 ssh_config). So,
I believe that it only leads to confusion, even if we try to document
it.

__By default__, Org should produce more expected behavior - what users
would get from running a script file rather than from redirecting stdin.
We can optionally leave the stdin redirection as an option to be used by
the users who understand the peculiarities.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Bruno Barbier
Matt  writes:

>   On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
>  
> It's still not clear to me if this is "what Emacs does".  However, that's the 
> best I could come up with.  
>
> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
> cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine 
> and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Agreed.



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Matt


  On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
 
 > But, you're right.  To be safe, from now on, I'll use:
 > 
 > cat /tmp/test.sh | bash -c bash

It's still not clear to me if this is "what Emacs does".  However, that's the 
best I could come up with.  

Evaluating the following 

#+name: /tmp/test.sh
#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
#+end_src

does exactly what

cat /tmp/test.sh | bash -c bash

does.  Both create a file "foo_file" containing "foo" on the remote machine and 
neither execute the second line.

So, I would say that what happens in Org is the "expected" behavior.



Re: bash source code block: problem after ssh commands

2023-11-18 Thread Matt
To clarify a previous typo I made.  Everything I've written was also done using:

1. emacs -q
2. C-x b "*scratch*"
3. M-x org-mode
4. Execute 

#+begin_src emacs-lisp :results none
(org-babel-do-load-languages
   'org-babel-load-languages
   '((shell . t)))
#+end_src

This corresponds to Org mode version 9.6.6 (release_9.6.6 @ 
/gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/)

I had previously, and incorrectly, given my version used with my full init 
(which uses a different Org version).

  On Sat, 18 Nov 2023 09:19:13 +0100  Bruno Barbier 
 
 > It seems your SSH failed to connect.  In that case, I cannot swallow the
 > second command; thus the command "echo bar" is executed.
 > 
 > I can reproduce what you see on my side if I force the connection to fail:
 > 
 > #+begin_src bash :results output
 >   ssh WRONG_REMOTE "echo foo>foo_file"
 >   echo "bar"
 > #+end_src
 > 
 > #+RESULTS:
 > : bar
 
Thank you!  That makes sense.   I forgot that without set -e Bash keeps 
evaluating when errors happen.

Yes, once it connects, I get the same behavior as reported.

Here is how I connected without a password prompt:

1. ssh-keygen with id_babel, , 
2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys
3. chmod og-wx ~/.ssh/authorized_keys
4. ssh-add .ssh/id_babel



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Bruno Barbier
Matt  writes:

>   On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 
>
>  > IIUC, what Max is saying is that you should not concentrate on
>  > *that specific command* because that command doesn't do what you think
>  > it does.
>
> Cool, it sounds like we're agreed (albeit for different reasons).

Great :-)


>  > To reproduce, I'm personally still using:
>  > 
>  > cat /tmp/test.sh | bash
>  > 
>  > which is, IIUC, what:
>  > 
>  > (process-file "bash" "/tmp/test.sh")
>
> Yes, agreed.  I think that's more like what's happening.  
>
> What about the ("-c" "bash") passed into process-file?  

Useless indirection when the command is "bash" is the same as
'shell-file-name', like in our case ?

Maybe ...

But, you're right.  To be safe, from now on, I'll use:

cat /tmp/test.sh | bash -c bash


Thanks.

Bruno




Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Matt


  On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 

 > IIUC, what Max is saying is that you should not concentrate on
 > *that specific command* because that command doesn't do what you think
 > it does.

Cool, it sounds like we're agreed (albeit for different reasons).  

 > To reproduce, I'm personally still using:
 > 
 > cat /tmp/test.sh | bash
 > 
 > which is, IIUC, what:
 > 
 > (process-file "bash" "/tmp/test.sh")

Yes, agreed.  I think that's more like what's happening.  

What about the ("-c" "bash") passed into process-file?  

The whole call looks like this:

(process-file
   "bash"
   "/tmp/two-lines.sh"
   '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
   nil
   "-c" "bash")




Re: bash source code block: problem after ssh commands

2023-11-18 Thread Bruno Barbier
Max Nikulin  writes:

> On 25/10/2023 18:17, alain.coch...@unistra.fr wrote:
>> By contrast, it works with this one:
>> 
>> #+begin_src bash :results output
>> sshcoch...@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
>> #+end_src
>
> What about
>
> #+begin_src bash :results output
> ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
> echo more
> #+end_src
>
> ?

For me:

 #+begin_src bash :results output
 ssh phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

 #+RESULTS:
 : bar

And, telling SSH to not swallow the remaining commands (option '-n'):

 #+begin_src bash :results output
 ssh -n phone "echo foo>foo_file" ; echo "bar"
 echo more
 #+end_src

#+RESULTS:
: bar
: more


Bruno



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Bruno Barbier
Matt  writes:

>   On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 
>
>  > >  bash -c bash /tmp/two-lines.sh
>  > 
>  >  From my point of view it was a plain mistake in attempts to simulate 
>  > the issue outside of Emacs. There is no point to concentrate on this 
>  > command. I tried to explain that it is incorrect usage of "-c" shell 
>  > option and what is the actual effect of this call, but I seems I failed.
>
> As an ob-shell user, my expectation is that execution within Org produces the 
> same behavior as outside of Emacs.  This is why I've focused on the command.  
> It acts as a guide for what is "correct."  Maybe this is misguided for a 
> reason I don't yet see?

IIUC, what Max is saying is that you should not concentrate on
*that specific command* because that command doesn't do what you think
it does.


To reproduce, I'm personally still using:

cat /tmp/test.sh | bash

which is, IIUC, what:

(process-file "bash" "/tmp/test.sh")

is doing, that is:

   The program’s input comes from file INFILE


Bruno





Re: bash source code block: problem after ssh commands

2023-11-18 Thread Bruno Barbier


Hi Matt,

Thanks this summary and for working on this!

Just a few comments/corrections about some specific points, hoping it
might help.

Matt  writes:

>   On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 
>
>  > This has nothing to do with Emacs comint and this is also not a bug in
>  > Emacs 
>
> Ihor, there were two claims made in the original report.  I was referring to 
> Claim 2.  That deals with M-x shell and therefore comint-mode.
>
> Regarding Claim 1:
>
> - Can anyone verify Claim 1?

I do: the file is created and the command "echo bar" is NOT executed.

Here is my code block and its results:

#+begin_src bash :results output
  ssh phone "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:

No results (the echo command is NOT executed).

The file "foo_file" is created on the remote; its content is "foo".

#+begin_src bash :results output
  date
  ssh -n phone "ls -alh foo_file"
  ssh -n phone "cat foo_file"
#+end_src

#+RESULTS:
: Sat Nov 18 08:33:59 CET 2023
: -rw--- 1 u0_a256 u0_a256 4 Nov 18 08:26 foo_file
: foo



> - What versions are people using?
>   + M-x org-version
>   + M-x emacs-version

#+begin_src elisp
  (list emacs-version org-version)
#+end_src

#+RESULTS:
| 30.0.50 | 9.7-pre |

GNU/Linux gentoo

> ...

> * Comments about the claims:

> ** Comment 1.
> ...
> I am unable to reproduce the reported behavior (of
> "bar" not returning).  Instead, I get an ssh-askpass permission denied
> error, foo_file is not created, and "bar" is given as the result.  I
> do not see anywhere in the thread that the original claim was
> reproduced.

It seems your SSH failed to connect.  In that case, I cannot swallow the
second command; thus the command "echo bar" is executed.

I can reproduce what you see on my side if I force the connection to fail:

#+begin_src bash :results output
  ssh WRONG_REMOTE "echo foo>foo_file"
  echo "bar"
#+end_src

#+RESULTS:
: bar


>
> The thread preceded something like follows.
>
> Leo Butler suggested two work arounds:
>
> - add the -f to the ssh command


> - add a semi-colon and line continuation to the first line.
>
> Russell Adams suggested another work around:
>
> - add -n to the ssh command

That's the one I use; the option -n is enough for me ('-n' = Redirects
stdin from /dev/null). The option '-f' means SSH will go to background;
I'm not sure I want that.

> ...

> ... 
> He then proposes an experiment to close stdin.  To do this, he calls
>
> #+begin_src shell :results output
> exec 0>&-
> echo OK
> #+end_src
>
> He claims that "exec 0<&-" closes stdin.  I believe there is a typo.
> ...

You're right. Good catch, thanks!

Although it seems to work either way on my side.

#+begin_src shell :results output
  exec 0<&-
  echo OK
#+end_src

#+RESULTS:

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

#+RESULTS:


> What Bruno writes corresponds to "closing output file descriptor 0".  I 
> honestly don't know what the difference is between an "output file 
> descriptor" and an "input file descriptor".  I had no luck finding this 
> information in man bash or info bash.
>

My point was: the commands are read the standard input, thus, any
command that modifies that standard input will modify what gets
executed.


> ...
> This is what we see in Org.  I'll be honest, though, I don't
> really know what to expect with exec 0>&- and exec 0<&-.  When I call
> them in the terminal, it kills the terminal.

Let's forget about 'exec 0<&-' (closing the standard input/outputs):
this is bringing other corner cases.  But, yes, I would expect a
terminal to close itself automatically if its input is closed.

> ...
> As far as I can tell, though, that's not what prevents "bar" from being 
> returned.  As far as I can reproduce, calling
>
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar"
> #+end_src
>
> *does* give "bar" for results even though it shouldn't.

Does it echo bar when the SSH connection succeeds too ?


Thanks again for working on this.


Bruno



Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-18 Thread Matt


  On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 

 > >  bash -c bash /tmp/two-lines.sh
 > 
 >  From my point of view it was a plain mistake in attempts to simulate 
 > the issue outside of Emacs. There is no point to concentrate on this 
 > command. I tried to explain that it is incorrect usage of "-c" shell 
 > option and what is the actual effect of this call, but I seems I failed.

As an ob-shell user, my expectation is that execution within Org produces the 
same behavior as outside of Emacs.  This is why I've focused on the command.  
It acts as a guide for what is "correct."  Maybe this is misguided for a reason 
I don't yet see?

I concluded that the command is not "what Emacs does" and therefore isn't a 
valid source of "truth".  I agree it's not worth concentrating on it further.





Re: bash source code block: problem after ssh commands

2023-11-18 Thread Max Nikulin

On 25/10/2023 18:17, alain.coch...@unistra.fr wrote:

By contrast, it works with this one:

#+begin_src bash :results output
sshcoch...@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
#+end_src


What about

   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
   echo more
   #+end_src

?




Re: bash source code block: problem after ssh commands

2023-11-18 Thread Max Nikulin

On 17/11/2023 17:17, Ihor Radchenko wrote:

I was only able to reproduce your problem with ssh asking a password.
We are discussing the reproduced case.


I see bash vs. dash difference with public key authorization, so no need 
for password prompts. I have not figured out how to construct an example 
without ssh since this command *may* read stdin, but does not do it in a 
same way as e.g. cat(1). Perhaps a small program performing single 
non-blocking read will allow it. The following behavior observed for a 
regular shell prompt, Emacs is not involved. Debian 12 bookworm.


cat ssh-script.sh
ssh -p  127.0.0.1 'echo foo>/tmp/foo'
echo done

Read commands from a script file:

dash ssh-script.sh
done

bash ssh-script.sh
done

Read commands from stdin

dash dash reads a block from stdin (whole file in this case) and interprets 
commands.


BASH reads just the ssh command and executes it. SSH reads "echo done" 
from stdin, so when control is returned to bash, stdin is exhausted and 
no commands remain to execute by BASH. SSH can not "unread" part of 
input not consumed by the remote command despite it might be possible in 
the case of the regular file as stdin.


Actually bash reads the whole script file as well when called as it is 
shown above, but it calls lseek before executing ssh. To make difference 
more apparent (e.g. for strace), force creation of pipe(7) for which 
lseek is not supported


cat ssh-script.sh | strace -o /tmp/bash.strace bash

I am unsure if POSIX specifies exact behavior of shell when commands are 
read from stdin. I think, the suggested earlier "-n" ssh option (or 
stdin. There is too much room for heuristics: interactive vs. 
non-interactive shell, a terminal vs. a regular file vs. a pipe as 
standard input. Be explicit to get reliable behavior.


ssh -n user@host 'command'

or

tar cvf - . | ssh user@host 'tar xvf -'

without "-n" when ssh needs stdin and it is explicitly specified.

I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
vs. dash issue.





Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

2023-11-17 Thread Max Nikulin

On 18/11/2023 05:07, Matt wrote:

Consider calling

 bash -c bash /tmp/two-lines.sh


Matt, could you, please, describe what this command should do 
accordingly to your expectations?


From my point of view it was a plain mistake in attempts to simulate 
the issue outside of Emacs. There is no point to concentrate on this 
command. I tried to explain that it is incorrect usage of "-c" shell 
option and what is the actual effect of this call, but I seems I failed.


Hints:
- Try this command in a regular shell prompt outside of Emacs
- Try "echo $$" before and after this command
- Try it with a script having a clear side effect and check whether it 
is produced, e.g. "date >>/tmp/touch-tmp-file.txt"





Re: bash source code block: problem after ssh commands

2023-11-17 Thread Matt


  On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 

 > This has nothing to do with Emacs comint and this is also not a bug in
 > Emacs 

Ihor, there were two claims made in the original report.  I was referring to 
Claim 2.  That deals with M-x shell and therefore comint-mode.

Regarding Claim 1:

- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
  + M-x org-version
  + M-x emacs-version

I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ 
/home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, 
x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).

* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with 
contents "foo" as well as give "bar" as the result.

#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src

The reported behavior is that "foo_file" is created with "foo" (with "foo" is 
not stated, but implied) and "bar" is *not* given as the result.

** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like 
xfce4-terminal executes the ssh line as expected and outputs the result of the 
second line.  It was noted that this does not happen with M-x shell.

* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result.  The result 
is "bar" for me.

The exact "expected behavior" for a shell block is a little fuzzy.  According 
to my analysis (given below), what Alain reports (remote file and no "bar") is 
the "expected" behavior.  What I see (no remote file and "bar") is actually 
"unexpected".

I used the following to test the claim:

#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar"
#+end_src

I am unable to reproduce the reported behavior (of "bar" not returning).  
Instead, I get an ssh-askpass permission denied error, foo_file is not created, 
and "bar" is given as the result.  I do not see anywhere in the thread that the 
original claim was reproduced.

The thread preceded something like follows.

Leo Butler suggested two work arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Russell Adams suggested another work around:

- add -n to the ssh command

Ihor identified that a non-session call does something like the following 
command:

bash -c bash /tmp/temp-file-with-source-block-code.sh

where

- /tmp/temp-file-with-source-block-code -
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
-

The second line (significantly different from the original report) pipes the 
echo result to stdout and to a file, bar.txt.  Writing to a file allows us to 
confirm if that line was executed.

Ihor suggested that

bash -c bash /tmp/temp-file-with-source-block-code.sh

does not run the second line because an interactive password prompt is 
displayed by ssh.  The reasoning is that the prompt hangs the process while 
waiting for input and the second line never runs.  Indeed, running the command 
does not produce /tmp/bar.txt.

Ihor is correct about prompts messing with shell blocks (this is not the first 
time he's seen this).  However, the way it's stated does not demonstrate it. 
This is because Emacs does *not* make a call like

bash -c bash /tmp/temp-file-with-source-block-code.sh

Alain responded by pointing out that

bash -c bash /tmp/temp-file-with-source-block-code.sh

does not execute the first line.  This is true.  Consider calling

bash -c bash /tmp/two-lines.sh

where

-- /tmp/two-lines.sh --
echo "first" > /tmp/first.txt
echo "second" > /tmm/second.txt
---

Neither first.txt or second.txt are created.

Max Nikulin interjected with a helpful reminder that Bash scripting is a 
snakepit of footguns.  (What Max said is more than that and interesting.  I 
skip it here because it depends on the form of the call.)

Before trying to untangle what a given Bash command does, we need to be sure 
what command is actually called.  Unfortunately, there's not a clear Bash 
command corresponding to how Emacs makes the call.

What happens goes something like this:

1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path 
to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object 
implemented in C
3. DEFUN ("call-process") forwards this information to the C function 
call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess

A lot of cleaning and setup happens which is dependent on the system (GNU, 
Window, Darwin, etc.).  There's a call to openp which looks for the executable. 
 An emacs_pipe is set up 

Re: bash source code block: problem after ssh commands

2023-11-17 Thread Bruno Barbier


Hi Matt, Ihor, Alain,

Ihor Radchenko  writes:

> alain.coch...@unistra.fr writes:
>
>> At the most basic user level (i.e., non lisp aware), why is it not
>> necessarily a bug if "something" does the expected in an X terminal
>> but not in an emacs terminal?  I think Matt and I (and others) are on
>> the same page here, and he already essentially drafted the bug report
>> I would have sent (only better).
>
> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
> the problem with M-x shell is not the problem you originally ran to. It
> is a different problem (also, we ran into it in the past).

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


>> What also confuses me is that it seems to me that, in the minimum
>> working example, you consider the ssh command and the read command as
>> equivalent.  But I don't even enter the password when using ssh...

I've introduced the 'read' example, as a simpler way to modify the
standard input and get the same kind of "unexpected" results, without
relying on SSH.  It seems that I have only caused confusion, sorry about
that.

IIUC, the OP example is not working because SSH is modifying the
standard input (with or without passwords).

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.
>

I'm not the OP, but, my SSH is configured to work without passwords and
SSH is still consuming lines from standard input:

#+begin_src bash :results output
ssh phone echo "remote"
echo "local"
#+end_src

#+RESULTS:
: remote

It looks like it is a known SSH "feature" (see
https://unix.stackexchange.com/a/688024):

#+begin_src bash :results output
seq 100 | (ssh phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 675173

Same block, but asking SSH to not use stdin (using '-n' as mentionned in
this thread):

#+begin_src bash :results output
seq 100 | (ssh -n phone sleep 1; wc -l)
#+end_src

#+RESULTS:
: 100

IMHO, what ob-shell is doing today seems a valid way of evaluating
source blocks (and it seems to have been like that for a long time).  It
should probably be documented somewhere, so that users know how to write
their source blocks, or switch to another way, like adding a :cmdline
parameter as mentionned in this thread.


Hoping I didn't increase the confusion again,
:-)


Bruno




> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-17 Thread Leo Butler
On Fri, Nov 17 2023, Ihor Radchenko  wrote:

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.

I think you are confused, Ihor. The bug was confirmed here:

https://list.orgmode.org/orgmode/87bkcmlor9@t14.reltub.ca/

I mis-identified the culprit; you corrected me by pointing out the bug
is in `process-file'. And you confirmed the bug:

https://list.orgmode.org/orgmode/87fs1xbis1.fsf@localhost/

Russell Adams and I suggested ssh-specific work-arounds that prevent ssh
from grabbing stdin (either -n or -f).

Leo


Re: bash source code block: problem after ssh commands

2023-11-17 Thread Ihor Radchenko
alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Fri 17 Nov 2023 09:22:
>
>  > I think that I need to clarify here.
>
> Thank you.
>
> Well, thank you for _trying_: most of the discussion in this thread is
> way beyond my pay grade.

My reply was to Matthew, the current ob-shell maintainer.
If you want, you need to examine ob-shell (`org-babel-sh-evaluate') and
ob-eval (`org-babel-eval') code, and try to step through the code using
debugger to understand technical details in thread better.

> At the most basic user level (i.e., non lisp aware), why is it not
> necessarily a bug if "something" does the expected in an X terminal
> but not in an emacs terminal?  I think Matt and I (and others) are on
> the same page here, and he already essentially drafted the bug report
> I would have sent (only better).

WRT M-x shell, feel free to submit a bug report. I mostly pointed that
the problem with M-x shell is not the problem you originally ran to. It
is a different problem (also, we ran into it in the past).

> What also confuses me is that it seems to me that, in the minimum
> working example, you consider the ssh command and the read command as
> equivalent.  But I don't even enter the password when using ssh...

I was only able to reproduce your problem with ssh asking a password.
We are discussing the reproduced case.

If you see problems with
   #+begin_src bash :results output
   ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src
even when ssh does not ask for a password, please provide more detailed
reproducer that we can replicate locally without guessing your ssh config.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-17 Thread Alain . Cochard
Ihor Radchenko writes on Fri 17 Nov 2023 09:22:

 > I think that I need to clarify here.

Thank you.

Well, thank you for _trying_: most of the discussion in this thread is
way beyond my pay grade.

At the most basic user level (i.e., non lisp aware), why is it not
necessarily a bug if "something" does the expected in an X terminal
but not in an emacs terminal?  I think Matt and I (and others) are on
the same page here, and he already essentially drafted the bug report
I would have sent (only better).

What also confuses me is that it seems to me that, in the minimum
working example, you consider the ssh command and the read command as
equivalent.  But I don't even enter the password when using ssh...


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-11-17 Thread Ihor Radchenko
Matt  writes:

> Thank you for clarifying.
>
>  > Anyway, this gives me the opportunity to come back to the question of
>  > whether or not there is a problem with emacs itself (like some people
>  > here thought), and if some message should be sent to some emacs list.
>  > Again my argument was that the 2 commands copy/yank'ed in an emacs
>  > terminal don't work as expected, while they do if the same is done in
>  > an X terminal.
>
> Okay, I follow you now.
>
> Yes, I agree with what others have said, it's related to Emacs (probably 
> comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's 
> hard to reproduce with M-x term because term-char-mode doesn't allow 
> copy-paste.  M-x with term-line-mode doesn' t print "bar".

I think that I need to clarify here.
We are talking about two different things:
1. Bash src block without session
2. Bash src block with session

The original report used bash src blocks _without_ session.
In such scenario, comint (and M-x shell) is not relevant.
Org simply uses `process-file' with INFILE argument.
This is equivalent of someone opening __X shell__, and literally typing

  ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
  echo "bar"

In the scenario described in the report, when ssh asks for password, it
is equivalent to

$ ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
Password: 

This has nothing to do with Emacs comint and this is also not a bug in
Emacs - we use INFILE argument, that is equivalent to the above as _per
docstring_. So, Org mode is simply not using `process-file' function as
users expect - instead of executing a bash script, we emulate
interactive user input to bash.

(Side note: for someone aware about these details, it is possible to do
something like

#+begin_src bash
ssh remote-server;
<>
#+end_src

or, as a demo

#+begin_src bash
read x;
value2
echo "We just read \"$x\"";
#+end_src

#+RESULTS:
: We just read "value2"

)

Another case is (2), when we do use comint is session is what we
discussed in
https://list.orgmode.org/orgmode/cal1eyujntgbxy6a794qm7ptbxh3diu1af6oays7clf3kobs...@mail.gmail.com/
I guess we can try to report this as a bug, especially since it also
manifests itself when comint is used interactively.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-16 Thread Matt
alaincochard alain.coch...@unistra.fr writes:

 > Yes.  (So I think that explains the 'no' above.) 

Thank you for clarifying.

 > Anyway, this gives me the opportunity to come back to the question of
 > whether or not there is a problem with emacs itself (like some people
 > here thought), and if some message should be sent to some emacs list.
 > Again my argument was that the 2 commands copy/yank'ed in an emacs
 > terminal don't work as expected, while they do if the same is done in
 > an X terminal.

Okay, I follow you now.

Yes, I agree with what others have said, it's related to Emacs (probably 
comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's hard 
to reproduce with M-x term because term-char-mode doesn't allow copy-paste.  
M-x with term-line-mode doesn' t print "bar".

 > I guess it is not important for Org, because workarounds have be
 > proposed, but I think if there is a problem with emacs it should be
 > reported, right?

I think it's worth reporting.  I'm headed to bed and can submit it tomorrow for 
you if you're not comfortable with doing it or don't have the time.

Here are the steps I followed to reproduce with M-x shell:

1. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

2. Paste them into a shell (tested with xterm or xfce4-terminal) and press 
return
3. After entering the correct password, "bar" is echoed
4. Paste them into the buffer associated with M-x shell
5. After entering the correct password, "bar" is NOT echoed

Here are the steps I followed to reproduce with M-x term:

1. M-x term
2. C-c C-j to switch to term-line-mode
3. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

4. Paste them into the buffer associated with M-x term
5. After entering the correct password, "bar" is NOT echoed



Re: bash source code block: problem after ssh commands

2023-11-16 Thread Alain . Cochard
Matt writes on Thu 16 Nov 2023 20:03:
 > 
 >   On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 
 > 
 >  > Or we can make use of TRAMP as we usually do to access remote
 >  > environment. IMHO, it is more reliable as TRAMP takes care about
 >  > arranging all the Elisp FS interaction to work on remote system.
 >  
 > Agreed.  Also, I wasn't aware TRAMP worked with shell block
 > evaluation.  That's nice.
 > 
 > Sure enough, these both work:
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user 
 > :session *remote*
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > Of course, both calls happen on the remote.
 > 
 > I wonder, would this have helped Alain?

No (see below).

 > I'm admittedly a little confused about the original report,  
 > 
 > #+name: original-report
 > #+begin_src bash :results output
 > ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
 > echo "bar"
 > #+end_src
 > 
 > Clearly, the ssh line is intended to happen on the remote server.
 > Was the echo "bar" intended to happen locally?

Yes.  (So I think that explains the 'no' above.)  Then I'm confused
about your confusion...

Anyway, this gives me the opportunity to come back to the question of
whether or not there is a problem with emacs itself (like some people
here thought), and if some message should be sent to some emacs list.
Again my argument was that the 2 commands copy/yank'ed in an emacs
terminal don't work as expected, while they do if the same is done in
an X terminal.

I guess it is not important for Org, because workarounds have be
proposed, but I think if there is a problem with emacs it should be
reported, right?

Thank you.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-11-16 Thread Matt


  On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 

 > Or we can make use of TRAMP as we usually do to access remote
 > environment. IMHO, it is more reliable as TRAMP takes care about
 > arranging all the Elisp FS interaction to work on remote system.
 
Agreed.  Also, I wasn't aware TRAMP worked with shell block evaluation.  That's 
nice.

Sure enough, these both work:

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session 
*remote*
echo foo>foo_file
echo "bar"
#+end_src

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user
echo foo>foo_file
echo "bar"
#+end_src

Of course, both calls happen on the remote.

I wonder, would this have helped Alain?

I'm admittedly a little confused about the original report,  

#+name: original-report
#+begin_src bash :results output
ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src

Clearly, the ssh line is intended to happen on the remote server.  Was the echo 
"bar" intended to happen locally?

PS:  I've updated the WORG page for ob-shell with a :dir section 
(https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54)



Re: bash source code block: problem after ssh commands

2023-11-16 Thread Ihor Radchenko
Matt  writes:

> So, rather than start a comint and then ssh from there, it's possible to let 
> Emacs start the subprocess, manage the ssh connection, and just read that.  
> Emacs makes a buffer from a network stream by calling start-process` using 
> `shell-file-name`.  `org-babel-shell-initialize` closes around 
> `shell-file-name` with whatever shell language is used.  Therefore, we could 
> provide header arguments to pass parameters and the destination to ssh.  

Or we can make use of TRAMP as we usually do to access remote
environment. IMHO, it is more reliable as TRAMP takes care about
arranging all the Elisp FS interaction to work on remote system.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-15 Thread Matt


  On Wed, 15 Nov 2023 17:32:19 +0100  Matt  wrote --- 

 > (let ((buff "*localhost-process-buffer*"))
 >   (switch-to-buffer
 >(make-comint-in-buffer
 > "localhost-process"
 > buff
 > "ssh"
 > nil
 > (format "%s@localhost" (getenv "USER")

It looks like the way this is called actually uses `start-file-process` rather 
than `open-network-stream`.  It's not clear to me how to call it so that 
`open-network-stream` is used.  However, that's more a question for the Emacs 
mailing list and, honestly, not one that I think I care about right now.  
Posting this just to correct what I said before.



Re: bash source code block: problem after ssh commands

2023-11-15 Thread Matt


I was poking around, learning how sessions are started.  Basically, `shell` 
creates the comint buffer using `make-comint-in-buffer`.  What I find 
interesting is that `make-comint-in-buffer` can also create a comint buffer 
from a network stream:

(let ((buff "*localhost-process-buffer*"))
  (switch-to-buffer
   (make-comint-in-buffer
"localhost-process"
buff
"ssh"
nil
(format "%s@localhost" (getenv "USER")

So, rather than start a comint and then ssh from there, it's possible to let 
Emacs start the subprocess, manage the ssh connection, and just read that.  
Emacs makes a buffer from a network stream by calling start-process` using 
`shell-file-name`.  `org-babel-shell-initialize` closes around 
`shell-file-name` with whatever shell language is used.  Therefore, we could 
provide header arguments to pass parameters and the destination to ssh.  

I haven't made a judgment yet about whether any of this is good or bad.   I 
thought it was interesting and figured I'd share.



Re: bash source code block: problem after ssh commands

2023-11-09 Thread Matt


  On Thu, 09 Nov 2023 13:13:36 +0100  Ihor Radchenko  wrote --- 

 > A bigger problem is that errors *Org Babel Error
 > Output* are not clickable:
 
Interesting.

 > So, it looks like we may need an alternative `org-babel-eval' function
 > that works with script files rather than input and arranges
 > stdout/stderr separation.

Yes, it seems like a separate handler may be in order.

 > We should reuse `process-file' in this scenario as well,
 > but utilizing its ARGS argument + arrange to generate the temporary
 > script file in the appropriate context - see `make-nearby-temp-file'.

I was unaware of `make-nearby-temp-file'.  Thank you.  It's helpful for this 
situation.

I'm not in a position to address these points right now.  My main focus is 
re-familiarizing myself with the =ob-shell= and =ob-comint= source.  When I was 
implementing async shell evaluatation earlier this year, I remember having the 
impression that execution paths, especially `org-babel-comint-with-output', 
could probably be simplified.  It seems to me that everything we're talking 
about in this thread aligns with that objective.  I've made a note of your 
points and have every intention to return to them when I get to writing code.  
However, don't let me stop you if this is something you were interested in 
doing.  I'll do what I can to assist.



Re: bash source code block: problem after ssh commands

2023-11-09 Thread Ihor Radchenko
Matt  writes:

> ... Aside from permission, any code we
> insert needs to be correct. For example, a shebang would need to point
> to the correct application and any arguments would need to correspond
> to the implementation being called. I doubt we'd need anything beyond
> /bin/. FWIW, it looks like there's been at least one instance
> where :shebang's formatting was questioned
> (https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/).

It is a problem, but I am not sure how useful it is to solve the line
numbering there. A bigger problem is that errors *Org Babel Error
Output* are not clickable:

Try
#+begin_src bash
cdf
#+end_src
and you will get
-- *Org Babel Error Output* -
bash: line 1: cdf: command not found
[ Babel evaluation exited with code 127 ]
-

Clicking on "line 1" will yield file opening dialogue.

However, there is yet another problem revealed when we try

#+begin_src bash :cmdline bash
cdf
#+end_src

#+RESULTS:
: /tmp/babel-47DarV/sh-script-vkbRfJ: line 2: cdf: command not found

There is no *Org Babel Error Output* buffer at all.

So, it looks like we may need an alternative `org-babel-eval' function
that works with script files rather than input and arranges
stdout/stderr separation.

> We'd also need to control for what environment the script runs in.
> That was another issue I saw raised in the list
> (https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/). Of
> course, we'd need to read the stdout and stderr. This is handled by
> =process-file=. Any step I missed or some kind of failure I didn't
> consider?

Valid concern. We should reuse `process-file' in this scenario as well,
but utilizing its ARGS argument + arrange to generate the temporary
script file in the appropriate context - see `make-nearby-temp-file'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-08 Thread Matt


  On Tue, 07 Nov 2023 09:53:46 +0100  Ihor Radchenko  wrote --- 
 > Matt m...@excalamus.com> writes:
 > 
 > >   On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > >  > I am wondering about the possible downsides of using script approach
 > >  > instead of stdin redirection.
 > >  
 > > I'm curious to hear more about what you're thinking.
 > 
 > I am thinking to change the
 >   (t (org-babel-eval shell-file-name (org-trim body)))
 > clause in `org-babel-sh-evaluate' to something that uses a script file.
 > 
 > It will clearly solve the discussed problem, possibly at the cost of
 > small overhead to write the script file to disk.
 
Thanks for clarifying.  I've been away from the codebase for a bit and, now 
that the FSF paperwork is signed (still need to get Craig a copy), I'm 
reviewing =ob-shell= with an eye for what could be cleaned or improved.   I 
feel like =org-babel-sh-evaluate= could use some attention.

I'm open to your suggestion.  The good is that using a script is how :shebang 
and :cmdline are processed currently (like you pointed out) so there's 
precedence and experience with it.  Also, it would make all non-session 
execution use the same model (script versus comint).  I like how that would 
create a clean separation.

For bad, nothing jumps out to me as obviously a problem.  Let me "think out 
loud" for a moment.  We'd need to write to disk.  Like you say, this incurs 
overhead opening, writing, and closing the file.  It's not like we'd forget to 
close it, though.  Nor is running out of space or inodes our problem.  Writing 
requires permission.  That's not an issue with /tmp.  Then, it needs to 
execute.  Aside from permission, any code we insert needs to be correct.  For 
example, a shebang would need to point to the correct application and any 
arguments would need to correspond to the implementation being called.  I doubt 
we'd need anything beyond /bin/.   FWIW, it looks like there's been at 
least one instance where :shebang's formatting was questioned 
(https://yhetil.org/orgmode/ca+a2izz1vmmkiuf4fem1au7ca1m9gqap+bkvrosz+0bxrt6...@mail.gmail.com/).
  We'd also need to control for what environment the script runs in.  That was 
another issue I saw raised in the list 
(https://yhetil.org/orgmode/87609ug5ae@luisa.c0t0d0s0.de/).  Of course, 
we'd need to read the stdout and stderr.  This is handled by =process-file=.  
Any step I missed or some kind of failure I didn't consider?




Re: bash source code block: problem after ssh commands

2023-11-07 Thread Ihor Radchenko
Matt  writes:

>   On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
>  > I am wondering about the possible downsides of using script approach
>  > instead of stdin redirection.
>  
> I'm curious to hear more about what you're thinking.

I am thinking to change the
  (t (org-babel-eval shell-file-name (org-trim body)))
clause in `org-babel-sh-evaluate' to something that uses a script file.

It will clearly solve the discussed problem, possibly at the cost of
small overhead to write the script file to disk.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-11-06 Thread Alain . Cochard
Matt writes on Mon  6 Nov 2023 19:01:

 > I want to confirm, are you able to accomplish your task by using
 > '-n'?

Yes.  Evaluating this block (with C-c C-c):

   #+begin_src bash :results output
   ssh -n coch...@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src

gives

   #+RESULTS:
   : bar

(without the '-n', #+RESULTS: is empty).  Same conclusion if I
copy/yank the 2 bash instructions in an emacs terminal (M-x shell). 

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-11-06 Thread Matt


  On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > I am wondering about the possible downsides of using script approach
 > instead of stdin redirection.
 
I'm curious to hear more about what you're thinking.



Re: bash source code block: problem after ssh commands

2023-11-06 Thread Matt


  On Fri, 27 Oct 2023 13:47:24 +0200  alain.coch...@unistra.fr wrote --- 
 > Russell Adams writes on Thu 26 Oct 2023 16:44:
 >  > [...] are you using an SSH key to connect, or entering a password?
 >  > If you use an SSH key for passwordless access, try adding -n (ie:
 > 
 >  > "ssh -n derp@host mycommand"). 
 > 
 > Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.
 
Hi Alain,

I'm the supposed maintainer of ob-shell who's been missing in action for this 
whole discussion.

I want to confirm, are you able to accomplish your task by using '-n'?

Matt



Re: bash source code block: problem after ssh commands

2023-11-06 Thread Ihor Radchenko
Bruno Barbier  writes:

> IIUC, the elisp expression:
>
> (process-file "bash" "/tmp/test.sh")
>
> is more equivalent to:
>
> cat /tmp/test.sh | bash
>
> i.e. the shell is getting the commands from stdin.  Thus, any command
> that uses stdin might change what gets executed or not.

Looking at `org-babel-sh-evaluate', it should be enough to specify
:shebang, :cmdline, or :stdin header argument to force using script
rather than channel source block as input to bash:

#+begin_src shell :results output :cmdline bash
exec 0>&-
echo OK
#+end_src

#+RESULTS:
: OK


#+begin_src shell :results output :cmdline bash
echo 1
read -p "Next command? " NEXT_COMMAND
echo 2
echo 3
#+end_src

#+RESULTS:
: 1
: 2
: 3

I am wondering about the possible downsides of using script approach
instead of stdin redirection.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-10-30 Thread Bruno Barbier


Hi,

alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 13:44:
>
>  > I can now reproduce the problem locally.
>  > 
>  > It boils down to
>  > 
>  > (setq exit-status
>  >  (process-file shell-file-name input-file
>  >(if error-file
>  >(list t error-file)
>  >  t)
>  >nil shell-command-switch command))
>  > 
>  > that is an equivalent of
>  > 
>  > bash -c bash /path/to/file-containing-the-source-code.sh
>  > 
...
> I am confused about what you specifically do to "evaluate the above".
> To start with, I have to use quotes to make your command be performed:
>
>bash -c "bash /path/to/file-containing-the-source-code.sh"
>
...
>
> But most importantly, the second line *is* produced, either if I use
> an SSH key for passwordless access or if I enter the password
> manually.

IIUC, the elisp expression:

(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash

i.e. the shell is getting the commands from stdin.  Thus, any command
that uses stdin might change what gets executed or not.

I'm able to reproduce using the following minimal script, without
passwords nor SSH (where the cryptic first line closes stdin).

#+begin_src shell :results output
  exec 0>&-
  echo OK
#+end_src

The result is "OK" only when commenting out the first line; else, the
echo command is not executed (because stdin has been closed).

Here is an other example, where the second echo is eaten by the script
itself:

#+begin_src shell :results output
  echo 1
  read -p "Next command? " NEXT_COMMAND
  echo 2
  echo 3
#+end_src

#+RESULTS:
: 1
: 3

Bruno



Re: bash source code block: problem after ssh commands

2023-10-27 Thread Max Nikulin

On 28/10/2023 01:26, alain.coch...@unistra.fr wrote:

bash -c "bash /path/to/file-containing-the-source-code.sh"

Without the quotes, the 1st line is not executed either.  NB: I get
the same behavior with simply


Without quotes it means: execute shell script (just "bash" this case) 
with "$0" set to "/path/to/file-containing-the-source-code.sh". 
"/path/..." is ignored because shell script does not contain "$0". It is 
a way to safely pass arguments to shell, however "$0" is confusing:


sh -c 'echo "arg0: $0"; echo  args: "$@"' arg0 arg1 arg2
arg0: arg0
args: arg1 arg2

In this example "arg0: $0"; echo  args: "$@"'  has the same role as 
"bash" in the cited command.


If you need to pass another variable into "sh -c" the use

var="/bin/sh"
sh -c 'ls -l "$1"' argv0-for-sh "$var"

Unsafe variant ("var" contain quotes): sh -c "ls -l '$var'". The same 
warning is applicable to "emacs --eval "(func \"$var\")".



or, after making the script executable with 'chmod +x':

/path/to/file-containing-the-source-code.sh


Always add a shebang ("#!/bin/sh", "#!/bin/bash", etc.) to executable 
scripts. A chance to face a bug is a shell is higher when it is left up 
to the shell and the kernel what to do with files without shebangs.


Sorry, I have not tried to debug the original issue.



Re: bash source code block: problem after ssh commands

2023-10-27 Thread Alain . Cochard
Ihor Radchenko writes on Thu 26 Oct 2023 13:44:

 > I can now reproduce the problem locally.
 > 
 > It boils down to
 > 
 > (setq exit-status
 >(process-file shell-file-name input-file
 >  (if error-file
 >  (list t error-file)
 >t)
 >  nil shell-command-switch command))
 > 
 > that is an equivalent of
 > 
 > bash -c bash /path/to/file-containing-the-source-code.sh
 > 
 > - file-containing-the-source-code.sh 
 >  ssh localhost 'echo foo>/tmp/foo_file'
 >   echo $(uname -a) |tee /tmp/uname1.txt
 > -

 > If one tries to evaluate the above, the second line is not produced. It
 > has nothing to do with Emacs itself.

I am confused about what you specifically do to "evaluate the above".
To start with, I have to use quotes to make your command be performed:

   bash -c "bash /path/to/file-containing-the-source-code.sh"

Without the quotes, the 1st line is not executed either.  NB: I get
the same behavior with simply

   bash /path/to/file-containing-the-source-code.sh

or, after making the script executable with 'chmod +x':

   /path/to/file-containing-the-source-code.sh

But most importantly, the second line *is* produced, either if I use
an SSH key for passwordless access or if I enter the password
manually.

 > However, if I try
 > 
 > (process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
 > produced.

Here too, it is not clear to me how you "try".  The way I know is to
highlight

   (process-file "bash" "/path/to/file-containing-the-source-code.sh")

and do

   M-x eval-region

In this case, indeed, the 1st line is executed but not the 2nd one
(again, whether or not I use an SSH key).


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-10-27 Thread Alain . Cochard
Russell Adams writes on Thu 26 Oct 2023 16:44:

 > [...] are you using an SSH key to connect, or entering a password?

 > If you use an SSH key for passwordless access, try adding -n (ie:
 > "ssh -n derp@host mycommand"). 

Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-10-26 Thread Russell Adams
On Wed, Oct 25, 2023 at 01:17:42PM +0200, alain.coch...@unistra.fr wrote:
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src

I know that Ihor has already reproduced, but are you using an SSH key
to connect, or entering a password?

If entering a password, I'd expect there is some mixup. That I defer
to Ihor.

If you use an SSH key for passwordless access, try adding -n (ie: "ssh
-n derp@host mycommand"). The "-n" flag helps prevent the remote
command from interfering with your local terminal by redirecting
/dev/null as the remote stdin. I have to use this commonly in systems
administration, scripts, and tools like Ansible.

Thanks.

--
Russell Adamsrlad...@adamsinfoserv.com
https://www.adamsinfoserv.com/



Re: bash source code block: problem after ssh commands

2023-10-26 Thread Ihor Radchenko
alain.coch...@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 08:44:
>
>  > I currently do not have a setup to test ssh commands,
>
> Sorry if that's irrelevant: I realized that one can ssh to the *same*
> machine.

You are indeed right.
I can now reproduce the problem locally.

It boils down to

(setq exit-status
  (process-file shell-file-name input-file
(if error-file
(list t error-file)
  t)
nil shell-command-switch command))

that is an equivalent of

bash -c bash /path/to/file-containing-the-source-code.sh

- file-containing-the-source-code.sh 
 ssh localhost 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
-

If one tries to evaluate the above, the second line is not produced. It
has nothing to do with Emacs itself.

However, if I try

(process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
produced.

This time, it is Emacs problem.
The problem is caused by interactive password prompt displayed by ssh.

I guess we may try to report this upstream, though I am not sure if this
is something that is supported to start with.

>From Org perspective, it is not the first time when interactive shell
programs are causing problems. I'd say that we have no technical ability
to support them reliably and one has to use ob-screen or similar
libraries to properly interact with such code blocks.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-10-26 Thread Alain . Cochard
Ihor Radchenko writes on Thu 26 Oct 2023 08:44:

 > I currently do not have a setup to test ssh commands,

Sorry if that's irrelevant: I realized that one can ssh to the *same*
machine.



Re: bash source code block: problem after ssh commands

2023-10-26 Thread Ihor Radchenko
Leo Butler  writes:

> It looks like an issue in comint-mode and the way that it is handling
> the temporary ssh session. This is not an issue with org-mode.
> ...
> Not working:
>
> #+begin_src bash :results output
>   ssh SSH-HOST 'echo foo>/tmp/foo_file'
>   echo $(uname -a) |tee /tmp/uname1.txt
> #+end_src

By default (no session), the above command should not use comint-mode.
So, something else might be going on here.

I currently do not have a setup to test ssh commands, so I would
appreciate someone trying to dig why `org-babel-eval' apparently not
returning the expected result.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 



Re: bash source code block: problem after ssh commands

2023-10-25 Thread yaxp




So it is an issue with emacs, right?


comint-mode

--
(yaxp me) => t




Re: bash source code block: problem after ssh commands

2023-10-25 Thread Leo Butler
On Wed, Oct 25 2023, alain.coch...@unistra.fr wrote:

> Thanks much for the detailed explanations.
>
>  > It looks like an issue in comint-mode and the way that it is
>  > handling the temporary ssh session. This is not an issue with
>  > org-mode.
>
> So it is an issue with emacs, right?

Yes, that is my understanding. Perhaps you can report it to the
emacs-devel mailing list.

Leo


Re: bash source code block: problem after ssh commands

2023-10-25 Thread Alain . Cochard


Thanks much for the detailed explanations.

 > It looks like an issue in comint-mode and the way that it is
 > handling the temporary ssh session. This is not an issue with
 > org-mode.

So it is an issue with emacs, right?

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.coch...@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France | [ slot available for rent ]




Re: bash source code block: problem after ssh commands

2023-10-25 Thread Leo Butler
On Wed, Oct 25 2023, alain.coch...@unistra.fr wrote:

> Hello.
>
> For me, 'C-c C-c' on the following group
>
>#+begin_src bash :results output
>echo "foo"
>echo "bar"
>#+end_src
>
>
> works as I expect: 'foo' and 'bar' are echo'ed.  But it fails on this
> one:
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file"
>echo "bar"
>#+end_src
>
>
> The file 'foo_file' is created on the remote machine, but 'bar' is not
> echo'ed.  I have tried to insert other commands between the ssh and
> echo commands, or append semicolons at the end of the commands,
> without success.
>
> By contrast, it works with this one:
>
>#+begin_src bash :results output
>ssh coch...@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
>#+end_src
>
> Perhaps it is worth noting that I observe the same behavior if I
> simply copy/yank the commands (I mean: the two commands yank'ed at
> once) in an emacs terminal (obtained with 'M-x shell').  However,
> everything works as I expect if I copy/yank the commands (all at once)
> in a "regular" X terminal (xfce in my case).

It looks like an issue in comint-mode and the way that it is handling
the temporary ssh session. This is not an issue with org-mode.

Here are a couple work-arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Working examples:

#+begin_src bash :results output
  ssh -f SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) 
x86_64 GNU/Linux

#+begin_src bash :results output
  ssh pearce-120 'echo foo>/tmp/foo_file' &&\
  echo $(uname -a) | tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) 
x86_64 GNU/Linux

Not working:

#+begin_src bash :results output
  ssh SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
#+end_src

If you look at /tmp after running the third example, you see that that
second command line has not been executed.

Leo