Re: bash sets O_NONBLOCK on pts

[Stephane Chazelas]

2019-10-03 02:49:36 +0900, Andrew Church:
> >Well, it's not so uncommon, I had it a few times. Reading on internet
> >it seems that other users have it but don't notice it.
> 
> The fault could be in some other program accessing the terminal.  Bash
> does not clear O_NONBLOCK on displaying a prompt, so if a previously
> executed program sets O_NONBLOCK on stdin and then exits, that state
> will remain until some other program unsets it.  For example:
> 
> $ cat >foo.c
> #include 
> int main(void) {fcntl(0, F_SETFL, O_NONBLOCK); return 0;}
> ^D
> $ cc foo.c
> $ ./a.out
> $ cat
> cat: -: Resource temporarily unavailable
[...]

Good point.

I see a difference between versions of bash there:

With GNU dd:

~$ bash5 --norc
bash5-5.0$ dd iflag=nonblock
dd: error reading 'standard input': Resource temporarily unavailable
0+0 records in
0+0 records out
0 bytes copied, 0.000150515 s, 0.0 kB/s
bash5-5.0$ cat
^C
bash5-5.0$ exit

~$ bash --norc
bash-4.4$ dd iflag=nonblock
dd: error reading 'standard input': Resource temporarily unavailable
0+0 records in
0+0 records out
0 bytes copied, 0.000126312 s, 0.0 kB/s
bash-4.4$ cat
cat: -: Resource temporarily unavailable

In bash5, with strace, we see:

fcntl(0, F_GETFL)   = 0x8802 (flags 
O_RDWR|O_NONBLOCK|O_LARGEFILE)
fcntl(0, F_SETFL, O_RDWR|O_LARGEFILE)   = 0

That seems to be done by sh_unset_nodelay_mode()

Which points to this change:

commit bc371472444f900d44050414e3472f7349a7aec7
Author: Chet Ramey 
Date:   Mon Jan 30 15:50:08 2017 -0500

commit bash-20170127 snapshot

diff --git a/CWRU/CWRU.chlog b/CWRU/CWRU.chlog
index b8436d64..74a0463e 100644
--- a/CWRU/CWRU.chlog
+++ b/CWRU/CWRU.chlog
@@ -13027,3 +13027,21 @@ subst.c
  after reading a double-quoted string, make sure the W_NOCOMSUB and
  W_NOPROCSUB flags make it to the recursive invocation.  Fixes bug
  reported by Jens Heyens 
+
+  1/23
+  
+lib/readline/signals.c
+   - _rl_orig_sigset: original signal mask, set and restored by
+ rl_set_signals (rl_clear_signals doesn't block signals).  If we
+ are not installing signal handlers, just save signal mask each
+ time rl_set_signals is called
+
+lib/readline/input.c
+   - rl_getc: use _rl_orig_sigmask in the call to pselect(), so we block
+ the set of signals originally blocked by the calling application.
+ Fixes bug reported by Frédéric Brière 
+
+parse.y
+   - yy_readline_get: try to unset NONBLOCK mode on readline's input
+ file descriptor before calling readline(). Inspired by report from
+ Siteshwar Vashisht 



Given that the OP is running 5.0.7, they should have that change already.

Maybe the fcntl(O_NONBLOCK) is done by a command run by a completion
widget.

-- 
Stephane

Re: bash sets O_NONBLOCK on pts

[Matteo Croce]

On Wed, Oct 2, 2019 at 8:16 PM Andreas Schwab  wrote:
>
> On Okt 02 2019, Matteo Croce  wrote:
>
> > It seems to me that bash restores the flag, cat prints an error when not:
> >
> > $ cat
> > $
> >
> > the same is not true if running multiple commands:
> >
> > $ ./foo; cat
> > cat: -: Resource temporarily unavailable
> > $
> >
> > Why this different behaviour?
>
> Because it's reset immediately before reading the next line with
> readline.
>
> Andreas.
>
> --
> Andreas Schwab, sch...@linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."

Exactly, while in my bug I can do any command I want without resetting it.
Only `exec bash` restores it.

-- 
Matteo Croce
per aspera ad upstream

Re: bash sets O_NONBLOCK on pts

[Andreas Schwab]

On Okt 02 2019, Matteo Croce  wrote:

> It seems to me that bash restores the flag, cat prints an error when not:
>
> $ cat
> $
>
> the same is not true if running multiple commands:
>
> $ ./foo; cat
> cat: -: Resource temporarily unavailable
> $
>
> Why this different behaviour?

Because it's reset immediately before reading the next line with
readline.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: bash sets O_NONBLOCK on pts

[Matteo Croce]

On Wed, Oct 2, 2019 at 7:50 PM Andrew Church  wrote:
>
> >Well, it's not so uncommon, I had it a few times. Reading on internet
> >it seems that other users have it but don't notice it.
>
> The fault could be in some other program accessing the terminal.  Bash
> does not clear O_NONBLOCK on displaying a prompt, so if a previously
> executed program sets O_NONBLOCK on stdin and then exits, that state
> will remain until some other program unsets it.  For example:
>
> $ cat >foo.c
> #include 
> int main(void) {fcntl(0, F_SETFL, O_NONBLOCK); return 0;}
> ^D
> $ cc foo.c
> $ ./a.out
> $ cat
> cat: -: Resource temporarily unavailable
>
>   --Andrew Church
> http://achurch.org/

It seems to me that bash restores the flag, cat prints an error when not:

$ cat
$

the same is not true if running multiple commands:

$ ./foo; cat
cat: -: Resource temporarily unavailable
$

Why this different behaviour?

Regards,

--
Matteo Croce
per aspera ad upstream

Re: bash sets O_NONBLOCK on pts

[Andrew Church]

>Well, it's not so uncommon, I had it a few times. Reading on internet
>it seems that other users have it but don't notice it.

The fault could be in some other program accessing the terminal.  Bash
does not clear O_NONBLOCK on displaying a prompt, so if a previously
executed program sets O_NONBLOCK on stdin and then exits, that state
will remain until some other program unsets it.  For example:

$ cat >foo.c
#include 
int main(void) {fcntl(0, F_SETFL, O_NONBLOCK); return 0;}
^D
$ cc foo.c
$ ./a.out
$ cat
cat: -: Resource temporarily unavailable

  --Andrew Church
http://achurch.org/

Re: bash sets O_NONBLOCK on pts

[Matteo Croce]

On Wed, Oct 2, 2019 at 4:28 PM Chet Ramey  wrote:
>
> On 10/2/19 8:27 AM, Matteo Croce wrote:
>
> > Bash Version: 5.0
> > Patch Level: 7
> > Release Status: release
> >
> > Description:
> > Sometimes bash leaves the pts with O_NONBLOCK set, and all programs
> > reading from stdin will get an EAGAIN:
>
> Without a way to reliably reproduce this, it's just a transient issue.
>

Well, it's not so uncommon, I had it a few times. Reading on internet
it seems that other users have it but don't notice it.
A symptom is an interactive tool like apt which exits at the first prompt.

-- 
Matteo Croce
per aspera ad upstream

Re: bash sets O_NONBLOCK on pts

[Stephane Chazelas]

2019-10-02 14:27:48 +0200, Matteo Croce:
[...]
> Sometimes bash leaves the pts with O_NONBLOCK set, and all programs
> reading from stdin will get an EAGAIN:
[...]

Can you reproduce it with

   bash --norc

Or with:

   INPUTRC=/dev/null bash --norc

?

If you could reproduce it with:

   strace -o strace.log bash --norc

that would allow us to see where a O_NONBLOCK flag is set and
not reset.

BTW, what's the point of the check_dev_tty() function? It seems
it just attempts to open the tty (the controlling one or the one
open on stdin), closes it, but doesn't return anything about the
success of failure in doing so.

On my system (Debian amd64, 5.0.3(1)-release, bash started from
a regular terminal emulator), it's the only place where I see
O_NONBLOCK being used (and that's on a new fd that is closed
straight after, so it could not have any bearing on the OP's
issue).

-- 
Stephane

Re: shouldn't it the comma operator has the lowerest precedence intheshell arithmetic expression?

[Chet Ramey]

On 10/2/19 10:50 AM, hkadeveloper wrote:
> Yes. I understand what you are saying. I mean isn’t it a little
> inconsistent about the comment, the macro name(EXP_HIGHEST) and the macro
> value(expcomma)?

I know what it means. ;-)

It's just the one you call first to start the recursive descent.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: shouldn't it the comma operator has the lowerest precedence intheshell arithmetic expression?

[hkadeveloper]

   Yes. I understand what you are saying. I mean isn’t it a little
   inconsistent about the comment, the macro name(EXP_HIGHEST) and the
   macro value(expcomma)?

   Thanks for you reply!

   -- Original --
   From: "Chet Ramey" ; <"Chet Ramey"
   ;>
   Date: Wed,Oct 2,2019 10:28 PM
   To: hkadeveloper 
   Subject: Re: shouldn't it the comma operator has the lowerest
   precedence inthe shell arithmetic expression?
On 10/1/19 8:35 PM, hk wrote:
> Configuration Information :
> Bash Version: 5.0
> Patch Level: 0
> Release Status: release
>
> Description:
> the code snippet from expr.c starting from line 141:
>
>> /* This should be the function corresponding to the operator with the
>>highest precedence. */
>> #define EXP_HIGHEST expcomma
>
>
> Am I understanding it wrong or is it a typo?

The bash arithmetic parser does things in reverse order, in a way. So
the comma operator is the first thing you call, and it calls functions
that implement the other operators in ascending priority order. You
didn't misunderstand it.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: bash sets O_NONBLOCK on pts

[Chet Ramey]

On 10/2/19 8:27 AM, Matteo Croce wrote:

> Bash Version: 5.0
> Patch Level: 7
> Release Status: release
> 
> Description:
> Sometimes bash leaves the pts with O_NONBLOCK set, and all programs
> reading from stdin will get an EAGAIN:

Without a way to reliably reproduce this, it's just a transient issue.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

bash sets O_NONBLOCK on pts

[Matteo Croce]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-Wno-parentheses -Wno-format-security
uname output: Linux mcroce-redhat 5.3.1-matteo #87 SMP Mon Sep 30
14:20:06 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 5.0
Patch Level: 7
Release Status: release

Description:
Sometimes bash leaves the pts with O_NONBLOCK set, and all programs
reading from stdin will get an EAGAIN:

# cat
cat: -: Resource temporarily unavailable

# strace -e read cat
read(0, 0x7f2a70511000, 131072) = -1 EAGAIN (Resource
temporarily unavailable)
: Resource temporarily unavailable
+++ exited with 1 +++

Initially I thought it was a kernel bug, so I filed a bug on the
kernel bugzilla:

https://bugzilla.kernel.org/show_bug.cgi?id=201641

but then I tested with fcntl() that the stding had te flag set.

Repeat-By:
random after some usage

Fix:
`exec bash` resolves it

-- 
Matteo Croce
per aspera ad upstream

Re: shouldn't it the comma operator has the lowerest precedence in the shell arithmetic expression?

[Chet Ramey]

On 10/1/19 8:35 PM, hk wrote:
> Configuration Information :
> Bash Version: 5.0
> Patch Level: 0
> Release Status: release
> 
> Description:
> the code snippet from expr.c starting from line 141:
> 
>> /* This should be the function corresponding to the operator with the
>>highest precedence. */
>> #define EXP_HIGHEST expcomma
> 
> 
> Am I understanding it wrong or is it a typo?

The bash arithmetic parser does things in reverse order, in a way. So
the comma operator is the first thing you call, and it calls functions
that implement the other operators in ascending priority order. You
didn't misunderstand it.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: shebang-less script execution not resetting some options

[L A Walsh]

On 2019/10/02 03:49, Ilkka Virta wrote:
>
>  If the execl() function fails due to an error equivalent to the
>  [ENOEXEC] [...] the shell shall execute a command equivalent to
>  having a shell invoked with the pathname resulting from the
>  search as its first operand, [...]
>
> [ 
> https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html#tag_18_09_01_01
>  
> ]
>
> -
>
> I think that last sentence above is the relevant part. The standard only 
> says to "invoke a shell". It doesn't say which shell, probably because 
> it only specifies one.
>   
POSIX probably doesn't mandate secure passwords, firewalls, malware
detection,
etc.  Just because posix doesn't require it, doesn't mean it is a good idea
to ignore it.

Re: shebang-less script execution not resetting some options

[Greg Wooledge]

On Wed, Oct 02, 2019 at 03:11:18AM -0700, L A Walsh wrote:
> > Bash allows a child of itself (a subshell) to read the commands.
> > GNU find -exec uses /bin/sh to run it.
> > zsh and csh both use /bin/sh to run it, I think.

> So if a user has 'rbash' in /etc/passwd, they might get a real shell
> because various programs ignore what /etc/passwd says?

Well, yeah.  If a user in a "restricted shell" is allowed to run bash,
or find, or zsh, or csh, then they can "escape" from the "restricted
shell" trivially.

It's exactly the same as if you allowed them to run vi, or emacs, or
less, or nethack, or any other program that has a shell escape.  Or perl,
or tclsh, or gcc, or any other programming environment that lets them
write a program that runs an arbitrary shell command.

This is why "restricted shells" are basically a joke.  And not even a
funny one.  Nobody has taken them seriously as a security measure in
decades.

But that's completely unrelated to the original question, which was
"how does ___ handle a script that's missing the shebang".  The answer
to that is, "everyone handles it differently".

I did some more poking around after my previous email in this thread.
Various people have written "ployglot scripts" that can run under
different interpreters.  I chose this one

and placed it in a file and gave it execute permissions.

Then I played around with it, just to verify my recollections:

wooledg:~$ echo "$BASH_VERSION"
5.0.3(1)-release
wooledg:~$ which_interpreter
bash 5.0.3(1)-release
wooledg:~$ zsh
wooledg:~% which_interpreter
POSIX shell
wooledg:~% exit
wooledg:~$ tcsh
wooledg:~> which_interpreter
POSIX shell
wooledg:~> exit
exit
wooledg:~$ find .bashrc -exec which_interpreter {} \;
POSIX shell

Re: ARGV[@] Not Always Populated

[Chet Ramey]

On 10/1/19 6:33 PM, Adam Danischewski wrote:

> If you don't access the ARGV array before you drop into a function, ARGV is
> not populated.
>   [...]
> If you uncomment the echo at the top of the script then the printing is as
> expected.
> I realize the Bash Documentation states:
>BASH_ARGV
>   An  array variable containing all of the parameters in the
> current bash
>   execution call stack.  The final parameter of the last
> subroutine  call
>   is  at the top of the stack; the first parameter of the
> initial call is
>   at the bottom.  When a subroutine is executed, the parameters
>  supplied
>   are  pushed  onto BASH_ARGV.  The shell sets BASH_ARGV only
> when in ex‐
>   tended debugging mode (see the description of the  extdebug
>  option  to
>   the shopt builtin below).  Setting extdebug after the shell
> has started
>   to execute a script, or
> *referencing this variable when extdebug is  not  set, may
> result in inconsistent values.*

By default, the array is lazily populated: if you reference BASH_ARGV, and
it hasn't been set up, it will be initialized using the current positional
parameters.

> Many people are using BASH_ARGV:
> e.g.
> https://stackoverflow.com/questions/2740906/how-to-access-command-line-arguments-of-the-caller-inside-a-function
> If it's not too much trouble I think a lot of people would be glad if you
> could make this work in a standardized way.

Doesn't that suggest that the best way to use it in a `standardized' way is
to run `shopt -s extdebug' at the script's top level, before trying to
reference BASH_ARGV?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: shebang-less script execution not resetting some options

[Ilkka Virta]

On 2.10. 13:11, L A Walsh wrote:

On 2019/10/01 05:41, Greg Wooledge wrote:

On Tue, Oct 01, 2019 at 04:14:00AM -0700, L A Walsh wrote:
   

On 2019/09/30 14:39, Grisha Levit wrote:
 

A few of the recently-added shopt options aren't getting reset when
running a shebang-less script, this should fix it up:
   
   

Suppose the shebang-less script is being run by an earlier version
of bash.  Won't the new patch radically change the behavior of of
such programs?
 


Bash allows a child of itself (a subshell) to read the commands.
GNU find -exec uses /bin/sh to run it.
zsh and csh both use /bin/sh to run it, I think.
   


 So if a user has 'rbash' in /etc/passwd, they might get a real shell
because various programs ignore what /etc/passwd says?

 UmI suppose no one cares for one reason or another.


---
  2.9 Shell Commands
  Command Search and Execution

  If the command name does not contain any  characters, the first
  successful step in the following sequence shall occur:

  [a to d: functions, special builtins, stuff like that]

  e. Otherwise, the command shall be searched for using the PATH
 environment variable
 [...]
 b. Otherwise, the shell executes the utility in a separate utility
environment (see Shell Execution Environment) with actions
equivalent to calling the execl() function...

If the execl() function fails due to an error equivalent to the
[ENOEXEC] [...] the shell shall execute a command equivalent to
having a shell invoked with the pathname resulting from the
search as its first operand, [...]

[ 
https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html#tag_18_09_01_01 
]


-

I think that last sentence above is the relevant part. The standard only 
says to "invoke a shell". It doesn't say which shell, probably because 
it only specifies one.


Incidentally, it doesn't really specify the hashbang either. As far as I 
can tell, it only mentions it as one of two ways that implementations 
have "historically" recognized shell scripts. (The above being the other.)


[ 
https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/functions/execl.html 
]



As for rbash, does it matter? If you let a user exec() something, 
they'll get that binary, or the interpreter specified in the hashbang if 
it's a script. The kernel doesn't look at /etc/passwd to recognize rbash 
or such either, so if you want to restrict a user from launching 
particular commands, you'll have to do it before exec() is attempted.


That is to say, don't let those users run (other) unrestricted shells, 
or any of the various programs that allow forking off other programs, 
including find, xargs, many editors, etc.



--
Ilkka Virta / itvi...@iki.fi

Re: shebang-less script execution not resetting some options

[L A Walsh]

On 2019/10/01 05:41, Greg Wooledge wrote:
> On Tue, Oct 01, 2019 at 04:14:00AM -0700, L A Walsh wrote:
>   
>> On 2019/09/30 14:39, Grisha Levit wrote:
>> 
>>> A few of the recently-added shopt options aren't getting reset when
>>> running a shebang-less script, this should fix it up:
>>>   
>>>   
>> Suppose the shebang-less script is being run by an earlier version
>> of bash.  Won't the new patch radically change the behavior of of
>> such programs?
>> 
>
> Bash allows a child of itself (a subshell) to read the commands.
> GNU find -exec uses /bin/sh to run it.
> zsh and csh both use /bin/sh to run it, I think.
>   

So if a user has 'rbash' in /etc/passwd, they might get a real shell
because various programs ignore what /etc/passwd says?

UmI suppose no one cares for one reason or another.