Re: wait inside subshell waits for sibling

[Chet Ramey]

On 10/24/22 4:52 AM, Emanuele Torre wrote:

On 24/10/2022, Robert Elz  wrote:



There's no reason that wait (the command) cannot check for candidate
jobs which are still running, and wait for any of those, and then
return as soon as there are none left.   That the shell happens to
have forked other processes for its own reasons should make no
difference at all.


Process substitutions are considered background jobs, and the wait
builtin waits for them.


They're not jobs: they don't go in the jobs list, `jobs' doesn't report
on them, and fg/bg don't affect them. They're asynchronous processes
that happen to set $! (which may have been a mistake, but too late now).

`wait' can wait for them because they set $!, but as documented `wait'
without arguments only waits for the last one, and only if its pid is
the same as $!.

In POSIX terms, they are "known to the current shell execution
environment."

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: wait inside subshell waits for sibling

[Chet Ramey]

On 10/23/22 10:25 PM, Emanuele Torre wrote:

I don't think the process running `cat' is a sibling of the subshell.


It's not, but in this case it doesn't matter. `wait' without arguments
should wait for the last process substitution only if it's the same as
$!. You can wait for other process substitutions individually if you
save $! after they're created.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: wait inside subshell waits for sibling

[Oğuz]

27 Ekim 2022 Perşembe tarihinde Chet Ramey  yazdı:
>
> Yep, this is a problem. I'll have a fix in the next devel branch push.
>

Thanks


-- 
Oğuz

Re: wait inside subshell waits for sibling

[Chet Ramey]

On 10/23/22 3:47 AM, Oğuz İsmail Uysal wrote:

To reproduce:

    $ echo $BASH_VERSION
    5.2.2(3)-release
    $ ( : & wait ) > >(cat)
    *hangs*

It should return immediately, but hangs instead.


Yep, this is a problem. I'll have a fix in the next devel branch push.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: wait inside subshell waits for sibling

[Robert Elz]

Date:Mon, 24 Oct 2022 16:16:21 +0200
From:Emanuele Torre 
Message-ID:  


  | It does not make any sense that you keep insisting that this is about
  | bash internals, or low level system call (especially `wait', `fork',
  | etc),

Since that is the exact opposite of what I have been saying, it
seems that you haven't been bothering to read any of it.

  | and I keep having to repeat that this is only caused by
  |
  | (CMDS) REDIRS
  |
  | being sematically equivalent, *in bash*, to
  |
  | (exec REDIRS; CMDS)

I have just spent some time rereading the bash man page, and
I'm afraid I couldn't find that specified anywhere.

Are you sure you're not just inventing this?

Note: I'm not saying that that isn't how bash actually works,
but whether it is anywhere specified to work that way.   bash
has had bugs before, after all.

Further, I don't care if it is implemented that way, provided
that it is done properly.


  | I don't get why you accept that:
  |
  |   'externalcmd' 'a' 'b' > "$(( ++i ))"
  |
  | is not equivalent to:
  |
  |   { 'externalcmd' 'a' 'b' ;} > "$(( ++i ))"

Where did I ever say that?   All I have ever said about
those examples is that what happens to variables altered
as a side effect of an expansion in a redirect is irrelevant
to the question at hand.

  | This behaviour probably originated in ksh (irrelevant note: and was
  | probably implemented because applying redirection after fork() is easier
  | and safer, [...]

Doing redirections in process created to run an external command is
nothing new, it probably came from the original Bourne shell, long
before ksh was even a consideration.

As long as the semantics are maintained as expected, that's fine.
In this case, that will be where the "side effects in redirects
don't work" came from (I expect), as while arithmetic (and lots more)
didn't exist in the original Bourne sh, ${var=default} did.

  | In bash, this same behaviour is also applied, to subshell compound
  | commands. Which makes sense (if you think that the behaviour for simple
  | commands that run external programs makes sense, which I don't
  | necessarily do personally), since they also have a "mandatory" fork()
  | that can be exploited to prevent the work needed to restore file
  | descriptors at the end, and is what inspired the simple command
  | behaviour in the first place.

Once again, it is just fine to do the redirects in the subshell.
The problem here is what is being added to the job table (async process
table) in that subshell, in a form that makes it available for wait,
when it shouldn't be.

In
( cmd & wait )
the only background command in the subshell, and hence the only thing
wait can wait for, is cmd (it doesn't matter what that is).   Nothing
from outside should be possible to wait upon.  Ever.

Any redirect applied to that (as opposed to added inside the subshell)
is not in the subshell, wherever, or however, the shell decides to
implement the redirect, and should not be able to create something that
the wait command inside the shell can ever see.

  | I can agree that I would also like if bash expanded everything in the
  | main shell, and only applied the redirection of the expanded values in
  | the subshell.

You might like that, but that isn't what I said should happen.  It
might be permitted by the standards to do it that way, but it isn't
required.   But being able to wait upon processes in a subshell evironment
not intended to be created there is wrong.

  | I do not agree at all, though, with your idea that variable
  | modifications *should* not be evaluated in the main shell, but
  | everything else should: all the expansions, and/or redirections,
  | regardless of what they are, should be evaluated by the same shell; if
  | they are not, it simply does not make sense.

If you apply "make sense" as the test of how any shell works, you're
in for lots of disappointments.   Much of what we have has been inherited
by what was required to fit the Bourne shell into the limited space
available for a process in a PDP-11.   That resulted in all kinds of
weirdness (including the bizarre quoting rules around ${var-word} type
expansions.

  | I have used a variable modification in the example simply because it is
  | easier to show.

Sure, but it doesn't help if the rules are different.  They are.

  | How doesn't that make any sense? And what justfies your reasoning since
  | also other shells that apply this change in behaviour (ksh) run
  |
  | external < <(cmd1) > "$(cmd2)"
  |
  | with `cmd1' and `cmd2' as child processes of the subshell that ends up
  | exec()-ing `external'.

Once again, I don't care what process is a child of what other.   That's
never been what this is about, but you don't seem able to grasp that.

The question is what wait should wait for.   This gets back to the first
point of my first reply.  wait in the shell (the command) is not the same
as wait(2) (the system call).   The 

Re: wait inside subshell waits for sibling

[Emanuele Torre]

On 24/10/2022, Robert Elz  wrote:
>   | The problem that was described is caused by that
>   | optimisation (not the version that is applied to simple commands that
>   | run external program, but to subshell compound command as mentioned).
>
> I understand that.
>
>   | In case you did not understand, the issue is not that "wait inside
>   | subshell waits for sibling", because the process that runs `cat' IS NOT
>   | a sibling of the process that runs `wait', it is a CHILD.
>
> I understand that, but you are missing the point.   This is where the
> "see below" that appeared earlier applies.   You're thinking of how it
> all exists at the unix process level.   And you're probably right (I don't
> know bash internals at all) that at that level the process substitution is
> a child of the shell environment that is running wait.

It does not make any sense that you keep insisting that this is about
bash internals, or low level system call (especially `wait', `fork',
etc), and I keep having to repeat that this is only caused by

(CMDS) REDIRS

being sematically equivalent, *in bash*, to

(exec REDIRS; CMDS)
# N.B. this is actually exactly the same change in behaviour applied
#  to simple commands that run external programs, but with
#  subshell compound commands it makes sense to say that it is
#  equivalent to (exec REDIRS; CMDS) since subshell compound
#  commands themselves only expand redirections, and not regular
#  words, and temporary environment assignment words.

which is totally intended.

>   | > That is an entirely different issue, and is working as is supposed
>   | > to work.   That it isn't what some people might expect is irrelevant.
>   |
>   | What do you mean?
>
> The ineffectiveness of var modifications in redirects is a direct
> result of how they work in general, they've always been (when a
> fork is required) evaluated in the subshell (though it is only barely
> that, just a forked process between the fork and the exec).

That is a modification of how the code works.

> But that's not the way we (or the shell) should be looking at this.  At
> shell script level, the command substitution is not a child of the
> subshell,
> and a wait command in the subshell environment should only see processes
> that were created inside that environment, at the shell script level.
>
> Any optimisation should be exactly that - retain the precise effects of
> the original code, but run faster (or smaller, or whatever).

I don't get why you accept that:

  'externalcmd' 'a' 'b' > "$(( ++i ))"

is not equivalent to:

  { 'externalcmd' 'a' 'b' ;} > "$(( ++i ))"

but can't accept that, *in bash*, this exact same change in behaviour
to subshell compound commands which.

This behaviour probably originated in ksh (irrelevant note: and was
probably implemented because applying redirection after fork() is easier
and safer, than having to dup() the original file descriptors, applying
the redirections, and finally having to restore the original file
descriptors with dup2() and close(), and the final behaviour is
equivalent "in most cases"), and bash copied it.

But note that that behaviour is even implemented by every sh; for
example, dash doesn't have it, and always runs redirections in the main
shell and then reverts them.

In bash, this same behaviour is also applied, to subshell compound
commands. Which makes sense (if you think that the behaviour for simple
commands that run external programs makes sense, which I don't
necessarily do personally), since they also have a "mandatory" fork()
that can be exploited to prevent the work needed to restore file
descriptors at the end, and is what inspired the simple command
behaviour in the first place.

> If an optimisation alters the way the code works, then it is broken, and
> needs to be fixed, or removed.
>
> That's as true of a shell as it is of a compiler, or anything else.

Perhaps you are mislead by the use of the word "optimisation" into
thinking that the shell is making an attempt at making the code not
change behaviour, it is not making any attempt at that, and it is
emulating an old (probably) ksh behaviour that allows it to do less work
achieving a similar behaviour most of the times.

'external' > "$(( ++i ))"
is not equivalent to
{ 'external' ;} > "$(( ++i ))"

nor is
'external' > "$(printf %s\\n "$PPID")"
equivalent to
{ 'external' ;} > "$(printf %s\\n "$PPID")"

or
(builtin echo hi) > "$BASHPID"
to
{ (builtin echo hi) ;} > "$BASHPID"

>   | bash will always evalute  (CMDS) REDIRS  as  (exec REDIRS; CMDS).
>
> That's just fine, but it needs to avoid having anything in REDIRS
> affect

Re: wait inside subshell waits for sibling

[Robert Elz]

Date:Mon, 24 Oct 2022 10:52:22 +0200
From:Emanuele Torre 
Message-ID:  


  | I don't see how this is relevant.

See below.

  | Process substitutions are considered background jobs, and the wait
  | builtin waits for them.

That's fine.

  | > That is an entirely different issue, and is working as is supposed
  | > to work.   That it isn't what some people might expect is irrelevant.
  |
  | What do you mean?

The ineffectiveness of var modifications in redirects is a direct
result of how they work in general, they've always been (when a
fork is required) evaluated in the subshell (though it is only barely
that, just a forked process between the fork and the exec).

  | The problem that was described is caused by that
  | optimisation (not the version that is applied to simple commands that
  | run external program, but to subshell compound command as mentioned).

I understand that.

  | In case you did not understand, the issue is not that "wait inside
  | subshell waits for sibling", because the process that runs `cat' IS NOT
  | a sibling of the process that runs `wait', it is a CHILD.

I understand that, but you are missing the point.   This is where the
"see below" that appeared earlier applies.   You're thinking of how it
all exists at the unix process level.   And you're probably right (I don't
know bash internals at all) that at that level the process substitution is
a child of the shell environment that is running wait.

But that's not the way we (or the shell) should be looking at this.  At
shell script level, the command substitution is not a child of the subshell,
and a wait command in the subshell environment should only see processes
that were created inside that environment, at the shell script level.

Any optimisation should be exactly that - retain the precise effects of
the original code, but run faster (or smaller, or whatever).

If an optimisation alters the way the code works, then it is broken, and
needs to be fixed, or removed.

That's as true of a shell as it is of a compiler, or anything else.

  | bash will always evalute  (CMDS) REDIRS  as  (exec REDIRS; CMDS).

That's just fine, but it needs to avoid having anything in REDIRS
affect the execution environment of the subshell.

  | To show that this optimisation also affects subshell compound commands,

Once again, examples showing variable modifications made in redirections
are 100% irrelevant, and have no bearing on this.

Let me give you an example

bash -c 'sleep 3 & (wait)'

how long should that run before you get the next prompt back?
3 seconds, or as quickly as the commands involved can be started?

Note that there is absolutely no reason to actually fork to run the
wait command, it is the final command, once wait is done, its subshell
exits, and the script exits - that subshell can be (and in many shells
is) optimised away.   I don't know about bash.

But any shell that takes 3 seconds to run that script is broken, as
(wait) as a shell command is identical to (:) as the subshell cannot
possibly have any children, and so that wait cannot possibly have
anything to wait upon.

That the background sleep might have happened to be started in the
same shell environment as the wait command is run, such that its
process is a child of the shell running the wait command is irrelevant.

wait(2) would wait for it, wait(1) must not.   That's what I meant
in the previous message about them not being the same thing.

  | (CMDS) REDIRS  =>  (exec REDIRS; CMDS)  is arguably an incorrect
  | optimisation (afaik, only bash peforms it; ksh93 performs the simple
  | command optimisation, but not this one),

I would agree, but the optimisation could be fixed, it doesn't need
to be removed.

  | but that is what is causing the issue,

I don't care what is causing the issue, that's not my problem, my
only concern is with correct behaviour.

  | This is not a problem with the `wait' builtin waiting for a sibling
  | since `cat' is not its sibling;

It is.   You cannot optimise away a relationship that exists.  Anything
that's doing that is reinterpreting the code to mean something different
than what was written, and consequently is broken, and should be fixed.

kre

Re: wait inside subshell waits for sibling

[Emanuele Torre]

On 24/10/2022, Robert Elz  wrote:
> It need not make a difference.   The wait command is not intended to
> be just the wait(2) (or wait(3) whichever it is on the system in
> question) system call (or wrapper around another system call).

I don't see how this is relevant.

> There's no reason that wait (the command) cannot check for candidate
> jobs which are still running, and wait for any of those, and then
> return as soon as there are none left.   That the shell happens to
> have forked other processes for its own reasons should make no
> difference at all.

Process substitutions are considered background jobs, and the wait
builtin waits for them.

bash-5.1$ printf '%()T\n'; exec {fd}> >(sleep 3); wait; printf '%()T\n'
10:22:19
10:22:22

> That is an entirely different issue, and is working as is supposed
> to work.   That it isn't what some people might expect is irrelevant.

What do you mean? The problem that was described is caused by that
optimisation (not the version that is applied to simple commands that
run external program, but to subshell compound command as mentioned).

In case you did not understand, the issue is not that "wait inside
subshell waits for sibling", because the process that runs `cat' IS NOT
a sibling of the process that runs `wait', it is a CHILD.

bash will always evalute  (CMDS) REDIRS  as  (exec REDIRS; CMDS).

To show that this optimisation also affects subshell compound commands,
and not just simple commands that run external programs, here is an
example script that breaks just like the previous example (i.e. loops
infinitely and keeps running `echo hi > /tmp/file1'), and does so even
thought `echo' is a shell builtin:

i=0
while [ "$i" -lt 10 ]; do
  (echo hi) > "/tmp/file$(( ++i ))"
  sleep 1
done

In this case,  (: & wait) > >(cat)  is run as (exec > >(cat); : & wait).
Since it is that subshell that evaluates `> >(cat)', and not the shell
that runs the subshell compound command, `cat' is a background job of
the subshell, and wait, in the subshell, will wait for it.

(CMDS) REDIRS  =>  (exec REDIRS; CMDS)  is arguably an incorrect
optimisation (afaik, only bash peforms it; ksh93 performs the simple
command optimisation, but not this one), but that is what is causing the
issue, and, as i previously noted, bash has been performing this
optimisation (just like the simple command version) for a while.

This is not a problem with the `wait' builtin waiting for a sibling
since `cat' is not its sibling; and this is not a problem caused by
`wait' incorrectly waiting for process substitution processes (`wait'
should wait for those!).

emanuele6

Re: wait inside subshell waits for sibling

[Oğuz]

24 Ekim 2022 Pazartesi tarihinde Robert Elz  yazdı:

> [...] That the shell happens to
> have forked other processes for its own reasons should make no
> difference at all.


Agreed.

In the case of process substitutions it is documented that their use may
change wait's behavior

> If id is not
> given, wait waits for all running background jobs and
> the last-executed process substitution, if its process
> id is the same as $!,

but what the manual says is not what actually happens; in the OP $! is the
PID of : & when wait is called. And it doesn't even wait for only the
last-executed one, it waits for all of them:

$ ( : & wait ) > >(sleep 3) > >(:)
*hangs for 3 seconds*


-- 
Oğuz

Re: wait inside subshell waits for sibling

[Robert Elz]

Date:Mon, 24 Oct 2022 04:25:44 +0200
From:Emanuele Torre 
Message-ID:  


  | bash performs an optimisation that runs redirections applied to simple
  | commands that run external programs, or subshell compound commands after
  | the fork(). (to avoid having to restore file descriptors after running
  | the command.)

It need not make a difference.   The wait command is not intended to
be just the wait(2) (or wait(3) whichever it is on the system in
question) system call (or wrapper around another system call).

There's no reason that wait (the command) cannot check for candidate
jobs which are still running, and wait for any of those, and then
return as soon as there are none left.   That the shell happens to
have forked other processes for its own reasons should make no
difference at all.

  | This causes other weird pitfalls, e.g. the following script loops
  | infinitely, and always runs `sleep 1 > /tmp/file1':
  |
  | i=0
  | while [ "$i" -lt 10 ]
  | do sleep 1 > "/tmp/file$(( ++i ))"
  | done

That is an entirely different issue, and is working as is supposed
to work.   That it isn't what some people might expect is irrelevant.

kre

Re: wait inside subshell waits for sibling

[Oğuz]

24 Ekim 2022 Pazartesi tarihinde Emanuele Torre 
yazdı:
>
> To inhibit this optimisation, you can wrap your subshell compound
> command (or simple command) in a group command, and apply the
> redirections to it instead of the subshell command:
>
> { (: & wait) ;} > >(cat)
>
> Or, in your specific case, use "$!" to only wait for the most recent
> background job.
>
> (: & wait -- "$!") > >(cat)
>

Thanks, I'm not looking for a workaround.


-- 
Oğuz

Re: wait inside subshell waits for sibling

[Emanuele Torre]

I don't think the process running `cat' is a sibling of the subshell.

bash performs an optimisation that runs redirections applied to simple
commands that run external programs, or subshell compound commands after
the fork(). (to avoid having to restore file descriptors after running
the command.)

This causes other weird pitfalls, e.g. the following script loops
infinitely, and always runs `sleep 1 > /tmp/file1':

i=0
while [ "$i" -lt 10 ]
do sleep 1 > "/tmp/file$(( ++i ))"
done

On 23/10/2022, Oğuz İsmail Uysal  wrote:
> To reproduce:
>
> $ echo $BASH_VERSION
> 5.2.2(3)-release
> $ ( : & wait ) > >(cat)
> *hangs*
>
> It should return immediately, but hangs instead.

This behaviour has existed for a while in bash, it was not introduced in
5.2.

It causes many pitfalls that are hard to spot; see [1] (which I just
updated to mention a problematic use case like yours), and [2].

[1]: 
[2]: https://www.vidarholen.net/contents/blog/?p=865

I have never encountered a problem like the one you described before,
but that is probably one of the worse pitfalls caused by this
optiomisation. :^)

Effectively, because of the optimisation, your command is run like:

(exec > >(cat); : & wait)

So the subshell process that runs `cat' is a child of the subshell that
runs `: & wait', so wait will also wait for it.

To inhibit this optimisation, you can wrap your subshell compound
command (or simple command) in a group command, and apply the
redirections to it instead of the subshell command:

{ (: & wait) ;} > >(cat)

Or, in your specific case, use "$!" to only wait for the most recent
background job.

(: & wait -- "$!") > >(cat)

Cheers.
 emanuele6

wait inside subshell waits for sibling

[Oğuz İsmail Uysal]

To reproduce:

   $ echo $BASH_VERSION
   5.2.2(3)-release
   $ ( : & wait ) > >(cat)
   *hangs*

It should return immediately, but hangs instead.