Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 16/03/20 11:44, Joerg Schilling wrote:
> Dirk Fieldhouse  wrote:
>
>...>
>
> Could you explain why you mention Ultrix that comes with a really
ancient shell?
>
>...
>

Given that the original intention of POSIX was to use ksh88 as a boiler plate
for the POSIX shell, it may be an unwanted break to change that 36 year old
behavior now.


That was the oldest source code (SVR2 or a near derivative) that I could
easily find online that implemented functions and return; I understand
that the System V Bourne shell of the time was the pattern for ksh88.

The code shows the original behaviour of 'return' which has been
maintained in later shells, and why it was easy to implement it in that
way rather than, for instance, maintaining some state (in addition to
the nesting level) that would have allowed return from a subshell to be
detected and perhaps flagged as an error, or even somehow communicating
back to the parent shell.

Now it's clear what the 36-year-old behaviour is and that such behaviour
is what POSIX intends, it's just a question of correctly documenting it
in the standard ...

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Joerg Schilling]

Robert Elz  wrote:


> I was referring to "in the standard".   The standard is not static,
> things are added (or move from unspecified to specified) as the
> implementations converge upon some common definition for something new
> or previously unspecified, and things are deleted when they become
> obsolete, and are implemented only to satisfy the standard, or move
> to being unspecified when it is clear that despite the standard there
> is in fact no common definition.   Of course all of this happens at
> a glacial pace.

Being not static does not mean that unexpected changes or changes that break 
compatibility may occur.

>   | Is there any suggestion that the 'exit'-like behaviour of any shell that
>   | implements it for 'return' in such contexts is subtly different from 
> 'exit'?
>
> Not that I am aware of.   exit is kind of blunt, it is quite hard to
> be subtly different - I suppose the one difference might be whether the
> EXIT trap (if there is one) is run, but (without testing yet) my guess
> is that it would be (whether the subshell exits because of a return, an
> explicit exit, or just by falling off the end should make no difference
> to exit trap processing).

Well, neither the documentation of the Bournr Shell nor the POSIX standard 
claim that return is an equivalent to exit. This is a deviation of ksh88 and 
descendents or lookalikes.

> Actually, I have just tested it ... bosh and yash do not run the EXIT
> trap when a subshell (with a trap in it .. the test is shown below)
> exits because of a "return" command in the subshell.   All other shells
> I have to test against do.   When "return" is changed to "exit" all shells
> run the exit trap, as they do when the statement is omitted comletely
> (in which case the following echo obviously runs, and the exit status from
> the subshell is 0).
>
> The test is:
>
> $SHELL -c 'f() { (trap "echo FOO" EXIT; return 5; echo BAR); echo "$?"; }; f'
>
> The shells that behave properly (all of them when "return" is replaced
> by "exit") print FOO, and then 5, bosh and yash simply print 5 for this case.

You are mistaken, all shells except ksh88, ksh93 and lookalikes behave properly 
and do not call the EXIT trap since return is not an alias for exit.

ksh behaves according to the ksh decumentation but not according to the POSIX 
text.

> ps: one "interesting" thing from this, is that when that was run in the
> shells interactively, rather than via "$SHELL -c" all acted the same way
> when "return" was used (the same way as reported above for each of them)
> but when "exit" is used (ie: in the case where the trap is actually run)
> bosh caused the subshell to stop (SIGTTOU) - foregrounding it simply makes
> causes it to stop again - that is, for some reason, when interactive (ie:
> with set -m) bosh seems to be running that subshell in a different process
> group (which is correct) but not changing the terminal to be in that pg.
> That's a bug (since it is a foreground process).
>
> bosh $ echo ${.sh.version}
> version bosh 2020/01/24 a+ (amd64-unknown-netbsd8.99.30)
>
> It came from schily-2020-02-11.tar.bz2
>
> Even more weird, when the "exit 5;" is simply omitted, that doesn't
> happen, bosh prints BAR FOO 0 (on 3 lines of course) as it should, without
> anything stopping - which indicates that in this case, either it runs the
> subshell in the same process group as the parent (unlikely) or switches the
> terminal to the process group of the subsjell (more likely) which it doesn't
> do when the exit command is present (either explicitly, or "hidden" using
> X=exit and then using $X instead of "exit" in the function).

I am not sure whether I understand you correctly, but I am unable to repeat 
your claims on Solaris or Linux, so this may be a bug in the kernel of your 
platform.

Please send an exact description of what you did, e.g. by a typescript to allow 
me to verify this.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Joerg Schilling]

Dirk Fieldhouse  wrote:

> You're reading too much into "If". I think it's clear that design
> decisions from several decades ago take precedence, regardless of what
> anyone would prefer. Eg
> 
> where apparently the case of "not in a function" is handled and
> otherwise a (per-process) flag is set so that nothing else is executed
> until the next function call is unwound, or the process exits, whichever
> comes first.

Could you explain why you mention Ultrix that comes with a really ancient shell?

If you like to refer to the Bourne Shell, you could have a look at bosh (see 
directory "sh" in the schilytools package). the advantage of bosh is that it is 
a POSIX compliant enhancement based on the Bourne Shell and that you may get a 
compile variant called "obosh" that behaves like the Svr4 Bourne Shell from 
OpenSolaris - just highly portable.

The problem in the Bourne Shell however is that there is an "in function" flag
that does not become invalid when a sub-shell is created.

This is the Bourne Shell (and ksh IIRC) behavior since functions have been 
introduced in 1984.

Given that the original intention of POSIX was to use ksh88 as a boiler plate 
for the POSIX shell, it may be an unwanted break to change that 36 year old 
behavior now.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 15/03/20 07:26, Robert Elz wrote:
>[a test on a range of current shell implementations show that 'return'
from a function, when run in an execution environment different from
that in which the function's compound-command is run, behaves like
'exit' except that implementations don't agree on EXIT traps]

Another test relevant to the HvD option below is

f() ( trap "echo FOO" EXIT; return 5; echo BAR ); f

which doesn't run the EXIT trap in bash-4.3 and dash-0.5.8.2, and so
indicates that this case should be considered as return-like rather than
exit-like.

And on 13/03/20 13:42, Dirk Fieldhouse proposed some more specific text
for the normative description of 'return', based on Don Cragun's
suggestion...

Given the test results I revised that proposal as below to put the case
tested above in the non-normative text, which I have also drafted in
terms that I hope are not controversial; please also see the questions
at the end.

2.14  return

DESCRIPTION

[Replace existing text entirely, up to "OPTIONS"]
The return utility shall cause the shell to stop executing the
current function or dot script, if any, as defined below; if
there is no such function or dot script, the results are
unspecified.

The current function shall be the function, if any,

a)  the compound-command of whose function definition (see
Section 2.9.5) contains the return utility, and

[Alternative HvD: allow 'f() (... return ...)', where ... marks 0 or
more components of the compound-list]
b)  whose compound-command is being executed by the shell in
the same execution environment (see Section 2.12) in which the
shell is executing the return utility.
[Alternative 0: only 'f() {... return ...}']
b)  that was invoked by the shell in the same execution
environment (see Section 2.12) in which the shell is executing
the return utility.
[end Alternatives]

The current dot script shall be the dot script, if any,

a)  that contains the return utility, and

b)  whose dot command (see dot) was executed by the shell in
the same execution environment (see Section 2.12) in which the
shell is executing the return utility.

...

APPLICATION USAGE

[Replace existing "None."]
The DESCRIPTION leaves unspecified the behaviour of the return
utility in a function when the utility is run [otherwise than
in the selected option above]: see RATIONALE.

Historically many implementations have treated such uses of
return as equivalent to exit. It is recommended to use the exit
utility explicitly instead of return when the behavior of
return might be unspecified, since exit does have defined
behavior in those cases. For instance, it is not specified
whether the stages of a pipeline are executed in one or separate
execution environments (see the final example below).

EXAMPLES

[Replace existing "None."]
The command "f $n && echo OK || echo NO" prints

Before return
OK

if n is 0, or if n is in the range 1-255 inclusive

Before return
NO

for each of the following definitions of f().

f() {
retval=$1
echo "Before return";
return $retval
echo "Failed to return" >&2; exit 1
}

f() (
retval=$1
echo "Before return";
return $retval
echo "Failed to return" >&2; exit 1
)

f() {
retval=$1
while true; do
echo "Before return"
return $retval
echo "Failed to return" >&2; exit 1
done
}

f() {
retval=$1
# avoid using return in a possibly unspecified context
echo "Before return" |
{ read -r line; echo $line; exit $retval; }
return
echo "Failed to return" >&2; exit 1
}

RATIONALE

[Insert before existing "The behavior of return when not ..."]
The standard requires utilities that incorporate a command
language using certain control flow keywords from the ISO C
Standard to implement those keywords in a way equivalent to
their C specifications (see Section 1.1.2).

As shell functions may invoke the full range of shell
capabilities, including subshells and asynchronous commands, the
behavior of the return utility from a function could not
generally be defined as equivalent to the use of return in C.

As discussed in XRAT Section C.2.9.5, shell functions were
originally specified as belonging to the shell execution
environment in which they were executed. It was sometimes
rationalised that after creating a separate execution

Re: XCU: 'return' from subshell

[Robert Elz]

Date:Sat, 14 Mar 2020 22:43:05 +
From:Dirk Fieldhouse 
Message-ID:  <06785188-5da9-1494-2af3-4456759cb...@gmx.net>

  | The fact that a behaviour exists in a wide range of implementations, in
  | a context held by many experts to be unspecified by the current
  | standard, doesn't mean that scripts should be able to rely upon it.

No it doesn't, but it is a candidate for the standard to switch from
being unspecified to defining the hehaviour, given that all the shells
implement it anyway - after which scripts can rely upon it.That's how
the standard evolves.

  |  > ...  we document what works, not what we think is "better".
  |
  | Documenting, fine. Standardising, perhaps not so much?

I was referring to "in the standard".   The standard is not static,
things are added (or move from unspecified to specified) as the
implementations converge upon some common definition for something new
or previously unspecified, and things are deleted when they become
obsolete, and are implemented only to satisfy the standard, or move
to being unspecified when it is clear that despite the standard there
is in fact no common definition.   Of course all of this happens at
a glacial pace.

  | Is there any suggestion that the 'exit'-like behaviour of any shell that
  | implements it for 'return' in such contexts is subtly different from 'exit'?

Not that I am aware of.   exit is kind of blunt, it is quite hard to
be subtly different - I suppose the one difference might be whether the
EXIT trap (if there is one) is run, but (without testing yet) my guess
is that it would be (whether the subshell exits because of a return, an
explicit exit, or just by falling off the end should make no difference
to exit trap processing).

Actually, I have just tested it ... bosh and yash do not run the EXIT
trap when a subshell (with a trap in it .. the test is shown below)
exits because of a "return" command in the subshell.   All other shells
I have to test against do.   When "return" is changed to "exit" all shells
run the exit trap, as they do when the statement is omitted comletely
(in which case the following echo obviously runs, and the exit status from
the subshell is 0).

The test is:

$SHELL -c 'f() { (trap "echo FOO" EXIT; return 5; echo BAR); echo "$?"; }; f'

The shells that behave properly (all of them when "return" is replaced
by "exit") print FOO, and then 5, bosh and yash simply print 5 for this case.

kre

ps: one "interesting" thing from this, is that when that was run in the
shells interactively, rather than via "$SHELL -c" all acted the same way
when "return" was used (the same way as reported above for each of them)
but when "exit" is used (ie: in the case where the trap is actually run)
bosh caused the subshell to stop (SIGTTOU) - foregrounding it simply makes
causes it to stop again - that is, for some reason, when interactive (ie:
with set -m) bosh seems to be running that subshell in a different process
group (which is correct) but not changing the terminal to be in that pg.
That's a bug (since it is a foreground process).

bosh $ echo ${.sh.version}
version bosh 2020/01/24 a+ (amd64-unknown-netbsd8.99.30)

It came from schily-2020-02-11.tar.bz2

Even more weird, when the "exit 5;" is simply omitted, that doesn't
happen, bosh prints BAR FOO 0 (on 3 lines of course) as it should, without
anything stopping - which indicates that in this case, either it runs the
subshell in the same process group as the parent (unlikely) or switches the
terminal to the process group of the subsjell (more likely) which it doesn't
do when the exit command is present (either explicitly, or "hidden" using
X=exit and then using $X instead of "exit" in the function).

That's simply bizarre.

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 14/03/20 01:26, Robert Elz wrote:

 Date:Fri, 13 Mar 2020 16:49:14 +
 From:Dirk Fieldhouse 
 Message-ID:  

   | But as 'return' is behaving like 'exit', and not actually returning (ie,
   | setting $?=7 in the invoking shell and not printing no or 7) in this
   | case, might it not be better to leave such usage for a commentary in
   | non-normative text?

The standard says what the shells do, and what scripts can rely upon.
That is what this is.


The fact that a behaviour exists in a wide range of implementations, in
a context held by many experts to be unspecified by the current
standard, doesn't mean that scripts should be able to rely upon it. If a
script relying on that behaviour works now, it's only through
suck-it-and-see programming; far be it from me to say that it deserves
to break, or indeed to promote a POSIX-conformant shell implementation
that fails an attempted 'return' from a subshell.

When the name 'return' was introduced into the shell it was already a
well understood programming concept used by that name in many languages.

Owing to the execution model of the shell, that concept couldn't be
implemented in the shell without constraints that were never clearly
documented, but must have been well known to implementers.

Since the constraints needed for 'return' to do so have not been widely
understood (eg
), there is
obvious benefit in standardising them, but since the 'exit'-like
behaviour could never previously have been relied upon standardising it
would just muddy the concept implied by the name 'return'. You say later:

> ...  we document what works, not what we think is "better".

Documenting, fine. Standardising, perhaps not so much?


   | If 'return' doesn't return from general subshell contexts,

Return in sh is *never* going to return the way you want it to.   Never.


You're reading too much into "If". I think it's clear that design
decisions from several decades ago take precedence, regardless of what
anyone would prefer. Eg

where apparently the case of "not in a function" is handled and
otherwise a (per-process) flag is set so that nothing else is executed
until the next function call is unwound, or the process exits, whichever
comes first.


   | shell programmers have to live with work-arounds like
   |
   | f() { (exit 7; echo no); return; }
   |
   | instead of
   |
   | f() { (return 7; echo no); }

Those two achieve the same thing, in a slightly different way. ...


Is there any suggestion that the 'exit'-like behaviour of any shell that
implements it for 'return' in such contexts is subtly different from 'exit'?


... there are times when things need to be written carefully to achieve
particular effects (the same when one wants to save side-effect results
from a pipeline - "return" isn't the only thing that gets affected by the
way subshells (that is, separate execution environments) work in the shell.


Indeed, but the issue of variable scope in a pipeline seems to be more
widely understood, eg .


And once again, if you don't like the way the shell language works, go
and invent a new one, then implement and distribute it.
...


I'm certainly not angling for that.

There are plenty of (whisper it, and without attaching any  blame -- not
like this guy
)
better designed scripting languages around, but not ones that you can
expect to see installed by default on a POSIX-conformant environment and
even raise defect reports against if they don't do what POSIX specifies.

And if the standard lays down when 'return' returns and when it might do
something else, that's potentially a validation case for shellcheck.net.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Robert Elz]

Date:Fri, 13 Mar 2020 16:49:14 +
From:Dirk Fieldhouse 
Message-ID:  

  | But as 'return' is behaving like 'exit', and not actually returning (ie,
  | setting $?=7 in the invoking shell and not printing no or 7) in this
  | case, might it not be better to leave such usage for a commentary in
  | non-normative text?

The standard says what the shells do, and what scripts can rely upon.
That is what this is.

  | If 'return' doesn't return from general subshell contexts,

Return in sh is *never* going to return the way you want it to.   Never.

  | shell programmers have to live with work-arounds like
  |
  | f() { (exit 7; echo no); return; }
  |
  | instead of
  |
  | f() { (return 7; echo no); }

Those two achieve the same thing, in a slightly different way.   But
yes, there are times when things need to be written carefully to achieve
particular effects (the same when one wants to save side-effect results
from a pipeline - "return" isn't the only thing that gets affected by the
way subshells (that is, separate execution environments) work in the shell.

And once again, if you don't like the way the shell language works, go
and invent a new one, then implement and distribute it.

  | There's no point encouraging people to use 'return' in a context where
  | it may not, let alone where it doesn't; they should use 'exit' which
  | already has specified behaviour in that context.

I agree with that as well, that's the better way for programmers to
write sh code, but once again, we document what works, not what we think
is "better".

kre

Re: XCU: 'return' from subshell

[Robert Elz]

Date:Fri, 13 Mar 2020 14:14:24 +
From:Harald van Dijk 
Message-ID:  


  | I don't see how you can allow that without also allowing
  |
  |f() { (return 7; echo no); echo $?; }; f
  |
  | If that also works in all shells (meaning it doesn't print no, and does 
  | print 7), then by all means standardise it.

Everything I have available to test prints '7' and doesn't print 'no'.

  | > Since current shells do not all treat return in a subshell as exit,
  | > I think we should leave that behavior unspecified.

When the subshell (separate execution environment) is not in a function
that is the thing to do.

  | I agree with that, I think it makes sense to limit it to what works in 
  | all current shells, that's why I limited my proposed change to what I 
  | know works. Not all current shells treat return in a subshell as exit, 
  | but if all current shells do treat return in a subshell *in a function* 
  | as exit, it makes sense to also allow that.

Agreed.   And it seems that they do (or at least all the ones that
we traditionally care about, I don't know about fish mrsh rc ...)

kre

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 13/03/20 14:57, Martijn Dekker wrote:

Op 13-03-20 om 15:17 schreef Chet Ramey:

On 3/13/20 10:14 AM, Harald van Dijk wrote:

[...]

I don't see how you can allow that without also allowing

   f() { (return 7; echo no); echo $?; }; f

...>

My testing says it acts identically (outputs '7') on the following
shells: ash variants (Busybox ash, dash, FreeBSD sh, gwsh, NetBSD sh),
bash, bosh, ksh88, ksh93, mksh, pdksh, yash, and zsh.


But as 'return' is behaving like 'exit', and not actually returning (ie,
setting $?=7 in the invoking shell and not printing no or 7) in this
case, might it not be better to leave such usage for a commentary in
non-normative text?

If 'return' doesn't return from general subshell contexts, shell
programmers have to live with work-arounds like

f() { (exit 7; echo no); return; }

instead of

f() { (return 7; echo no); }

There's no point encouraging people to use 'return' in a context where
it may not, let alone where it doesn't; they should use 'exit' which
already has specified behaviour in that context.

Therefore I don't think the normative text should cover the case offered
by Chet Ramey on 13/03/20 13:10:
>...>
>

We should be able to do better than that. I don't see why "if not executing
in the same shell execution environment as the compound-list ..." can't
cover the f() { (return 1) } case as well, and seems to work in all shells.


It would be more accurate to say that it fails weirdly in all shells.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Martijn Dekker]

Op 13-03-20 om 15:17 schreef Chet Ramey:

On 3/13/20 10:14 AM, Harald van Dijk wrote:

[...]

I don't see how you can allow that without also allowing

   f() { (return 7; echo no); echo $?; }; f

If that also works in all shells (meaning it doesn't print no, and does
print 7), then by all means standardise it.


I can't find one that doesn't in my quick initial testing, but I don't
have binaries for every shell under the sun.


My testing says it acts identically (outputs '7') on the following 
shells: ash variants (Busybox ash, dash, FreeBSD sh, gwsh, NetBSD sh), 
bash, bosh, ksh88, ksh93, mksh, pdksh, yash, and zsh.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/13/20 10:14 AM, Harald van Dijk wrote:

>>> Can this instead say "in the same shell execution environment as the
>>> compound-list of the compound-command of the function definition", so that
>>>
>>>    f() (return 1)
>>>
>>> which is fairly sensible and works in all shells[*] remains well-defined,
>>> but only something along the lines of f() { (return 1) } or
>>> f() ( (return 1) ) becomes unspecified?
>>
>> We should be able to do better than that. I don't see why "if not executing
>> in the same shell execution environment as the compound-list ..." can't
>> cover the f() { (return 1) } case as well, and seems to work in all shells.
> 
> I don't see how you can allow that without also allowing
> 
>   f() { (return 7; echo no); echo $?; }; f
> 
> If that also works in all shells (meaning it doesn't print no, and does
> print 7), then by all means standardise it.

I can't find one that doesn't in my quick initial testing, but I don't
have binaries for every shell under the sun.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Harald van Dijk]

On 13/03/2020 13:10, Chet Ramey wrote:

On 3/12/20 4:21 PM, Harald van Dijk wrote:

On 11/03/2020 17:44, Don Cragun wrote:

Would this issue be resolved if we change the last sentence of the
description section of the return Special Built-In Utility from:
  If the shell is not currently executing a function
  or dot script, the results are unspecified.
to:
  If the shell is not currently executing a function
  or dot script running in the same shell execution
  environment as the command that invoked the function
  or dot script, the results are unspecified.
?


Can this instead say "in the same shell execution environment as the
compound-list of the compound-command of the function definition", so that

   f() (return 1)

which is fairly sensible and works in all shells[*] remains well-defined,
but only something along the lines of f() { (return 1) } or
f() ( (return 1) ) becomes unspecified?


We should be able to do better than that. I don't see why "if not executing
in the same shell execution environment as the compound-list ..." can't
cover the f() { (return 1) } case as well, and seems to work in all shells.


I don't see how you can allow that without also allowing

  f() { (return 7; echo no); echo $?; }; f

If that also works in all shells (meaning it doesn't print no, and does 
print 7), then by all means standardise it.


Don Cragun had written:


Since current shells do not all treat return in a subshell as exit, I think we 
should leave that behavior unspecified.


I agree with that, I think it makes sense to limit it to what works in 
all current shells, that's why I limited my proposed change to what I 
know works. Not all current shells treat return in a subshell as exit, 
but if all current shells do treat return in a subshell *in a function* 
as exit, it makes sense to also allow that.


Cheers,
Harald van Dijk

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 12/03/20 20:59, Dirk Fieldhouse wrote:

...>

If Harald's suggestion is accepted, the handling of functions (defined
by a compound-command) and dot scripts (not so defined) would be
different and the DESCRIPTION of 2.14 'return' should probably be split
into two cases to avoid confusion.

For instance, like the text below, which I think also deals with
discriminating which context is supposed to be returned from when a
function is called from a dot script and vice versa; I included
alternative wording for Harald's and Chet's suggestions:

2.14  return

DESCRIPTION

The return utility shall cause the shell to stop executing the
current function or dot script, if any, as defined below; if
there is no such function or dot script, the results are
unspecified.

The current function shall be the function, if any,

a)  the compound-command of whose function definition (see
Section 2.9.5) contains the return utility, and

[Alternative HvD: allow 'f() (... return ...)', where ... marks 0 or
more components of the compound-list]
b)  whose compound-command is being executed by the shell in
the same execution environment (see Section 2.12) in which the
shell is executing the return utility.
[Alternative CR: also allow 'f() (... (return) ...)']
b)  the compound-list of whose compound-command is being
executed by the shell in the same execution environment (see
Section 2.12) in which the shell is executing the return
utility.
[Alternative 0: only 'f() {... return ...}']
b)  that was invoked by the shell in the same execution
environment (see Section 2.12) in which the shell is executing
the return utility.
[end Alternatives]

The current dot script shall be the dot script, if any,

a)  that contains the return utility, and

b)  whose dot command (see dot) was executed by the shell in
the same execution environment (see Section 2.12) in which the
shell is executing the return utility.

OPTIONS
...

Is it necessary to qualify the phrase "contains the return utility" in
each case with something like "other than inside a function definition
command", as in DR 842 ?

Is there a conflict between this normal use of "contains" and the
definition of "contains" used for 'break' and 'continue' as a result of
DR 842 ?

Should a '(return)' at the top-level of a dot script do so (bash 4.3,
dash 0.5.8 don't) ?

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/12/20 4:21 PM, Harald van Dijk wrote:
> On 11/03/2020 17:44, Don Cragun wrote:
>> Would this issue be resolved if we change the last sentence of the
>> description section of the return Special Built-In Utility from:
>>  If the shell is not currently executing a function
>>  or dot script, the results are unspecified.
>> to:
>>  If the shell is not currently executing a function
>>  or dot script running in the same shell execution
>>  environment as the command that invoked the function
>>  or dot script, the results are unspecified.
>> ?
> 
> Can this instead say "in the same shell execution environment as the
> compound-list of the compound-command of the function definition", so that
> 
>   f() (return 1)
> 
> which is fairly sensible and works in all shells[*] remains well-defined,
> but only something along the lines of f() { (return 1) } or
> f() ( (return 1) ) becomes unspecified?

We should be able to do better than that. I don't see why "if not executing
in the same shell execution environment as the compound-list ..." can't
cover the f() { (return 1) } case as well, and seems to work in all shells.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 12/03/20 20:21, Harald van Dijk wrote:

On 11/03/2020 17:44, Don Cragun wrote:

Would this issue be resolved if we change the last sentence of the
description section of the return Special Built-In Utility ...

 If the shell is not currently executing a function
 or dot script running in the same shell execution
 environment as the command that invoked the function
 or dot script, the results are unspecified.




Can this instead say "in the same shell execution environment as the
compound-list of the compound-command of the function definition", so that

   f() (return 1)

which is fairly sensible and works in all shells[*] remains
well-defined, but only something along the lines of f() { (return 1) } or
f() ( (return 1) ) becomes unspecified?


If Harald's suggestion is accepted, the handling of functions (defined
by a compound-command) and dot scripts (not so defined) would be
different and the DESCRIPTION of 2.14 'return' should probably be split
into two cases to avoid confusion.

In addition, consider 'return' in a function executed by a dot script,
all in the same execution environment.

Neither Don's proposal nor my revisions of it address this case where
there is both a current function and a current dot script in the same
execution environment as the 'return'; although obviously the intention
is to resume the dot script at the next command after the function
invocation, the texts permit the deluded interpretation that the dot
script should be returned from.

Some DR 842-style wording about textual nesting may be inescapable.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 5:24 PM, Dirk Fieldhouse wrote:

> Even with this wording, it isn't clear that there is "the function or
> dot script, if any" (ie just one, or none) without first applying the
> restriction to the same execution environment, depending on whether you
> think that asynchronous commands in a function definition are counted in
> the "current function", so this perhaps would be better:
> 
> The return utility shall cause the shell to stop executing the
> current function or dot script, if any, that is being executed
>  using the same shell execution environment (see 2.12) as the
> command that invoked the function or dot script. Otherwise the
> results are unspecified.

I think "executed in the same shell execution environment" is better
wording, since it parallels usage elsewhere in the standard, such as

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01

steps 1 and 2.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 1:32 PM, Don Cragun wrote:
> Would this issue be resolved if we change the last sentence of the 
> description section of the return Special Built-In Utility from:
>   If the shell is not currently executing a function
>   or dot script, the results are unspecified.
> to:
>   If the shell is not currently executing a function
>   or dot script running in the same shell execution
>   environment as the command that invoked the function
>   or dot script, the results are unspecified.

I think this is heading in the right direction.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 1:07 PM, Dirk Fieldhouse wrote:
> On 11/03/20 15:23, Chet Ramey wrote:
>> ...>
>> What does a `return from the execution environment' mean, exactly? ...
> 
> To clarify, what I wrote was shorthand for "return from the function if
> the 'return' is executed in the same execution environment as" the
> function's defining command, or otherwise (ii) exit or (iii) unspecified
> behaviour.

I don't think `defining command' is right. It's where the function is
executed that is the issue. So different execution environments is the
way to proceed, but using something like caller instead of defining
command.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 12:15 PM, Dirk Fieldhouse wrote:
>
>> All shells I am aware of print foo and bar
> 
> The discussion seems to have confirmed that this is the general existing
> practice, and not just in the few cases I tested, but IMO only a shell
> implementer could see the suggested behaviour of these examples as
> baffling, based on the wording of the standard (not to mention "man sh",
> etc, so I won't). 

If it's the wording that implies possible behavior that no shell
implements, let's fix the wording.

> The question is what, if any, rewording of the standard should be made.
> There are plenty of choices for better designed scripting languages, so
> arguably making the specification agree with existing practice would be
> an acceptable resolution. The example of DR 842 for 'break' and
> 'continue' shows that this should not be seen as an unnecessary change.

We can use 842 as a model for the changes. Someone needs to propose new
wording that's comprehensive enough to cover the different cases.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/12/20 6:02 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>> I use Mac OS X. I test on Linux.
>>
>> By far, the biggest difference between older Mac OS X/Linux and current Mac
>> OS X is using lldb instead of gdb for debugging.
> 
> OK, I develop on Solaris and like dbx in favor of gxb at al. but it seems 
> to be a pity that the Solaris compilers do not get updates for OpenSolaris 
> anymore.
> 
> I test on various platforms and as a result, I recently discovered that 
> waitid() on Mac OS is still not usable even though there is a POSIX 
> certification. The still remaining problem is that it always returns a signal 
> number of 0 if the child has been killed by a signal. So for a portable 
> program like bash, it seems that OSX and Linux are not sufficient.

It might be, if bash used waitid().

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Joerg Schilling]

Chet Ramey  wrote:

> I use Mac OS X. I test on Linux.
>
> By far, the biggest difference between older Mac OS X/Linux and current Mac
> OS X is using lldb instead of gdb for debugging.

OK, I develop on Solaris and like dbx in favor of gxb at al. but it seems 
to be a pity that the Solaris compilers do not get updates for OpenSolaris 
anymore.

I test on various platforms and as a result, I recently discovered that 
waitid() on Mac OS is still not usable even though there is a POSIX 
certification. The still remaining problem is that it always returns a signal 
number of 0 if the child has been killed by a signal. So for a portable 
program like bash, it seems that OSX and Linux are not sufficient.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 17:32, Don Cragun wrote:


... notice that I said "in the same shell execution environment as the command that invoked 
..."; not "in the same shell execution environment as the function's defining 
command".  If a function is invoked in a subshell of the environment that defined the function 
that should not affect the behavior of return as long as the function doesn't doesn't invoke return 
in a new shell execution environment


In my "in the same shell execution environment as the function's
defining command ..." please complete the ellipsis with "was invoked". I
think we're saying the same thing.

As function definitions are part of the definition of 'execution
environment', it shouldn't matter where a function was defined.

Your wording seems to cover the cases from 2.12, which are

-   command substitution

-   subshell compound-group

-   asynchronous list

-   multi-command pipeline.

But it only goes to unspecified behaviour and there seems to be support
for more specific wording to cover the excluded cases, if only in the
APPLICATION USAGE or RATIONALE sections of 2.14 'return. Leaving that
for the  moment ...

Would "using", or "with", be a better term than "running in" in your
proposal? No suitable precedent seems to exist in the current standard
or rationale. Also the two potentially contradictory requirements should
be combined, and there should be a reference out.

As in:

The return utility shall cause the shell to stop executing the
current function or dot script_, if any, and if the shell is _
currently executing _the_ function or dot script _using_ the
same shell execution environment _(see 2.12) _as the command
that invoked the function or dot script_. Otherwise_ the results
are unspecified.

Even with this wording, it isn't clear that there is "the function or
dot script, if any" (ie just one, or none) without first applying the
restriction to the same execution environment, depending on whether you
think that asynchronous commands in a function definition are counted in
the "current function", so this perhaps would be better:

The return utility shall cause the shell to stop executing the
current function or dot script, if any, that is being executed
using the same shell execution environment (see 2.12) as the
command that invoked the function or dot script. Otherwise the
results are unspecified.

APPLICATION USAGE might then point out that 'exit' has defined semantics
in the cases where 'return' has unspecified behaviour, and some
distillation of the issues discussed here might be included in the
RATIONALE or in the RATIONALE for function definition (C.2.9.5).

>...>

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[shwaresyst]

I agree this is something suitable for a TC type resolution. However, given the 
amount of existing practice, I do not see we should leave anything unspecified 
for Issue 8, even if some implementations break as a result. 

Either it always stops function execution or dot file processing, which can be 
implemented with process shared mutexes or semaphores for synchronous 
subshells, or they are ignored in a subshell context to the extent 
interpretation continues after the closing ')'. As the shell stays in the 
foreground after initiating any asynchronous subshells, interpretation inside 
the definition or dot file is already required to continue. It may not get far 
past before the async job is permitted to get to the foreground, but this is 
still past. It may need to be explicit, for that context, that return is also 
required to raise SIGCHLD the same as a normal exit would.
On Wednesday, March 11, 2020 Don Cragun  wrote:
Would this issue be resolved if we change the last sentence of the description 
section of the return Special Built-In Utility from:
    If the shell is not currently executing a function
    or dot script, the results are unspecified.
to:
    If the shell is not currently executing a function
    or dot script running in the same shell execution
    environment as the command that invoked the function
    or dot script, the results are unspecified.
?

Dirk, notice that I said "in the same shell execution environment as the 
command that invoked ..."; not "in the same shell execution environment as the 
function's defining command".  If a function is invoked in a subshell of the 
environment that defined the function that should not affect the behavior of 
return as long as the function doesn't doesn't invoke return in a new shell 
execution environment that it created.)

Since current shells do not all treat return in a subshell as exit, I think we 
should leave that behavior unspecified.  (I see no reason why a conforming 
shell should not be able to report that return was invoked in a shell execution 
environment that is not current execution environment as the command that 
invoked the function or dot script.)

Cheers,
Don

> On Mar 11, 2020, at 10:07 AM, Dirk Fieldhouse  wrote:
> 
> On 11/03/20 15:23, Chet Ramey wrote:
>> ...>
>> What does a `return from the execution environment' mean, exactly? ...
> 
> To clarify, what I wrote was shorthand for "return from the function if
> the 'return' is executed in the same execution environment as" the
> function's defining command, or otherwise (ii) exit or (iii) unspecified
> behaviour.
> 
> So I think your agreement with case (i) means that you would be looking
> for some text like that in my originally (10/03/20 15:22) proposed
> Resolution (a), and therefore (with the clarification above) a
> definition of when a 'return' is "in" a function for case (i) that would
> make it equivalent to case (iii).
> 
> Then the remaining question is whether (case (ii)) there is benefit in
> codifying, normatively or not, the 'exit'-like behaviour when a 'return'
> that is textually contained in a function definition is run in a
> 'separate execution environment' from the function's defining command,
> rather than leaving all other cases unspecified.
> 
> The case (naively) of 'return' used textually outside a function
> definition would be covered by existing 2.14 text "If the shell is not
> currently executing a function or dot script", except that it is widely
> held that "not currently executing a function" includes the situation of
> the previous paragraph. Whatever the wording, it appears controversial
> whether the 'exit'-like behaviour should apply here.
> 
> Without wishing to extend the scope of discussion, the model against
> which readers are likely to interpret the standard's text on functions
> is that of a programming language like C, where a function foo() is said
> to be executing while any statements textually contained in the function
> are executing, and possibly more than once if it performed a successful
> fork(). Any revised wording needs to invalidate such a mental model, if
> that is what the standard intends.
> 
> /df
> 
> --
> London SW6
> UK
> 

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 11:52 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>> On 3/11/20 11:46 AM, Joerg Schilling wrote:
>>
>>> Since you most likely develop on Linux
>>
>> I don't; don't make assumptions.
> 
> Interesting, where do you develop?

I use Mac OS X. I test on Linux.

By far, the biggest difference between older Mac OS X/Linux and current Mac
OS X is using lldb instead of gdb for debugging.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Don Cragun]

Would this issue be resolved if we change the last sentence of the description 
section of the return Special Built-In Utility from:
If the shell is not currently executing a function
or dot script, the results are unspecified.
to:
If the shell is not currently executing a function
or dot script running in the same shell execution
environment as the command that invoked the function
or dot script, the results are unspecified.
?

Dirk, notice that I said "in the same shell execution environment as the 
command that invoked ..."; not "in the same shell execution environment as the 
function's defining command".  If a function is invoked in a subshell of the 
environment that defined the function that should not affect the behavior of 
return as long as the function doesn't doesn't invoke return in a new shell 
execution environment that it created.)

Since current shells do not all treat return in a subshell as exit, I think we 
should leave that behavior unspecified.  (I see no reason why a conforming 
shell should not be able to report that return was invoked in a shell execution 
environment that is not current execution environment as the command that 
invoked the function or dot script.)

Cheers,
Don

> On Mar 11, 2020, at 10:07 AM, Dirk Fieldhouse  wrote:
> 
> On 11/03/20 15:23, Chet Ramey wrote:
>> ...>
>> What does a `return from the execution environment' mean, exactly? ...
> 
> To clarify, what I wrote was shorthand for "return from the function if
> the 'return' is executed in the same execution environment as" the
> function's defining command, or otherwise (ii) exit or (iii) unspecified
> behaviour.
> 
> So I think your agreement with case (i) means that you would be looking
> for some text like that in my originally (10/03/20 15:22) proposed
> Resolution (a), and therefore (with the clarification above) a
> definition of when a 'return' is "in" a function for case (i) that would
> make it equivalent to case (iii).
> 
> Then the remaining question is whether (case (ii)) there is benefit in
> codifying, normatively or not, the 'exit'-like behaviour when a 'return'
> that is textually contained in a function definition is run in a
> 'separate execution environment' from the function's defining command,
> rather than leaving all other cases unspecified.
> 
> The case (naively) of 'return' used textually outside a function
> definition would be covered by existing 2.14 text "If the shell is not
> currently executing a function or dot script", except that it is widely
> held that "not currently executing a function" includes the situation of
> the previous paragraph. Whatever the wording, it appears controversial
> whether the 'exit'-like behaviour should apply here.
> 
> Without wishing to extend the scope of discussion, the model against
> which readers are likely to interpret the standard's text on functions
> is that of a programming language like C, where a function foo() is said
> to be executing while any statements textually contained in the function
> are executing, and possibly more than once if it performed a successful
> fork(). Any revised wording needs to invalidate such a mental model, if
> that is what the standard intends.
> 
> /df
> 
> --
> London SW6
> UK
> 

Re: XCU: 'return' from subshell

[Joerg Schilling]

Don Cragun  wrote:

> Would this issue be resolved if we change the last sentence of the 
> description section of the return Special Built-In Utility from:
>   If the shell is not currently executing a function
>   or dot script, the results are unspecified.
> to:
>   If the shell is not currently executing a function
>   or dot script running in the same shell execution
>   environment as the command that invoked the function
>   or dot script, the results are unspecified.
> ?

For me, this would be helful since

f() {
(return)
echo foo
}

prints foo in all shells I tried.

So the hint to the same execution environment is definitely helpful.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 15:23, Chet Ramey wrote:

...>
What does a `return from the execution environment' mean, exactly? ...


To clarify, what I wrote was shorthand for "return from the function if
the 'return' is executed in the same execution environment as" the
function's defining command, or otherwise (ii) exit or (iii) unspecified
behaviour.

So I think your agreement with case (i) means that you would be looking
for some text like that in my originally (10/03/20 15:22) proposed
Resolution (a), and therefore (with the clarification above) a
definition of when a 'return' is "in" a function for case (i) that would
make it equivalent to case (iii).

Then the remaining question is whether (case (ii)) there is benefit in
codifying, normatively or not, the 'exit'-like behaviour when a 'return'
that is textually contained in a function definition is run in a
'separate execution environment' from the function's defining command,
rather than leaving all other cases unspecified.

The case (naively) of 'return' used textually outside a function
definition would be covered by existing 2.14 text "If the shell is not
currently executing a function or dot script", except that it is widely
held that "not currently executing a function" includes the situation of
the previous paragraph. Whatever the wording, it appears controversial
whether the 'exit'-like behaviour should apply here.

Without wishing to extend the scope of discussion, the model against
which readers are likely to interpret the standard's text on functions
is that of a programming language like C, where a function foo() is said
to be executing while any statements textually contained in the function
are executing, and possibly more than once if it performed a successful
fork(). Any revised wording needs to invalidate such a mental model, if
that is what the standard intends.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 15:01, Joerg Schilling wrote:
> Dirk Fieldhouse  wrote:
>
>> As to "baffling semantics", I suggest that these are two examples
>> where 'return' is meaningful (and far from baffling) and that "foo"
>> and not "bar" should be printed in each case:
>>
>> f1() {
>> ( echo foo; return )
>> echo bar
>> }
>
> Do you know a single shell that prints only foo in this case?
>
> All shells I am aware of print foo and bar

The discussion seems to have confirmed that this is the general existing
practice, and not just in the few cases I tested, but IMO only a shell
implementer could see the suggested behaviour of these examples as
baffling, based on the wording of the standard (not to mention "man sh",
etc, so I won't). For instance, Chet Ramey's (11/03/20 15:23) comments
on examples f1() and f2() draw on under-the-hood rationale that does not
appear to be reflected in the standard text. In particular I don't see
any suggestion that either usage might be "unspecified".

The question is what, if any, rewording of the standard should be made.
There are plenty of choices for better designed scripting languages, so
arguably making the specification agree with existing practice would be
an acceptable resolution. The example of DR 842 for 'break' and
'continue' shows that this should not be seen as an unnecessary change.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Joerg Schilling]

Chet Ramey  wrote:

> On 3/11/20 11:46 AM, Joerg Schilling wrote:
>
> > Since you most likely develop on Linux
>
> I don't; don't make assumptions.

Interesting, where do you develop?

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 11:46 AM, Joerg Schilling wrote:

> Since you most likely develop on Linux

I don't; don't make assumptions.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Joerg Schilling]

Chet Ramey  wrote:

> > I don't see that we should do this, but id you like to be able to reably 
> > get a
> > 
> > ``NOEXEC'' or ``NOTFOUND''
> > 
> > from expanding "$/", there is a need for interprocess communication unless 
> > you 
> > use vfork() for that specific command.
>
> What is "$/"? Nobody, with perhaps the exception of bosh, implements that.

This has been proposed by Don Cragun a few years ago when we had a waitid() 
related discussion where I introduced the ${.sh.code}, ${.sh.status}, ...
that exist in bosh since 5 years. $/ is intended as something similar to $? 
but with better usability.

Since you most likely develop on Linux, which is a platform with broken 
waitid(), you probably have not come to that point before, but SVr4 introduced 
waitid() in 1988 to be able to get the full 32 bits from exit() and UNOS (the 
first UNIX clone from 1980) has already something similar.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 11:30 AM, Joerg Schilling wrote:
> Chet Ramey  wrote:
> 
>>> and that "foo" and not
>>> "bar" should be printed in each case:
>>>
>>> f1() {
>>>   ( echo foo; return )
>>>   echo bar
>>> }
>>
>> This implies some interprocess communication between the parent and child
>> that simply doesn't exist, and nothing in the standard indicates that it
>> should.
> 
> I don't see that we should do this, but id you like to be able to reably get a
> 
>   ``NOEXEC'' or ``NOTFOUND''
> 
> from expanding "$/", there is a need for interprocess communication unless 
> you 
> use vfork() for that specific command.

What is "$/"? Nobody, with perhaps the exception of bosh, implements that.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 11:13 AM, Stephane Chazelas wrote:
> 2020-03-11 09:55:57 -0400, Chet Ramey:
>> On 3/11/20 5:43 AM, Stephane Chazelas wrote:
>>
>>> AFAIK, bash and bosh are the only shells that complain when you
>>> use return outside of functions/sourced scripts (bash also
>>> doesn't exit upon that failing "return" special builtin in that
>>> case which could be seen as a conformance bug). 
>>
>> You really should try posix mode.
> [...]
> 
> As it happens, I did in that case, and I found it behaved the
> same in or outside of POSIX mode:
> 
> $ bash -c 'return; echo "$?"'
> bash: line 0: return: can only `return' from a function or sourced script
> 1
> $ bash -o posix -c 'return; echo "$?"'
> bash: line 0: return: can only `return' from a function or sourced script

This wasn't covered in my previous message; it was changed in January 2019.

$ ./bash -o posix -c 'return ; echo after'
./bash: line 0: return: can only `return' from a function or sourced script

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Joerg Schilling]

Chet Ramey  wrote:

> > and that "foo" and not
> > "bar" should be printed in each case:
> > 
> > f1() {
> >   ( echo foo; return )
> >   echo bar
> > }
>
> This implies some interprocess communication between the parent and child
> that simply doesn't exist, and nothing in the standard indicates that it
> should.

I don't see that we should do this, but id you like to be able to reably get a

``NOEXEC'' or ``NOTFOUND''

from expanding "$/", there is a need for interprocess communication unless you 
use vfork() for that specific command.

bosh maps a single page of data for this purpose...



Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 11:13 AM, Stephane Chazelas wrote:
> 2020-03-11 09:55:57 -0400, Chet Ramey:
>> On 3/11/20 5:43 AM, Stephane Chazelas wrote:
>>
>>> AFAIK, bash and bosh are the only shells that complain when you
>>> use return outside of functions/sourced scripts (bash also
>>> doesn't exit upon that failing "return" special builtin in that
>>> case which could be seen as a conformance bug). 
>>
>> You really should try posix mode.
> [...]
> 
> As it happens, I did in that case, and I found it behaved the
> same in or outside of POSIX mode:

$ ./bash ./x5
./x5: line 1: return: can only `return' from a function or sourced script
after
$ ./bash -o posix ./x5
./x5: line 1: return: can only `return' from a function or sourced script
$ cat x5
return 7
echo after



-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 10:49 AM, Dirk Fieldhouse wrote:
> On 11/03/20 14:03, Chet Ramey wrote:
>>...>
>> So what's the goal here? That the function continue execution in the
>> subshell so `return' has consistent, if baffling, semantics? That we
>> tighten up the language to make the unspecified specific? What is this
>> discussion intended to accomplish?
> 
> I refer you to this excerpt:
> 
>> On 3/11/20 9:12 AM, Dirk Fieldhouse wrote:>...>
>>> a)    the wording of the standard about 'return' doesn't say this (or
>>> as you said,
 what [I] believe it appears to say, and what it
 actually means, are probably not the same thing.
>>> which is not a good look for a standard);
>>...>
> 
> If the standard can easily be misinterpreted, it ought to be reworded.

So the latter, then. We can move on to proposing language.

> The interesting discussion prompted by my original post indicates that
> even experts don't agree on the interpretation of the text that
> specifies 'return'.
> 
> Did the wise authors of that text mean 'return' to cause:
> 
> i)    a return from the function's lexical scope, subject to some missing
> definition of that scope, or

Yes.

> 
> ii)    a return from the execution environment of the function's defining
> command, or otherwise like 'exit', or

In the case of a subshell or other separate execution environment, the
`exit' seems the most reasonable action. It would be far worse if the
function continued execution in a subshell.

> 
> iii)    a return from the execution environment of the function's defining
> command, or otherwise unspecified?
> 
> Perhaps the answer is (i) but owing to existing practice the standard
> should say (ii) or (iii).

What does a `return from the execution environment' mean, exactly? Does
it mean that the calling shell should exit somehow? Since functions are
executed in the same execution environment as the caller, and subshells
are created as necessary as part of the function body execution, does
the `defining command' mean the caller, or something else?

> 
> As to "baffling semantics", I suggest that these are two examples where
> 'return' is meaningful (and far from baffling) 

I assert that having the function (and the rest of any script) continue
to execute in a subshell because a `return' appeared in a subshell would
be baffling and difficult to explain to users.


> and that "foo" and not
> "bar" should be printed in each case:
> 
> f1() {
>   ( echo foo; return )
>   echo bar
> }

This implies some interprocess communication between the parent and child
that simply doesn't exist, and nothing in the standard indicates that it
should.

> 
> and
> 
> f2() {
>   echo foo |
>     if read -r xx && [ "$xx" = foo ]; then
>    echo "$xx"; return
>     else
>    echo "$xx"
>     fi
>   echo bar
> }

This is unspecified, and has been ever since ksh decided to run the last
pipeline element in the current shell process (or execution environment,
if you prefer). You can't rely on either behavior. The number of shells
that print `bar' exceeds the number that don't, for what that's worth.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Stephane Chazelas]

2020-03-11 09:55:57 -0400, Chet Ramey:
> On 3/11/20 5:43 AM, Stephane Chazelas wrote:
> 
> > AFAIK, bash and bosh are the only shells that complain when you
> > use return outside of functions/sourced scripts (bash also
> > doesn't exit upon that failing "return" special builtin in that
> > case which could be seen as a conformance bug). 
> 
> You really should try posix mode.
[...]

As it happens, I did in that case, and I found it behaved the
same in or outside of POSIX mode:

$ bash -c 'return; echo "$?"'
bash: line 0: return: can only `return' from a function or sourced script
1
$ bash -o posix -c 'return; echo "$?"'
bash: line 0: return: can only `return' from a function or sourced script
1
$ bash -c 'printf "%s\n" "$BASH_VERSION"'
5.0.3(1)-release

-- 
Stephane

Re: XCU: 'return' from subshell

[Joerg Schilling]

Dirk Fieldhouse  wrote:

> As to "baffling semantics", I suggest that these are two examples where
> 'return' is meaningful (and far from baffling) and that "foo" and not
> "bar" should be printed in each case:
>
> f1() {
>( echo foo; return )
>echo bar
> }

Dop yo know a single shell that prints only foo in this case?

All shells I am aware of print foo and bar

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 14:03, Chet Ramey wrote:
>...>
> So what's the goal here? That the function continue execution in the
> subshell so `return' has consistent, if baffling, semantics? That we
> tighten up the language to make the unspecified specific? What is this
> discussion intended to accomplish?

I refer you to this excerpt:

> On 3/11/20 9:12 AM, Dirk Fieldhouse wrote:>...>
>> a)the wording of the standard about 'return' doesn't say this (or
>> as you said,
>>> what [I] believe it appears to say, and what it
>>> actually means, are probably not the same thing.
>> which is not a good look for a standard);
>...>

If the standard can easily be misinterpreted, it ought to be reworded.
The interesting discussion prompted by my original post indicates that
even experts don't agree on the interpretation of the text that
specifies 'return'.

Did the wise authors of that text mean 'return' to cause:

i)  a return from the function's lexical scope, subject to some missing
definition of that scope, or

ii) a return from the execution environment of the function's defining
command, or otherwise like 'exit', or

iii)a return from the execution environment of the function's defining
command, or otherwise unspecified?

Perhaps the answer is (i) but owing to existing practice the standard
should say (ii) or (iii).

As to "baffling semantics", I suggest that these are two examples where
'return' is meaningful (and far from baffling) and that "foo" and not
"bar" should be printed in each case:

f1() {
  ( echo foo; return )
  echo bar
}

and

f2() {
  echo foo |
if read -r xx && [ "$xx" = foo ]; then
   echo "$xx"; return
else
   echo "$xx"
fi
  echo bar
}

Without committing to the usefulness of any example, in a usage like
f1() the author might want to use some variables without overwriting them.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 9:12 AM, Dirk Fieldhouse wrote:
> On 11/03/20 06:25, Robert Elz wrote:
>>
>> ... The standard by [referring to 'subshell environment'] is trying ...
>> to avoid constraining the implementation, things that an implementation
>> can work out how to do without forking it can do (which will make it
>> faster, and less expensive to run) - but it must preserve the fiction
>> that it has forked, as other shells do, and scripts are allowed to
>> rely upon that, so no side effects (such as a return in a subshell
>> causing a function in the parent to return) are permitted.
> 
> Absolutely, but
> 
> a)    the wording of the standard about 'return' doesn't say this (or as
> you said,
>> what [I] believe it appears to say, and what it
>> actually means, are probably not the same thing.
> which is not a good look for a standard);
> 
> b)    in particular, returning from a subshell is not one of the forbidden
> side-effects that is mentioned in or can be inferred from the text of 2.12.
> 
> Isn't returning on use of 'return' an effect (ie, the actual behaviour
> expected by the script author) rather than a side-effect?

So what's the goal here? That the function continue execution in the
subshell so `return' has consistent, if baffling, semantics? That we
tighten up the language to make the unspecified specific? What is this
discussion intended to accomplish?

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Chet Ramey]

On 3/11/20 5:43 AM, Stephane Chazelas wrote:

> AFAIK, bash and bosh are the only shells that complain when you
> use return outside of functions/sourced scripts (bash also
> doesn't exit upon that failing "return" special builtin in that
> case which could be seen as a conformance bug). 

You really should try posix mode.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 06:25, Robert Elz wrote:


... The standard by [referring to 'subshell environment'] is trying ...
to avoid constraining the implementation, things that an implementation
can work out how to do without forking it can do (which will make it
faster, and less expensive to run) - but it must preserve the fiction
that it has forked, as other shells do, and scripts are allowed to
rely upon that, so no side effects (such as a return in a subshell
causing a function in the parent to return) are permitted.


Absolutely, but

a)  the wording of the standard about 'return' doesn't say this (or as
you said,

what [I] believe it appears to say, and what it
actually means, are probably not the same thing.

which is not a good look for a standard);

b)  in particular, returning from a subshell is not one of the forbidden
side-effects that is mentioned in or can be inferred from the text of 2.12.

Isn't returning on use of 'return' an effect (ie, the actual behaviour
expected by the script author) rather than a side-effect?

Clearly there are cases when it would be essentially meaningless or
undesirable but a context that is synchronous wrt the calling script
such as an and-or-list in a &-less pipeline or subshell compound-command
could be a reasonable place from which to return from the enclosing
function as the standard implies, or appears to some readers to do.


...


/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 11/03/20 06:31, Stephane Chazelas wrote:

IMO,

the best and only reasonable way to address it is to specify
what all shells do: return works like exit when called in a
subshell.


But behaving like 'exit' is not the same as exiting the function and
resuming at the next command, which is clearly stated as a requirement,
albeit difficult to interpret, by the standard.

In

f() (
   foo || return
   bar
)

that return should be guaranteed to exit the function (with
foo's exit status) if foo fails.


And without calling bar, as the name of the 'return' utility implies.


It's also important to keep the guarantee that nothing you do in
a subshell should affect the parent shell (except of course
explicit process killing or poweroff or other pathological
cases). ...


This guarantee (as often) doesn't have such extensive coverage:

"2.9.4 ... ( compound-list ) ...Variable assignments and built-in
commands that affect the environment shall not remain in effect after
the list finishes."

"2.12 ... A shell execution environment consists of the following:
[ a list of things not affected by 'return' that are preserved in the
parent shell ]"

So there's no restriction on 'return' from a subshell that I can find,
absent additional wording in 2.14.


I don't think many would expect that in:

f() {
   return | cat
   echo "$(return)"
   return &
   (return)
   echo done
}
any of those returns would return from the function. I don't see
it desirable that it does. I'm ok for the first one to be left
unspecified but only on the ground that POSIX currently leaves
it unspecified which component of a pipeline may not be run in a
subshell environment (though in practice, shells run either all
components in subshells or only the rightmost one).


"2.12 ... each command of a multi-command pipeline is in a subshell
environment; as an extension [whatever that means], however, any or all
commands in a pipeline may be executed in the current environment..."

So the portable script author has to assume that "all commands" in a
pipeline are run in subshells.

Leaving aside the rationale for each usage, from reading the standard I
would expect all of these examples to return from the function, except
the asynchronous case because its weirdness breaks the necessary
suspension of disbelief.

I suggest that what a script author would prefer is that at least
"return | cat" and "(return)" cause the function to exit without
printing "done", and the other cases to behave like 'exit' on the
principle of "do what I mean".

And on 11/03/20 09:43, Stephane Chazelas wrote:
> On 2020-03-11 15:38:10 +0700, Robert Elz:
>...>
>>

So, it looks as if we could define that return in a subshell in a function
causes the subshell to exit - that said, I'm not sure that there is a lot
of need, it is a confusing usage, and does the same as exit does, so it
is probably better for users to simply use exit instead.

> [...]
>
> But coding patterns like:
>
> f() (
>foo || return
>...
> )
>
> are common.
>
> If only for that, I think POSIX should require "return" to
> behave like exit in that case.

Agreed that this is a common pattern, but wouldn't people be unhappy if
calling f() with failing foo caused the calling script to exit when they
expected it just to return from the function? I know I would be.

> ...

And finally on 11/03/20 06:37, Stephane Chazelas wrote:
>...>
>
> All the ones I tried output 42 and only 42 in
>
> sh -c 'f() { (return 42; echo x); echo "$?"; }; f'

This example is overloaded: is 42 a H2G2 reference, or to the other
memorable event of 1978 (+42=2020)?

"S. R. Bourne, UNIX Time-Sharing System: The UNIX Shell, Bell System
Technical Journal, Volume 57 number 6, July-August 1978"

But SRB, perhaps prudently, omitted functions from his shell: they had
arrived by System V. The paper apparently listed the {} and ()
command-list syntaxes without defining them; you just can't get the
referees.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Stephane Chazelas]

2020-03-11 18:42:48 +0700, Robert Elz:
[...]
>   | f() (
>   |   foo || return
>   |   ...
>   | )
>   |
>   | are common.
> 
> Really?   I have only rarely ever seen (aside from the occasional one
> I write myself) sub-shell functions like that very rarely.

I certainly use it and have seen it been used.

And already had that discussion before that POSIX required those
functions to be exited with "exit" instead of "return", probably
here though I couldn't tell for sure. Might have been
comp.unix.shell. It did come up on the dash mailing list (which
I suspect is where this current discussion is spawning from).

[...]
> 
>   | AFAIK, bash and bosh are the only shells that complain when you
>   | use return outside of functions/sourced scripts (bash also
>   | doesn't exit upon that failing "return" special builtin in that
>   | case which could be seen as a conformance bug).
> 
> To the NetBSD sh, return in a script (not in a function) is a no-op.
> Not an error, just a nothing (I don't think it even sets $? but I might
> be mistaken about that.)
> 
> You can sit in front of an interactive shell and type return as many
> times as you like, and none of them will do a thing.
[...]

Indeed. Seems like it's unique to ash-based shells. dash,
FreeBSD sh, busybox ash all exit upon return, interactive or
not.

I agree though that "return" exiting an interactive shell seems
slightly wrong. It being silently ignored feels a lot wronger
though, as there's no way the user could expect "cmd" to be run
in "return; cmd". Is that really an intentional change in NetBSD
sh?

zsh and yash's return also won't exit interactive shells (they
still exit subshells of interactive shells).

yash's:

$ return; echo "$?"
return: cannot be used in the interactive mode
1

(return still doesn't return there but at least, like in
bash/bosh, users are notified that they did something wrong).

zsh's "return" returns to the prompt (with $? set accordingly)
which seems the best approach to me.

-- 
Stephane

Re: XCU: 'return' from subshell

[Robert Elz]

Date:Wed, 11 Mar 2020 09:43:06 +
From:Stephane Chazelas 
Message-ID:  <20200311094306.kqqdsjyn7v4ik...@chazelas.org>


  | But coding patterns like:
  |
  | f() (
  |   foo || return
  |   ...
  | )
  |
  | are common.

Really?   I have only rarely ever seen (aside from the occasional one
I write myself) sub-shell functions like that very rarely.   I suspect
that most users don't even know it is legal.

I have occasionally seen

f() {(
# whetever, not usually including return
)}

which is just weird, but fits people's notions of what a sh function
should look like...

  | If only for that, I think POSIX should require "return" to
  | behave like exit in that case.

Yes, as I said we could do that, since it seems everyone already does.
I just don't think scripts should be written that way, subshells exit,
they don't return, and so it is clearer and more understandable to write
it that way,

  | It would make sense for "return" to act as exit even outside of
  | functions or sourced files. So you can have a script that can be
  | both sourced and executed (or used as autoloadable function
  | where supported).

Yes, there are times I would have liked that - but that definitely doesn't
work in general now - it would be new.

  | AFAIK, bash and bosh are the only shells that complain when you
  | use return outside of functions/sourced scripts (bash also
  | doesn't exit upon that failing "return" special builtin in that
  | case which could be seen as a conformance bug).

To the NetBSD sh, return in a script (not in a function) is a no-op.
Not an error, just a nothing (I don't think it even sets $? but I might
be mistaken about that.)

You can sit in front of an interactive shell and type return as many
times as you like, and none of them will do a thing.

  | They're also
  | the only shells that don't support some form or other of
  | autoloadable functions (though support has also been removed in
  | recent versions of some ash-based shells) which may be linked.

That would be unrelated.   I think it is more that the auto-load stuff
was barely documented, and in practice, never used, or almost never.
NetBSD doesn't even bother installing the small handful of scripts
written with the intent to use that (providing pushd  and a few
more similar things).

kre

Re: XCU: 'return' from subshell

[Stephane Chazelas]

2020-03-11 15:38:10 +0700, Robert Elz:
> Date:Wed, 11 Mar 2020 06:37:41 +
> From:Stephane Chazelas 
> Message-ID:  <20200311063741.wpkuiffb7jsyv...@chazelas.org>
> 
>   | What shell implementation "ignores" the return or "treat it as an
>   | error" here.
> 
> You're right, I must have been remembering (always dangerous when it is
> me doing it) a different test of return in an inappropriate context.
> 
> So, it looks as if we could define that return in a subshell in a function
> causes the subshell to exit - that said, I'm not sure that there is a lot
> of need, it is a confusing usage, and does the same as exit does, so it
> is probably ybetter for users to simply use exit instead.
[...]

But coding patterns like:

f() (
  foo || return
  ...
)

are common.

If only for that, I think POSIX should require "return" to
behave like exit in that case.

It would make sense for "return" to act as exit even outside of
functions or sourced files. So you can have a script that can be
both sourced and executed (or used as autoloadable function
where supported). And that would make the interface more
consistent: return returns  from the immediate subshell or
function or sourced file or the shell.

AFAIK, bash and bosh are the only shells that complain when you
use return outside of functions/sourced scripts (bash also
doesn't exit upon that failing "return" special builtin in that
case which could be seen as a conformance bug). They're also
the only shells that don't support some form or other of
autoloadable functions (though support has also been removed in
recent versions of some ash-based shells) which may be linked.

-- 
Stephane

Re: XCU: 'return' from subshell

[Robert Elz]

Date:Wed, 11 Mar 2020 06:37:41 +
From:Stephane Chazelas 
Message-ID:  <20200311063741.wpkuiffb7jsyv...@chazelas.org>

  | What shell implementation "ignores" the return or "treat it as an
  | error" here.

You're right, I must have been remembering (always dangerous when it is
me doing it) a different test of return in an inappropriate context.

So, it looks as if we could define that return in a subshell in a function
causes the subshell to exit - that said, I'm not sure that there is a lot
of need, it is a confusing usage, and does the same as exit does, so it
is probably ybetter for users to simply use exit instead.

kre

Re: XCU: 'return' from subshell

[Stephane Chazelas]

2020-03-11 13:25:40 +0700, Robert Elz:
[...]
> What a return (or break or continue) in a subshell environment (but
> not in a function or dot script (or loop for break and continue) in that
> subshell does is unspecified.   Some shells make it equivalent to exit,
> others ignore it, and others treat it as an error (which would then cause
> the sub-shell to exit).   Scripts cannot rely upon anything here, so
> should never use return in a subshell (outside of a function called in
> that subshell).
[...]

What shell implementation "ignores" the return or "treat it as an
error" here.

All the ones I tried output 42 and only 42 in

sh -c 'f() { (return 42; echo x); echo "$?"; }; f'

-- 
Stephane

Re: XCU: 'return' from subshell

[Stephane Chazelas]

IMO,

the best and only reasonable way to address it is to specify
what all shells do: return works like exit when called in a
subshell.

In

f() (
  foo || return
  bar
)

that return should be guaranteed to exit the function (with
foo's exit status) if foo fails.

It's also important to keep the guarantee that nothing you do in
a subshell should affect the parent shell (except of course
explicit process killing or poweroff or other pathological
cases). in particular:

f() {
  (eval "$something"; foo); echo done
}

The $something evaluation might cause the subshell to exit, but
"done" should always be printed.


I don't think many would expect that in:

f() {
  return | cat
  echo "$(return)"
  return &
  (return)
  echo done
}

any of those returns would return from the function. I don't see
it desirable that it does. I'm ok for the first one to be left
unspecified but only on the ground that POSIX currently leaves
it unspecified which component of a pipeline may not be run in a
subshell environment (though in practice, shells run either all
components in subshells or only the rightmost one).

-- 
Stephane

Re: XCU: 'return' from subshell

[Robert Elz]

If there is anything that needs fixing about this (I haven't checked
the whole standard for all the places where there might be relevant
text) it is to make it more clear that a sub-shell environment cannot
affect the parent shell in any way at all - other than via explicit I/O
(the output from a command substitution, data through pipes, etc),
the exit status of the subshell, and explicit signals/traps.

None of the semi-syntax commands (they're all executable commands) can
possibly have any effect upon anything happening in the parent shell.

If that needs to be made more clear, then it should be done once,
for all of them (break/continue/return/exit) - that is, if, personally
this was never something that ever occurred to me.

Part of the issue might be the way the standard refers to subshell
environment, where what generally happens is a fork - and a whole
new process.  The standard by doing that is trying (for good reason)
to avoid constraining the implementation, things that an implementation
can work out how to do without forking it can do (which will make it
faster, and less expensive to run) - but it must preserve the fiction
that it has forked, as other shells do, and scripts are allowed to
rely upon that, so no side effects (such as a return in a subshell
causing a function in the parent to return) are permitted.

What a return (or break or continue) in a subshell environment (but
not in a function or dot script (or loop for break and continue) in that
subshell does is unspecified.   Some shells make it equivalent to exit,
others ignore it, and others treat it as an error (which would then cause
the sub-shell to exit).   Scripts cannot rely upon anything here, so
should never use return in a subshell (outside of a function called in
that subshell).

Also there is no "scope" really anywhere in the shell, or rather there
is global scope, but that's really it.   For more on this, and why, see
the reply I intend to send soon to Donn Terry's message.

For a couple of explicit statements in this message:

 |  But this is restating the wording of the standard, unless "in a shell
 |  execution environment" means "in a shell execution environment, and not
 |  in a subshell environment thereof",

That is exactly what it means - a subshell environment is a different
environment from the parent shells environment - they do not nest, they
are distinct (and once again, as above, for "execution environment"
think "process" - a "subshell execution environment" is the child of
a shell that has forked, just in standards language).

wrt your possible resolutions:

   a)   the existing 2.14 and 2.95 text means to permit the interpretation
that 'return' from a function when in a subshell context may just exit
the subshell and not return from the function;

It cannot exit the function, that's impossible while remaining consistent
with the rest of the standard.   The "may just exit" isn't guaranteed either,
as your oracle stated, what the return there does is unspecified, it might
simply be "nothing".

   b)   the existing 2.14 text is consistent with the observed behaviour
but the 2.95 text specifying function definition must be changed
to restrict the types of command that can be used in the definition;

I have no idea what that means - any commands can be used in a function,
the only issue is whether they have specified results or not.   Note that
not everyone cares about that - scripts written for a particular shell can
make use of whatever it happens to do  for what the standard leaves as
unspecified - they just won't be portable.

c)  the text of 2.14 and 2.95 means what it appears to say

That's correct, but what you believe it appears to say, and what it
actually means, are probably not the same thing.

kre

Re: XCU: 'return' from subshell

[shwaresyst]

After some thought, I believe I'd be in favor more of: a) adding explicitly 
that '{' and '(' introduce a new lexical scope, balanced by '}' and ')' 
respectively; b) when such scopes are asynchronous return shall function the 
same as exit, with, as App Usage, it is up to any trap action for CHLD to store 
a return or exit parameter value for that result to be available to any 
monitoring loop in the main script; and c) something like "return &;" shall be 
equivalent to "{ return &; }" in effect, for that corner case.
On Tuesday, March 10, 2020 Dirk Fieldhouse  wrote:
On 10/03/20 18:25, shwaresyst wrote:
 >
 > I basically agree this is an issue - I see return as more for being
interpreted as a lexical scope abort, whatever the execution context,
and exit an execution scope abort, such as a subshell or separate script
utility environment, as their basic intent. ...

As this is what the reader with no shell source code would take from the
standard, the problem (one of them) is how to converge the spec and
implementations with the least overall pain.

 >                    ... Further complicating things, I don't see that the 
 >standard or
the fixes proposed here adequately addresses the expectations when such
a return is part of a brace group or subshell executed as an
asynchronous command, using '&'.
 >...

The issue of asynchronous lists came to me, obviously, walking the dog
after posting.

If 'return' is a lexical scope abort, an asynchronous list is not a
meaningful place for it to be used. So one's expectations based on the
standard should be for such usage to invoke unspecified behaviour, such
as ignoring the 'return' with/out a warning or generating an error.

The proposed wording in my Resolution (a) to restrict the specification
of 'return' to cases where it is "executing in the same execution
environment as the function's defining compound-command" would in fact
cover asynchronous lists. But it would leave this unspecified:

f() {
  ( echo "What will happen next is Unspecified"; return )
  echo "This probably should not be reached, but typically is"
}

New wording would be required in my Resolution (c), eg in 2.14 'return'
(inside_ _):

"... If the shell is not currently executing a function or dot script,
_or if the current execution context is asynchronous with respect to the
execution context of the function's defining command or the caller of
the dot script,_ the results are unspecified."

Unsurprisingly, the tested implementations are consistent in treating
'return' in an asynchronous context like 'exit' as well.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 10/03/20 18:25, shwaresyst wrote:
>
> I basically agree this is an issue - I see return as more for being
interpreted as a lexical scope abort, whatever the execution context,
and exit an execution scope abort, such as a subshell or separate script
utility environment, as their basic intent. ...

As this is what the reader with no shell source code would take from the
standard, the problem (one of them) is how to converge the spec and
implementations with the least overall pain.

>... Further complicating things, I don't 
see that the standard or
the fixes proposed here adequately addresses the expectations when such
a return is part of a brace group or subshell executed as an
asynchronous command, using '&'.
>...

The issue of asynchronous lists came to me, obviously, walking the dog
after posting.

If 'return' is a lexical scope abort, an asynchronous list is not a
meaningful place for it to be used. So one's expectations based on the
standard should be for such usage to invoke unspecified behaviour, such
as ignoring the 'return' with/out a warning or generating an error.

The proposed wording in my Resolution (a) to restrict the specification
of 'return' to cases where it is "executing in the same execution
environment as the function's defining compound-command" would in fact
cover asynchronous lists. But it would leave this unspecified:

f() {
  ( echo "What will happen next is Unspecified"; return )
  echo "This probably should not be reached, but typically is"
}

New wording would be required in my Resolution (c), eg in 2.14 'return'
(inside_ _):

"... If the shell is not currently executing a function or dot script,
_or if the current execution context is asynchronous with respect to the
execution context of the function's defining command or the caller of
the dot script,_ the results are unspecified."

Unsurprisingly, the tested implementations are consistent in treating
'return' in an asynchronous context like 'exit' as well.

/df

--
London SW6
UK

RE: XCU: 'return' from subshell

[shwaresyst]

I basically agree this is an issue - I see return as more for being interpreted 
as a lexical scope abort, whatever the execution context, and exit an execution 
scope abort, such as a subshell or separate script utility environment, as 
their basic intent. Further complicating things, I don't see that the standard 
or the fixes proposed here adequately addresses the expectations when such a 
return is part of a brace group or subshell executed as an asynchronous 
command, using '&'.
On Tuesday, March 10, 2020 Dirk Fieldhouse  wrote:
1    Summary

XCU Ch2.14 states that 'return' shall cause the shell to leave the
current function or dot script, if any. Ch2.95 says that execution shall
continue with the next command after the function call. Implementations
that claim conformance consistently contradict this specification, if
the function has created a subshell. They can't both be right. As the
specification was in part intended to codify existing practice, how did
this contradiction arise?

2    Description

2.1    Relevant specifications

POSIX Vol3 (XCU) Ch2.14 'return'

says, and has since at least 2004:

> The return utility shall cause the shell to stop executing the current 
> function or dot script. If the shell is not currently executing a function or 
> dot script, the results are unspecified.
In case this wasn't clear enough, in Vol3 Ch2.9.5
:

> A function is a user-defined name that is used as a simple command to call a 
> compound command with new positional parameters. ...
>
> The compound-command shall be executed whenever the function name is 
> specified as the name of a simple command... If the special built-in return 
> ... is executed in the compound-command, the function completes and execution 
> shall resume with the next command after the function call.

So not with some other command inside the function, you'd imagine.

2.2    The issue

Implementations consistently contradict the above wording when the
'return' is in a 'subshell' (probably any 'separate shell execution
context'), treating this use of return as if the script used 'exit'. Yet
suppliers have been claiming conformance to the existing wording.

Are they all wrong, or is an there an adaptation or interpretation of
the specification that would align it with reality?

2.3    Example

A simple test case is

foo() {
    ( if [ "$1" = fum ]; then echo EQ; return 0; fi )
    echo NE; return 1
}

with expected result

$ foo fum || echo WTF
EQ
$

but in the several modern shells (dash-0.5.9.1 and earlier,
bash-4.3-14ubuntu1.4, busybox-static-1:1.22.0-15ubuntu1.4) tested, we get

$ foo fum || echo WTF
EQ
NE
WTF
$

2.4    Interpretations

One oracle has said:

> In the subshell, the shell should not be considered to still be executing a 
> function or dot script. As such, the results should be unspecified, and any 
> behaviour should be valid. The standard may be underspecified here, but any 
> other interpretation is not reasonable.

But if you read the standard without having knowledge of existing shell
internals, it's entirely reasonable (and IMO desirable) to consider that
a shell function is a lexical group, like a script file, which is being
executed as long as any command within the function's defining
compound-command is running; as the spec refers to subshells explicitly
elsewhere (eg 'exit') the reader would have to believe that "subshell"
was accidentally omitted from the list of contexts that 'return' should
return from, to interpret the text as quoted above.

The resolution of a related rejected Defect Report 1042
 says:

> ... the results of using return when you are not in a shell execution 
> environment running a function or a dot script is unspecified.

But this is restating the wording of the standard, unless "in a shell
execution environment" means "in a shell execution environment, and not
in a subshell environment thereof", which, as argued above, is
additional to that wording.

Under DR 842, clarifications
 have been made
on the scope of the 'break' and 'continue' special utilities so that the
expected behaviour matches the specification; 'return' did not receive
such attention.

3    Resolution

I considered these possible resolutions, though others may exist:

a)    the existing 2.14 and 2.95 text means to permit the interpretation
that 'return' from a function when in a subshell context may just exit
the subshell and not return from the function;

b)    the existing 2.14 text is consistent with the observed behaviour but
the 2.95 text specifying function definition must be changed to restrict
the types of command that can be used in the definition;

c)    the text of 2.14 and 2.95 means what it appears to say.

3.1