XCU: 'return' from subshell

[Dirk Fieldhouse]

1   Summary

XCU Ch2.14 states that 'return' shall cause the shell to leave the
current function or dot script, if any. Ch2.95 says that execution shall
continue with the next command after the function call. Implementations
that claim conformance consistently contradict this specification, if
the function has created a subshell. They can't both be right. As the
specification was in part intended to codify existing practice, how did
this contradiction arise?

2   Description

2.1 Relevant specifications

POSIX Vol3 (XCU) Ch2.14 'return'

says, and has since at least 2004:


The return utility shall cause the shell to stop executing the current function 
or dot script. If the shell is not currently executing a function or dot 
script, the results are unspecified.

In case this wasn't clear enough, in Vol3 Ch2.9.5
:


A function is a user-defined name that is used as a simple command to call a 
compound command with new positional parameters. ...

The compound-command shall be executed whenever the function name is specified 
as the name of a simple command... If the special built-in return ... is 
executed in the compound-command, the function completes and execution shall 
resume with the next command after the function call.


So not with some other command inside the function, you'd imagine.

2.2 The issue

Implementations consistently contradict the above wording when the
'return' is in a 'subshell' (probably any 'separate shell execution
context'), treating this use of return as if the script used 'exit'. Yet
suppliers have been claiming conformance to the existing wording.

Are they all wrong, or is an there an adaptation or interpretation of
the specification that would align it with reality?

2.3 Example

A simple test case is

foo() {
   ( if [ "$1" = fum ]; then echo EQ; return 0; fi )
   echo NE; return 1
}

with expected result

$ foo fum || echo WTF
EQ
$

but in the several modern shells (dash-0.5.9.1 and earlier,
bash-4.3-14ubuntu1.4, busybox-static-1:1.22.0-15ubuntu1.4) tested, we get

$ foo fum || echo WTF
EQ
NE
WTF
$

2.4 Interpretations

One oracle has said:


In the subshell, the shell should not be considered to still be executing a 
function or dot script. As such, the results should be unspecified, and any 
behaviour should be valid. The standard may be underspecified here, but any 
other interpretation is not reasonable.


But if you read the standard without having knowledge of existing shell
internals, it's entirely reasonable (and IMO desirable) to consider that
a shell function is a lexical group, like a script file, which is being
executed as long as any command within the function's defining
compound-command is running; as the spec refers to subshells explicitly
elsewhere (eg 'exit') the reader would have to believe that "subshell"
was accidentally omitted from the list of contexts that 'return' should
return from, to interpret the text as quoted above.

The resolution of a related rejected Defect Report 1042
 says:


... the results of using return when you are not in a shell execution 
environment running a function or a dot script is unspecified.


But this is restating the wording of the standard, unless "in a shell
execution environment" means "in a shell execution environment, and not
in a subshell environment thereof", which, as argued above, is
additional to that wording.

Under DR 842, clarifications
 have been made
on the scope of the 'break' and 'continue' special utilities so that the
expected behaviour matches the specification; 'return' did not receive
such attention.

3   Resolution

I considered these possible resolutions, though others may exist:

a)  the existing 2.14 and 2.95 text means to permit the interpretation
that 'return' from a function when in a subshell context may just exit
the subshell and not return from the function;

b)  the existing 2.14 text is consistent with the observed behaviour but
the 2.95 text specifying function definition must be changed to restrict
the types of command that can be used in the definition;

c)  the text of 2.14 and 2.95 means what it appears to say.

3.1 Resolution (a)

A non-normative clarification should be added to avoid misleading the
naive reader, or the normative text might be revised to add 'separate
shell execution context' to the list of contexts that 'return' returns
from, or to restrict what is meant by executing 'return' "in" a function.

However existing scripts should not be affected, and most, if not all,
shell implementers can rest easy. Shell programmers would still have to
avoid 'return' or use work-arounds to avoid uncertainty as to whether
the shell implementation may decide to 

RE: XCU: 'return' from subshell

[shwaresyst]

I basically agree this is an issue - I see return as more for being interpreted 
as a lexical scope abort, whatever the execution context, and exit an execution 
scope abort, such as a subshell or separate script utility environment, as 
their basic intent. Further complicating things, I don't see that the standard 
or the fixes proposed here adequately addresses the expectations when such a 
return is part of a brace group or subshell executed as an asynchronous 
command, using '&'.
On Tuesday, March 10, 2020 Dirk Fieldhouse  wrote:
1    Summary

XCU Ch2.14 states that 'return' shall cause the shell to leave the
current function or dot script, if any. Ch2.95 says that execution shall
continue with the next command after the function call. Implementations
that claim conformance consistently contradict this specification, if
the function has created a subshell. They can't both be right. As the
specification was in part intended to codify existing practice, how did
this contradiction arise?

2    Description

2.1    Relevant specifications

POSIX Vol3 (XCU) Ch2.14 'return'

says, and has since at least 2004:

> The return utility shall cause the shell to stop executing the current 
> function or dot script. If the shell is not currently executing a function or 
> dot script, the results are unspecified.
In case this wasn't clear enough, in Vol3 Ch2.9.5
:

> A function is a user-defined name that is used as a simple command to call a 
> compound command with new positional parameters. ...
>
> The compound-command shall be executed whenever the function name is 
> specified as the name of a simple command... If the special built-in return 
> ... is executed in the compound-command, the function completes and execution 
> shall resume with the next command after the function call.

So not with some other command inside the function, you'd imagine.

2.2    The issue

Implementations consistently contradict the above wording when the
'return' is in a 'subshell' (probably any 'separate shell execution
context'), treating this use of return as if the script used 'exit'. Yet
suppliers have been claiming conformance to the existing wording.

Are they all wrong, or is an there an adaptation or interpretation of
the specification that would align it with reality?

2.3    Example

A simple test case is

foo() {
    ( if [ "$1" = fum ]; then echo EQ; return 0; fi )
    echo NE; return 1
}

with expected result

$ foo fum || echo WTF
EQ
$

but in the several modern shells (dash-0.5.9.1 and earlier,
bash-4.3-14ubuntu1.4, busybox-static-1:1.22.0-15ubuntu1.4) tested, we get

$ foo fum || echo WTF
EQ
NE
WTF
$

2.4    Interpretations

One oracle has said:

> In the subshell, the shell should not be considered to still be executing a 
> function or dot script. As such, the results should be unspecified, and any 
> behaviour should be valid. The standard may be underspecified here, but any 
> other interpretation is not reasonable.

But if you read the standard without having knowledge of existing shell
internals, it's entirely reasonable (and IMO desirable) to consider that
a shell function is a lexical group, like a script file, which is being
executed as long as any command within the function's defining
compound-command is running; as the spec refers to subshells explicitly
elsewhere (eg 'exit') the reader would have to believe that "subshell"
was accidentally omitted from the list of contexts that 'return' should
return from, to interpret the text as quoted above.

The resolution of a related rejected Defect Report 1042
 says:

> ... the results of using return when you are not in a shell execution 
> environment running a function or a dot script is unspecified.

But this is restating the wording of the standard, unless "in a shell
execution environment" means "in a shell execution environment, and not
in a subshell environment thereof", which, as argued above, is
additional to that wording.

Under DR 842, clarifications
 have been made
on the scope of the 'break' and 'continue' special utilities so that the
expected behaviour matches the specification; 'return' did not receive
such attention.

3    Resolution

I considered these possible resolutions, though others may exist:

a)    the existing 2.14 and 2.95 text means to permit the interpretation
that 'return' from a function when in a subshell context may just exit
the subshell and not return from the function;

b)    the existing 2.14 text is consistent with the observed behaviour but
the 2.95 text specifying function definition must be changed to restrict
the types of command that can be used in the definition;

c)    the text of 2.14 and 2.95 means what it appears to say.

3.1    

Re: XCU: 'return' from subshell

[Dirk Fieldhouse]

On 10/03/20 18:25, shwaresyst wrote:
>
> I basically agree this is an issue - I see return as more for being
interpreted as a lexical scope abort, whatever the execution context,
and exit an execution scope abort, such as a subshell or separate script
utility environment, as their basic intent. ...

As this is what the reader with no shell source code would take from the
standard, the problem (one of them) is how to converge the spec and
implementations with the least overall pain.

>... Further complicating things, I don't 
see that the standard or
the fixes proposed here adequately addresses the expectations when such
a return is part of a brace group or subshell executed as an
asynchronous command, using '&'.
>...

The issue of asynchronous lists came to me, obviously, walking the dog
after posting.

If 'return' is a lexical scope abort, an asynchronous list is not a
meaningful place for it to be used. So one's expectations based on the
standard should be for such usage to invoke unspecified behaviour, such
as ignoring the 'return' with/out a warning or generating an error.

The proposed wording in my Resolution (a) to restrict the specification
of 'return' to cases where it is "executing in the same execution
environment as the function's defining compound-command" would in fact
cover asynchronous lists. But it would leave this unspecified:

f() {
  ( echo "What will happen next is Unspecified"; return )
  echo "This probably should not be reached, but typically is"
}

New wording would be required in my Resolution (c), eg in 2.14 'return'
(inside_ _):

"... If the shell is not currently executing a function or dot script,
_or if the current execution context is asynchronous with respect to the
execution context of the function's defining command or the caller of
the dot script,_ the results are unspecified."

Unsurprisingly, the tested implementations are consistent in treating
'return' in an asynchronous context like 'exit' as well.

/df

--
London SW6
UK

Re: XCU: 'return' from subshell

[shwaresyst]

After some thought, I believe I'd be in favor more of: a) adding explicitly 
that '{' and '(' introduce a new lexical scope, balanced by '}' and ')' 
respectively; b) when such scopes are asynchronous return shall function the 
same as exit, with, as App Usage, it is up to any trap action for CHLD to store 
a return or exit parameter value for that result to be available to any 
monitoring loop in the main script; and c) something like "return &;" shall be 
equivalent to "{ return &; }" in effect, for that corner case.
On Tuesday, March 10, 2020 Dirk Fieldhouse  wrote:
On 10/03/20 18:25, shwaresyst wrote:
 >
 > I basically agree this is an issue - I see return as more for being
interpreted as a lexical scope abort, whatever the execution context,
and exit an execution scope abort, such as a subshell or separate script
utility environment, as their basic intent. ...

As this is what the reader with no shell source code would take from the
standard, the problem (one of them) is how to converge the spec and
implementations with the least overall pain.

 >                    ... Further complicating things, I don't see that the 
 >standard or
the fixes proposed here adequately addresses the expectations when such
a return is part of a brace group or subshell executed as an
asynchronous command, using '&'.
 >...

The issue of asynchronous lists came to me, obviously, walking the dog
after posting.

If 'return' is a lexical scope abort, an asynchronous list is not a
meaningful place for it to be used. So one's expectations based on the
standard should be for such usage to invoke unspecified behaviour, such
as ignoring the 'return' with/out a warning or generating an error.

The proposed wording in my Resolution (a) to restrict the specification
of 'return' to cases where it is "executing in the same execution
environment as the function's defining compound-command" would in fact
cover asynchronous lists. But it would leave this unspecified:

f() {
  ( echo "What will happen next is Unspecified"; return )
  echo "This probably should not be reached, but typically is"
}

New wording would be required in my Resolution (c), eg in 2.14 'return'
(inside_ _):

"... If the shell is not currently executing a function or dot script,
_or if the current execution context is asynchronous with respect to the
execution context of the function's defining command or the caller of
the dot script,_ the results are unspecified."

Unsurprisingly, the tested implementations are consistent in treating
'return' in an asynchronous context like 'exit' as well.

/df

--
London SW6
UK

Thinking a bit outside of/stepping back from the box

[Donn Terry]

I've mostly been lurking in this group for decades(!), but there are
certain recurring themes that might be worth considering for the next
revision.

Yes, I fully recognize that the goal is "standardizing existing practice"
is a requirement of the standard, but as was pointed out, I believe in the
Portland OR meeting (or the one after that) a lot of years ago, "we don't
standardize brokenness".

The recurring discussions on the exact syntax and semantics of the shell
(prompted by the current discussion of 'return') really reflect the history
of the various shells, which were all initially simple ad-hoc interpreters.
Some of the original interpreters have (thankfully) fallen by the wayside,
but several still survive with conflicting implementations because they are
trying to deal with a language that, from a formal point of view, is
"broken", with its irregular, ambiguous, and context-sensitive syntax and
semantics.

Rather than continuing these discussions into the a never-ending future, it
might make more sense for the POSIX committee to solicit a proposal (and
reference implementation, making it "existing practice") for a replacement
language that is "substantially similar" to the current shell, but is
explicitly NOT required to be fully compatible, but which IS well-formed as
a language by today's standards. (No, not csh... something closer to
Bourne/ksh than that.)

A useful measure of success of such a language is that there is a
well-formed, non-ambiguous grammar that accepts substantially most commands
that a user would type, and for scripts that an automatic translator that
works for at least 95% of current shell scripts is implemented. (I could
imagine that such a translator varied somewhat depending on which actual
shell implementation the script being translated runs on, since we know
that the current implementations vary on details, both those explicitly
allowed by the standard and those still "unknown" (to the standard)).

Over the next couple of revisions the old shell standard could be
deprecated in favor of a more regular and maintainable, but still familiar,
shell language.  (I'd like to see specific objections to such a proposed
language that boiled down to "but it's not like MY implementation"
(particularly when referring to an existing shell implementation) be
declared out of scope.)

Another "outside the box" item is "ancient history" that could be addressed
in the next revision. There are some features of the standard (particularly
those that were needed to get the standard past various "political parties"
during initial approval) that may no longer have a constituency of actual
users. It would seem to make sense to go through the standard (probably at
a "page" level) and suggest that certain features have fallen out of use,
and see if there is an objection to moving them to a deprecated state. I
suspect there are both commands and functions that would be hard-pressed to
find a constituency today.  Instances of "unspecified" and
"implementation-defined" (particularly those used to create a compromise
way-back-then) might well also be candidates.

I would advocate that if there was even a small constituency for a feature
that it be retained, but I expect that there would be things that found no
constituency among users of the current revision of the standard.

Donn Terry