Re: $? in a simple command with no command name

2021-09-02 Thread Robert Elz via austin-group-l at The Open Group
Date:Wed, 1 Sep 2021 19:04:12 +0100
From:Harald van Dijk 
Message-ID:  <837d3b5b-ac61-98eb-2741-d667a78e2...@gigawatt.nl>

  | $? is defined as "Expands to the decimal exit status of the most recent 
  | pipeline (see Pipelines)."

I suspect this is just a slightly more precisely specified version of
what the original Bourne shell (7th edition) man page said of $?

.B ?
The value returned by the last executed command
in decimal.

"The value returned" has turned into "exit status" which is
clearly what is intended (not something sent to stdout or whatever),
and "command" has been replaced by "pipeline" (of which a command,
as specified in the 7th edition man page, is just the degenerate case),
which is correct.

The 7th edition man page says nothing about the exit status of a
command substitution, and nothing about commands without command words,
or what exit status they might have.   It also does not say that
command substitutions are executed in a subshell environment.

So one could read its man page as implying that $? is set whenever
a command terminates, including commands for command substititions
(and since those are, and always were, actually executed in a subshell
environment, the final status of that environment instead of the individual
commands (or pipelines) that form the command substitution).

Further, we now know from what has been reported here in other messages
that that was how the implementation worked (I assume, that whenever it
did a wait(2) for a child, it set $? to the exit status (or signal number
modified so that can be detected) (as well as generating the exit status
from the built-in commands).

It seems likely that when this was all specified for the older POSIX/SUS
standards, the wording for $? was taken from the sh man page, corrected
as much as was known it should be, and simply then forgotten.   That is,
there was probably never any intent to explicitly say that $? can only be
set when a pipeline in the current execution environment terminates (and
is waited upon), that most probably simply is the way things ended up written.

Now, because of that, we can no longer reasonably say that $? is set
as each command substitution is performed, and then (normally) overridden
by the exit status of the command (if there is one), as the current
wording, with its (probably unintended) implications has been around for
too long now - even if it didn't match shells of the time.   But we can
say that in the presence of command substitutions, when the value of $?
used in the same command line is unspecified (used elsewhere there's no
issue).

kre



Re: $? in a simple command with no command name

2021-09-01 Thread Chet Ramey via austin-group-l at The Open Group

On 9/1/21 4:59 PM, (Joerg Schilling) wrote:

"Chet Ramey via austin-group-l at The Open Group" 
 wrote:


Given the following:

(exit 42)
a=$? b=`false` b=$?

echo $? $a $b

Bash prints 1 42 1.

The original (v7) bourne shell and the rest of the research line through v9
prints 1 1 (b is set to the empty string). That implies that it executes
the assignment statements in reverse order, in addition to carrying $?
through the sequence of assignments.


You are right, the original Bourne Shell for unknown reasons did evaluate a
series of shell variable assignments in reverse order. That was changed in
ksh88 and in bosh.
  

The SVR4.2 shell prints 1 42 1. I imagine the rest of the SVR4 line sh is
the same.


Something called SVR4.2 does not really exist. It was a minor change compared
to SvR4 announced by Novell short before they sold the Copyright to SCO. >
I know of no customers for SVR4.2... even SCO seems to only used it internally
in their project Monterey that was abandoned by IBM.


And yet it existed as a product. Univel probably had Unixware customers
they didn't tell you about.

You can find it for download if you look for it, I suspect.


There have been major changes in the Bourne Shell for SvR4, but the $? was not
touched. So you are mistaken.


Sure, I didn't have SVR4 to test against when I wrote that.


The important thing to know here is that the Bourne Shell has some
checkpoints that update the intermediate value of $?. Since that changed in
ksh88 and since POSIX requires a different behavior compared to the Bourne
Shell, I modified one checkpoint in bosh to let it match POSIX.


Interesting, since ksh88 (Solaris 10 11/16/88i) and ksh93 (93u+ 2012-08-01)
both print

1 42 1

Odd that POSIX would specify something different, isn't it?



(exit 42); a=$? b=`false` b=$?; echo $? $a $b

prints

1 42 42

in bosh and

1 1

in the SvR4 Bourne Shell.


It echoes 255 255 with the Solaris 10 /bin/sh (b is again null). It looks
like /bin/false exits with status 255, the Solaris 10 sh still performs
the assignments in reverse order, and the Solaris 10 version of the SVR4 sh
sets $? from the result of each command substitution.

In any case, kre's point stands: the original Bourne shell (and, for that
matter, the POSIX base implementation) set $? as each command substitution
finishes.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: $? in a simple command with no command name

2021-09-01 Thread Joerg Schilling via austin-group-l at The Open Group
Scott Lurndal  wrote:

> On Wed, Sep 01, 2021 at 10:59:46PM +0200, Joerg Schilling via austin-group-l 
> at The Open Group wrote:
> > 
> > Something called SVR4.2 does not really exist. It was a minor change 
> > compared 
> > to SvR4 announced by Novell short before they sold the Copyright to SCO.
> 
> SVR4.2 ES/MP became Unixware 2.01.

Sure? I had such a machine for a while because I enhanced the cdrom filesystem 
driver to support Joliet and ISO-9660:1999, but I did not realize that the 
shell was changed.

Well, this of course is a change that needs testing to discover.

Jörg

-- 
EMail:jo...@schily.net  Jörg Schilling D-13353 Berlin
Blog: http://schily.blogspot.com/
URL:  http://cdrecord.org/private/ 
http://sourceforge.net/projects/schilytools/files/



Re: $? in a simple command with no command name

2021-09-01 Thread Scott Lurndal via austin-group-l at The Open Group
On Wed, Sep 01, 2021 at 10:59:46PM +0200, Joerg Schilling via austin-group-l at 
The Open Group wrote:
> 
> Something called SVR4.2 does not really exist. It was a minor change compared 
> to SvR4 announced by Novell short before they sold the Copyright to SCO.

SVR4.2 ES/MP became Unixware 2.01.

They both were fairly widely used back in the day.

scott



Re: $? in a simple command with no command name

2021-09-01 Thread Joerg Schilling via austin-group-l at The Open Group
"Chet Ramey via austin-group-l at The Open Group" 
 wrote:

> Given the following:
> 
> (exit 42)
> a=$? b=`false` b=$?
> 
> echo $? $a $b
> 
> Bash prints 1 42 1.
> 
> The original (v7) bourne shell and the rest of the research line through v9
> prints 1 1 (b is set to the empty string). That implies that it executes
> the assignment statements in reverse order, in addition to carrying $?
> through the sequence of assignments.

You are right, the original Bourne Shell for unknown reasons did evaluate a 
series of shell variable assignments in reverse order. That was changed in 
ksh88 and in bosh.
 
> The SVR4.2 shell prints 1 42 1. I imagine the rest of the SVR4 line sh is
> the same.

Something called SVR4.2 does not really exist. It was a minor change compared 
to SvR4 announced by Novell short before they sold the Copyright to SCO.

I know of no customers for SVR4.2... even SCO seems to only used it internally 
in their project Monterey that was abandoned by IBM.

There have been major changes in the Bourne Shell for SvR4, but the $? was not 
touched. So you are mistaken.

The important thing to know here is that the Bourne Shell has some 
checkpoints that update the intermediate value of $?. Since that changed in 
ksh88 and since POSIX requires a different behavior compared to the Bourne 
Shell, I modified one checkpoint in bosh to let it match POSIX.

(exit 42); a=$? b=`false` b=$?; echo $? $a $b

prints

1 42 42

in bosh and

1 1

in the SvR4 Bourne Shell.



Jörg

-- 
EMail:jo...@schily.net  Jörg Schilling D-13353 Berlin
Blog: http://schily.blogspot.com/
URL:  http://cdrecord.org/private/ 
http://sourceforge.net/projects/schilytools/files/



Re: $? in a simple command with no command name

2021-09-01 Thread Chet Ramey via austin-group-l at The Open Group
On 9/1/21 2:23 PM, Robert Elz via austin-group-l at The Open Group wrote:
> Date:Wed, 1 Sep 2021 19:04:12 +0100
> From:Harald van Dijk 
> Message-ID:  <837d3b5b-ac61-98eb-2741-d667a78e2...@gigawatt.nl>
> 
>   | Is there any statement that overrides the general definition to 
>   | explicitly make this unspecified? If not, the general definition applies 
>   | and $? must expand to 0 both times it appears on line 2.
> 
> Perhaps as currently written that's correct, but if so, the standard
> probably needs to be updated, as it is fairly clear that shells which
> set $? as each command substitution finishes have always existed (in
> fact, that might have been what the original Bourne shell did, I haven't
> checked) and the standard should allow for that.

Given the following:

(exit 42)
a=$? b=`false` b=$?

echo $? $a $b

Bash prints 1 42 1.

The original (v7) bourne shell and the rest of the research line through v9
prints 1 1 (b is set to the empty string). That implies that it executes
the assignment statements in reverse order, in addition to carrying $?
through the sequence of assignments.

The SVR4.2 shell prints 1 42 1. I imagine the rest of the SVR4 line sh is
the same.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



Re: $? in a simple command with no command name

2021-09-01 Thread Robert Elz via austin-group-l at The Open Group
Date:Wed, 1 Sep 2021 19:04:12 +0100
From:Harald van Dijk 
Message-ID:  <837d3b5b-ac61-98eb-2741-d667a78e2...@gigawatt.nl>

  | Is there any statement that overrides the general definition to 
  | explicitly make this unspecified? If not, the general definition applies 
  | and $? must expand to 0 both times it appears on line 2.

Perhaps as currently written that's correct, but if so, the standard
probably needs to be updated, as it is fairly clear that shells which
set $? as each command substitution finishes have always existed (in
fact, that might have been what the original Bourne shell did, I haven't
checked) and the standard should allow for that.

kre



Re: $? in a simple command with no command name

2021-09-01 Thread Harald van Dijk via austin-group-l at The Open Group

On 01/09/2021 18:48, Robert Elz via austin-group-l at The Open Group wrote:

 Date:Wed, 1 Sep 2021 16:38:00 +0300
 From:"=?UTF-8?B?T8SfdXo=?= via austin-group-l at The Open Group" 

 Message-ID:  


   | true
   | a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2`
   | echo $? $a $b

   | Now, I wonder, what did I miss?

That $? (the exit status) is defined only when a command completes.
What it is in the middle of a command execution isn't specified.


$? is defined as "Expands to the decimal exit status of the most recent 
pipeline (see Pipelines)." During the a=$? and b=$? assignments, the 
most recent pipeline in that shell execution environment was 'true' at 
line 1. Is there any statement that overrides the general definition to 
explicitly make this unspecified? If not, the general definition applies 
and $? must expand to 0 both times it appears on line 2.


Cheers,
Harald van Dijk



Re: $? in a simple command with no command name

2021-09-01 Thread Oğuz via austin-group-l at The Open Group
1 Eylül 2021 Çarşamba tarihinde Robert Elz  yazdı:

> Date:Wed, 1 Sep 2021 16:38:00 +0300
> From:"=?UTF-8?B?T8SfdXo=?= via austin-group-l at The Open
> Group" 
> Message-ID:   qvsysq5nfzfxetr7yeq...@mail.gmail.com>
>
>   | true
>   | a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2`
>   | echo $? $a $b
>
>   | Now, I wonder, what did I miss?
>
> That $? (the exit status) is defined only when a command completes.
> What it is in the middle of a command execution isn't specified.
>
>   | Where does it say in the standard that
>   | the value of parameter `?' is affected by redirections,
>
> It isn't the redirection, but the command substitution that generates
> the file name for the redirection.
>
>   | and that its
>   | value depends on assignments that appear earlier in the same command?
>
> All this stuff is just unspecified, the order in which all of this is
> carried out, and precisely when $? gets updated, isn't specified, if you
> want to achieve meaningful results, you need to split the one
> multi-assignemnt
> with redirect command into multiple commands.
>
> You get 1 0 0 from a shell which only sets $? when a command completes
> (and which does the redirect before the arg expansions).
>
> Getting 2 for $? comes because it all depends which of the two command
> substitutions is executed first (the last one executed provides $?), which
> depends upon whether the redirect happens first or the arg expansions, and
> for a command with no actual command word, that's unspecified
> (implementations
> are allowed to do it either way - which the standard allows, as as you see,
> different implementations do it different ways, and users need to be aware
> of that).
>
> Getting a non-zero value for a or b just means that $? is being internally
> updated when a value for it becomes available, rather than only when a
> command completes, about which as best I remember, nothing is said in the
> standard at all.


>
Oh, okay then. Thanks


> kre
>
>

-- 
Oğuz


Re: $? in a simple command with no command name

2021-09-01 Thread Robert Elz via austin-group-l at The Open Group
Date:Wed, 1 Sep 2021 16:38:00 +0300
From:"=?UTF-8?B?T8SfdXo=?= via austin-group-l at The Open Group" 

Message-ID:  


  | true
  | a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2`
  | echo $? $a $b

  | Now, I wonder, what did I miss?

That $? (the exit status) is defined only when a command completes.
What it is in the middle of a command execution isn't specified.

  | Where does it say in the standard that
  | the value of parameter `?' is affected by redirections,

It isn't the redirection, but the command substitution that generates
the file name for the redirection.

  | and that its
  | value depends on assignments that appear earlier in the same command?

All this stuff is just unspecified, the order in which all of this is
carried out, and precisely when $? gets updated, isn't specified, if you
want to achieve meaningful results, you need to split the one multi-assignemnt
with redirect command into multiple commands.

You get 1 0 0 from a shell which only sets $? when a command completes
(and which does the redirect before the arg expansions).

Getting 2 for $? comes because it all depends which of the two command
substitutions is executed first (the last one executed provides $?), which
depends upon whether the redirect happens first or the arg expansions, and
for a command with no actual command word, that's unspecified (implementations
are allowed to do it either way - which the standard allows, as as you see,
different implementations do it different ways, and users need to be aware
of that).

Getting a non-zero value for a or b just means that $? is being internally
updated when a value for it becomes available, rather than only when a
command completes, about which as best I remember, nothing is said in the
standard at all.

kre




Re: $? in a simple command with no command name

2021-09-01 Thread Harald van Dijk via austin-group-l at The Open Group

On 01/09/2021 17:22, Geoff Clare via austin-group-l at The Open Group wrote:

Harald van Dijk wrote, on 01 Sep 2021:


The second problem is the redirection. Based on the above, command
substitutions in a redirection are supposed to affect the exit status just
as any other command substitution, but the standard says:

   3. Redirections shall be performed as described in Redirection.

   4. Each variable assignment shall be expanded for tilde expansion,
  parameter expansion, command substitution, arithmetic expansion,
  and quote removal prior to assigning the value.

This means the command substitution in your redirection is supposed to be
performed prior to that in the variable assignment.


After the list, the standard says:

 In the preceding list, the order of steps 3 and 4 may be reversed
 if no command name results from step 2 or if the command name
 matches the name of a special built-in utility


Ah, thank you. That means the outputs "1 0 0" and "2 0 0" are both 
equally correct. I assume the outputs "1 2 1" and "2 0 1" do indicate 
shell bugs.


Cheers,
Harald van Dijk



Re: $? in a simple command with no command name

2021-09-01 Thread Geoff Clare via austin-group-l at The Open Group
Harald van Dijk wrote, on 01 Sep 2021:
> 
> The second problem is the redirection. Based on the above, command
> substitutions in a redirection are supposed to affect the exit status just
> as any other command substitution, but the standard says:
> 
>   3. Redirections shall be performed as described in Redirection.
> 
>   4. Each variable assignment shall be expanded for tilde expansion,
>  parameter expansion, command substitution, arithmetic expansion,
>  and quote removal prior to assigning the value.
> 
> This means the command substitution in your redirection is supposed to be
> performed prior to that in the variable assignment.

After the list, the standard says:

In the preceding list, the order of steps 3 and 4 may be reversed
if no command name results from step 2 or if the command name
matches the name of a special built-in utility

-- 
Geoff Clare 
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England



Re: $? in a simple command with no command name

2021-09-01 Thread Harald van Dijk via austin-group-l at The Open Group

Hi,

On 01/09/2021 14:38, Oğuz via austin-group-l at The Open Group wrote:

Consider the following:

true
a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2`
echo $? $a $b

Having read the relevant sections of the standard a couple times, I
would expect the output to be `1 0 0';


I agree.


   but that's not the case with
most shells. Below are what each shell I have on my computer output
for that script.

 ?  a  b
bash2  0  1
bosh2  0  0
dash1  0  0
ksh 2  0  1
mksh1  0  0
yash2  0  0
zsh 1  2  1

Now, I wonder, what did I miss? Where does it say in the standard that
the value of parameter `?' is affected by redirections, and that its
value depends on assignments that appear earlier in the same command?


There are two problems here. The first is the value of $? during 
expansion. The standard says:


  If there is no command name, but the command contained a command
  substitution, the command shall complete with the exit status of the
  last command substitution performed. Otherwise, the command shall
  complete with a zero exit status.

Some shells implement this by modifying $? as soon as a command 
substitution finishes. As all command substitutions are processed in 
order, the end result is that $? is based on the last command 
substitution performed. However, there is nothing that allows $? to be 
modified earlier, so this looks like a bug (#1) in those shells.


The second problem is the redirection. Based on the above, command 
substitutions in a redirection are supposed to affect the exit status 
just as any other command substitution, but the standard says:


  3. Redirections shall be performed as described in Redirection.

  4. Each variable assignment shall be expanded for tilde expansion,
 parameter expansion, command substitution, arithmetic expansion,
 and quote removal prior to assigning the value.

This means the command substitution in your redirection is supposed to 
be performed prior to that in the variable assignment. Some shells do 
not do this in the specified order, which looks like a bug (#2) in those 
shells. This can be confirmed by running under 'set -x'.


Bug #1 leads to the output "1 2 1".
Bug #2 leads to the output "2 0 0".
Bug #1 & bug #2 combined leads to the output "2 0 1".

Cheers,
Harald van Dijk