Re: Pushing/restoring a file descriptor for a compound command

[Chet Ramey]

On 5/2/18 5:52 AM, Joerg Schilling wrote:

> 2)bash ignores the "new" rules.

Bash reserves most of these historical accidents for posix mode.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Pushing/restoring a file descriptor for a compound command

[Stephane CHAZELAS]

2018-05-02 11:52:09 +0200, Joerg Schilling:
> Stephane Chazelas  wrote:
> 
> > 2018-04-27 20:28:57 +0200, Martijn Dekker:
> > [...]
> > > : <&8 || echo "oops, closed"
> > [...]
> >
> > Remember ":" is a special builtin, so its failure causes the
> > shell to exit. Another one of those accidents of implementation
> > of the Bourne shell that ended up being specified by POSIX, but
> > which makes little sense (but that other shells ended up
> > implementing for conformance).
> 
> The background seems to be a bit more complex:
> 
> sh -c ' : <&8; echo bla' # Solaris 11 Bourne Shell
> bla

Here the problem is that the Bourne shell silently ignored the
failing of those "dup2()" redirections.

$ echo test | sh -c 'cat <&8; echo OK'
test
OK

That bug was fixed in ksh and all other shells.

But otherwise, a failing ":" still causes the shell to exit:

$ sh -c 'echo < /x; echo OK'
sh: /x: cannot open


[...]
> bash -c ' : <&8; echo bla'
> bash: 8: Falsche Dateinummer
> bla
> 
> So:
> 
> 1)before POSIX, ":" could not fail
> 
> 2)bash ignores the "new" rules.
[...]

Not really, it's POSIX again specifying an accident of
implementation in the Korn shell (in this case inherited from
the Bourne shell, so a "very old" rule).

Here, it's a silly requirement. Nobody would expect a  failure
of the no-op command to exit the shell.

That's why bash, zsh and yash only do it in conformance mode
(when called as "sh").

$ (exec -a sh bash -c ': <&8; echo X')
sh: 8: Bad file descriptor


-- 
Stephane

Re: Pushing/restoring a file descriptor for a compound command

[Joerg Schilling]

Stephane Chazelas  wrote:

> 2018-04-27 20:28:57 +0200, Martijn Dekker:
> [...]
> > : <&8 || echo "oops, closed"
> [...]
>
> Remember ":" is a special builtin, so its failure causes the
> shell to exit. Another one of those accidents of implementation
> of the Bourne shell that ended up being specified by POSIX, but
> which makes little sense (but that other shells ended up
> implementing for conformance).

The background seems to be a bit more complex:

sh -c ' : <&8; echo bla' # Solaris 11 Bourne Shell
bla

osh -c ' : <&8; echo bla' # Schily Solaris 11 emulation
bla

ksh -c ' : <&8; echo bla'  # ksh88
ksh: 8: bad file unit number

bosh -c ' : <&8; echo bla'
bosh: 8: Falsche Dateinummer

bash -c ' : <&8; echo bla'
bash: 8: Falsche Dateinummer
bla

So:

1)  before POSIX, ":" could not fail

2)  bash ignores the "new" rules.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: Pushing/restoring a file descriptor for a compound command

[Joerg Schilling]

Martijn Dekker  wrote:

> I need references and opinions about the following, please.
>
> Consider:
>
> {
>   exec 8   ...stuff using file descriptor 8...
> } 8<&-
>
> Should the effect of the 'exec' persist past the compound command (the 
> curly braces block)?
>
> My expectation is that the '8<&-' should push file descriptor 8 onto the 
> shell's internal stack in a closed state, so that it is restored at the 
> end of the block.

The way it is implemented does not matter.


> I think this should allow the effect of 'exec' to be local to that 
> compound command, and I think this construct should be nestable.

OK, and the fact that this is supported by the historic Bourne Shell, ksh88 and 
ksh93 is a strung hint that this should also be in POSIX since it is something 
that looks a "natural" feature.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: Pushing/restoring a file descriptor for a compound command

[Chet Ramey]

On 4/27/18 9:53 PM, Martijn Dekker wrote:

> That's what I've got. Is that a sane interpretation?
> 
> It would be nice if there were something more unequivocal in the standard,
> but it seems there isn't...
> 
>> You might have more luck with bash (perhaps.)
> 
> Chet, what do you think?

There's nothing in Posix that specifies the behavior one way or another, so
it's just an incompatibility between shells. I'll take a look.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Pushing/restoring a file descriptor for a compound command

[Stephane Chazelas]

2018-04-28 00:23:51 +0200, Martijn Dekker:
[...]
> That said, do you have any opinion on whether something like
>{ ... ; } 3>&-
> should push/restore a closed file descriptor if it's already closed, so that
> the effect of exec-ing that descriptor within the compound command is local
> to that compound command?
[...]

Actually, in my list of "issues" I plan to some day raise to the
Austin Group, I had:

sh -c '{ exec > a; } > b; echo x' > c

which sounds like it's the same.

Do we honour the compound command asking for stdout to be
redirected, or do we honour the "> b" redirection being
temporary and apply only to the command substitution?

I'm undecided here, though I'd tend to lean the same way as you
do.

The problem can also be expressed as:

redir_stdout() {
  exec > "$1"
}

redir_stdout file1 > file2

Or:

close() { eval "exec $1>&-"; }
close 1 > file

f() {
  close 1
  redir_stdout file
  echo  foo
  close 1
}

f > /dev/null

POSIX could arbitrate based on the number of implementations
that go one or the other way. Or leave it unspecified (I've not
checked what it currently says)..

-- 
Stephane

Re: Pushing/restoring a file descriptor for a compound command

[Robert Elz]

Date:Sat, 28 Apr 2018 03:53:12 +0200
From:Martijn Dekker 
Message-ID:  <81b93245-e42f-ad62-4005-8ad676733...@inlv.org>

  |  How does NetBSD sh handle this?

This isn't really the best place for code samples, but ...

"fd" is the file descriptor in question:

if ((flags & REDIR_PUSH) && !is_renamed(sv->renamed, fd)) {
INTOFF;
if (big_sh_fd < 10)
find_big_fd();
if ((i = fcntl(fd, F_DUPFD, big_sh_fd)) == -1) {
switch (errno) {
case EBADF:
i = CLOSED;
break;
case EMFILE:
case EINVAL:
find_big_fd();
i = fcntl(fd, F_DUPFD, big_sh_fd);
if (i >= 0)
break;
/* FALLTHRU */
default:
i = errno;
INTON;/* XXX not needed here ? */
error("%d: %s", fd, strerror(i));
/* NOTREACHED */
}
}
if (i >= 0)
(void)fcntl(i, F_SETFD, FD_CLOEXEC);
fd_rename(sv, fd, i);

CLOSED is < 0 

fd_rename() does (aside from bookkeeping)

rl->orig = from;
rl->into = to;

The "sv" arg is the data struct where all this is
saved (rl is alloc'd memory, linked to it).

and then later, when things are being restored

if (rl->into < 0)
close(rl->orig);
else
movefd(rl->into, rl->orig);

movefd() ends up translating into dup2() with
error, and close-on-exec handling.

The FreeBSD sh is similar, but simpler - they don't deal
with user fd's >= 10, which makes the data structs needed
simpler, and much easier to avoid user fd's stepping all
over the shell's internal fds - we allow user fds to be anything
the system allows, and the shell makes sure it moves its
own fds around if needed to avoid issues.

The EINVAL and EMFILE handling is dealing with the
consequences of the user/script playing with ulimit, or
at least as much as is possible.

  | If it's a bug, surely it would meet that requirement.

Not for me to say, but it it doesn't have to be different
(rather than just would be nicer if it was different) then
it can be hard to justify making changes.

kre

Re: Pushing/restoring a file descriptor for a compound command

[Martijn Dekker]

Op 28-04-18 om 01:55 schreef Robert Elz:

 Date:Sat, 28 Apr 2018 00:23:51 +0200
 From:Martijn Dekker 
 Message-ID:  <8800d6d5-67ea-fad4-19c3-dac4bbfd8...@inlv.org>

   | That said, do you have any opinion on whether something like
   | { ... ; } 3>&-
   | should push/restore a closed file descriptor if it's already closed,

I suspect that it is just a bug, the usual way to do this is

saved_fd = fcntl(3, F_DUPFD, BIG_NUM);
close(3);

and later

dup2(saved_fd, 3);
close(saved_fd);

Which works fine, provided fd 3 is open at the beginning, otherwise
all of those fail with EBADF, which tends to just be ignored (and the
dup2() failing leaves fd 3 with whatever it was set to in between.)


It works fine on NetBSD sh though, and on every other ash derivative I 
know of except dash -- even for a file descriptor 3 that is closed at 
the beginning. How does NetBSD sh handle this?



Fixing it means adding code - as I understand it, in dash, that is
something they avoid unless it is absolutely essential


If it's a bug, surely it would meet that requirement.


 - and if no-one
can show where POSIX requires it, probably just won't happen.


The closest thing I've found is the first sentence of POSIX 2.7 
Redirection: "Redirection is used to open and close files for the 
current shell execution environment (see Shell Execution Environment) or 
for any command."


The first part of that sentence refers to exec'ing it, so is irrelevant 
here. The relevant bit is "or for any command". This includes compound 
commands.


I claim that this implies that a redirection added to a compound command 
should always cause the specified descriptor to be saved and restored, 
no matter the initial state of it, or the state initialised by the 
redirection.


The alternative seems inherently broken: the effect of an 'exec 3&- redirection for the same FD creates a reasonable 
expectation that it should restore that FD's state when leaving the 
compound command.


Moreover, every current POSIX-compliant shell I know of works as I would 
expect, except bash and dash. The behaviour of dash appears to be unique 
for current ash derivatives, as FreeBSD sh, NetBSD sh, and Busybox ash 
also work as I would expect. I think that's further evidence that the 
behaviour of bash and dash should be considered a bug.


That's what I've got. Is that a sane interpretation?

It would be nice if there were something more unequivocal in the 
standard, but it seems there isn't...



You might have more luck with bash (perhaps.)


Chet, what do you think?

Thanks,

- M.

Re: Pushing/restoring a file descriptor for a compound command

[Robert Elz]

Date:Sat, 28 Apr 2018 00:23:51 +0200
From:Martijn Dekker 
Message-ID:  <8800d6d5-67ea-fad4-19c3-dac4bbfd8...@inlv.org>

  | That said, do you have any opinion on whether something like
  | { ... ; } 3>&-
  | should push/restore a closed file descriptor if it's already closed,

I suspect that it is just a bug, the usual way to do this is

saved_fd = fcntl(3, F_DUPFD, BIG_NUM);
close(3);

and later

dup2(saved_fd, 3);
close(saved_fd);

Which works fine, provided fd 3 is open at the beginning, otherwise
all of those fail with EBADF, which tends to just be ignored (and the
dup2() failing leaves fd 3 with whatever it was set to in between.)

Fixing it means adding code - as I understand it, in dash, that is
something they avoid unless it is absolutely essential - and if no-one
can show where POSIX requires it, probably just won't happen.

You might have more luck with bash (perhaps.)

kre

Re: Pushing/restoring a file descriptor for a compound command

[Martijn Dekker]

Op 27-04-18 om 23:38 schreef Stephane Chazelas:

2018-04-27 20:28:57 +0200, Martijn Dekker:
[...]

: <&8 || echo "oops, closed"

[...]

Remember ":" is a special builtin, so its failure causes the
shell to exit. Another one of those accidents of implementation
of the Bourne shell that ended up being specified by POSIX, but
which makes little sense (but that other shells ended up
implementing for conformance).


Good point. Ignore the '|| echo "oops, closed"', it's pointless because 
a failed direction prints an error message anyway. I should have said 
that no shell produces an error there.


That said, do you have any opinion on whether something like
   { ... ; } 3>&-
should push/restore a closed file descriptor if it's already closed, so 
that the effect of exec-ing that descriptor within the compound command 
is local to that compound command?


- M.

Re: Pushing/restoring a file descriptor for a compound command

[Stephane Chazelas]

2018-04-27 20:28:57 +0200, Martijn Dekker:
[...]
> : <&8 || echo "oops, closed"
[...]

Remember ":" is a special builtin, so its failure causes the
shell to exit. Another one of those accidents of implementation
of the Bourne shell that ended up being specified by POSIX, but
which makes little sense (but that other shells ended up
implementing for conformance).

-- 
Stephane

Re: Pushing/restoring a file descriptor for a compound command

[Martijn Dekker]

Op 27-04-18 om 18:49 schreef Martijn Dekker:
The author of dash, Herbert Xu, said in response to my bug report that 
dash is not obligated to push and restore that file descriptor if it is 
already closed when entering the compound command -- implying that a 
file descriptor should not be pushed if the local state would be 
different from the parent state.


Actually, that implication that I understood is not even correct. No 
shell, not even bash and dash, outputs "oops, closed" for the following:


exec 8To rephrase my request: I would welcome opinions on whether the 
dash/bash behaviour shown in the original message should reasonably 
considered a bug in standards terms.


Thanks,

- M.

Pushing/restoring a file descriptor for a compound command

[Martijn Dekker]

I need references and opinions about the following, please.

Consider:

{
exec 8Should the effect of the 'exec' persist past the compound command (the 
curly braces block)?


My expectation is that the '8<&-' should push file descriptor 8 onto the 
shell's internal stack in a closed state, so that it is restored at the 
end of the block.


I think this should allow the effect of 'exec' to be local to that 
compound command, and I think this construct should be nestable.


According to my testing, nearly all shells do this. However, two very 
widely used shells, dash and bash, leave the file descriptor open beyond 
the block.


The author of dash, Herbert Xu, said in response to my bug report that 
dash is not obligated to push and restore that file descriptor if it is 
already closed when entering the compound command -- implying that a 
file descriptor should not be pushed if the local state would be 
different from the parent state.


He asked me for a POSIX reference proving otherwise. I can't find any.

Without this behaviour, an awkward workaround is required. To guarantee 
that the FD will be restored at the end of the block, you'd need to 
attempt to push it twice in two different states in two nested compound 
command blocks, for instance:


{
{
exec 8/dev/null

Does anyone have pointers to the POSIX text, or other strong evidence, 
that this workaround should not be necessary?


Thanks,

- M.