Date: Thu, 21 Oct 2021 10:31:17 +0100
From: "Geoff Clare via austin-group-l at The Open Group"
<austin-group-l@opengroup.org>
Message-ID: <20211021093117.GA14037@localhost>
| For O_APPEND it would certainly not be unusual. It is required to be
| set when >> redirection is used in the shell.
Sure, but when that kind of thing happens it is unusual for that fd to
be (in any way), shared with the others, unless via a construct like
>>file 2>&1 in which case it is clear that both fds are intended to be
O_APPEND.
The more interesting question is when the fd's are "accidentally" the
same thing, rather than when they are deliberately made so. I'm not sure
what (if anything) can be said about this, I have no specific goal in mind,
but I wonder if perhaps, somewhere, something should be said about the
effects of file descriptors that an application inherits open, and not
necessarily knowing if they are dups or completely separate, in such a way
that the application cannot know whether operations on one might be affecting
one or more of the others.
Further, I know of no good way to discover if this has occurred, fstat()
doesn't reveal anything useful, about the only way I can think of (which
doesn't mean there isn't a better way I don't know) is to try some operation
on one fd (setting of clearing one of the O_ flags, other than CLOEXEC
and CLOFORK) and seeing if the other fd is affected, or perhaps testing via
lseek() (but that's only meaningful on fds where lseek works, so is not
generally workable).
| For other flags, applications might get unexpected behaviour from
| the calls they make. For example, if fd 0 is a pipe with O_NONBLOCK
| set then calls to read() could give an EAGAIN error.
Sure, that's what is going to happen - the relevant question here is whether
the standard should say anything about that, and indicate whose responsibility
it is to avoid it happening (other than by deliberate agreed design).
Otherwise we get parent application claiming that there's nothing that says
it cannot pass down fd 0 with O_NONBLOCK or any other random bits like that
set, and the child application should be able to cope - and (naturally) the
child saying that responsible parents don't act like that, and they're
entitled to assume that fds open all come in the "normal" way, that most
applications do it.
One of the goals of any standard should be to avoid (or at least, be able to
clearly answer) any disputes of that kind, so I wonder if we should add to
the section that currently says about the necessity for applications to be
started with fd 0 open for reading, and fds 1 & 2 open for writing, what other
attributes they can assume. That would clearly include the admonition that
applications cannot assume that lseek() will work on any of those, that any
combination of the 3 of them may refer to a common file description (is that
the correct terminology to mean "struct file *" in the kernel? If not, then
you know what I mean anyway) that fds 1 and 2 might have O_APPEND set (and
because 0 might be the same, potentially, so might it), but that other (later
than open() time affecting O_FLAGS should all be clear on those fds (or
whatever we agree is reasonable for them) (I mean things like O_CREAT and
O_TRUNC which only matter at open time aren't included in this).
| In practice
| this would only happen if they inherit an fd with unusual flag settings
| from another application that had them set, and I think most people
| would consider this to be a bug in the parent application.
Perhaps - but convincing the autors of that parent application of that
is not always easy. Something in the standard about what applications
are entitled to assume at startup would be useful. That is, not only
the state of the fds, but what signals might or might not be masked, or
ignored (running a shell with O_SIGCHLD ignored or blocked is "fun" if
it doesn't ignore the requirements and simply reset it to something sane
for example), what the umask might be (here there should probably be an
"anything is possible, alter it if needed" admonition) what the various
resource limits can be expected to be "ulimit -n 1; utility" tends not
to work well for any utility that wants to open files, etc) and everything
else that gets inherited, even with a bigger (any) RLIMIT_NOFILE the following
is probably not acceptable, just as one example:
errno = 0;
while (errno != EMFILE)
(void)dup(0);
execl(.....);
No sane application would do that, but where do we actually say that
they should not, and that applications can assume there are some available
fds open for them to use?
Some applications do something like
for (fd=3; fd < BIG; fd++)
(void)close(fd);
just to be "safe", but that can be dangerous -- though it isn't in POSIX,
many systems allow a file name of "/dev/fd/13" or something similar to
be passed as one of the files for the utility to access - which does not
work if the application has already closed fd 13.
With regard to bugnote 5508 (
https://austingroupbugs.net/view.php?id=1526#c5508 )
I won't add a new note (now anyway) as nothing I have to say right now
deserves that, but:
austin-group-l@opengroup.org (really Geoff) said:
| So I think the statement for 'w': "If the open file description referenced
| by fildes has O_APPEND set, it shall remain set" should be added to 'r' as
| well.
Agreed, that's about what I expected.
then initially quoting me in bugnote 5507:
| > I also wonder about the requirement that 'b' be ignored for fdopen().
| It matches the requirement that 'b' is ignored for fopen() and freopen().
I think that's a bug in fopen() and will open a bug about that one (sometime
soonish), and while the effect is probably not going to really alter, for any
standard filesystem anyway, the way it is stated isn't really what we should
be saying. It is even worse when applied to fdopen() than the others (which
can perhaps detect when 'b' can simply be ignored).
I think we need wording about 'b' in fdopen() that is similar to that which
applies to 'a' ("a" I guess), except that we need to add O_BINARY to <fcntl.h>
first (that will be part of the fopen bug when I file it).
While no standard filesystem does anything with O_BINARY, non-standard add
on filesystems might, and there needs to be a way for an application using
stdio to access the O_BINARY flag. That's what 'b' does. Saying that 'b'
is ignored makes that impossible to implement. 'b' needs to set O_BINARY if
O_BINARY is supported by the implementation (which would not be required).
Then in fdopen we need words about not clearing O_BINARY if it is set, and
setting it if 'b' is given, just like we do for O_APPEND and "a" (except
'b' properly applies to both reads and writes).
| > Lastly, is there any prohibition (or stated to be undefined or unspecified
| > anywhere) on the same underlying fd being the fd for more than one stdio
| > stream?
| There is no prohibition, but there are rules (in XSH 2.5.1) that
| applications must follow when changing from one "handle" to another
Yes, that stuff is what I meant by "Obviously it would be up to the
application to sequence output so it makes sense", XSH 2.5.1 is really
only about the effect of changing the file position (avoiding overwriting
etc), it says so:
A file descriptor that is never used in an operation that could
affect the file offset (for example, read( ), write( ), or lseek( ))
is not considered a handle for this discussion,
Whether turning on, or off, O_APPEND is "an operation that could affect the
file offset" isn't clear, but I can note that the rules of legal interpretation
mean that the "for example..." would mean "no, changing O_APPEND is not sucb
an operation" as all of the examples are examples of operations which actually
immediately alter the file offset, and setting or clearning O_APPEND is
different as it does not immediately affect the file offset.
(The rule is, if I have the Latin correct, which I probably don't,
"Expressio unius est exclusio alterius" or something like that.]
But that's just O_APPEND, there are several other O_XXXs that might apply to
a file description, which have nothing at all to do with the file offset, and
which are thus clearly excluded for anything related to 2.5.1.
| The second fdopen() must behave exactly as specified - the fact that
| there is already another stream that has the same fd underlying it has
| no bearing on how fdopen() behaves.
While that's reasonable, and I expect reflects what is standard practice,
it could perhaps be further clarified in the APPLICATION USAGE, that an
fdopen() on one file descriptor might affect the state of another FILE *
(or other fd) that has already been opened. I suspect that might be a
surprise to some people.
I suspect this might also be why in some implementations fdopen(fd, "a") on
a fd that doesn't already have O_APPEND set doesn't cause it to be set -
because doing so might affect some other fd without explicit action by the
application.
Note that once upon a time, the "a" stdio open modes did not set O_APPEND
(long after O_APPEND existed). Rather when in "a" mode, stdio simply did
an lseek() to the current EOF before writing.
This stuff all gets quite messy, very quickly, and actually specifying how
everything works, or should work as I am not convinced there actually is
a standard here, would end up being a lot of "is unspecified", so much so
that application writers would start wondering if there's anything actually
defined to work...
Note that in this regard, almost all of what is in XSH 2.5.1 is fantasy land.
It looks to have been written by someone trying to work out what the rules
would need to be to make everything safe wrt file position changes and multiple
handles into the same file. However it totally fails to reflect what anything
actually does (ie: it is not describing anything pertaining to standard
behavious).
That is, it is entirely normal for an application to be written like
if (fopen("myfile", "w") == NULL)
perror("myfile"), exit(1);
(or something roughly equivalent). But according to what XSH 2.5.1
specifies, that is not just unspecified behaviour (which applications
could live with) but is undefined behaviour (which is totally unacceptable).
That's because it happens that the application was started:
app >file 2>&1
Which makes stdout buffered, stderr line buffered,
stderr and sdtout handles share a file description in the XSH 2.5.1 sense,
and nothing bothered to fflush(stdout) before the perror() which writes
to stderr.
Yet applications do this, and it works well enough, all the time. Making
this be undefined is ludicrous. It is somewhat unspecified, as the
intermixing of what appears from stdout and stderr (in "file") isn't
necessarily going to work very well, but apart from that, everything
operates as it should, the exit actually exits with status 1 (were things
undefined we couldn't promise that to happen, the perror() might have the
effect of longjmp() to almost anywhere).
WHat's in XSH 2.5.1 should be mostly removed from the normative text, and
moved to an application usage note somewhere (as a method that can be used
to try and make I/O all work ideally) and 2.5.1 just make it be unspecified
what happens when I/O is intermixed via different handles accessing the same
file.
kre
