Re: utilities and write errors
Date:Thu, 29 Jul 2021 10:38:25 +0100 From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20210729093825.GA7847@localhost> This is definitely my last message in this thread (and given your new bug #1492 I believe that you now accept that the current standard does not say what has been claimed that it says, and needs to be fixed to make it clear ... we can have more discussion in the context of that bug, later). | This seems to be the crux of our disagreement. You think the presence | of the word "successful" in "0 Successful completion" isn't sufficient | to require that the write must have succeeded in order for the exit | status to be 0. Yes. | I think that (for pwd) it is sufficient. The required "(for pwd)" there is telling. The same words need to mean the same thing every place they're used (in a document like a standard) or we end up with chaos, where everyone invents a new meaning for the phrase to suit whatever they want it to mean in any particular instance. | There could be some ambiguity if that wording was used for a utility | that performs multiple different actions It isn't ambiguity, so much as the words cannot mean what you think they mean. | but pwd is required to perform one and only one action - to write a | pathname to file descriptor 1. (Everything else the standard requires | of it is about how it decides what pathname to write.) The "everything else" is what can fail, and lead to an unsuccessful completion. | Therefore, in the case of pwd, it is crystal clear that "Successful | completion" means that it successfully completed the one action it is | required to perform, and thus the exit status can only be 0 if the | write to file descriptor 1 succeeded. No, you seem to have forgotten (or ignored) the CONSEQUENCE OF ERRORS section ... 104925 CONSEQUENCES OF ERRORS 104926 If an error is detected, output shall not be written to standard output, a diagnostic message shall 104927 be written to standard error, and the exit status is not zero. Now clearly, since "output shall not be written" when there's an error, the errors that are of consequence there are ones that occur before the write is to happen. So, something else must be happening; pwd is not simply "echo $PWD" (or printf, with appropriate quoting etc). The errors that matter to pwd are in determining that pathname to output, and it is entirely reasonable to conclude that an exit status of 1 is intended to mean that the correct pathname for the current working directory could not be determined, rather than that it could be, but writing it failed. The section quoted just above makes that kind of clear. That's a particularly appealing interpretation when one also understands that this is exactly how implementations of pwd behaved, and so is exactly what the standard should have said users could expect from pwd. | If what you claim was correct, then the standard would allow any utility | whose exit status 0 is described as "Successful completion" to exit with | status 0 as long as it was not terminated by a signal, regardless of | whether any actions is it is required to perform succeeded or not. No, it wouldn't, you might have a point if it was just "successful", but the "completion" requires that all the steps have been carried out - if any of the requirements along the way fail, then the command cannot be completed - except for the very last one, which once performed (requested) ends the process. | Yes, some would have reported the data loss. However, I remain convinced | that neither they nor the recipients of the report would suspect that a | a utility neglecting to report a write error could be the cause, I very much doubt that. | and so would have no reason to investigate whether that might have been | the cause. They are much more likely to blame it on a hardware glitch. Some people blame the hardware, others blame the compiler ("the write call must have been optimised out" ... despite it being there every other time), some people have no clue what they're talking about - anyone competent would investigate all possibilities, and if they even suspected a "hardware glitch" (as absurd as that would be for an issue like this - hardware doesn't act like that) the system error logs would be examined, and the "out of space" on the filesystem, at just about the right time, would be noticed. And that would lead to further investigation and testing. | Okay, but all that does is make it more likely that the data loss | would be noticed. It would remain unexplained. No, if it happened, or happened anywhere near as often as you're claiming it "must" happen, then it would be explained. There are plenty of very good diagnosticians around, someone would have worked it out, and you'd have your smoking gun. But you don't, because in practice this just does not happen in any real application (in a situation
[Issue 8 drafts 0001471]: Add an orthogonal interface for immediate macro expansion definitions to make
A NOTE has been added to this issue. == https://austingroupbugs.net/view.php?id=1471 == Reported By:joerg Assigned To: == Project:Issue 8 drafts Issue ID: 1471 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Jörg Schilling Organization: User Reference: Section:make Page Number:2888- Line Number:97001- Final Accepted Text: == Date Submitted: 2021-05-16 12:15 UTC Last Modified: 2021-07-29 18:07 UTC == Summary:Add an orthogonal interface for immediate macro expansion definitions to make == -- (0005423) psmith (developer) - 2021-07-29 18:07 https://austingroupbugs.net/view.php?id=1471#c5423 -- Until someone can explain to me the advantage of adding Yet Another Assignment Operator to make, I'm not excited about spending effort adding it to GNU make. If there's behavior which is already implemented across all variants of make then that's obviously appropriate for standardization. If there's behavior which is used widely in makefiles, proving it has real utility for users, AND it provides behaviors which are not currently available in the standard, then that could be appropriate for standardization as well. Just because a facility _exists_ in a subset of variants of make, even if it has existed for a long time, is not by itself sufficient justification IMO. I haven't seen evidence that the difference between the behavior of POSIX ::= and BSD make := is something that users care about and that makefiles want to take advantage of, such that it warrants a new operator. I've also pointed out above that the behavior of BSD make := can be obtained in terms of the existing POSIX ::= operator (although you do need to add an extra variable to do it). Issue History Date ModifiedUsername FieldChange == 2021-05-16 12:15 joerg New Issue 2021-05-16 12:15 joerg Name => Jörg Schilling 2021-05-16 12:15 joerg Section => make 2021-05-16 12:15 joerg Page Number => 2888- 2021-05-16 12:15 joerg Line Number => 97001- 2021-05-16 14:35 shware_systems Note Added: 0005356 2021-05-16 17:18 psmith Note Added: 0005357 2021-05-16 19:02 shware_systems Note Added: 0005358 2021-05-16 19:12 kreNote Added: 0005359 2021-05-16 19:49 shware_systems Note Added: 0005360 2021-05-16 21:26 joerg Note Added: 0005361 2021-05-16 21:27 joerg Note Edited: 0005361 2021-05-22 19:02 psmith Note Added: 0005363 2021-07-08 16:43 geoffclare Note Added: 0005393 2021-07-08 16:57 hvdNote Added: 0005394 2021-07-08 19:30 steffenNote Added: 0005395 2021-07-08 19:32 steffenNote Added: 0005396 2021-07-08 20:01 kreNote Added: 0005397 2021-07-08 20:21 steffenNote Added: 0005398 2021-07-08 20:34 steffenNote Added: 0005399 2021-07-08 21:15 kreNote Added: 0005400 2021-07-08 22:33 steffenNote Added: 0005401 2021-07-08 22:33 steffenNote Added: 0005402 2021-07-10 12:30 joerg Note Added: 0005403 2021-07-10 18:03 psmith Note Added: 0005404 2021-07-10 21:26 mirabilos Note Added: 0005405 2021-07-15 14:51 joerg Note Added: 0005406 2021-07-15 14:52 joerg Note Edited: 0005406 2021-07-17 10:35 joerg
[1003.1(2016/18)/Issue7+TC2 0001493]: move XCU 2.7 definition of "file descriptor" into XBD 3
A NOTE has been added to this issue. == https://austingroupbugs.net/view.php?id=1493 == Reported By:geoffclare Assigned To: == Project:1003.1(2016/18)/Issue7+TC2 Issue ID: 1493 Category: Shell and Utilities Type: Enhancement Request Severity: Comment Priority: normal Status: New Name: Geoff Clare Organization: The Open Group User Reference: Section:2.7 Page Number:2360 Line Number:75306 Interp Status: --- Final Accepted Text: == Date Submitted: 2021-07-29 14:33 UTC Last Modified: 2021-07-29 16:49 UTC == Summary:move XCU 2.7 definition of "file descriptor" into XBD 3 == -- (0005422) kre (reporter) - 2021-07-29 16:49 https://austingroupbugs.net/view.php?id=1493#c5422 -- I have no problem with this change in general, though there are a couple of wording changes (just cleanups) I'll suggest in some later note if no-one else gets there first, but this part: The file descriptor underlying stdin is initially 0; this cannot change through the use of interfaces defined in this standard, (and the similar limitations expresseed for stdout and stderr) looks to be simply wrong to me. I'm also not sure there's any need to actually make this point, even if it were true. But consider close(fileno(stdin)); fd = open("/dev/null", 0); freopen("/my/file", "r", stdin); (use fdopen() instead of open() if you prefer, but not fclose() as any use of any stream after it has been fclose'd is undefined, as its data struct may have been discarded). I'm not sure what file descriptor you expect freopen() to assign in that case (assuming the open of /my/file succeeds, etc, of course), but it isn't usually going to be 0, and this is using only interfaces defined by the standard, I believe. Note that freopen() specifically says: If pathname is not a null pointer, freopen() shall close any file descriptor associated with stream. Failure to close the file descriptor successfully shall be ignored. That is, freopen() is defined to work properly if the file descriptor has already been closed (some systems may have other reasons for the close failing, allowing for a different fd to be returned from the open, but that would be using something beyond the standard interfaces I think). Issue History Date ModifiedUsername FieldChange == 2021-07-29 14:33 geoffclare New Issue 2021-07-29 14:33 geoffclare Name => Geoff Clare 2021-07-29 14:33 geoffclare Organization => The Open Group 2021-07-29 14:33 geoffclare Section => 2.7 2021-07-29 14:33 geoffclare Page Number => 2360 2021-07-29 14:33 geoffclare Line Number => 75306 2021-07-29 14:33 geoffclare Interp Status => --- 2021-07-29 16:49 kreNote Added: 0005422 ==
[Issue 8 drafts 0001489]: malloc RATIONALE awkward wording
The following issue has been RESOLVED. == https://austingroupbugs.net/view.php?id=1489 == Reported By:rhansen Assigned To: == Project:Issue 8 drafts Issue ID: 1489 Category: System Interfaces Type: Error Severity: Editorial Priority: normal Status: Resolved Name: Richard Hansen Organization: User Reference: Section:malloc RATIONALE Page Number:1272 Line Number:42599 Final Accepted Text: Resolution: Accepted Fixed in Version: == Date Submitted: 2021-07-12 16:09 UTC Last Modified: 2021-07-29 15:36 UTC == Summary:malloc RATIONALE awkward wording == Relationships ID Summary -- related to 0001387 Should EAGAIN be acceptable for malloc ... == Issue History Date ModifiedUsername FieldChange == 2021-07-12 16:09 rhansenNew Issue 2021-07-12 16:09 rhansenName => Richard Hansen 2021-07-12 16:09 rhansenSection => malloc RATIONALE 2021-07-12 16:09 rhansenPage Number => 1272 2021-07-12 16:09 rhansenLine Number => 42599 2021-07-12 16:10 rhansenRelationship added related to 0001387 2021-07-29 15:36 geoffclare Status New => Resolved 2021-07-29 15:36 geoffclare Resolution Open => Accepted ==
[1003.1(2016/18)/Issue7+TC2 0001493]: move XCU 2.7 definition of "file descriptor" into XBD 3
The following issue has been SUBMITTED. == https://austingroupbugs.net/view.php?id=1493 == Reported By:geoffclare Assigned To: == Project:1003.1(2016/18)/Issue7+TC2 Issue ID: 1493 Category: Shell and Utilities Type: Enhancement Request Severity: Comment Priority: normal Status: New Name: Geoff Clare Organization: The Open Group User Reference: Section:2.7 Page Number:2360 Line Number:75306 Interp Status: --- Final Accepted Text: == Date Submitted: 2021-07-29 14:33 UTC Last Modified: 2021-07-29 14:33 UTC == Summary:move XCU 2.7 definition of "file descriptor" into XBD 3 Description: XCU 2.7 Redirection has a definition of "file descriptor" that should be incorporated into the definition in XBD chapter 3 instead of being tucked away there. Also, the definitions of standard error/input/output currently say they are streams, but they are also used in XCU (and perhaps elsewhere) to refer to file descriptors. Desired Action: On page 60 line 1783 section 3.166 File Descriptor, after:A per-process unique, non-negative integer used to identify an open file for the purpose of file access.append:The values 0, 1, and 2 have special meaning and conventional uses, and are referred to as standard input, standard output, and standard error, respectively. Programs usually take their input from standard input, and write output on standard output. Diagnostic messages are usually written on standard error. On page 92 line 2575 section 3.366 Standard Error, change:An output stream usually intended to be used for diagnostic messages.to:In the context of file descriptors (see [xref to 3.166 File Descriptor]), file descriptor number 2. In the context of standard I/O streams (see [xref to XSH 2.5]), an output stream usually intended to be used for diagnostic messages, and accessed using the global variable stderr. Note: The file descriptor underlying stderr is initially 2, but it can be changed by freopen() to 0 or 1 (and implementations may have extensions that allow it to be changed to other numbers). Therefore, writing to the standard error stream does not always produce output on the standard error file descriptor. On page 92 line 2577 section 3.367 Standard Input, change:An input stream usually intended to be used for primary data input.to:In the context of file descriptors (see [xref to 3.166 File Descriptor]), file descriptor number 0. In the context of standard I/O streams (see [xref to XSH 2.5]), an input stream usually intended to be used for primary data input, and accessed using the global variable stdin. Note: The file descriptor underlying stdin is initially 0; this cannot change through the use of interfaces defined in this standard, but implementations may have extensions that allow it to be changed. Therefore, in conforming applications using extensions, reading from the standard input stream does not always obtain input from the standard input file descriptor. On page 92 line 2579 section 3.368 Standard Output, change:An output stream usually intended to be used for primary data output.to:In the context of file descriptors (see [xref to 3.166 File Descriptor]), file descriptor number 1. In the context of standard I/O streams (see [xref to XSH 2.5]), an output stream usually intended to be used for primary data output, and accessed using the global variable stdout. Note: The file descriptor underlying stdout is initially 1, but it can be changed by freopen() to 0 (and implementations may have extensions that allow it to be changed to other numbers). Therefore, writing to the standard output stream does not always produce output on the standard output file descriptor. On page 2360 line 75294 section 2.7 Redirection, change:The number n is an optional decimal number designating the file descriptor number; the application shall ensure it is delimited from any preceding text and immediately precede the redirection operator redir-op.to:The number n is an optional one or more digit decimal number designating the file descriptor number; the application shall ensure it is delimited from any preceding text and immediately precedes the redirection operator redir-op (with no intervening characters allowed). On page 2360 line 75304 section 2.7 Redirection, change:Open files are represented by decimal numbers starting with zero. The
[1003.1(2016/18)/Issue7+TC2 0001492]: clarify what "successful completion" means in EXIT STATUS
The following issue has been SUBMITTED. == https://austingroupbugs.net/view.php?id=1492 == Reported By:geoffclare Assigned To: == Project:1003.1(2016/18)/Issue7+TC2 Issue ID: 1492 Category: Shell and Utilities Type: Clarification Requested Severity: Objection Priority: normal Status: New Name: Geoff Clare Organization: The Open Group User Reference: Section:1.4 Utility Description Defaults Page Number:2341 Line Number:74538 Interp Status: --- Final Accepted Text: == Date Submitted: 2021-07-29 13:53 UTC Last Modified: 2021-07-29 13:53 UTC == Summary:clarify what "successful completion" means in EXIT STATUS Description: There are some utilities for which the EXIT STATUS section describes exit status 0 as "Successful completion", but the utility is required to perform more than one action, meaning it is unclear whether all of those actions must succeed in order for the exit status to be 0. The suggested change is to require that it means all actions must succeed, but also to stop using "Successful completion" for some utilities where this would not be appropriate. Desired Action: On page 2341 line 74538 section 1.4 Utility Description Defaults, add to the end of EXIT STATUS:Default Behavior: When the description of exit status 0 is ``Successful completion'', it means that exit status 0 shall indicate that all of the actions the utility is required to perform were completed successfully. On page 2870 line 94579 section jobs, after:When jobs reports the termination status of a job, the shell shall remove its process ID from the list of those ``known in the current shell execution environment''; see [xref to 2.9.3.1].add:If a write error occurs when jobs writes to standard output, some process IDs might have been removed from the list but not successfully reported. On page 2872 line 94663 section jobs, change:0 Successful completionto:0 The output specified in STDOUT was successfully written to standard output. On page 2873 line 94704 section jobs, add to RATIONALE:If jobs uses buffered writes to standard output, a write error could be detected when attempting to flush a buffer containing multiple reports of terminated jobs, resulting in some unreported jobs having their process IDs removed from the list of those known in the current shell execution environment (because they were removed when the report was added to the buffer). On page 2982 line 99046 section make (EXIT STATUS with -q), change:0 Successful completion 1 The target was not up-to-date.to:0 All specified targets were already up-to-date. 1 One or more targets were not up-to-date. On page 2983 line 99050 section make (EXIT STATUS without -q), change:0 Successful completion >0 An error occurred.to:0 All specified targets were already up-to-date, or all commands executed to bring targets up-to-date either exited with status 0 or had a non-zero exit status that was specified (via the -i option, the special target .IGNORE, or a '-' command prefix) to be ignored. >0 An error occurred, or at least one command executed to bring a target up-to-date exited with a non-zero exit status that was not specified to be ignored. == Issue History Date ModifiedUsername FieldChange == 2021-07-29 13:53 geoffclare New Issue 2021-07-29 13:53 geoffclare Name => Geoff Clare 2021-07-29 13:53 geoffclare Organization => The Open Group 2021-07-29 13:53 geoffclare Section => 1.4 Utility Description Defaults 2021-07-29 13:53 geoffclare Page Number => 2341 2021-07-29 13:53 geoffclare Line Number => 74538 2021-07-29 13:53 geoffclare Interp Status => --- ==
Re: utilities and write errors
[Catching up after a few days' vacation] Robert Elz wrote, on 25 Jul 2021: > > Date:Thu, 15 Jul 2021 10:19:17 +0100 > From:"Geoff Clare via austin-group-l at The Open Group" > > Message-ID: <20210715091917.GA13523@localhost> > > For pwd "0 Successful completion" has never been in doubt, the question > has always been "what is required of pwd for it to be considered successful?" > That's where we started. > > You contend that it must successfully write to standard output, because, > well, just because nothing else seems sane, if I understand what you've > been saying all this time. > > I say that pwd must write to standard output, but that nothing says that > that write must actually succeed - because there simply is no text that > requires that (if there were, someone would have quoted it by now). This seems to be the crux of our disagreement. You think the presence of the word "successful" in "0 Successful completion" isn't sufficient to require that the write must have succeeded in order for the exit status to be 0. I think that (for pwd) it is sufficient. There could be some ambiguity if that wording was used for a utility that performs multiple different actions (which is why it isn't used for cd; it has three actions it is required to perform and only two of the three are required to succeed for the exit status to be 0), but pwd is required to perform one and only one action - to write a pathname to file descriptor 1. (Everything else the standard requires of it is about how it decides what pathname to write.) Therefore, in the case of pwd, it is crystal clear that "Successful completion" means that it successfully completed the one action it is required to perform, and thus the exit status can only be 0 if the write to file descriptor 1 succeeded. > | The word "successful" is in the description of exit status 0. > > It is, but they are different things being successful, one is the write > (which isn't so required) the other is completion of the command, which > can be successful (as insane as it looks) when the write has failed. If what you claim was correct, then the standard would allow any utility whose exit status 0 is described as "Successful completion" to exit with status 0 as long as it was not terminated by a signal, regardless of whether any actions is it is required to perform succeeded or not. This would effectively mean a strictly conforming application cannot rely on the exit status to tell it anything other than whether the utility was terminated by a signal or not. That certainly is insane. > | I didn't claim these events aren't rare - I said they had undoubtedly > | happened many times. That's across dozens of utilities used by millions > | of users over almost three decades. > > And if they had, someone would have noticed, at least a few times, and > reported it. Yes, some would have reported the data loss. However, I remain convinced that neither they nor the recipients of the report would suspect that a a utility neglecting to report a write error could be the cause, and so would have no reason to investigate whether that might have been the cause. They are much more likely to blame it on a hardware glitch. > | You kick off a "find ... -exec grep -l ... {} +" command and go and > | make a coffee. > [...] > > | During the short time the disk was full, one of your grep commands > | failed to write some output but did not report it as an error. > > That might happen. > > | When you return from your coffee break your find command has finished > | and there is no indication that anything went wrong. > > But not that. Sure, one of the grep commands might have "failed to write > some output", but the commands in something like that, and the blocks in > the filesystem aren't synchronised. What you're almost certainly going to > get is partial output from that grep that failed, up to the end of the block > that was part full before the filesystem was filled up, but not the rest of > it - when the next grep (after the filesystem has had space returned) runs > and produces output, its output will be shoved in what appears to be the > middle of the previous output. Okay, but all that does is make it more likely that the data loss would be noticed. It would remain unexplained. > I am not arguing that it is a good thing that commands don't detect write > errors on standard output (except perhaps in cases like cd, and rm -v, and > another one or two I will mention below, and similar cases, where the > output is ancillary to the actual work being done) Good. So please go ahead and fix this bug in NetBSD sh. > As I was saying, to conclude, consider two more commands which are required > to write to standard output, and where exiting 0, even in the face of write > errors is probably the right thing to do. > > First, make: > > 98356 STDOUT > 98357 The make utility shall write all commands to
[Issue 8 drafts 0001471]: Add an orthogonal interface for immediate macro expansion definitions to make
A NOTE has been added to this issue. == https://austingroupbugs.net/view.php?id=1471 == Reported By:joerg Assigned To: == Project:Issue 8 drafts Issue ID: 1471 Category: Shell and Utilities Type: Enhancement Request Severity: Editorial Priority: normal Status: New Name: Jörg Schilling Organization: User Reference: Section:make Page Number:2888- Line Number:97001- Final Accepted Text: == Date Submitted: 2021-05-16 12:15 UTC Last Modified: 2021-07-29 08:17 UTC == Summary:Add an orthogonal interface for immediate macro expansion definitions to make == -- (0005421) geoffclare (manager) - 2021-07-29 08:17 https://austingroupbugs.net/view.php?id=1471#c5421 -- Re: https://austingroupbugs.net/view.php?id=1471#c5414 I agree that not expanding $$ is likely only needed in rare cases. However, I think the need to expand $$ is even rarer, and probably non-existent. The vast majority of uses of := work the same either way. Before the changes in 2016, the only reason anyone had to put on the rhs of a := would be as a work-around for it being expanded twice (or more). I assume the reason the default is not to preserve $$ is because there were existing makefiles where people has used this work-around, and defaulting to preserving $$ would break them. If $$ had always been preserved, those makefiles would not have needed to use in the first place. We should not make the same mistake with :::= Issue History Date ModifiedUsername FieldChange == 2021-05-16 12:15 joerg New Issue 2021-05-16 12:15 joerg Name => Jörg Schilling 2021-05-16 12:15 joerg Section => make 2021-05-16 12:15 joerg Page Number => 2888- 2021-05-16 12:15 joerg Line Number => 97001- 2021-05-16 14:35 shware_systems Note Added: 0005356 2021-05-16 17:18 psmith Note Added: 0005357 2021-05-16 19:02 shware_systems Note Added: 0005358 2021-05-16 19:12 kreNote Added: 0005359 2021-05-16 19:49 shware_systems Note Added: 0005360 2021-05-16 21:26 joerg Note Added: 0005361 2021-05-16 21:27 joerg Note Edited: 0005361 2021-05-22 19:02 psmith Note Added: 0005363 2021-07-08 16:43 geoffclare Note Added: 0005393 2021-07-08 16:57 hvdNote Added: 0005394 2021-07-08 19:30 steffenNote Added: 0005395 2021-07-08 19:32 steffenNote Added: 0005396 2021-07-08 20:01 kreNote Added: 0005397 2021-07-08 20:21 steffenNote Added: 0005398 2021-07-08 20:34 steffenNote Added: 0005399 2021-07-08 21:15 kreNote Added: 0005400 2021-07-08 22:33 steffenNote Added: 0005401 2021-07-08 22:33 steffenNote Added: 0005402 2021-07-10 12:30 joerg Note Added: 0005403 2021-07-10 18:03 psmith Note Added: 0005404 2021-07-10 21:26 mirabilos Note Added: 0005405 2021-07-15 14:51 joerg Note Added: 0005406 2021-07-15 14:52 joerg Note Edited: 0005406 2021-07-17 10:35 joerg Note Added: 0005407 2021-07-18 03:22 psmith Note Added: 0005408 2021-07-19 08:43 geoffclare Note Added: 0005409 2021-07-19 12:30 steffenNote Added: 0005410 2021-07-19 13:41 joerg Note