A NOTE has been added to this issue. ====================================================================== https://www.austingroupbugs.net/view.php?id=1649 ====================================================================== Reported By: kre Assigned To: ====================================================================== Project: Issue 8 drafts Issue ID: 1649 Category: Shell and Utilities Type: Error Severity: Objection Priority: normal Status: Resolved Name: Robert Elz Organization: User Reference: Section: XCU 2.6.5 Page Number: 2476 Line Number: 80478 - 80504 Final Accepted Text: See https://www.austingroupbugs.net/view.php?id=1649#c6488. Resolution: Accepted As Marked Fixed in Version: ====================================================================== Date Submitted: 2023-03-31 01:55 UTC Last Modified: 2023-10-01 22:22 UTC ====================================================================== Summary: Field splitting is woefully under specified, and in places, simply wrong ====================================================================== Relationships ID Summary ---------------------------------------------------------------------- related to 0001560 clarify wording of command substitution ======================================================================
---------------------------------------------------------------------- (0006497) kre (reporter) - 2023-10-01 22:22 https://www.austingroupbugs.net/view.php?id=1649#c6497 ---------------------------------------------------------------------- Re: https://www.austingroupbugs.net/view.php?id=1649#c6495 - thank you, yes, all of those changes look good to me, obviously at least the final one (no idea how that slipped through all of the re-reading that has been happening - to make it easier, that's in the paragraph which begins "For the purposes" which is about the 12th para I think). For the rest, those changes, or something similar, look to be an improvement. One of the reasons that I never expect wording I write to survive intact, is that I tend to write long run-on sentences, which can go on forever sometimes, even though I know that's not good writing. Corrections by those better at this than I am are always welcome. Re https://www.austingroupbugs.net/view.php?id=1649#c6496 No, there is no implication anything like you are suggesting. That "last sentence" you mean includes (requoting it here) subsequent data bytes that are not from the results of an expansion, or that do not form IFS white-space characters are required for a new field to begin. which is the part of it I believe you're focussing on. Let me slightly restate your example, inspired by the later PATH related example, just to avoid there being spaces (any white space) here - just because that is hard to display cleanly in text like this (and the PATH example shows that the issue isn't in any way related to the IFS white space delimiter your original example used). var='var:' sh -fc 'IFS=:; printf "[%s]\n" $var"" ' After variable expansion (there are no arith erxpansions or command subs to muddy the waters here) we have 3 initial fields in that printf command. The first two are just the "printf" word, and the format string, neither of which contain any expansion results, and so aren't touched by field splitting (not that the note was suggesting anything different, just for completeness here). So we have just the third field, which after variable expansion will contain the following byte sequence (into which I have inserted white space just for this note, to make things more obvious - this is why I needed to change the example from the original - just for easier visualisation in text like this): v a r : " " in which the first 4 bytes are the results of the variable expansion, and the final 2 are not. Now we do field splitting, the ':' in this case is an IFS char (in the original a space was used, which is also IFS white space, but in this example that makes no difference) when we see that the candidate contains "var" so is not empty, and becomes an output field. Now we're at the point where the sentence in question is thought to apply subsequent data bytes that are not from the results of an expansion, however does apply. there are still those two " (double quote) characters present, which are not from the results of an expansion, and hence satisfy that condition, which then allow a new candidate field to start. That one has no delimiter, so we reach the end of the input with the candidate containing two double quote characters, which is not empty, hence forms another output field - so now after field splitting we have (in this example) 4 fields printf "[%s]\n" var "" Subsequently (after pathname expansion, which is disabled in this example, so does nothing, not that there are any unquoted glob magic chars to invoke it anyway) we will do quote removal, which never alters the number of fields simply removes the quoting chars, still leaving the 4 fields, with all the " characters removed (so the final field will be a null string). That whole sentence though is redundant with the actual algorithm, which shows how that works - it was added merely as added reinforcement that if the input had not had those two trailing quote characters, then there would not be an extra field produced, just 3; which in the case you gave was never in doubt, IFS white space is always handled like that, but in the case with a ':' delimiter, many people assume there should be an empty field produced as the IFS are "field separators" implied by the 'S' in the var name - the whole sentence can (and probably should) be removed if it is thought to potentially cause more confusion than it removes (but this example isn't really a case of that I think - the problem here was that the quote characters weren't considered as being in the input, as if they'd already been replaced by a null string. The sentence earlier in the proposed spec (as it currently appears in https://www.austingroupbugs.net/view.php?id=1649#c6488): Note that this occurs before quote removal, any input field that contains any quoting characters can never be empty at this point. was another sentence I feared might be considered redundant, as other text in the standard makes that clear already - it should not really need to be restated here - but I have found that some people find it difficult to comprehend this part of the abstract algorithms in the standard - particularly when real implementations rarely act quite like this, so I included it as an additional reinforcement of this issue. True, it is in a paragraph that's primarily concerned with what do do when IFS='' applies - but whether or not quotes have already been removed from the input fields cannot (and does not) differ in that case, so I considered that just making sure that any reader has in mind that quote characters (in the abstract implementation) are still present, unaltered, in the fields in this one place would be sufficient. I'll leave it for others to decide whether more, or less, or adjusted, forms of these two redundant sentences should remain in the final version. Issue History Date Modified Username Field Change ====================================================================== 2023-03-31 01:55 kre New Issue 2023-03-31 01:55 kre File Added: ifs 2023-03-31 01:55 kre Name => Robert Elz 2023-03-31 01:55 kre Section => XCU 2.6.5 2023-03-31 01:55 kre Page Number => 2476 2023-03-31 01:55 kre Line Number => 80478 - 80504 2023-07-31 16:13 Don Cragun Note Added: 0006412 2023-09-07 14:14 kre Note Added: 0006459 2023-09-07 14:15 kre Note Added: 0006460 2023-09-07 14:30 kre Note Added: 0006462 2023-09-07 14:32 kre Note Added: 0006463 2023-09-07 14:41 kre Note Deleted: 0006463 2023-09-07 14:43 kre Note Edited: 0006462 2023-09-07 14:45 kre Note Added: 0006464 2023-09-07 14:54 kre Note Added: 0006465 2023-09-07 15:01 kre Note Added: 0006466 2023-09-07 15:03 kre Note Added: 0006467 2023-09-07 15:06 kre File Added: IFS-test 2023-09-07 15:07 kre File Added: POSIX-bug-1649-impl.sh 2023-09-07 15:09 kre File Added: Expected-Results 2023-09-07 15:15 kre Note Added: 0006468 2023-09-07 15:21 kre Note Added: 0006469 2023-09-07 15:22 kre Note Edited: 0006469 2023-09-07 16:12 kre Note Added: 0006471 2023-09-07 16:40 geoffclare Note Added: 0006472 2023-09-07 16:41 geoffclare Relationship added related to 0001560 2023-09-07 17:06 kre Note Added: 0006473 2023-09-07 17:09 kre Note Edited: 0006473 2023-09-11 03:35 kre Note Added: 0006477 2023-09-11 03:36 kre Note Added: 0006478 2023-09-11 03:39 kre Note Edited: 0006469 2023-09-11 03:59 kre Note Added: 0006479 2023-09-11 03:59 kre File Added: Revised-bug-1649-suggestion 2023-09-11 04:18 kre Note Added: 0006480 2023-09-11 04:20 kre Note Edited: 0006480 2023-09-21 15:51 Don Cragun Note Added: 0006482 2023-09-21 15:58 Don Cragun Note Deleted: 0006482 2023-09-21 16:48 Don Cragun Note Added: 0006483 2023-09-21 16:52 Don Cragun Note Edited: 0006483 2023-09-21 17:01 Don Cragun Note Edited: 0006483 2023-09-21 17:03 Don Cragun Final Accepted Text => See https://www.austingroupbugs.net/view.php?id=1649#c6483. 2023-09-21 17:03 Don Cragun Status New => Resolved 2023-09-21 17:03 Don Cragun Resolution Open => Accepted As Marked 2023-09-21 17:04 Don Cragun Tag Attached: issue8 2023-09-21 19:28 kre Note Added: 0006485 2023-09-25 09:43 Don Cragun Note Edited: 0006483 2023-09-25 09:47 Don Cragun Note Edited: 0006483 2023-09-25 14:21 Don Cragun Note Added: 0006487 2023-09-25 14:21 Don Cragun Resolution Accepted As Marked => Reopened 2023-09-25 15:43 Don Cragun Note Added: 0006488 2023-09-25 15:46 Don Cragun Final Accepted Text See https://www.austingroupbugs.net/view.php?id=1649#c6483. => See https://www.austingroupbugs.net/view.php?id=1649#c6488. 2023-09-25 15:46 Don Cragun Resolution Reopened => Accepted As Marked 2023-09-26 13:14 kre Note Added: 0006490 2023-09-29 01:50 Don Cragun Note Edited: 0006488 2023-09-29 01:58 Don Cragun Note Edited: 0006488 2023-09-29 02:04 Don Cragun Note Edited: 0006488 2023-09-29 02:16 Don Cragun Note Edited: 0006488 2023-09-29 02:28 Don Cragun Note Added: 0006491 2023-09-29 08:59 kre Note Added: 0006492 2023-09-29 20:42 larryv Note Added: 0006495 2023-10-01 16:00 stephane Note Added: 0006496 2023-10-01 22:22 kre Note Added: 0006497 ======================================================================
