Re: [1003.1(2024)/Issue8 0001852]: Clarify "$@[:$@:]" (with $# -eq 0)
On 03/09/2024 19:52, Martijn Dekker via austin-group-l at The Open Group wrote: Op 02-09-2024 om 21:30 schreef Steffen Nurpmeso via austin-group-l at The Open Group: At times you even get answers like "we already do more than required", but noone understands why "+10++11" is a correctly handled case, whereas "+10++VAR" is not, and if you generate In ksh, this is an error: $ echo $((+10++11)) -ksh: +10++11: assignment requires lvalue Does POSIX require this to work? ksh is finding an incorrect use of the ++ increment prefix operator. Correct is: $ echo $((+10+ +11)) 21 POSIX does not require ++ to be implemented but does not forbid it either. It says: The sizeof() operator and the prefix and postfix "++" and "--" operators are not required. It is fine for shells to implement features that are not required, and if the "++" feature is implemented, it is correct for it to result in an error in this example, the C rules that POSIX defers to specify that this is treated as a ++, not two consecutive +s, even if that then results in a syntax error. Cheers, Harald van Dijk
Re: [1003.1(2024)/Issue8 0001852]: Clarify "$@[:$@:]" (with $# -eq 0)
On 03/09/2024 09:29, Austin Group Bug Tracker via austin-group-l at The Open Group wrote: A NOTE has been added to this issue. == https://austingroupbugs.net/view.php?id=1852 == -- (0006872) geoffclare (manager) - 2024-09-03 08:29 https://austingroupbugs.net/view.php?id=1852#c6872 -- The standard is clear that the expansion of $@ in that case results in no fields, but it says nothing about the surrounding "s. Actually it does. You need to read the $@ description in 2.5.2 in combination with the text in 2.6 that references it: The shell shall create multiple fields or no fields from a single word only as a result of field splitting, pathname expansion, or the following cases: Parameter expansion of the special parameters '@' and '*', as described in Section 2.5.2, can create multiple fields or no fields from a single word. When this says "single word" it is referring to the whole word (which includes any quotes). Let me follow up on the list to clear up potential misunderstandings before responding on the bug tracker. The problem that I see with this interpretation is that there is no wording to distinguish between "$@" and "[$@]". When there are no positional arguments, we know "$@" must result in no fields, and we know "[$@]" must result in one field. But the description of $@ does not address this, it says when there are no positional arguments, it creates no fields, and if I understand your interpretation correctly, the fact that $@ appears in the word also means 2.6 does not specify that it results in a field. I was previously under the impression that the "[ and ]" cause the creation of that field, but I think your logic is saying that it does not, and if it does not, I cannot see what else does either. What am I missing? Cheers, Harald van Dijk
Re: [1003.1(2024)/Issue8 0001852]: Clarify "$@[:$@:]" (with $# -eq 0)
On 02/09/2024 23:06, Steffen Nurpmeso wrote:
Harald van Dijk wrote in
<696ea7fe-f40c-4a7d-b10a-62a9329d0...@gigawatt.nl>:
|On 02/09/2024 21:18, Steffen Nurpmeso via austin-group-l at The Open
|Group wrote:
|> hvd@: if you want to have a quote mechanism, you should
|> open an issue. On the list kre@ has shown how a POSIX
|> shell can already today do that a couple of years ago:
|
|I am not asking for a quote mechanism and am not sure where you got that
|idea from. I commented only on the standard's requirements for "$@"
|being unclear, and on potential implementation difficulty in meeting the
|current/suggested (depending on interpretation) requirements for "$@$@".
You linked to the message which responded to the ${@Q} quote
mechanism that had been brought up.
I did not. The page I linked to has every message in that thread, but I
linked to a specific message on that page. Considering that message
specifically speaks about unclarity in the standard wrt "$@$@", and my
comment was about unclarity in the standard wrt "$@$@", the logical
interpretation would be that that was indeed the message I intended to
link to.
Regarding the issue i follow what Geoff said, and am thankful for
his interpretation indeed (all these things are complicated enough
as is).
(It is hard to imagine Xu of dash would not "fix" the codebase.)
Your response suggests that I was considering the possibility Herbert Xu
would be unwilling to change dash to match Geoff's interpretation of
POSIX (regardless of whether it is the correct interpretation). That is
not what I said or meant. I pointed out specific implementation
difficulty in changing dash accordingly, which it would be fair to get
his input on rather than making assumptions.
Cheers,
Harald van Dijk
Re: [1003.1(2024)/Issue8 0001852]: Clarify "$@[:$@:]" (with $# -eq 0)
On 02/09/2024 21:18, Steffen Nurpmeso via austin-group-l at The Open Group wrote: hvd@: if you want to have a quote mechanism, you should open an issue. On the list kre@ has shown how a POSIX shell can already today do that a couple of years ago: I am not asking for a quote mechanism and am not sure where you got that idea from. I commented only on the standard's requirements for "$@" being unclear, and on potential implementation difficulty in meeting the current/suggested (depending on interpretation) requirements for "$@$@". Cheers, Harald van Dijk
Re: sh 'continue' shenanigans: negating
On 13/02/2024 22:38, Harald van Dijk via austin-group-l at The Open Group wrote: On 13/02/2024 21:04, Thorsten Glaser via austin-group-l at The Open Group wrote: > After all, the continue utility doesn't know it's called by the ! construct. In ash-derived shells, this basically works because internally, the continue utility, like all utilities, does know it's called by the ! construct, because the call is wrapped in a function that checks if the result was negated. But there are other ways to make this work too. Actually, it looks like I misremembered how this works internally, I was remembering another part of the shell, and a further look is making me wonder about a more complicated scenario. for x in y; do ! break done echo $? for x in y; do ! eval break done echo $? Note that I changed the example from "! continue" to "! break" because that makes it a more realistic and useful scenario: it allows an early exit to be effectively reflected in the exit status of the "for" loop. Per bug 842, it is unspecified whether the loop is considered to enclose the break command inside the eval. I am only considering shells where it does (which is all the ones I have tested). Because the eval command parses a command from a string, here, the second ! is not part of the last pipeline. The last pipeline is just "break". The "eval" command's exit status would be negated, but stops prematurely. Therefore, as long as the break command exits from the loop, I believe POSIX requires this to print 1 0. All shells I can test print either 1 1, or 0 0. If no shell implements what I believe POSIX specifies, then probably either my understanding of POSIX is wrong, or POSIX is wrong, and it would be a mistake to try to implement my understanding of the spec as it currently stands. What is the intended behaviour here? Cheers, Harald van Dijk
Re: sh 'continue' shenanigans: negating
On 14/02/2024 07:54, Oğuz wrote: On Wed, Feb 14, 2024 at 9:58 AM Harald van Dijk wrote: The test script with 'return'? I mean this one: for x in y; do ! continue done echo $? Ah, thanks for the clarification. I still do not see the same results as you (I still see a difference between yash 2.53 and 2.54, not 2.54 and 2.55), but in dash, gwsh, and yash alike, it looks like it is that same fix for the return command that also affected the break and continue commands. I believe that is simply a bug, and unless the conclusion in this thread is that POSIX is wrong to specify what it does, I will see to it that at least in gwsh, this is restored for the break and continue commands. Cheers, Harald van Dijk
Re: sh 'continue' shenanigans: negating
On 14/02/2024 06:16, Oğuz wrote: On Wed, Feb 14, 2024 at 8:47 AM Harald van Dijk wrote: POSIX specifies: > The value of the special parameter '?' shall be set to n, an unsigned decimal integer, or to the exit status of the last command executed if n is not specified. In your example, n is 1, therefore $? shall be set to 1. Insofar as there is an exit status of the 'return' command itself, it is ignored because the 'return' command specifies that the usual rules for $?, those where it picks up the last pipeline's exit status, do not apply. As the negation only applies to the pipeline's exit status, that is ignored as well. ksh behaves the same way. Other shells do not, and return 0. I agree with you that 0 strikes me as a more logical result as well, but I cannot see an argument that POSIX permits so, I cannot see it permit any result other than 1. Thanks. I was just asking to clarify, I don't have a preference for either behavior yet. Given the script in OP, gwsh and dash 0.5.12 also print 0 by the way, but dash 0.5.11 prints 1. The test script with 'return'? That is not what I am seeing. I am seeing dash 0.5.11 print 0, and dash 0.5.12 and gwsh print 1 (matching bash). This was an intentional change: I had reported the old behaviour as a bug on the dash mailing list. yash 2.54 and yash 2.55 also differ in the same way. I am seeing yash 2.54 and 2.55 print 1. yash 2.53, however, prints 0. This too was an intentional change: someone had reported my dash bug report on the yash issue tracker and its maintainer agreed and fixed it. You think these recent changes result from a misinterpretation of 202x draft? Considering the changes match my understanding of what POSIX requires, I do not think there is any misinterpretation, no. Cheers, Harald van Dijk
Re: sh 'continue' shenanigans: negating
On 14/02/2024 05:06, Oğuz via austin-group-l at The Open Group wrote:
On Tuesday, February 13, 2024, Chet Ramey via austin-group-l at The Open
Group <mailto:austin-group-l@opengroup.org>> wrote:
`continue' is a builtin; continue has a return status; `!' says to
negate it. It seems easy to come to the conclusion that the script
should return 1.
The same can be said about `return'. But bash disagrees:
POSIX specifies the return command in a way that by my reading requires
bash's behaviour. Whether this is intentional I cannot say.
$ bash -c 'f(){ ! return 1;}; f; echo $?'
1
$
Does POSIX allow this or is it another case where bash diverges from POSIX?
POSIX specifies:
> The value of the special parameter '?' shall be set to n, an unsigned
decimal integer, or to the exit status of the last command executed if n
is not specified.
In your example, n is 1, therefore $? shall be set to 1. Insofar as
there is an exit status of the 'return' command itself, it is ignored
because the 'return' command specifies that the usual rules for $?,
those where it picks up the last pipeline's exit status, do not apply.
As the negation only applies to the pipeline's exit status, that is
ignored as well.
ksh behaves the same way.
Other shells do not, and return 0. I agree with you that 0 strikes me as
a more logical result as well, but I cannot see an argument that POSIX
permits so, I cannot see it permit any result other than 1.
Cheers,
Harald van Dijk
Re: sh 'continue' shenanigans: negating
On 13/02/2024 21:04, Thorsten Glaser via austin-group-l at The Open Group wrote: Chet Ramey wrote, according to the list archive ontinue' is a builtin; continue has a return status; !' says to negate it. It seems easy to come to the conclusion that the script should return 1. Yes, I can see how people could come to the conclusion. But “continue”, if it unwinds the loop, doesn’t really return, so its return status does not pass to the ! construct but is directly set before the next loop round or exit, in mksh (and other pdsh-derived shells). That's an implementation detail that is not part of the spec. The spec says that $? reports "the decimal exit status of the most recent pipeline". The most recent pipeline is "! continue". You're interpreting "! continue" as something like "continue; (exit $(( !$? )))" so that "continue" is a pipeline of its own, but that's neither how it's specified nor how other shells implement it. You write: > After all, the continue utility doesn't know it's called by the ! construct. In ash-derived shells, this basically works because internally, the continue utility, like all utilities, does know it's called by the ! construct, because the call is wrapped in a function that checks if the result was negated. But there are other ways to make this work too. Cheers, Harald van Dijk
Re: probable UB in recently accepted text
On 28/07/2023 21:51, Thorsten Glaser via austin-group-l at The Open Group wrote: Davin McCall via austin-group-l at The Open Group dixit: Since the operand isn't evaluated, there is no null pointer dereference. It is That’s what I told them, but they weren’t convinced. This has been definitively answered in DRs #109 and #132 in the context of division by zero, but the same reasoning applies in full to null pointer dereferences. Even if an expression has operands that would otherwise permit evaluation as a constant expression, if the expression would have undefined behaviour if evaluated, but the expression isn't evaluated, and is not used in a context that requires a constant expression, the implementation must accept it. https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_109.html https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_132.html Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 05/04/2023 18:05, Harald van Dijk via austin-group-l at The Open Group wrote: On 05/04/2023 17:44, Oğuz wrote: 5 Nisan 2023 Çarşamba tarihinde Harald van Dijk <mailto:a...@gigawatt.nl>> yazdı: I am not sure which other ash based shells you were looking at, /bin/sh on NetBSD and FreeBSD Thanks. I indeed see the same results as you on a recent version of FreeBSD sh (the one on the FreeBSD 13.1 installation media). There is a legitimate benefit to this: swapping variables without an additional helper variable actually works in that implementation. a=1 b=2 a=$b b=$a echo $b $a Actually, I am seeing another legitimate benefit: The standard is clear that in export a="$b" b="$a" the arguments to the 'export' command are expanded first, and then the 'export' command is run which performs the assignments, thus swapping the values of 'a' and 'b', but also exporting their values. It seems rather a shame that export a=<...> b=<...> and a=<...> b=<...> export a b do not have the same effect, so that the 'export a b' line can be taken out when exporting is not desired. Under the traditional ash behaviour, these do have the same effect. This consistency, to me, is a good argument in favour of the traditional behaviour and in favour of POSIX permitting that behaviour. Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 06/04/2023 20:03, Chet Ramey via austin-group-l at The Open Group wrote: On 4/6/23 1:55 PM, Harald van Dijk wrote: One additional data point: in schily-2021-09-18, Jörg's last release, obosh, the legacy non-POSIX shell that is just there for existing scripts and for portability testing, prints 0 (using `` rather than $()), whereas pbosh and sh, the minimal and extended POSIX versions of the shell, print 1. This does provide extra support for the view that this was a change that POSIX demanded, that the deviation from historical practice was intentional, but does not answer what the reasoning might have been. I doubt it was `demanded'; the bosh change immediately followed an austin- group discussion (we both participated) about this exact issue. Maybe he thought it was the right thing based on that discussion. Possibly something lost in translation here. What I meant by "that POSIX demanded" was just "that was intended to be required for POSIX conformance". As part of the discussion, he wrote: > The important thing to know here is that the Bourne Shell has some > checkpoints that update the intermediate value of $?. Since that changed in > ksh88 and since POSIX requires a different behavior compared to the Bourne > Shell, I modified one checkpoint in bosh to let it match POSIX. so he had already been modifying that behavior before 2021, maybe after interp 1150. And again. I can see how you would read my message as saying that this is something he changed in his last release. That wasn't what I was going for, I only meant that his last release was the one that I tested. It did not seem relevant to test all earlier versions to figure out when this changed. Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 06/04/2023 16:17, Chet Ramey wrote: On 4/5/23 12:36 PM, Harald van Dijk wrote: On 05/04/2023 15:35, Chet Ramey via austin-group-l at The Open Group wrote: On 4/5/23 9:06 AM, Martijn Dekker via austin-group-l at The Open Group wrote: Consider: false || echo $(true) $? dash, mksh and yash print 1. bash, ksh93 and zsh print 0. Which is right? I believe dash, mksh, yash are already right based on the current wording of the standard. As Martijn wrote, the rule is that $? "Expands to the decimal exit status of the most recent pipeline", the most recent pipeline in the shell environment in which $? is evaluated is "false", and changes in the subshell environment shall not affect the parent shell environment, including changes in the subshell environment to $?. That's certainly one interpretation, and may indeed be what the 1992 authors intended. My question is why they would choose something other than what the so-called reference implementations (SVR4 sh, ksh88) did. That is a good question that I have no answer for. One additional data point: in schily-2021-09-18, Jörg's last release, obosh, the legacy non-POSIX shell that is just there for existing scripts and for portability testing, prints 0 (using `` rather than $()), whereas pbosh and sh, the minimal and extended POSIX versions of the shell, print 1. This does provide extra support for the view that this was a change that POSIX demanded, that the deviation from historical practice was intentional, but does not answer what the reasoning might have been. Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 05/04/2023 17:44, Oğuz wrote: 5 Nisan 2023 Çarşamba tarihinde Harald van Dijk <mailto:a...@gigawatt.nl>> yazdı: I am not sure which other ash based shells you were looking at, /bin/sh on NetBSD and FreeBSD Thanks. I indeed see the same results as you on a recent version of FreeBSD sh (the one on the FreeBSD 13.1 installation media). There is a legitimate benefit to this: swapping variables without an additional helper variable actually works in that implementation. a=1 b=2 a=$b b=$a echo $b $a As it turns out, the at the moment still rather incomplete mrsh <https://mrsh.sh/> also behaves this way. Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 05/04/2023 16:25, Oğuz via austin-group-l at The Open Group wrote: 5 Nisan 2023 Çarşamba tarihinde Chet Ramey via austin-group-l at The Open Group <mailto:austin-group-l@opengroup.org>> yazdı: but should it be set fron the command substitution for the assignment to c? I think it'd be practical, is there a reason why it shouldn't? And while we're at it, is there a reason why assignments in a simple command shouldn't be applied sequentially, from left to right? Most shells already do it that way; that is, a=1 b=$a c=$b echo $a $b $c prints `1 1 1' in them. Outliers are ash based shells; they apply assignments concurrently but it isn't useful at all. Note that in dash, this only applies to older versions. dash 0.5.7 (released in 2011) and higher print "1 1 1" like almost all other shells, and other dash-derived shells (busybox ash, and my own) keep it that way. I am not sure which other ash based shells you were looking at, but it may be worth making sure you are on a current version. Cheers, Harald van Dijk
Re: $? behaviour after comsub in same command
On 05/04/2023 15:35, Chet Ramey via austin-group-l at The Open Group wrote: On 4/5/23 9:06 AM, Martijn Dekker via austin-group-l at The Open Group wrote: Consider: false || echo $(true) $? dash, mksh and yash print 1. bash, ksh93 and zsh print 0. Which is right? I believe dash, mksh, yash are already right based on the current wording of the standard. As Martijn wrote, the rule is that $? "Expands to the decimal exit status of the most recent pipeline", the most recent pipeline in the shell environment in which $? is evaluated is "false", and changes in the subshell environment shall not affect the parent shell environment, including changes in the subshell environment to $?. A variant with slightly different semantics: (exit 8) a=4 b=$(exit 42) c=$? Likewise, the most recent pipeline in the shell environment in which c=$? is evaluated is "(exit 8)", so c should be set to 8. echo status:$? c=$c Here, as shells agree, $? should expand to 42. The standard is clear about what $? should be for the echo, but should it be set fron the command substitution for the assignment to c? The way the standard specifies that the final $? should expand to 42 is in 2.9.1 Simple Commands: > [...] If there is no command name, but the command contained a command substitution, the command shall complete with the exit status of the last command substitution performed. Otherwise, the command shall complete with a zero exit status. That is, the command substitution itself does not cause $? in the parent environment to change, rather it is the fact that the entire "a=4 b=$(exit 42) c=$?" command exits with status 42 that causes $? to change. Nothing else in the standard indicates that the parent shell environment's $? value changes; while the expansion of the second command is in place, "the most recent pipeline" is still the previous one. You have linked to <https://www.austingroupbugs.net/view.php?id=1150> in a subsequent reply. This does not see the current wording as sufficiently clear, but does clear up with new wording that regardless of whether this can be inferred from the current wording of the standard, this interpretation is the intended one. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 15/03/2023 00:17, Chet Ramey wrote: On 3/14/23 4:58 PM, Harald van Dijk wrote: On 14/03/2023 20:41, Chet Ramey wrote: On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. Isn't it? The mode bash gets into when invoked as sh is described in the manpage (looking at the 5.2.15 manpage) as: If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [...] When invoked as sh, bash enters posix mode after the startup files are read. The mode bash gets into when POSIXLY_CORRECT is set, the mode that can also be obtained with --posix, is described in the manpage as: When bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. Right. When you force posix mode immediately, as I said above, bash won't read the startup files. A login shell named sh or -sh reads the historical startup fles, then enters posix mode. We are clear on how bash behaves, I believe. What is not clear to me is how I should have worded my message so that you would not have taken issue with it. As far as I can tell, you are saying the same thing I was. I wrote that bash appears to disables the reading of .profile in POSIX mode entirely. It does. I do not see any contradiction between that and what you wrote: while a clarification that this does not apply to bash being invoked as sh or -sh is useful, I made no comment about what happens when bash is invoked as sh or -sh. My comment was about POSIX mode, and when invoked as sh or -sh, per the manpage and your own message, the startup file processing happens while bash is still in non-POSIX mode. So what should I have written instead? If I had written bash appears to disable the reading of .profile in POSIX mode (i.e. bash --posix) entirely. would that have been sufficiently clear to avoid this discussion? If so I will keep that in mind. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 14/03/2023 20:41, Chet Ramey wrote: On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. Isn't it? The mode bash gets into when invoked as sh is described in the manpage (looking at the 5.2.15 manpage) as: If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [...] When invoked as sh, bash enters posix mode after the startup files are read. The mode bash gets into when POSIXLY_CORRECT is set, the mode that can also be obtained with --posix, is described in the manpage as: When bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. My wording appears to be consistent with how bash describes itself in its manpage: there is POSIX mode during startup file processing, and POSIX mode after startup file processing. Only the POSIXLY_CORRECT / --posix behaviour causes startup file processing to happen in POSIX mode, and results in the reading of .profile being disabled. But if I am misunderstanding how you intend for the different modes of bash to be referred to, if you can clarify I can take care to be more careful with my wording in the future. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 12/03/2023 19:10, Robert Elz wrote:
Date:Fri, 10 Mar 2023 23:40:18 +
From:"Harald van Dijk via austin-group-l at The Open Group"
Message-ID:
| Based on past experiences, I am assuming the e-mail this is a reply to
| was meant to be sent to the list and I am quoting it in full and
| replying on the list for that reason.
Thanks, and yes, it was - my MUA absolutely believes in the one true
meaning of Reply-To (where the author of the message to which the reply
is being sent requests that replies be sent -- to addresses in that field,
and no others). I need to manually override it when I choose to ignore
that request and send to different addresses (which is allowed, but in
general, done only with proper consideration of why). This list always
directs that all replies go only to the author of the message, and never
to the list itself. Irritating...
I know. Even more frustrating is the reasoning we have been given as to
the reasoning for it ("to better handle DMARC email authentication for
messages" -- no, DMARC absolutely does not require that).
| Sourcing arbitrary script fragments and having assurance that they do
| not exit the shell is not reasonable, as the arbitrary script fragment
| could contain an 'exit' command.
Of course, deliberate exits aren't the issue, only accidental ones.
| Beyond shell options and variable assignments not persisting in the
| parent shell, are there any other issues you see with running them in a
| subshell?
The whole point of many . scripts is to alter the shell's environment,
if they were just arbitrary commands, not intended to affect the current
shell, they'd just be sh scripts, and run the normal way. The very act
of using the '.' command more or less means "must run in the current shell).
"(. file)" is silly, "file" would accomplish the same thing (if executable,
otherwise "sh < file" after finding the path to file) in a more obvious way.
This does not result in the same behaviour. (. file) can be used to
inherit unexported shell variables, shell functions, aliases, etc. from
the parent shell that would not be available to an explicitly launched
sh subprocess.
I agree that it is likely that a . script would be sourced in order to
change the current shell environment, but I do not agree that it is the
only legitimate use of it, and I had assumed based on your "without
risking the shell exiting" that you were considering one of those other
uses.
Apart from options and variables, . files often define functions, change
the umask and perhaps ulimit, and may alter the current directory, set exit
(or other) traps, ... anything in fact.
Sure, they are in the same category as options and variables.
As an example, consider what you might put in your .profile or $ENV
file - those are run in more or less the same way as a '.' file (just
without the PATH search to locate the file). XRAT C.2.5.3 says almost
exactly that about ENV.
Indeed. Worth explicitly pointing out here is that they are only used by
interactive shells, so in this comparison, only the behaviour of the '.'
command in interactive shells is relevant.
(Strangely though, even though .profile is
mentioned several times as a place where things can be set, it doesn't
appear in the standard (as something that shells process) at all - which
is kind of odd really, since it is considerably older then ENV, and as best
I can tell, supported by everything. The closest that we get is a mention
in XRAT that "some shells" run it at startup of a login shell. Which are
the other shells? That is, the ones that don't run .profile? And I don't
mean in situations like bash, which prefers .bash_profile if it exists.
bash appears to disables the reading of .profile in POSIX mode entirely.
yash only ever appears to source .yash_profile, it does not fall back to
.profile if no .yash_profile exists.
I doubt that you'd want those scripts run in a subshell environment,
I also doubt that you want the shell to exit if there's an error in
one of them. How would you ever be able to log in (and start a shell)
if it exited before you ever had a chance to run a command? If you can't
log in, because your shell won't start, how would you ever fix the problem?
As best I can tell (I have done very limited testing of this) shells tend
to simply abort processing one of those scripts upon encountering an error
(like a syntax error, etc - not executing "exit" - that should exit) and
just go on to the next step of initialising the shell. They don't just
exit because there's a syntax error - most shells report the error (not all),
but I couldn't find one which exits.
This is indeed a problem. My own
Re: [1003.1(2016)/Issue7+TC2 0001234]: in most shells, backslash doesn't have two meaning wrt pattern matching
On 27/06/2019 09:29, Stephane Chazelas wrote:
2019-06-27 08:59:29 +0100, Harald van Dijk:
[...]
If there is no fnmatch() implementation that behaves that way, then agreed
that it makes sense to just specify that. That pax example in the rationale
should then also be changed to not escape any parenthesis.
What did this pax example come from, though? Was that based on a real pax
implementation that did have special treatment of parentheses, not just an
invention?
[...]
That's something I also wondered.
I do have a vague recollection that some early glob()
implementations were actually calling "sh" to expand globs (so
for instance glob("*`reboot`*") would reboot (which is currently
allowed by the spec as ` is a shell special character). Could it
be linked to that? I wouldn't expect it to apply to fnmatch()
though.
Or maybe I'm confusing with perl globs that used to call csh.
I hate to revisit this thread, but I found out today that this was very
much an intentional decision and the resolution of bug 1234 goes
completely contrary to that intent.
Quoting https://unix.org/whitepapers/shdiffs.html:
However, the command:
find . -name 'a(b*' -print
did work.
Now, this form requires escaping of the shell character to avoid side
effects from implementation extensions:
find . -name 'a\(b*' -print
This makes it as explicit as it can possibly be that the wording had
always been intended to be interpreted as exactly what it said, that
metacharacters *do* require escaping here and have required so ever
since the first version of POSIX, that it was intentional that an
unescaped '(' in a find -name pattern may result in other behaviour than
matching a literal '('.
Cheers,
Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
Based on past experiences, I am assuming the e-mail this is a reply to was meant to be sent to the list and I am quoting it in full and replying on the list for that reason. On 10/03/2023 21:24, Robert Elz wrote: Date:Fri, 10 Mar 2023 18:13:00 + From:"Harald van Dijk via austin-group-l at The Open Group" Message-ID: | Other shells that exit are bosh, yash, and my own. It's both what POSIX | currently requires (contrary to what kre wrote on the bug) That's not how I intended what I wrote to be interpreted, I meant exactly what you said - when I wrote "most shells are doing what shells always have, and what the standard requires", I meant "exiting". I'm glad that that is what you meant. I wrote my mail before reading your mail where you said you tested something different, so when I read that you wrote that what most shells are doing is what the standard requires, it could only be interpreted to mean the opposite of what you intended. But as I wrote in my previous message, I was actually testing "command eval" rather than "command ." which I would normally expect to work about the same way in this regard, but it turns out that not shells all do. Further, and subsequent to when I sent that last message, I went and looked at my tests again - the way I do these is by (for tests like this one) composing a command line as input to one shell (each has its own xterm in my shell testing root window page - they're all tiled), then pasting it into the windows for all the others - then I can see the results from all of them, at the same time, and easily compare what happened (the command always starts $SHELL kind of like the example Geoff showed, except I do not quote that, because sometimes SHELL needs to be "bash -o posix" or similar, and I want that field split, not treated as a quoted word. For this, I tested both without, and with, "command" present ... but it turns out that somehow, for some of the shells, instead of running both tests, I managed to paste the wrong command, and ran the one without "command" twice, without noticing. That even included the NetBSD sh test, which contrary to what I said before, turns out does do the same thing for "." and "eval" in both cases (exit without command, not exit with it) which is what I had expected, before I saw the results of the incorrect test - before I noticed it was incorrect. | and what I think is probably the right thing for shells to do. I don't. I want to be able to source arbitrary script fragments, and eval arbitrary strings (there are no security issues here, the fragments and strings, are all provided by the user running the shell - anything that could be done buried one of those other ways, could simply be done as a command without subterfuge) without risking the shell exiting. Sometimes running them in a subshell works, but only sometimes. Sourcing arbitrary script fragments and having assurance that they do not exit the shell is not reasonable, as the arbitrary script fragment could contain an 'exit' command. In order to be able to do this, a subshell is needed. That will mean that variable assignments etc. do not persist in the parent shell, but that is unavoidable. Either the sourced script affects the parent shell, in which case it can cause the parent shell to exit, or it does not affect the parent shell. Beyond shell options and variable assignments not persisting in the parent shell, are there any other issues you see with running them in a subshell? | Whether bug 1629 should introduce a significant shell consistency issue | is not separate from bug 1629. Perhaps that one, and some new one, yet to be submitted, should be considered together, but resolving 1629 the right way should not be held hostage by other ancient weirdness that might not be so easy to alter. But perhaps after all, it might be - if it is only yash, bosh and your shell not already continuing after "command . file" fails because of a syntax error, then those might not matter, and those, plus, I think, mksh and ancient pdksh (and consequently, probably ksh88 as well) for "command eval 'gibberish<;)'" failing the same way then I'd guess mksh can get changed, and the others also no longer really matter. You have left out bash 4 here. For 'command eval', that version behaves the same way as for 'command .' in that it causes the shell to exit on syntax errors. | Bug 1629 started as trying to see what | shell authors are willing to implement. No, it started because read errors were not being handled in a rational way. A proposed solution depended upon what shell authors are willing to implement. True, I should have said "the discussion on bug 1629 started" rather than &qu
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 10/03/2023 17:12, Geoff Clare via austin-group-l at The Open Group wrote: -- (0006200) kre (reporter) - 2023-03-10 14:52 https://www.austingroupbugs.net/view.php?id=1629#c6200 -- Re https://www.austingroupbugs.net/view.php?id=1629#c6197 https://www.austingroupbugs.net/view.php?id=1629#c6198 https://www.austingroupbugs.net/view.php?id=1629#c6199 : I agree it is probably a problem that shells exit on syntax errors after "command ." (also "command eval") but it is a problem in the sense that it would be better if that isn't what happened - and it isn't just "some shells" that exit, it is most of them. None of the shells I tried exited. I used a file containing: foo < and ran: "$shell" -c 'command . ./file ; echo $?' All of bash, ksh88, ksh93, dash and mksh reported the syntax error and then executed the echo command (which output a non-zero number). For bash, it depends on the version. Version 4 will exit. Other shells that exit are bosh, yash, and my own. It's both what POSIX currently requires (contrary to what kre wrote on the bug) and what I think is probably the right thing for shells to do. In any case, this isn't a problem with the proposed resolution of this issue (read error behaviour) - a new bug needs to be filed and processed to get the behaviour of "command ." and "command eval" altered to be "not exit" (and for this, it may need to start out just being "may exit" rather than "shall not exit" with a future directions indicating that the stronger form will apply in a later version. I agree this should be handled as a separate issue, hence my replying by email instead of a note in the bug. Whether bug 1629 should introduce a significant shell consistency issue is not separate from bug 1629. Bug 1629 started as trying to see what shell authors are willing to implement. In that case, let me be clear, as it stands this is not something I am going to implement as written. I know you have no reason to care what I do in my shell, and I know bosh sadly isn't going to see an update anyway, but I would hope that authors of the other shells also have the good sense to implement something that makes sense to them and keep it internally consistent, whether that is by ignoring what 1629 specifies, or by ignoring what is required for syntax errors. There are more issues that make me doubt this is going to be implemented as written in other shells too, but that's for the other authors to raise. Cheers, Harald van Dijk
Re: Writing the "[job] pid" for async and-or lists
On 24/02/2023 15:59, Robert Elz via austin-group-l at The Open Group wrote: In new text added by bug 1254, in the (renamed) section "Asynchronous AND-OR Lists", the following text has been added: If the shell is interactive and the asynchronous AND-OR list became a background job, the job number and the process ID associated with the job shall be written to standard error using the format: "[%d] %d\n", , If the shell is interactive and the asynchronous AND-OR list did not become a background job, the process ID associated with the asynchronous AND-OR list shall be written to standard error in an unspecified format. That is, in all cases, in an interactive shell, when an async AND-OR list (which really just translates as an async anything) is run, something must be written to standard error.(What the something is, isn't the issue here). For simple commands like sleep 1 & most shells do that (ash derived shells don't, which is why I am looking at this particular issue, the NetBSD shell is one of those, and I was planning to change it, at least in "posix" more - not sure our users would appreciate this happening by default). yash writes the info, but in jobs format, rather than that specified above. But in other cases, like for example (sleep 1&) only bosh and an oldish ksh93 (of shells I have to test) write anything. (A more updated ksh93 does not). For this case : $(sleep 1&) even bosh doesn't write the job number &/or pid, only the oldish ksh93 does. What is written above seems to demand the old ksh93 behaviour, which I doubt is intended. Can we agree (and then after Draft 3 is available, submit a bug report) that this text should only apply to the top level of an interactive shell, and not to any subshell environments. I agree, but a subshell of an interactive shell is effectively non-interactive anyway, and many of the special rules for interactive shells should not apply to subshells of interactive shells and already don't in various existing shells (but the extent to which they don't varies from shell to shell). Rather than make a special exception for process IDs, could this be made a general rule? Cheers, Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001614]: XSH 3/mktime does not specify EINVAL and should
On 25/11/2022 05:40, Robert Elz via austin-group-l at The Open Group wrote: Date:Thu, 24 Nov 2022 15:49:49 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: | Combining the above with the TZ rules, if TZ=EST5EDT then POSIX requires | that mktime() calculates seconds since the Epoch as specified in XBD 4.16 | then applies a timezone adjustment of 5 hours and (depending on tm_isdst | and the specified date and time) a seasonal adjustment of 1 hour (with | implementation-defined start and end times, but we can eliminate that | by including a rule in the TZ value). There is nothing unspecified | here at all. I could (if I needed) dispute more of your message than this, but there is no need, this is enough for the purpose here. In the case where tm_isdst == -1 (which is the relevant one here) and where the broken down time referenced by timeptr specifies a time in the gap, that is, a time which never existed (or ever will) and so is not summer time, and is not standard time, it is not any kind of local time at all (except erroneous) and the application has not told us which to pretend it should be, where exactly is the specification of which offset is supposed to apply? Don't bother hunting, there is none, and as you have said on various topics many times, that which is not specified is unspecified. Note that it is not unspecified whether it is the standard time offset, or the summer time offset, it is simply unspecified. So, there is something here unspecified, and if the application invokes unspecified behaviour, the implementation is free to produce any result that pleases it, right?Hence an error return is acceptable. And if that is true, an errno value ought to have been assigned. To possibly avoid some of the discussion of whether EINVAL is appropriate: Geoff wrote "then applies a timezone adjustment of 5 hours and (depending on tm_isdst and the specified date and time) a seasonal adjustment of 1 hour". Does POSIX actually specify the seasonal adjustment, if applied, has to be 1 hour? If it does, that's a bug, as DST isn't always 1 hour everywhere on the world. (Lord Howe Island appears to switch between UTC+10:30 and UTC+11:00.) If it doesn't, and the time zone doesn't specify what the seasonal adjustment is for this non-existent local time, is an implementation permitted to deduce a seasonal adjustment of 1e99 hours? If that is valid, that would make the result out of time_t's range, and require EINVAL. And if that is valid, that is indistinguishable from an implementation that uses EINVAL directly without going through that logic, and an implementation that is indistinguishable from a conforming one is itself conforming. Cheers, Harald van Dijk
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 20/05/2022 01:11, Christoph Anton Mitterer wrote:
On Thu, 2022-05-19 at 09:05 +0100, Harald van Dijk wrote:
The above, AFAIU, mean that any shell/fnmatch matches a valid
multibyte
character... but also a byte that is not a character in the locale.
Correct, though as I wrote later on, the way they go about it is
different.
And I think, for any real standardisation of this (which I'd still love
to see) quite a few things would need to be reasonably defined,
including but most likely not limited to:
- Does * match bytes (by which I mean 1-n which don't form valid
characters in the current locale).
This is another one of those that seemed obvious enough to me that I did
not think to check explicitly. As far as I can tell at a quick glance, *
matches any number (zero or more) of ?, whatever ? means, except in the
case of a particular shell bug that also breaks scripts already required
by the standard to work.
- The same for ? ... and if that matches bytes at all - only 1 or n?
It matches a single character or a single byte that is not part of a
character.
- In which "direction" is the matching done, which AFAIU would be
important, e.g.
\303\244\244
*if* '?' were to match also bytes, is '?\244' meant to be matching
the character followed by the byte:
(\303\244)\244
or could it non-match the byte followed by more bytes:
(\303)\244\244
In an UTF-8 locale, \303\244\244 is an invalid character string. As the
test results have shown, in some of the implementations, that causes
pattern matching to be done as if under the C locale. In those
implementations, it does not match ?\244, but it does match ??\244. In
other implementations, only the final invalid byte \244 is given special
treatment, in which case the whole string does match ?\244.
- And I guess these questions would also pop up for the ##, #, %% and %
forms of parameter expansions, especially when one has a local like
Big-5.
In the sense of, can one strip of a character (or byte) that forms
part of another character.
This does pop up there too but the questions are not new, they are the
same questions that already pop up for regular pattern matching.
${var#pat} strips a leading pat of $var if and only if $var matches
pat*. ${var%pat} strips a trailing pat of $var if and only if $var
matches *pat. That said, in those cases where shells disagree over
whether $var matches pat* / *pat, that is those cases where I would
propose making the result unspecified, the results may also be
inconsistent with the same shell's pattern matching in other contexts.
If shells were required not to decompose such valid characters (that
contain another valid character, when looked at it from right to
left), then it would also need to be defined how the strings needed
to be interpreted (most likely of course: as defined by the
respective char encoding).
This is where the example with β comes in. The current standard, as far
as I can tell, *already* requires
var=β
echo ${var%]}
case $var in
*]) echo match
esac
to print "β", and not print "match", regardless of how that β is
encoded. There are no invalid bytes here. This can only be done by
processing the string left-to-right.
So for all these cases it might additionally be required to check how
the different shells behave when trying to ##, #, %%, % ...
And AFAIU, some actually allow to "decompose" a character.
Yes, this is expected and consistent with regular pattern matching.
And even if the standard were to say, that it must check whether the
matched part is part of a bigger multibyte character (like in the
BIG5 case) and then not allow to decompose that would it still
be allowed to do so when the pattern contains bytes that are
themselves not valid characters)
Yes, they should be allowed to do so. As we have seen, bash and GNU
fnmatch() simply fall back to single-byte-character-set matching if the
string or pattern is not valid in the current locale, and what you
describe would be the natural result of that.
- Are there any undesired side effects? Like bash, has the nocasematch
shell option... which IIRC affects patterns... would we break
anything in such fields?
How could we? What bash does if a non-standard shell option is set is
not covered by POSIX, nor should it be.
- I think it already is defined (more or less) which locale is actually
used for the matching, i.e. the current one as set by LC_CTYPE and
not e.g. the "lexical" locale defined on the start of the shell.
Agreed.
I tested this now. In that same list of shells, and in glibc
fnmatch(),
? only matches a single invalid byte. Tested in an UTF-8 locale with
the
string \200\200 and the patterns ? and ??. With ?, they do not match.
With ??, they do.
The next question that would come to my mind:
Do t
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 15/05/2022 16:14, Harald van Dijk via austin-group-l at The Open Group wrote: On 19/04/2022 01:52, Harald van Dijk via austin-group-l at The Open Group wrote: On 15/04/2022 04:57, Christoph Anton Mitterer wrote: On Fri, 2022-04-15 at 00:44 +0100, Harald van Dijk via austin-group-l at The Open Group wrote: If there is interest in getting this standardised, I can spend some more time on creating some hopefully comprehensive tests for this to confirm in what cases shells agree and disagree, and use that as a basis for proposing wording to cover it. I'd love to see that and if you'd actually do so, I'd kindly ask Geoff to defer any changes in the ticket #1564 of mine, until it can be said whether it might be possible to get that standardised. Very well, I will post tests and test results as soon I can make the time for it. Please see the tests and results here. Apologies for the HTML mail but this is hard to make readable in plain text. String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \303\244 [\303\244] no match match match match match match match \303\244 ? no match match match match match match match \303 [\303] match match match match match match match \303 ? match match match match match match match \303.\303\244 [\303].[\303\244] no match no match no match match match match match \303.\303\244 ?.? no match no match no match match match match match \303\303\244 [\303][\303\244] no match no match no match match match match match \303\303\244 ?? no match no match no match match match match match \303\244.\303 [\303\244].[\303] no match no match no match match match match match \303\244.\303 ?.? no match no match no match match match match match \303\244\303 [\303\244][\303] no match no match no match match match match match \303\244\303 ?? no match no match no match match match match match \303\244 \303* match match match match no match match no match \303\244 \303? match match match no match no match match no match \303\244 [\303]* match match match match no match match no match \303\244 [\303]? match match match no match no match match no match \303\244 *\204 match match match no match no match no match match \303\244 ?\204 match match match no match no match no match no match \303\244 *[\204] match match match no match no match no match no match \303\244 ?[\204] match match match no match no match no match no match \243] [\243]] match match match match match match match \243] ? no match match match match match match match \243 ? match match match match match match match \243 [\243] match match match match no match no match error \243 [\243!] match match match match match match match \243] [\243!]] match match no match no match no match match no match \243] ?] match match no match no match no match no match no match \243] *] match match no match no match no match no match m
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 19/05/2022 02:46, Christoph Anton Mitterer wrote: On Sun, 2022-05-15 at 16:14 +0100, Harald van Dijk wrote: Please see the tests and results here. So dash/ash/mksh/posh/pdksh,... and every other shell that doesn't handle locales at all (and thus works in the C locale)... is anyway always right (except for bugs), since any (non-NUL) byte is treated as a character. Correct. For the other shells (and fncmatch): String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \303\244 [\303\244] no match match match match match match match \303\244 ? no match match match match match match match \303 [\303] match match match match match match match \303 ? match match match match match match match The above, AFAIU, mean that any shell/fnmatch matches a valid multibyte character... but also a byte that is not a character in the locale. Correct, though as I wrote later on, the way they go about it is different. String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \303.\303\244 [\303].[\303\244] no match no match no match match match match match \303.\303\244 ?.? no match no match no match match match match match \303\303\244 [\303][\303\244] no match no match no match match match match match \303\303\244 ?? no match no match no match match match match match \303\244.\303 [\303\244].[\303] no match no match no match match match match match \303\244.\303 ?.? no match no match no match match match match match \303\244\303 [\303\244][\303] no match no match no match match match match match \303\244\303 ?? no match no match no match match match match match The above, I'm not quite sure what these tell/prove... I assume the ones with '?': that for all except bash/fnmatch '?' matches both, valid characters and a single byte that is no character. Correct. And the ones with bracket expression, that these also work when the BE has either a valid character or a byte (that is not a character) and vice-versa? Correct. If Chet is reading along, is the above intended in bash, or considered a bug? IMO it would have been interesting to see whether ? would also match multiple bytes that are each for themselves and together no valid character... cause for '*' one can kinda assume that it has this "match anything" meaning... one could also say that is more or less reasonable that '?' matches a single invalid byte... but why not several of them? I tested this now. In that same list of shells, and in glibc fnmatch(), ? only matches a single invalid byte. Tested in an UTF-8 locale with the string \200\200 and the patterns ? and ??. With ?, they do not match. With ??, they do. String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \303\244 \303* match match match match no match match no match \303\244 \303? match match match no match no match match no match \303\244 [\303]* match match match match no match match no match \303\244 [\303]? match match match no match no match match no match \303\244 *\204 match match match no match no match no match match \303\244 ?\204 match match match no match no match no match no match \303\244 *[\204] match match match no match no match no match no match \303\244 ?[\204] match match match no match no match no match no match So unlike before, in the above bash/fnmatch do seem to let '?' match a single byte that is not a character... and the remaining ones have quite mixed feelings Not quite: all of them always let ? match a single invalid byte, but here we have a single byte that is invalid on its own, valid as part of a character, and appears in the string as part of that character. When processing \303\244, most shells don't process this as the single byte \303 followed by the single byte \244, they preprocess this so that by the time they actually check whether it matches, they just see the character U+00C4, so that if a pattern looks for \303 on its own, it will not be found. String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \243] [\243]] match match match match match match match \243] ? no match match match match match match match \243 ? match match match match match match match \243 [\243] match match match match no match no match error \243 [\243!] match match match match match match match \243] [\243!]] match match no match no match no match match no match \243] ?] match match no match no match no match no match no match \243] *] match match no match no match no match no match match The tests involving \243 are run in a Big5 environment. In Big5, \243\135 is the representation of β, a single valid character, even though \135 on its own is still the single character ]. Seem also a bit strange to me,... all shells match \243 against ? ... i.e. ? matches a single byte that is not a character... but later on it doesn't work aga
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 19/04/2022 01:52, Harald van Dijk via austin-group-l at The Open Group wrote: On 15/04/2022 04:57, Christoph Anton Mitterer wrote: On Fri, 2022-04-15 at 00:44 +0100, Harald van Dijk via austin-group-l at The Open Group wrote: If there is interest in getting this standardised, I can spend some more time on creating some hopefully comprehensive tests for this to confirm in what cases shells agree and disagree, and use that as a basis for proposing wording to cover it. I'd love to see that and if you'd actually do so, I'd kindly ask Geoff to defer any changes in the ticket #1564 of mine, until it can be said whether it might be possible to get that standardised. Very well, I will post tests and test results as soon I can make the time for it. Please see the tests and results here. Apologies for the HTML mail but this is hard to make readable in plain text. String Pattern dash, busybox ash, mksh, posh, pdksh glibc fnmatch bash bosh gwsh ksh zsh \303\244 [\303\244] no match match match match match match match \303\244 ? no match match match match match match match \303 [\303] match match match match match match match \303 ? match match match match match match match \303.\303\244 [\303].[\303\244] no match no match no match match match match match \303.\303\244 ?.? no match no match no match match match match match \303\303\244 [\303][\303\244] no match no match no match match match match match \303\303\244 ?? no match no match no match match match match match \303\244.\303 [\303\244].[\303] no match no match no match match match match match \303\244.\303 ?.? no match no match no match match match match match \303\244\303 [\303\244][\303] no match no match no match match match match match \303\244\303 ?? no match no match no match match match match match \303\244 \303* match match match match no match match no match \303\244 \303? match match match no match no match match no match \303\244 [\303]* match match match match no match match no match \303\244 [\303]? match match match no match no match match no match \303\244 *\204 match match match no match no match no match match \303\244 ?\204 match match match no match no match no match no match \303\244 *[\204] match match match no match no match no match no match \303\244 ?[\204] match match match no match no match no match no match \243] [\243]] match match match match match match match \243] ? no match match match match match match match \243 ? match match match match match match match \243 [\243] match match match match no match no match error \243 [\243!] match match match match match match match \243] [\243!]] match match no match no match no match match no match \243] ?] match match no match no match no match no match no match \243] *] match match no match no match no match no match match The tests involving \303 and \244 are run in an UTF-8 environment. In UTF-8,
Re: POSIX bind_textdomain_codeset(): some invalid codeset arguments
On 13/05/2022 14:28, Steffen Nurpmeso wrote: You again strip content of follow-up RFCs. In my previous message, I quoted your message that I replied to in full. I stripped absolutely nothing and do not appreciate your utterly false claim here. Retract it. Harald van Dijk
Re: POSIX bind_textdomain_codeset(): some invalid codeset arguments
On 12/05/2022 23:10, Steffen Nurpmeso wrote: Harald van Dijk wrote in : |On 12/05/2022 18:19, Steffen Nurpmeso via austin-group-l at The Open |Group wrote: |> Bruno Haible wrote in |> <4298913.vrqWZg68TM@omega>: |>|Steffen Nurpmeso wrote: |>|> ... |>|>| [.] "UTF-7"." |>|> |>|> That is overshoot. |>| |>|No. UTF-7 is invalid here because it produces output that is not NUL |>|terminated. See: |>| |>|$ printf 'ab\0' | iconv -t UTF-7 | od -t c |>|000 a b + A A A - |>|007 |>| |>|strlen() on such a return value makes invalid memory accesses. |>|You can convince yourself by running |>|$ OUTPUT_CHARSET=UTF-7 valgrind ls --help |> |> This is then surely bogus? UTF-7 is a normal single byte |> character set and is to be terminated like anything else. Nothing |> in RFC 2152 nor RFC 3501 if you want makes me think something |> else. | |RFC 2152's rules 1 and 3 only allow specifying the listed characters as |their ASCII form. All other characters, including U+, must be |encoded using rule 2. GNU iconv is doing what the RFC specifies here. No really, please. And please do not strip important content, I didn't think I did. You didn't read the RFC properly, I replied to show where and how the RFC specifies exactly what GNU iconv does, the rest of your message looks like it's based on the false assumption that the RFC specifies something other than what it does, which becomes irrelevant when that assumption is corrected. Looking in more detail, there is one thing I should have responded to. Included here. UTF-7. Heck, how about that, for example: LC_ALL=C printf 'ab\0' | iconv -f iso-8859-1 -t utf-16 | od -t c 000 \0 \0 a \0 b \0 \0 \0 Two leading NULs? This is not what GNU iconv prints at all, at least not on my system, which just uses the GNU version unmodified. Rather, it prints 000 377 376 a \0 b \0 \0 \0 010 That is, it includes a BOM, just like it showed in your SunOS output. Both the GNU iconv that is shipped as part of GNU libc 2.35, and the GNU iconv that is shipped as part of GNU libiconv 1.16, print this. Those are the current releases. If you are testing an older release, or a modified version, that is important information missing from your message. If you are seeing the leading null bytes in a current version, you may want to report this, including steps on how to get a GNU iconv that behaves this way. i am neither Chinese nor Russian, and especially not one of the other 7 billion that do not count. (I said surely bogus because i alone see the shiny light of having found give-me-five GNU iconv errors. Or even beyond that.) This makes absolutely zero sense. I am including it only to pre-empt you again claiming I am stripping important content. Cheers, Harald van Dijk
Re: POSIX bind_textdomain_codeset(): some invalid codeset arguments
On 12/05/2022 18:19, Steffen Nurpmeso via austin-group-l at The Open Group wrote: Bruno Haible wrote in <4298913.vrqWZg68TM@omega>: |Steffen Nurpmeso wrote: |> ... |>| [.] "UTF-7"." |> |> That is overshoot. | |No. UTF-7 is invalid here because it produces output that is not NUL |terminated. See: | |$ printf 'ab\0' | iconv -t UTF-7 | od -t c |000 a b + A A A - |007 | |strlen() on such a return value makes invalid memory accesses. |You can convince yourself by running |$ OUTPUT_CHARSET=UTF-7 valgrind ls --help This is then surely bogus? UTF-7 is a normal single byte character set and is to be terminated like anything else. Nothing in RFC 2152 nor RFC 3501 if you want makes me think something else. RFC 2152's rules 1 and 3 only allow specifying the listed characters as their ASCII form. All other characters, including U+, must be encoded using rule 2. GNU iconv is doing what the RFC specifies here. Cheers, Harald van Dijk
Re: When can shells remove "known" process IDs from the list?
On 29/04/2022 19:38, Robert Elz via austin-group-l at The Open Group wrote: Date:Fri, 29 Apr 2022 15:39:23 +0100 From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20220429143923.GA22521@localhost> | It also appears that dash still implements remove-before-prompting. pdksh does as well. However, pdksh is no longer maintained and the maintained shells that derive from pdksh have changed this. busybox ash and my shell do as well, but both are derived from dash and have merely retained dash's behaviour. Does anyone not? bash does not. bosh does not. ksh does not. mksh does not. posh does not. yash does not. zsh does not. You can test this by doing true & wait $!; echo $? This should print 0. Then do the same, except with the first command changed to false &. That should print 1. For the record, in my shell, a fork of dash, I have retained the dash behaviour for now because it is unclear to me what the expected behaviour is if a later background job gets the same PID as a previous background job. I consider the dash behaviour a bug, but do not want to fix it in a way that introduces another bug. Cheers, Harald van Dijk
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
[accidentally replied privately, re-sending to list] On 22/04/2022 10:54, Geoff Clare via austin-group-l at The Open Group wrote: Harald van Dijk wrote, on 15 Apr 2022: For the most part(*), those shells that support locales appear to already be in agreement that single bytes that do not form a valid multi-byte character are interpreted as single characters that can be matched with *, with ?, and with those single bytes themselves. Shells are not in agreement on whether such single bytes can be matched with [...], nor in those shells where they can be, whether multiple bracket expressions can be used to match the individual bytes of a valid multi-byte character. Shells are not the only issue here. Pattern matching or expansion is also performed by find, pax, fnmatch() and glob(), and there is no agreement between those. E.g. GNU find -name does not match such bytes with * or ?, whereas Solaris find does, and the glibc fnmatch() returns an error (not just "no match") if the string to be matched does not consist of valid characters. Good point that those need to be checked too. However, what you describe is only what older versions of GNU fnmatch() did. It was regarded as a bug, and fixed. Cheers, Harald van Dijk
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 15/04/2022 04:57, Christoph Anton Mitterer wrote:
On Fri, 2022-04-15 at 00:44 +0100, Harald van Dijk via austin-group-l
at The Open Group wrote:
Hmm, I would.
I like that :-D This would have been the preferred alternative I've
asked for to look at, in the ticket.
Shells
are not in agreement on whether such single bytes can be matched with
[...], nor in those shells where they can be, whether multiple
bracket
expressions can be used to match the individual bytes of a valid
multi-byte character.
The cases with [...] only come up when scripts themselves use
patterns
that are not valid character strings
You mean in the lexical locale?
I do not, but interesting question. I am one of the few, if not only,
shell authors that actually implemented "Changing the value of LC_CTYPE
after the shell has started shall not affect the lexical processing of
shell commands in the current shell execution environment or its
subshells" rule. Even I did not to apply this to pattern matching. The
lexical locale, the locale used for lexing, is only used for lexing,
i.e. for recognising tokens, not to how those tokens are then
interpreted later on. If locale comes into play for that, as it does in
pattern matching, it is the then-current value of LC_CTYPE that comes
into play, as it does in other shells.
they are unlikely to affect
existing scripts and I imagine there is not much harm in leaving
those
unspecified.
It should however be clearly described that behaviour in this field is
undefined, perhaps with some "future directions" that this might change
some day.
I prefer explicit over implicit as well myself. Perhaps it does not even
need to be undefined though, perhaps unspecified with a few limited
options is good enough. I am not sure at this time whether that is feasible.
As for future directions, no opinion on that from me.
The cases with * and ? do come up in existing scripts, but
if shells are in agreement as they appear to be, there is no need to
coordinate with shell authors on whether they would be willing to
change
their implementations, it is possible to change POSIX to describe the
shells' current behaviour.
Well but it's not only * and ? ... it's also a single character
matching that character in a byte string that contains bytes or
sequences thereof which do not form any valid character ... both before
or after that character to be matched.
Yes, I did mention those earlier on in my message but forgot to repeat
it here. It's where shells also appear to be in agreement, except in the
same corner case that also applies to [...] where an invalid byte in a
pattern is used to match part of a valid character in the string.
And since pattern matching notation isn't just used for matching alone,
but e.g. also for string manipulation in parameter expansion (e.g.
"${foo%.}" case)... these shells would also need to agree how to handle
that, wouldn't they?
I would not think this should be a special case: «${foo%.}» should strip
a trailing «.» in exactly those cases where the shell considers foo to
match the pattern «*.». However, I can see value in doing some extra
tests to verify that this matches what shells do.
If there is interest in getting this standardised, I can spend some
more
time on creating some hopefully comprehensive tests for this to
confirm
in what cases shells agree and disagree, and use that as a basis for
proposing wording to cover it.
I'd love to see that and if you'd actually do so, I'd kindly ask
Geoff to defer any changes in the ticket #1564 of mine, until it can be
said whether it might be possible to get that standardised.
Very well, I will post tests and test results as soon I can make the
time for it.
Cheers,
Harald van Dijk
Re: [Issue 8 drafts 0001564]: clariy on what (character/byte) strings pattern matching notation should work
On 15/04/2022 00:03, Robert Elz via austin-group-l at The Open Group wrote: Date:Thu, 14 Apr 2022 09:42:37 +0100 From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20220414084237.GA15370@localhost> | That is how things are at present. The suggested changes just make it | explicit. Yes, I know, but that's what I am suggesting that we not do in this one case. | Do you have an alternative proposal? Only to the extent of "do nothing". I am certainly not suggesting that we attempt to solve the problem. Hmm, I would. Except perhaps it might be worth adding something to the Rationale (but about what, ie: where there, I have no idea) along the lines of: It is often unclear whether a string is to be interpreted as characters in some locale, or as an arbitrary byte string. While it would have been possible to arbitrarily make the various cases more explicit, or explicitly unspecifried, it was considered better, in this version of to make no changes, as it is believed that much additional work is required to enable a standards-worthy specification possible. This work is beyond the scope of this standard. The problem I see, is that any specification at all of any of this, allows implementors to just say "that is what posix requires" and do nothing at all, where we really need some innovation, by someone who actually understands the issues and how to deal with them in a rational way - or at least who can come up with some kind of plan, and without any possibility of being considered a non-conformant implementation because of it. For the most part(*), those shells that support locales appear to already be in agreement that single bytes that do not form a valid multi-byte character are interpreted as single characters that can be matched with *, with ?, and with those single bytes themselves. Shells are not in agreement on whether such single bytes can be matched with [...], nor in those shells where they can be, whether multiple bracket expressions can be used to match the individual bytes of a valid multi-byte character. The cases with [...] only come up when scripts themselves use patterns that are not valid character strings, they are unlikely to affect existing scripts and I imagine there is not much harm in leaving those unspecified. The cases with * and ? do come up in existing scripts, but if shells are in agreement as they appear to be, there is no need to coordinate with shell authors on whether they would be willing to change their implementations, it is possible to change POSIX to describe the shells' current behaviour. If there is interest in getting this standardised, I can spend some more time on creating some hopefully comprehensive tests for this to confirm in what cases shells agree and disagree, and use that as a basis for proposing wording to cover it. Cheers, Harald van Dijk (*) The notable exception here is yash, which internally processes strings as wide strings and cannot handle any string that cannot be converted to a wide string. This was already said to be contrary to what POSIX intends in other cases.
Re: wait and stopped jobs
On 28/02/2022 23:20, Robert Elz via austin-group-l at The Open Group wrote:
It is worth pointing out though that testing this in interactive shells
is somewhat of a weird case - it should probably be done in a script
instead. That's because of the effect of the (implicit) jobs command
before each PS1 output, that should have reported the background job as
stopped when the Ctl-Z was sent, before the wait command is issued.
I know, that was intentional for two reasons.
1: Multiple shells turn out to have logic that amounts to roughly
while (job->state == RUNNING) {
wait(job->pid);
/* ... */
}
where a simpler test leaves job->state as RUNNING even though the job
has stopped already, simply because the shell has not yet picked up on
the state change, and so the shell actually does end up waiting for the
stopped job. Sort of, anyway. I did not want to complicate the test with
such matters.
2: Sending SIGCONT from another process could well be a wrong thing to
do. An interactive shell is responsible for switching the TTY
foreground process group as jobs are started, stopped, resumed. A
SIGCONT from another process would just send SIGCONT, nothing else, as
if the job was resumed by the bg command. Given that the child process
would have been resumed and the shell would be waiting for that child
process to finish, resuming as if by the fg command would likely be more
appropriate, but that is not something an external process is capable of
doing, that is something only the shell itself can do.
Thanks all for the answers, all. If this is something where the standard
says shells must behave one way, but it remains unclear whether it says
so intentionally and there may be good reasons why implementations
don't, I will need to think carefully about what to do in my shell.
Cheers,
Harald van Dijk
wait and stopped jobs
Hi, wait is specified as: If one or more pid operands are specified that represent known process IDs, the wait utility shall wait until all of them have terminated. In no shell does the wait utility appear to behave this way. To test this, in an interactive shell, I tried: sleep 10 & fg wait $! kill -l $? The results for the shells I can test right now are: bash, busybox ash, dash, gwsh, yash: no wait, print TSTP bosh:no wait, print EXIT ksh93: resume sleep command, print EXIT mksh, pdksh: no wait, print 0 zsh: resume sleep command, print 255 Based on the specification, I would think the correct behaviour is to wait indefinitely until some other process sends SIGCONT to the sleep command. However, as no shell appears to behave this way I am wondering if I am not simply misreading the specification. Am I? If not, does waiting indefinitely match the intent? Cheers, Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 28/01/2022 01:49, Christoph Anton Mitterer wrote: And hasn't it always been the case that stdout/in can contain arbitrary binary data? This is done by gazillions of tools, including such standardised by POSIX (cat, tr, even printf via \ooo sequences). Yes, but it has also been the case that not all utilities can process arbitrary binary data. There are many utilities that specify input must be a text file. One way or another it should be made clear whether that is a restriction that also applies to command substitutions. Cheers, Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 28/01/2022 01:48, Christoph Anton Mitterer wrote:
On Thu, 2022-01-27 at 15:18 +, Harald van Dijk via austin-group-l
at The Open Group wrote:
The benefit of this that when the
shell's locale changes, variables still hold their original text (as
opposed to their original bytes).
But doesn't that by itself already violate POSIX?
There is "2.5.3 Shell Variables", which AFAIU says that setting
LANG/LC_* must take effect during the shell runtime.
The way it works in yash, it does take effect at runtime, just not in
the same way it does in other shells.
LC_CTYPE says:
"Determine the interpretation of sequences of bytes of text data as
characters (for example, single-byte as opposed to multi-byte
characters), which characters are defined as letters (character class
alpha) and characters (character class blank), and the behavior
of character classes within pattern matching. Changing the value of
LC_CTYPE after the shell has started shall not affect the lexical
processing of shell commands in the current shell execution environment
or its subshells. Invoking a shell script or performing exec sh
subjects the new shell to the changes in LC_CTYPE."
=> lexical scanning of the current script stays
=> everything else, changes
including e.g. printf, or things like ${#var}, ${var##}, etc.
Right?
${#var}, ${var##}, etc. are supposed to work at the character level. In
other shells that internally hold values as byte strings, yes, LC_CTYPE
needs to be considered here. What the other shells effectively do is
convert to a wide string (not necessarily holding the full wide string
in memory). yash normally converts the wide strings to multibyte strings
as needed. Here, that would mean converting the wide string to a
multibyte string and immediately back a wide string again, which can be
optimised by just acting on the wide string directly. That said...
So if the shell would keep holding it's original text/characters.. this
wouldn't work (or the shell would need to convert every time)?
...for anything that cannot be done by keeping the strings as wide
strings, including calling any non-builtin command, yash does convert
every time the values are used according to the current locale.
Cheers,
Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 28/01/2022 01:49, Christoph Anton Mitterer wrote:
On Thu, 2022-01-27 at 17:07 +, Harald van Dijk via austin-group-l
at The Open Group wrote:
That is not what POSIX says. It says "The value of an environment
variable is a string of characters" (8.1 Environment Variable
Definition), and "character" is defined as "a sequence of one or more
bytes representing a single graphic symbol or control code" (3
Definitions), with a note that says it corresponds to what C calls a
multi-byte character. Environment variables are not specified to
allow
arbitrary bytes.
And later in the same chapter it says:
"These strings have the form name=value; names shall not contain the
character '='. For values to be portable across systems conforming to
POSIX.1-2017, the value shall be composed of characters from the
portable character set (except NUL and as indicated below)."
BUT...
further down it says:
"The values that the environment variables may be assigned are not
restricted except that they are considered to end with a null byte and
the total space used to store the environment and the arguments to the
process is limited to {ARG_MAX} bytes."
That is a good point. I don't have an answer here, this looks to me like
a contradiction in the standard.
Cheers,
Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 27/01/2022 17:43, Geoff Clare via austin-group-l at The Open Group wrote: Harald van Dijk wrote, on 27 Jan 2022: On 27/01/2022 12:44, Geoff Clare via austin-group-l at The Open Group wrote: Christoph Anton Mitterer wrote, on 26 Jan 2022: 3) Does POSIX define anywhere which values a shell variable is required to be able to store? I only found that NUL is excluded, but that alone doesn't mean that any other byte value is required to work. Kind of circular, but POSIX clearly requires that a variable can be assigned any value obtained from a command substitution that does not include a NUL byte, and specifies utilities that can be used to generate arbitrary byte values, therefore a variable can contain any sequence of bytes that does not include a NUL byte. Is it really clear that POSIX requires that? The fact that it refers to "characters" of the output implies the bytes need to be interpreted as characters according to the current locale, which is a process that can fail. The only relevant uses of "character" I can see are part of the phrase " character". Since is required to be a single-byte character, and the byte that encodes it is not allowed to be part of any other character, changing the text to " byte" would not make any difference to the requirements. I have to disagree. The use of " character" to me clearly means that the output of the command is processed as a sequence of characters, as opposed to a sequence of bytes. Implementations may (and likely will) implement this by treating it as a sequence of bytes when they can prove that this is equivalent, but that is an optimisation, not what POSIX specifies. In a UTF-8 locale, if a command outputs the bytes 0x80 and 0x0A, does it end in a character? I say it is the same as asking if it ends in the M_PI constant: we are not dealing with a sequence of floating-point values, so the question does not make sense and has no answer. Cheers, Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 27/01/2022 16:06, Chet Ramey via austin-group-l at The Open Group wrote: Wow, that seems like a bug. Environment variables can contain sequences of arbitrary non-NULL bytes, and, as long as the portion before the `=' is a valid NAME, the shell is required to create a variable with the remainder of the string as its value and pass it to child processes in the environment. That is not what POSIX says. It says "The value of an environment variable is a string of characters" (8.1 Environment Variable Definition), and "character" is defined as "a sequence of one or more bytes representing a single graphic symbol or control code" (3 Definitions), with a note that says it corresponds to what C calls a multi-byte character. Environment variables are not specified to allow arbitrary bytes. Cheers, Harald van Dijk
Re: how do to cmd subst with trailing newlines portable (was: does POSIX mandate whether the output…)
On 27/01/2022 12:44, Geoff Clare via austin-group-l at The Open Group wrote: Christoph Anton Mitterer wrote, on 26 Jan 2022: 3) Does POSIX define anywhere which values a shell variable is required to be able to store? I only found that NUL is excluded, but that alone doesn't mean that any other byte value is required to work. Kind of circular, but POSIX clearly requires that a variable can be assigned any value obtained from a command substitution that does not include a NUL byte, and specifies utilities that can be used to generate arbitrary byte values, therefore a variable can contain any sequence of bytes that does not include a NUL byte. Is it really clear that POSIX requires that? The fact that it refers to "characters" of the output implies the bytes need to be interpreted as characters according to the current locale, which is a process that can fail. In at least one shell (yash), bytes that do not form a valid character are discarded, which makes sense since yash internally stores variables etc. as wide strings. The benefit of this that when the shell's locale changes, variables still hold their original text (as opposed to their original bytes). However, there is no (standard) way to convert invalid multibyte characters to invalid wide characters in such a way that they can be back-converted to the original invalid multibyte characters. Should I take your comment as saying it is fundamentally invalid for shells to internally store text as wide strings? If so, it would be useful if POSIX were to explicitly require this. yash is written based on the POSIX specification; the fact that they implemented it like this is a clear indication that such a requirement was not clear at all to them. Cheers, Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001558]: require [^...] in addition to [!...] for bracket expression negation
On 24/01/2022 08:51, Austin Group Bug Tracker via austin-group-l at The Open Group wrote: The following issue has been SUBMITTED. == https://www.austingroupbugs.net/view.php?id=1558 == [...] All of bash, zsh, yash, dash, BSDs (except those like OpenBSD, MirBSD that use pdksh) allow [^...]. Most fnmatch() implementations do (including on OpenBSD making it a discrepancy between sh and fnmatch()/find...). Desired Action: Require [^...] to negate the bracket expression in addition to [!...] in issue8, or at least make it a "future direction" that it *will* be required in a future version, so as to at last remove that unnecessary discrepancy between regexp and shell pattern syntax. This is highly misleading. Yes, all of those allow [^...] but not all of them treat the ^ in it as negation. Whether they treat it as negation is what's relevant, not whether they allow it. Cheers, Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001440]: Calling `system("-some-tool")` fails (although it is a valid `sh` command)
On 31/10/2021 01:21, Wayne Pollock via austin-group-l at The Open Group wrote: Is it guaranteed that on conforming systems nohup (and friends) must not accept or delete the first "--"? For the example to work, nohup must not discard the "--". But might it? For context, the example was . nohup should be, and as far as I know, is required to support invocations as , treating that first <--> as the end of the options. The GNU extension that options and the end-of-options indicator can also follow operands is as far as I know not supported by POSIX in the first place (Utility Syntax Guideline 9: "All options should precede operands on the command line.") and accordingly disabled in GNU utilities when invoked in POSIX mode, so in any POSIX utility, if the arguments start with sh -c --, I think the fact that sh can only be interpreted as an operand already means that -c and -- must also be interpreted as operands. Cheers, Harald van Dijk
Re: Adding %n$ conversions to the printf utility (printf(1))
On 10/09/2021 22:54, Joerg Schilling via austin-group-l at The Open Group wrote: "Harald van Dijk via austin-group-l at The Open Group" wrote: Either the distinction matters or it doesn't. If it matters, then it was important to switch back to talk about what O?uz wrote. If it doesn't matter, then it should not be a problem that I switched back to talk about what O?uz wrote. It may be that the distinction doesn't matter. If it helps with understanding, I will answer again based on the integer version. O?uz's suggested output, translated to the integer version, would be (please correct me if I am misstating anything here): $ printf '%3$d %1$d\n' 1 2 3 4 0 1 0 3 This is not correct, the correct output is: printf '%3$d %1$d\n' 1 2 3 4 3 1 0 4 so please read the documentation, e.g. from: http://schilytools.sourceforge.net/man/man1/bosh.1.html play a bit with that feature and rethink what you wrote... I never said that output was what any implementation of printf would generate, I said that that output was what Oğuz suggested printf _should_ generate. That's why I wrote "Oğuz's suggested output" in that message. You, meanwhile, said that what Oğuz suggested printf _should_ do is what it _did_ do. That is utterly false and you still have not retracted that false claim. I do not know whether you actually believe what you are writing at this point. I do know there is no point in continuing this because you are misinterpreting them to such a point that communication just completely breaks down. I do not know whether this is intentional. If it is, it is an outright form of dishonesty. I really hope it is not. If it is not, it is still a giant waste of everyone's time. Either way, shame on you.
Re: Adding %n$ conversions to the printf utility (printf(1))
On 10/09/2021 22:14, (Joerg Schilling) wrote: Harald van Dijk wrote: Not correct: ksh93 prints the same as bosh Indeed, something went wrong with the copying there. > and pleasew note that my example is > using an integer format instead of a string format. Irrelevant. You wrote: > That is exactly what existing implementations do. Well, I modified the example to make the result easier to understand, since the POSIX standard requires an empty string to be printed for a missing string argument and a 0 for a missing integer argument. Either the distinction matters or it doesn't. If it matters, then it was important to switch back to talk about what Oğuz wrote. If it doesn't matter, then it should not be a problem that I switched back to talk about what Oğuz wrote. It may be that the distinction doesn't matter. If it helps with understanding, I will answer again based on the integer version. Oğuz's suggested output, translated to the integer version, would be (please correct me if I am misstating anything here): $ printf '%3$d %1$d\n' 1 2 3 4 0 1 0 3 You wrote that what Oğuz suggested is what existing implementations do. As your tests have shown, even for the integer version, it is not.
Re: Adding %n$ conversions to the printf utility (printf(1))
On 10/09/2021 21:34, (Joerg Schilling) wrote: Harald van Dijk wrote: No, it isn't. The second command prints ksh93: c a d zsh: printf: 3: argument specifier out of range c a bosh: c a d Not correct: ksh93 prints the same as bosh Indeed, something went wrong with the copying there. > and pleasew note that my example is > using an integer format instead of a string format. Irrelevant. You wrote: > That is exactly what existing implementations do. in response to Oğuz's > That is, the empty string would be taken for the third argument. That can only be taken to be about Oğuz's example, not your modified example which had not even been posted yet.
Re: Adding %n$ conversions to the printf utility (printf(1))
On 10/09/2021 20:27, Joerg Schilling via austin-group-l at The Open Group wrote: "O?uz via austin-group-l at The Open Group" wrote: Wouldn't it be more useful if, in printf(1), `n' in `%n$' referred to the nth argument in the current set of arguments being processed? For example, the command: printf '%2$s %1$s\n' a b c d would print: b a d c And the command: printf '%3$s %1$s\n' a b c d would print: a c That is, the empty string would be taken for the third argument. That is exactly what existing implementations do. No, it isn't. The second command prints ksh93: c a d zsh: printf: 3: argument specifier out of range c a bosh: c a d In no shell that I can find does it print a c
Re: shell: swapping var values in one command, plus some here doc stuff
On 08/09/2021 08:15, Oğuz via austin-group-l at The Open Group wrote: Sorry for butting in, but according to the standard, is there really a syntax error in the following? sh -c ': << do | for x in xxx do do echo $x done' busybox sh, dash, gwsh, netbsd sh, and freebsd sh complain about a missing `done'. This is not about 'do' as a delimiter, but about a heredoc right after the items in a for command. This fails too: sh -c ': << x | for x in xxx x do echo $x done' It results in an error about a missing 'do'. And this works, but should not: sh -c ': << x | for x in xxx do echo $x x done' Only speaking for gwsh, but yes, I consider this is a bug and will make sure to fix it. What's happening is a missing check for pending heredocs, causing the heredoc to only start on line 3, rather than on line 2 as it should. There is nothing that I am aware of in POSIX that allows this. Cheers, Harald van Dijk
Re: shell: swapping var values in one command, plus some here doc stuff
On 03/09/2021 13:23, Robert Elz via austin-group-l at The Open Group wrote: and ksh93 did: ksh93 $ cat < It looks like ksh93 fails to properly handle backslash-newline pairs that occur during determination of whether the current line is the heredoc delimiter, so any backslash-newline after a prefix of the heredoc delimiter gets mishandled. The way the first backslash-newline gets mishandled then means the second backslash-newline occurs in a context where the shell has already determined the current line is not the heredoc delimiter, so that one does get processed correctly. That wouldn't be some unexpected interpretation of any POSIX rule, that would simply be a bug. Cheers, Harald van Dijk
Re: $? in a simple command with no command name
On 01/09/2021 18:48, Robert Elz via austin-group-l at The Open Group wrote: Date:Wed, 1 Sep 2021 16:38:00 +0300 From:"=?UTF-8?B?T8SfdXo=?= via austin-group-l at The Open Group" Message-ID: | true | a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2` | echo $? $a $b | Now, I wonder, what did I miss? That $? (the exit status) is defined only when a command completes. What it is in the middle of a command execution isn't specified. $? is defined as "Expands to the decimal exit status of the most recent pipeline (see Pipelines)." During the a=$? and b=$? assignments, the most recent pipeline in that shell execution environment was 'true' at line 1. Is there any statement that overrides the general definition to explicitly make this unspecified? If not, the general definition applies and $? must expand to 0 both times it appears on line 2. Cheers, Harald van Dijk
Re: $? in a simple command with no command name
On 01/09/2021 17:22, Geoff Clare via austin-group-l at The Open Group wrote: Harald van Dijk wrote, on 01 Sep 2021: The second problem is the redirection. Based on the above, command substitutions in a redirection are supposed to affect the exit status just as any other command substitution, but the standard says: 3. Redirections shall be performed as described in Redirection. 4. Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value. This means the command substitution in your redirection is supposed to be performed prior to that in the variable assignment. After the list, the standard says: In the preceding list, the order of steps 3 and 4 may be reversed if no command name results from step 2 or if the command name matches the name of a special built-in utility Ah, thank you. That means the outputs "1 0 0" and "2 0 0" are both equally correct. I assume the outputs "1 2 1" and "2 0 1" do indicate shell bugs. Cheers, Harald van Dijk
Re: $? in a simple command with no command name
Hi, On 01/09/2021 14:38, Oğuz via austin-group-l at The Open Group wrote: Consider the following: true a=$? b=`exit 1` b=$? >`echo /dev/null; exit 2` echo $? $a $b Having read the relevant sections of the standard a couple times, I would expect the output to be `1 0 0'; I agree. but that's not the case with most shells. Below are what each shell I have on my computer output for that script. ? a b bash2 0 1 bosh2 0 0 dash1 0 0 ksh 2 0 1 mksh1 0 0 yash2 0 0 zsh 1 2 1 Now, I wonder, what did I miss? Where does it say in the standard that the value of parameter `?' is affected by redirections, and that its value depends on assignments that appear earlier in the same command? There are two problems here. The first is the value of $? during expansion. The standard says: If there is no command name, but the command contained a command substitution, the command shall complete with the exit status of the last command substitution performed. Otherwise, the command shall complete with a zero exit status. Some shells implement this by modifying $? as soon as a command substitution finishes. As all command substitutions are processed in order, the end result is that $? is based on the last command substitution performed. However, there is nothing that allows $? to be modified earlier, so this looks like a bug (#1) in those shells. The second problem is the redirection. Based on the above, command substitutions in a redirection are supposed to affect the exit status just as any other command substitution, but the standard says: 3. Redirections shall be performed as described in Redirection. 4. Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value. This means the command substitution in your redirection is supposed to be performed prior to that in the variable assignment. Some shells do not do this in the specified order, which looks like a bug (#2) in those shells. This can be confirmed by running under 'set -x'. Bug #1 leads to the output "1 2 1". Bug #2 leads to the output "2 0 0". Bug #1 & bug #2 combined leads to the output "2 0 1". Cheers, Harald van Dijk
Re: utilities and write errors
On 04/07/2021 21:54, Robert Elz wrote: Date:Sun, 4 Jul 2021 19:26:25 +0100 From:Harald van Dijk Message-ID: <40fc0b0c-364f-8ff8-0613-a76c887d4...@gigawatt.nl> | P.S.: The fact that the underlying file descriptor of standard output is | fd 1 rather than some other fd is specified under "stderr, stdin, stdout | - standard I/O streams", look for "STDOUT_FILENO", but I think that is | ultimately no longer relevant. But for interest, what section are you referring to, I don't have a (useful) PDF grep tool I was looking at the HTML version. Aside from its search function, you can find it there under System Interfaces (top left frame), then System Interfaces (bottom left frame), then stdout (bottom left frame). Cheers, Harald van Dijk
Re: utilities and write errors
On 04/07/2021 18:32, Robert Elz via austin-group-l at The Open Group wrote: Date:Thu, 1 Jul 2021 11:45:40 +0100 From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20210701104540.GA4023@localhost> | it just requires pwd to write the directory to file descriptor 1. Where does it say that? I think the definition of "write", in Base Definitions, tries to say so, but I also think the definition is wrong. 3.449 Write To output characters to a file, such as standard output or standard error. Unless otherwise stated, standard output is the default output destination for all uses of the term "write''; see the distinction between display and write in Display. Note the "to a file", not "to a stream". If an implementation uses stdio to achieve a write, the write has only completed once the stream has been flushed: until that happens, no file access has been performed. However, if that interpretation is correct, the rest of the definition is wrong, as standard output and standard error are streams, not files, so the definition should read something like > 3.449 Write > > To output characters to a file, such as the files associated with > standard output or standard error. Unless otherwise stated, the file > associated with standard output is the default output destination for > all uses of the term "write''; see the distinction between display and > write in Display. Alternatively, "write" could be defined in a way such that writing to a stream is defined as writing to the file associated with that stream. Cheers, Harald van Dijk P.S.: The fact that the underlying file descriptor of standard output is fd 1 rather than some other fd is specified under "stderr, stdin, stdout - standard I/O streams", look for "STDOUT_FILENO", but I think that is ultimately no longer relevant.
Re: utilities and write errors
On 30/06/2021 16:56, Chet Ramey via austin-group-l at The Open Group wrote: On 6/30/21 11:49 AM, Joerg Schilling via austin-group-l at The Open Group wrote: Erm, yes. For some reason, I assumed the OP wrote &> instead of >& which have the same meaning in GNU bash (but &> is the parse-trouble one even if the bash manpage actively recommends it). I guess their ?~>&? confused me. My point of _please_ using ?>file 2>&1? instead is still valid, ofc. BTW: I would not call it a hard parse error but a semantic problem, since the standard only mentions numbers after >& It does not. The redirection is specified as `[n]>&word'. [...] mirabilos wrote that it was a misreading of >& as &>. For &>, it is a different story: echo hello &>/dev/null In POSIX sh, this runs 'echo hello' in the background, and independent of that, opens /dev/null for writing and immediately closes it, unless this is explicitly made unspecified in some section I have missed. In bash, even in POSIX mode, this runs 'echo hello' with both stdout and stderr redirected to /dev/null. Cheers, Harald van Dijk
Re: execve(2) optimisation for last command
On 15/04/2021 21:36, Martijn Dekker via austin-group-l at The Open Group wrote: My question: why is this? I would have thought that a script is a script and that it should make no essential difference whether a script is taken from a -c argument or loaded from a file. What makes the optimisation appropriate for one but not the other? The parsing of script files and -c arguments is different and is meant to be different. With -c, the whole command string is known and parsed in advance. With script files, in general, only as much is read as necessary to parse the next command, that command is executed, and so on. With -c, all shells already know at the time that a command is executed whether it is the last command the shell will execute. When a script file is executed, in the general case, the shell not only does not know, it cannot know when a command is the last command in the script file. The command will almost always be followed by a newline, and determining whether that newline is the last byte of the file requires trying to read at least one more byte. Trying to read at least one more byte is problematic if the file is non-seekable so that the shell cannot restore the state to what it was before that read (needed when the shell is reading from standard input and the fd is shared with other processes), and is problematic if the read may block. In specific cases, the shell may be able to detect that these issues cannot be a problem. In those specific cases, the optimisation should be perfectly valid and any reasons why it should or should not be performed are not of a technical nature. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 13/04/2021 14:43, Chet Ramey wrote: On 4/13/21 5:16 AM, Harald van Dijk via austin-group-l at The Open Group wrote: Please note again that POSIX's Command Search and Execution doesn't say "continue until execve() doesn't fail". It says "Otherwise, the command shall be searched for using the PATH environment variable as described in XBD Environment Variables", and then what happens to the result of that search. It very clearly separates the search from the attempt to execute. The complicating factor is POSIX's definition of "executable file." You search "until an executable file with the specified name and appropriate execution permissions is found." An executable file is a "regular file acceptable as a new process image file by the equivalent of the exec family of functions." Good spot. We know that it is intended for command search and execution to also find shell scripts that are then interpreted by the shell, shell scripts that execve() cannot process, and all shells implement this as intended rather than as specified. This must be a defect in the current wording. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 12/04/2021 21:57, Robert Elz wrote:
Date:Mon, 12 Apr 2021 18:42:03 +0100
From:Harald van Dijk
Message-ID:
| No, not anything. It still has to be shell start-up activity.
And your definition of what is a shell start-up activity comes from ?
When words don't have a specific definition, we have to assume they are
used in their ordinary English meaning. That may in some cases not
always be clear and unambiguous. I can imagine some examples where some
people would reasonably say it counts as start-up activity, and others
would reasonably say it doesn't. That said...
| The starting a thread would be shell start-up activity. The actions
| performed on that thread while some other thread is running the script
| clearly aren't.
Nonsense. They are the result of the start-up activity. Nothing defines
how long that is allowed to take to complete, does it? Nothing specifies
what else is allowed to execute in parallel, does it?
...your hypothetical example is not one of them, IMO. There is nothing
reasonable about saying that an activity that continues throughout the
execution of the script counts as start-up activity. Your interpretation
would mean *all* activity can be considered start-up activity just by
virtue of being performed on a different thread where that different
thread was launched at shell startup, and then of course as that is just
an implementation detail invisible to a user of the shell, it doesn't
actually have to be performed on a different thread. If that were the
intent, why would the standard say "shell start-up activity" in the
first place?
| It cannot do this either, parsing the whole script in advance is not
| only not allowed
Of course it is allowed.
| (it would break the use of aliases defined in the script, at the least)
Apart from that being (IMO) a good thing, who cares? We're building a
bash table, if a few entries are added (words that turn out to be aliases)
that aren't needed, no problem. If some get missed, either from alias
expansions or the results of other expansions, then big deal, those can
be looked up and added later.
In the context of a shell, parsing a script already has a clear and
unambiguous meaning, and it seems you're using it to mean something
else. I take no responsibility for the miscommunication that happened
because of that.
Consider the case where the shell is reading from standard input,
standard input is a pipe, and the first line of standard input is
'exit'. With your idea, the shell cannot avoid reading beyond that first
line, and cannot restore the state to pretend it had not read beyond
that first line.
I am aware that many shells, including mine, do already read beyond that
first line. I consider that a bug in my shell.
| I am not convinced that that is the intent at all.
Recall, once upon a time there was no hash table, the code simply
walked $PATH trying exec until either it succeeded, or $PATH ran out.
The hash table was an invention to speed that up, not change the
algorithm.The results should be unchanged.
The language in the standard might not be written as clearly as
we might like, but it is attempting to describe that (aside from
the hash command, it should probably say nothing about the hash
table, or remembering results from previous scans of PATH at all).
Please note again that POSIX's Command Search and Execution doesn't say
"continue until execve() doesn't fail". It says "Otherwise, the command
shall be searched for using the PATH environment variable as described
in XBD Environment Variables", and then what happens to the result of
that search. It very clearly separates the search from the attempt to
execute. If shells combine the two in a way that changes the outcome,
that is a conformance bug in those shells. If what POSIX says does not
match historical practice, that is a separate issue, that should be
explained in the rationale at the very least.
| This code is shared between ordinary command execution and the exec
| builtin.
True, but that's irrelevant. When there is a command for the exec
builtin (that is, when it isn't being used just to make redirections
persist in the shell, etc) it behaves identically to command execution,
just skipping the alias/function/builtin tests. There is no difference
at all in how the PATH search happens. When exec was used the shell
doesn't actually fork, it just jumps directly to the child side of the
normally post-fork code, so the code fragment I should would really
start something like
if (exec_was_used || fork() == 0) {
(and of course, the real code would be saving the result from fork()
for the parent code, but we're not concerned with that stuff here).
We were talking about dash internals. In dash, the ordinary command
execu
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 12/04/2021 20:22, (Joerg Schilling) wrote: Harald van Dijk wrote: On 12/04/2021 12:47, (Joerg Schilling) wrote: Do you have a private variant og ksh93v? I get the same behavior from ksh88, the ksh93 from OpenSolaris and ksh93v. I don't. I was testing with ksh built from <https://github.com/att/ast/archive/93v.tar.gz>. I will try to figure out why I am getting different results from you. OK, it depends on usage, so you may have tested the "wrong" way: ksh93 -c 'PATH=/tmp/:$PATH; gcc' gcc: no input files ksh93 -ic 'PATH=/tmp/:$PATH; gcc' ksh93: gcc: not found [Datei oder Verzeichnis nicht gefunden] ksh93 -c 'PATH=/tmp/:$PATH; gcc; hash' ksh93[1]: gcc: not found [Datei oder Verzeichnis nicht gefunden] gcc=/tmp/gcc ksh93 -c 'PATH=/tmp/:$PATH; gcc; echo' ksh93[1]: gcc: not found [Datei oder Verzeichnis nicht gefunden] this looks strange... Ah, good catch, thanks. Multiple shells have an optimisation where a command invocation at the last step, when no traps are active, is implicitly performed by exec without a fork, as if the exec builtin had been used. ksh apparently has a mismatch between the command lookup in the two cases. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 12/04/2021 12:47, (Joerg Schilling) wrote: Harald van Dijk wrote: If they are mistakes, they are widespread mistakes. As hinted in the links, with PATH=/bin:/usr/bin, /bin/gcc and /usr/bin/gcc both existing as files with execute permission, but /bin/gcc as a text file containing #!/bad so that any attempt to execute it will fail, there are a lot of shells where command -v gcc returns /bin/gcc, but running gcc actually executes /usr/bin/gcc instead without reporting any error: this behaviour is common to bosh, dash and variants (including mine), ksh, and zsh. My tests show that ksh, bash, yash, mksh do not find gcc in that case. Huh. My tests with ksh were with 93v, it's possible different versions behave differently. Do you have a private variant og ksh93v? I get the same behavior from ksh88, the ksh93 from OpenSolaris and ksh93v. I don't. I was testing with ksh built from <https://github.com/att/ast/archive/93v.tar.gz>. I will try to figure out why I am getting different results from you. [...] I don't think command -v should do more, I think ordinary command lookup should do less. The behaviour of shells of continuing command lookup after a failed execve() is not supported by what POSIX says in "Command Search and Execution". Command lookup is supposed to stop as soon as "an executable file with the specified name and appropriate execution permissions is found" (per the referenced "Other Environment Variables", "PATH"). In my example that results in /bin/gcc. The shell should attempt to execute /bin/gcc, and once that fails, stop. Given that #!/bad causes an ENOENT, this seems to be a missinterpreation from you. The shell cannot distinct the missconfigured script from a missing file. It can. bash already does. You are correct that is not possible to check this just by looking at execve()'s error code, but there are other ways to handle this. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 12/04/2021 00:25, Robert Elz wrote: Date:Sun, 11 Apr 2021 22:27:19 +0100 From:Harald van Dijk Message-ID: <79b98e30-46ba-d468-153f-c1a2a0416...@gigawatt.nl> | Okay, but that is a technicality. The pre-seeding is only permitted at | startup time, No, what it says is "an unspecifiedshell start-up activity". "unspecified" means it can be anything. No, not anything. It still has to be shell start-up activity. Anything includes starting a thread which monitors what commands are about to be executed and loads the hash table just in time. Or one which populates the hash table with every possible command every tenth of a micro-second. Anything. It is unspecified. The starting a thread would be shell start-up activity. The actions performed on that thread while some other thread is running the script clearly aren't. | so cannot depend on the contents of the script. Of course, it can, the script is available at startup time of the shell, the startup activity can read the entire script, parse it, find all the command names and possible command names, and add them to the hash table. It cannot do this either, parsing the whole script in advance is not only not allowed (it would break the use of aliases defined in the script, at the least) but also impossible as command names need not be named literally inside the script. Alternatively, it can examine PATH and load every executable in every directory in PATH into the hash table. zsh (seems to) do something like the latter. This is something that I agree is valid for a shell to do. It does not make any fundamental difference. Incidentally, I only see this in zsh's interactive mode. I am not sure whether this depends on interactive mode directly, or on another option automatically turned on or off in interactive mode. | I want to say this is a theoretical concern, that there are no shells | where hash -r is implemented as doing anything other than clearing the | hash table. I cannot prove this but will be quite disappointed if any to | turn out to do something else. zsh comes close, it appears to empty the hash table on "hash -r", but do anything at all, and it fills up again. And I mean fills. And I understand that - if you're going to search the directories in PATH over and over again, every time a command is executed, better to read them once, and remember what they contain - no more useless I/O. (I vaguely recall deciding that zsh read as many directories as needed to find the command, and then stopped - getting a "command not found" would result in everything possible from PATH now being in the hash table.) Yes, that is exactly what it is doing. | > That is, find an entry for cmd in PATH for which exec() succeeds. | > Only fail if there is none. | | Yes, that is what dash is doing. The way PATH searches should be done. | Well, that is sort of what dash does. dash takes an extra integer that | specifies which PATH component was hashed and uses that as the starting | point for the search, I know. This is irrelevant here. If this algorithm doesn't produce the required results, that would be a bug, and like most bugs, if it is considered serious enough, it can be fixed. The important issue, is that the intent is to examine each element in PATH, until we get success from exec(), (or ENOEXEC with a file we're willing to treat as a script, and so exec a shell to interpret it). So, if there is a /bin/gcc that is "#!/bad" and a later one in path that is a real executable, we should exec the later one, right? I am not convinced that that is the intent at all. This code is shared between ordinary command execution and the exec builtin. The former needs no second PATH lookup if the hashing was done correctly, the latter does: no hashing happens for exec, as it would be useless. The code looks like the simplest way to shoehorn both into a single function based on the assumption that the first execve() in the ordinary command execution case would not fail. The fact that it can fail tells us nothing about what was intended to happen in such a case. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 22:05, Robert Elz wrote: Date:Sun, 11 Apr 2021 19:46:36 +0100 From:Harald van Dijk Message-ID: <9ab286f9-125d-55a4-a65f-08d4af04d...@gigawatt.nl> | Sure, that's why I then switched to a different example that did not | have an earlier "command -v" to point out how this leads to inconsistent | behaviour. But while it is possible to (at least probabilistically - it is a hash table after all, effectively a cache) ensure that an entry exists, it is not possible to ensure that one doesn't. Recall this part from POSIX (still 2.9.1.1 1. e. i.) Once a utility has been searched for and found (either as a result of this specific search or as part of an unspecified shell start-up activity), That is, a shell is permitted to pre-seed the hash table at startup time, and if allowed then, exactly when it happens between when main() of the shell is first called, and when a lookup for a command is actually done, is unknowable. That means it is OK for the shell to pre-seed the hash table for a command when the command name is seen, and then it will be there when the search for that command is done. Okay, but that is a technicality. The pre-seeding is only permitted at startup time, so cannot depend on the contents of the script. Replace gcc by any utility that is not hashed at startup and you will still have the same problem. Or, as you say, clear the hash table explicitly. Even hash -r (which removes everything) doesn't guarantee that everything isn't immediately replaced (with up to date values of course) before that command even finishes. Actually, if hashing commands is only allowed "as a result of this specific search or as part of an unspecified shell start-up activity", then after "hash -r" has executed, before a new command search has been performed, the hash table must be empty. I want to say this is a theoretical concern, that there are no shells where hash -r is implemented as doing anything other than clearing the hash table. I cannot prove this but will be quite disappointed if any to turn out to do something else. But all of this is really irrelevant, it is based upon a flawed assumption about what is happening (and even what should happen). What dash and the others, I presume, are doing, is not really the "subsequent command" thing (that was just an interesting argument to make), it is rather an implementation of the original Bourne shell strategy (pre hash table), which was, more or less (not this code, I don't write algol68, just a similar effect): [...] That is, find an entry for cmd in PATH for which exec() succeeds. Only fail if there is none. Yes, that is what dash is doing. The addition of the hash table should allow that algorithm to run faster (with the occasional problem when after a hash entry is created, someone inserts an entry earlier in PATH than it was before) but it should not normally change the outcome of that algorithm. Well, that is sort of what dash does. dash takes an extra integer that specifies which PATH component was hashed and uses that as the starting point for the search, but otherwise it is the same algorithm. So if PATH=/a:/b:/c and the hash table says x is found in /b, the search in the shell child will look for /b/x, and if that fails, /c/x. It will not search for /a/x unless the hash table is cleared. This does not seem useful to me. If the command is no longer present in /b, it should be checked in all PATH components. Commands may legitimately move from /usr/bin to /bin by system upgrades just as well as the other way around. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 21:17, shwaresyst wrote: The requirement explicitly specified behavior shall be implemented as specified takes priority. Some conforming script authors may simply want the first line to be a # IMPORTANT USAGE NOTE headline, or similar, not want a utility named "!!!" to be exec'd. If you are really saying that when POSIX says "If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.", it actually means the results are well-defined, you are either seriously deluded, or trolling. I cannot tell which and have no interest in wasting time figuring it out. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 17:50, Robert Elz wrote: Date:Sun, 11 Apr 2021 17:04:05 +0100 From:Harald van Dijk Message-ID: <92113e70-5605-10f4-8e57-47c9f64cd...@gigawatt.nl> | This only applies when a remembered location exists at all, though. Yes, but in the examples I showed, it did (you can see that from the output of the hash command before the attempt to execute cmd). It was put there by "command -v". I haven't checked again, but I think all shells do that. Sure, that's why I then switched to a different example that did not have an earlier "command -v" to point out how this leads to inconsistent behaviour. | Then, if you accept that, for consistency, "the shell shall repeat the | search" can only mean to repeat the full search and again stop at the | first file with execute permissions, as it would be batshit crazy to | have a shell that, when presented with "gcc; gcc", for the first gcc | issues an error because /bin/gcc cannot be executed, and for the second | gcc to find /usr/bin/gcc because /bin/gcc failed to execute. Actually, in my, and I suspect most, implementations, even the first will invoke the "subsequent" clause, as the (parent) shell first searches PATH to find the executable, and enters it in the hash table. Then it forks, and the child repeats the whole thing (after redirects etc have all been done). This one is the subsequent search, which starts out with what is already in the hash table (assuming the command was found at all) and then if that fails, goes ahead and looks for another. That is an implementation detail. As far as POSIX is concerned, there is only a single command search when a command is executed, so "a subsequent invocation" can only refer to the shell script attempting to execute the same command again at a later time. POSIX does not even require the shell to fork at all, the shell may use some other system-specific way of creating a new process. This isn't hypothetical, such other system-specific ways of creating new processes were the reason posix_spawn was added, and posix_spawn appears to be used by at least one shell (ksh). | I am pretty sure you are not suggesting that that is reasonable, Don't be too sure, I would not object to an implementation that did work the way you described, and I suspect most users wouldn't either. We're a pragmatic bunch, if something goes wrong the first time, and fixes itself the second time (and subsequently), people tend to be fairly happy. Not deliriously, just fairly... Ha, that's bad enough for an interactive shell, but for a non-interactive shell script that executes gcc and exits if it fails, retrying wouldn't even work. | I think that is easier to explain than the other way around, myself. | Suppose PATH is intentionally modified so that an uClibc-linked version | of GCC appears first in $PATH, but the user messed up, For almost everything we do, we can find instances where the results are sub-optimal. Throwing away everything where that could occur leaves us with almost nothing. The best way to avoid this would be to remove PATH completely (not revert to the Thompson shell fixed search path, but require all commands to be always specified by full pathname). I doubt that would be well received as a solution however. If it were a case of choosing your poison, then sure, but we do now have multiple benefits in this thread of the bash behaviour, and no scenario in which I am seeing the dash behaviour as clearly better. If possible, I will stick with choosing no poison. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 17:09, shwaresyst via austin-group-l at The Open Group wrote: No, it's not nonsense. The definition of comment has all characters, including '!', shall be ignored until newline or end-of-file being conforming. Then tokenization which might discover an operator, keyword or command continues. This precludes "#!" being recognized as any of those. There is NO allowance for '!' being the second character as reserved for implementation extensions. This is wrong on two counts. The first is that you're assuming that this will be interpreted by a shell. If execve() succeeds (and the #! line does not name a shell), it will not be interpreted by a shell at all, and the shell syntax for comments is irrelevant. The second is about what happens when it does get interpreted by a shell: POSIX allows shells to treat files starting with "#!" specially: "If the first line of a file of shell commands starts with the characters "#!", the results are unspecified." Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 16:33, Robert Elz wrote: Date:Sun, 11 Apr 2021 13:25:46 +0100 From:Harald van Dijk Message-ID: | > My tests show that ksh, bash, yash, mksh do not find gcc in that case. | | Huh. My tests with ksh were with 93v, it's possible different versions | behave differently. I see the same results as Joerg. I'm using ksh93u. Interesting. Will need to re-test with that later. [...] Note that POSIX says (this is from 8 D1.1 XCU 2.9.1.1 1. e. i.) Once a utility has been searched for and found (either as a result of this specific search or as part of an unspecified shell start-up activity), an implementation may remember its location and need not search for the utility again unless the PATH variable has been the subject of an assignment. Aside from the lack of mention of hash -r there, that much is fine. It goes on: If the remembered location fails for a subsequent invocation, the shell shall repeat the search to find the new location for the utility, if any. Note: "fails" not "utility is is not found at" or similar, and "the shell shall". What it means in these circumstances to "repeat the search to find the new location for the utility, if any" is less clear - but a reasonable interpretation (adopted by about half the shells) is that it should look through PATH, see if it can find a copy of the utility that does not fail to invoke, and invoke that one.Also note that it does not say that it is OK to replace the remembered location with that of the newly located command. This only applies when a remembered location exists at all, though. If no remembered location exists, the invocation is not a "subsequent invocation" and the paragraph does not apply. Then, if you accept that, for consistency, "the shell shall repeat the search" can only mean to repeat the full search and again stop at the first file with execute permissions, as it would be batshit crazy to have a shell that, when presented with "gcc; gcc", for the first gcc issues an error because /bin/gcc cannot be executed, and for the second gcc to find /usr/bin/gcc because /bin/gcc failed to execute. I am pretty sure you are not suggesting that that is reasonable, but I think that is a bad consequence of your interpretation of the wording. And if "shall repeat the search" does refer to the exact same search that was initially performed, then "Once a utility has been searched for and found [...] an implementation may remember its location" arguably applies to that repeated search as well, but that is less clear. You have asked questions about that later on. They are good questions to think about. I am not sure about those yet, so am skipping them for now. I agree with that. Nothing else is rationally possible, except failing to exec the command (like bash and the ksh's do), but it is hard to explain how failing to run a command when one that is runnable exists in $PATH, is a better outcome than running it. I think that is easier to explain than the other way around, myself. Suppose PATH is intentionally modified so that an uClibc-linked version of GCC appears first in $PATH, but the user messed up, the dynamic linker of uClibc is actually not yet installed, or is installed in the wrong location. It is clearly the user's intention to execute the uClibc-linked version, and attempting to execute that and reporting the error is what bash and others would do. Silently executing some other version that the user didn't want is, in my opinion, doing the user a disservice. (Disclaimer: I am not certain whether all shells would treat this exactly the same way as the '#!/bad' example.) Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 11/04/2021 13:02, Joerg Schilling via austin-group-l at The Open Group wrote: "Harald van Dijk via austin-group-l at The Open Group" wrote: If they are mistakes, they are widespread mistakes. As hinted in the links, with PATH=/bin:/usr/bin, /bin/gcc and /usr/bin/gcc both existing as files with execute permission, but /bin/gcc as a text file containing #!/bad so that any attempt to execute it will fail, there are a lot of shells where command -v gcc returns /bin/gcc, but running gcc actually executes /usr/bin/gcc instead without reporting any error: this behaviour is common to bosh, dash and variants (including mine), ksh, and zsh. My tests show that ksh, bash, yash, mksh do not find gcc in that case. Huh. My tests with ksh were with 93v, it's possible different versions behave differently. I am assuming that by "do not find gcc" you mean "do not find /usr/bin/gcc" here. bosh and dash execute the correct gcc binary, but still have the wrong script path in their hash after calling gcc. I believe what bosh and dash do is the best behavior. None of the known shells opens the file with "command -v something" and thus cannot know whether the content is a script, a useless #! script or even a binary for the wrong architecture. Earlier, you did not see the problem that prompted this thread, and now you say that the behaviour where command -v lookup does not match execution lookup is the best behaviour. I trust that you do see now the problem that prompted this thread: there is, in these shells at least, no reliable way to perform command lookup separate from execution. This is a result of the layering that has been introduced in the past 50 years of UNIX. If command -v should become able to do more, we would need to invent a way to execute _any_ utility (regardless of whether it is a binary or script) to execute in a harmless way without side-effects. I don't think command -v should do more, I think ordinary command lookup should do less. The behaviour of shells of continuing command lookup after a failed execve() is not supported by what POSIX says in "Command Search and Execution". Command lookup is supposed to stop as soon as "an executable file with the specified name and appropriate execution permissions is found" (per the referenced "Other Environment Variables", "PATH"). In my example that results in /bin/gcc. The shell should attempt to execute /bin/gcc, and once that fails, stop. This is what the other shells do, including bash, and what I intend to implement in mine. There is still a problem: only bosh and ksh could in therory add the right entry into the hash, since they are using vfork() and could report back the final result via shared memory. I have that probability in mind for bosh since I introduced vfork() support to bosh in 2014. That's an interesting thought. The approach taken by the other shells avoids the problem entirely and makes this unnecessary though. Cheers, Harald van Dijk
Re: [Shell Command Language][shortcomings of command utlity][improving robustness of POSIX shells]
On 10/04/2021 17:08, Robert Elz via austin-group-l at The Open Group wrote: Date:Sat, 10 Apr 2021 11:54:34 +0200 From:"Jan Hafer via austin-group-l at The Open Group" Message-ID: <15c15a5b-2808-3c14-7218-885e704cc...@rwth-aachen.de> | my inquiry is a question about the potential unexpected behavior of the | shell execution environment on names. It is related to shortcomings of | the command utility. I'm not sure I understand. I read the rest of the message, and I couldn't find anything really about any shortcomings, other than perhaps some mistakes in interpretation, and usage. If they are mistakes, they are widespread mistakes. As hinted in the links, with PATH=/bin:/usr/bin, /bin/gcc and /usr/bin/gcc both existing as files with execute permission, but /bin/gcc as a text file containing #!/bad so that any attempt to execute it will fail, there are a lot of shells where command -v gcc returns /bin/gcc, but running gcc actually executes /usr/bin/gcc instead without reporting any error: this behaviour is common to bosh, dash and variants (including mine), ksh, and zsh. Cheers, Harald van Dijk
Re: sh: Assignments to PATH and XCU 2.9.1
On 19/03/2021 16:41, Robert Elz via austin-group-l at The Open Group wrote:
In note 5287 to bug 1460 (an innocuous change to the hash command)
( https://austingroupbugs.net/view.php?id=1460#c5287 )
Geoff said:
[...] that statement about PATH is pointing out a normative requirement
from XCU 2.9.1. In Issue 8 draft 1.1 it's on page 2292 line 74177:
Once a utility has been searched for and found (either as a result of
this specific search or as part of an unspecified shell start-up activity),
an implementation may remember its location and need not search for the
utility again unless the PATH variable has been the subject of an
assignment.
The "that statement" referred to, is the following text in the Application
Usage section of the page in XCU 3 detailing the hash command:
The effects of hash -r can also be achieved portably by resetting
the value of PATH; in the simplest form, this can be:
PATH="$PATH"
This statement only follows from what 2.9.1 says if what 2.9.1 says is
that any assignment to PATH must clear the hash table. But that is not
what it says (nor does the definition of PATH in XCU 2.5.3 add anything
relevant).
This is technically true, but if there is no conforming shell that
implements 2.9.1 other than by clearing the hash table (see below) and
no reason for an implementation to do anything other than clearing the
hash table, the standard may as well just go ahead and require that.
[...]
I also just did a test, first I made sure that "ls" was in the hash
table ("hash ls") and then verified that ("hash | grep /bin/ls") and
in all shells I tested that worked (ls is in /bin on NetBSD). Then I
did
PATH=$PATH; hash | grep /bin/ls
and both dash and the NetBSD shell still had /bin/ls in the hash table.
Other shells did not.
Consider the scenario where PATH=/bin:/usr/bin. Consider a hashed
command that exists in /usr/bin, let's say /usr/bin/bash. Now consider a
system upgrade that installs both /bin/bash and /usr/bin/bash (one
likely as a compatibility symlink to the other). POSIX clearly requires
in 2.9.1 that after PATH=$PATH, the bash command in /bin is found,
though I agree it does not require that this result be achieved by
clearing the hash table. In dash, and presumably NetBSD sh, it will not
be, so the optimisation that dash performs here is not permitted by
POSIX. In my shell, a fork of dash, I removed the optimisation for
exactly this reason in 2018.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 22/02/2021 13:13, Joerg Schilling via austin-group-l at The Open
Group wrote:
"Harald van Dijk via austin-group-l at The Open Group"
wrote:
$ bosh -c 'case x in ( (x) echo match ;; esac'
bosh: syntax error at line 1: `(' unexpected
It may be that you are missinterpreting the results.
I'm not. You say there's no state change that happens as a result of the
first opening parenthesis. However, before the first opening
parenthesis, an opening parenthesis is accepted, while after the first
opening parenthesis, an opening parenthesis is not accepted. Since the
two states differ in what they accept, they cannot be the same state.
Therefore, the opening parenthesis did cause a state change.
Given that only "(" is thrown away without modifying internal variables, there
is no state change.
Describing state in terms of variables is possible, but if you do, it
requires including implicit variables such as program counter as part of
the state.
The problem here is that the lexer sees the '(' as a separate token and thus it
is seen as pattern. The next token is the "x" and since "x" is neither ')',
nor '|', this syntax is not accepted.
The error message in both bosh and ksh is "`(' unexpected". This error
message means the shell rejected the '(' token, not whatever follows. Of
course it did: a pattern cannot be any arbitrary token, it has to be
what the standard calls a "word", which '(' is not.
Well, I should have been a bit more specific.
The code is:
if (wdval || (word() != ')' && wdval != '|'))
synbad();
and wdval is the return value from the previous word() call. Wdval is == 0
in case that the parsed word is recognized as a string. As I mentioned, '(' is
not recognized as a string as I mentioned before.
This is not being more specific, this is not "as [you] mentioned" or "as
[you] mentioned before", this is the opposite of what you wrote before.
You previously wrote "The problem here is that the lexer sees the '(' as
a separate token and thus it is seen as pattern." You did get it right
this time though: because of the code that you are quoting now, the '('
is not seen as a pattern.
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 22/02/2021 09:51, Geoff Clare via austin-group-l at The Open Group wrote:
I think your suggestion works, but having a separate production to
add the ')' seems unnecessary. I'd also rename "pattern" since it is
really just WORD that constitutes a pattern to be matched. So how
about this:
case_item_ns : pattern_list ')' linebreak
| pattern_list ')' compound_list
;
case_item : pattern_list ')' linebreak DSEMI linebreak
| pattern_list ')' compound_list DSEMI linebreak
;
pattern_list : WORD /* Apply rule 4 */
| '(' WORD /* Do not apply rule 4 */
| pattern_list '|' WORD /* Do not apply rule 4 */
;
Sure, that looks just fine to me. I tried avoiding having only one of
'(' or ')' part of the pattern (now pattern_list) rule for readability
reasons, but if you prefer a shorter grammar instead, that makes sense
to me too.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 21/02/2021 18:17, Joerg Schilling via austin-group-l at The Open
Group wrote:
"Harald van Dijk via austin-group-l at The Open Group"
wrote:
On 21/02/2021 17:18, Joerg Schilling via austin-group-l at The Open
Group wrote:
"Harald van Dijk via austin-group-l at The Open Group"
wrote:
That is neither what the standard says nor what shells do, though.
case x in ( (x) echo match ;; esac
is rejected because that first '(' does change the parse state, making
the second '(' invalid.
That state change does not happen in ksh and the Bourne Shell as mentioned
before.
That state change clearly does happen and the exact example that I
included and you quoted shows that it does.
$ bosh -c 'case x in ( (x) echo match ;; esac'
bosh: syntax error at line 1: `(' unexpected
It may be that you are missinterpreting the results.
I'm not. You say there's no state change that happens as a result of the
first opening parenthesis. However, before the first opening
parenthesis, an opening parenthesis is accepted, while after the first
opening parenthesis, an opening parenthesis is not accepted. Since the
two states differ in what they accept, they cannot be the same state.
Therefore, the opening parenthesis did cause a state change.
The problem here is that the lexer sees the '(' as a separate token and thus it
is seen as pattern. The next token is the "x" and since "x" is neither ')',
nor '|', this syntax is not accepted.
The error message in both bosh and ksh is "`(' unexpected". This error
message means the shell rejected the '(' token, not whatever follows. Of
course it did: a pattern cannot be any arbitrary token, it has to be
what the standard calls a "word", which '(' is not.
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 21/02/2021 17:18, Joerg Schilling via austin-group-l at The Open
Group wrote:
"Harald van Dijk via austin-group-l at The Open Group"
wrote:
That is neither what the standard says nor what shells do, though.
case x in ( (x) echo match ;; esac
is rejected because that first '(' does change the parse state, making
the second '(' invalid.
That state change does not happen in ksh and the Bourne Shell as mentioned
before.
That state change clearly does happen and the exact example that I
included and you quoted shows that it does.
$ bosh -c 'case x in ( (x) echo match ;; esac'
bosh: syntax error at line 1: `(' unexpected
$ pbosh -c 'case x in ( (x) echo match ;; esac'
pbosh: syntax error at line 1: `(' unexpected
$ ksh -c 'case x in ( (x) echo match ;; esac'
ksh: syntax error at line 1: `(' unexpected
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 19/02/2021 16:21, Geoff Clare via austin-group-l at The Open Group wrote:
It may be prudent to clarify matters by rearranging things in the
grammar so that it ends up saying "Do not apply rule 4" when a '('
has just been seen, like it does when a '|' has just been seen.
That sounds good. I think that can be done by moving '(' and ')' from
the case_item[_ns] rules into the pattern rule, as follows:
case_item_ns : pattern linebreak
| pattern compound_list
;
case_item : pattern linebreak DSEMI linebreak
| pattern compound_list DSEMI linebreak
;
pattern : pattern_head ')'
;
pattern_head : WORD /* Apply rule 4 */
| '(' WORD /* Do not apply rule 4 */
| pattern_head '|' WORD /* Do not apply rule 4 */
;
It is a bit odd to have '(' and ')' part of the pattern rule like this,
since they are not part of any pattern as far as the semantics of the
case statement are concerned, but neither is '|' and that is already
part of the pattern rule too.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 19/02/2021 17:56, Robert Elz wrote:
Date:Fri, 19 Feb 2021 15:11:58 +
From:"Harald van Dijk via austin-group-l at The Open Group"
Message-ID: <4b4f2cbf-2a2e-f0bf-34ca-a7357f99c...@gigawatt.nl>
| Observe that rule 4 is applied for the first word in a pattern even if
| that pattern follows an opening parenthesis.
Yes.
| Because of that, in my
| example, the esac in parentheses is interpreted as the esac keyword
| token, not a regular WORD token that makes for a valid pattern.
Yes.
| Your change would make it so that since the esac keyword is not
| acceptable at that point in the grammar,
Is it not? Why not?
The statement "case foo in (esac" is valid according to the grammar,
just as "case foo in esac" is.
The grammar only allows the '(' in a case_item or case_item_ns. In your
example, you don't have one and there is no other relevant part of the
grammar that allows '(' to appear.
When the '(' was added, it was added (in shells) as a throw away token,
which changes nothing about the parse state, and is permitted merely to
match the ')' required after the pattern (both for user style reasons,
and to handle this common usage of a lone ')' for shells that used
parentheses counting to find the end of a $() cmdsub ... the latter doesn't
work anyway, and is largely gone everywhere now, but the optional (and
supposedly meaningless) '(' remains.
That is neither what the standard says nor what shells do, though.
case x in ( (x) echo match ;; esac
is rejected because that first '(' does change the parse state, making
the second '(' invalid.
(I inserted a space to account for shells that treat '((' as a special
token.)
The issue here is that you seem to be expecting the shell to convert
"esac" to Esac only when the complete statement would then be valid.
No, that's not neither what I'm expecting nor what I believe POSIX says.
I agree that that would be absurd.
What I believe POSIX currently specifies is that "esac" is treated as an
Esac token even when it follows '(', resulting in a syntax error.
What I believe shells should do is what Geoff believes POSIX already
requires, which is treat "esac" as a WORD token when it follows "(".
Even if my interpretation of the spec is correct and this is currently a
syntax error, shells are already permitted to do the right thing, and
many do.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 19/02/2021 15:33, Geoff Clare via austin-group-l at The Open Group wrote:
Harald van Dijk wrote, on 19 Feb 2021:
On 19/02/2021 15:04, Geoff Clare via austin-group-l at The Open Group wrote:
Harald van Dijk wrote, on 19 Feb 2021:
On 19/02/2021 09:59, Geoff Clare via austin-group-l at The Open Group wrote:
How about changing that problem sentence in 2.10.1 to:
When a TOKEN is seen where one of those annotated productions could
be used to reduce the symbol, the applicable rule shall be applied to
convert the token identifier type of the TOKEN to:
* The token identifier of the recognized reserved word, for rule 1.
* A token identifier acceptable at that point in the grammar, for
all other rules.
This change would allow
case esac in
(esac) echo match
esac
which is currently invalid and rejected by bash, and when invoked in POSIX
mode, also rejected by yash and zsh. Should that become valid, or should
that remain an error?
It's currently valid and the above change wouldn't affect that (since
the change only affects when rule 1 can be applied, and this example
does not involve rule 1 except for the initial "case").
It's currently invalid. The grammar reads:
case_item: pattern ')' linebreak DSEMI linebreak
| pattern ')' compound_list DSEMI linebreak
| '(' pattern ')' linebreak DSEMI linebreak
| '(' pattern ')' compound_list DSEMI linebreak
;
pattern : WORD /* Apply rule 4 */
| pattern '|' WORD /* Do not apply rule 4 */
Observe that rule 4 is applied for the first word in a pattern even if that
pattern follows an opening parenthesis. Because of that, in my example, the
esac in parentheses is interpreted as the esac keyword token, not a regular
WORD token that makes for a valid pattern.
Yes, rule 4 is applied there, but your mistake is in assuming that
the *result* of rule 4 is that the token is converted to an Esac.
Looking more closely, I do see now that you are correct that the
"acceptable at that point in the grammar" requirement already appears in
the current standard. I missed that there, you indeed did not change
that. This is an issue separate from your wording changes.
There is no way to apply rule 4 to produce "a token identifier
acceptable at that point in the grammar". The only token identifier
acceptable at that point in the grammar is WORD, and rule 4 does not
produce WORD. Rule 4 reads:
When the TOKEN is exactly the reserved word esac, the token identifier
for esac shall result. Otherwise, the token WORD shall be returned.
Here, the TOKEN is exactly the reserved word esac, and you agree that
this rule is applied. This therefore produces the token identifier for
esac. There is nothing else that turns it into WORD, which is needed to
parse it as a pattern.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 19/02/2021 15:04, Geoff Clare via austin-group-l at The Open Group wrote:
Harald van Dijk wrote, on 19 Feb 2021:
On 19/02/2021 09:59, Geoff Clare via austin-group-l at The Open Group wrote:
How about changing that problem sentence in 2.10.1 to:
When a TOKEN is seen where one of those annotated productions could
be used to reduce the symbol, the applicable rule shall be applied to
convert the token identifier type of the TOKEN to:
* The token identifier of the recognized reserved word, for rule 1.
* A token identifier acceptable at that point in the grammar, for
all other rules.
This change would allow
case esac in
(esac) echo match
esac
which is currently invalid and rejected by bash, and when invoked in POSIX
mode, also rejected by yash and zsh. Should that become valid, or should
that remain an error?
It's currently valid and the above change wouldn't affect that (since
the change only affects when rule 1 can be applied, and this example
does not involve rule 1 except for the initial "case").
It's currently invalid. The grammar reads:
case_item: pattern ')' linebreak DSEMI linebreak
| pattern ')' compound_list DSEMI linebreak
| '(' pattern ')' linebreak DSEMI linebreak
| '(' pattern ')' compound_list DSEMI linebreak
;
pattern : WORD /* Apply rule 4 */
| pattern '|' WORD /* Do not apply rule 4 */
Observe that rule 4 is applied for the first word in a pattern even if
that pattern follows an opening parenthesis. Because of that, in my
example, the esac in parentheses is interpreted as the esac keyword
token, not a regular WORD token that makes for a valid pattern.
Your change would make it so that since the esac keyword is not
acceptable at that point in the grammar, it would not be interpreted as
the esac keyword, unless I am misreading your change.
Cheers,
Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001454]: Conflict between "case" description and grammar
On 19/02/2021 09:59, Geoff Clare via austin-group-l at The Open Group wrote: How about changing that problem sentence in 2.10.1 to: When a TOKEN is seen where one of those annotated productions could be used to reduce the symbol, the applicable rule shall be applied to convert the token identifier type of the TOKEN to: * The token identifier of the recognized reserved word, for rule 1. * A token identifier acceptable at that point in the grammar, for all other rules. This change would allow case esac in (esac) echo match esac which is currently invalid and rejected by bash, and when invoked in POSIX mode, also rejected by yash and zsh. Should that become valid, or should that remain an error? (Accidentally sent only to Geoff, re-sending to the list.) Cheers, Harald van Dijk
Re: [1003.1(2008)/Issue 7 0000249]: Add standard support for $'...' in shell
On 06/02/2021 23:38, Robert Elz via austin-group-l at The Open Group wrote: Date:Sat, 06 Feb 2021 21:55:19 +0100 From:Steffen Nurpmeso Message-ID: <20210206205519.43rln%stef...@sdaoden.eu> | Fiddling with bytes is something completely different. But how is the shell supposed to know? Consider U1=$'\u021c' U2=$'\u0a47' X1=$'\310\234' X2=$'\340\251\207' [...] Ignoring the bit about converting to other replacement chars, here, since I'm concerned with valid codepoints only, I don't think the shell should be converting this kind of thing via iconv() ... utilities might (including built-ins in sh, like echo or printf) but not the shell itself. In the above (assuming I did the conversions correctly) it should always be the case that $U1 = $X1 and $U1 = $X2, regardless of any locale settings. If I cannot assume that when writing a script then I have no idea how I would ever do anything with non-ascii chars reliably. bash, ksh and zsh, all of which support $'\u', do convert the Unicode code point to the current locale, and I support this and implemented the same in my shell. For \u sequences that ask for a Unicode code point that is not representable in the current locale, the \u sequence is left unconverted (bash, ksh, my shell) or causes the shell to report an error (zsh). This is useful for scripts that aim to work in a limited selection of locales and know that certain characters are valid in all the supported locales, but are not encoded the same way in all of them. If they want to print a Euro symbol, for instance, they can write echo $'\u20AC' and be assured it works everywhere the Euro symbol is supported. If they instead write echo '€' where the script is saved as UTF-8, the script will needlessly break when it is run in an ISO-8859-15 environment. Cheers, Harald van Dijk
Re: [1003.1(2016/18)/Issue7+TC2 0001384]: Subshell of an interactive shell is effectively non-interactive
On 11/01/2021 16:45, Austin Group Bug Tracker via austin-group-l at The Open Group wrote: shall be set to the default action.If the shell is interactive, the subshell shall behave as a non-interactive shell in all respects except:The expansion of the special parameter '-' and the output of set -o and set +o shall continue to indicate that it is interactive.The set -n option may be ignored. The output of set -o and set +o need not indicate whether the shell is interactive. Most (not all) shells include it in set -o, most (not all) shells exclude it from set +o. Those shells that do not indicate that the shell is interactive cannot continue to indicate so. All shells that I know of do include i in $- when the shell is interactive, but pdksh and derivatives (posh, mksh) remove it from $- in subshells. There does not seem to be a good reason to require a change to these shells so that an effectively non-interactive subshell starts to pretend that it is interactive. Cheers, Harald van Dijk
Re: More issues with pattern matching
On 05/08/2020 15:54, Geoff Clare via austin-group-l at The Open Group wrote: Harald van Dijk wrote, on 31 Jul 2020: Take the previous example glibc's cy_GB.UTF-8 locale, but with a different collating element: in this locale, "dd" is a single collating element too. Therefore, this must be matchable by bracket expressions. Incorrect. I think you overlooked these statements in XBD 9.3.5 items 2 and 3: It is unspecified whether a matching list expression matches a multi-character collating element that is matched by one of the expressions. It is unspecified whether a non-matching list expression matches a multi-character collating element that is not matched by any of the expressions. My message was indirectly in reply to your message where you claimed that shells were required to support this. I'm very happy to see that this is actually not true, thanks for that. Cheers, Harald van Dijk
Re: ksh93 job control behaviour [was: Draft suggestion: Job control and subshells]
On 29/07/2020 16:07, Geoff Clare wrote: Harald van Dijk wrote, on 29 Jul 2020: On 29/07/2020 09:45, Geoff Clare wrote: It's only easy because (most/all?) shells take the easy option and do a lexical analysis of the command to be substituted. trap (without -p) in a command substitution prints the trap actions of the parent shell in bash, busybox ash, gwsh, ksh and yash. Of these, only busybox ash does it by lexical analysis. Perhaps things have changed since 2009 when this first came up, but (as I reported in bug 53) at that time, ksh93 was the only shell I could find which handled things like: var=trap; save_traps=$($var) and save_traps=$(trap; true) correctly. I do not know which shells were the ones you tried at the time, but you should have included bash. Those exact commands print the trap actions of the parent shell at least as far back as bash 2.05b from 2002. Cheers, Harald van Dijk
Re: More issues with pattern matching
On 31/07/2020 00:10, Harald van Dijk wrote: Take the previous example glibc's cy_GB.UTF-8 locale, but with a different collating element: in this locale, "dd" is a single collating element too. Therefore, this must be matchable by bracket expressions. However, "d" individually must *also* be matched by pattern expressions. "dd" can be matched by both [!x] and [!x][!x]. A shell cannot use regcomp()+regexec() to find the longest match for [!x] and assume that that is matched: a shell where case dd in [!x]d) echo match ;; esac does not print "match" does not implement what POSIX requires. A shell where case dd in [!x]) echo match ;; esac does not print "match" does not implement what POSIX requires either. Using regcomp()+regexec() to bind [!x] to either "d" or "dd" without taking the rest of the pattern into account will fail to match in one of these cases. And it needn't be the same way for all bracket expressions in a single pattern: case ddd in [!x][!x]) echo match ;; esac Shells are required by POSIX to consider both the possibility that [!x] picks up "d" and that it picks up "dd" for each bracket expression individually. A followup example: it seems downright crazy that POSIX would require that case ddd in *[!d]*) echo match ;; esac prints "match", yet that appears to be exactly what it does require, and exactly what yash implements: "dd" is a collating element which is not "d", and therefore must be matched by [!d]. And this is something where GNU fail to implement the POSIX-specified behaviour even in regular expressions. If the regular expression support does not work as specified, shells cannot implement pattern matching on top of regular expressions and expect correct results. $ echo ddd | LC_ALL=cy_GB.UTF-8 grep '[^d]' $ echo ddd | LC_ALL=cy_GB.UTF-8 grep '.[^d]' ddd It's clear that if the second prints 'ddd', so should the first, so it's clear that this result indicates a bug. What's not clear to me is whether the second should print 'ddd'. When the string 'ddd' is part of a set of strings to be sorted, the first collating element is 'dd' and the second is 'd'. The second and third character do not together form a collating element. Is it correct that grep nevertheless uses that second and third 'd' to match '[^d]'? If that is not correct, then shells cannot use regexec() at a given starting position: that given starting position may yield different collating elements compared to when the string is searched from the beginning. Cheers, Harald van Dijk
Re: More issues with pattern matching
On 26/09/2019 10:20, Geoff Clare wrote: Geoff Clare wrote, on 26 Sep 2019: Are shells required to support this, and are shells therefore implicitly required to translate patterns to regular expressions, or should it be okay to implement this with single character support only? Shells are required to support it. They don't need to translate entire patterns to regular expressions - they can use either regcomp()+regexec() or fnmatch() to see if the bracket expression matches the next character. Sorry, I should have written "matches *at* the next character" here; I didn't mean to imply checking against a single character. For example, if using regcomp()+regexec() the shell could try to match the bracket expression against the remainder of the string and see how much of it regexec() reported as matching. To use fnmatch() I suppose you would have to use it in a loop, passing it first one character, then two, etc. (stopping at the number of characters between the '.'s). As I had replied at the time, it is fundamentally impossible in the general case as POSIX does not provide any mechanism to escape characters and there is nothing in POSIX that rules out the possibility of a collating element containing "=]" or ".]". However, ignoring that aspect of it, looking at implementing this once again, implementing it the way you specified is incorrect, fixing it to make it correct cannot possibly be done efficiently with standard library support, and shells in general don't bother to implement what POSIX specifies here. Take the previous example glibc's cy_GB.UTF-8 locale, but with a different collating element: in this locale, "dd" is a single collating element too. Therefore, this must be matchable by bracket expressions. However, "d" individually must *also* be matched by pattern expressions. "dd" can be matched by both [!x] and [!x][!x]. A shell cannot use regcomp()+regexec() to find the longest match for [!x] and assume that that is matched: a shell where case dd in [!x]d) echo match ;; esac does not print "match" does not implement what POSIX requires. A shell where case dd in [!x]) echo match ;; esac does not print "match" does not implement what POSIX requires either. Using regcomp()+regexec() to bind [!x] to either "d" or "dd" without taking the rest of the pattern into account will fail to match in one of these cases. And it needn't be the same way for all bracket expressions in a single pattern: case ddd in [!x][!x]) echo match ;; esac Shells are required by POSIX to consider both the possibility that [!x] picks up "d" and that it picks up "dd" for each bracket expression individually. This means that in the worst case, if every bracket expression in a pattern has X ways to match, and a pattern has Y bracket expressions, the shell is required to consider X^Y possibilities. This is completely unreasonable and it's obvious why no shell actually does this. The complexity can be reduced in theory, but POSIX does not expose enough information to allow that to be implemented in a shell. The only way around this mess is by translating the whole pattern to a regular expression, as only the C library has enough detailed knowledge about the locale that it can implement it efficiently.[*] Doing that has its own new set of problems though: translating the whole pattern to a regular expression means the shell no longer has the option to decide how to handle invalid byte sequences (byte sequences that lead to EILSEQ) that shells in general try to tolerate, and the shell no longer has the option to decide how to handle invalid patterns (patterns containing non-existent character classes or collating elements) which shells in general also aim to tolerate. Cheers, Harald van Dijk [*] I have not investigated whether implementations actually do do this efficiently.
Re: ksh93 job control behaviour [was: Draft suggestion: Job control and subshells]
On 29/07/2020 09:45, Geoff Clare wrote: It's only easy because (most/all?) shells take the easy option and do a lexical analysis of the command to be substituted. trap (without -p) in a command substitution prints the trap actions of the parent shell in bash, busybox ash, gwsh, ksh and yash. Of these, only busybox ash does it by lexical analysis. jobs -p in a command substitution prints the PIDs of the jobs of the parent shell in bash, busybox ash, ksh, mksh, pdksh. Of these, only busybox ash does it by lexical analysis. Almost no shell takes the lexical analysis option. Cheers, Harald van Dijk
Re: sh: aliases in command substitutions
On 19/04/2020 22:04, Robert Elz wrote:
If I do
alias switch=case
then every shell I can find, processes
switch foo in foo) echo ok;; esac
correctly (and writes "ok" to stdout).Aside from that I
don't think we should really have aliases at all (not relevant
directly here), that is fine.
However if I want to do
echo "$( switch foo in foo) echo ok;; esac )"
then XCU 2.2.3 (2.2 is "Quoting" 2.2.3 is "Double-Quotes")
(lines 74718-22, Issue 7 TC2 - 2018 edition) says ...
The input characters within the quoted string that are also enclosed
between "$(" and the matching ')' shall not be affected by the
double-quotes, but rather shall define that command whose output
replaces the "$(...)" when the word is expanded. The tokenizing rules
in Section 2.3, not including the alias substitutions in Section 2.3.1,
shall be applied recursively to find the matching ')'.
The relevant part here being the "not including the alias substitutions
in Section 2.3.1".
[...]
But perhaps that is actually what the standard says must happen - we
don't use the alias for finding the matching ')', but then do when
parsing the command inside.That would be a recipe for disaster,
but if it is what old versions of ksh did/do then perhaps the standard
really is requiring that? If so, it is time for a change, as nothing
relevant acts like that any more (not mksh, not ksh93, not bosh, ...)
The text you quoted about not performing alias substitution only applies
to finding the matching ")", so yes, that is what the standard is
saying, I believe. Once the matching ")" is located, everything between
"$(" and ")" is parsed as normal, including alias substitution. However,
as parsing $(switch foo in foo) by itself results in a syntax error as
far as POSIX is concerned, shells are permitted to parse the full $(
switch foo in foo) echo ok;; esac ) as a command substitution as an
extension instead. All the results you see in shells appear to me to be
permitted by the standard.
Cheers,
Harald van Dijk
Re: XCU 2.14: Exit status by no-argument `return' in shell trap handlers
On 19/04/2020 14:32, Koichi Murase wrote:
2020-04-19 21:55 Harald van Dijk :
My reading was that interpretation B must be what is intended, which
is why I had modified my shell, a fork of dash, to change dash's A
behaviour to B in late 2018.
Thank you. Is your shell this https://github.com/hvdijk/gwsh ? I
tried gwsh-0.5.9.1. Here is the updated list:
Yes, that is my shell. Pay no attention to that version number, that is
dash's version number that I have not updated because I never did a
proper release yet. :)
(A) `zsh', `dash', `busybox', `heirloom-sh'
(B) `bash-4.4', `mksh', `ksh', `gwsh'
(C) `yash'
(D) none
Not Implemented: `bash-4.3', `posh'
My reasoning for that is that the description of the return
commanhd ("When return is executed in a trap action, the last
command is considered to be the command that executed immediately
preceding the trap action.") is identical to that of the exit
command ("When exit is executed in a trap action, the last command
is considered to be the command that executed immediately preceding
the trap action.")
Thank you. This is a good point. Maybe this is the origin of the
current wording.
and for the exit command, almost all shells, including dash, are in
agreement that it applies even when the exit command is invoked
indirectly. [...]
It is reasonable that indirect `exit' in function calls in trap
handlers are affected because it actually brings the completion of
trap action which is more consistent with the following description of
`trap'. While, the trap action will not be completed by indirect
`return', so it is not surprising that the behavior can be different
between `exit' and `return'.
True, and if the intent is that exit and return behave differently and
the standard is updated to clearly require that, I have no problem
changing the shell back to the prior behaviour.
It does still mean that anyone writing a function needs to beware that
exit and exit $? do not have the same effect, if there is any
possibility that the function will be invoked from a trap action. I
suspect most people writing functions will not be aware of that.
Another question is what the behaviour should be in a subshell. That is:
f() { false; return; } # or exit
trap '(f; echo $?)' EXIT
Is the return or exit here still considered to execute within a trap
action, or does the fact that subshells reset traps mean that whatever
was executing in the parent shell can no longer be considered a trap
action within the subshell?
Cheers,
Harald van Dijk
Re: XCU 2.14: Exit status by no-argument `return' in shell trap handlers
On 19/04/2020 12:27, Robert Elz wrote:
My first reading (my immediate impression) is that your interpretation A
is what is intended, and only a return command executing directly from the
trap action itself, and not one in a function that happens to be called,
should be affected. (And of course, the former is only specified if the
trap occurs while running a function or dot script).
My reading was that interpretation B must be what is intended, which is
why I had modified my shell, a fork of dash, to change dash's A
behaviour to B in late 2018. My reasoning for that is that the
description of the return commanhd ("When return is executed in a trap
action, the last command is considered to be the command that executed
immediately preceding the trap action.") is identical to that of the
exit command ("When exit is executed in a trap action, the last command
is considered to be the command that executed immediately preceding the
trap action."), and for the exit command, almost all shells, including
dash, are in agreement that it applies even when the exit command is
invoked indirectly. If different behaviour is intended for the exit and
return commands, but identical wording is used for both, that must
surely be a defect in the standard.
Cheers,
Harald van Dijk
Re: [1003.1(2013)/Issue7+TC1 0000854]: requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice
On 28/03/2020 16:31, Robert Elz wrote: Date:Sat, 28 Mar 2020 15:12:48 + From:Harald van Dijk Message-ID: <865ce3c3-9976-9de4-1a65-9aebe80b8...@gigawatt.nl> | Not to disagree with the main point of your message, but printf is | something of a portability problem too. Not really if used correctly. There are always extensions, to everything, and scripts that use them suffer portability issues (I'm sure you've seen all the scripts that insist that test "$1" == foo is valid...). "Correctly" is tricky. A script that uses test ==, or hex escape sequences with printf, can be a "Conforming POSIX Application Using Extensions" provided it documents this requirement. Using extensions makes scripts non-portable, but that's only incorrect if they are meant to be portable in the first place. But yes, test == is probably the number one issue making scripts that aim to be portable fail to be so. If the only extension they use, whether documented or not, is in the test utility, or in the printf utility, I can see some value in having a simple way of only overriding the test utility, or only overriding the printf utility. Aside from educating people to stick to what is specified to work, everywhere, there's not a lot we can do about this (use /usr/bin/printf on NetBSD you get the exact same printf as is built into /bin/sh - but at extra cost). If shells behaved the way the standard specified, it would have been possible to install GNU coreutils on NetBSD into a non-standard prefix and prepend that to $PATH for scripts that rely on test == or printf \x. Actually, in some ash-based shells, there is a way to do that: explicitly place %builtin somewhere in PATH, and directories that precede it take precedence. Some ash-based shells that traditionally supported this already removed it (at least FreeBSD sh), but support for it is still present in dash and in my shell. That said, I do think this is a mistake and have been thinking about removing it already as well, even though the issue's resolution allows it (it says PATH entries containing % make the behaviour implementation-defined), because I think that it is worse than the problem it aims to solve: it is a potential security risk for environment variables that are inherited from parent processes by child processes to have a completely different meaning between parent and child, and external processes that look up utilities in $PATH cannot possibly replicate the shell's behaviour. Cheers, Harald van Dijk
Re: [1003.1(2013)/Issue7+TC1 0000854]: requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice
On 28/03/2020 06:40, Robert Elz wrote: What's more, that the list of intrinsic builtins has kill and pwd in it is simply absurd - if you were looking for builtins that people might want to override, then things like printf (which the text gives as an example as one that should be able to be overridden) almost certainly isn't what would be wanted, Not to disagree with the main point of your message, but printf is something of a portability problem too. The use of hexadecimal escape sequences is not specified by the standard, not implemented in all shells and I have no intention of implementing it myself, but it has been used often enough in scripts that sometimes I did feel like taking the easy route and making the scripts use /usr/bin/printf (which on my system is the GNU coreutils version) rather than fixing up the uses to use octal escape sequences. Cheers, Harald van Dijk
Re: sed '\n\nnd'
On 25/03/2020 23:30, shwaresyst wrote: yes, without them the argument would be "nnnd", after quote removal by the shell. The reasoning in first reply was meant to show that the non-GNU versions might be erroneously treating the second '\' as "do contol alias processing always", ignoring that its use as delimeter overrides that interpretation, to get the results observed. Again, it's the BSD version that treats the second \n as , treating the backslash in there as just escaping the delimiter character. You have it backwards. The GNU version is the one that treats the second \n as . ---- On Wednesday, March 25, 2020 Harald van Dijk wrote: On 25/03/2020 21:09, shwaresyst wrote: > If it wasn't in single quotes, then that might be plausible, but I don't > see it as the intent since no other aliases are excluded as > possibilities for after the '/'. The initial "\n" makes 'n' the > delimiter, the 2nd overrides it as being the BRE terminator, and the > following 'n' is the terminator, before the 'd' command. Should there be > something explicit about aliases not being usable when repurposed as > delimiter, maybe. This reply makes no sense to me, sorry. The single quotes are processed at the shell level. Without single quotes, there would be no backslash for sed to process. Regardless, the only thing I wrote was that you simultaneously considered the GNU version more correct and explained it in a way that led me to believe you actually consider the BSD version more correct. I wrote absolutely nothing about what the standard says or intends to say. > ---- > On Wednesday, March 25, 2020 Harald van Dijk <mailto:a...@gigawatt.nl>> wrote: > > On 25/03/2020 19:44, shwaresyst wrote: > > The GNU version is more correct, in my opinion, in that the use of n as > > a delimiter should take precedence over its use as control character > > alias with the wording as is. The other versions appear to consider the > > BRE as so does not match 'n'. > > You have that backwards, don't you? The GNU version lets the use of \n > as a control character take precedence over its use as a delimiter. > That's why n gets printed: \n\nn is treated as /\n/, which can never > match any single-line string, so nothing gets deleted. > > Likewise, > > echo n | sed '\n[^\n]nd' > > prints nothing with GNU sed, but prints n with FreeBSD sed for the same > reason: 'n' does contain a character that is not , but does not > contain any character that is not . > > > > > > On Wednesday, March 25, 2020 Oğuz <mailto:oguzismailuy...@gmail.com> > <mailto:oguzismailuy...@gmail.com <mailto:oguzismailuy...@gmail.com>>> wrote: > > > > echo n | sed '\n\nnd' > > > > Above command returns 'n' with GNU sed, and nothing with BSD seds and > > OmniOS sed. [...]
Re: sed '\n\nnd'
On 25/03/2020 21:09, shwaresyst wrote: If it wasn't in single quotes, then that might be plausible, but I don't see it as the intent since no other aliases are excluded as possibilities for after the '/'. The initial "\n" makes 'n' the delimiter, the 2nd overrides it as being the BRE terminator, and the following 'n' is the terminator, before the 'd' command. Should there be something explicit about aliases not being usable when repurposed as delimiter, maybe. This reply makes no sense to me, sorry. The single quotes are processed at the shell level. Without single quotes, there would be no backslash for sed to process. Regardless, the only thing I wrote was that you simultaneously considered the GNU version more correct and explained it in a way that led me to believe you actually consider the BSD version more correct. I wrote absolutely nothing about what the standard says or intends to say. ------------ On Wednesday, March 25, 2020 Harald van Dijk wrote: On 25/03/2020 19:44, shwaresyst wrote: > The GNU version is more correct, in my opinion, in that the use of n as > a delimiter should take precedence over its use as control character > alias with the wording as is. The other versions appear to consider the > BRE as so does not match 'n'. You have that backwards, don't you? The GNU version lets the use of \n as a control character take precedence over its use as a delimiter. That's why n gets printed: \n\nn is treated as /\n/, which can never match any single-line string, so nothing gets deleted. Likewise, echo n | sed '\n[^\n]nd' prints nothing with GNU sed, but prints n with FreeBSD sed for the same reason: 'n' does contain a character that is not , but does not contain any character that is not . > > On Wednesday, March 25, 2020 Oğuz <mailto:oguzismailuy...@gmail.com>> wrote: > > echo n | sed '\n\nnd' > > Above command returns 'n' with GNU sed, and nothing with BSD seds and > OmniOS sed. [...]
Re: sed '\n\nnd'
On 25/03/2020 19:44, shwaresyst wrote: The GNU version is more correct, in my opinion, in that the use of n as a delimiter should take precedence over its use as control character alias with the wording as is. The other versions appear to consider the BRE as so does not match 'n'. You have that backwards, don't you? The GNU version lets the use of \n as a control character take precedence over its use as a delimiter. That's why n gets printed: \n\nn is treated as /\n/, which can never match any single-line string, so nothing gets deleted. Likewise, echo n | sed '\n[^\n]nd' prints nothing with GNU sed, but prints n with FreeBSD sed for the same reason: 'n' does contain a character that is not , but does not contain any character that is not . On Wednesday, March 25, 2020 Oğuz wrote: echo n | sed '\n\nnd' Above command returns 'n' with GNU sed, and nothing with BSD seds and OmniOS sed. [...]
Re: XCU: 'exit' trap condition
On 15/03/2020 19:00, Dirk Fieldhouse wrote:
[...] The careful reader might
wonder if it is significant that the first "terminate" is said to be
done by the "shell utility" while the second is said to be done by just
the "shell", but there is no answer.
Ah, okay. You suggested you were under the impression that the standard
required ("The standard's wording on 'exit' appears to require [...]")
something that it shouldn't, something that shells don't do. If you had
already seen that there was a legitimate interpretation of the text that
does match what shells do, and you weren't actually under the impression
that the standard required otherwise, if you merely felt the standard
could be misinterpreted in such a way, ignore my reply.
Cheers,
Harald van Dijk
Re: XCU: 'exit' trap condition [was:Re: XCU: 'return' from subshell]
On 15/03/2020 14:44, Dirk Fieldhouse wrote:
On 15/03/20 07:26, Robert Elz wrote:
>...>
>[I wrote:]> | Is there any suggestion that the 'exit'-like
behaviour of any shell that
| implements it for 'return' in such contexts is subtly different
from 'exit'?
Not that I am aware of. exit is kind of blunt, it is quite hard to
be subtly different - I suppose the one difference might be whether the
EXIT trap (if there is one) is run, ...
>...>
Unfortunately this brings a differently flavoured can of worms.
The standard's wording on 'exit' appears to require that the EXIT trap
is run when exiting a top level shell ("before the shell terminates,"),
and not when execution "continue[s] in the environment from which that
subshell environment was invoked" -- which the standard contrasts with
"otherwise, the shell utility shall terminate...".
"Before the shell terminates" is not limited to "before the top level
shell terminates". If a shell terminates, even if it is a subshell that
terminates, any EXIT trap action should be run.
Cheers,
Harald van Dijk
Re: Weird possibility with async processes, $!, and long running scripts
On 15/03/2020 08:36, Robert Elz wrote: I'd love to hear from anyone who has (or can even imagine, regardless of whether it is currently implemented anywhere) a better solution for this issue. Or if for some reason I am not understanding this isn't even a potential (certainly it is extremely unlikely) problem, then why. This is a real problem. The Linux kernel has made changes in 5.4 to make PID reuse far less likely, because all sorts of code was not prepared to handle that possibility correctly, not just shells. Of course it does not and cannot guarantee that PIDs do not get reused at all. The one way I can see that would fix this is to mandate limited support for job ID notation even when job control is disabled, and for scripts to use that instead of tracking PIDs. Cheers, Harald van Dijk
Re: XCU: 'return' from subshell
On 13/03/2020 13:10, Chet Ramey wrote:
On 3/12/20 4:21 PM, Harald van Dijk wrote:
On 11/03/2020 17:44, Don Cragun wrote:
Would this issue be resolved if we change the last sentence of the
description section of the return Special Built-In Utility from:
If the shell is not currently executing a function
or dot script, the results are unspecified.
to:
If the shell is not currently executing a function
or dot script running in the same shell execution
environment as the command that invoked the function
or dot script, the results are unspecified.
?
Can this instead say "in the same shell execution environment as the
compound-list of the compound-command of the function definition", so that
f() (return 1)
which is fairly sensible and works in all shells[*] remains well-defined,
but only something along the lines of f() { (return 1) } or
f() ( (return 1) ) becomes unspecified?
We should be able to do better than that. I don't see why "if not executing
in the same shell execution environment as the compound-list ..." can't
cover the f() { (return 1) } case as well, and seems to work in all shells.
I don't see how you can allow that without also allowing
f() { (return 7; echo no); echo $?; }; f
If that also works in all shells (meaning it doesn't print no, and does
print 7), then by all means standardise it.
Don Cragun had written:
Since current shells do not all treat return in a subshell as exit, I think we
should leave that behavior unspecified.
I agree with that, I think it makes sense to limit it to what works in
all current shells, that's why I limited my proposed change to what I
know works. Not all current shells treat return in a subshell as exit,
but if all current shells do treat return in a subshell *in a function*
as exit, it makes sense to also allow that.
Cheers,
Harald van Dijk
Re: [1003.1(2016)/Issue7+TC2 0001295]: Left brackets in shell patterns may cause other pattern matching characters to be taken literally in all contexts
On 15/12/2019 12:38, Robert Elz wrote: Date:Sat, 14 Dec 2019 14:41:34 + From:Harald van Dijk Message-ID: <9cbcf0fd-83d1-b449-cfe8-eafdee14b...@gigawatt.nl> | This was not an exhaustive list, nor was it intended to be. Still, if one relies (at all) upon behaviouir of pdksh and posh in an argument, that, of itself, is fair evidence that the major shells (including the ones that posix most relies upon) do not act that way. No, it is not. The claim I was responding to was that there would be no shell that behaved this way. These two were the first counterexamples I had readily available. "The ones that posix most relies upon" includes ksh, does it not? It behaves the same way. I did not mention ksh because it used to be the other shell where the string [0-9] was considered to match the pattern [0-9] for a long time, but in its current version (2020.0.0), it no longer does, it now behaves the same way pdksh and posh do. | Then can you please open a bug asking for exactly that change to be made | to the standard? I can, but not until there is a new updated doc available for review - adding more changes upon changes to what is currently being edited doesn't make a lot of sense to me ... something entirely new would be reasonable (something which hasn't been considered recently) but where we're discussing text that is being changed, I'd like to be sure what state it ends up in, thyat is, what we're working with, first. As long as it is before we have a new version of the standard maintaining the current inconsistency, sure, no problem. | You wrote "too lazy to do things correctly", which is not a criticism of | shells, but of the motivations of their authors. That wasn't what was intended either (and the distinction between a shell and its author/maintainer is tenuous at best anyway) - what it was intended as was an admonition to those who make accept/reject decisions not to take that kind of argument as a reason for adding more unspecified wording to the standard. Shell authors are free to do whatever they like, for whatever reasons they like, but they don't have the right to demand that the standard be dumbed down so they can label their shell as conforming. I do not see how this shows laziness on the shell authors' part. | "Implementing it would be unreasonable", however, can be a good reason | not to change shell behaviour to comply with POSIX I disagree. Some shells sometimes cannot conform for backwards compat reasons, that's understandable, but it isn't an excuse to change the standard to permit their behaviour. Then we have a different understanding of the goals of POSIX. My understanding is that it is both a specification for implementations to work towards, and a specification for applications to use as the common baseline of implementations. For the former goal, changing the standard to permit their behaviour is not useful. For the latter, it is. | depending on why the author considers it unreasonable, Some shells simply refuse to implement parts of posix because it is considered, stupid, or even dangerous - the former is why NetBSD have decided to ignore the "must implement wait/cd/umask/... as execable utilities" (that one I don't much care about, doing it is trivial, but it is useless, and clutters the bin directories with nonsense, so the community decided to reject that); the latter is why we don't (won't) do "cd -L" - the difference between "cd .." and "ls .." is unconscionable, and that someone might do ls .. # and verify that user is where expected to be cd ..; rm -fr * # or just rm -fr ../* (or some other pattern) is simply too big a risk to even consider. In all cases (assuming no other file system manuipulations are happening in parallel), the ".." filename handed to ls (or any other utility) - either as the complete pathname or a prefix to a longer pathname, must reference the exact same directory as the ".." in the same pathname handed to the "cd' builtin. Worth mentioning is that if you see this as a serious risk, it is still possible to implement it in a way that mitigates that risk. It is possible to implement the -L and -P options, but keep -P as the default. It is then also possible to change that default based on an environment variable or some other configuration setting, either a common one such as GNU's POSIXLY_CORRECT, or a cd-specific one such as CDOPT=-L, and document that that environment variable or configuration setting is needed to get NetBSD to be POSIX-conforming. I personally do rely on the cd -L behaviour in my everyday work, despite not being convinced it is a good idea. But that NetBSD's sh (or system) doesn't implement those isn'
