Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
2020-02-04 19:56:05 +0700, Robert Elz: [...] > What I am at the very least unclear about, is that the way that this > group chose to require "--" processing, appears to me to also require > that any arg (before that "--" or an arg not starting with "-") which > does start with "-" be treated as an option - which since we have no > options (not sure if any shell does, I haven't encountered any) would > *require* issuing an "invalid option" error. [...] AFAICT, the behaviour of exec -a foo bar would be unspecified because -a is not an option of POSIX exec. So an implementation can choose to do what they please including what many shells do and treat it as an option that takes an argv[0] as parameter, or to treat that -a as an unknown option and report an error, or to consider it as the command to execute. Ideally, applications should ideally treat it as an option (and error out if it's not supported) for consistency, but some implementations may not want to, for instance for backward compatibility (but what will they do then when POSIX specifies exec -a?). See how some implementations of commands like true, false, printf don't error out for some arguments that start with "-" but are all required to support "--". true --help true x behaviour is unspecified But true true -- is. -- Stephane
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
On 2/4/20 9:29 AM, Eric Blake wrote: On 2/4/20 9:16 AM, Robert Elz wrote: I am putting this in a new thread, as it isn't really important, more just amusing, but the solution to this issue, with respect to the "." command, is I think, causing that command to be in violation of the standard (in a completely different way than what the previous discussion is about). The resolution of the issue (as was previously noted) adds the words: The dot special built-in shall support XBD Section 12.2 (on page 215). Last time I looked, "." was neither a lower case letter, nor a digit, in any character set. Hence, the resolution of this issue has caused a contradiction in the standard - those guidelines are both required, and ignored, all in the same command. We could fix all this by changing the name of the "." command, probably to "source" as that's already supported by some shells, but is this degree of penantry really important, or do we just live with the standard being inconsistent with itself? Good catch. However, I don't think we can require the name 'source'; better would be a fix along the lines of what we do for 'tail', in documenting explicit exceptions to the XBD guidelines. Something like: The dot special built-in shall support XBD Section 12.2 (on page 215), except that it does not comply to Guideline 1 or 2 due to its name. or maybe The dot special built-in shall support Guidelines 3-14 of XBD Section 12.2 (on page 215). For what it's worth, the standard was already self-contradictory for the [ utility; it is also required to support XBD Section 12.2 with an exception for Guideline 10; but would need a similar exemption for Guideline 1 and 2. I don't think reopening bug 252 is correct, but a new bug fixing both '.' and '[' would be in order. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
On 2/4/20 9:16 AM, Robert Elz wrote: I am putting this in a new thread, as it isn't really important, more just amusing, but the solution to this issue, with respect to the "." command, is I think, causing that command to be in violation of the standard (in a completely different way than what the previous discussion is about). The resolution of the issue (as was previously noted) adds the words: The dot special built-in shall support XBD Section 12.2 (on page 215). Last time I looked, "." was neither a lower case letter, nor a digit, in any character set. Hence, the resolution of this issue has caused a contradiction in the standard - those guidelines are both required, and ignored, all in the same command. We could fix all this by changing the name of the "." command, probably to "source" as that's already supported by some shells, but is this degree of penantry really important, or do we just live with the standard being inconsistent with itself? Good catch. However, I don't think we can require the name 'source'; better would be a fix along the lines of what we do for 'tail', in documenting explicit exceptions to the XBD guidelines. Something like: The dot special built-in shall support XBD Section 12.2 (on page 215), except that it does not comply to Guideline 1 or 2 due to its name. or maybe The dot special built-in shall support Guidelines 3-14 of XBD Section 12.2 (on page 215). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
I am putting this in a new thread, as it isn't really important, more just amusing, but the solution to this issue, with respect to the "." command, is I think, causing that command to be in violation of the standard (in a completely different way than what the previous discussion is about). The resolution of the issue (as was previously noted) adds the words: The dot special built-in shall support XBD Section 12.2 (on page 215). XBD 2.15 (on page 217) [aside: unless things have changed in the new version, 12.2 starts on page 216, not 215] lines 7344-6 says: The utilities in the Shell and Utilities volume of POSIX.1-2017 that claim conformance to these guidelines shall conform completely to these guidelines as if these guidelines contained the term ``shall'' instead of ``should''. That is, now that "." claims conformance to XBD 2.12 (which it didn't before) it *shall* conform completely (and with all uses of "should" in the guidelines replaced by "shall"). But. Guideline 1, with s/should/shall/ ... Guideline 1: Utility names shall be between two and nine characters, inclusive. "." is only one character, which is not between two and nine (inclusive or not). And guideline 2, again with the s/should/shall/ applied ... Guideline 2: Utility names should include lowercase letters (the lower character classification) and digits only from the portable character set. Last time I looked, "." was neither a lower case letter, nor a digit, in any character set. Hence, the resolution of this issue has caused a contradiction in the standard - those guidelines are both required, and ignored, all in the same command. We could fix all this by changing the name of the "." command, probably to "source" as that's already supported by some shells, but is this degree of penantry really important, or do we just live with the standard being inconsistent with itself? kre
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Date:Tue, 4 Feb 2020 07:58:43 -0600
From:Eric Blake
Message-ID: <4e4b95da-9e30-5e55-faa9-58be84ed8...@redhat.com>
| Implementations are allowed to provide any number of options as extensions.
Sure, that's not an issue.
| No, the standard still leaves plenty of latitude for your implementation
| of '.' to behave like 'chmod', where '-x' is an undocumented option that
| behaves as if it is a request for '.' to instead source the file named
| '-x' wherever it appears in the search path used by '.', as if the user
| had typed '. -- -x'.
OK, I guess if I read it really closely, XBD 2.12 doesn't actuallly say
that (when options are still possible) anything that starts with a '-'
must be an option ("-" alone and "--" ignored for this paragraph).
Thanks.
| Nothing is stopping you from treating ALL unrecognized options as a
| magic extension option that instead sources a file by that name, thus
| restoring back-compat behavior for all except '--' (which, as you say,
| would be unlikely to be a real issue).
OK, then good in that case I have no more problems.
| But I think you've overlooked the fact that you ARE allowed to have the
| extension behavior for all except '--' that preserves your goal of "if
| there is one argument, it is a filename even if it starts with '-'",
Overlooked .. not perhaps the right word, did not read XBD 2.12 that way
would be closer to it.
| while still remaining compliant to the standard's "'. -- "$arg"' must
| treat $arg as a filename even if it starts with '-'".
Yes, of course.
If there is no-one here who disagrees with Eric's reading of all of this,
then I'm happy (as I suggested in the first message, what I was looking for
was some confirmation that the way I was reading things was wrong).
And while I am here...
konrad.schw...@siemens.com said:
| I would often like to pass arguments to dot, like so,
| . ./myscript arg1 arg2 ...
| temporarily setting $0 $1 $2 ..., similar to shell functions.
| Currently, this is not permitted by POSIX.
That's wrong, and there are shells that do just that. There are also
shells that (whether they mean anything or not) process options to the "."
and exec commands. All that is (has been) compliant. No issues about
any of that. You don't need to invent an option to allow args to be handed
to the script. Nor does option processing need to be "allowed" - it already
is, it simply isn't required (wasn't).
kre
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Schwarz, Konrad wrote, on 04 Feb 2020: > > I would often like to pass arguments to dot, like so, > > . ./myscript arg1 arg2 ... > > temporarily setting $0 $1 $2 ..., similar to shell functions. > > Currently, this is not permitted by POSIX. It's an allowed extension. It is even mentioned in the RATIONALE that the KornShell does it and that it is a valid extension. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
On 2/4/20 6:56 AM, Robert Elz wrote: What I am at the very least unclear about, is that the way that this group chose to require "--" processing, appears to me to also require that any arg (before that "--" or an arg not starting with "-") which does start with "-" be treated as an option - which since we have no options (not sure if any shell does, I haven't encountered any) would *require* issuing an "invalid option" error. Implementations are allowed to provide any number of options as extensions. For example, the following is not strictly compliant: chmod -x file because it requires -x to be treated as an option, rather than as a mode string. The standard-conformant way to write that is: chmod -- -x file But in practice, most (or even all?) implementations have an extension where if the -x option is present, then an alternative command syntax where there is no explicit 'mode' argument is used (that is, historically, 'chmod -x' is an extension option implemented by most implementations which behaves as if you had called 'chmod -- -x'). And implementations would be crazy to NOT have such an extension, because so much (non-compliant) code exists in the wild that assumes the existence of this extension, even if POSIX does not require it. This means that if I were to do PATH=.:/whatever/else:/and/more echo "echo foo" > -x . -x after the new standard appears shells would be required to either do whatever their 'x' option to the "." command does, and then do whatever they do with a "." command without a filename, or to issue an error if they have no 'x' option (which I think is all shells). No, the standard still leaves plenty of latitude for your implementation of '.' to behave like 'chmod', where '-x' is an undocumented option that behaves as if it is a request for '.' to instead source the file named '-x' wherever it appears in the search path used by '.', as if the user had typed '. -- -x'. Until now, some shells did that (error from the unknown 'x' option) whereas others would output "foo" - that is, would source the "-x" file and run the echo command contained in it. Both behaviours were permitted by the standard, and both exist in the wild. While (as earlier stated) I have no idea if anyone ever actually uses that, I am not willing to break backward compatibility by preventing it from working, so even though treating the "--" in ". --" as an option ender, and not a file name, also breaks backward compat, that one I consider so unlikely, and anyone who relied upon it as so moronic, that I'll ignore that issue - doing the same for all file names that start with "-" is just too much. Nothing is stopping you from treating ALL unrecognized options as a magic extension option that instead sources a file by that name, thus restoring back-compat behavior for all except '--' (which, as you say, would be unlikely to be a real issue). So, the code in the NetBSD shell will ignore a "--" (more or less simply pretend it is not there) no other option processing, of any kind will be done (once it is committed). That seems compliant to me. Of course, I might be misreading the effect of the words: The dot special built-in shall support XBD Section 12.2 (on page 215). (and the same thing, with s/dot/exec/ for "exec"), and that no requirement to regonise ". -x" as being the 'x' option to the "." command is thereby imposed - but at least until that is explained to me (how we can use the words added to mean "--" must be supported, but "-x" need not be) I will be (once again) ignoring the standard. Alternatively, perhaps that wording could change to (something like) The dot special built-in shall support at least enough of XBD Section 12.2 (on page 215) to recognise and process the "--" string as specified there, however no other option processing is required. One complication for the "." command in particular, is that our shell (so a comment in its source says anyway) copies behaviour of some SysVR2 shell, and treats the command . (that is, the dot command without a filename) as identical to ":" - ie: all it does is set the exit status to 0.POSIX does not make the file name optional, so scripts should not use that, but, once again, I have no idea if, and if so, how many, scripts in the wild might, for some reason, be relying upon this, it after all allows one to write code like ... mv file1 long_name_1 mv file2 long_name_2 . . . which in a script makes no sense really, but in an example in a manual or book might ... and users might literally copy the whole thing (they do things just like that, I've seen it, often) - and currently with our shell (and dash, bosh, and pdksh - so maybe ksh88) it works just fine. Keeping that complicates all of this - otherwise I would simply implement "if there is exactly one arg it must be intended to be the required file name,
RE: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
> -Original Message- > From: Robert Elz > Sent: Tuesday, February 4, 2020 1:56 PM > To: Steffen Nurpmeso > Cc: austin-group-l@opengroup.org > Subject: Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax > Guidelines > > What I am at the very least unclear about, is that the way that this group > chose to require "--" processing, > appears to me to also require that any arg (before that "--" or an arg not > starting with "-") which does start > with "-" be treated as an option - which since we have no options (not sure > if any shell does, I haven't > encountered any) would > *require* issuing an "invalid option" error. I can think of a reason for requiring option processing in the dot command: I would often like to pass arguments to dot, like so, . ./myscript arg1 arg2 ... temporarily setting $0 $1 $2 ..., similar to shell functions. Currently, this is not permitted by POSIX. Allowing for option processing would enable future extension to allow such usage, e.g., . -x ./myscript arg1 arg2 ...
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Date:Mon, 03 Feb 2020 23:46:26 +0100 From:Steffen Nurpmeso Message-ID: <20200203224626.fhclq%stef...@sdaoden.eu> | I see, after having read the issue, it was from 2010. Yes, I only noticed it when I saw the message saying that the resolution (from way back then) had been applied. | as you possibly remember. Sorry, no, but that's likely because my memory is pathetic ... | But yes, of course, having -- is an improvement. I agree. With the way "." and "exec" are implemented by current shells, there is a real problem with path names that might start with a dash, and requiring the "--" be allowed to be used by scripts to avoid that problem is a simple solution. That we need to change our shell to support that is not a problem, like I said the code for it exists already (very unlikely to be in NetBSD 9, which is coming soon, but would be in 10 - whenever that happens). What I am at the very least unclear about, is that the way that this group chose to require "--" processing, appears to me to also require that any arg (before that "--" or an arg not starting with "-") which does start with "-" be treated as an option - which since we have no options (not sure if any shell does, I haven't encountered any) would *require* issuing an "invalid option" error. This means that if I were to do PATH=.:/whatever/else:/and/more echo "echo foo" > -x . -x after the new standard appears shells would be required to either do whatever their 'x' option to the "." command does, and then do whatever they do with a "." command without a filename, or to issue an error if they have no 'x' option (which I think is all shells). Until now, some shells did that (error from the unknown 'x' option) whereas others would output "foo" - that is, would source the "-x" file and run the echo command contained in it. Both behaviours were permitted by the standard, and both exist in the wild. While (as earlier stated) I have no idea if anyone ever actually uses that, I am not willing to break backward compatibility by preventing it from working, so even though treating the "--" in ". --" as an option ender, and not a file name, also breaks backward compat, that one I consider so unlikely, and anyone who relied upon it as so moronic, that I'll ignore that issue - doing the same for all file names that start with "-" is just too much. So, the code in the NetBSD shell will ignore a "--" (more or less simply pretend it is not there) no other option processing, of any kind will be done (once it is committed). Of course, I might be misreading the effect of the words: The dot special built-in shall support XBD Section 12.2 (on page 215). (and the same thing, with s/dot/exec/ for "exec"), and that no requirement to regonise ". -x" as being the 'x' option to the "." command is thereby imposed - but at least until that is explained to me (how we can use the words added to mean "--" must be supported, but "-x" need not be) I will be (once again) ignoring the standard. Alternatively, perhaps that wording could change to (something like) The dot special built-in shall support at least enough of XBD Section 12.2 (on page 215) to recognise and process the "--" string as specified there, however no other option processing is required. One complication for the "." command in particular, is that our shell (so a comment in its source says anyway) copies behaviour of some SysVR2 shell, and treats the command . (that is, the dot command without a filename) as identical to ":" - ie: all it does is set the exit status to 0.POSIX does not make the file name optional, so scripts should not use that, but, once again, I have no idea if, and if so, how many, scripts in the wild might, for some reason, be relying upon this, it after all allows one to write code like ... mv file1 long_name_1 mv file2 long_name_2 . . . which in a script makes no sense really, but in an example in a manual or book might ... and users might literally copy the whole thing (they do things just like that, I've seen it, often) - and currently with our shell (and dash, bosh, and pdksh - so maybe ksh88) it works just fine. Keeping that complicates all of this - otherwise I would simply implement "if there is exactly one arg it must be intended to be the required file name, whatever it looks like, otherwise..." and things would be a little simpler. kre ps: if it turns out I know someone in the balloting group for final approval of this, I'd suggest to them that they do not approve the new version while it contains the kind of incompatibility and breakage for no particularly good reason that seems to me to exist.
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Robert Elz wrote in <28486.1580735...@jinx.noi.kre.to>: |Date:Fri, 31 Jan 2020 23:18:31 +0100 |From:Steffen Nurpmeso |Message-ID: <20200131221831.vapcz%stef...@sdaoden.eu> | || May i ask whether you have numbers on how often "." is used with || $PATH searching? | |Of course you may ask. | |But to save you actually doing that, no, I have no idea, like many |other things in this area all that we really know is that it might |be, as that's how it was specified to work - even if none of us |usually ever use it that way. |But the relevant change here was identical for both "." and "exec" |and "exec" using PATH search is rather more likely. | || But, then: for explicit relative || file names, using ./ is a way to accomplish escaping. | |Of course, but if that were adequate for the original issue, the |reported problem would not have needed a change at all. That's |a different kettle of fish. I'm assuming (without any personal |evidence of it) that there actually is/was a problem to fix, and |then simply asking about whether the fix that was made perhaps over |does things a little (requires more than was strictly necessary). I see, after having read the issue, it was from 2010. I use functions which do the path search for me, you and Stephane have fixed bugs in it, as you possibly remember. This is because command -v is not really usable. So i, among others, have found ways to workaround issues. I have found only one script which does './exec "$shellvar"' where $shellvar is not a readily prepared path, and that is in a release script which is known to run locally. But yes, of course, having -- is an improvement. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Date:Fri, 31 Jan 2020 23:18:31 +0100 From:Steffen Nurpmeso Message-ID: <20200131221831.vapcz%stef...@sdaoden.eu> | May i ask whether you have numbers on how often "." is used with | $PATH searching? Of course you may ask. But to save you actually doing that, no, I have no idea, like many other things in this area all that we really know is that it might be, as that's how it was specified to work - even if none of us usually ever use it that way. But the relevant change here was identical for both "." and "exec" and "exec" using PATH search is rather more likely. | But, then: for explicit relative | file names, using ./ is a way to accomplish escaping. Of course, but if that were adequate for the original issue, the reported problem would not have needed a change at all. That's a different kettle of fish. I'm assuming (without any personal evidence of it) that there actually is/was a problem to fix, and then simply asking about whether the fix that was made perhaps over does things a little (requires more than was strictly necessary). kre
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Robert Elz wrote in <6289.1580506...@jinx.noi.kre.to>: |Date:Fri, 31 Jan 2020 21:56:25 +0100 |From:Steffen Nurpmeso |Message-ID: <20200131205625.lu2lw%stef...@sdaoden.eu> | || But the usual and portable way to escape a file starting with || a hyphen-minus is to use relative paths, as in ". ./-x"? | |No, that is an explicit file name, the original searched $PATH |to find a file "-x". | || In fact this is even necessary to get things done portably, for || example: | |That's merely illustrating that .profile is not in your $PATH. That's |an entirely different issue. May i ask whether you have numbers on how often "." is used with $PATH searching? You have of course trapped me regarding the paths search, but from a look into the OpenBSD source tree for example i see not one such use case (the ones using variables expand to paths, as far as i understand autotools-generated configure files). Interesting. But, then: for explicit relative file names, using ./ is a way to accomplish escaping. Files in $PATH with leading hyphen-minus i do not have around. They are also all executable.. Thank you. And a nice weekend i wish. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Re: [1003.1(2008)/Issue 7 0000252]: dot should follow Utility Syntax Guidelines
Date:Fri, 31 Jan 2020 14:55:24 + From:Austin Group Bug Tracker Message-ID: <970ec2df057f7ff8c00b28cfbf548...@austingroupbugs.net> I had never seen this bug before... | The following issue has a resolution that has been APPLIED. | == | https://austingroupbugs.net/view.php?id=252 For those unaware (everything about this issue was done 10 years ago), this requires that "." and "exec" special built in commands in sh handle the "--" option syntax. That I have no problem with, the NetBSD sh didn't do that, but I have just made it do so (not yet committed but the code exists). However, I have a question about the way that this was accomplished in the standard (this is for "." the change for "exec" is the same): At line 74031, add a new paragraph to the dot DESCRIPTION: The dot special built-in shall support XBD Section 12.2 (on page 215). As I understand it (and I am open to correction) this means that if run as . -x then the shell (assuming it hasn't implemented some 'x' option to the dot command, which none I have looked at have done) is now required to issue an "unknown option" error, and abort the dot command. That I cannot do, for backward compatibility - I don't think I'll have a problem with not processing the file "--" if found in a PATH search, but forbidding all files that start with "-" is going too far. I know a script could do ". -- -x" but that has never worked before (still doesn't in anyone's netbsd sh binary, except mine) so nothing is going to be doing it that way. I tested my usual set of shells, and find it split, some shells do option processing for "." (I have no idea if any support any options - what I mean is that they object to an unknown option) and others simply search for the arg in $PATH (and process it if found). The netbsd sh, freebsd sh, zsh, and bosh are in the second camp, bash, yash, dask, and the ksh shells are in the former. Assuming that my interpretation of the meaning of the change is correct, is there any way we can make a less intrusive change, which mandates "--" processing (and perhaps even that only if there is another arg following the "--" though that's less important) but does not require that "options" (args starting with '-') be treated as options, when no options exist, and in the past nothing has ever required that kind of processing. kre
