Re: feature request: echo --
Bob Proulx writes: > To be totally honest I thought echo was already very well documented. > And I thought we already were strongly recommending people to use > printf for arbitrary data. Therefore honestly I thought by agreeing I > thought we were done. I do not agree. The only recommendation to use printf that I found is in the "echo invocation" page of the coreutil info documentation. And this is what it says: POSIX does not require support for any options, and says that the behavior of ‘echo’ is implementation-defined if any STRING contains a backslash or if the first argument is ‘-n’. Portable programs can use the ‘printf’ command if they need to omit trailing newlines or output control characters or backslashes. *Note printf invocation::. And I believe that this paragraph misses the point. It suggests that the only issue with "echo" is a portability issue, and that only portable programs should use "printf". My point is that even without considering portability, users shouldn't use the echo command to process arbitrary strings. IMHO, the two notes sent by Glenn Golden and Frank Seifferth are much more accurate to describe the issue, and provide a real reason (besides portability) for programmers to use printf rather than echo. Also, as far as I could see, users tend to read only the "echo --help" message to grasp the usage of a command. Some of them may look at the man page to get more details. Only few read the info documentation. Therefore, to help the community to write robust programs, I believe that an accurate explanation of the echo command caveats should be displayed in the man page, if not in the --help message. I believe that good software deserves accurate documentation. These are the 2 great notes that describe the issue with the echo command. note by Glenn Golden: - NOTE For historical and back-compatibility reasons, certain bare option-like strings cannot be emitted without setting POSIXLY_CORRECT, and the bare string '-n' cannot be emitted at all. Prefixing or suffixing such strings with quoted whitespace (e.g. ' -n') can be used as a workaround for this peculiarity. More generally, printf(1) is recommended as a more modern and flexible replacement for tasks historically performed by echo(1). - note by Frank Seifferth: For historical and back-compatibility reasons, certain bare option-like strings cannot be passed to echo as non-option arguments. The only way to echo the string '-n', for instance, is to specify the dash in either octal or hexadecimal representation (e.g. 'echo -e "\x2dn"'). It is therefore not advisable to use echo(1) for printing unknown or variable arguments. More generally, printf(1) is recommended as a more modern and flexible replacement for tasks historically performed by echo(1). Regards, Florent
Re: feature request: echo --
Florent Flament wrote: > Bob Proulx writes: > >> In which case, thoroughly documenting the edge cases of the 'echo' > >> command and inviting programmers to use 'printf' instead on its > >> manpage (I know 'printf' is mentioned on the coreutils info page, > >> but it's one additional level of indirection) would probably be > >> helpful for the community. I'd gladly try to update the manpage if > >> that be the way to go. > > > > That's an excellent idea. I agree completely. We should document > > echo behavior and invite programmers to use printf instead. :-) > > Let's do that then. To be totally honest I thought echo was already very well documented. And I thought we already were strongly recommending people to use printf for arbitrary data. Therefore honestly I thought by agreeing I thought we were done. Bob
Re: feature request: echo --
Bob Proulx writes: > The "-n" in the "string" position is listed specifically as an > exception to the general rule previously stated. Basically an > interpretation might be don't implement getopt option processing in > general but look specifically at the first argument for this specific > case, it being implementation defined. Ok, this looks like a reasonable interpretation. Though in my mind, exceptions need to be more explicit (like in "Hey ho ! Beware ! This is a EXCEPTION ! ..."). Otherwise I tend to see "Don't do X. When doing X do Y" as an inconsistency. > Originally POSIX was not the design document it has become in later > years. Originally POSIX was an operating system feature > non-proliferation treaty trying to keep the differences among systems > from becoming even more different. At the time it was created there > were already two major flavors of echo on different systems. And it > was already a portability problem. Therefore the standard tried to > freeze the common feature set and allow the existing differences. I trust you on this. >> > Unix v7 echo accepted -n as an option and did not print it. Same >> > with BSD. It is too late to mess with it now. >> >> Then these implementations are broken as well. > > That point is similar to saying that one should choose their parents > better. It may be true! But it is pointless because one does not > have the possibility of choosing their parents. > > And similarly when echo was originally created it was done without the > benefit of 50 years of hindsight looking back at the problems the > choices made then have caused over the years and still continue to > cause now. > >> Also, I can understand that one may not want to change the behavior >> of a command that has been used for 30 years. > > That's the main point of the discussion. It's been 50 years and some > expectation of stability should be maintained. Ok, I understand your point. > On the other hand if people want to fix everything then I think they > should be free to create their own new operating system and do so > there. That's fine. New systems don't have legacy and don't need to > be backward compatible. So break whatever you want there. But don't > break the existing system for others along the way. I'm not there yet ; ) >> On a practical level, I have seen the 'echo' command being used in >> many shell scripts to process arbitrary strings. Which means that in >> some cases (when processing the strings "-n", "-e" or "-E") these >> scripts won't behave as expected by the programmer. > > Then those are the bugs to have fixed. Indeed, 'printf' for the win ! (Like in "if you can't fix something use something else" ; ) > I still use echo to this very day. > > echo "Hello world!" > > But that does not include any options and does not include any > backslash escape sequences. If it did or if there was any possibility > of it doing so then I would program it differently. > > printf "What do you want me to say? " > IFS= read -r phrase > echo $phrase # N! > > printf "What do you want me to say? " > IFS= read -r phrase > printf "%s\n" "$phrase" # Yes! > > The above shows input from an untrusted source. It's "tainted". It > should not be used without caution. Indeed, this looks like a fair use of echo and printf. >> That is also true, though chances are that the shell builtin >> implementations of 'echo' be probably inspired (if not copied) from >> the GNU echo implementation. > > Actually no. Their behaviors were based upon existing echo > implementations that pre-date GNU echo that GNU echo was itself also > based upon. Ok, I trust you on this one as well. >> What I would expect from a good implementation of the 'echo' command >> is to interpret the '--' argument to signify the end of the options, >> therefore removing any ambiguity in the interpretation of the strings >> to display. That would be the (non-posix) default behavior. Then the >> command would have a POSIX compliant behavior with the POSIXLY_CORRECT >> flag, which wouldn't interpret any option at all. > > Please no. That would create yet another different behavior case. It > would be yet again one of the different odd unusual unique things that > would need to be programmed around with horrid code. Plus we have > already been there and done that. The result of which was "printf" > which was created as the answer to the problem. Fair enough. >> Besides, I understand that the current implementation of the GNU >> 'echo' command has been used for several decades now, and that it >> may be difficult to change its behavior. > > Yes. A very long time. Try 50 years of "echo"! Didn't think an issue could last that long ; ) >> In which case, thoroughly documenting the edge cases of the 'echo' >> command and inviting programmers to use 'printf' instead on its >> manpage (I know 'printf' is mentioned on the coreutils info page, >> but it's one additional level of
Re: feature request: echo --
Florent Flament wrote: > Bob Proulx wrote: > However, I believe that the use of "shall not" makes the POSIX echo > definition ambiguous: > > The echo utility shall not recognize the "--" argument in the > manner specified by Guideline 10 of XBD Utility Syntax Guidelines; > "--" shall be recognized as a string operand. > > Implementations shall not support any options. That seems fairly unambiguous to me. > In the case of '--', "shall not" seems to mean "is forbidden to", > because '--' "must" be recognized as a string operand. In the case of > options, "shall not" seems to mean "don't have to", because it is then > mentioned that the behavior is implementation dependent when using the > '-n' flag. The "-n" in the "string" position is listed specifically as an exception to the general rule previously stated. Basically an interpretation might be don't implement getopt option processing in general but look specifically at the first argument for this specific case, it being implementation defined. > That said, I believe that the POSIX echo definition is broken, because > it tried to reconcile incompatible existing implementations. Originally POSIX was not the design document it has become in later years. Originally POSIX was an operating system feature non-proliferation treaty trying to keep the differences among systems from becoming even more different. At the time it was created there were already two major flavors of echo on different systems. And it was already a portability problem. Therefore the standard tried to freeze the common feature set and allow the existing differences. > > Unix v7 echo accepted -n as an option and did not print it. Same > > with BSD. It is too late to mess with it now. > > Then these implementations are broken as well. That point is similar to saying that one should choose their parents better. It may be true! But it is pointless because one does not have the possibility of choosing their parents. And similarly when echo was originally created it was done without the benefit of 50 years of hindsight looking back at the problems the choices made then have caused over the years and still continue to cause now. > Also, I can understand that one may not want to change the behavior > of a command that has been used for 30 years. That's the main point of the discussion. It's been 50 years and some expectation of stability should be maintained. On the other hand if people want to fix everything then I think they should be free to create their own new operating system and do so there. That's fine. New systems don't have legacy and don't need to be backward compatible. So break whatever you want there. But don't break the existing system for others along the way. > > On a practical level why is it an issue at all? If there is any > > concern about printing -n then use 'printf' as that is a much better > > solution for arbitrary strings. > > On a practical level, I have seen the 'echo' command being used in > many shell scripts to process arbitrary strings. Which means that in > some cases (when processing the strings "-n", "-e" or "-E") these > scripts won't behave as expected by the programmer. Then those are the bugs to have fixed. > I agree that 'printf' should have been used instead, but it seems > that programmers have been taught to use 'echo' to print strings in > shell scripts for many years (that's my case as well). I still use echo to this very day. echo "Hello world!" But that does not include any options and does not include any backslash escape sequences. If it did or if there was any possibility of it doing so then I would program it differently. printf "What do you want me to say? " IFS= read -r phrase echo $phrase # N! printf "What do you want me to say? " IFS= read -r phrase printf "%s\n" "$phrase" # Yes! The above shows input from an untrusted source. It's "tainted". It should not be used without caution. > > Also note that most shells include 'echo' as a builtin command which > > will be different from the coreutils standalone executable. Most > > users should look to their shell for their echo implementation > > instead. > > That is also true, though chances are that the shell builtin > implementations of 'echo' be probably inspired (if not copied) from > the GNU echo implementation. Actually no. Their behaviors were based upon existing echo implementations that pre-date GNU echo that GNU echo was itself also based upon. > What I would expect from a good implementation of the 'echo' command > is to interpret the '--' argument to signify the end of the options, > therefore removing any ambiguity in the interpretation of the strings > to display. That would be the (non-posix) default behavior. Then the > command would have a POSIX compliant behavior with the POSIXLY_CORRECT > flag, which wouldn't interpret any option at all. Please no. That would create yet another
Re: feature request: echo --
On Tue, 2021-10-19 at 01:06 -0600, Bob Proulx wrote: > Florent Flament wrote: > > Out of curiosity, would it possible to have the `echo` command > > output the string "-n" ? > > > > ``` > > $ POSIXLY_CORRECT=1 /bin/echo -n > > ``` > > But the standards do actually mention -n. The behavior you see with > POSIXLY_CORRECT=1 is conforming behavior. > > https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html#tag_20_37 > > If the first operand is -n, or if any of the operands contain a > character, the results are implementation-defined. I agree that the GNU echo command complies with the sentence mentioned above, when using the POSIXLY_CORRECT flag. However, I believe that the use of "shall not" makes the POSIX echo definition ambiguous: The echo utility shall not recognize the "--" argument in the manner specified by Guideline 10 of XBD Utility Syntax Guidelines; "--" shall be recognized as a string operand. Implementations shall not support any options. In the case of '--', "shall not" seems to mean "is forbidden to", because '--' "must" be recognized as a string operand. In the case of options, "shall not" seems to mean "don't have to", because it is then mentioned that the behavior is implementation dependent when using the '-n' flag. That said, I believe that the POSIX echo definition is broken, because it tried to reconcile incompatible existing implementations. The two different historical versions of echo vary in fatally incompatible ways. > Unix v7 echo accepted -n as an option and did not print it. Same > with BSD. It is too late to mess with it now. Then these implementations are broken as well. Also, I can understand that one may not want to change the behavior of a command that has been used for 30 years. > On a practical level why is it an issue at all? If there is any > concern about printing -n then use 'printf' as that is a much better > solution for arbitrary strings. On a practical level, I have seen the 'echo' command being used in many shell scripts to process arbitrary strings. Which means that in some cases (when processing the strings "-n", "-e" or "-E") these scripts won't behave as expected by the programmer. I agree that 'printf' should have been used instead, but it seems that programmers have been taught to use 'echo' to print strings in shell scripts for many years (that's my case as well). > Also note that most shells include 'echo' as a builtin command which > will be different from the coreutils standalone executable. Most > users should look to their shell for their echo implementation > instead. That is also true, though chances are that the shell builtin implementations of 'echo' be probably inspired (if not copied) from the GNU echo implementation. What I would expect from a good implementation of the 'echo' command is to interpret the '--' argument to signify the end of the options, therefore removing any ambiguity in the interpretation of the strings to display. That would be the (non-posix) default behavior. Then the command would have a POSIX compliant behavior with the POSIXLY_CORRECT flag, which wouldn't interpret any option at all. Besides, I understand that the current implementation of the GNU 'echo' command has been used for several decades now, and that it may be difficult to change its behavior. In which case, thoroughly documenting the edge cases of the 'echo' command and inviting programmers to use 'printf' instead on its manpage (I know 'printf' is mentioned on the coreutils info page, but it's one additional level of indirection) would probably be helpful for the community. I'd gladly try to update the manpage if that be the way to go. Regards, Florent Flament
Re: feature request: echo --
Florent Flament wrote: > Out of curiosity, would it possible to have the `echo` command output > the string "-n" ? > > ``` > $ POSIXLY_CORRECT=1 /bin/echo -n > ``` But the standards do actually mention -n. The behavior you see with POSIXLY_CORRECT=1 is conforming behavior. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html#tag_20_37 If the first operand is -n, or if any of the operands contain a character, the results are implementation-defined. Unix v7 echo accepted -n as an option and did not print it. Same with BSD. It is too late to mess with it now. On a practical level why is it an issue at all? If there is any concern about printing -n then use 'printf' as that is a much better solution for arbitrary strings. Also note that most shells include 'echo' as a builtin command which will be different from the coreutils standalone executable. Most users should look to their shell for their echo implementation instead. Bob
Re: feature request: echo --
On Fri, 2021-10-15 at 23:37 +0100, Philip Rowlands wrote: > On Fri, 15 Oct 2021, at 18:42, Roger Pack wrote: > > It came to my attention recently that it seems not possible to > > "echo" the > > string "-e" > > $ echo "-e" > > Nitpick: the double quotes aren't doing anything here. > > > Perhaps echo could add a "--" style param like > > > > $ echo -- -e > > These are both covered by the documentation > https://www.gnu.org/software/coreutils/manual/html_node/echo-invocation.html > > "... the normally-special argument ‘--’ has no special meaning and is > treated like any other string." > > "If the POSIXLY_CORRECT environment variable is set, then when echo’s > first argument is not -n it outputs option-like arguments instead of > treating them as options." > > Thus the way to make this work is: > > $ POSIXLY_CORRECT=1 /bin/echo -e > -e > > > Cheers, > Phil Hi, Out of curiosity, would it possible to have the `echo` command output the string "-n" ? ``` $ POSIXLY_CORRECT=1 /bin/echo -n ``` Regards, Florent Flament
Re: feature request: echo --
I just use printf "%s\n" "-e"-e Regards Leslie Leslie Satenstein Montréal Québec, Canada On Friday, October 15, 2021, 06:38:19 p.m. GMT-4, Philip Rowlands wrote: On Fri, 15 Oct 2021, at 18:42, Roger Pack wrote: > It came to my attention recently that it seems not possible to "echo" the > string "-e" > $ echo "-e" Nitpick: the double quotes aren't doing anything here. > Perhaps echo could add a "--" style param like > > $ echo -- -e These are both covered by the documentation https://www.gnu.org/software/coreutils/manual/html_node/echo-invocation.html "... the normally-special argument ‘--’ has no special meaning and is treated like any other string." "If the POSIXLY_CORRECT environment variable is set, then when echo’s first argument is not -n it outputs option-like arguments instead of treating them as options." Thus the way to make this work is: $ POSIXLY_CORRECT=1 /bin/echo -e -e Cheers, Phil
Re: feature request: echo --
On 15/10/2021 23:37, Philip Rowlands wrote: On Fri, 15 Oct 2021, at 18:42, Roger Pack wrote: It came to my attention recently that it seems not possible to "echo" the string "-e" $ echo "-e" Nitpick: the double quotes aren't doing anything here. Perhaps echo could add a "--" style param like $ echo -- -e These are both covered by the documentation https://www.gnu.org/software/coreutils/manual/html_node/echo-invocation.html "... the normally-special argument ‘--’ has no special meaning and is treated like any other string." "If the POSIXLY_CORRECT environment variable is set, then when echo’s first argument is not -n it outputs option-like arguments instead of treating them as options." Thus the way to make this work is: $ POSIXLY_CORRECT=1 /bin/echo -e -e Also worth mentioning that printf(1) is preferable. It has less edge cases and portability issues. cheers, Pádraig
Re: feature request: echo --
On Fri, 15 Oct 2021, at 18:42, Roger Pack wrote: > It came to my attention recently that it seems not possible to "echo" the > string "-e" > $ echo "-e" Nitpick: the double quotes aren't doing anything here. > Perhaps echo could add a "--" style param like > > $ echo -- -e These are both covered by the documentation https://www.gnu.org/software/coreutils/manual/html_node/echo-invocation.html "... the normally-special argument ‘--’ has no special meaning and is treated like any other string." "If the POSIXLY_CORRECT environment variable is set, then when echo’s first argument is not -n it outputs option-like arguments instead of treating them as options." Thus the way to make this work is: $ POSIXLY_CORRECT=1 /bin/echo -e -e Cheers, Phil
