Re: Word splitting for $@ in variable assignment

[Ilkka Virta]

On Thu, Jun 24, 2021 at 5:20 PM Chet Ramey  wrote:

> On 6/24/21 4:09 AM, Ilkka Virta wrote:
>
> > But 3.4 Shell Parameters is a bit confusing: "Word splitting is not
> > performed, with the exception of "$@" as explained below."
>
> This means that "$@" expands to multiple words, even though double quotes
> would usually inhibit that.
>

Like Nora already mentioned, that quote is from the paragraph about scalar
variable assignment, where it doesn't appear to
produce more than one word, but instead behaves exactly as described in
Special Parameters for the case where word-
splitting does _not_ happen.

As far as I can tell, the behaviour is the same as when $@ is used in the
tested word or one of the patterns in a 'case'
statement or inside [[. The description for 'case' omits any mention of
word splitting, and the description for [[ explicitly
mentions it's not done, but neither/none of those mention any exceptions.

As an aside, the description of $* could perhaps also be changed to also
mention those non-word-splitting contexts, and not
only quoted and unquoted contexts, since in all the non-splitting cases
mentioned above, $* seems to expand to just
a single word even if it's not quoted. (Using the first character of IFS as
joiner, of course).

Re: Word splitting for $@ in variable assignment

[Nora Platiel]

On 2021-06-26 01:05, Robert Elz wrote:
>   | If Chet feels that a change is needed here, I would remove the "with
>   | the exception of" clause entirely.  Just say "Word splitting is not
>   | performed."
> 
> My suspicion (I'm guessing) is that exception is to cover
> 
> x=("$@")

`name=[value]' as described under 3.4 Shell Parameters doesn't include arrays.
That case is documented under 6.7 Arrays:

| name=(value1 value2 … )
| [...]
| Each value in the list undergoes all the shell expansions described
| above (see Shell Expansions).

Which includes word splitting, and not only with "$@" but with anything (e.g. 
`x=($y)' is also subjected to word splitting, while `x=$y' isn't).

In the context of `name=[value]' I don't see any exception.

> where the "$@" is expanded to multiple words (but still this isn't word
> splitting).

Yes, "$@" expanding to multiple words is not "word splitting", but it happens 
in contexts where word splitting is performed.
In contexts where word splitting is not performed it expands to a single word.

Re: Word splitting for $@ in variable assignment

[Nora Platiel]

On 2021-06-25 12:11, Greg Wooledge wrote:
> > | A variable may be assigned to by a statement of the form
> > | name=[value]
> > | [...]
> > | Word splitting is not performed, with the exception of "$@" as
> > | explained below.
>
> This isn't the full sentence in the current man page.  In the bash 5.1
> man page, it says:
>
>Word splitting is not
>performed, with the exception of "$@" as explained below under  Special
>Parameters.

Ok, so I quoted the web version which is slightly out of date, but the meaning 
of "below" was already clear to me. I linked to that part in my previous 
message.

I repeat for the 3rd time: the part under Special Parameters is already clear 
to me and I see no problem with it.
The part I have a problem with is `with the exception of "$@"'.

Try to follow my reasoning and tell me where our point of divergence is:
(This is all in the context of `name=[value]'.)

- doc: "word splitting is not performed, with the exception of "$@" [...]"

therefore

- in case of "$@" (as an exception), word splitting *is* performed

therefore

- the behavior of var="$@" should be the one described under Special Parameters 
when word splitting *is* performed

- but the behavior of var="$@" is actually the one described under Special 
Parameters when word splitting is *not* performed

therefore

- even for "$@", word splitting is *not* performed

therefore

- there is no exception: word splitting is *not* performed, period.


> If Chet feels that a change is needed here, I would remove the "with
> the exception of" clause entirely.  Just say "Word splitting is not
> performed."

This is what I've been proposing from the start. Because "with the exception 
of" makes it logically incorrect.

Re: Word splitting for $@ in variable assignment

[Robert Elz]

Date:Fri, 25 Jun 2021 12:11:33 -0400
From:Greg Wooledge 
Message-ID:  

  | That matches the behavior that I saw (and pasted on this mailing list the
  | other day).  (Which by the way is *not* the same as "$*" unless IFS happens
  | to be unset or to begin with a space.)

Hmm, never saw that difference before, I wonder why - mksh and ksh93 do
it that way as well.   Other shells (except bosh, which is simply broken
in this area, but might be intended to be the same as bash) - that is
all the ash descendents (including dash), plus yash, and zsh, all treat $@
as being just the same as $* in contexts where only one word is to be
produced.   After all, why not, reuse the code...

POSIX just says (or will say if it doesn't already) it is unspecified,
so either is OK (and applications simply should not do that).

Either way the quotes in x="$@" (or x="$*") do nothing at all, as is true
of any other var assign, except where the value to be assigned (as written
in the script or command line) contains a sh magic character (operator,
quoting char, white space).

  | If Chet feels that a change is needed here, I would remove the "with
  | the exception of" clause entirely.  Just say "Word splitting is not
  | performed."

My suspicion (I'm guessing) is that exception is to cover

x=("$@")

where the "$@" is expanded to multiple words (but still this isn't word
splitting).   (Here "value" in the man page is ("$@") (including the
parentheses and quotes) in the example).

kre

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Fri, Jun 25, 2021 at 03:15:25PM +0200, Nora Platiel wrote:
> But at least it is documented, and that part of the doc is clear to me.
> The part of the doc that I'm complaining about is this one:
> 
> | A variable may be assigned to by a statement of the form
> | name=[value]
> | [...]
> | Word splitting is not performed, with the exception of "$@" as
> | explained below.

This isn't the full sentence in the current man page.  In the bash 5.1
man page, it says:

   Word splitting is not
   performed, with the exception of "$@" as explained below under  Special
   Parameters.

So, we need to look at Special Parameters:

   @  Expands to the positional parameters,  starting  from  one.   In
  contexts  where  word  splitting is performed, this expands each
  positional parameter to a separate word; if  not  within  double
  quotes,  these words are subject to word splitting.  In contexts
  where word splitting is not performed, this expands to a  single
  word  with each positional parameter separated by a space.

That matches the behavior that I saw (and pasted on this mailing list the
other day).  (Which by the way is *not* the same as "$*" unless IFS happens
to be unset or to begin with a space.)

If Chet feels that a change is needed here, I would remove the "with
the exception of" clause entirely.  Just say "Word splitting is not
performed."

Re: Word splitting for $@ in variable assignment

[Robert Elz]

Date:Thu, 24 Jun 2021 21:03:22 -0400
From:Greg Wooledge 
Message-ID:  

  | Bash is ALL about these special cases.  If you don't like it, don't write
  | code that uses it.  In any sensible programming language, var="$@"
  | would have been an error.  In bash, it's not.  But that doesn't mean
  | you have to *write* it.

It isn't bash, it is Bourne sh (of which bash is an implementation, with
extensions) - there are two distinct contexts where expansions happen - one
where multiple words are expected (pathname expansions etc happen) and the
other where only one word is rational.

The rules are different for the two cases.   "$@" is simply "$*" in the
latter case (so you're right, it is kind of dumb to write it, and
simply creates confusion).

I would however never call what "$@" does (in the former case) "word
splitting", that isn't what happens at all, nothing is ever split, there
are simply some number ($#) of input words, which are individually quoted
and inserted in whatever list is being created.   This one is really a
very simple thing to understand, it is all of the other $@ and $* expansions
that are the weird(er) ones.

kre

ps: "word" above is in the sh syntax meaning, nothing to do with
natural language words.

Re: Word splitting for $@ in variable assignment

[Nora Platiel]

On 2021-06-25 7:36, Oğuz wrote:
> > To me, "$@" expanding to multiple words would mean that:
> >
> > $ var="$@" foo
> >
> > for $# > 0, behaves the same as:
> >
> > $ var="$1" "${@:2}" foo
> 
> But it wouldn't make any sense. `foo' would be subjected to alias
> substitution even though it isn't known for sure to be a command name. How
> would that work?

I'm not saying that `var="$@" foo' should behave the same as `var="$1" "${@:2}" 
foo', I'm just showing what I think it means to "expand to multiple words".
I know that `foo' is subjected to alias substitution in the former case but not 
in the latter (regardless of how many parameters we have).

Re: Word splitting for $@ in variable assignment

[Nora Platiel]

On 2021-06-24 21:03, Greg Wooledge wrote:
> "$@" expands like "$1 "$2" ... when used in most contexts.  For example,

Yes, that is clear to me.

> The assignment-ness of the command overrides everything else.  It
> completely changes how the "$@" expansion occurs.  Now, instead of
> expanding like "$1" "$2" ... it expands like "$1 $2 ...".  It's
> completely unique.  "$@" does not act like that in any other context.

There are others, like `case "$@" in'.

The doc makes a clear distinction about contexts where word splitting is 
performed and contexts where word splitting is not performed.
In the former case "$@" is multiple words while in the latter case it is a 
single word.

https://www.gnu.org/software/bash/manual/html_node/Special-Parameters.html

> Bash is ALL about these special cases.  If you don't like it, don't write
> code that uses it.  In any sensible programming language, var="$@"
> would have been an error.  In bash, it's not.  But that doesn't mean
> you have to *write* it.

I can see why you think that allowing `@' only in contexts where word splitting 
is performed would have been a better design. And I'm not encouraging the use 
of `@' in contexts where word splitting is not performed, since we have `*' 
which makes more sense there.

But at least it is documented, and that part of the doc is clear to me.
The part of the doc that I'm complaining about is this one:

| A variable may be assigned to by a statement of the form
| name=[value]
| [...]
| Word splitting is not performed, with the exception of "$@" as
| explained below.

https://www.gnu.org/software/bash/manual/html_node/Shell-Parameters.html

In the context of `name=[value]' word splitting is not performed, period, no 
exceptions.
That's why var="$@" behaves as described in the paragraph about $@ under the 
no-word-splitting case (i.e. "$@" is a single word).

Regards,
NP

Re: Word splitting for $@ in variable assignment

[Oğuz]

25 Haziran 2021 Cuma tarihinde Nora Platiel  yazdı:
>
> To me, "$@" expanding to multiple words would mean that:
>
> $ var="$@" foo
>
> for $# > 0, behaves the same as:
>
> $ var="$1" "${@:2}" foo
>

But it wouldn't make any sense. `foo' would be subjected to alias
substitution even though it isn't known for sure to be a command name. How
would that work?


-- 
Oğuz

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Fri, Jun 25, 2021 at 01:47:01AM +0200, Nora Platiel wrote:
> To me, "$@" expanding to multiple words would mean that:
> 
> $ var="$@" foo
> 
> for $# > 0, behaves the same as:
> 
> $ var="$1" "${@:2}" foo
> 
> which is obviously not the case.

"$@" expands like "$1 "$2" ... when used in most contexts.  For example,

foo --bar "$@"

passes the script's arguments along to foo, exactly as received, with
the extra --bar argument in front of them.

Likewise,

args=("$@")

stores the script's arguments as the elements of an indexed array.  Each
argument is retained as a separate word, becoming one element of the
new array.

var="$@" is simply a special case.  It's an assignment *first*, and it's
parsed as such.

The assignment-ness of the command overrides everything else.  It
completely changes how the "$@" expansion occurs.  Now, instead of
expanding like "$1" "$2" ... it expands like "$1 $2 ...".  It's
completely unique.  "$@" does not act like that in any other context.

Bash is ALL about these special cases.  If you don't like it, don't write
code that uses it.  In any sensible programming language, var="$@"
would have been an error.  In bash, it's not.  But that doesn't mean
you have to *write* it.

Re: Word splitting for $@ in variable assignment

[Nora Platiel]

I was also confused by the same statement.

On 2021-06-24 10:20 Chet Ramey wrote:
> > But 3.4 Shell Parameters is a bit confusing: "Word splitting is not
> > performed, with the exception of "$@" as explained below."
>
> This means that "$@" expands to multiple words, even though double quotes
> would usually inhibit that.

To me, "$@" expanding to multiple words would mean that:

$ var="$@" foo

for $# > 0, behaves the same as:

$ var="$1" "${@:2}" foo

which is obviously not the case.

I don't know how this is implemented, maybe they are actually split into 
multiple words, and such words somehow remain in the context of the assignment 
and are rejoined later.
But the user doesn't see that. The user just see that on the RHS of an 
assignment, "$@" produces a single string (by joining arguments), and such 
string is not split in any way.

I second the proposal of removing the part: `with the exception of "$@" as 
explained below', because I think it serves no useful purpose and adds 
confusion.

Regards,
NP

Re: Word splitting for $@ in variable assignment

[Chet Ramey]

On 6/24/21 4:09 AM, Ilkka Virta wrote:


But 3.4 Shell Parameters is a bit confusing: "Word splitting is not
performed, with the exception of "$@" as explained below." 


This means that "$@" expands to multiple words, even though double quotes
would usually inhibit that.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: Word splitting for $@ in variable assignment

[Greg Wooledge]

On Thu, Jun 24, 2021 at 05:52:47PM +1000, Alvin Seville wrote:
>  Hello! I want to understand why the following code doesn't produce any
> error:
> set -- "a b" "c"
> a="$@"

It doesn't produce an error because the designers of the shell tried
to make it "user-friendly" by having it guess what you really want,
whenever you do silly things.  Sometimes it guesses correctly.  Sometimes
not.

In this case, you've written code that makes no sense to an experienced
programmer, because on the surface you're assigning a vector (list, array)
value to a scalar (string) variable.

But the shell decided it would try to "help" you by assuming you really
meant a="$*".  Except, it's not actually that simple, and it just gets
more and more ugly the deeper you dive into it.

> ? I expected smth like: main.sh: line 2: b: command not found due to word
> splitting according to documentation
> .
> What am I missing?

Word splitting does not occur on the right hand side of a string variable
assignment.  That's why you can do this:

a='two words'
b=$a

This is well-defined behavior, completely portable across every Bourne-type
shell ever written.

The difference between this and your case (a="$@") is that "$@" doesn't
actually involve word splitting.  It's like... the opposite of word
splitting.  Word retention, maybe?  It doesn't have a name.

But it doesn't make sense for a retained list of words to be assigned to
a string variable.  There has to be a single string that gets assigned.
So the shell tries to come up with a result that it thinks will make
sense.

What it ends up doing is taking the list of words generated by "$@" and
concatenating them all into a single string with spaces between them.
It does NOT use the contents of IFS -- it simply chooses "space" as
the separator.

unicorn:~$ bash
unicorn:~$ IFS=x
unicorn:~$ set -- "a b" c
unicorn:~$ a="$*"; echo "$a"
a bxc
unicorn:~$ a="$@"; echo "$a"
a b c

This surprising leniency is one of the reasons there are so many terrible
shell scripts (and shell script writers) in the world.  The shell lets you
do many ridiculous and even dangerous things, with no warnings of any kind.

Re: Word splitting for $@ in variable assignment

[Ilkka Virta]

On Thu, Jun 24, 2021 at 10:53 AM Alvin Seville 
wrote:

>  Hello! I want to understand why the following code doesn't produce any
> error:
> set -- "a b" "c"
> a="$@"
> ? I expected smth like: main.sh: line 2: b: command not found due to word
> splitting according to documentation
> <
> https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameters
> >.
> What am I missing?
>

It's a bit oddly put in the manual. It doesn't actually get word-split as
such, it just gives the individual positional parameters joined with spaces
to a single string.

3.4.2 Special Parameters explains that: "$@: In contexts where word
splitting is not performed, this expands to a single word with each
positional parameter separated by a space."

But 3.4 Shell Parameters is a bit confusing: "Word splitting is not
performed, with the exception of "$@" as explained below." I'm not sure if
this is meant to refer to that sentence 3.4.2, but it's a bit confusing
since that one refers to cases without word splitting. It would probably be
clearer to not call it an exception and just refer to Special Parameters
for how $@ behaves when not word-split.

Anyway, it may be better to use a="$*" as it's treated more consistently
between different shells. a="$@" uses the first character of IFS as the
separator in some shells (Zsh, Busybox, Debian/Ubuntu's Dash), and a space
always in others (Bash, Ksh), while a="$*" consistently uses the first
character of IFS. With the default IFS, the result is of course the same.

Word splitting for $@ in variable assignment

[Alvin Seville]

 Hello! I want to understand why the following code doesn't produce any
error:
set -- "a b" "c"
a="$@"
? I expected smth like: main.sh: line 2: b: command not found due to word
splitting according to documentation
.
What am I missing?

-- 
alvinseville7cf@Alvins-MacBook-Pro ~ $* echo *-e "Best regards, \e[33m$(
whoami)\e[0m."* && exit *$SUCCESS_EC