Re: More fun with IFS
On Friday, March 01, 2013 01:06:27 PM Thorsten Glaser wrote: > Hrm, but the docs, both, specifically say that (unquoted) $@ behaves > like $* except in the face of no arguments, so I cannot do that. > > But thanks for the feedback. My reading differed, but you have a > point, and the others can be kept as-is. > I think the root of the problem is trying to force unquoted $@ to be like $* instead of the other way around. That's how bash (if not for the bug) and ksh93 manage to do this while remaining consistent with the spec. -- Dan Douglas
Re: More fun with IFS
Dan Douglas dixit: >For "$@" that sounds about right. I think it would be preferable if x="$@" and >x=$@ were the same. If a user wants IFS-delimited they should probably use Hrm, but the docs, both, specifically say that (unquoted) $@ behaves like $* except in the face of no arguments, so I cannot do that. But thanks for the feedback. My reading differed, but you have a point, and the others can be kept as-is. bye, //mirabilos -- FWIW, I'm quite impressed with mksh interactively. I thought it was much *much* more bare bones. But it turns out it beats the living hell out of ksh93 in that respect. I'd even consider it for my daily use if I hadn't wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh
Re: More fun with IFS
On Friday, March 01, 2013 11:49:37 AM Thorsten Glaser wrote:
> Dan Douglas dixit:
>
> >Well, ok then. I'm just nitpicking here. I think this makes sense because
> >distinguishes between $@ and $* when assigning to a scalar, so that the end
> >result of $@ is always space-separated, as spaces delimit words during
> […]
> >Consider for example if you ever implement "${@@Q}". Because of this
>
> Hrm, you do have a point there. Now, how to do it for consistency…
> re-reading the manual snippet:
[…]
> This means “"$*"” needs to be “one:::two:three:::four”, and POSIX
> doesn’t say anything about “$*” or “$@”… the manpage says “separate
> words (which are subjected to word splitting)”, so I’d say we need
> “one:::two:three:::four” for “$*” and thus also “$@”.
>
> So basically, only “"$@"” needs to change to implement what comes
> out of your point, and the other three cases need to stay the same.
>
> Do you agree?
In the case of $* I don't really care. It's probably going to differ from Bash
no matter what because as pointed out earlier in this thread, Bash expands
unquoted $* to multiple words even with a null IFS, whereas mksh doesn't (if
multiple words result it's from word splitting). “one:::two:three:::four”
makes sense for the way mksh is doing it.
For "$@" that sounds about right. I think it would be preferable if x="$@" and
x=$@ were the same. If a user wants IFS-delimited they should probably use
x=$* because that's what it's for. Even if you decide not to do that, at least
there will be one way to get that outcome, and people will just have to know
that $@ behaves like $* when unquoted even in contexts without word splitting.
--
Dan Douglas
Re: More fun with IFS
Dan Douglas dixit:
>Well, ok then. I'm just nitpicking here. I think this makes sense because it
>distinguishes between $@ and $* when assigning to a scalar, so that the end
>result of $@ is always space-separated, as spaces delimit words during command
[…]
>Consider for example if you ever implement "${@@Q}". Because of this behavior,
Hrm, you do have a point there. Now, how to do it for consistency…
re-reading the manual snippet:
* All positional parameters (except 0), i.e. $1, $2, $3, ...
If used outside of double quotes, parameters are separate words
(which are subjected to word splitting); if used within double
quotes, parameters are separated by the first character of the
IFS parameter (or the empty string if IFS is NULL).
@ Same as $*, unless it is used inside double quotes, in which case
a separate word is generated for each positional parameter. If
there are no positional parameters, no word is generated. $@ can
be used to access arguments, verbatim, without losing NULL argu-
ments or splitting arguments with spaces.
So this basically means, for “"$@"” generate
“one:::two three:::four” so it can be used verbatim.
*
Expands to the positional parameters, starting from one.
When the expansion occurs within a double-quoted string
(see [54]Double-Quotes ), it shall expand to a single
field with the value of each parameter separated by the
first character of the IFS variable, or by a if
IFS is unset. If IFS is set to a null string, this is not
equivalent to unsetting it; its first character does not
exist, so the parameter values are concatenated.
This means “"$*"” needs to be “one:::two:three:::four”, and POSIX
doesn’t say anything about “$*” or “$@”… the manpage says “separate
words (which are subjected to word splitting)”, so I’d say we need
“one:::two:three:::four” for “$*” and thus also “$@”.
So basically, only “"$@"” needs to change to implement what comes
out of your point, and the other three cases need to stay the same.
Do you agree?
>> usually abstract everything eval into little functions of their
>> own and then just use those.
>I agree that's an excellent strategy. :)
;-)
bye,
//mirabilos
--
Support mksh as /bin/sh and RoQA dash NOW!
‣ src:bash (257 (276) bugs: 0 RC, 178 (192) I&N, 79 (84) M&W, 0 (0) F&P)
‣ src:dash (84 (98) bugs: 3 RC, 39 (43) I&N, 42 (52) M&W, 0 F&P)
‣ src:mksh (1 bug: 0 RC, 0 I&N, 1 M&W, 0 F&P)
http://qa.debian.org/data/bts/graphs/d/dash.png is pretty red, innit?
Re: More fun with IFS
On Wednesday, February 27, 2013 01:31:58 PM Thorsten Glaser wrote:
> Why whitespace? $IFS certainly contains none. And the usual
> insertion rules all specify the first character of $IFS and
> specify what to do if $IFS is empty or unset (which it isn’t
> in these examples).
Well, ok then. I'm just nitpicking here. I think this makes sense because it
distinguishes between $@ and $* when assigning to a scalar, so that the end
result of $@ is always space-separated, as spaces delimit words during command
parsing. Your way would make more sense to me if this were the Bourne shell
where IFS is in charge of both the initial argument splitting and field
splitting. In this case though it seems strange to use IFS to represent
separate words.
Consider for example if you ever implement "${@@Q}". Because of this behavior,
the integrity of the result can only be guaranteed with a default IFS during
assignment. This can be demonstrated with zsh which implements the same
expansion (with different syntax) and uses the same assignment rules as mksh.
$ zsh -s <<\EOF
emulate ksh
typeset -a cmd
cmd=(echo w:x y:z)
IFS=: x=${(q)cmd[@]} # Now we're in trouble
typeset -p x
unset -v IFS # Different problem whether or not we go back to default.
eval $x
EOF
typeset x=echo:w:x:y:z
zsh:1: command not found: echo:w:x:y:z
> Yeah, of course, it’s the only way to do some things… I personally
> usually abstract everything eval into little functions of their
> own and then just use those.
>
I agree that's an excellent strategy. :)
--
Dan Douglas
Re: More fun with IFS
Dan Douglas dixit:
>of any reason it should be inserting a '':'' between the two arguments,
>especially for the ''$@'' variants, either quoted or unquoted. It certainly
>can't be because of a word splitting step.
‘:’ is ${IFS::1} and inserted because of the word *concatenation*
(not splitting) that occurs when assigning a vector to a scalar.
>Most script writers treat assignments as identical whether quoted or not.
We had a discussion about that on the Austin ML (when I still
managed to somewhat follow it) and specifically changed it so
shells MUST distinguish between unquoted and quoted $*/$@.
> $ mksh -c 'set one:::two three:::four; IFS=:; cat <<<$@'
>one:::two:three:::four
> $ mksh -c 'set one:::two three:::four; IFS=:; cat <<<"$@"'
>one:::two:three:::four
I think that is completely reasonable and correct here.
>I tend to think AT&T ksh is doing the most reasonable thing here by making the
>concatenated result exactly the same as if expanded as arguments in a quoted
>context, with whitespace separating them.
Why whitespace? $IFS certainly contains none. And the usual
insertion rules all specify the first character of $IFS and
specify what to do if $IFS is empty or unset (which it isn’t
in these examples).
>> In other words, “don’t do that then” (rely on this behaviour).
>
>I wouldn't bother with this language if the only non-random behavior was that
>specified by POSIX. "POSIX doesn't specify it" is a horrible reason to do
>anything.
Right, “POSIX says so” oftentimes also is, after all…
which is why I tried above to argue without it.
>> I think eval is evil anyway ;-)
>
>Eval is frowned upon because it almost always misused. Until shells add first-
>class closures and higher-order functions I'll continue using it.
Yeah, of course, it’s the only way to do some things… I personally
usually abstract everything eval into little functions of their
own and then just use those.
bye,
//mirabilos
--
„nein: BerliOS und Sourceforge sind Plattformen für Projekte, github ist
eine Plattform für Einzelkämpfer“
-- dieses Zitat ist ein Beweis dafür, daß auch ein blindes Huhn
mal ein Korn findet, bzw. – in diesem Fall – Recht haben kann
Re: More fun with IFS
On Sunday, February 24, 2013 10:26:52 PM Thorsten Glaser wrote: > Dan Douglas dixit: > > >Zsh and pdkshes produce: > > > >one:::two:three:::four > > > >For all of the above, which I think is wrong for the last 4. ksh93 > >produces: > > Why is it incorrect? This test was intended to demonstrate expansions within assignment contexts. ''one:::two'' and ''three:::four'' are separate arguments to begin with. Word splitting and pathname expansion shouldn't occur within (ordinary) assignment contexts. I think the mksh (and zsh) results are wrong because I can't think of any reason it should be inserting a '':'' between the two arguments, especially for the ''$@'' variants, either quoted or unquoted. It certainly can't be because of a word splitting step. It's already been pointed out that different shells interpret unquoted @ and * differently. I think Chet's interpretation of the spec is the most reasonable (but you could argue otherwise): http://lists.gnu.org/archive/html/bug-bash/2013-01/msg00180.html Most script writers treat assignments as identical whether quoted or not. Quotes should only be needed to assign whitespace-containing strings and $'...' quotes, but shouldn't affect globbing or word splitting or modify the expansion in any other way. You'll notice the same thing in the case of herestrings. $ mksh -c 'set one:::two three:::four; IFS=:; cat <<<$@' one:::two:three:::four $ mksh -c 'set one:::two three:::four; IFS=:; cat <<<"$@"' one:::two:three:::four $ ksh -c 'set one:::two three:::four; IFS=:; cat <<<"$@"' one:::two three:::four $ ksh -c 'set one:::two three:::four; IFS=:; cat <<<$@' one:::two three:::four $ bash -c 'set one:::two three:::four; IFS=:; cat <<<$@' one two three four $ bash -c 'set one:::two three:::four; IFS=:; cat <<<"$@"' one:::two three:::four I tend to think AT&T ksh is doing the most reasonable thing here by making the concatenated result exactly the same as if expanded as arguments in a quoted context, with whitespace separating them. > In other words, “don’t do that then” (rely on this behaviour). I wouldn't bother with this language if the only non-random behavior was that specified by POSIX. "POSIX doesn't specify it" is a horrible reason to do anything. > I think eval is evil anyway ;-) Eval is frowned upon because it almost always misused. Until shells add first- class closures and higher-order functions I'll continue using it. > (Thanks to ormaaj for pointing out this posting.) :) -- Dan Douglas
Re: More fun with IFS
Dan Douglas dixit: >Zsh and pdkshes produce: > >one:::two:three:::four > >For all of the above, which I think is wrong for the last 4. ksh93 produces: Why is it incorrect? The mksh manpage documents $@ behaving like $*: @ Same as $*, unless it is used inside double quotes, in which case a separate word is generated for each positional parameter. If there are no positional parameters, no word is generated. $@ can be used to access arguments, verbatim, without losing NULL argu- ments or splitting arguments with spaces. And $* uses the first char of IFS: * All positional parameters (except 0), i.e. $1, $2, $3, ... If used outside of double quotes, parameters are separate words (which are subjected to word splitting); if used within double quotes, parameters are separated by the first character of the IFS parameter (or the empty string if IFS is NULL). POSIX is just as explicit on $* (with better wording for the two distinguished cases of IFS being unset or empty which the mksh code implements correctly, though): * Expands to the positional parameters, starting from one. When the expansion occurs within a double-quoted string (see [54]Double-Quotes ), it shall expand to a single field with the value of each parameter separated by the first character of the IFS variable, or by a if IFS is unset. If IFS is set to a null string, this is not equivalent to unsetting it; its first character does not exist, so the parameter values are concatenated. And POSIX on $@ doesn’t specify anything different for when the result of $@ is used where it isn’t multiple fields: @ Expands to the positional parameters, starting from one. When the expansion occurs within double-quotes, and where field splitting (see [53]Field Splitting ) is performed, each positional parameter shall expand as a separate field, with the provision that the expansion of the first parameter shall still be joined with the beginning part of the original word (assuming that the expanded parameter was embedded within a word), and the expansion of the last parameter shall still be joined with the last part of the original word. If there are no positional parameters, the expansion of '@' shall generate zero fields, even when '@' is double-quoted. So I think mksh at least behaves as specified, and the standard doesn’t contradict it. Inside the code, there’s even special-casing for “ifs0”, so I believe this is no accident. In other words, “don’t do that then” (rely on this behaviour). I think eval is evil anyway ;-) (Thanks to ormaaj for pointing out this posting.) bye, //mirabilos -- “It is inappropriate to require that a time represented as seconds since the Epoch precisely represent the number of seconds between the referenced time and the Epoch.” -- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2
Re: More fun with IFS
On 1/30/13 1:03 AM, Dan Douglas wrote:
> Hi everyone, and welcome to another edition of IBOTD (IFS-bug-of-the-day),
> featuring everyone's favorite Bourne shell kludge: word-splitting!
>
> On today's episode - inconsistencies within assignments that depend upon
> quoting. Though I can't take credit for discovering this -- it was pointed
> out
> to me by some guys on IRC after demonstrating some other stuff.
>
> And a quick test:
>
> function expassign {
> typeset -a a
> a=("$@")
> typeset var asn
>
> while IFS= read -r asn; do
> IFS=: command eval "$asn"
> printf '%-14s... %s\n' "$asn" "$var"
> done <<\EOF
> var=${a[*]}
> var="${a[*]}"
> var=$*
> var="$*"
> var=${a[@]}
> var="${a[@]}"
> var=$@
> var="$@"
> EOF
> }
>
> ${ZSH_VERSION+:} false && emulate ksh
> expassign one:::two three:::four
Thanks for the test case. I had to fix the expansions of ${a[@]} and
${a[*]}, and those fixes will be in the next release.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: More fun with IFS
On Wednesday, January 30, 2013 11:35:55 AM Chet Ramey wrote: > On 1/30/13 2:47 AM, Dan Douglas wrote: > > > No, $* always expands to a single word. If multiple words result, those > > are > > the result of field-splitting, not an intrinsic multi-word expansion as in > > the > > case of $@. Though POSIX says very little about the unquoted cases. > > I haven't looked at the rest of this, but the situation is clearly not as > absolute as you've phrased it. > I know, I agree. I've looked at what happens with empty IFSes on $* before, which is pretty much the only way I can think of to even test whether the words are due to field splitting or parameter expansion. The way bash does that is fine. That's probably the least important of these points anyway and I wouldn't really expect it to have any impact on these tests involving assignments. -- Dan Douglas
Re: More fun with IFS
On 1/30/13 2:47 AM, Dan Douglas wrote: > No, $* always expands to a single word. If multiple words result, those are > the result of field-splitting, not an intrinsic multi-word expansion as in > the > case of $@. Though POSIX says very little about the unquoted cases. I haven't looked at the rest of this, but the situation is clearly not as absolute as you've phrased it. In a scenario where word splitting is not performed, say when IFS is null, there are shells that expand $* to multiple words (bash, ksh93, FreeBSD sh), those that expand $* to a single word separated by spaces (posh, mksh, SVR4.2 sh), and those that expand $* to a single word with the positional parameters concatenated (dash). Personally, I think the phrase "expands to the positional parameters, starting from one" supports the bash/ksh93 behavior. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: More fun with IFS
"Chris F.A. Johnson" writes:
> On Wed, 30 Jan 2013, Andreas Schwab wrote:
>
>> "Chris F.A. Johnson" writes:
>>
var=${a[*]} ... one two three four # bad
>>>
>>>Looks good to me. It expands to multiple words, just as an unquoted
>>>$* would do.
>>
>> But no field splitting is performed on the expansion, so why are the
>> colons lost?
>
>How is it printed if it is not expanded?
??? Parameter expansion and field splitting are separate steps of word
expansion.
Andreas.
--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Re: More fun with IFS
On Wed, 30 Jan 2013, Andreas Schwab wrote:
"Chris F.A. Johnson" writes:
var=${a[*]} ... one two three four # bad
Looks good to me. It expands to multiple words, just as an unquoted
$* would do.
But no field splitting is performed on the expansion, so why are the
colons lost?
How is it printed if it is not expanded?
The expansion is done when it is printed.
--
Chris F.A. Johnson,
Re: More fun with IFS
"Chris F.A. Johnson" writes:
>> var=${a[*]} ... one two three four # bad
>
>Looks good to me. It expands to multiple words, just as an unquoted
>$* would do.
But no field splitting is performed on the expansion, so why are the
colons lost?
Andreas.
--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Re: More fun with IFS
On Wednesday, January 30, 2013 02:00:26 AM Chris F.A. Johnson wrote:
> On Wed, 30 Jan 2013, Dan Douglas wrote:
>
> > Hi everyone, and welcome to another edition of IBOTD (IFS-bug-of-the-day),
> > featuring everyone's favorite Bourne shell kludge: word-splitting!
> >
> > On today's episode - inconsistencies within assignments that depend upon
> > quoting. I can't take credit for discovering this -- it was pointed out
> > to me by some guys on IRC after demonstrating some other stuff.
> >
> > And a quick test:
> >
> > function expassign {
> > typeset -a a
> > a=("$@")
> > typeset var asn
> >
> > while IFS= read -r asn; do
> > IFS=: command eval "$asn"
> > printf '%-14s... %s\n' "$asn" "$var"
> > done <<\EOF
> > var=${a[*]}
> > var="${a[*]}"
> > var=$*
> > var="$*"
> > var=${a[@]}
> > var="${a[@]}"
> > var=$@
> > var="$@"
> > EOF
> > }
> >
> > ${ZSH_VERSION+:} false && emulate ksh
> > expassign one:::two three:::four
> >
> > Bash output: # I think...
> > var=${a[*]} ... one two three four # bad
>
> Looks good to me. It expands to multiple words, just as an unquoted
> $* would do.
No, $* always expands to a single word. If multiple words result, those are
the result of field-splitting, not an intrinsic multi-word expansion as in the
case of $@. Though POSIX says very little about the unquoted cases.
Secondly, at least one of these would almost have to be wrong unless it
somehow makes sense for $* and ${a[*]} to be treated differently (it doesn't).
Third, field-splitting doesn't occur within assignments so quoting shouldn't
matter here.
I'll grant that POSIX is extremely unclear about whether and when multiple
words are actually the result of a multi-word expansion, or a field splitting
step. It gets much worse when it comes to the alternate-value expansions where
it's then additionally unclear whether and how word-splitting and/or the
implicit multi-wordness of $@ occur recursively when nested (or not). Almost
all shells disagree on that, but I need to do more research for those cases
because it's hard to test what's going on. ksh93 is especially bizarre in its
nested expansions.
> > var="${a[*]}" ... one:::two:three:::four # good
> > var=$*... one:::two:three:::four # good
> > var="$*" ... one:::two:three:::four # good
> > var=${a[@]} ... one two three four # bad
>
> As above.
No, the ::: shouldn't be removed here. The * and @ cases are separate issues
(I should have been clear that there are multiple problems going on).
> > var="${a[@]}" ... one:::two three:::four # good
> > var=$@... one two three four # bad
>
> Ditto.
>
> > var="$@" ... one:::two three:::four # good
> >
> > Zsh and pdkshes produce:
> >
> > one:::two:three:::four
> >
> > For all of the above, which I think is wrong for the last 4. ksh93
produces:
> >
> > one:::two three:::four
> >
> > for the last 4, which I think is correct.
> >
> >
>
>
--
Dan Douglas
Re: More fun with IFS
On Wed, 30 Jan 2013, Dan Douglas wrote:
Hi everyone, and welcome to another edition of IBOTD (IFS-bug-of-the-day),
featuring everyone's favorite Bourne shell kludge: word-splitting!
On today's episode - inconsistencies within assignments that depend upon
quoting. Though I can't take credit for discovering this -- it was pointed out
to me by some guys on IRC after demonstrating some other stuff.
And a quick test:
function expassign {
typeset -a a
a=("$@")
typeset var asn
while IFS= read -r asn; do
IFS=: command eval "$asn"
printf '%-14s... %s\n' "$asn" "$var"
done <<\EOF
var=${a[*]}
var="${a[*]}"
var=$*
var="$*"
var=${a[@]}
var="${a[@]}"
var=$@
var="$@"
EOF
}
${ZSH_VERSION+:} false && emulate ksh
expassign one:::two three:::four
Bash output: # I think...
var=${a[*]} ... one two three four # bad
Looks good to me. It expands to multiple words, just as an unquoted
$* would do.
var="${a[*]}" ... one:::two:three:::four # good
var=$*... one:::two:three:::four # good
var="$*" ... one:::two:three:::four # good
var=${a[@]} ... one two three four # bad
As above.
var="${a[@]}" ... one:::two three:::four # good
var=$@... one two three four # bad
Ditto.
var="$@" ... one:::two three:::four # good
Zsh and pdkshes produce:
one:::two:three:::four
For all of the above, which I think is wrong for the last 4. ksh93 produces:
one:::two three:::four
for the last 4, which I think is correct.
--
Chris F.A. Johnson,
More fun with IFS
Hi everyone, and welcome to another edition of IBOTD (IFS-bug-of-the-day),
featuring everyone's favorite Bourne shell kludge: word-splitting!
On today's episode - inconsistencies within assignments that depend upon
quoting. Though I can't take credit for discovering this -- it was pointed out
to me by some guys on IRC after demonstrating some other stuff.
And a quick test:
function expassign {
typeset -a a
a=("$@")
typeset var asn
while IFS= read -r asn; do
IFS=: command eval "$asn"
printf '%-14s... %s\n' "$asn" "$var"
done <<\EOF
var=${a[*]}
var="${a[*]}"
var=$*
var="$*"
var=${a[@]}
var="${a[@]}"
var=$@
var="$@"
EOF
}
${ZSH_VERSION+:} false && emulate ksh
expassign one:::two three:::four
Bash output: # I think...
var=${a[*]} ... one two three four # bad
var="${a[*]}" ... one:::two:three:::four # good
var=$*... one:::two:three:::four # good
var="$*" ... one:::two:three:::four # good
var=${a[@]} ... one two three four # bad
var="${a[@]}" ... one:::two three:::four # good
var=$@... one two three four # bad
var="$@" ... one:::two three:::four # good
Zsh and pdkshes produce:
one:::two:three:::four
For all of the above, which I think is wrong for the last 4. ksh93 produces:
one:::two three:::four
for the last 4, which I think is correct.
--
Dan Douglas
