Date: Wed, 19 Aug 2026 11:17:45 +0200
From: Edgar =?iso-8859-1?B?RnXf?= <[email protected]>
Message-ID: <[email protected]>
I have been meaning to reply to this, but hadn't found any round tuits.
As there have been other messages on the topic in the past day or
so, now seems like an opportune time:
| > like "for f in ${allf}" where the ${allf} value needs to be field split
| > - that's not really safe, but the alternative is messy.
| Why is that unsafe?
| What else could one do?
The first of those has been partially answered now by dholland@
[email protected] said:
| Suppose you have:
| allf=$(ls *.jpg)
| for f in $allf; do
| printf 'File: %s\n' "$f"
| done
| Now imagine that the files in $CWD are foo.jpg, bar.jpg, and silly filename
| with spaces.jpg...
It is more than just spaces/newlines of course, there can also be
meta chars (* etc) which also mess things up.
David goes on:
| There is no easy way to make it more robust, unfortunately.
which, depending upon how robust you need it, how easy you need it
to be, and what assumptions, or potential defects you're willing to
live with (any code like the above is clearly willing to assume lots
about the file names it is dealing with - since the precursor to this
discussion was /etc/rc.d/* files, which largely means, managed by root,
and since root has much easier ways to break booting (or whatever they
would hope to achieve) than by making funny rc.d/script names) that's
probably not unreasonable - which is why I just said "the alternative
is messy" and left it like that (certainly no new issues were being
created by the patches proposed, now installed, than were already present).
However, it is not all that bleak:
Ken Hornstein (nice to see you back in these lists Ken) has part
of the answer (or one technique anyway).
[email protected] said:
| My limited, imperfect understanding is that in _this_ particular case,
| this is safe:
| for f in *.jpg
| [...]
| Since word splitting happens before pathname expansion.
yes, that is fine, as long as one quotes "$f" when used. But that
isn't the question (though dholland's example might make it seem as
if this might be an alternative). In the real application, we need
to take the list of files, manipulate that list (at the very least sort
it in a kind of peculiar way) and then use it. That can't be done as
a simple "for f in *.jpg" it would need at least to be a
"for f in $(some commands *.jpg) and as soon as you do that you're
processing the list of file names twice, just as dholland@'s example
showed (that one just didn't explain why it was written that way.)
Ken goes on:
| If you need to save the results of the pathname expansion, you could do:
| set -- *.jpg
| And then use "$@", as that expansion is special.
which is on the right path to a solution. He suggests this problem:
| The downside is that overwrites the argument list.
but that issue is easy to solve, you just do all this in a function,
created for the purpose. Functions have their own arg list, and once
any args they need have been saved in (usually local) variables, the
function's private arg list is open to be used however the function needs
it. The problem then is that we need to export the list of filenames
from the function.
[email protected] said:
| However, I am not sure there is an portable way of dealing with the
| output of a command that may contain spaces;
And that is getting to the real crux of the issue, though it isn't
spaces, or even newlines, which are the real problem that might exist
in the filenames, if that was all we needed to worry about, we'd just
do:
filelist()
{
local names=$1
set -- $names # $names, from $1, will be something like *.jpg
[ "$#" -gt 0 ] && printf "'%s' " "$@"
}
where the set and then use of "$@" is really just for the benefit
of another example to appear later; then use
list=$(filelist '*.jpg')
or whatever you need. That works just fine, later when we need to use
the list, we can do something like (another function)
runlist()
{
eval set -- $1
command "$@"
}
used as
runlist "$list"
(with enough effort, the command to run can be later args to runlist,
rather than embedded in it, but doing that properly is also messy, and
not related to the point here.)
The separate eval set -- rather than just using "eval command $1" is
in case the "command" part and any other args it needs, aren't suitable
to be used via eval. Being sure about that can be tricky sometimes, so
this method which avoids using eval on anything except where we know it
is safe (nothing will happen with "set" or with "--" and everything in $1
has been nicely quoted 'file 1.jpg' 'files*.*.jpg' .... and so no harm
can come from any of those as well).
Of course, nothing is ever quite that simple, and the "if that was all
we needed to worry about" above probably was enough of a hint that this
doesn't actually work in general. Spaces, newlines, *'s etc are no
problem at all (\0's would be, but they can't exist in file names, so
we're safe from that). The problem is files with ' characters in their
names. Taking a file
O'Hare.jpg
and quoting it like
'O'Hare.jpg'
as is done above, just doesn't produce anything even semi-useful
for this kind of purpose.
What we do next depends upon what assumptions we're willing to make.
If (which would probably be reasonable for rc.d/* files) we would be
willing to assume there will be no files containing ' characters, then
we're good. We don't have to just blindly press ahead assuming that
of course, we can check for it, and refuse to continue (or with a little
extra work, just skip any such file names, if that would be acceptable)
by changing the filelist function to be:
filelist()
{
local names=$1
local A
set -- $names # $names, from $1, will be something like *.jpg
for A
do
case "$A" in
*\'*) ;; # bad filename , do nothing
*) printf "'%s' " "$A";; # good filename, keep
esac
done
}
Doing it that way, you can even not generate the meaingless trailing ' '
at the end of the list if you want to (though there's no need, the way it
is to be used).
If that limitation isn't possible though, we need a way to correctly
quote filenames containing '. The easy answer would be to just use "
quotes for that case, but that's close to the worst thing you can do.
That's because if the same filename also contains '$', or '`', and
potentially '\', or, of course, '"', then you have just made far more
problems than existed before. So we won't do that.
One solution would be to give printf a %q operation (bash has one, which
would work for this, though its updated %Q which fixes a stupidity in the
design of its %q is a better model - either would work for this however).
so the printf in the original filelist() can just be
[ "$#" -gt 0 ] && printf '%q ' "$@"
and the problem is solved, and properly solved. %q arranges to correctly
quote its arg (each arg in this case) regardless of what it contains.
But our printf doesn't currently have that (I have code for it, there
has just never been any demand for it to be included, so I haven't).
The alternative, where it gets messier, is to do the quoting, properly,
in sh itself. It can be done, just a SMOP, I have a few different
versions of it floating around on my system - I won't include any here,
as then I'd need to explain it, and this message is already long enough
(and there is one more point from Edgar's message to answer yet).
So, it is possible in sh (and just using sh features) to do this all
perfectly safely, it is, as I said in the original message, just messy
to do it. I doubt we really need it in the rc scripts, but we could
add it if needed - this kind of thing tends to be more important in
scripts which scan random user files, downloaded tarballs, ... where
the control of what is included is nonexistent, than in rc.d files,
where the control should be quite good, and bizarre filenames are unlikely
(and can easily be prevented at install time). Also note, the ' char
only becomes a problem when we start adding quoting, and using eval, without
that the problem characters are the whitespace chars (we need one of those,
or something at least, to be in IFS so we can split the variable) and the
shell meta chars (those we can make safe using "set -f" at the appropriate
time.) If spaces are possible in filenames (and newlines too), but we can
assume some other character isn't likely (like perhaps control-A or something)
we could use that as the separator char, and set IFS=$'\1' to split things.
(If that is done, make sure the stray excess separator is at the end of the
list, not at the beginning - with space it makes no difference, with any
non whitespace character (which for this means space, tab, or newline)
the excess one (to make writing the printf format string easy) needs to
be after each filename, not before it, or we would get a stray empty
"filename" when things are split.
| > The quotes in "${ans}" are needed, those around yes are silly.
| Well, it looks more symmetric to the eye and even in
| foo="foo bar"
Ignoring the issue with the space, which was corrected in a later
message, this is mostly about style, I said "silly" not "wrong", and
if you feel strongly about it, then nothing will break because of
this kind of unnecessary quoting (if it were foo="foo" with no space).
But:
| At least for someone like me skipping between programming languages
that is a poor reason. Any language should be treated and used according
to its own rules and idioms, attempting to make them look all the same
will just lead to confusion when there is something that just cannot
be done the same way, or something quite similar, but with different
boundary conditions or side effects.
In the above, I notice you didn't write
At least for someone like me who is between programming Languages
skipping
which, I think, from my half-a-century old unused German knowledge,
something more like you would write it in German (and even there I
probably don't have things in the mandated order for German sentences,
as I have no idea what that is any more.)
I'm sure you switch backwards and forwards between German and English
all the time, and the English you write always looks like English, not
like German translated by a 3rd grader with a dictionary and nothing
else.
Programming languages should be treated the same way, what is right for
one is often not the accepted way in another. Each needs to be written
in its own style, and according to its own conventions.
These redundant quotes are really a minor issue, as long as it is
understood that they are redundant, and that there is little difference
between '' quoting and "" quoting - except that each can include the
other without doing anything special, but '' cannot embed a ' by any
means at all, whereas "" can \" a " into the string. And "" quoting
permits variable/arith/cmdsub expansions to occur within the string,
whereas whatever is inside a '' string is inviolate, nothing changes it.
For those who really understand all this, there are no problems, style
is style, and we all have our preferences. I tend to harp on this issue
somewhat, as I get the impression that lots of people don't really
understand what is happening, and misunderstand what quoting is doing,
and its purpose, and quote things because they have some belief (often
from cargo-culting other code) that it is needed in some context.
So, when I see something with quoting that isn't needed, I tend to
point it out. If you know it isn't needed, and are just doing it
for style reasons, fine, just ignore me.
kre
