Re: Command substitution with null bytes generates warning
On Wed, Oct 05, 2016 at 09:20:58AM -0400, Chet Ramey wrote: > Try the attached patch, which reduces the number of warnings to 1 per call > to command substitution. I don't agree with this approach -- I think you should be able to turn off the warning completely, so I am not interested in testing the patch. In the script where I ran into this problem, I simply suppressed the warning using 2>&-, and that's what I will continue to do if I will knowingly read files that contain null bytes. Eric
Re: Command substitution with null bytes generates warning
On 9/16/16 1:51 AM, Eric Pruitt wrote: > Bash Version: 4.4 > Patch Level: 0 > Release Status: release > > Description: > I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* > ]];`. > All /proc/*/cmdline include null bytes, and as of Bash 4.4, this > results in > a warning being spewed on stderr which did not happen in Bash 4.3. > > Repeat-By: > echo "$(<"/proc/$$/cmdline")" > > Fix: > Is this even an intentional change? I looked at some of the other > internal_warning invocations, and they were commented out using "#if 0 > ... > #endif." Try the attached patch, which reduces the number of warnings to 1 per call to command substitution. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/ *** ../bash-4.4/subst.c 2016-08-30 16:46:38.0 -0400 --- subst.c 2016-09-26 10:20:19.0 -0400 *** *** 5932,5935 --- 5933,5937 int istring_index, istring_size, c, tflag, skip_ctlesc, skip_ctlnul; ssize_t bufn; + int nullbyte; istring = (char *)NULL; *** *** 5939,5942 --- 5941,5946 skip_ctlesc |= *s == CTLESC, skip_ctlnul |= *s == CTLNUL; + nullbyte = 0; + /* Read the output of the command through the pipe. This may need to be changed to understand multibyte characters in the future. */ *** *** 5957,5961 { #if 1 ! internal_warning ("%s", _("command substitution: ignored null byte in input")); #endif continue; --- 5961,5969 { #if 1 ! if (nullbyte == 0) ! { ! internal_warning ("%s", _("command substitution: ignored null byte in input")); ! nullbyte = 1; ! } #endif continue;
Re: Command substitution with null bytes generates warning
The script is 5 years old. I don't remember any of the design decisions that went in to it -- so you wanna pick it apart, go ahead. I am already quite able to do so myself on stuff I wrote even 3 years ago, often, so 5 years, hey, go for it. I wasn't even using aliases to help my coding back then or have "include" files. Shame bash doesn't have those natively. (include files only get included once by default and can be specified by a path relative to PATH...) BTW, did I mention that was only 1 version of that and that it was intended as a one-of hack?
Re: Command substitution with null bytes generates warning
On Tue, Sep 20, 2016 at 12:17:37PM -0700, L. A. Walsh wrote: > /sys/class/net/br0> /tmp/showvals > addr_assign_type: 1 > addr_len: 6 > address: 00:15:17:bf:be:b2 > /tmp/showvals: line 63: printf: `invalid format character > brforward:'`f#?? > 7ridge/ageing_time:3 > bridge/bridge_id: 8000.001517bfbeb2 So you are just "catting" each file in the current working directory? Or maybe you just want the first "line" of each file? Why not simply do: wooledg@wooledg:/sys/class/net/eth0$ for f in *; do [[ -f $f ]] || continue; read -r line < "$f" 2>/dev/null; [[ $line ]] && printf '%-25.25s %s\n' "$f:" "$line"; done addr_assign_type: 0 address: 6c:3b:e5:2b:f7:19 addr_len: 6 broadcast:ff:ff:ff:ff:ff:ff carrier: 1 carrier_changes: 2 ... Your example script didn't even *have* a command substitution with a NUL-byte-containing file in it, as far as I could tell. Of course, the script was 20 times as large & complex as it needed to be, so I might have missed some crazy dynamically-constructed thing. (If you also want it to show the contents of subdirectories, then it becomes slightly larger. I won't bother writing that one out.)
Re: Command substitution with null bytes generates warning
I probably shouldn't even bother at this point, but morbid curiosity compels me to foolish ends. What are you DOING with these files that contain NUL bytes that makes it permissible to simply drop the NUL bytes on the floor, with no explicit step like tr -d \\0 to remove them? How is your script anything but buggy, if it is relying on a program (shell) to drop certain input bytes, when there is no documentation stating that it does this? If you are reading a file that uses \0 as a separator between fields, then how it is *OK* for your script to mash the fields all together into one giant mess? Why aren't you reading the fields separately? Take for example the file /proc/1/cmdline on Debian 8 (with mostly default settings): wooledg@wooledg:~$ cat -vtue /proc/1/cmdline /lib/systemd/systemd^@--system^@--deserialize^@16^@wooledg@wooledg:~$ I count 4 NUL bytes there, each one presumably terminating a field of some kind. If you simply use $(< /proc/1/cmdline) in some naive script running under bash 2.0 ~ 4.3 then you get something like /lib/systemd/systemd--system--deserialize16 Again, how is a script that produces this as a value not simply buggy? Perhaps what you really wanted was something like: { read -r -d '' a; read -r -d '' b; read -r -d '' c; read -r -d '' d; } \ < /proc/1/cmdline which gives: wooledg@wooledg:~$ declare -p a b c d declare -- a="/lib/systemd/systemd" declare -- b="--system" declare -- c="--deserialize" declare -- d="16" I don't know what these fields *mean*, but that's OK, because I'm not the one using them in a script. But this would be a much better way of getting them, than a blind command substitution.
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: Don't assume that every use of something like this has to do with /proc. Here's a representative report: "I was wondering what would happen if I'd do something like that: $ foo="$(cat file_containing_ascii_null_byte)" or faster $ foo="$(
Re: Command substitution with null bytes generates warning
On 9/19/16 12:28 PM, L. A. Walsh wrote: > > > Chet Ramey wrote: >> On 9/16/16 1:51 AM, Eric Pruitt wrote: >> >> >>> Bash Version: 4.4 >>> Patch Level: 0 >>> Release Status: release >>> >>> Description: >>> I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* >>> ]];`. >>> All /proc/*/cmdline include null bytes, and as of Bash 4.4, this >>> results in >>> a warning being spewed on stderr which did not happen in Bash 4.3. >>> >> >> Other users have expectations that differ from yours. I received messages >> reporting the the bash-4.3 behavior (the longtime bash behavior) as a bug. >> Warning the user that bash discards some characters from the command >> substitution output seemed like the course that would let everyone know >> what's happening regardless of their expectations. >> > --- >If users were relying on this behavior (I know I have scripts that read > things from proc -- a text interface that uses \0 to display values similar > to MS's multi-string Values in the Windows registry. Don't assume that every use of something like this has to do with /proc. Here's a representative report: "I was wondering what would happen if I'd do something like that: $ foo="$(cat file_containing_ascii_null_byte)" or faster $ foo="$(
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 01:10:56PM -0700, L. A. Walsh wrote: > Does readarray allow specifying the nulls as line-terminators? Yes, as of bash 4.4.
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: On 9/19/16 2:41 PM, Greg Wooledge wrote: Bash has only three choices that I can think of: it can silently drop the NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). There is a fourth choice: terminate the string read from the command substitution at the first NUL. Pretty sure nobody wants that, even though that's what would happen if bash didn't drop the NULs. Hmm Does seems like it might be reasonable to allow 'null' in the list of field separators to allow read to read such lines into separate vars, as well as allowing it in the line-separator field to allow such data to be read into arrays. Does readarray allow specifying the nulls as line-terminators? I'm generally against issuing warnings for "normal" or "default" behavior, as it is likely to trigger spurious messages. Where are the posts of people complaining about normal behavior? Was there any discussion from the peanut...err, mailing list when this issue came up? I can't believe I missed the large number of complaints about the standard behavior. In the same vein, is the idea of issuing warnings for improper characters when reading in UTF-8 mode, as that will also result in data corruption or loss of data. Do you really want to start bash being responsible to issue warnings for any input that might not get stored correctly? That sounds like another nasty can of worms.
Re: Command substitution with null bytes generates warning
On 9/19/16 2:41 PM, Greg Wooledge wrote: > Bash has only three choices that I can think of: it can silently drop the > NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an > error, or it can drop the NUL bytes with a warning (4.4 behavior). There is a fourth choice: terminate the string read from the command substitution at the first NUL. Pretty sure nobody wants that, even though that's what would happen if bash didn't drop the NULs. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
Greg Wooledge wrote: Bash has only three choices that I can think of: it can silently drop the NUL bytes behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). now who is being disingenuous? It was silent not just in 4.3, but in all previous versions. There is far more weight behind the being silent (the unix design philosophy) than being needlessly noisy. Calling that an "illegally modified input" is disingenuous. Claiming it needed a warning for something that is "impossible" is equally ridiculous. Why not warn that bash can't do your laundry? If bash is to warn about everything it ***can't*** do, it would be unusable. So why make a special case to warn about the "impossible"?
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 11:30:56AM -0700, Linda Walsh wrote: > How about, w/r/t the new warning -- I complain because the null bytes > are missing after bash knowingly detected them and illegally modified > the input. Putting out a warning about null bytes, doesn't mean it's > "ok" to drop them. Now it's just compounded with an anti-unix (silence > is golden) warning message. Bash *cannot* store the NUL bytes in the result of a command substitution. It's completely impossible. Calling that an "illegally modified input" is disingenuous. Bash has only three choices that I can think of: it can silently drop the NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). I understand that you, personally, prefer the 4.3 behavior, and that you want Chet to change it back. Well, I'm not Chet, and I can't change it back for you, but I can give you a workaround to suppress the warning. That's what I did.
Re: Command substitution with null bytes generates warning
Eric Blake wrote: On 09/19/2016 11:58 AM, Greg Wooledge wrote: wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) wooledg@wooledg:~$ Or: $ x=$(< /proc/$$/cmdline tr -d \\0) Admittedly better, as it doesn't suppress valid messages, but still requires tracking down source code mods on multiple machines and making changes for something that wasn't a problem. Why should people who didn't expect them to make any difference (the valid behavior) be penalized because others didn't know. How about, w/r/t the new warning -- I complain because the null bytes are missing after bash knowingly detected them and illegally modified the input. Putting out a warning about null bytes, doesn't mean it's "ok" to drop them. Now it's just compounded with an anti-unix (silence is golden) warning message. ^^ see, you can still complain about the null bytes regardless of the warning. The warning is just adding salt to the wound -- like "nyaa nyaa, I know about the null bytes, and am still refusing to return them to you!!!" What's the purpose in making emit warnings about everything that doesn't work? Could just fix it and use the C++ definition of Strings that have a builtin count and can handle nulls in the middle of the data... ;^/
Re: Command substitution with null bytes generates warning
On 09/19/2016 11:58 AM, Greg Wooledge wrote: > On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: >>If users were relying on this behavior (I know I have scripts that read >> things from proc -- a text interface that uses \0 to display values similar >> to MS's multi-string Values in the Windows registry. > >>Could you change it back or provide a way to suppress "kiddy-scripter" >> seatbelts? > > wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) > bash-4.4: warning: command substitution: ignored null byte in input > wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) > wooledg@wooledg:~$ Or: $ x=$(< /proc/$$/cmdline tr -d \\0) -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: Command substitution with null bytes generates warning
Greg Wooledge wrote: On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: Could you change it back or provide a way to suppress "kiddy-scripter" seatbelts? wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) Not useful: 1) it requires finding all scripts that have the problem and creating new versions of such, testing that something else didn't break. 1a) it requires finding all the locations in those scripts. Not all will be obvious as some may only be triggered under specific circumstances. 2) It suppresses all warnings -- it isn't just suppressing the new warning, but all warnings and messages. That's generally considered "bad practice". I'm surprised you would suggest such a hack.
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: >If users were relying on this behavior (I know I have scripts that read > things from proc -- a text interface that uses \0 to display values similar > to MS's multi-string Values in the Windows registry. >Could you change it back or provide a way to suppress "kiddy-scripter" > seatbelts? wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) wooledg@wooledg:~$ Voila.
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: On 9/16/16 1:51 AM, Eric Pruitt wrote: Bash Version: 4.4 Patch Level: 0 Release Status: release Description: I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* ]];`. All /proc/*/cmdline include null bytes, and as of Bash 4.4, this results in a warning being spewed on stderr which did not happen in Bash 4.3. Other users have expectations that differ from yours. I received messages reporting the the bash-4.3 behavior (the longtime bash behavior) as a bug. Warning the user that bash discards some characters from the command substitution output seemed like the course that would let everyone know what's happening regardless of their expectations. --- If users were relying on this behavior (I know I have scripts that read things from proc -- a text interface that uses \0 to display values similar to MS's multi-string Values in the Windows registry. Perhaps worse, not sure, but cygwin has a /proc/registry to allow programs/scripts to read values from the registry. I have several that read such values. Of course, I know they are dropped -- I've complained about the fact that they are dropped in the past and wanted a way to read them, but that's not this problem. The "other users" you are talking about, are ones who don't bother reading documentation or understanding the technology. Changing working features to throw diag-messages is dumbing down bash to the lowest common denominator. Could you change it back or provide a way to suppress "kiddy-scripter" seatbelts? Just because people who don't know anything about how strings are handled on unix, doesn't mean you should dump to change something to shut them up. Chet
Re: Command substitution with null bytes generates warning
On 9/16/16 1:51 AM, Eric Pruitt wrote: > Bash Version: 4.4 > Patch Level: 0 > Release Status: release > > Description: > I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* > ]];`. > All /proc/*/cmdline include null bytes, and as of Bash 4.4, this > results in > a warning being spewed on stderr which did not happen in Bash 4.3. Other users have expectations that differ from yours. I received messages reporting the the bash-4.3 behavior (the longtime bash behavior) as a bug. Warning the user that bash discards some characters from the command substitution output seemed like the course that would let everyone know what's happening regardless of their expectations. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
On Thu, Sep 15, 2016 at 10:51:22PM -0700, Eric Pruitt wrote: > Fix: > Is this even an intentional change? I looked at some of the other > internal_warning invocations, and they were commented out using "#if 0 > ... > #endif." In 4.4-beta2, I see them in subst.c and parse.y and y.tab.c. In subst.c they are inside an "#if 1" block. In the other two, they are inside "#if 0". I do not understand the discrepancy, but I would assume Chet made the change intentionally. It seems unlikely to be something one would do accidentally.