Short doc for readarray synonym for mapfile needs to be updated
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/local/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -g -O2 -Wno-parentheses -Wno-format-security uname output: Linux jh2 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-unknown-linux-gnu Bash Version: 4.4 Patch Level: 0 Release Status: release Description: bash-4.4 mapfile supports -d DELIM, and the short doc in the help has been updated to reflect this. However, the short doc for the readarray synonym hasn't been updated. Repeat-By: $ help mapfile $ help readarray Fix: Here is a patch for mapfile.def...I didn't know if all of the .po files needed --- bash-4.4/builtins/mapfile.def.orig 2016-09-19 18:21:04.350497510 -0500 +++ bash-4.4/builtins/mapfile.def 2016-09-19 18:21:53.760427393 -0500 @@ -59,7 +59,7 @@ $BUILTIN readarray $FUNCTION mapfile_builtin -$SHORT_DOC readarray [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] +$SHORT_DOC readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] Read lines from a file into an array variable. A synonym for `mapfile'. -- This e-mail is intended only for the recipient and may contain confidential or proprietary information. If you are not the intended recipient, the review, distribution, duplication or retention of this message and its attachments is prohibited. Please notify the sender of this error immediately by reply e-mail, and permanently delete this message and its attachments in any form in which they may have been preserved.
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 01:10:56PM -0700, L. A. Walsh wrote: > Does readarray allow specifying the nulls as line-terminators? Yes, as of bash 4.4.
Re: Is SIGWINCH fixed to re-enable it's pre-4.3 behavior? If not, why not?
Chet Ramey wrote: On 9/19/16 12:38 PM, L. A. Walsh wrote: It sounds like, from the release notes, that allowing real-time processing of signal handlers in read should no longer be an issue if pselect is used to before read to allow waiting for and processing of real-time events before reading a character. Correct. SIGWINCH will interrupt a pselect() and allow the shell, or any other application using readline, to run a signal hook outside of any signal handling context. Awesome! (I hope!)
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: On 9/19/16 2:41 PM, Greg Wooledge wrote: Bash has only three choices that I can think of: it can silently drop the NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). There is a fourth choice: terminate the string read from the command substitution at the first NUL. Pretty sure nobody wants that, even though that's what would happen if bash didn't drop the NULs. Hmm Does seems like it might be reasonable to allow 'null' in the list of field separators to allow read to read such lines into separate vars, as well as allowing it in the line-separator field to allow such data to be read into arrays. Does readarray allow specifying the nulls as line-terminators? I'm generally against issuing warnings for "normal" or "default" behavior, as it is likely to trigger spurious messages. Where are the posts of people complaining about normal behavior? Was there any discussion from the peanut...err, mailing list when this issue came up? I can't believe I missed the large number of complaints about the standard behavior. In the same vein, is the idea of issuing warnings for improper characters when reading in UTF-8 mode, as that will also result in data corruption or loss of data. Do you really want to start bash being responsible to issue warnings for any input that might not get stored correctly? That sounds like another nasty can of worms.
Re: Command substitution with null bytes generates warning
On 9/19/16 2:41 PM, Greg Wooledge wrote: > Bash has only three choices that I can think of: it can silently drop the > NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an > error, or it can drop the NUL bytes with a warning (4.4 behavior). There is a fourth choice: terminate the string read from the command substitution at the first NUL. Pretty sure nobody wants that, even though that's what would happen if bash didn't drop the NULs. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Is SIGWINCH fixed to re-enable it's pre-4.3 behavior? If not, why not?
On 9/19/16 12:38 PM, L. A. Walsh wrote: > It sounds like, from the release notes, that allowing real-time processing > of signal handlers in read should no longer be an issue if pselect > is used to before read to allow waiting for and processing of real-time > events before reading a character. Correct. SIGWINCH will interrupt a pselect() and allow the shell, or any other application using readline, to run a signal hook outside of any signal handling context. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
Greg Wooledge wrote: Bash has only three choices that I can think of: it can silently drop the NUL bytes behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). now who is being disingenuous? It was silent not just in 4.3, but in all previous versions. There is far more weight behind the being silent (the unix design philosophy) than being needlessly noisy. Calling that an "illegally modified input" is disingenuous. Claiming it needed a warning for something that is "impossible" is equally ridiculous. Why not warn that bash can't do your laundry? If bash is to warn about everything it ***can't*** do, it would be unusable. So why make a special case to warn about the "impossible"?
Re: repeated extended pattern substitution incredibly slow w/large variables
On 9/18/16 5:32 AM, xa...@t-online.de wrote:
> Bash Version: 4.4
> Patch Level: 0
> Release Status: rc2 / release
>
> Description:
> The tests below were performed with 4.4.0-rc2. However, the problem is
> still present in 4.4.0-release, only execution times are even higher
> for about 20%.
>
> Repeated pattern substitution (here: removal) using an extended pattern
> and variables of considerable size is incredibly time and cpu consuming.
> The command that revealed the problem was:
>
>D=${C//\[+([0-9])\]=}
This is absolutely true. The extended pattern matching code is very slow
and does far more work, in the sense that the matcher traverses the string
to be matched, than it needs to. It's never been optimized, and the
primitives are not quite suited for this kind of scenario: multiple
short matches in a long string.
This is one of the things on my list to look at for bash-5.0.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 11:30:56AM -0700, Linda Walsh wrote: > How about, w/r/t the new warning -- I complain because the null bytes > are missing after bash knowingly detected them and illegally modified > the input. Putting out a warning about null bytes, doesn't mean it's > "ok" to drop them. Now it's just compounded with an anti-unix (silence > is golden) warning message. Bash *cannot* store the NUL bytes in the result of a command substitution. It's completely impossible. Calling that an "illegally modified input" is disingenuous. Bash has only three choices that I can think of: it can silently drop the NUL bytes (4.3 behavior), it can drop ALL of the bytes and return an error, or it can drop the NUL bytes with a warning (4.4 behavior). I understand that you, personally, prefer the 4.3 behavior, and that you want Chet to change it back. Well, I'm not Chet, and I can't change it back for you, but I can give you a workaround to suppress the warning. That's what I did.
Re: Command substitution with null bytes generates warning
Eric Blake wrote: On 09/19/2016 11:58 AM, Greg Wooledge wrote: wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) wooledg@wooledg:~$ Or: $ x=$(< /proc/$$/cmdline tr -d \\0) Admittedly better, as it doesn't suppress valid messages, but still requires tracking down source code mods on multiple machines and making changes for something that wasn't a problem. Why should people who didn't expect them to make any difference (the valid behavior) be penalized because others didn't know. How about, w/r/t the new warning -- I complain because the null bytes are missing after bash knowingly detected them and illegally modified the input. Putting out a warning about null bytes, doesn't mean it's "ok" to drop them. Now it's just compounded with an anti-unix (silence is golden) warning message. ^^ see, you can still complain about the null bytes regardless of the warning. The warning is just adding salt to the wound -- like "nyaa nyaa, I know about the null bytes, and am still refusing to return them to you!!!" What's the purpose in making emit warnings about everything that doesn't work? Could just fix it and use the C++ definition of Strings that have a builtin count and can handle nulls in the middle of the data... ;^/
Re: Command substitution with null bytes generates warning
On 09/19/2016 11:58 AM, Greg Wooledge wrote: > On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: >>If users were relying on this behavior (I know I have scripts that read >> things from proc -- a text interface that uses \0 to display values similar >> to MS's multi-string Values in the Windows registry. > >>Could you change it back or provide a way to suppress "kiddy-scripter" >> seatbelts? > > wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) > bash-4.4: warning: command substitution: ignored null byte in input > wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) > wooledg@wooledg:~$ Or: $ x=$(< /proc/$$/cmdline tr -d \\0) -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: Command substitution with null bytes generates warning
Greg Wooledge wrote: On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: Could you change it back or provide a way to suppress "kiddy-scripter" seatbelts? wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) Not useful: 1) it requires finding all scripts that have the problem and creating new versions of such, testing that something else didn't break. 1a) it requires finding all the locations in those scripts. Not all will be obvious as some may only be triggered under specific circumstances. 2) It suppresses all warnings -- it isn't just suppressing the new warning, but all warnings and messages. That's generally considered "bad practice". I'm surprised you would suggest such a hack.
Re: Command substitution with null bytes generates warning
On Mon, Sep 19, 2016 at 09:28:53AM -0700, L. A. Walsh wrote: >If users were relying on this behavior (I know I have scripts that read > things from proc -- a text interface that uses \0 to display values similar > to MS's multi-string Values in the Windows registry. >Could you change it back or provide a way to suppress "kiddy-scripter" > seatbelts? wooledg@wooledg:~$ x=$(< /proc/$$/cmdline) bash-4.4: warning: command substitution: ignored null byte in input wooledg@wooledg:~$ x=$(< /proc/$$/cmdline 2>/dev/null) wooledg@wooledg:~$ Voila.
Is SIGWINCH fixed to re-enable it's pre-4.3 behavior? If not, why not?
From the below, it sounds like you've fixed the problem disallowing signal handlers to process signals. SIGWINCH is specifically listed to use wait methods that allow it to be processed outside of 'read' -- which apparently caused problems such that signal handling was disabled in reads and not processed in real-time anymore, but when returning from the read (when the user pressed a key). This caused breakage in scripts that relied on this behavior (wanting to process ALARMs, reap-children, process window-size changes, etc...). It sounds like, from the release notes, that allowing real-time processing of signal handlers in read should no longer be an issue if pselect is used to before read to allow waiting for and processing of real-time events before reading a character. So has the regression introduced in 4.3 been fixed? If not, could it be in a patch? Thanks, Linda Chet Ramey wrote: +== CHANGES == d. Readline's input handler now performs signal processing if read(2) is interrupted by SIGALRM or SIGVTALRM. --- It sounds like the ability for signal handlers to be called while waiting for input has been restored. h. Use pselect(2), if available, to wait for input before calling read(2), so a SIGWINCH can interrupt it, since it doesn't interrupt read(2). Is it the case that pselect is used to allow SIGALRM and SIGVTALRM to be called while waiting for input? Does this mean SIGWINCH also causes signal processing, allowing signal handlers to called again? It seems if it is possible to call other signal handlers while doing a read, then SIGWINCH's sig-handler should also be callable, is that now the case?
Re: Command substitution with null bytes generates warning
Chet Ramey wrote: On 9/16/16 1:51 AM, Eric Pruitt wrote: Bash Version: 4.4 Patch Level: 0 Release Status: release Description: I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux* ]];`. All /proc/*/cmdline include null bytes, and as of Bash 4.4, this results in a warning being spewed on stderr which did not happen in Bash 4.3. Other users have expectations that differ from yours. I received messages reporting the the bash-4.3 behavior (the longtime bash behavior) as a bug. Warning the user that bash discards some characters from the command substitution output seemed like the course that would let everyone know what's happening regardless of their expectations. --- If users were relying on this behavior (I know I have scripts that read things from proc -- a text interface that uses \0 to display values similar to MS's multi-string Values in the Windows registry. Perhaps worse, not sure, but cygwin has a /proc/registry to allow programs/scripts to read values from the registry. I have several that read such values. Of course, I know they are dropped -- I've complained about the fact that they are dropped in the past and wanted a way to read them, but that's not this problem. The "other users" you are talking about, are ones who don't bother reading documentation or understanding the technology. Changing working features to throw diag-messages is dumbing down bash to the lowest common denominator. Could you change it back or provide a way to suppress "kiddy-scripter" seatbelts? Just because people who don't know anything about how strings are handled on unix, doesn't mean you should dump to change something to shut them up. Chet
Re: Broken PIPESTATUS with --disable-job-control
Chet Ramey wrote: > On 9/18/16 11:20 PM, Felix Janda wrote: > > >>> Notice that the configure script disables job-control when a run-time > >>> test (which could easily be a built-time test) fails. So by default, > >>> a cross-compiled bash will have this bug. > >> > >> Which test? > > > > I am referring to BASH_SYS_JOB_CONTROL_MISSING. > > Sure. I'm asking which part of that run-time test can easily be converted > to a build-time test that handles conditional compilation and definitions. Everything could easily be a preprocessor test: #include #ifdef HAVE_SYS_WAIT_H #include #endif #ifdef HAVE_UNISTD_H #include #endif #include /* signal type */ #if !defined (HAVE_POSIX_SIGNALS) && !defined (HAVE_BSD_SIGNALS) #error #endif /* signals and tty control. */ #if !defined (SIGTSTP) || !defined (SIGSTOP) || !defined (SIGCONT) #error #endif /* process control */ #if !defined (WNOHANG) || !defined (WUNTRACED) #error #endif /* Posix systems have tcgetpgrp and waitpid. */ #if defined (_POSIX_VERSION) && !defined (HAVE_TCGETPGRP) #error #endif #if defined (_POSIX_VERSION) && !defined (HAVE_WAITPID) #error #endif /* Other systems have TIOCSPGRP/TIOCGPRGP and wait3. */ #if !defined (_POSIX_VERSION) && !defined (HAVE_WAIT3) #error #endif I think that the only reason that it is currently a run-time test is related to the comment: /* Add more tests in here as appropriate. */ So it was conceived that in future run-time only tests might be necessary. Felix
Re: Broken PIPESTATUS with --disable-job-control
On 9/18/16 11:20 PM, Felix Janda wrote: >>> Notice that the configure script disables job-control when a run-time >>> test (which could easily be a built-time test) fails. So by default, >>> a cross-compiled bash will have this bug. >> >> Which test? > > I am referring to BASH_SYS_JOB_CONTROL_MISSING. Sure. I'm asking which part of that run-time test can easily be converted to a build-time test that handles conditional compilation and definitions. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: repeated extended pattern substitution incredibly slow w/large variables
> maybe I do not fully follow your example, but wouldn't you instead of:
>
> time D="${C//\[+([0-9])\]=}" # rm '[]='
>
> want:
>
> time D="${C//\[[0-9]*\]=}" # rm '[]='
this would also find [!asd]. And there are other possibilities.
but that was not my problem / question.
> your example copies a lot to D and thats what takes time, I guess.
... but only if _extended_ pattern matching is used, see my examples. Why is
extended pattern matching so slow as compared to simple pattern matching.
That's the question.
Gesendet mit Telekom Mail - kostenlos und
sicher für alle!
