Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug
Reply to Eric Blake,
At this time, I do not have a Linux image available to me. If
you saw the same behavior on Fedora, then I suggest the behavior
originates upstream at or close the the GNU source-code level.
Mr. Penny's response asserted the observed behavior "is intended
behavior", in which case there should exist a GNU specification
document describing the intended pipe STDIN re-direction
restrictions for 'mapfile', 'read' and possibly other "builtins".
Lacking such a reference nothing can be said about intent.
Implementation-as-intent implies errors and conceptual and
behavioral inconsistencies in the implementation were intended.
I decline to think that of the distributed BASH maintenance team.
He provided a reference to
http://mywiki.wooledge.org/BashFAQ/024, where, if you read the
page, the behavior inconsistency I reported is shown under the
heading of "More broken stuff:". I take that "More broken
stuff:" opinion to mean yet others read the same surface
discrepancy between doc and behavior as I did.
I believe what we have at this point is:
A) GNU doc lacking nuance, attention to consistent terminology,
and helpful rule-statement to code example/counter-example
illustrations adjacent to rule statements. For a doc
counter-example, the Open Group doc at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
does make an effort to distinguish behaviors between "2.14
Special Built-In Utilities" and "2.9.1 Simple Commands" whereas
the GNU doc indiscriminately mixes the words "commands" (section
3.2) and "builtin" and "command" and the phrase "builtin command"
(section 4) as if their behaviors are identical under the section
4 title of "Shell Builtin Commands"
(https://www.gnu.org/software/bash/manual/html_node/Shell-Builtin-Commands.html#Shell-Builtin-Commands).
The GNU doc for shopt lastpipeis fairly opaque unless you have
deep knowledge and that knowledge is cued when considering the
possible meanings of "current shell environment" for built-ins
(same process) vs. external child-process executables.
B) However conceptually inconsistent, an obsessive BASH doc
reader could imply the observed bash built-ins behavior by
integrating multiple hints and rule statements scattered across
the GNU doc and then, crucially, doubting the plain meaning of
the unqualified doc statements of "Read lines from the standard
input..." .
Thank you for your response.
Regards,
UN*X Since '85
On 7/20/2018 12:08 PM, Eric Blake wrote:
On 07/17/2018 08:52 PM, BloomingAzaleas wrote:
Reply to Steven Penny :
no mis-behaving: this is intended behavior - you yourself
have given
workarounds: either redirect output to a file that can be
later read, or pipe to
command grouping ala {} or () and read stdin from inside
the subshell
I suggest the following adjustment to the man pages inserting
a parenthetical cue regards behavior in pipes:
Is the behavior you are complaining about unique to Cygwin, or
can it be reproduced on a GNU/Linux box? If the latter, then
an upstream bug report is better than asking for a
cygwin-specific patch. [Hint - as the maintainer of the cygwin
bash port, I don't recall adding any cygwin-specific tweaks for
mapfile - and a quick test on Fedora shows the same behaviors]
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug
Reply to Steven Penny :
no mis-behaving: this is intended behavior - you yourself have given
workarounds: either redirect output to a file that can be later read, or
pipe to
command grouping ala {} or () and read stdin from inside the subshell
im not sure what you are asking here - it seems you have a grasp already of
the
"problem" and "solution" - so nothing more needs to be said really, as
nothing
needs to be fixed - another option is "lastpipe"
If the reported behavior is the intended behavior, then this
complaint is now a documentation complaint. Neither the Cygwin
man pages (quoted in the original report) nor the upstream GNU
documentation indicate any exception to stdin redirect with pipe
in the doc paragraphs for the subject builtins. What confuses
the situation is that some stdin re-directions work for these
builtins in a parent shell context (left-chevs) but not others
(pipes) UNLESS performed strictly at mid-night under a full moon
using obscure incantation hidden in pieces throughout a 178-page
pdf. In the GNU doc, the sub-shell for builtins in a pipeline
rule involved is obscured by burial in paragraph six of section
3.7.3 Command Execution Environment.
I suggest the following adjustment to the man pages inserting a
parenthetical cue regards behavior in pipes:
mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u
fd] [-C callback] [-c quantum] [array]
readarray [-d delim] [-n count] [-O origin] [-s count] [-t]
[-u fd] [-C callback] [-c quantum] [array]
Read lines from the standard input (see pipeline
restriction) into the indexed array variable array, or ...
read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N
nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
One line is read from the standard input (see pipeline
restriction), or ...
At least the reader gets a cue to modify expectations and
research the "pipeline restriction" compared to all other
commands that work just peachy with stdin pipe re-direction.
Regards the shopt lastpipe doc single-sentence paragraph in GNU
section 4.3.2 The Shopt Builtin, it could be improved by adding:
"This option will change the parent shell context behavior of
shell builtins mapfile, readarray and read when they are used as
the last command of a pipeline."
As to the left-chev work-arounds, yes they work, but that would
require some script re-structuring. In any case, "what you are
asking here" is that there appears to be a difference between a
plain reading of the doc paragraphs for the builtins and the
builtins behavior and so a bug against one or the other. If this
were a unix/linux kernel environment, the triage emphasis would
be documentation wording rather than implementation bug. Given
the Cygwin context of necessary hidden emulation chicanery on
Windows, triage emphasis is not that obvious. I exerted the
effort to construct a reasonably complete report with multiple
illustration cases contrasted to the doc to improve the FOSS, be
it implementation or documentation. I didn't need too.
Regards.
On 7/15/2018 7:01 PM, BloomingAzaleas wrote:
Windows 10 Pro 10.0.17134 N/A Build 17134 patched through 15
July 2018
Cygwin 2.10.0(0.325/5/3)
Bash 4.4.12(3)
Cygwin man pages show:
mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u
fd] [-C callback] [-c quantum] [array]
readarray [-d delim] [-n count] [-O origin] [-s count] [-t]
[-u fd] [-C callback] [-c quantum] [array]
Read lines from the standard input into the indexed array
variable array, or ...
read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N
nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
One line is read from the standard input, or ...
So the expectation is that all bash stdin re-directions such as
pipes and left chevs should work. For mapfile/readarry, the
output array variable MAPFILE is described as being created if
not array_var argument is given to mapfile.
A) Searched cygwin email list with terms 'mapfile', 'readarray'
and 'read builtin'. No obvious hits of recent vintage in
summary result list.
B) Confirm, not using mapfile/readarry/read, that bash stdin
redirs work as expected.
echo multi-line_arg | /bin/cat
/bin/cat < some_unix_fmt_file
/bin/cat <C) Test mapfile/readarray/read in various stdin redir
situations. Basic harnesses are :
mapfile/readarry HARNESS:
stdin or fd 0 feed to mapfile
then
: "${MAPFILE[@]:?MAPFILE null or unset}"
if get past the ${:?} test then dump MAPFILE with a simple /for
/loop as: for ent in "${MAPFILE[@]}" ; do echo ="${ent}"= ; done
read HARNESS:
stdin feed to 'read foo'
then
: "${foo:?foo null or unset}"
if get past the ${:?} test then dump variable foo with echo
="${foo}"=
CASES:
echo multi-line_arg | mapfile
cat multi-line_unix_fm
BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe
Windows 10 Pro 10.0.17134 N/A Build 17134 patched through 15 July 2018 Cygwin 2.10.0(0.325/5/3) Bash 4.4.12(3) Cygwin man pages show: mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] Read lines from the standard input into the indexed array variable array, or ... read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...] One line is read from the standard input, or ... So the expectation is that all bash stdin re-directions such as pipes and left chevs should work. For mapfile/readarry, the output array variable MAPFILE is described as being created if not array_var argument is given to mapfile. A) Searched cygwin email list with terms 'mapfile', 'readarray' and 'read builtin'. No obvious hits of recent vintage in summary result list. B) Confirm, not using mapfile/readarry/read, that bash stdin redirs work as expected. echo multi-line_arg | /bin/cat /bin/cat < some_unix_fmt_file /bin/cat
