Re: caller returns wrong line number in command substitution
On 10/18/22 1:47 PM, Andrew Neff via Bug reports for the GNU Bourne Again
SHell wrote:
Description:
Using the "caller" command on a line of bash code in a process substitution
has been incorrect from bash 3.2 through 5.1, but I could write my code in such a way to
work around it. Some of these workarounds no longer function in bash 5.2. I saw that you
made some changes to this code [see below], however, I think they introduced another
regression to how caller calculates the line offset. In the following tests, caller
should always return the last line in a multi-line bash call, but that is not the case
now in a process substitution in bash 5.2 (test 5)
Getting line numbers correct for multi-line simple commands has always been
tricky, given the bottom-up way you have to build commands with bison. You
don't really know you're parsing a simple command until you've parsed it,
so you know where you are when you're finished, but not really where you
started.
For most single-line cases, it doesn't matter, but multi-line cases use the
line number when the parser finishes parsing the simple command.
So let's take a look at what happens for this script:
1 function foobar()
2 {
3 caller >&2
4 }
5 true <(
6foobar "test2
7 ...
8 bar")
Running this script prints `10'. Why?
Well, it's a combination of things. First, the line number associated with
the `true' command is 8, since that's the line number when bison resolves
the `simple_command' production and we go off and create it.
Now, before that happens, we have to run the process substitution through
the parser so we can find the ending right paren. We use the same code used
to parse command substitutions, since they have the same requirements.
Rather than keep the original text around, which is difficult to do with
how the parser and lexer are structured, bash reconstitutes the text of the
process substitution from the command it parses. This is syntactically
equivalent, but has the effect of throwing away leading and trailing
sequences of newlines, since those don't contribute to the command. (Your
suspicion about the new command substitution parsing code having an effect
here is correct.)
So when we go to perform word expansion on the process substitution, the
internal line number is 8, since that's the line number of the simple
command, and the command that's run by the shell forked to execute the
process substitution starts with `foobar' (not `\nfoobar').
The shell forks a child, and that child parses the string, starting at
line 8. The command consists of three lines: lines 8, 9, and 10. Since
the current line number is 10 when we finish parsing the simple command
and resolve the production, that's the line number associated with that
command. When the function is called, and `caller' does its thing, the
line number associated with the currently-executing command that called
the function is 10, and that's what it prints.
You should be able to puzzle out what happens for the other cases based
on this description.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: caller returns wrong line number in command substitution
Oops, my mistake. Got some terms mixed up a little there. Yes, every time I command substitution, I meant process substitution. So that release note for "Rewrote the command substitution" most likely nothing to do with this. From: Robert Elz Sent: Tuesday, October 18, 2022 7:09 PM To: Andrew Neff Cc: bug-bash@gnu.org Subject: Re: caller returns wrong line number in command substitution There are no command substitutions in any of your examples. A command substitution is what you get with $( ) (or if you really like obsolete syntax for some reason, ``). What you're showing is process substitution. This has nothing to do with whether or not there is an issue here that's worth fixing however - but bash's management of line numbers has always been "close enough" rather than "accurate". kre
Re: caller returns wrong line number in command substitution
There are no command substitutions in any of your examples. A command substitution is what you get with $( ) (or if you really like obsolete syntax for some reason, ``). What you're showing is process substitution. This has nothing to do with whether or not there is an issue here that's worth fixing however - but bash's management of line numbers has always been "close enough" rather than "accurate". kre
caller returns wrong line number in command substitution
Machine: x86_64
OS: linux-musl
Compiler: gcc
Compilation CFLAGS: -g -O2
uname output: Linux cfa1574b05c7 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed
Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux
uname output: Linux 3beae0f31cdf 5.18.18-100.fc35.x86_64 #1 SMP PREEMPT_DYNAMIC
Wed Aug 17 16:09:22 UTC 2022 x86_64 Linux
Machine Type: x86_64-pc-linux-musl
Docker: docker run -it --rm bash:5.2
Description:
Using the "caller" command on a line of bash code in a process substitution
has been incorrect from bash 3.2 through 5.1, but I could write my code in such
a way to work around it. Some of these workarounds no longer function in bash
5.2. I saw that you made some changes to this code [see below], however, I
think they introduced another regression to how caller calculates the line
offset. In the following tests, caller should always return the last line in a
multi-line bash call, but that is not the case now in a process substitution in
bash 5.2 (test 5)
[quote]
c. Rewrote the command substitution parsing code to call the parser
recursively and rebuild the command string from the parsed command. This allows
better syntax checking and catches errors much earlier. Along with this, if
command substitution parsing completes with here-documents remaining to be
read, the shell prints a warning message and reads the here-document bodies
from the current input stream
Repeat-By:
Here's a small script to repeat the problem:
#!/usr/bin/env bash
function foobar()
{
caller >&2
}
foobar "test0
...
bar"
true <(foobar "test1
...
bar")
true <(
foobar "test2
...
bar")
true <(\
foobar "test3
...
bar")
read -rd '' foo < <(foobar "test4
...
bar") || :
read -rd '' foo < <(
foobar "test5
...
bar") || :
read -rd '' foo < <(\
foobar "test6
...
bar")
read -rd '' foo < \
<(foobar "test7
...
bar")
Results:
Test Ans 5.1 5.2
08 8 8
111 13 13
215 18 17
319 21 21
422 22 22
526 26 25
630 29 29
734 33 33
Summary:
Test 0: no command substitution, it works.
Test 1: the answer is off by the number of lines. So 1 line is right, 2 lines
if off by one line, 3 lines (as show) is off by 2. If it was 10 lines, the
answer would be off by 9.
Test 2: Same offset as Test 1. On bash 3.2-5.1 off by one additional line
Test 3: Same as test 1
Test 4: Actually right
Test 5: Off by -1 lines in bash 5.2, right on bash 3.2-5.1
Test 6: Always off by -1 lines
Test 7: Same as Test 6
Fix/Workarounds:
Only the Test0 and Test4 consistently give the right answer. However for
readability and other reasons, I don't always want those syntaxes.
