Re: bug in bash

[Greg Wooledge]

On Sun, May 12, 2024 at 03:33:12PM +0200, Andreas Schwab wrote:
> > On Sun, May 12, 2024 at 03:55:21AM +0200, Quốc Trị Đỗ wrote:
> >> I found a bug when i tried with syntax <(cmd). this is an example
> >> cat <(wc -l) < bk

> Since the redirection fails and the cat command is never started, bash
> doesn't switch the terminal process group, and the background wc command
> goes on competing with bash for the terminal.

Ah... I assumed bk was an existing file.

hobbit:~$ cat <(wc -l) <.bashrc
wc: 'standard input': Input/output error
0
hobbit:~$ 

Re: bug in bash

[Greg Wooledge]

On Sun, May 12, 2024 at 03:55:21AM +0200, Quốc Trị Đỗ wrote:
> I found a bug when i tried with syntax <(cmd). this is an example
> cat <(wc -l) < bk

What is "wc -l" supposed to read from?  It counts lines of standard input,
until EOF is reached.  But its standard input is a terminal.  And you're
running it as a background process.

I would *expect* this command to fail with an error message of some
kind, because a background process shouldn't be allowed to read
input from a terminal.

> How to fix it? It doesn't last that long. After a while, it will show "wc:
> stdin: read: Input/output error". Or we can ctrl C.

Sounds appropriate.

What's the "bug in bash" supposed to be?  What did you think this command
would do?

Re: [sr #111058] Problem transmitting script arguments

[Greg Wooledge]

On Wed, May 08, 2024 at 02:07:55PM -0400, Dale R. Worley wrote:
> "Kerin Millar"  writes:
> > On Mon, 6 May 2024, at 7:01 PM, Dale R. Worley wrote:
> >> anonymous  writes:
> >>> [...]
> 
> > It's likely that your reply will never be seen by the anonymous
> > Savannah issue filer.
> 
> OK.  Now does that mean that there is no way for me to effectively
> suggest a solution (and so I shouldn't have bothered), or that I should
> have done so by some different method?

Any replies you make here will go into the list archives, where someone
might find them if they search for this topic.  Maybe even the original
bug submitter, though the chances of that seem low, given that they
submitted this bug rather than searching for any existing literature
or discussions.

So, you just need to decide whether the time you'll spend writing a
reply here is justified.

Re: [PATCH 0/4] Add import builtin

[Greg Wooledge]

On Sun, May 05, 2024 at 03:32:04PM -0400, Lawrence Velázquez wrote:
> Much like the periodic requests for XDG-organized startup files, a
> BASH_SOURCE_PATH might be convenient but is not groundbreaking and
> probably doesn't merit significant changes to the shell.  I'm not
> even convinced it merits a new "source" option.

I don't really have any strong opinions on this issue, but the proposed
patches seem a bit overkill-ish to me.

The idea to add a BASH_SOURCE_PATH variable that gets searched before, or
instead of, PATH when using the source builtin -- that sounds good.  I
don't really understand the concepts behind the rest of the discussion.

Re: [Help-bash] difference of $? and ${PIPESTATUS[0]}

[Greg Wooledge]

On Mon, Apr 22, 2024 at 08:13:16AM +0200, felix wrote:
> Then after some tests:
> 
>   if ls /wrong/path | wc | cat - /wrong/path | sed 'w/wrong/path' >/dev/null 
> ; then
>   echo Don't print this'
>   fi ; echo ${?@Q} ${PIPESTATUS[@]@A}  $(( $? ${PIPESTATUS[@]/#/+} ))
> 
>   ls: cannot access '/wrong/path': No such file or directory
>   cat: /wrong/path: No such file or directory
>   sed: couldn't open file /wrong/path: No such file or directory
>   '0' declare -a PIPESTATUS=([0]="2" [1]="0" [2]="1" [3]="4") 7
> 
> Where $PIPESTATUS[0]=>2 and $?=>0 !!
> 
> I could explain that '$?' is result of bash's if...then...fi group command
> executed correctly [...]

That is indeed the issue here.  $? contains the exit status of the "if"
command, not of the pipeline.

hobbit:~$ help if
[...]
The exit status of the entire construct is the exit status of the
last command executed, or zero if no condition tested true.
[...]

hobbit:~$ if (exit 42); then :; fi
hobbit:~$ echo $?
0

If you remove the 'if' from your example, you get a very different result:

hobbit:~$ /wrong/path | wc | cat - /wrong/path | sed 'w/wrong/path' >/dev/null
bash: /wrong/path: No such file or directory
sed: couldn't open file /wrong/path: No such file or directory
hobbit:~$ echo "<$?> <${PIPESTATUS[@]@A}>"
<4> 

Here, $? is the exit status of the last command in the pipeline, as
it should be.

I don't know where your notion that $? and PIPESTATUS[0] should be
equivalent came from, but it is never right, except in the case where the
"pipeline" is just a single command.

Introducing the 'if' separates the value of $? and the values of the
PIPESTATUS array entirely.  They no longer refer to the same pipeline
at all.  $? contains the exit status of 'if' itself, and PIPESTATUS
contains the exit statuses of the commands from the pipeline.  If
you want $? to contain the exit status of the pipeline instead, you
need to reference it from within the 'if' statement.

hobbit:~$ if (exit 42); then echo "yes $?"; else echo "no $?"; fi
no 42

Once the 'if' terminates, the exit status of the pipeline is no longer
reliably available through $?.

Re: Erasing sensitive data from memory?

[Greg Wooledge]

On Sun, Apr 21, 2024 at 02:16:57PM -0400, Zachary Santer wrote:
> $ IFS='' read -e -r -s -p 'password: ' password
> password:

 seems to be relevant here.
I won't say that you have malicious intent here, but a script that
behaves in this way is just a step or two away from being a password
intercepter.

Re: [sr #111051] New commands: `-h`, `--help`

[Greg Wooledge]

On Thu, Apr 18, 2024 at 03:20:21PM +0700, Robert Elz wrote:
> ps: bash could probably lose the "be a login shell" '-' from argv[0][0]
> for error messages.   It isn't helpful.

It's a tiny bit helpful for someone who knows what it means, and harmless
for people who don't know.  I'd prefer it to be left as is.

Re: [sr #111051] New commands: `-h`, `--help`

[Greg Wooledge]

On Thu, Apr 18, 2024 at 03:03:32AM -0400, anonymous wrote:
> they gave me reply:
> 
> 'There isn't command `-h` on my Limux'
> 
> Therefore, after calling -h/--help, I suggest displaying a message like:

Adding a /usr/bin/-h command or whatever sounds like overkill to me.
I wouldn't want that to be present on every system in the world.

If you're maintaining a system that has extremely novice users on it,
you're going to be the one selecting their shell for them, so you can
just customize their shell.

E.g. add a -h() function in /etc/bash.bashrc (or whatever your Linux
distribution uses for that filename) which prints the message you'd
like your users to see.

Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators

[Greg Wooledge]

On Mon, Apr 15, 2024 at 08:13:23PM +0200, Emanuel Attila Czirai wrote:
> On Mon, Apr 15, 2024 at 7:56 PM Greg Wooledge  wrote:
> > Sounds like you've found a nontrivial bug in FreeBSD (in the adduser
> > script, not in sh).  I hope this gets reported and fixed, and in any case,
> > good work and thank you.
> >
> It's nothing really,
> 
> there's code in adduser that does this:
> [ -z ">" -a -z ">" ] && continue
> which errors like:
> [: -a: unexpected operator
> but the > are $passwordvars

And that's a bug.  That code is wrong, and it should be written this way
instead:

[ -z "$var1" ] && [ -z "$var2" ] && continue

Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators

[Greg Wooledge]

On Mon, Apr 15, 2024 at 07:04:23PM +0200, Emanuel Attila Czirai wrote:
> In my superficial report, I definitely didn't think of that. I even forgot
> to mention that it works when escaped like "\>"
> 
> I've encountered it in the "adduser" FreeBSD sh script that runs as root,
> while trying to set a one char password like ">", so I thought I'd mention
> it here as well in case it might be helpful since I saw it happens in bash
> as well.

Sounds like you've found a nontrivial bug in FreeBSD (in the adduser
script, not in sh).  I hope this gets reported and fixed, and in any case,
good work and thank you.

Re: syntax error with lone > or < as string in [ ] tests with -a or -o operators

[Greg Wooledge]

On Sun, Apr 14, 2024 at 11:16:27AM +0200, Emanuel Attila Czirai wrote:
> $ [ -n ">" -a -n "something" ] || echo hmm
> bash: [: syntax error: `-n' unexpected
> hmm

Don't do this.  You're in the land of unspecified behavior here.

Use multiple test or [ commands instead, or use bash's [[ command.

[ -n ">" ] && [ -n "something" ]

[[ -n ">" && -n "something" ]]

> Note, the issue is present also in (current latest) FreeBSD's '/bin/sh' and
> 'bash' and `/bin/[`.

That's because they all try to implement the POSIX rules.

> But doesn't happen on Gentoo's /usr/bin/[ which is from
> sys-apps/coreutils-9.5::gentoo

Who knows what that one does.

Re: echo test >& "quote'test"

[Greg Wooledge]

On Tue, Apr 09, 2024 at 01:37:08AM +, squeaky wrote:
> Bash Version: 5.2 Patch Level: 21 Release Status: release
> 
> Description:
> 
> Running
> echo test >& "quote'test"
> should create the file "quote'test", but it creates "quotetest" 
> instead.

I can confirm this all the way back to bash 2.05b, and only with the >&
redirection operator.  Not with &> or > or >> .

Re: Potential Bash Script Vulnerability

[Greg Wooledge]

On Mon, Apr 08, 2024 at 02:23:18PM +0300, ad...@osrc.rip wrote:
> Btw wouldn't it be possible (and worth) temporarily revoking write access to
> the user while it's being executed as root, and restoring original rights
> after execution?

I think that would be a huge overreach.  It would also lead to a whole
lot of breakage.

Imagine that we implement this change.  It would have to be done in
the shell, since the kernel simply offloads script execution to the
interpreter.  So, your change would essentially add code to the shell
which causes it to change the permissions on a script that it's
reading, if that script is given as a command-line argument, and if
the shell's EUID is 0.  Presumably it would change the permissions
back to normal at exit.

Now imagine what happens if the shell is killed by a SIGKILL, or if
the system simply crashes during the script's execution.  The script
is left with altered permissions.

Re: Potential Bash Script Vulnerability

[Greg Wooledge]

On Mon, Apr 08, 2024 at 12:40:55PM +0700, Robert Elz wrote:
> or perhaps better just:
> 
>   main() { ... } ; main "$@"

You'd want to add an "exit" as well, to protect against new lines of
code being appended to the script.

Re: Potential Bash Script Vulnerability

[Greg Wooledge]

On Mon, Apr 08, 2024 at 12:23:38AM +0300, ad...@osrc.rip wrote:
> - Looks for list of PIDs started by the user, whether it's started in 
> terminal or command line, and saves them into $DotShProcessList

> - Takes $DotShProcessList and filters out those that don't have root access. 
> Those that do are saved into $UserScriptsRunningAsRoot

> - Searches for file names of $UserScriptsRunningAsRoot processes in 
> /home/$USER (aka ~) and save it to $ScriptFiles

So your "vulnerability" requires that the attacker has unprivileged
access to the system, and locates a shell script which is owned by a
second unprivileged user, and for some reason has world write access,
and is also currently being executed by root?

In that scenario I would say the real problem is that the second user
is leaving world-writable files sitting around.  If the attacker finds
such scripts, they can edit them ahead of time, and simply wait for
the second user to execute them via sudo.  There's no need to find the
script being executed in real time.

Re: Scope change in loops with "read" built-in

[Greg Wooledge]

On Thu, Apr 04, 2024 at 08:39:51PM -0400, Dale R. Worley wrote:
> To circumvent that, I've sometimes done things like
> 
> exec 3<( ... command to generate stuff ... )
> while read VAR <&3; do ... commands to process stuff ... ; done
> exec 3<-

Please note that the syntax for closing an FD is 3<&-

I may be a bit hypersensitive to this one, after
.

Re: Scope change in loops with "read" built-in

[Greg Wooledge]

On Tue, Apr 02, 2024 at 08:08:57PM +, Linde, Evan wrote:
> In a loop constructed like `... | while read ...`, changes to 
> variables declared outside the loop only have a loop local
> scope, unlike other "while" or "for" loops.

https://mywiki.wooledge.org/BashFAQ/024

Re: > /dev/stderr 2>&-

[Greg Wooledge]

On Fri, Mar 29, 2024 at 09:02:12PM +1100, Reuben wrote:
> $ echo cat /dev/stderr > bug
> $ bash bug 2>&-
> cat /dev/stderr

I don't understand what you were trying to do here.

> calling bash script 2>&- on linux
> seems to make /dev/stderr refer to script,
> though &2 seems unaffected.
> using 2>&- inside script does not trigger this bug.
> i assume it is a bug and not 'historical compatibility'.
> the bug does not seem to appear in bash on openbsd.
> the bug does not seem to appear in dash or pdksh.

As a first guess, bash was given the name of a file to read as a script,
so it opened that file using the first available file descriptor.  Since
you had already closed FD 2 before invoking bash, FD 2 was the first
available, and therefore the script was opened as FD 2.

To me, this seems like a case of "Doctor, it hurts when I bend my arm
this way."  Maybe Chet will disagree.

Re: Docco

[Greg Wooledge]

On Wed, Mar 27, 2024 at 10:00:06AM +0100, Phi Debian wrote:
> $ man bash
> ...
> CONDITIONAL EXPRESSIONS
> ...
> 
>-a file
>   True if file exists.
>-e file
>   True if file exists.
> ...
> 
> 'May be' would be nice for newbies to precise which options are [ specific
> vs [[ specific for instance
> 
>-a file
>   True if file exists ([[ only, for [ see test builtin)
> 
> This to avoid things like
> 
> $ [   -a /tmp ] && echo ok || echo nok
> ok
> $ [ ! -a /tmp ] && echo ok || echo nok
> ok
> 
> I know it is obvious, unless this is intended to force a complete
> multi-pass man read...

I wouldn't say it's "obvious" what's happening here.  The problem is
that there are two different "-a" operators, one unary, and one binary.
"help test" documents both of them:

hobbit:~$ help test | grep -- -a
  -a FILETrue if file exists.
  EXPR1 -a EXPR2 True if both expr1 AND expr2 are true.

In your first example, you are using the unary -a operator on a file:

> $ [   -a /tmp ] && echo ok || echo nok
> ok

This one returns true because there are two arguments, and the first one
is not '!', and is a "unary primary" (POSIX wording).  Therefore it uses
the unary  -a and tests for existence of the second argument as a file.

In your second example, you are using the binary -a operator:

> $ [ ! -a /tmp ] && echo ok || echo nok
> ok

Here, you have three arguments, and argument 2 is a "binary primary"
(POSIX wording again), so it's treated as if you had written this:

[ ! ] && [ /tmp ] && echo ok || echo nok

This is simply performing two string length tests.  Both strings are
non-empty (the first is one character, and the second is four), so
the result is true.

The check for whether the first argument is '!' is not performed,
because the "$2 is a binary primary" check comes first.  This is how
POSIX documents it.

So... how do you work around this?  Well, the easiest way would be
to stop using -a entirely.  Both the unary *and* binary forms.  The
unary form can be replaced by -e, and then everything works as you
expect.  The binary form should be discarded along with "-o", and
never used.  You are much better off stringing together multiple
test or [ commands instead:

if [ -e "$logdir" ] && [ -e "$outputdir" ]; then ...

This removes all ambiguity, and is in fact the only supported way
to write this under POSIX restrictions (">4 arguments: The results
are unspecified.")

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Sun, Mar 24, 2024 at 03:54:10PM -0500, Dennis Williamson wrote:
> The @K transform outputs key value pairs for indexed arrays as well as
> associative arrays (you used the @k transform which does word splitting and
> loses the k-v sequence).

The @K (capital) transformation gives you quoted strings which need to
be eval'ed.  Very Bourne-shell-ish.

The @k (lowercase) transformation gives you a list of alternating raw
key/value strings, like what you'd expect from a Tcl command.

hobbit:~$ unset -v hash kvlist
hobbit:~$ declare -A hash=([key 1]='value 1' [key 2]='value 2')
hobbit:~$ kvlist=( "${hash[@]@k}" )
hobbit:~$ declare -p kvlist
declare -a kvlist=([0]="key 2" [1]="value 2" [2]="key 1" [3]="value 1")
hobbit:~$ kvlist2=( "${hash[@]@K}" )
hobbit:~$ declare -p kvlist2
declare -a kvlist2=([0]="\"key 2\" \"value 2\" \"key 1\" \"value 1\" ")
hobbit:~$ eval kvlist3=\("${hash[@]@K}"\)
hobbit:~$ declare -p kvlist3
declare -a kvlist3=([0]="key 2" [1]="value 2" [2]="key 1" [3]="value 1")

kvlist2 is an undesired result.  Don't do that.  kvlist and kvlist3 are
both usable.

> Thus the @K allows preserving indices in a sparse
> indexed array.

Both of them do that:

hobbit:~$ sparse=(a b [42]=z)
hobbit:~$ echo "${sparse[@]@K}"
0 "a" 1 "b" 42 "z"
hobbit:~$ echo "${sparse[@]@k}"
0 a 1 b 42 z

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Sun, Mar 24, 2024 at 07:46:46PM +0200, Oğuz wrote:
> On Sunday, March 24, 2024, Zachary Santer  wrote:
> >
> > Yeah, but what can you do with @k?
> 
> 
> It helps when reconstructing an associative array as a JSON object in JQ
> 
> $ declare -A a=([x]=1 [y]=2)
> $ jq --args -n '[$ARGS.positional | _nwise(2) | {(.[0]): .[1]}] | add'
> "${a[@]@k}"
> {
>   "y": "2",
>   "x": "1"
> }

Conceptually that looks great, but how do you avoid "Argument list
too long" with larger inputs?

I've got this example on hand, but it doesn't include a hash:

hobbit:~$ a=("an array" "of strings"); b="a string"; printf '%s\0' "${a[@]}" | 
jq -R --arg b "$b" '{list: split("\u"), string: $b}'
{
  "list": [
"an array",
"of strings"
  ],
  "string": "a string"
}

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Sun, Mar 24, 2024 at 01:04:38PM -0400, Zachary Santer wrote:
> On Fri, Mar 22, 2024 at 11:23 AM Chet Ramey  wrote:
> >
> > This is what you can do with @K.
> >
> > https://lists.gnu.org/archive/html/bug-bash/2021-08/msg00119.html
> >
> > Word splitting doesn't happen on the rhs of an assignment statement, so you
> > use eval. The @K quoting is eval-safe.
> 
> Yeah, but what can you do with @k?

You can write the keys and values to a stream/file.  That's pretty
much it -- but that's actually not too shabby.  Imagine sharing a bash
associative array with some other programming language and wanting to
import it into that other language's dictionary/hash data structure.
This is what you'd want.

> The difference in expansion behavior between indexed and associative
> array compound assignment statements doesn't make sense. As nice as it
> is to have expansions that expand to eval-safe expressions, needing
> eval less would be nicer.

It would be pretty reasonable to have a builtin that could take an array
name plus any number of additional argument pairs, and load those pairs
as keys/values into said array.  Then you could do something like this:

declare -A hash=(...)
kvlist=( "${hash[@]@k}" )

declare -A newhash
addtoarray newhash "${kvlist[@]}"

Some readers may observe that this looks a bit like Tcl's "array set"
command.  That's not a coincidence.  Whenever I see "a list of alternating
keys and values", that's where my mind goes.

I'm debating mentally whether this hypothetical new builtin would only
work with associative arrays, or also double as an "lappend" (append new
elements to the end of a list/array) if given an indexed array name.
I'm leaning toward having it be both.  It wouldn't be any *more* confusing
than the current situation already is.  The main difference would be that
with an indexed array, every argument after the array name becomes an
array element, instead of half of them being keys and the other half
being values.

Then again, I'm not likely to attempt to implement it, so anyone who
actually writes the code gets to make all the decisions.

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Fri, Mar 22, 2024 at 02:09:23PM -0400, Lawrence Velázquez wrote:
> On Fri, Mar 22, 2024, at 12:54 PM, Greg Wooledge wrote:
> > Also, I don't see the lower-case k transformation in the man page.
> 
> It's at the end of the list:
> 
> https://git.savannah.gnu.org/cgit/bash.git/tree/doc/bash.1?id=f3b6bd1#n3525

Ah... for some reason I had a bash.1 page from bash 5.1 in my
/usr/local/share/man/man1 directory, which was being pulled up instead
of Debian's bash 5.2 page.  My fault.

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Fri, Mar 22, 2024 at 11:23:35AM -0400, Chet Ramey wrote:
> This is what you can do with @K.
> 
> https://lists.gnu.org/archive/html/bug-bash/2021-08/msg00119.html
> 
> Word splitting doesn't happen on the rhs of an assignment statement, so you
> use eval. The @K quoting is eval-safe.

It would be good to include that information in the manual.  I only
see this in the man page:

  K  Produces a possibly-quoted version of the value of param‐
 eter, except that it prints the values of indexed and as‐
 sociative arrays as a sequence of quoted key-value  pairs
 (see Arrays above).

The reader is left wondering whether it's eval-safe.

Also, I don't see the lower-case k transformation in the man page.

Re: ${var@A}; hypothetical, related parameter transformations

[Greg Wooledge]

On Wed, Mar 20, 2024 at 03:05:56PM -0400, Zachary Santer wrote:
> I want a declare command, no matter what ${var@A} gives me. I have to
> write a function for that: generate_declare_command (). That function
> can work a couple of reasonable ways:

This seems rather theoretical to me.  If the associative array has
nothing in it, does it really matter whether it's nonexistent, only
declared as a scope placeholder (I think that's a thing...), or fully
declared but empty?  The second script which receives the serialized data
doesn't really care how the first script stored it, does it?  All you
really need is to have the keys and values replicated faithfully in the
second script's associative array.  If that gives you an empty hash,
then that's what you use.

I simply struggle to figure out what the real-world application is for
some of these tangents.

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Wed, Mar 20, 2024 at 12:53:07PM +0100, alex xmb sw ratchev wrote:
> On Wed, Mar 20, 2024, 12:49 Greg Wooledge  wrote:
> 
> > On Wed, Mar 20, 2024 at 07:11:34AM -0400, Zachary Santer wrote:
> > > On Wed, Mar 20, 2024 at 12:29 AM Lawrence Velázquez 
> > wrote:
> > > > A couple of previous discussions:
> > > >   - https://lists.gnu.org/archive/html/bug-bash/2020-12/msg00066.html
> > > >   - https://lists.gnu.org/archive/html/bug-bash/2023-06/msg00128.html
> > >
> > > There I go, reporting a bug that isn't a bug again.
> > >
> > > One would think that enabling this behavior would be the entire
> > > purpose of the alternate ( key value ) syntax. If it doesn't do that,
> > > what benefit does it give over the standard ( [key]=value ) syntax?
> > > Maybe it;s easier to use eval with?
> >
> > I believe the "X" in this X-Y problem is "I want to serialize an
> > associative array to a string, send the string to another bash script,
> > and de-serialize it back into an associative array there."
> >
> 
> as to that , simply declare -p and run that back

That will work in some cases, yes.  The problem is that it locks you
in to having the same array name and the same attributes (if any) as
the original array had.  It also runs "declare" which can create issues
with scope, if it's done inside a function.  Perhaps you wanted to
populate an array that was previously declared at some scope other
than your own, and other than global (which means adding -g wouldn't
work either).

I suppose you might argue that you could run declare -p, and then
edit out the "declare" and the attributes, and then while you're at it,
also edit the array name.  I guess that's a possible solution, but it
really seems ugly.

Re: "${assoc[@]@k}" doesn't get expanded to separate words within compound assignment syntax

[Greg Wooledge]

On Wed, Mar 20, 2024 at 07:11:34AM -0400, Zachary Santer wrote:
> On Wed, Mar 20, 2024 at 12:29 AM Lawrence Velázquez  wrote:
> > A couple of previous discussions:
> >   - https://lists.gnu.org/archive/html/bug-bash/2020-12/msg00066.html
> >   - https://lists.gnu.org/archive/html/bug-bash/2023-06/msg00128.html
> 
> There I go, reporting a bug that isn't a bug again.
> 
> One would think that enabling this behavior would be the entire
> purpose of the alternate ( key value ) syntax. If it doesn't do that,
> what benefit does it give over the standard ( [key]=value ) syntax?
> Maybe it;s easier to use eval with?

I believe the "X" in this X-Y problem is "I want to serialize an
associative array to a string, send the string to another bash script,
and de-serialize it back into an associative array there."

So, the first thing we observe is the behavior of the @k and @K expansions:

hobbit:~$ declare -A aa=('key 1' 'value 1' 'key 2' 'value 2')
hobbit:~$ declare -p aa
declare -A aa=(["key 2"]="value 2" ["key 1"]="value 1" )
hobbit:~$ printf '<%s> ' "${aa[@]@k}"; echo

hobbit:~$ printf '<%s> ' "${aa[@]@K}"; echo
<"key 2" "value 2" "key 1" "value 1" > 

Essentially, @k serializes the associative array to a list of alternating
keys/values, while @K serializes it to a string.

For the purposes of sending the serialization to another script, the
list is not suitable unless we NUL-terminate each element.  We could
pursue that, but it's going to involve more work, I suspect.

Given the string serialization we get from @K, it makes sense to start
there, and try to determine whether it's eval-safe.

hobbit:~$ declare -A mess=('*' star '?' qmark $'\n' nl '$(date)' cmdsub '`id`' 
backticks)
hobbit:~$ declare -p mess
declare -A mess=([$'\n']="nl" ["?"]="qmark" ["*"]="star" ["\$(date)"]="cmdsub" 
["\`id\`"]="backticks" )

That's not comprehensive, but it's a start.

hobbit:~$ printf '<%s> ' "${mess[@]@K}"; echo
<$'\n' "nl" "?" "qmark" "*" "star" "\$(date)" "cmdsub" "\`id\`" "backticks" > 
hobbit:~$ serial="${mess[@]@K}"
hobbit:~$ printf '<%s>\n' "$serial"
<$'\n' "nl" "?" "qmark" "*" "star" "\$(date)" "cmdsub" "\`id\`" "backticks" >

There's our serialization to a string.  Now we'll try to de-serialize:

hobbit:~$ eval "declare -A copy=($serial)"
hobbit:~$ declare -p copy
declare -A copy=([$'\n']="nl" ["?"]="qmark" ["*"]="star" ["\$(date)"]="cmdsub" 
["\`id\`"]="backticks" )

Looks OK.  Let's verify another way:

hobbit:~$ for i in "${!copy[@]}"; do printf '<%s>: <%s>\n' "$i" "${copy[$i]}"; 
done
<
>: 
: 
<*>: 
<$(date)>: 
<`id`>: 

Unless someone else comes up with a key or value that breaks the eval,
this looks like the simplest way.

By comparison, the NUL-terminated @k list can't even be stored in a
variable, so you'd need to transmit it to the other script in some way
that's equivalent to using a temp file.  Thus:

hobbit:~$ printf '%s\0' "${mess[@]@k}" > tmpfile
hobbit:~$ unset -v copy; declare -A copy; while IFS= read -rd '' key && IFS= 
read -rd '' value; do copy[$key]=$value; done < tmpfile
hobbit:~$ declare -p copy
declare -A copy=([$'\n']="nl" ["?"]="qmark" ["*"]="star" ["\$(date)"]="cmdsub" 
["\`id\`"]="backticks" )

I can't think of any way to restore an @k serialization other than a
IFS= read -rd '' loop.  The only advantage I can see here is that it
doesn't use eval, and therefore is visibly safe.  With the eval @K
solution, we're still left wondering whether the script is a ticking
time bomb.

Re: nameref and referenced variable scope, setting other attributes (was "local -g" declaration references local var in enclosing scope)

[Greg Wooledge]

On Thu, Mar 14, 2024 at 10:15:35AM -0400, Zachary Santer wrote:
> > $ cat ./nameref-what
> > #!/usr/bin/env bash
> >
> > func_1 () {
> >   local var_3='EGG'
> >   func_2
> >   printf '%s\n' "func_1:"
> >   local -p var_3
> > }
> >
> > func_2 () {
> >   local -n nameref_3='var_3'
> >   nameref_3='soufflé'
> >   local var_4='GROUND BEEF'
> >   local -n nameref_4='var_4'
> >   local -l nameref_4
> >   printf '%s\n' "func_2:"
> >   local -p nameref_3
> >   local -p var_3
> >   local -p nameref_4
> >   local -p var_4
> > }
> >
> > func_1
> >
> > $ ./nameref-what
> > func_2:
> > declare -n nameref_3="var_3"
> > ./nameref-what: line 31: local: var_3: not found
> > declare -n nameref_4="var_4"
> > declare -l var_4="GROUND BEEF"
> > func_1:
> > declare -- var_3="soufflé"
> 
> Not on a machine with bash right now. 'declare -p var_3' in func_2 ()
> said var_3 was not found, despite it having just been set by assigning
> a value to the nameref variable nameref_3.

I can't seem to duplicate this.  This is with bash 5.2:

hobbit:~$ cat foo
#!/bin/bash

outer() {
local var3=EGG
inner
}

inner() {
local -n ref3=var3
ref3=BANANA
declare -p ref3
declare -p var3
}

outer
hobbit:~$ ./foo
declare -n ref3="var3"
declare -- var3="BANANA"

And I get the same result in older versions as well:

hobbit:~$ bash-5.1 foo
declare -n ref3="var3"
declare -- var3="BANANA"
hobbit:~$ bash-5.0 foo
declare -n ref3="var3"
declare -- var3="BANANA"
hobbit:~$ bash-4.4 foo
declare -n ref3="var3"
declare -- var3="BANANA"
hobbit:~$ bash-4.3 foo
declare -n ref3="var3"
declare -- var3="BANANA"

Re: nameref and referenced variable scope, setting other attributes (was "local -g" declaration references local var in enclosing scope)

[Greg Wooledge]

On Thu, Mar 14, 2024 at 08:29:47AM -0400, Zachary Santer wrote:
> Alright, that's all fair. But this?
> 
> On Sun, Mar 10, 2024 at 7:29 PM Zachary Santer  wrote:
> >
> > Additionally, a nameref variable referencing a variable declared in a 
> > calling function hides that variable in the scope of the function where the 
> > nameref variable is declared.
> 

I don't quite understand what this is saying.  Do the variables have
different names, or the same name?  If they have different names, then
the nameref shouldn't "hide" the other variable.  But they can't have
the same name, because a nameref pointing to itself is a circular
reference and won't ever work under any circumstance.

hobbit:~$ f() { local -n var=var; var=1; }; f
bash: local: warning: var: circular name reference
bash: warning: var: circular name reference
bash: warning: var: circular name reference

You don't even need an outer calling function to see this.

Re: nameref and referenced variable scope, setting other attributes (was "local -g" declaration references local var in enclosing scope)

[Greg Wooledge]

On Thu, Mar 14, 2024 at 08:27:33AM +0100, alex xmb sw ratchev wrote:
> how to unset a nameref

Look at "help unset".  Specifically the -n option.

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote:
> Here's the script and the Makefile using "printf '<%s>'":

Sadly, your mail user agent chose to attach "Makefile" with content-type
application/octet-stream, which my MUA (mutt) refuses to show inline,
or to include in a reply as quoted text.

Here's the top part of it, inline:


STR[0]="one and two"
STR[1]=one\ and\ two

CFLAGS[0]=-D__STRINGIZED__=0 -D__STRING__=${STR[0]}
CFLAGS[1]=-D__STRINGIZED__=0 -D__STRING__=${STR[1]}
CFLAGS[2]=-D__STRINGIZED__=1 -D__STRING__=${STR[0]}
CFLAGS[3]=-D__STRINGIZED__=1 -D__STRING__=${STR[1]}


And here's what Martin said about it:

> On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey 
> wrote:
> > In section two, the problem is that quote removal is done BEFORE variables
> > are expanded, even though it prevents word splitting from being done AFTER
> > variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> > absolutely cannot do what you might expect; the embedded quote marks will
> > be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> > resulting string will be split on any embedded whitespace..


As someone said yesterday, you will need eval for this.


hobbit:/tmp/x$ cat Makefile
FOO=-D__x="one two three"

all:
@bash -c 'eval '\''CFLAGS=${FOO}'\''; declare -p CFLAGS'
hobbit:/tmp/x$ make
declare -- CFLAGS="-D__x=one two three"


Of course, using eval presents its own set of challenges, so proceed
with extreme caution.

I'd still like to hear why you aren't simply using "make -j".

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 02:15:38PM +0700, Robert Elz wrote:
> Date:Mon, 11 Mar 2024 17:25:57 -0400
> From:Chet Ramey 
> Message-ID:  <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>
> 
>   | OK, here's the longer answer. When the shell is interactive, and job
>   | control is enabled, the shell prints job completion notifications to
>   | stdout at command boundaries.
> 
> which is, IMO, yet another bogus bash misfeature.  That should
> happen, with the effect described, but only when PS1 is about
> to be printed - precisely so commands like the one described
> will work.   Sigh.
> 
> There aren't many complaints about this misfeature of bash,
> as almost no-one writes interactive command lines where it makes
> a difference.   That doesn't mean they should not be able to.

Yeah, it appears you're right.  In a script, this works as expected:

hobbit:~$ cat foo
#!/bin/bash
for i in {0..3}; do sleep 1 & done
for i in {0..3}; do
wait -n -p pid; e=$?
printf 'pid %s status %s\n' "$pid" "$e"
done
hobbit:~$ ./foo
pid 530359 status 0
pid 530360 status 0
pid 530361 status 0
pid 530362 status 0


But interactively, *even with bash -c and set +m*, it just fails:

hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; for i in {0..3}; 
do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e"; done'
pid 530407 status 0
pid 530410 status 0
pid  status 127
pid  status 127

Looks like a race condition, where some of the children get reaped and
tossed away before "wait -n -p pid" has a chance to grab their status.

If I stick a "sleep 2" in between the two loops, then it's even worse:

hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; sleep 2; for i 
in {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e"; 
done'pid  status 127
pid  status 127
pid  status 127
pid  status 127

ALL of the children are discarded before the second loop has a chance to
catch a single one of them.  This is clearly not working as expected.

Using "bash +i -c" doesn't change anything, either.  Or "bash +i +m -c".
Whatever magic it is that you get by putting the code in an actual
script, I can't figure out how to replicate it from a prompt.

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 10:33:36AM +0100, Mischa Baars wrote:
> bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
> sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
> "$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'

"wait -p pid" is not correct here.  In that command, pid is an output
variable, and you have not specified any process IDs to wait for -- so
wait is going to wait for *all* of the children to finish, and you'll
get zero as the exit status:

$ help wait
[...]

Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status.  If ID is not
given, waits for all currently active child processes, and the return
status is zero.  If ID is a job specification, waits for all processes
in that job's pipeline.

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
> On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
> > The pid can be obtained with the -p option, as of 5.1. Below is a
> > synthetic example of how it might be put into practice.

I'd forgotten about that one.  A recent addition, and one I've never used
yet.

> > #!/bin/bash
> >
> > declare -A job_by status_by
> > max_jobs=4
> > jobs=0
> >
> > wait_next() {
> > local pid
> > wait -n -p pid
> > status_by[$pid]=$?
> >
> 
> How exactly is this indexing implemented internally? Does a first number on
> index n take m bits (using a linked list) or does it take n * m bits (using
> realloc(3))?

"declare -A" makes it an associative array (hash table).

Without that declare -A, it would be a sparsely indexed array.

In either case, it doesn't just blindly allocate millions of bytes of
memory.  It uses something slimmer.

Re: multi-threaded compiling

[Greg Wooledge]

> On Mon, Mar 11, 2024, 20:13 Mischa Baars 
> wrote:
> 
> > Also I don't think that gives you an exit status for each 'exit $i'
> > started. I need that exit status.

"wait -n" without a PID won't help you, then.  You don't get the PID or
job ID that terminated, and you don't get the exit status.  It's only
of interest if you're trying to do something like "run these 100 jobs,
5 at a time" without storing their exit statuses.

If you need to reap each individual job and store its exit status indexed
by its PID, then you're probably going to need something like:


#!/bin/bash
i=0 pid=() status=()
for job in ...; do
longrunner "$job" & pid[i++]=$!
done

for ((i=0; i < ${#pid[@]}; i++)); do
wait "${pid[i]}"; status[i]=$#
done


You won't be able to take advantage of "wait -n"'s ability to react
to the first job that finishes.  You'll end up reaping each job in the
order they started, not the order they finished.

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 07:22:39PM +0100, Mischa Baars wrote:
> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
> wrote:
> 
> > i also completly dont get ur issue
> >
> > f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
> >
> >   while [[ -v f[++i] ]] ; do
> >  (( ++r > threads )) &&
> > wait -n
> > gcc -c "${f[i]}" &
> >   done
> >
> 
> How nice!
> 
> wait -n exit 1 & echo $?
> 
> You got me the solution :) Except that wait expects a pid after -n.

No, wait -n without a PID waits for any one job to complete.

hobbit:~$ sleep 123 & sleep 234 & sleep 3 & wait -n
[1] 513337
[2] 513338
[3] 513339
[3]+  Donesleep 3
hobbit:~$ ps
PID TTY  TIME CMD
   1138 pts/000:00:00 bash
 513337 pts/000:00:00 sleep
 513338 pts/000:00:00 sleep
 513341 pts/000:00:00 ps
hobbit:~$ 

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; 
> sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; 
> echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> /bin/bash: line 1: wait: 1747087: no such job
>   0 pid 1747087 exit 127
> /bin/bash: line 1: wait: 1747088: no such job
>   1 pid 1747088 exit 127

Without analyzing this in depth, one thing struck me immediately:
you're using the reserved variable SECONDS, which has special semantics
in bash.  ${SECONDS} is going to expand to an ever-increasing number,
beginning with 5 (since that's the value you assigned), and going up
by 1 per second that the script runs.  I'm assuming that was not your
intention.

In general, any variable whose name is all capital letters *may* have
special meaning to the shell and should not be used for general storage
purposes in scripts.

Re: "local -g" declaration references local var in enclosing scope

[Greg Wooledge]

On Sun, Mar 10, 2024 at 06:39:19PM -0400, Lawrence Velázquez wrote:
> On Sun, Mar 10, 2024, at 5:36 PM, Greg Wooledge wrote:
> > Here it is in action.  "local -g" (or "declare -g") without an assignment
> > in the same command definitely does things.
> >
> > hobbit:~$ f() { declare -g var; var=in_f; }
> > hobbit:~$ unset -v var; f; declare -p var
> > declare -- var="in_f"
> 
> This example appears to work the same without "declare -g":
> 
>   $ f() { var=in_f; }
>   $ unset -v var; f; declare -p var
>   declare -- var="in_f"

The impetus for adding it was that you *have* to declare associative
arrays.  If you want to create a global associative array from within a
function, you need the -g option.

For regular variables, it's mostly redundant, yes.

Re: "local -g" declaration references local var in enclosing scope

[Greg Wooledge]

On Sun, Mar 10, 2024 at 04:01:10PM -0400, Lawrence Velázquez wrote:
> Basically, without an assignment, "local -g" does nothing.

Well, the original purpose of -g was to create variables, especially
associative arrays, at the global scope from inside a function.

I think this thread has been asking about a completely different
application, namely to operate upon a global scope variable from
within a function where a non-global scope variable is shadowing the
global one.

(As far as I know, there isn't any sensible way to do that in bash.)


Here it is in action.  "local -g" (or "declare -g") without an assignment
in the same command definitely does things.

hobbit:~$ f() { declare -g var; var=in_f; }
hobbit:~$ unset -v var; f; declare -p var
declare -- var="in_f"


I think the key to understanding is that while "local -g var" creates
a variable at the global scope, any references to "var" within the
function still use the standard dynamic scoping rules.  They won't
necessarily *see* the global variable, if there's another one at a
more localized scope.

Re: Unable to close a file descriptor using exec command

[Greg Wooledge]

On Sun, Mar 03, 2024 at 10:29:17PM +, Venkat Raman via Bug reports for the 
GNU Bourne Again SHell wrote:
> Repeat-By:
> exec 2> commands freezes the terminal and unable to close the 
> fd.
> 
> Fix:
> [Is there a sane  way to close the fd?]

You're using the wrong syntax.

To open a file (for writing + truncation):  exec 2>file

To open a file (for appending):  exec 2>>file

To close a file descriptor:  exec 2>&-


The command that you ran (exec 2>) redirects stderr which is
normally opened to the terminal.  In order to get it back, you would
need to do something like  exec 2>&1   assuming stdout has not been
altered yet.

The shell prompt is normally written to stderr, so you won't see the
prompt until stderr is restored.

Re: possible bash bug bringing job. to foreground

[Greg Wooledge]

On Sat, Feb 17, 2024 at 07:41:43PM +, John Larew wrote:
> After further examination, the examples with "fg $$" and "fg $!" clearly do 
> not bring the subshell into the foreground, as they are evaluated prior to 
> the subshells background execution.
> I'm trying to bring the subshell to the foreground to perform an exit, after 
> a delay.
> Ultimately, it will be used as part of a terminal emulator inactivity timeout.

Bash already has a TMOUT variable which will cause an interactive shell
to exit after a specified length of inactivity.  Is that sufficient?
If not, how does your desired solution need to differ from TMOUT?

Re: possible bash bug bringing job. to foreground

[Greg Wooledge]

On Sat, Feb 17, 2024 at 01:30:00PM +, John Larew wrote:
> Repeat-By:  1: (sleep 15s; set -m; fg %%; exit ) & 2: (sleep 15s; set -m; fg 
> %+; exit ) & 

You're using %% or %+ inside a shell where there have NOT been any
background jobs created yet.  The sleep 15s runs in the foreground,
because it doesn't have a & after it.

> Fix: (sleep 15s; set -m; kill $PPID) &     Not a preferred solution; I prefer 
> a smaller hammer.

It's not clear to me what you're trying to do in the first examples.
Did you think that the subshell would somehow "take over" the terminal
and become the interactive shell, after the 15 second delay?  And what
would that achieve, if it worked?

In the "Fix" example, you're killing the main shell's parent, which
is most likely a terminal emulator.  Did you want to kill the main
shell instead of the terminal?  If so, you can get its PID with
the $$ variable.  Even inside a subshell, $$ always refers to the main
shell's PID.  If you want the subshell's PID, use BASHPID instead.

What exactly is it about the "Fix" example that you don't prefer?
What goal are you trying to achieve?

Re: printf - strange behaviour in debug mode (BASH_VERSION="5.2.21(1)-release")

[Greg Wooledge]

On Thu, Feb 08, 2024 at 05:48:18PM +0200, Timotei Campian wrote:
> The culprit function is *printargs() { printf "%s|" "$@"; echo; }*
> 
> calling it with no arguments it prints as expected the pipe: "|"
> while in debug mode (set -x), only a *blank line* is printed:
> 
> $ printargs
> printargs
> + printargs
> + printf %sx
> x+ echo
> 
> $

It looks like your session used a different definition of printargs,
which has printf "%sx" instead of printf "%s|".

Here's what I get with the correct function:

unicorn:~$ bash-5.2
unicorn:~$ printargs() { printf "%s|" "$@"; echo; }
unicorn:~$ set -x
unicorn:~$ printargs
+ printargs
+ printf '%s|'
|+ echo

unicorn:~$ 

This one has the pipeline where yours has the x, in both places.

Re: set-e and command expansion

[Greg Wooledge]

On Sun, Feb 04, 2024 at 08:27:56PM +0300, Van de Bugger wrote:
> Case 3: echo $(false)
> 
> $ cat ./test  
> #!/bin/bash  
> set -e  
> echo before  
> echo $(false)  
> echo after
> 
> $ ./test  
> before
> 
> after
> 
> Oops, in this case the script is NOT terminated before "echo after", but
> continues to the end. I would say this is a bug, but interaction between "set 
> -
> e" and command substitution is not well defined in the bash manual.

I'm sure you can find explanations somewhere in the manual, after you
read it half a dozen times and then ask the Internet about the current
state of the POSIX standard and the current interpretation of how set -e
should work.  (It changes over time.)

Or... you could simply stop trying to understand it.  Dump it in the
rubbish bin of history where it belongs.

 offers some more examples of
surprising behavior, along with explanations of each example.

Re: glob-expand-word and vi-command mode

[Greg Wooledge]

On Fri, Feb 02, 2024 at 03:39:54PM +0100, Mike Jonkmans wrote:
> [ mkdir test; cd test; touch file1 file2 ]
> 
> Going into `vi-command' mode on the line `ls *' puts the cursor on the `*'.
> Then `glob-expand-word' does nothing with the `*', it just inserts a space.
> Resulting in `ls  *' (cursor still on `*').
> Expected: nothing happens.

I'm not sure what keystrokes you're actually using, or what bind calls
you've done leading up to this, but in a vanilla instance of bash with
nothing done except 'set -o vi', typing

l s space * esc *

will replace the * with file1 file2 and another space, and also puts
you in insert mode for some reason.  Probably historical.

Re: incorrect cursor position when navigating with 'ALT-b'

[Greg Wooledge]

On Thu, Feb 01, 2024 at 01:55:50PM +0100, Korneel Dumon wrote:
> Repeat-By:
> Execute the following command:
> > curl -X POST -H "Content-Type: application/json"
> https://ec.europa.eu/taxation_customs/vies/rest-api/check-vat-test-service
> -d '{"countryCode":"BE", "vatNumber":"00"}'
> Then use "arrow up" to recall the command and use "ALT-b" to navigate
> to 'test-service' and
> change it to 'number'. Execute this command. It will fail and if you
> recall it again using "arrow up"
> you should see that the text was changed in the wrong place (offset of
> one character).

This might be caused by an incorrect PS1 (prompt) variable.  You must
ensure that any non-printing sequences in PS1 (colors, etc.) are
enclosed in \[ \] characters.  See
 for details.

If changing to PS1='\$ ' or something similar makes the problem go away,
then you know it's your PS1 causing it.

Re: wait -n misses signaled subprocess

[Greg Wooledge]

On Mon, Jan 29, 2024 at 08:52:37PM +0700, Robert Elz wrote:
> Date:Mon, 29 Jan 2024 13:54:10 +0100
> From:Andreas Schwab 
> Message-ID:  
> 
>   | n = next?

This was my assumption as well.

> That would be a reasonable interpretation, I guess, but
> unfortunately not one which helps the current question,
> as it doesn't answer "next what?"

For the record, with bash 5.2:


unicorn:~$ cat foo
#!/bin/bash

sleep 1 &
sleep 37 &
sleep 2
time wait -n
unicorn:~$ ./foo
real 0.001  user 0.000  sys 0.001
unicorn:~$ ps
PID TTY  TIME CMD
   1152 pts/300:00:00 bash
 542197 pts/300:00:00 sleep
 542200 pts/300:00:00 ps
unicorn:~$ ps -fp 542197
UID  PIDPPID  C STIME TTY  TIME CMD
greg  542197   1  0 08:59 pts/300:00:00 sleep 37


wait -n *does* appear to acknowledge the already-terminated child process,
despite a second child process still being active.

Re: wait -n misses signaled subprocess

[Greg Wooledge]

On Sun, Jan 28, 2024 at 10:26:27PM -0500, Dale R. Worley wrote:
> The man page doesn't make clear that if you don't specify "-n" and do
> supply ids and one of them has already terminated, you'll get its status
> (from the terminated table); the wording suggests that "wait" will
> always *wait for* a termination.

This might be a result of C programmers who already know the semantics
of wait(2) writing documentation which assumes the reader *also* knows
these semantics.

wait(2) and its brethren return immediately if the process in question
has already terminated.  It's how you reap the zombie and free up the
process table slot, while also retrieving its exit status.  If it's
not already dead, then wait(2) blocks until death occurs.

The shell's "wait" command is meant to mimic this behavior, at its core.
There are some differences, however -- notably, the shell aggressively
reaps zombies and stores their exit statuses in memory, revealing them
to you in the event that you call "wait".  Normally this change is
invisible, but if you were *counting* on the zombie to be there, holding
on to that PID, preventing it from being reused until you could observe
the death and react to it, then you're screwed.  Don't use the shell for
this.

Anyway... a script writer who has a basic familiarity with wait(2) and
who reads about "wait -n" will probably assume that wait -n will return
immediately if a child process has already terminated and hasn't been
"pseudo-reaped" by a previous "wait" command yet.  If three children
have terminated, then the next three "wait -n" commands should return
immediately, and the fourth should block (assuming a fourth child exists).

Re: Path of exported variable not set as source

[Greg Wooledge]

On Fri, Jan 26, 2024 at 03:28:11PM +0200, Ricky Tigg wrote:
> $ export EDITOR='/usr/bin/nano' && source $HOME/.bashrc
> $ echo $EDITOR
> /usr/bin/vim
> 
> Expected: Latest command's output to be "/usr/bin/nano" instaed of
> "/usr/bin/vim".

It's really unclear to me why you expected this.  You're changing
the variable, and then sourcing a file that changes the variable
again.  The last change wins.

It's no different from something like this:

unicorn:~$ a=5 && a=7
unicorn:~$ echo "$a"
7

You can see why the value is set to 7, right?  It was set to 5, and then
set to 7, and the last change is the one that sticks around.

> I unexpectedly get the desired behaviour without invoking 'source'.
> 
> $ export EDITOR='/usr/bin/nano'
> $ echo $EDITOR
> /usr/bin/nano

This *surprises* you?  Why are you surprised that the value you set it
to is the value that it has?

Why are you even calling "source" at all?

> Hopefully, my interpretation of the 'source' command is correct.

I don't think I understand what your interpretation is.  What do you
think 'source' does?

For the record:

unicorn:~$ help source
source: source filename [arguments]
Execute commands from a file in the current shell.

Read and execute commands from FILENAME in the current shell.  The
entries in $PATH are used to find the directory containing FILENAME.
If any ARGUMENTS are supplied, they become the positional parameters
when FILENAME is executed.

Exit Status:
Returns the status of the last command executed in FILENAME; fails if
FILENAME cannot be read.

Re: ./script doesn't work in completion function

[Greg Wooledge]

On Mon, Jan 22, 2024 at 12:17:12PM +0200, Oğuz wrote:
> On Monday, January 22, 2024, Martin D Kealey 
> wrote:
> >
> > You seem to have created an invalid executable. It seems that scripts
> > without a #! can only be run with help from the debugger library
> >
> 
> Hi Martin, POSIX shells interpret such executables as shell scripts, I
> don't think bash would take a hacky approach like that for implementing
> this basic functionality. It must be something else

Bash attempts to execute the command normally, and when the kernel
returns ENOEXEC, bash decides to try a different approach.  It opens
the file, reads a hundred or so bytes, and scans them for NUL bytes.
If it doesn't find a NUL, it forks a subshell of itself, and uses
that to read the file as a script.  If a NUL is found, bash aborts with
an error message.

Without a NUL:

unicorn:~$ printf 'echo foo' > foo
unicorn:~$ strace -f bash -c ./foo
execve("/usr/bin/bash", ["bash", "-c", "./foo"], 0x7ffe1badab60 /* 53 vars */) 
= 0
[...]
execve("./foo", ["./foo"], 0x562def3ca230 /* 53 vars */) = -1 ENOEXEC (Exec 
format error)
openat(AT_FDCWD, "./foo", O_RDONLY) = 3
read(3, "echo foo", 128)= 8
close(3)= 0
[...]

With a NUL:

unicorn:~$ printf 'echo foo\0' > foo
unicorn:~$ bash -c ./foo
bash: line 1: ./foo: cannot execute binary file: Exec format error

I don't currently have a build of bash-5.3 (prerelease) to test the
original bug report, so I can neither confirm nor deny it.

But in any case, writing a "script" without a shebang and then counting
on the shell to work around the broken script is not ideal.  You get
differing behaviors depending on which shell you're currently in.
Bash forks a subshell.  Zsh and GNU find fork /bin/sh.  Systemd will
just report the original error and will not do any workarounds.

Re: document that read built-in can't return zero-length string in the middle of input

[Greg Wooledge]

On Sat, Jan 13, 2024 at 02:06:08AM +0700, Robert Elz wrote:
> Date:Fri, 12 Jan 2024 07:15:35 -0500
> From:    Greg Wooledge 
> Message-ID:  
> 
>   | This was one of the things I tested:
> 
> Perhaps intended to, but didn't, or not in this example:
> 
>   | { read -N1; read -r b c; } < <(printf \\nabc); declare -p REPLY a b c
> 
> Rewrite that, correctly for the purpose, as:
> 
> { read -N1; read -r b c; } < <(printf \\nabc); declare -p REPLY b c
> 
> or a bit more cleanly as:
> 
> { read -N1; read -r b c; } < <(printf '\\\nabc'); declare -p REPLY b c

Yes, I made an error there.  Sorry for the confusion.

However, you're testing something slightly different from what I was
testing.  Here's what I originally intended:

unicorn:~$ { read -N1; read -r b c; } < <(printf %s \\nabc); declare -p REPLY b 
c
declare -- REPLY="n"
declare -- b="abc"
declare -- c=""

The important part of the result, for me, is that read -N1 still consumes
two characters of input, and stores one character into the variable.
The stored character is 'n' instead of a newline, after fixing the test.

And here's what you thought I was trying to test:

unicorn:~$ { read -N1; read -r b c; } < <(printf %s $'\\\nabc'); declare -p 
REPLY b c
declare -- REPLY="a"
declare -- b="bc"
declare -- c=""

In this case, read -N1 consumes *three* characters of input, and stores
one character into the variable.  So the character count difference is
even more dramatic.

> ps: the use of process substitution there is just silly, it would work
> just as well, and be easier to understand if written:
> 
>   printf '\\\nabc' | { read -N1; read -r b c; }; declare -p REPLY b c

That requires lastpipe, or moving the declare -p inside the curly braces.

> or{ read -N1; read -r b c; } <<< $'\\\nabc' ; declare -p REPLY b c

<<< unavoidably adds a newline character to the input, which isn't
actually harmful in this case, but isn't what I was trying to test.
I generally go with < <(printf %s ...) when I want to send input that
may not end with a newline.  I simply forgot the %s.

I actually ran several tests before this, which I didn't show in the
previous email.  The first was:

unicorn:~$ { read -r -N1 a b c; } < <(printf abc); declare -p a b c
declare -- a="a"
declare -- b=""
declare -- c=""

I kept making changes to this, ultimately ending up with the test with
the error in it.  I should've started with printf %s in the first place,
but I was lazy.  There's a lesson somewhere in there.

In any case, the important part of this thread is the proposed
documentation change.  My wording may not be ideal, so if you have
any improvement suggestions, feel free to make them.

Re: document that read built-in can't return zero-length string in the middle of input

[Greg Wooledge]

On Fri, Jan 12, 2024 at 03:26:31PM +0700, Robert Elz wrote:
> Date:Thu, 11 Jan 2024 20:02:04 -0500
> From:    Greg Wooledge 
> Message-ID:  
> 
>   | What actually counts is how many
>   | characters are *stored* in the variable, not how many characters are
>   | *read* from the input.
> 
> I suspect that is the root of the issue here, you're considering
> the word "read" to mean read(2) whereas as it is in the documentation
> for the read command, I suspect it means that one, how many characters
> the read command obtains, rather than how many the read system call
> processes (like elided newlines. when -r is not used, would also not
> be counted, or expected to be, right?)

This was one of the things I tested:


unicorn:~$ { read -N1; read -r b c; } < <(printf \\nabc); declare -p REPLY a b 
cdeclare -- REPLY=$'\n'
declare -- a=$'\n'
declare -- b="abc"
declare -- c=""


(The value of 'a' was from a previous test.)  read -N1 consumed two
characters of the input, and stored one character in the variable.
As a script writer, those are the things I care about.  I don't know
or care about the internals of the shell.

Also note that I'm assuming the implementation isn't going to change,
and that our sole goal in this thread is to document its behavior.
Therefore I don't speak about what's "expected" -- merely what is.

I would, however, strongly recommend that script writers who use
read -N *always* include the -r option.

Re: document that read built-in can't return zero-length string in the middle of input

[Greg Wooledge]

On Thu, Jan 11, 2024 at 08:02:04PM -0500, Greg Wooledge wrote:
> And this for the help text:
> 
> -N nchars  return only after storing exactly NCHARS characters, unless
>EOF is encountered or read times out, ignoring any NUL or
>delimiter characters

Actually, that's not good wording.  It implies that NUL and delimiter
characters are treated in the same way, but they aren't.

-N nchars  return only after storing exactly NCHARS characters, unless
   EOF is encountered or read times out, ignoring any NULs, and
   not stopping for delimiters

Re: document that read built-in can't return zero-length string in the middle of input

[Greg Wooledge]

On Fri, Jan 12, 2024 at 01:29:19AM +0100, Ángel wrote:
> One might say "reading exactly nchars characters into the name",

I would still find that confusing.  What actually counts is how many
characters are *stored* in the variable, not how many characters are
*read* from the input.

> but
> given that there's no mention that the NULs are never stored in
> variables, I would tend to add a line below saying e.g. "NUL characters
> cannot be stored in bash variables and are always ignored by read".

I would be as explicit as possible.  Don't require the reader to put
any pieces together themselves.

How about this for the man page:

-N nchars
read returns after storing exactly nchars characters in the
first named variable (or REPLY if no variable is named), unless
EOF is encountered or read times out.  read does not wait for
a complete line of input; any delimiter characters encountered
in the input are not treated specially, and do not cause read to
return before storing nchars characters.  NUL characters are
ignored, as they cannot be stored in variables.  The result is
not split on the characters in IFS; the intent is that the
variable is assigned exactly the characters read (with the
exceptions of NUL and backslash; see the -r option below).  If
multiple variable names are given, input is only stored in the
first; all other variables will be empty.

And this for the help text:

-N nchars  return only after storing exactly NCHARS characters, unless
   EOF is encountered or read times out, ignoring any NUL or
   delimiter characters

Re: document that read built-in can't return zero-length string in the middle of input

[Greg Wooledge]

On Thu, Jan 11, 2024 at 09:29:03AM -0500, Chet Ramey wrote:
> The read builtin skips over NUL characters because you can't store them
> as part of the value of a shell variable. That seems obvious.

I would argue that it's not obvious at all when using -N.  The help
text for -N says "return only after reading exactly NCHARS characters".
Having it return after reading NCHARS + 1 characters, just because one
of them was NUL, is a surprise.

> What would you like to see documented? That NUL characters don't count
> towards the number of characters read to satisfy -N?

That would work for me, though I can't speak for the OP.

Re: mapfile weirdness ^@ is added it would seem if output is quoted

[Greg Wooledge]

On Mon, Jan 01, 2024 at 04:38:20PM -0500, Federer Fanatic wrote:
> Consider the following:
> 
> shopt -s extdebug
> mapfile -d" " < <(declare -F  Results in the array MAPFILE having 3 entries with '2' entry containing the
> filename
> of bash function initial defining line

I don't think this is correct.  You're showing the command "declare -F < file"
but declare -F doesn't read stdin at all.

unicorn:~$ declare -F
declare -f crawl
declare -f fput
declare -f git
[...]
unicorn:~$ echo 'f() { echo f; }' > x
unicorn:~$ mapfile -d" " < <(declare -F  $ vim "${MAPFILE[2}}" causes failure, not just for vim but cat for
> example, however

*What* failure?

> $ vim ${MAPFILE[2]} does not add '^@' at the end of the fle name,

Without seeing what's actually in your MAPFILE array, it's hard to
understand what you're saying here.  ^@ is how vim displays a NUL byte.
Are you simply trying to edit a binary file?  That would very likely
give you some instances of ^@ on your screen, one for each NUL byte
in the currently-visible part of the file.

Also, your bug report doesn't include which version of bash you're
running.

Please tell us:

 * Which version of bash.
 * What's in the file you're using.
 * The *exact* command you're running, including the filename.
 * The actual output you get.

Please be sure to include "declare -p MAPFILE" to show the contents of
the array.

Re: Multi-line PS1 color disappearance problem

[Greg Wooledge]

On Fri, Dec 22, 2023 at 07:25:30AM +, email--- via Bug reports for the GNU 
Bourne Again SHell wrote:
> On 2023-12-21 17:32, Alex Ratchev wrote:
> 
> > do u really PS1=\[\033 etc ?
> > 
> > try PS1='\[\e...'
> > 
> > in ' quotes
> 
> I forgot quotes from the email. But I did not know that '\e' is valid
> ANSI escape sequence prefix.

It's not ANSI.  It's bash.  Specifically, bash allows \e as a synonym
for \033 when expanding PS1, and inside $'...' quotes, and in readline
key sequence definitions, and in arguments to echo with the -e option.
It also allows \e in the printf format argument, though I don't see
this mentioned in the documentation.

Re: Unexpected Quick Substitution in string literals

[Greg Wooledge]

On Thu, Dec 14, 2023 at 09:41:17AM +0530, Sundeep Agarwal wrote:
> Thanks for the correction on my second example. I had assumed ^ wasn't
> special inside double quotes since the documentation mentions only the !
> character for history expansion (
> https://www.gnu.org/software/bash/manual/bash.html#Double-Quotes).
> 
> However, no character should be treated specially inside single quotes,
> right?
> 
> $ echo 'fig
> ^mango'
> fig
> !!:s^mango

This one also depends on history expansion being active.

unicorn:~$ bash-5.2
unicorn:~$ set -H
unicorn:~$ echo 'fig
> ^mango'
fig
!!:s^mango
unicorn:~$ set +H
unicorn:~$ echo 'fig
> ^mango'
fig
^mango

Is that the correct behavior?  I have no idea.  I don't understand the
rationale behind this piece of histexpand *at all*, so I don't know
whether it's supposed to trigger inside single quotes.  Or even what
it's *doing*.

I turned off histexpand decades ago, and haven't missed it.  But that's
just me.

Re: funsub questions

[Greg Wooledge]

On Thu, Dec 14, 2023 at 04:44:07AM +, Kerin Millar wrote:
> On Wed, 13 Dec 2023 23:16:11 -0500
> Zachary Santer  wrote:
> 
> > On Wed, Dec 13, 2023 at 11:06 PM Greg Wooledge  wrote:
> > > Is that on a system that lacks a process manager?  Something like
> > > "systemctl reload ssh" or "service ssh reload" would be preferred from
> > > a system admin POV, on systems that have process managers.
> > I am not super knowledgeable in this kind of stuff, but would that not
> > cause you to lose your SSH connection?
> 
> It would not. Nor would even a restart, owing to the way privilege separation 
> is implemented in sshd(8).

To expand on this: in most process managers used on Linux systems, the
"reload" action sends a SIGHUP or equivalent, if the service supports
it.  The "restart" action stops and then starts the service.

sshd is a special case, with each active connection being managed by a
child of the main daemon.  Stopping and restarting the main daemon does
not cause the existing children to be terminated.  This was true even
before privilege separation was added.

Re: funsub questions

[Greg Wooledge]

On Wed, Dec 13, 2023 at 10:48:59PM -0500, Zachary Santer wrote:
> On Wed, Dec 13, 2023 at 9:17 PM Greg Wooledge  wrote:
> > If you'd like to read the contents of a file into a variable, the
> > "read" and "readarray" (aka "mapfile") builtins are usually better
> > choices anyway.  $(< file) would only be useful if you want the entire
> > content in a single string variable, which is a questionable idea in
> > most programs.
> sudo kill -s SIGHUP "$(< /var/run/sshd.pid)"
> The one thing I've done with this that wasn't a bad idea.

Is that on a system that lacks a process manager?  Something like
"systemctl reload ssh" or "service ssh reload" would be preferred from
a system admin POV, on systems that have process managers.

And before bash 5.2,

read -r pid < /var/run/sshd.pid && sudo kill -s SIGHUP "$pid"

would be more efficient, on systems where no process manager is in use.

Re: funsub questions

[Greg Wooledge]

On Thu, Dec 14, 2023 at 02:39:04AM +, Kerin Millar wrote:
> On Wed, 13 Dec 2023 21:17:05 -0500
> Greg Wooledge  wrote:
> 
> > On Wed, Dec 13, 2023 at 08:50:48PM -0500, Zachary Santer wrote:
> > > Would there be a purpose in implementing ${< *file*; } to be the 
> > > equivalent
> > > of $(< *file* )? Does $(< *file* ) itself actually fork a subshell?
> > 
> > $(< file) does indeed fork.  The only difference between $(< file) and
> > $(cat file) is the latter also does an exec (but it's portable).
> 
> This stopped being the case with the release of 5.2 or thereabouts.

unicorn:~$ strace -e clone bash-5.1 -c ': $(< /etc/profile)'
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, 
child_tidptr=0x7f274f5cba10) = 105452
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=105452, si_uid=1000, 
si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

unicorn:~$ strace -e clone bash-5.2 -c ': $(< /etc/profile)'
+++ exited with 0 +++

Huh... so it seems.  OK then.

Re: funsub questions

[Greg Wooledge]

On Wed, Dec 13, 2023 at 08:50:48PM -0500, Zachary Santer wrote:
> Would there be a purpose in implementing ${< *file*; } to be the equivalent
> of $(< *file* )? Does $(< *file* ) itself actually fork a subshell?

$(< file) does indeed fork.  The only difference between $(< file) and
$(cat file) is the latter also does an exec (but it's portable).

If you'd like to read the contents of a file into a variable, the
"read" and "readarray" (aka "mapfile") builtins are usually better
choices anyway.  $(< file) would only be useful if you want the entire
content in a single string variable, which is a questionable idea in
most programs.

Re: Unexpected Quick Substitution in string literals

[Greg Wooledge]

On Wed, Dec 13, 2023 at 10:50:16AM +0530, Sundeep Agarwal wrote:
> $ echo "fig
> ^mango"
> bash: :s^mango": substitution failed

I can confirm this happens in every version of bash, at least back to
bash-2.05b which is as far as I can go, but only when history expansion
is enabled (set -H or set -o histexpand).

I think this is intended behavior, although I'm someone who routinely
disables histexpand, so I'm not as familiar with all of its features as
others might be.

Re: Add example of bind readline-command-line

[Greg Wooledge]

On Tue, Dec 12, 2023 at 10:06:49PM +0800, Dan Jacobson wrote:
> bash man page says
>   -v Display  readline variable names and values in such a way
>  that they can be re-read.
> Perhaps add an example of rereading via the bind command:

The man page also includes

   bind [-m keymap] [-lpsvPSVX]
   bind [-m keymap] [-q function] [-u function] [-r keyseq]
   bind [-m keymap] -f filename
   bind [-m keymap] -x keyseq:shell-command
   bind [-m keymap] keyseq:function-name
   bind [-m keymap] keyseq:readline-command

Given that the output of "bind -v" is several lines of text, I would try
the "bind -f filename" command first, to re-read such output.

Re: $((expr)) allows the hexadecimal constant "0x"

[Greg Wooledge]

On Tue, Dec 12, 2023 at 12:10:02AM +0900, Koichi Murase wrote:
> https://lists.gnu.org/archive/html/bug-bash/2019-06/threads.html#00039

Four years later,  i=$(( ${i%%[!+-]*}10#${i#[-+]} ))  is still
horrible.  Well, to be fair, five years since the original suggestion
.

No offense to Pierre; this really is the best we've got.  It's just sad
that the best answer looks like *this*.

Re: bash encountered a coredump issue with stepping on memory

[Greg Wooledge]

On Wed, Dec 06, 2023 at 05:28:19PM +0800, wang yuhang via Bug reports for the 
GNU Bourne Again SHell wrote:
> When the for loop reaches env, the values of each env are: 
>  
[...]
> (gdb) p env[16] 
> $21 = 0x7ffce3c2e25a "DIRNAME_ALIAS" 
> (gdb) p env[17] 
> $22 = 0x7ffce3c2e279 "PID="

The missing '=' in DIRNAME_ALIAS appears to be noteworthy.

I could imagine a library function scanning through these strings, and
for each one, iterating from the start and looking for '='.  In the
case of DIRNAME_ALIAS, it never finds one, so it marches off past the
edge of the allocated memory chunk.

This is pure speculation, of course, until someone actually tests it.

Re: Missing documentation "-bash_input"

[Greg Wooledge]

On Wed, Nov 29, 2023 at 12:37:55AM +, Klaus Frank wrote:
> One thing though, I probably should already know that, but why is a $0
> needed even though a command was already specified? Shouldn't the command
> itself be $0?

It's simply how sh -c has always worked.  The first argument after -c
is the script to be executed.  The second argument after -c becomes $0
of the script, the third becomes $1, and so on.

If you want to pass positional parameters to the -c script, therefore,
you must put *something* in the slot between the script and the parameters.
That something becomes $0.  Common choices are sh, bash, _ and x but you
can use pretty much any string.

Re: Missing documentation "-bash_input"

[Greg Wooledge]

On Tue, Nov 28, 2023 at 10:33:20PM +, Klaus Frank wrote:
> sorry, but this is not true, I can clearly see that it exists. It may be an
> distro addition though. Is it specific to ArchLinux?

Here's what I get on Debian:

unicorn:~$ bash -bash_input foobar -c 'read; declare -p REPLY'
bash: -_: invalid option
Usage:  bash [GNU long option] [option] ...
bash [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
[...]

I get the same thing if I use a bash that I compiled from upstream source.

If I try --bash_input I get this:

unicorn:~$ bash --bash_input foobar -c 'read; declare -p REPLY'
bash: --bash_input: invalid option
Usage:  bash [GNU long option] [option] ...
bash [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
[...]

> Bash source code on GitHub mirror:
> https://github.com/bminor/bash/blob/master/y.tab.c#L3967
> 
> "(--bash_input.location.string) = c;"

That's the pre-decrement operator in C.  Here's the full context for
that snippet:

static int
yy_string_unget (c)
 int c;
{
  *(--bash_input.location.string) = c;
  return (c);
}

It's decreasing the value of bash_input by 1, before accessing it.
It has nothing to do with the presence or absence of "--bash_input" or
"-bash_input" as a command line argument.

Re: Command hangs when using process substitution

[Greg Wooledge]

On Sat, Nov 18, 2023 at 08:36:06AM -0500, dbarrett--- via Bug reports for the 
GNU Bourne Again SHell wrote:
> echo foo | tee >(xclip -i) | tr o x
> 
> The command does print "fxx" but then it hangs.
> 
> The same command behaves correctly when run in zsh.

For the record, here's what processes are running on the terminal while
it's in the "hanging" state:

unicorn:~$ ps f -ft pts/28
UID  PIDPPID  C STIME TTY  STAT   TIME CMD
greg 1082506 1082504  0 09:21 pts/28   Ss 0:00 bash
greg 1082862 1082506  0 09:35 pts/28   S+ 0:00  \_ tr o x
greg 1082864   1  0 09:35 pts/28   S+ 0:00 xclip -i

The tee process has exited, but somehow the tr process has not --
which must mean that tr's stdin is still open.

One additional observation: if I highlight something (e.g. that ps
output), this causes the pipeline to terminate.  I assume the xclip
process somehow notices the highlighting and exits, which causes
tr to exit, because (presumably) it's the orphaned xclip process
whose output was still connected to tr's input.

Without copy/pasting anything, it'll be a bit annoying to diagnose, but
let's try:

unicorn:~$ ps f -ft pts/28
UID  PIDPPID  C STIME TTY  STAT   TIME CMD
greg 1082506 1082504  0 09:21 pts/28   Ss 0:00 bash
greg 1082907 1082506  0 09:39 pts/28   S+ 0:00  \_ tr o x
greg 1082909   1  0 09:39 pts/28   S+ 0:00 xclip -i
unicorn:~$ ls -l /proc/1082907/fd
total 0
lr-x-- 1 greg greg 64 Nov 18 09:39 0 -> 'pipe:[29847034]'
lrwx-- 1 greg greg 64 Nov 18 09:39 1 -> /dev/pts/28
lrwx-- 1 greg greg 64 Nov 18 09:39 2 -> /dev/pts/28
unicorn:~$ ls -l /proc/1082909/fd
total 0
lr-x-- 1 greg greg 64 Nov 18 09:39 0 -> 'pipe:[29848673]'
l-wx-- 1 greg greg 64 Nov 18 09:39 1 -> 'pipe:[29847034]'
lrwx-- 1 greg greg 64 Nov 18 09:39 2 -> /dev/pts/28
lrwx-- 1 greg greg 64 Nov 18 09:39 3 -> 'socket:[29847035]'

In this run, I can confirm that the stdout of xclip is indeed attached
to the stdin of tr, via pipe:[29847034].  Therefore, as a workaround,
I would suggest:

unicorn:~$ echo foo | tee >(xclip -i >/dev/null) | tr o x
fxx
unicorn:~$ 

That should work as desired, in whichever shell you're using that
has process substitutions.

Re: posix command search and execution

[Greg Wooledge]

On Sat, Nov 11, 2023 at 09:14:41AM +0100, Mike Jonkmans wrote:
> On Fri, Nov 10, 2023 at 08:07:31PM -0500, Chet Ramey wrote:
> > On 11/9/23 11:17 AM, Mike Jonkmans wrote:
> > > On Thu, Nov 09, 2023 at 10:12:06PM +0700, Robert Elz wrote:
> > > 
> > > On Ubuntu 22.04 `getconf path' returns /bin:/usr/bin
> > > and these are symlinked.
> > Then that's an issue to take up with Debian/Ubuntu, right? If they're
> > symlinked by the distro, then getconf PATH should only return one of them.
> 
> Yes, that would be more efficient than a shell checking for symlinks in PATH.
> 
> I'll try to convince them.

Good luck with that.  The problem with "usrmerge" in Debian is that
it's still optional.  There are Debian infrastructure systems (virtual
machines especially) that don't use it.  Therefore, simply changing
the static answer in confstr(3) across all Debian systems would be
incorrect.  They would need to make it check the actual /bin to see
whether it's a symbolic link to usr/bin, and give an answer accordingly.

It's a lot less work to just leave it as is.

Re: the portability of seq(1) (was: Idea: jobs(1) -i to print only :%ID:s)

[Greg Wooledge]

On Fri, Nov 10, 2023 at 01:09:29PM -0600, G. Branden Robinson wrote:
> At 2023-11-10T10:54:52-0800, Eric Pruitt wrote:
> > From _seq(1)_ on FreeBSD:
> > 
> > > The seq command first appeared in Version 8 AT UNIX. A seq command
> > > appeared in NetBSD 3.0, and was ported to FreeBSD 9.0. This command
  ^^
> > > was based on the command of the same name in Plan 9 from Bell Labs
> > > and the GNU core utilities. The GNU seq command first appeared in
> > > the 1.13 shell utilities release.

> That leaves NetBSD (data point requested), and I stumbled into seq's
> absence when building groff 1.23.0 release candidates for Solaris 10
> (maybe 11 too).

Apparently present in NetBSD 3.0.  Easily confirmed by going to
 as well.

Re: Idea: jobs(1) -i to print only :%ID:s

[Greg Wooledge]

On Fri, Nov 10, 2023 at 10:54:52AM -0800, Eric Pruitt wrote:
> > A seq command appeared in Version 8 AT UNIX. This version of seq
> > appeared in NetBSD 3.0 and was ported to OpenBSD 7.1.

Ah, I'm a year and a half behind.  OpenBSD 7.1 was released April 2022.

Re: Idea: jobs(1) -i to print only :%ID:s

[Greg Wooledge]

On Fri, Nov 10, 2023 at 06:59:10PM +0100, Steffen Nurpmeso wrote:
> Sequences are also bash-only (though seq(1) is
> everywhere).

It most definitely is *not* everywhere.  It's part of GNU coreutils,
and is generally not present on any system that does't use those (BSDs
and commercial Unixes for example).

Re: Idea: jobs(1) -i to print only :%ID:s

[Greg Wooledge]

On Thu, Nov 09, 2023 at 10:17:35PM +0100, Andreas Schwab wrote:
> On Nov 09 2023, Greg Wooledge wrote:
> 
> > re='^\[([0-9]+)\]'
> > jobspecs=()
> > while IFS= read -r line; do
> > if [[ $line =~ $re ]]; then
> > jobspecs+=( "%${BASH_REMATCH[1]}" )
> > fi
> > done < <(jobs -l)
> 
> That fails for multi-line commands that happen to contain matches for
> re.
> 
> $ (sleep 100; printf $'\n[100]\n') &

Fair enough.  I believe *nothing* would work in that case; the output of
"jobs -l" cannot be safely parsed in its current form.

Adding a NUL-delimited output option might work around that, but if we're
modifying the jobs builtin, we might as well just add the -i or -j option
instead.

Re: Idea: jobs(1) -i to print only :%ID:s

[Greg Wooledge]

On Thu, Nov 09, 2023 at 08:09:23PM +0100, Steffen Nurpmeso wrote:
>   j() {
> local j= a=${AWK:-awk}
> [ $# -gt 0 ] && j='&& $2 !~ /(^| )('$(echo "$@" | tr ' ' '|')')( |$)/'
> j=$(jobs -l | $a -F '[][]' '/^[[]/'"$j"'{print "%" $2}{next}')
> echo $j
>   }

Classic code injection vulnerability.

What are we even parsing?  Start with the input:

unicorn:~$ sleep 5 &
[1] 849028
unicorn:~$ jobs -l
[1]+ 849028 Running sleep 5 &

OK, so you wanted to strip the "1" from "[1]" and turn that into "%1",
yes?  That shouldn't be terribly hard in pure bash.

re='^\[([0-9]+)\]'
jobspecs=()
while IFS= read -r line; do
if [[ $line =~ $re ]]; then
jobspecs+=( "%${BASH_REMATCH[1]}" )
fi
done < <(jobs -l)

Wrap that in a function with local declarations, etc.

Re: nullglob is documented incorrectly

[Greg Wooledge]

On Mon, Nov 06, 2023 at 08:56:11AM -0500, Chet Ramey wrote:
> The null string (NULL) and the empty string ("") are not the same thing.

If this is true, then the man page has many inconsistencies that need
to be cleared up.  For example, in the definition of PATH:

   PATH   The search path for commands.  It is a colon-separated  list  of
  directories  in  which the shell looks for commands (see COMMAND
  EXECUTION below).  A zero-length (null) directory  name  in  the
  value of PATH indicates the current directory.  A null directory
  name may appear as two adjacent colons,  or  as  an  initial  or
  trailing  colon.

Obviously you can't have a NULL pointer in the middle of PATH's content,
when it's serialized as a single string full of colons.  So "null" here
clearly refers to an empty (zero-length) string, at least prior to
whatever internal splitting may occur.

The word "null" is used in many places throughout the man page, and in
most of these, the context seems to say that it means the same as the
empty string.  Parameter Expansion is one such place:

   ${parameter:-word}
  Use Default Values.  If parameter is unset or null,  the  expan‐
  sion  of word is substituted.  Otherwise, the value of parameter
  is substituted.

We know that if parameter contains the empty string, the Default Value
is used.  Since the empty string case is never mentioned explicitly, one
is led to believe that "null" covers that case, possibly in addition to
the case where the variable is/contains a NULL pointer internally.

As far as NULL variables go, I assume this is how it appears:

unicorn:~$ unset -v x; x=""; declare -p x
declare -- x=""
unicorn:~$ unset -v x; declare x; declare -p x
declare -- x

with the first being an empty string, and the second being a NULL pointer.
If that's true, then it wouldn't hurt to spell that out explicitly
somewhere.

Re: Regex: A case where the longest match isn't being found

[Greg Wooledge]

On Thu, Oct 26, 2023 at 10:50:13AM -0700, Dan Bornstein wrote:
> I found a case where the regex evaluator doesn't seem to be finding the 
> longest possible match for a given expression. The expression works as 
> expected on an older version of Bash (3.2.57(1)-release 
> (arm64-apple-darwin22)).

Bash uses the system's (libc) version of regex(3), so the difference
you're seeing is presumably caused by the apple-darwin22 part, rather
than the bash 3.2 part.  (Or conversely, caused by the linux-gnu part
rather than the bash 5.2 part.)

Re: BUG: Colorize background of whitespace

[Greg Wooledge]

On Wed, Oct 25, 2023 at 02:48:20PM +, David wrote:
> On Wed, 25 Oct 2023 at 11:51, Greg Wooledge  wrote:
> > On Wed, Oct 25, 2023 at 10:29:32AM +0200, Holger Klene wrote:
> > > Repeat-By:
> > > run the following command line:
> > > clear; seq 50; printf '\e[36;44m\nsome colored\ttext with\ttabs\e[m\n'
> > > Play with the parameter to seq, to keep the line within the first screen 
> > > or move it offscreen.

> The bug being reported is that the '\t' characters have a black background
> if the 'seq' argument is low enough that its lines 1 and 2 remain
> visible when run.
> But if the 'seq' argument is changed to be bigger, so that (at least)
> lines 1 and 2 both
> scroll off the top of the terminal window so that they are not visible, then 
> the
> '\t' characters then get the expected blue background.

Ahh.  That wasn't clear to me.  Thanks.

In an 80x24 rxvt-unicode or xterm terminal, "seq 22" shows the reported
problem, and "seq 23" does not.

Re: BUG: Colorize background of whitespace

[Greg Wooledge]

On Wed, Oct 25, 2023 at 10:29:32AM +0200, Holger Klene wrote:
> Description:
> The initial bash background is hardcoded to some default (e.g. black) and 
> cannot be colorized by printing "transparent" tabs/newlines with 
> ANSI-ESC-codes.
> Only after a vertical scrollbar appears, the whitespace beyond the window 
> hight will get the proper background color.

Terminals have colors (foreground and background).  Bash does not.  Bash
is just a command shell.

> Repeat-By:
> run the following command line:
> clear; seq 50; printf '\e[36;44m\nsome colored\ttext with\ttabs\e[m\n'
> Play with the parameter to seq, to keep the line within the first screen or 
> move it offscreen.
> 
> Reproduced in:
> - in Konsole on Kubuntu 23.04
> - in the git bash for windows mintty 3.6.1
> - in WSL cmd window on Windows 11

I ran this command in xterm (version 379) and rxvt-unicode (9.30) on
Debian, but I'm not sure what I'm supposed to be seeing.

In my case, the terminals are 80x24, and xterm has a default black
background, while rxvt has a default white background.  In both cases,
the expected rows of numbers are printed by seq, and then there's
a blank line, and then there's a line of dark blue background which
extends all the way across the terminal, containing "some colored" and
"text with" and "tabs" in a greenish foreground color.

Hitting Enter to scroll the text upward doesn't do anything surprising,
and neither does scrolling the terminal with Shift-PageUp or with
the mouse.

What are *you* seeing which surprises you?

In any case, this has nothing to do with bash.  It's strictly a terminal
issue, whatever the issue may be.

Re: bash tries to parse comsub in quoted PE pattern

[Greg Wooledge]

On Wed, Oct 18, 2023 at 09:39:36AM -0400, Zachary Santer wrote:
> I guess I still want to hear about "${#@}" and $[  ].

$[ ] is officially deprecated, and users are advised to stop using it.
It was originally going to be the syntax for arithmetic expansion, and
made it as far as some POSIX rough draft, I think.  But then the
decision was made to go with $(( )) instead.

Meanwhile, bash had already shipped releases with support for $[ ] and
people had already started using it.  So, it's still *there*, and still
works, despite being deprecated for literally decades.  I don't know
whether it'll ever actually be removed -- maybe it'll just linger until
POSIX decides to use $[ ] for something else and forces a change.

${#@} is simply the use of @ as a pseudo array name, which works in many
places, the most common being "$@".  For example,

unicorn:~$ set -- one two three four
unicorn:~$ echo "${@//o/x}"
xne twx three fxur

In this context, @ can be thought of as a shortcut for argv[@] where
argv contains the positional parameters.  ${#@} is therefore analogous
to ${#argv[@]} which counts the number of elements in an array named
argv.

It's not a perfect analogy, though.  For example, this doesn't work:

unicorn:~$ echo "${!@}"
bash: one two three four: invalid variable name

That's due to the syntax collision between ${!ref} and ${!array[@]}.
The parser tries to apply the former rather than the latter.

In practical terms, there is no reason to use ${#@}.  It's the same
as $# except that the latter is shorter and more portable.  But at the
same time, there's no reason to say anything about ${#@}, since nobody
ought to be using it anyway.  And if they do use it, what's the harm?

Re: bash tries to parse comsub in quoted PE pattern

[Greg Wooledge]

On Wed, Oct 18, 2023 at 08:19:35AM -0400, Zachary Santer wrote:
> On Tue, Oct 17, 2023 at 5:56 PM Emanuele Torre 
> wrote:
> 
> > bash-5.1$ letters=( {a..z} ); echo "${letters["{10..15}"]}"
> > k l m n o p
> >
> 
> Then there's the question "Was that even supposed to work like that?"

This is another one of those cases where some end user did something
wacky and it produced a result that they liked, so they kept doing it.
At some point, they convinced themselves that this was an intended
feature, and they'd probably be upset if it stopped "working".

It's a pretty straightforward application of brace expansion, but not
one I would have thought to use.

> If
> so, you'd think it would generalize to being able to pass a series of
> whitespace-delimited indices to an array expansion.

That's not what's happening, at all.

unicorn:~$ set -x
unicorn:~$ echo "a b"{c,d}"e f"
+ echo 'a bce f' 'a bde f'
a bce f a bde f

Brace expansion is extremely low-level text macro expansion.  In my
example above, the thing after the echo consists of a single parser-word
which contains two quoted sections, and a brace expansion.  The brace
expansion fires first, and causes two words to be generated: "a b"c"e f"
is the first, and "a b"d"e f" is the second.

After those words are generated, quote removal occurs, and the result
is what you see in the -x output.

> In Bash 5.2:
> $ array=( zero one two three four five six )
> $ printf '%s\n' "${array["{2..6..2}"]}"
> two
> four
> six

This one "works" because the final parser-word is a brace expansion with
quoted sections before and after it.  The brace expansion causes three
words to be generated: "${array["2"]}" is the first, and so on.

You're basically typing

printf '%s\n' "${array["2"]}" "${array["4"]}" "${array["6"]}"

or rather, you're letting the brace expansion type it for you.

It just so happens that "${array["2"]}" is accepted as an indexed array
element expansion, despite the extra quotes inside the square brackets.

> $ printf '%s\n' "${array[{2..6..2}]}"
> -bash: {2..6..2}: syntax error: operand expected (error token is
> "{2..6..2}")

Here, what would have been a valid brace expansion is inside quotes, so
it's not expanded.  "{2..6..2}" is not a valid indexed array index value.

> $ printf '%s\n' "${array["2 4 6"]}"
> -bash: 2 4 6: syntax error in expression (error token is "4 6")

"2 4 6" is not a valid indexed array index value either.

> $ printf '%s\n' "${array[2 4 6]}"
> -bash: 2 4 6: syntax error in expression (error token is "4 6")

Same here, just without the extra quotes.

> $ printf '%s\n' "${array[2,4,6]}"
> six

In this case, "2,4,6" *is* a valid indexed array index.  It's an
arithmetic expression, whose value is that of the thing after the
last comma.

A comma series in an arithmetic expression is intended to be used in
cases like:

unicorn:~$ let 'a=1,b=a+1'; declare -p a b
declare -- a="1"
declare -- b="2"

The evaluation of the comma series as the last element is mostly an
afterthought.  "What else would it evaluate to?"  We're more interested
in the side effects, rather than the final value.

This comes from C, where the most common use is something like:

for (i=1,j=2; i<10; i++,j+=2) {...}

The comma series lets you perform two assignments/alterations in a
place where the syntax only asks for one.

> $ indices=(2 4 6)
> $ printf '%s\n' "${array[${indices[@]}]}"
> -bash: 2 4 6: syntax error in expression (error token is "4 6")
> $ printf '%s\n' "${array[${indices[*]}]}"
> -bash: 2 4 6: syntax error in expression (error token is "4 6")

In these cases, the inner expression is evaluated, yielding an
invalid indexed array index value.  "2 4 6" isn't allowed as an index,
no matter where you pull it from.

> Considering I don't think this is documented anywhere, and what's in
> between the square brackets gets an arithmetic expansion applied to it, I'm
> going to guess "no."
> 
> So how important is it to maintain undocumented behavior?

There is one analogous case where the behavior *did* change.  Consider
a command like this:

unicorn:~$ echo <(printf '%s\n' {a..c})
/dev/fd/63

In bash 5.2, we get the result shown above.  The brace expansion happens
inside the process substitution.  It's as if we had typed

echo <(printf '%s\n' a b c)

In bash 3.2, however:

unicorn:~$ bash-3.2
unicorn:~$ echo <(printf '%s\n' {a..c})
/dev/fd/63 /dev/fd/62 /dev/fd/61

Here, the brace expansion happened *first*, and we got three separate
process substitution words out of it.  It's as if we had typed

echo <(printf '%s\n' a) <(printf '%s\n' b) <(printf '%s\n' c)

Changing this was significant, and caused some scripts to break, and
some people to become confused.  But it caused *more* people to be less
confused, because the current behavior looks a lot more reasonable to
more people vs. the previous behavior.

So, there's precedent for changing something that seems wrong, when it
comes to brace expansions.  The first question is whether the 

Re: wrong variable name in error message about unbound variable?

[Greg Wooledge]

On Tue, Oct 17, 2023 at 04:46:22AM +0200, Christoph Anton Mitterer wrote:
> But why does it even try to evaluate the subscript "key" as arithmetic
> expression?

Because that's how indexed arrays work.  Everything inside the square
brackets is an arithmetic expression, and in an arithmetic context,
bash treats anything that *can* be parsed as a variable name as a
variable name.

unicorn:~$ unset -v a b c array
unicorn:~$ a=b b=c c=42 array[a]=foo; declare -p array
declare -a array=([42]="foo")

> Yes it's defined for indexed arrays, but shouldn't it already know that
> there is no indexed array of the name "array" and any evaluation of the
> subscript is thus pointless?

There are two undefined variables at that point, and bash chooses one
of them to report.  Prioritizing "the important error" over "the less
important error" is not something that compilers/interpreters are good
at, generally.  If you have multiple errors, you may get a report of
both of them, or just one, which masks the second one.  You'll have
to fix both of them to get the correct result anyway, so it doesn't
really matter which one gets reported and fixed first.

Re: variable set in exec'ing shell cannot be unset by child shell

[Greg Wooledge]

On Sat, Oct 14, 2023 at 12:55:21PM -0400, Ti Strga wrote:
> it's just the "[[ -v
> foo ]]" tests to see where along the cloning process we are.

*Shudder*

I foresee so much more pain in your future.  Seriously, this is going
to blow up in your face at some point.  -v peeks into some incredibly
dark and spooky corners of the shell, and will expose *precisely* how
your assumptions about the shell differ with those of the bash author.
Also, it's been historically buggy.

I'm inclined to agree with Grisha Levit.  This whole thing looks like
a massively out-of-control X-Y problem.  If the *real* goal is to
overwrite a running script with a new version of itself, and then
re-exec it, then the correct solution is to wrap the script in a single
compound command so that it gets read and parsed up front, before
beginning execution of the main loop.  Either wrap the whole thing in
"{" ... "}" as Grisha suggested, or wrap the whole thing in a "main()"
function and then call main "$@".  That way, you can overwrite the file
without sabotaging running instances of the script.

Re: variable set in exec'ing shell cannot be unset by child shell

[Greg Wooledge]

On Fri, Oct 13, 2023 at 02:21:59PM -0400, Dale R. Worley wrote:
> I was too lazy to chew through your example, but I coded an instance of
> your description (above), and it does not show the dubious behavior that
> you report.  Specifically,
> 
> $ bash -version
> GNU bash, version 5.1.0(1)-release (x86_64-redhat-linux-gnu)
> ...
> $ ls -l
> total 20
> -rwxr-xr-x. 1 worley worley 112 Oct 13 14:14 inner
> -rw-r--r--. 1 worley worley 113 Oct 13 14:13 middle
> -rw-r--r--. 1 worley worley 123 Oct 13 14:12 outer

> $ cat inner
> echo "Value in ./inner at the beginning: $OUTSIDE"
> 
> unset OUTSIDE
> 
> echo "Value in ./inner at the end: $OUTSIDE"

I tried to understand the question as well.  I got as far as realizing
there were *three* scripts, not two as originally stated (your example
agrees).  But the OP's third script wraps around and re-invokes the
first one.  It's an ouroboros snake.

Your example doesn't circle around in the same way, so I don't know how
faithfully it's reproducing the original question.

Once I realized there was an infinite loop of scripts eating their own
tails, I gave up trying to understand any further.  Too complicated for
me.

Re: "here document" causing failures. Implementation is inconsistent and not documented.

[Greg Wooledge]

On Wed, Oct 11, 2023 at 11:56:42AM +1100, Jim McD wrote:
> Bug:
> Trailing white space after the delimiting tag cause the here document to
> fail with an error like
> /./: line : warning: here-document at line
>  delimited by end-of-file (wanted `msg_end')Trailing white
> space/
> 
> Trailing white space after the start of the here statement is ignored.

To be fair, whitespace *is* possible within the delimiting word, if
the word is quoted.

unicorn:~$ ebase -host localhost
unicorn:~$ cat <<'EOF  '
> EOF
> EOF  
EOF

In the example above, there are two spaces on the second line which
starts with '> EOF'.  The first line has no trailing whitespace, so
it doesn't match the delimiting word.  The second line has the two
trailing spaces, so it matches.

The shell is acting correctly here; if you feel the documentation could
be improved, that's fair.  But it seems fairly clear to me:

   Here Documents
   This type of redirection instructs the shell to  read  input  from  the
   current source until a line containing only delimiter (with no trailing
   blanks) is seen.

Re: error message lacks useful debugging information

[Greg Wooledge]

On Thu, Oct 05, 2023 at 07:04:26AM +0200, Phi Debian wrote:
> Since we are on the error path (not the perf path) may be the shell could
> go the extra miles and try some more diag, has it does for shebang, on
> ENOENT, the shell could try to open the a.out, if OK try some other
> euristics, [...]

Just for the record, trying to open the program for reading is only
possible for scripts.  It's not too uncommon for a compiled program
to have +x permission for the user, but *not* +r permission.  Such a
program can be executed, but not opened for reading.

This permission combo is primarily used for setuid programs on some
systems, where it's believed that letting the user read the program
might give them insight into finding security exploits in it.  Such
systems are typically not open source, although it wouldn't surprise
me one bit if there are also some BSD/Linux systems with programs
like this.

Another reason some people might use this permission combo is to
"protect their intellectual property", by allowing end users to run
the program, but not to make a copy of it.  This obviously only works
if the owner of the system (and anyone else with root access) is
trusted not to make copies.  So, it's an incredibly niche situation,
but it might still be in use somewhere.

Re: error message lacks useful debugging information

[Greg Wooledge]

On Wed, Oct 04, 2023 at 08:05:41PM -0400, Dave Cigna via Bug reports for the 
GNU Bourne Again SHell wrote:
> Attempting to tun an executable file (not a bash script) with the following
> command:
> 
> ./Candle
> 
> the following error message is reported by bash:
> 
> bash: ./Candle: cannot execute: required file not found
> 
> The executable file 'Candle' does exist in the current directory;
> if it didn't then bash would report a different error.

Ironically, this *is* the improved error message, which is supposed to
be less confusing than the previous message ("No such file or directory").

You're right that bash does not tell you which file is missing.  But
that's because bash doesn't know, either.  All bash knows is that it
asked the kernel to execute this command, and the kernel said
"No such file or directory" (ENOENT), but the command you asked to
run exists as a file.  Therefore, bash gives this message, instead of
"No such file or directory".

Since you claim it's not a script, your next steps would be to run "file"
against it, and if "file" claims it's a compiled executable, try running
"ldd" against it.  You're probably missing whatever loader is required
to run this program.  Loaders are programs like /lib/ld-linux.so which
should be visible in ldd's output, and often in file's output as well.

The inability to execute a compiled program usually means it was
compiled for a different platform/architecture than your system uses.
For example, you might be trying to run an i386 (32-bit Intel) compiled
program on an amd64 (64-bit AMD/Intel) system, without the correct i386
compatibility libraries installed.  Or you might be trying to run
something completely foreign, like an ARM program on an x86 computer.

> download:  candle_1.1.7.tar.gz
> from: https://github.com/Denvi/Candle

Unfortunately, this web site does not say *what* kind of Linux system
that program was built for.

You'll have to use "file" and/or "ldd" to diagnose it.  Or ask the
maintainers, or whatever support channels are available.

Re: bug:

[Greg Wooledge]

On Tue, Oct 03, 2023 at 12:54:45PM +, queency jones via Bug reports for the 
GNU Bourne Again SHell wrote:
> BASH_VERSION='5.1.4(1)-release'PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"no 
> compile needed
> i made a fifo file by typing: mkfifo gdbout.fifo
> i coded a resource file named  : command.bashthe file code is :aa(){ gdb 
> ./hello_wordl.out > gdbout.fifo & }
> i loaded the resource file by typing in bash shell:       .com.bash
> then run the function aa
> when i typed: ps i got 2 bash jobs instead of 1 bash and 1 gdb !

Your function runs a background subshell which opens a named pipe before
executing gdb.  Opening a named pipe will block, until the named pipe is
opened a *second* time in the opposite direction.  (Since your function
opens it for writing, you need a second process to open it for reading.)

Once the named pipe is opened a second time, the background subshell
will be able to continue, to execute gdb.  Until then, however, it's
still a bash process.

Re: Prompt messed up if PS1 contains ANSI escape sequences

[Greg Wooledge]

On Thu, Sep 07, 2023 at 04:03:39PM +0200, Gioele Barabucci wrote:
> The following snippet shows that, even with the final \], Bash produces the
> same erroneous output and miscalculates the cursor position (it just needs a
> longer prompt):
> 
> $ long_name="$(printf 'abcdef0123456789/%.0s' {0..20})"
> $ mkdir -p /tmp/$long_name
> $ cd /tmp/$long_name
> $ PS1=$'\n\[\e[1m\]\w\[\e[m\] \$ '
> 
> Now press the up arrow, then the down arrow)

I did this in bash 5.2.15 (Debian 12 package version 5.2.15-2+b2) in
rxvt-unicode (Debian 12 package version 9.30-2+b4), and I can't see
whatever problem you're reporting.  Typing a long command (one which
goes past the 80th column and scrolls the terminal's text upward) and
then backspacing all of it acts as expected.  As does pressing up arrow
until a sufficiently long command is recalled from history.

Have you tried testing with several different terminal emulators?  Be
sure to include xterm among them, as that one tends to be extremely
well-behaved in situations where the big fancy ones are not.  I'm a fan
of rxvt or rxvt-unicode, which have both been kind to me, but good old
xterm should always be considered.

Re: Prompt messed up if PS1 contains ANSI escape sequences

[Greg Wooledge]

On Thu, Sep 07, 2023 at 03:46:23PM +0200, Gioele Barabucci wrote:
> On 07/09/23 15:00, alex xmb ratchev wrote:
> > u have to \[ esc-seq \]
> > eg inside \[ and \]
> > 
> > PS1=$'\u\[\e[1m\]\h\[\e[0m- '
> > 
> > should display hostname bold
> 
> Thanks for the suggestion, but adding \] does not really fix the problem, it
> just masks it in many cases (better than nothing).

The \[ \] wrappings are required.  They're not "masking" the problem.
Your prompt is literally set incorrectly without them.

>From the man page:

PROMPTING
[...]
  \[ begin  a sequence of non-printing characters, which could
 be used to embed a terminal  control  sequence  into  the
 prompt
  \] end a sequence of non-printing characters

See also  for more details.

Re: [PATCH] printf: add %#s alias to %b

[Greg Wooledge]

On Wed, Sep 06, 2023 at 02:13:44AM +, William Bader wrote:
> Has bash ever had a change before that would break valid scripts?

Yes.  Many times, sadly.  Most recently, bash 5.2 introduced some changes
to how associative array indexes are handled, which will break
backward compatibility in several ways.

The changing of how "set -e" is handled (several times, mostly driven
by POSIX) has also caused scripts to work differently.

There are lots more examples, but those are the ones I think of first.

> Could the printf format change be settable by a variable or by an option like 
> the -e/-E in echo?
> 
> Is it necessary for bash printf to match C printf?
> 
> I suppose that it is already decided.

Chet has answered these.

> Does shellcheck track variables well enough to warn if the new %b is passed 
> something that doesn't look like a numeric string?

There isn't a "new %b", so this question is theoretical.  Shellcheck is
an independent project, and if a new printf format were to be introduced,
you could talk to its developer about getting the relevant changes into
shellcheck in a timely fashion -- again, theoretically.

Re: Warn upon "declare -ax"

[Greg Wooledge]

On Tue, Sep 05, 2023 at 05:09:52PM +0200, Andreas Kähäri wrote:
> On Tue, Sep 05, 2023 at 04:04:50PM +0200, alex xmb ratchev wrote:
> > On Mon, Sep 4, 2023, 15:19 Kerin Millar  wrote:
> > > Curiously, ARRAY_EXPORT can be defined in config-top.h. It's probably safe
> > > to say that nobody uses it (nor should anybody wish to upon realising how
> > > it works).
> > 
> > does it make too big copies or wha ..
> 
> On Unix systems, environment variables are plain key-value pairs of
> strings.  An array in bash is not a key-value pair of strings, but
> a list of strings (and corresponding indexes).  The ARRAY_EXPORT
> configuration option (which I believe isn't settable from the
> "configure" script, and which I've never tried enabling) presumably
> makes arrays exportable somehow (probably only to other bash shell
> instances, similar to how "exportable functions" work).

For the record, here's the snippet from config-top.h:

/* Define to 1 if you want to be able to export indexed arrays to processes
   using the foo=([0]=one [1]=two) and so on */
/* #define ARRAY_EXPORT 1 */

I've never tried it, so I don't know how it plays out in practice.  It
does make me think of bash's exportable functions, and the shellshock
vulnerability which was spotlighted a while back, but I don't have any
idea whether this would introduce any security vulnerabilities.

I wonder what specific issues Kerin is thinking about.

Re: set -x vs. n=($@)

[Greg Wooledge]

On Sun, Sep 03, 2023 at 08:39:25PM +0200, alex xmb ratchev wrote:
> On Sun, Sep 3, 2023, 12:08 Dan Jacobson  wrote:
> 
> > It's not fair:
> > set -x a b c
> >
> 
> you may want -v

*Sigh* No.  set -v shows lines as they are being READ by the shell.
set -x shows commands as they are being EXECUTED.

set -v is almost never useful.  It's especially unlikely to be useful
in any script that uses functions or loops, as it only shows the function
or the loop being READ, not being EXECUTED.

Re: bug attached

[Greg Wooledge]

On Thu, Aug 31, 2023 at 10:38:55PM +0300, queency3 jones wrote:
> > > function sub { aa=8;return_value=$aa; }
> > > function sub { aa=8; }
> > >
> > > function main { aa=3;sub;aa=$(($aa+1));printf "$aa\n"; }

> i don't think that main is significant when it declared in "Bash" but even
> though,
> if you change the function main , to function ain  the $aa should be local
> , but it ain't !!

Bash uses dynamic variable scope.  If a function needs to retrieve
the value from a variable, it looks at its own local variables first,
then at its caller's local variables, then it's caller's caller's
variables, and so on up the stack.

Since main calls sub, "aa=8" in sub look for a local variable in sub.
There isn't one, so it looks for a local variable in main.  And if
there isn't one in main, then it looks in main's caller, which happens
to be the global scope.

If you add "local aa" inside main, this doesn't change anything inside
main or sub.  sub will find the local variable aa inside main, and it
will change that one, instead of the global one.  The difference here
is that the modified aa is no longer visible at the global scope.  It
"disappears" when main is finished.

If you add "local aa" inside sub, this *does* change the result.  Now,
aa=8 is executed upon the copy of aa that's local to sub, rather than
the one that's local to main.  So, main's copy remains set to 3.

If you want *every* copy of aa to be treated as a local variable within
its own function, you must declare it as local inside every function
where it's used.  That's the way bash works: every variable is non-local
by default.

Re: Some incorrect behaviour for BASH arrays

[Greg Wooledge]

On Thu, Aug 31, 2023 at 04:13:09PM +0100, Kerin Millar wrote:
> This script is a mess because its functions begin by defining RESULT as an 
> ordinary, non-array variable (if not yet declared).
> 
> $ RESULT='string'
> $ declare -p RESULT # not yet an array variable
> declare -- RESULT='string'
> 
> It then proceeds to operate on the variable in such a way that it will be 
> transformed to an array.
> 
> $ RESULT[1]='second element'
> $ declare -p RESULT
> declare -a RESULT=([0]="string" [1]="second element")
> 
> Now, would RESULT='' empty this array? No, it would not.
> 
> $ RESULT='' # no different from RESULT[0]=''
> $ declare -p RESULT
> declare -a RESULT=([0]="" [1]="second element")
> 
> A correct way to initialise an empty array variable is RESULT=().
> 
> $ RESULT=()
> $ declare -p RESULT # now an empty array
> declare -a RESULT=()
> 
> You might also consider using the "local" builtin to declare the variable 
> with a function-local scope.

All of this is excellent advice.  If you'd like a tutorial that explains
more about bash arrays, I recommend:

https://mywiki.wooledge.org/BashFAQ/005

Re: using exec to close a fd in a var crashes bash

[Greg Wooledge]

On Wed, Aug 23, 2023 at 09:50:36AM -0400, Zachary Santer wrote:
> From the NEWS file [1]:
> 
> o. The new `varredir_close' shell option causes bash to automatically close
>file descriptors opened with {var}redirection unless they're arguments to the `exec' builtin.

Hmm, interesting.

> $ shopt -s varredir_close
> $ printf 'words\n' {fd}>&1 1>&2 2>&$fd {fd}>&-
> words
> $ ls /dev/fd
> 0  1  2  3

Of course, you can drop the {fd}>&- there, because it does nothing.  With
or without varredir_close, either way, it does nothing.

So... {var}> redirections were introduced in bash 4.1, and varredir_close
in bash 5.2.  That means that in all versions from bash 4.1 to 5.1, you
will need the separate "exec {fd}>&-" to close the temp FD.  At this
point, it hardly seems worthwhile to make use of a feature that only
works in bash 5.2, which is surely a tiny fraction of the set of installed
bash instances in the world.

Perhaps in a decade or so, when bash 5.2+ instances are the majority,
it will make sense to expect that feature to exist.  But you'd still
need fallback code for the rest.

Then again... leaving an FD open in a shell script usually won't matter,
because the script will exit, and that'll just take care of it.  The
only times it actually matters are long-running bash processes -- either
interactive shells, or some kind of weird daemon that's been implemented
in bash for some reason -- or scripts that open and (fail to) close lots
of temp FDs in a loop.

So, unless you're using this feature in an interactive shell function
or programmable completion or something, I guess it can mostly be ignored.

Re: using exec to close a fd in a var crashes bash

[Greg Wooledge]

On Wed, Aug 23, 2023 at 12:05:42PM +1000, Martin D Kealey wrote:
> On Wed, 23 Aug 2023, 05:29 Greg Wooledge,  wrote:
> 
> > Excuse me now, while I go and close several open FDs in the shell where
> > I tested something the other day, which I had no idea were left open.
> >
> 
> It's even worse than that; try:
> 
> echo Hi {X}>/dev/null
> ls -l "/proc/$$/fd/$X"

That's a really strange example.  I wonder what would lead someone to
write that particular redirection.  What's the intended outcome, in
terms of the variable X and the state of file descriptors?  If you
simply want to discard output, there's no reason I can think of to
use {X}>/dev/null instead of simply >/dev/null.

The thing I was testing was in response to this question, which was
posed as an exercise:

Swap stderr and stdout of a shell command.

I gave two answers:

cmd 3>&1 1>&2 2>&3 3>&-

and

cmd {fd}>&1 1>&2 2>&$fd {fd}>&-

As it turns out, the second one does *not* close the temporary file
descriptor, so each time I ran the function while testing, an extra
file descriptor was left open.

But this variant *does* close it:

cmd {fd}>&1 1>&2 2>&$fd
exec {fd}>&-

I find this incredibly confusing.

Re: using exec to close a fd in a var crashes bash

[Greg Wooledge]

On Tue, Aug 22, 2023 at 02:59:14PM -0400, Dale R. Worley wrote:
> The "{var}>..." mechanism *assigns* to $var, rather than
> taking its existing value.

... oh.  Well, today I learned something.

Excuse me now, while I go and close several open FDs in the shell where
I tested something the other day, which I had no idea were left open.

Re: using exec to close a fd in a var crashes bash

[Greg Wooledge]

On Sat, Aug 19, 2023 at 01:37:31PM -0400, jleivent wrote:
> Repeat-By:
> 
>   exec {foo}>/tmp/foo
>   exec "${foo}"<&-

Remove the quotes and the $ and it should work.

exec {foo}<&-

The way you've got it is essentially:

exec "$foo" <&-

This does't make bash "crash".  It's simply exec-ing a nonexistent
program.

Re: Assignment to RO variable

[Greg Wooledge]

On Tue, Aug 15, 2023 at 11:24:31PM -0500, Dennis Williamson wrote:
> Your use of colon as a comment character is confusing.

They're running with set -x, so presumably they used those : commands
to mark the set -x output with various labels.  Which seems nominally
clever, except they didn't *show* us the output.  Also, the example
is way more complicated than it needs to be, to the point where I
gave up trying to suss out what part of it they thought was behaving
incorrectly.

Compare with this example:


unicorn:~$ cat foo
#!/bin/bash

f() {
local x=in_function_f
echo "still in f"
}

readonly x=global
f
unicorn:~$ ./foo
./foo: line 4: local: x: readonly variable
still in f


As you stated, a variable that has been declared readonly at the global
scope is also considered readonly at every other scope.  This is the
intended behavior.

After the failed local variable assignment, the function continues
execution.  This is also the intended behavior, as this example does
not use any of the set -e variants.