Re: Examples of concurrent coproc usage?
On 4/4/24 7:23 PM, Martin D Kealey wrote:
I'm somewhat uneasy about having coprocs inaccessible to each other. I
can foresee reasonable cases where I'd want a coproc to utilize one or
more other coprocs.
In particular, I can see cases where a coproc is written to by one
process, and read from by another.
Can we at least have the auto-close behaviour be made optional, so that
it can be turned off when we want to do something more sophisticated?
With support for multiple coprocs, auto-closing the fds to other coprocs
when creating new ones is important in order to avoid deadlocks.
But if you're willing to take on management of those coproc fds yourself,
you can expose them to new coprocs by making your own copies with exec
redirections.
But this only "kind of" works, because for some reason bash seems to close
all pipe fds for external commands in coprocs, even the ones that the user
explicitly copies with exec redirections.
(More on that in a bit.)
On Mon, 8 Apr 2024, Chet Ramey wrote:
On 4/4/24 7:23 PM, Martin D Kealey wrote:
I'm somewhat uneasy about having coprocs inaccessible to each other. I
can foresee reasonable cases where I'd want a coproc to utilize one or
more other coprocs.
That's not the intended purpose,
Just a bit of levity here - i can picture Doc from back to the future
exclaiming, "Marty, it's perfect! You're just not thinking 4th
dimensionally!"
so I don't think not fixing a bug to accommodate some future
hypothetical use case is a good idea. That's why there's a warning
message when you try to use more than one coproc -- the shell doesn't
keep track of more than one.
If you want two processes to communicate (really three), you might want
to build with the multiple coproc support and use the shell as the
arbiter.
For what it's worth, my experience is that coprocesses in bash (rigged up
by means other than the coproc keyword) become very fun and interesting
when you allow for the possibility of communication between coprocesses.
(Most of my use cases for coprocesses fall under this category, actually.)
The most basic commands for tying multiple coprocesses together are tee(1)
and paste(1), for writing to or reading from multiple coprocesses at once.
You can do this already with process substitutions like
tee >(cmd1) >(cmd2)
paste <(cmd3) <(cmd4)
My claim here is that there are uses for this where these commands are all
separate coprocesses; that is, you'd want to read the output from cmd1 and
cmd2 separately, and provide input for cmd3 and cmd4 separately.
(I'll try to send some examples in a later email.)
Nevertheless it's still crucial to keep the shell's existing coprocess fds
out of new coprocesses, otherwise you easily run yourself into deadlock.
Now, if you built bash with multiple coproc support, I would have expected
you could still rig this up, by doing the redirection work explicitly
yourself. Something like this:
coproc UP { stdbuf -oL tr a-z A-Z; }
coproc DOWN { stdbuf -oL tr A-Z a-z; }
# make user-managed backup copies of coproc fds
exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}
coproc THREEWAY { tee /dev/fd/$up_w /dev/fd/$down_w; }
But the above doesn't actually work, as it seems that the coproc shell
(THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the
user-managed ones explicitly copied with exec.
As a result, you get back errors like this:
tee: /dev/fd/11: No such file or directory
tee: /dev/fd/13: No such file or directory
That's the case even if you do something more explicit like:
coproc UP_AND_OUT { tee /dev/fd/99 99>&$up_w; }
the '99>&$up_w' redirection succeeds, showing that the coproc does have
access to its backup fd $up_w (*), but apparently the shell closes fd 99
(as well as $up_w) before exec'ing the tee command.
Note the coproc shell only does this with pipes; it leaves other user
managed fds like files or directories alone.
I have no idea why that's the case, and i wonder whether it's intentional
or an oversight.
But anyway, i imagine that if one wants to use multi coproc support (which
requires automatically closing the shell's coproc fds for new coprocs),
and wants to set up multiple coprocs to communicate amongst themselves,
then the way to go would be explicit redirections.
(But again, this requires fixing this peculiar behavior where the coproc
shell closes even the user managed copies of pipe fds before exec'ing
external commands.)
(*) to prove that the coproc shell does have access to $up_w, we can make
a shell-only replacement for tee(1) : (actually works)
fdtee () {
local line fd
while read -r line; do
for fd; do
printf '%s\n' "$line" >&$fd;
done;
done;
}
coproc UP { stdbuf -oL tr a-z A-Z; }
Re: strtoimax test still broken
On 4/9/24 10:57 AM, Dag-Erling Smørgrav wrote: Chet Ramey writes: The fixed version of strtoimax.m4 has been in the devel branch since October 3, 2022. Since bash-5.2 built fine on FreeBSD 13 before I released it, I assume the link problem exist only on systems using musl. des@crash13 ~/src/bash-5.2% uname -r 13.3-STABLE des@crash13 ~/src/bash-5.2% ./configure --enable-static-link Ah, I test with the default build options but I see what you mean. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: strtoimax test still broken
Chet Ramey writes: > The fixed version of strtoimax.m4 has been in the devel branch since > October 3, 2022. Since bash-5.2 built fine on FreeBSD 13 before I released > it, I assume the link problem exist only on systems using musl. des@crash13 ~/src/bash-5.2% uname -r 13.3-STABLE des@crash13 ~/src/bash-5.2% ./configure --enable-static-link && make [...] cc -L./builtins -L./lib/readline -L./lib/readline -L./lib/glob -L./lib/tilde -L./lib/malloc -L./lib/sh -static -g -O2 -static -o bash shell.o eval.o y.tab.o general.o make_cmd.o print_cmd.o dispose_cmd.o execute_cmd.o variables.o copy_cmd.o error.o expr.o flags.o jobs.o subst.o hashcmd.o hashlib.o mailcheck.o trap.o input.o unwind_prot.o pathexp.o sig.o test.o version.o alias.o array.o arrayfunc.o assoc.o braces.o bracecomp.o bashhist.o bashline.o list.o stringlib.o locale.o findcmd.o redir.o pcomplete.o pcomplib.o syntax.o xmalloc.o -lbuiltins -lglob -lsh -lreadline -lhistory -ltermcap -ltilde -lmalloc lib/intl/libintl.a ld: error: duplicate symbol: strtoimax >>> defined at strtoimax.c:67 >>>strtoimax.o:(strtoimax) in archive ./lib/sh/libsh.a >>> defined at xlocale_private.h:201 >>> (/usr/src/lib/libc/locale/xlocale_private.h:201) >>>strtoimax.o:(.text+0x230) in archive /usr/lib/libc.a cc: error: linker command failed with exit code 1 (use -v to see invocation) *** Error code 1 Stop. make: stopped in /home/des/src/bash-5.2 DES -- Dag-Erling Smørgrav - d...@des.no
Re: Examples of concurrent coproc usage?
On Mon, Apr 8, 2024 at 3:50 PM Chet Ramey wrote:
>
> On 4/4/24 7:23 PM, Martin D Kealey wrote:
> > I'm somewhat uneasy about having coprocs inaccessible to each other.
> > I can foresee reasonable cases where I'd want a coproc to utilize one or
> > more other coprocs.
>
> That's not the intended purpose, so I don't think not fixing a bug to
> accommodate some future hypothetical use case is a good idea. That's
> why there's a warning message when you try to use more than one coproc --
> the shell doesn't keep track of more than one.
That use case is always going to be hypothetical if the support for it
isn't really there, though, isn't it?
> If you want two processes to communicate (really three), you might want
> to build with the multiple coproc support and use the shell as the
> arbiter.
If you've written a script for other people than just yourself,
expecting all of them to build their own bash install with a
non-default preprocessor directive is pretty unreasonable.
The part that I've been missing this whole time is that using exec
with the fds provided by the coproc keyword is actually a complete
solution for my use case, if I'm willing to close all the resultant
fds myself in background processes where I don't want them to go.
Which I am.
$ coproc CAT1 { cat; }
[1] 1769
$ exec {CAT1_2[0]}<&"${CAT1[0]}" {CAT1_2[1]}>&"${CAT1[1]}"
{CAT1[0]}<&- {CAT1[1]}>&-
$ declare -p CAT1 CAT1_2
declare -a CAT1=([0]="-1" [1]="-1")
declare -a CAT1_2=([0]="10" [1]="11")
$ coproc CAT2 { exec {CAT1_2[0]}<&- {CAT1_2[1]}>&-; cat; }
[2] 1771
$ exec {CAT2_2[0]}<&"${CAT2[0]}" {CAT2_2[1]}>&"${CAT2[1]}"
{CAT2[0]}<&- {CAT2[1]}>&-
$ declare -p CAT2 CAT2_2
declare -a CAT2=([0]="-1" [1]="-1")
declare -a CAT2_2=([0]="12" [1]="13")
$ printf 'dog\ncat\nrabbit\ntortoise\n' >&"${CAT1_2[1]}"
$ IFS='' read -r -u "${CAT1_2[0]}" line; printf '%s\n' "${?}:${line}"
0:dog
$ exec {CAT1_2[1]}>&-
$ IFS='' read -r -u "${CAT1_2[0]}" line; printf '%s\n' "${?}:${line}"
0:cat
[1]- Donecoproc CAT1 { cat; }
$ IFS='' read -r -u "${CAT1_2[0]}" line; printf '%s\n' "${?}:${line}"
0:rabbit
$ IFS='' read -r -u "${CAT1_2[0]}" line; printf '%s\n' "${?}:${line}"
0:tortoise
$ IFS='' read -r -u "${CAT1_2[0]}" line; printf '%s\n' "${?}:${line}"
1:
$ exec {CAT1_2[0]}<&- {CAT2_2[0]}<&- {CAT2_2[1]}>&-
$
[2]+ Done
No warning message when creating the CAT2 coproc. I swear, I was so
close to getting this figured out three years ago, unless the behavior
when a coproc still exists only because other non-coproc fds are
pointing to it has changed since whatever version of bash I was
testing in at the time.
I am completely satisfied with this solution.
The trial and error aspect to figuring this kind of stuff out is
really frustrating. Maybe I'll take some time and write a Wooledge
Wiki article on this at some point, if there isn't one already.
Whether the coproc fds should be automatically kept out of most kinds
of subshells, like it is now; or out of more kinds than currently; is
kind of beside the point to me now. But, having a builtin to ensure
the same behavior is applied to any arbitrary fd might be useful to
people, especially if those fds get removed from process substitutions
as well. If the code for coproc fds gets applied to these fds, then
you've got more chances to see that the logic actually works
correctly, if nothing else.
Re: strtoimax test still broken
On 4/7/24 8:36 AM, Dag-Erling Smørgrav wrote: The strtoimax() existence test in m4/strtoimax.m4 has been broken since its inception in September 2022. The fixed version of strtoimax.m4 has been in the devel branch since October 3, 2022. Since bash-5.2 built fine on FreeBSD 13 before I released it, I assume the link problem exist only on systems using musl. I don't generally release patches for configure, but let's see what I can do here. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: echo test >& "quote'test"
On Tue, Apr 09, 2024 at 01:37:08AM +, squeaky wrote: > Bash Version: 5.2 Patch Level: 21 Release Status: release > > Description: > > Running > echo test >& "quote'test" > should create the file "quote'test", but it creates "quotetest" > instead. I can confirm this all the way back to bash 2.05b, and only with the >& redirection operator. Not with &> or > or >> .
