Re: multi-threaded compiling

[Mischa Baars]

No? No further suggestions?

Then I'd like to thank you all for your help and your input.

I'll be off-list now.

Kind regards,
Mischa Baars.

On Thu, Mar 14, 2024 at 9:17 AM Mischa Baars 
wrote:

> Ok. Then this is what I was able to make of it. To be honest, I prefer the
> Makefile syntax over the script syntax, because of its direct substitutions.
>
> Other improvements that anyone likes to suggest? Is this the best bash can
> do?
>
>
> On Thu, Mar 14, 2024 at 3:47 AM Martin D Kealey 
> wrote:
>
>> On Wed, 13 Mar 2024, Mischa Baars wrote:
>>
>> > Date: Wed, 13 Mar 2024 22:52:16
>> > From: Mischa Baars 
>> > To: alex xmb sw ratchev 
>> > Cc: psm...@gnu.org, bug-bash , help-m...@gnu.org
>> > Subject: Re: multi-threaded compiling
>> >
>> > I found another mistake of mine:
>> >
>> > #ifndef __STRINGIZED__
>> >
>> > should be
>> >
>> > #if __STRINGIZED__ == 0
>> >
>> > or it will always go for the second statement in the conditional.
>> >
>> > Can someone please correct my script??? Things are starting to itch.
>>
>> The fundamental limitation is that Bash only provides lists of scalars
>> masquerading as arrays, and lists of scalar-scalar pairs masquerading as
>> maps (aka hashes, dicts, or "associative arrays").
>>
>> It does not support any more complex data structures, and in particular,
>> does not have arrays-of-arrays or lists-of-lists.
>>
>> So I would change
>>
>>   CFLAGS[0]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[0]}\""
>>   CFLAGS[1]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[1]}\""
>>   CFLAGS[2]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[0]}\""
>>   CFLAGS[3]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[1]}\""
>>
>> to
>>
>>   CFLAGS0=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}" )
>>   CFLAGS1=( -D__STRINGIZED__=0 -D__STRING__="${STR[1]}" )
>>   CFLAGS2=( -D__STRINGIZED__=1 -D__STRING__="${STR[0]}" )
>>   CFLAGS3=( -D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>>
>> and then change
>>
>>   gcc -o main main.c ${CFLAGS[0]} ; ./main
>>   gcc -o main main.c ${CFLAGS[1]} ; ./main
>>   gcc -o main main.c ${CFLAGS[2]} ; ./main
>>   gcc -o main main.c ${CFLAGS[3]} ; ./main
>>
>> to
>>
>>   gcc -o main main.c "${CFLAGS0[@]}" && ./main
>>   gcc -o main main.c "${CFLAGS1[@]}" && ./main
>>   gcc -o main main.c "${CFLAGS2[@]}" && ./main
>>   gcc -o main main.c "${CFLAGS3[@]}" && ./main
>>
>> If you absolutely must have some kind one indexed array for all the
>> permutations of cflags, then you could use a synthetic multidimensional
>> array, like this:
>>
>>   CFLAGS=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}"
>>-D__STRINGIZED__=0 -D__STRING__="${STR[1]}"
>>-D__STRINGIZED__=1 -D__STRING__="${STR[0]}"
>>-D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>>
>>   gcc -o main main.c "${CFLAGS[@]:0*2:2}" && ./main
>>   gcc -o main main.c "${CFLAGS[@]:1*2:2}" && ./main
>>   gcc -o main main.c "${CFLAGS[@]:2*2:2}" && ./main
>>   gcc -o main main.c "${CFLAGS[@]:3*2:2}" && ./main
>>
>> However, given the pattern, I'd have gone with:
>>
>>   for i in 0 1 2 3 ; do
>> gcc -o main main.c -D__STRINGIZED__=$((i/2))
>> -D__STRING__="${STR[i%2]}" &&
>> ./main
>>   done
>>
>> -Martin
>>
>

Re: multi-threaded compiling

[Mischa Baars]

Ok. Then this is what I was able to make of it. To be honest, I prefer the
Makefile syntax over the script syntax, because of its direct substitutions.

Other improvements that anyone likes to suggest? Is this the best bash can
do?


On Thu, Mar 14, 2024 at 3:47 AM Martin D Kealey 
wrote:

> On Wed, 13 Mar 2024, Mischa Baars wrote:
>
> > Date: Wed, 13 Mar 2024 22:52:16
> > From: Mischa Baars 
> > To: alex xmb sw ratchev 
> > Cc: psm...@gnu.org, bug-bash , help-m...@gnu.org
> > Subject: Re: multi-threaded compiling
> >
> > I found another mistake of mine:
> >
> > #ifndef __STRINGIZED__
> >
> > should be
> >
> > #if __STRINGIZED__ == 0
> >
> > or it will always go for the second statement in the conditional.
> >
> > Can someone please correct my script??? Things are starting to itch.
>
> The fundamental limitation is that Bash only provides lists of scalars
> masquerading as arrays, and lists of scalar-scalar pairs masquerading as
> maps (aka hashes, dicts, or "associative arrays").
>
> It does not support any more complex data structures, and in particular,
> does not have arrays-of-arrays or lists-of-lists.
>
> So I would change
>
>   CFLAGS[0]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[0]}\""
>   CFLAGS[1]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[1]}\""
>   CFLAGS[2]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[0]}\""
>   CFLAGS[3]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[1]}\""
>
> to
>
>   CFLAGS0=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}" )
>   CFLAGS1=( -D__STRINGIZED__=0 -D__STRING__="${STR[1]}" )
>   CFLAGS2=( -D__STRINGIZED__=1 -D__STRING__="${STR[0]}" )
>   CFLAGS3=( -D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>
> and then change
>
>   gcc -o main main.c ${CFLAGS[0]} ; ./main
>   gcc -o main main.c ${CFLAGS[1]} ; ./main
>   gcc -o main main.c ${CFLAGS[2]} ; ./main
>   gcc -o main main.c ${CFLAGS[3]} ; ./main
>
> to
>
>   gcc -o main main.c "${CFLAGS0[@]}" && ./main
>   gcc -o main main.c "${CFLAGS1[@]}" && ./main
>   gcc -o main main.c "${CFLAGS2[@]}" && ./main
>   gcc -o main main.c "${CFLAGS3[@]}" && ./main
>
> If you absolutely must have some kind one indexed array for all the
> permutations of cflags, then you could use a synthetic multidimensional
> array, like this:
>
>   CFLAGS=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}"
>-D__STRINGIZED__=0 -D__STRING__="${STR[1]}"
>-D__STRINGIZED__=1 -D__STRING__="${STR[0]}"
>-D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>
>   gcc -o main main.c "${CFLAGS[@]:0*2:2}" && ./main
>   gcc -o main main.c "${CFLAGS[@]:1*2:2}" && ./main
>   gcc -o main main.c "${CFLAGS[@]:2*2:2}" && ./main
>   gcc -o main main.c "${CFLAGS[@]:3*2:2}" && ./main
>
> However, given the pattern, I'd have gone with:
>
>   for i in 0 1 2 3 ; do
> gcc -o main main.c -D__STRINGIZED__=$((i/2))
> -D__STRING__="${STR[i%2]}" &&
> ./main
>   done
>
> -Martin
>


make.sh
Description: application/shellscript
/* * * *
 *	:set tabstop=9
 */

#include	

#define	str(x)	lit(x)
#define	lit(x)	#x

#if	__STRINGIZED__ == 0
const	char*	s  =	__STRING__ ;
#else
const	char*	s  =	str(__STRING__);
#endif

int	main	()
{
	printf	("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}


Makefile
Description: Binary data

Re: multi-threaded compiling

[Mischa Baars]

I found another mistake of mine:

#ifndef __STRINGIZED__

should be

#if __STRINGIZED__ == 0

or it will always go for the second statement in the conditional.

Can someone please correct my script??? Things are starting to itch.

On Wed, Mar 13, 2024 at 8:44 AM alex xmb sw ratchev 
wrote:

>
>
> On Wed, Mar 13, 2024, 08:26 Mischa Baars 
> wrote:
>
>> On Tue, Mar 12, 2024 at 10:00 PM Paul Smith  wrote:
>>
>> > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote:
>> > > > I'd still like to hear why you aren't simply using "make -j".
>> > >
>> > > That's because I don't want to define static compile and link targets
>> > > for every new project I start. The Makefile in question contains only
>> > > a few lines of code and some environment variables, but works on
>> > > every new project as long as you follow certain guidelines regarding
>> > > the directory structure. It scans, compiles and links on the fly.
>> >
>> > I don't find this argument compelling.  It's relatively straightforward
>> > to create a makefile environment that "works on every new project as
>> > long as you follow certain guidelines regarding the directory
>> > structure" while still using parallel builds, that will "scan, compile,
>> > and link" on the fly, using things like the wildcard function, pattern
>> > rules, etc.
>> >
>> > You are merely trading a bit of extra complexity in your makefile for a
>> > whole lot of complexity and tricky hacking in bash, as can be seen by
>> > following this thread.
>> >
>>
>> Only the Makefile is functional right now. The bash script is not working.
>> Good enough to reduce compile time from 1:43 to 0:10. I would have liked
>> to
>> see the script working.
>>
>
> u said u have fs naming conventions
> write em up here , i make the bash ( not sh )
>
> But if you prefer to re-invent make's parallel build capabilities in
>> > bash, certainly that's your prerogative.
>> >
>>
>


Makefile
Description: Binary data
/* * * *
 *	:set tabstop=9
 */

#include	

#define	str(x)	lit(x)
#define	lit(x)	#x

int	main	()
{
#ifndef	__STRINGIZED__
	const	char*	s  =	__STRING__ ;
#else
	const	char*	s  =	str(__STRING__);
#endif

	printf	("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Wed, Mar 13, 2024, 08:26 Mischa Baars 
wrote:

> On Tue, Mar 12, 2024 at 10:00 PM Paul Smith  wrote:
>
> > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote:
> > > > I'd still like to hear why you aren't simply using "make -j".
> > >
> > > That's because I don't want to define static compile and link targets
> > > for every new project I start. The Makefile in question contains only
> > > a few lines of code and some environment variables, but works on
> > > every new project as long as you follow certain guidelines regarding
> > > the directory structure. It scans, compiles and links on the fly.
> >
> > I don't find this argument compelling.  It's relatively straightforward
> > to create a makefile environment that "works on every new project as
> > long as you follow certain guidelines regarding the directory
> > structure" while still using parallel builds, that will "scan, compile,
> > and link" on the fly, using things like the wildcard function, pattern
> > rules, etc.
> >
> > You are merely trading a bit of extra complexity in your makefile for a
> > whole lot of complexity and tricky hacking in bash, as can be seen by
> > following this thread.
> >
>
> Only the Makefile is functional right now. The bash script is not working.
> Good enough to reduce compile time from 1:43 to 0:10. I would have liked to
> see the script working.
>

u said u have fs naming conventions
write em up here , i make the bash ( not sh )

But if you prefer to re-invent make's parallel build capabilities in
> > bash, certainly that's your prerogative.
> >
>

Re: multi-threaded compiling

[Mischa Baars]

On Wed, Mar 13, 2024 at 4:14 AM Martin D Kealey 
wrote:

>
> On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey  wrote:
>> > On 3/11/24 2:50 PM, Mischa Baars wrote:
>> > > Which sort of brings us back to the original question I suppose. Who
>> does
>> > > that line of code function from a script and why does it fail from the
>> > > command line?
>> >
>> > Job control and when the shell notifies the user about job completion,
>> > most likely, two of the relevant things that differ between interactive
>> > and non-interactive shells.
>>
>
> In this case, no.
>
> I inserted « echo $- ; shopt -p ; shopt -po ; » in front of each case, and
> the ONLY difference was that « echo $- » reported “hxBc” vs “hxB”. Not an
> “m” in sight. And no “i” or “l” either. (The “c” was expected, given how «
> make » invokes the shell.)
>
> -Martin
>

 

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 10:00 PM Paul Smith  wrote:

> On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote:
> > > I'd still like to hear why you aren't simply using "make -j".
> >
> > That's because I don't want to define static compile and link targets
> > for every new project I start. The Makefile in question contains only
> > a few lines of code and some environment variables, but works on
> > every new project as long as you follow certain guidelines regarding
> > the directory structure. It scans, compiles and links on the fly.
>
> I don't find this argument compelling.  It's relatively straightforward
> to create a makefile environment that "works on every new project as
> long as you follow certain guidelines regarding the directory
> structure" while still using parallel builds, that will "scan, compile,
> and link" on the fly, using things like the wildcard function, pattern
> rules, etc.
>
> You are merely trading a bit of extra complexity in your makefile for a
> whole lot of complexity and tricky hacking in bash, as can be seen by
> following this thread.
>

Only the Makefile is functional right now. The bash script is not working.
Good enough to reduce compile time from 1:43 to 0:10. I would have liked to
see the script working.

But if you prefer to re-invent make's parallel build capabilities in
> bash, certainly that's your prerogative.
>

Re: multi-threaded compiling

[Martin D Kealey]

> On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey  wrote:
> > On 3/11/24 2:50 PM, Mischa Baars wrote:
> > > Which sort of brings us back to the original question I suppose. Who
> does
> > > that line of code function from a script and why does it fail from the
> > > command line?
> >
> > Job control and when the shell notifies the user about job completion,
> > most likely, two of the relevant things that differ between interactive
> > and non-interactive shells.
>

In this case, no.

I inserted « echo $- ; shopt -p ; shopt -po ; » in front of each case, and
the ONLY difference was that « echo $- » reported “hxBc” vs “hxB”. Not an
“m” in sight. And no “i” or “l” either. (The “c” was expected, given how «
make » invokes the shell.)

-Martin

Re: multi-threaded compiling

[Paul Smith]

On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote:
> > I'd still like to hear why you aren't simply using "make -j".
> 
> That's because I don't want to define static compile and link targets
> for every new project I start. The Makefile in question contains only
> a few lines of code and some environment variables, but works on
> every new project as long as you follow certain guidelines regarding
> the directory structure. It scans, compiles and links on the fly.

I don't find this argument compelling.  It's relatively straightforward
to create a makefile environment that "works on every new project as
long as you follow certain guidelines regarding the directory
structure" while still using parallel builds, that will "scan, compile,
and link" on the fly, using things like the wildcard function, pattern
rules, etc.

You are merely trading a bit of extra complexity in your makefile for a
whole lot of complexity and tricky hacking in bash, as can be seen by
following this thread.

But if you prefer to re-invent make's parallel build capabilities in
bash, certainly that's your prerogative.

Re: multi-threaded compiling

[Chet Ramey]

On 3/12/24 4:23 AM, Mischa Baars wrote:


So... how do I fix it?


If you want double quotes in a variable expansion to have some meaning to
the parser, you need to run the expanded value through the parser, using
something like `eval'. Or you figure out a different way to glue the
arguments together.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: multi-threaded compiling

[Chet Ramey]

On 3/12/24 7:29 AM, Greg Wooledge wrote:


hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; for i in {0..3}; do wait -n -p pid; e=$?; 
printf "pid %s status %s\n" "$pid" "$e"; done'
pid 530407 status 0
pid 530410 status 0
pid  status 127
pid  status 127


Thanks for this example. This case is one where bash aggressively marks
terminated jobs as notified, making them eligible for removal from the
jobs list.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 12:49 PM Greg Wooledge  wrote:

> On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote:
> > Here's the script and the Makefile using "printf '<%s>'":
>
> Sadly, your mail user agent chose to attach "Makefile" with content-type
> application/octet-stream, which my MUA (mutt) refuses to show inline,
> or to include in a reply as quoted text.
>
> Here's the top part of it, inline:
>
>
> STR[0]="one and two"
> STR[1]=one\ and\ two
>
> CFLAGS[0]=-D__STRINGIZED__=0 -D__STRING__=${STR[0]}
> CFLAGS[1]=-D__STRINGIZED__=0 -D__STRING__=${STR[1]}
> CFLAGS[2]=-D__STRINGIZED__=1 -D__STRING__=${STR[0]}
> CFLAGS[3]=-D__STRINGIZED__=1 -D__STRING__=${STR[1]}
>
>
> And here's what Martin said about it:
>
> > On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey  >
> > wrote:
> > > In section two, the problem is that quote removal is done BEFORE
> variables
> > > are expanded, even though it prevents word splitting from being done
> AFTER
> > > variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> > > absolutely cannot do what you might expect; the embedded quote marks
> will
> > > be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> > > resulting string will be split on any embedded whitespace..
>
>
> As someone said yesterday, you will need eval for this.
>
>
> hobbit:/tmp/x$ cat Makefile
> FOO=-D__x="one two three"
>
> all:
> @bash -c 'eval '\''CFLAGS=${FOO}'\''; declare -p CFLAGS'
> hobbit:/tmp/x$ make
> declare -- CFLAGS="-D__x=one two three"
>

>
> Of course, using eval presents its own set of challenges, so proceed
> with extreme caution.
>

Indeed it does :) The Makefile was already working :) It is the script that
won't.


>
> I'd still like to hear why you aren't simply using "make -j".
>

That's because I don't want to define static compile and link targets for
every new project I start. The Makefile in question contains only a few
lines of code and some environment variables, but works on every new
project as long as you follow certain guidelines regarding the directory
structure. It scans, compiles and links on the fly.

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 12:30 PM Greg Wooledge  wrote:

> On Tue, Mar 12, 2024 at 02:15:38PM +0700, Robert Elz wrote:
> > Date:Mon, 11 Mar 2024 17:25:57 -0400
> > From:Chet Ramey 
> > Message-ID:  <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>
> >
> >   | OK, here's the longer answer. When the shell is interactive, and job
> >   | control is enabled, the shell prints job completion notifications to
> >   | stdout at command boundaries.
> >
> > which is, IMO, yet another bogus bash misfeature.  That should
> > happen, with the effect described, but only when PS1 is about
> > to be printed - precisely so commands like the one described
> > will work.   Sigh.
> >
> > There aren't many complaints about this misfeature of bash,
> > as almost no-one writes interactive command lines where it makes
> > a difference.   That doesn't mean they should not be able to.
>
> Yeah, it appears you're right.  In a script, this works as expected:
>
> hobbit:~$ cat foo
> #!/bin/bash
> for i in {0..3}; do sleep 1 & done
> for i in {0..3}; do
> wait -n -p pid; e=$?
> printf 'pid %s status %s\n' "$pid" "$e"
> done
> hobbit:~$ ./foo
> pid 530359 status 0
> pid 530360 status 0
> pid 530361 status 0
> pid 530362 status 0
>
>
> But interactively, *even with bash -c and set +m*, it just fails:
>
> hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; for i in
> {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e";
> done'
> pid 530407 status 0
> pid 530410 status 0
> pid  status 127
> pid  status 127
>
> Looks like a race condition, where some of the children get reaped and
> tossed away before "wait -n -p pid" has a chance to grab their status.
>
> If I stick a "sleep 2" in between the two loops, then it's even worse:
>
> hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; sleep 2;
> for i in {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n"
> "$pid" "$e"; done'pid  status 127
> pid  status 127
> pid  status 127
> pid  status 127
>
> ALL of the children are discarded before the second loop has a chance to
> catch a single one of them. * This is clearly not working as expected*.
>

Now we're talking :) To observe different output from script and from
Makefile certainly is somewhat peculiar and unexpected.


>
> Using "bash +i -c" doesn't change anything, either.  Or "bash +i +m -c".
> Whatever magic it is that you get by putting the code in an actual
> script, I can't figure out how to replicate it from a prompt.
>
>

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 12:19 PM Greg Wooledge  wrote:

> On Tue, Mar 12, 2024 at 10:33:36AM +0100, Mischa Baars wrote:
> > bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
> > sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
> > "$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
>
> "wait -p pid" is not correct here.  In that command, pid is an output
> variable, and you have not specified any process IDs to wait for -- so
> wait is going to wait for *all* of the children to finish, and you'll
> get zero as the exit status:
>

These are the two waits I'm now using:

wait ${pid[${i}]};
wait -np pid;

Which gives the following two lines to execute either from script or from
makefile:

script:

seconds=5; for (( i=0;i<32;i++ )); do exit ${i} & pid[${i}]=${!}; done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait ${pid[${i}]}; e=${?};
echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
seconds=5; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -np pid;  e=${?};
echo "$(printf %3u ${i}) pid ${pid} exit ${e}";   done; #!!! script
only !!!

makefile:

seconds=5; for (( i=0;i<32;i++ )); do exit $${i} & pid[$${i}]=$${!}; done;
sleep $${seconds}; for (( i=0;i<32;i++ )); do wait $${pid[$${i}]}; e=$${?};
echo "$$(printf %3u $${i}) pid $${pid[$${i}]} exit $${e}"; done;
seconds=5; for (( i=0;i<32;i++ )); do exit $${i} &   done;
sleep $${seconds}; for (( i=0;i<32;i++ )); do wait -np pid;e=$${?};
echo "$$(printf %3u $${i}) pid $${pid} exit $${e}";done; #???
script only ???

The first executes correctly from both the script and the Makefile. The
second only executes correctly from script.


> $ help wait
> [...]
>
> Waits for each process identified by an ID, which may be a process ID
> or a
> job specification, and reports its termination status.  If ID is not
> given, waits for all currently active child processes, and the return
> status is zero.  If ID is a job specification, waits for all processes
> in that job's pipeline.
>
>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Tue, Mar 12, 2024, 12:49 Greg Wooledge  wrote:

> On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote:
> > Here's the script and the Makefile using "printf '<%s>'":
>
> Sadly, your mail user agent chose to attach "Makefile" with content-type
> application/octet-stream, which my MUA (mutt) refuses to show inline,
> or to include in a reply as quoted text.
>
> Here's the top part of it, inline:
>
>
> STR[0]="one and two"
> STR[1]=one\ and\ two
>
> CFLAGS[0]=-D__STRINGIZED__=0 -D__STRING__=${STR[0]}
> CFLAGS[1]=-D__STRINGIZED__=0 -D__STRING__=${STR[1]}
> CFLAGS[2]=-D__STRINGIZED__=1 -D__STRING__=${STR[0]}
> CFLAGS[3]=-D__STRINGIZED__=1 -D__STRING__=${STR[1]}
>
>
> And here's what Martin said about it:
>
> > On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey  >
> > wrote:
> > > In section two, the problem is that quote removal is done BEFORE
> variables
> > > are expanded, even though it prevents word splitting from being done
> AFTER
> > > variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> > > absolutely cannot do what you might expect; the embedded quote marks
> will
> > > be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> > > resulting string will be split on any embedded whitespace..
>
>
> As someone said yesterday, you will need eval for this.
>
>
> hobbit:/tmp/x$ cat Makefile
> FOO=-D__x="one two three"
>
> all:
> @bash -c 'eval '\''CFLAGS=${FOO}'\''; declare -p CFLAGS'
> hobbit:/tmp/x$ make
> declare -- CFLAGS="-D__x=one two three"
>
>
> Of course, using eval presents its own set of challenges, so proceed
> with extreme caution.
>

declare instead of eval

I'd still like to hear why you aren't simply using "make -j".
>
>

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote:
> Here's the script and the Makefile using "printf '<%s>'":

Sadly, your mail user agent chose to attach "Makefile" with content-type
application/octet-stream, which my MUA (mutt) refuses to show inline,
or to include in a reply as quoted text.

Here's the top part of it, inline:


STR[0]="one and two"
STR[1]=one\ and\ two

CFLAGS[0]=-D__STRINGIZED__=0 -D__STRING__=${STR[0]}
CFLAGS[1]=-D__STRINGIZED__=0 -D__STRING__=${STR[1]}
CFLAGS[2]=-D__STRINGIZED__=1 -D__STRING__=${STR[0]}
CFLAGS[3]=-D__STRINGIZED__=1 -D__STRING__=${STR[1]}


And here's what Martin said about it:

> On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey 
> wrote:
> > In section two, the problem is that quote removal is done BEFORE variables
> > are expanded, even though it prevents word splitting from being done AFTER
> > variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> > absolutely cannot do what you might expect; the embedded quote marks will
> > be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> > resulting string will be split on any embedded whitespace..


As someone said yesterday, you will need eval for this.


hobbit:/tmp/x$ cat Makefile
FOO=-D__x="one two three"

all:
@bash -c 'eval '\''CFLAGS=${FOO}'\''; declare -p CFLAGS'
hobbit:/tmp/x$ make
declare -- CFLAGS="-D__x=one two three"


Of course, using eval presents its own set of challenges, so proceed
with extreme caution.

I'd still like to hear why you aren't simply using "make -j".

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 02:15:38PM +0700, Robert Elz wrote:
> Date:Mon, 11 Mar 2024 17:25:57 -0400
> From:Chet Ramey 
> Message-ID:  <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>
> 
>   | OK, here's the longer answer. When the shell is interactive, and job
>   | control is enabled, the shell prints job completion notifications to
>   | stdout at command boundaries.
> 
> which is, IMO, yet another bogus bash misfeature.  That should
> happen, with the effect described, but only when PS1 is about
> to be printed - precisely so commands like the one described
> will work.   Sigh.
> 
> There aren't many complaints about this misfeature of bash,
> as almost no-one writes interactive command lines where it makes
> a difference.   That doesn't mean they should not be able to.

Yeah, it appears you're right.  In a script, this works as expected:

hobbit:~$ cat foo
#!/bin/bash
for i in {0..3}; do sleep 1 & done
for i in {0..3}; do
wait -n -p pid; e=$?
printf 'pid %s status %s\n' "$pid" "$e"
done
hobbit:~$ ./foo
pid 530359 status 0
pid 530360 status 0
pid 530361 status 0
pid 530362 status 0


But interactively, *even with bash -c and set +m*, it just fails:

hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; for i in {0..3}; 
do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e"; done'
pid 530407 status 0
pid 530410 status 0
pid  status 127
pid  status 127

Looks like a race condition, where some of the children get reaped and
tossed away before "wait -n -p pid" has a chance to grab their status.

If I stick a "sleep 2" in between the two loops, then it's even worse:

hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; sleep 2; for i 
in {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e"; 
done'pid  status 127
pid  status 127
pid  status 127
pid  status 127

ALL of the children are discarded before the second loop has a chance to
catch a single one of them.  This is clearly not working as expected.

Using "bash +i -c" doesn't change anything, either.  Or "bash +i +m -c".
Whatever magic it is that you get by putting the code in an actual
script, I can't figure out how to replicate it from a prompt.

Re: multi-threaded compiling

[Greg Wooledge]

On Tue, Mar 12, 2024 at 10:33:36AM +0100, Mischa Baars wrote:
> bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
> sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
> "$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'

"wait -p pid" is not correct here.  In that command, pid is an output
variable, and you have not specified any process IDs to wait for -- so
wait is going to wait for *all* of the children to finish, and you'll
get zero as the exit status:

$ help wait
[...]

Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status.  If ID is not
given, waits for all currently active child processes, and the return
status is zero.  If ID is a job specification, waits for all processes
in that job's pipeline.

Re: multi-threaded compiling

[Robert Elz]

Date:Tue, 12 Mar 2024 18:32:19 +1000
From:Martin D Kealey 
Message-ID:  


  | You don't need '-n' when you specify a PID; the fix is simply to remove it.

That's OK, when there is exactly one PID, but for a task like this,

wait -n -p pid PID1 PID2 PID3 ...

would be a better invocation, to wait for whichever of the list finishes
first.   Still doesn't get around the bash idiocy though (for which there
is a very simple solution, I think, which I suggested to Chet off list a
while ago).

In a more realistic application, where the tasks all take longer than
the script takes to start them, and get to the point of waiting, solutions
using -n would mostly work, even in bash.   It just fails when the tasks
have finished before "wait -n" is performed (or if they are killed by a
signal).

kre

Re: multi-threaded compiling

[Robert Elz]

  | I think you forgot to look at the actual code in the Makefile here.

No, I didn't forget, I never intended to look, unpacking attached
tar.gz files is more effort than I'm willing to undertake.

  | Every '$' is written as a '$$', exactly as you say it should.

OK, I guessed incorrectly ... just based upon the typical kind of
mistake I'd expect from someone who doesn't really understand shell
basics like quoting and field splitting.

kre

Re: multi-threaded compiling

[Mischa Baars]

Got it!!!

bash -c 'set +m; seconds=5; for (( i=0;i<32;i++ )); do exit ${i} &
pid[${i}]=${!}; done; sleep ${seconds}; for (( i=0;i<32;i++ )); do wait
${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}";
done;'

One problem solved :) The problem was exactly the -n, and had nothing to do
with interactive / non-interactive shells. Thank you for helping! Now the
Makefile works just as good as the script.

I can now use at least Makefiles for the parallel compilation. Would be
nice if we got #3 in the script working as well. Because I'm still curious
how to do the job right.

Best regards,
Mischa.

On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey  wrote:

> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Mischa Baars]

Oh, sorry. Without the -n that is.

bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
bash -c 'set -m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'

disables / enables the notifications respectively, but doesn't do anything
otherwise.

On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey  wrote:

> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Mischa Baars]

Hi Chet,

The only thing I can find about this behavior is the set +m / set -m
option, which disables / enables the notifications, but has no effect
otherwise.

The line from the script won't execute from the command line with either
option, neither enclosed with a 'bash -c' as in:

set +m; bash -c 'seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -np pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'

On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey  wrote:

> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Mischa Baars]

Here's the script and the Makefile using "printf '<%s>'":

On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey 
wrote:

> In section one, the problem is that "wait -n" does not do what you think
> it does. (Lots of us think this behaviour is broken, and it may be fixed in
> an upcoming version of Bash.) You don't need '-n' when you specify a PID;
> the fix is simply to remove it.
>
> In section two, the problem is that quote removal is done BEFORE variables
> are expanded, even though it prevents word splitting from being done AFTER
> variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> absolutely cannot do what you might expect; the embedded quote marks will
> be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> resulting string will be split on any embedded whitespace..
>
>
> On Mon, 11 Mar 2024 at 18:56, Mischa Baars 
> wrote:
>
>> Hi,
>>
>> I'd like to compile a project of mine using multiple logical cores.
>>
>> I've attached the problem. It consists of two parts:
>>
>> 1) multi-threaded bash script and / or multi-threaded Makefile
>>
>> Running bash script functions as expected, but executing the same line of
>> code with make and / or the command line, does not function. Perhaps
>> someone could explain to me why?
>>
>> 2) passing a string argument from a bash script and / or Makefile to the
>> gcc -D option
>>
>> Running the makefile functions as expected, but I have not been able to
>> get
>> similar code to work from a bash script. Can someone please explain to me
>> what I'm doing wrong?
>>
>> Hope to hear from you soon.
>>
>> Best regards,
>> Mischa Baars.
>>
>


Makefile
Description: Binary data


make.sh
Description: application/shellscript
/* * * *
 *	:set tabstop=9
 */

#include	

#define	str(x)	lit(x)
#define	lit(x)	#x

int	main	()
{
#ifndef	__STRINGIZED__
	const	char*	s  =	__STRING__ ;
#else
	const	char*	s  =	str(__STRING__);
#endif

	printf	("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey 
wrote:

> In section one, the problem is that "wait -n" does not do what you think
> it does. (Lots of us think this behaviour is broken, and it may be fixed in
> an upcoming version of Bash.) You don't need '-n' when you specify a PID;
> the fix is simply to remove it.
>

Thanks. I already noticed the mistake yesterday and corrected it.


>
> In section two, the problem is that quote removal is done BEFORE variables
> are expanded, even though it prevents word splitting from being done AFTER
> variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> absolutely cannot do what you might expect; the embedded quote marks will
> be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> resulting string will be split on any embedded whitespace..
>
>
So.. how do I make the script produce the same output as the Makefile?
Using 'printf <%s>' as Chet said I should, does show the difference between
the two, but it doesn't fix the problem.


>
> On Mon, 11 Mar 2024 at 18:56, Mischa Baars 
> wrote:
>
>> Hi,
>>
>> I'd like to compile a project of mine using multiple logical cores.
>>
>> I've attached the problem. It consists of two parts:
>>
>> 1) multi-threaded bash script and / or multi-threaded Makefile
>>
>> Running bash script functions as expected, but executing the same line of
>> code with make and / or the command line, does not function. Perhaps
>> someone could explain to me why?
>>
>> 2) passing a string argument from a bash script and / or Makefile to the
>> gcc -D option
>>
>> Running the makefile functions as expected, but I have not been able to
>> get
>> similar code to work from a bash script. Can someone please explain to me
>> what I'm doing wrong?
>>
>> Hope to hear from you soon.
>>
>> Best regards,
>> Mischa Baars.
>>
>

Re: multi-threaded compiling

[Martin D Kealey]

In section one, the problem is that "wait -n" does not do what you think it
does. (Lots of us think this behaviour is broken, and it may be fixed in an
upcoming version of Bash.) You don't need '-n' when you specify a PID; the
fix is simply to remove it.

In section two, the problem is that quote removal is done BEFORE variables
are expanded, even though it prevents word splitting from being done AFTER
variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
absolutely cannot do what you might expect; the embedded quote marks will
be used literally, and then (because ${CFLAGS[0]} is not quoted) the
resulting string will be split on any embedded whitespace..


On Mon, 11 Mar 2024 at 18:56, Mischa Baars 
wrote:

> Hi,
>
> I'd like to compile a project of mine using multiple logical cores.
>
> I've attached the problem. It consists of two parts:
>
> 1) multi-threaded bash script and / or multi-threaded Makefile
>
> Running bash script functions as expected, but executing the same line of
> code with make and / or the command line, does not function. Perhaps
> someone could explain to me why?
>
> 2) passing a string argument from a bash script and / or Makefile to the
> gcc -D option
>
> Running the makefile functions as expected, but I have not been able to get
> similar code to work from a bash script. Can someone please explain to me
> what I'm doing wrong?
>
> Hope to hear from you soon.
>
> Best regards,
> Mischa Baars.
>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Tue, Mar 12, 2024, 09:26 Mischa Baars 
wrote:

> On Mon, Mar 11, 2024 at 10:26 PM Chet Ramey  wrote:
>
> > On 3/11/24 3:44 PM, Mischa Baars wrote:
> > > On Mon, 11 Mar 2024, 20:20 Chet Ramey,  > > > wrote:
> > >
> > > On 3/11/24 2:50 PM, Mischa Baars wrote:
> > >  > Which sort of brings us back to the original question I suppose.
> > Who
> > > does
> > >  > that line of code function from a script and why does it fail
> > from the
> > >  > command line?
> > >
> > > Job control and when the shell notifies the user about job
> > completion,
> > > most likely, two of the relevant things that differ between
> > interactive
> > > and non-interactive shells.
> > >
> > >
> > > Wasn't expecting any better answer, but thanks anyway.
> >
> > OK, here's the longer answer. When the shell is interactive, and job
> > control is enabled, the shell prints job completion notifications to
> > stdout at command boundaries. Once the shell notifies the user of a
> > job's completion, it is no longer `new' (or the `next', whichever
> > mnemonic you prefer), and `wait -n' will not return it.
> >
> > So if you include the complete output of an interactive shell executing
> > the above list (with the obvious corrections already discussed), you'll
> > get a series of
> >
> > 32 messages about job creation
> > 32 notifications of job exits, with exit status (they exit immediately,
> > after all)
> > 32 error messages because the user has already been notified of the
> > job's completion and the shell has reaped it (so it is no longer the
> > `next' job to exit)
> >
> > When the shell is not interactive, it only prints job notifications to
> > stdout when the job is terminated by a signal, so the jobs are available
> > to `wait -n'.
> >
> > There was quite a long discussion about this aspect of `wait -n' behavior
> > a couple of months ago, starting at
> >
> > https://lists.gnu.org/archive/html/bug-bash/2024-01/msg00104.html
> >
> >
> > > And the script from directory 'two'? How do I pass this string through
> > the
> > > CFLAGS variable to the compiler? What am I doing wrong in #3 of the
> > script?
> >
> > You're not taking word splitting into account. It seems like you want
> > selective word splitting -- for the embedded double quotes in the various
> > elements of CFLAGS to have some semantic meaning and somehow inhibit
> > word splitting. Quotes are special to the parser, not when they are in
> > the results of word expansion. If you want the parser to evaluate the
> > results of word expansion, you're going to have to run it through the
> > parser again, using something like `eval'.
> >
> > It's clearer if you replace `gcc' in your command with
> >
> > printf '<%s> '
> >
> > and `./main' with `echo'.
> >
> > Yes. Now I see a difference in all the different lines. I wasn't familiar
> with this printf construct, but the results are clear:
>
> Using make:
>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
>
> Using ./make.sh:
>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
>
> and either for unquoted variable in #3
>
> <-o><-D__STRINGIZED__=0><-D__STRING__="one>
> <-o><-D__STRINGIZED__=0><-D__STRING__="one>
> <-o><-D__STRINGIZED__=1><-D__STRING__="one>
> <-o><-D__STRINGIZED__=1><-D__STRING__="one>
>
> or for double quoted variable in #3
>
> <-o><-D__STRINGIZED__=0 -D__STRING__="one and two">
> <-o><-D__STRINGIZED__=0 -D__STRING__="one and two">
> <-o><-D__STRINGIZED__=1 -D__STRING__="one and two">
> <-o><-D__STRINGIZED__=1 -D__STRING__="one and two">
>
> So... how do I fix it?
>

if u need to expand quotes , use any builtin
like
declare -a "arr=( $var_to_expand )" and use arr[@]

> --
> > ``The lyf so short, the craft so long to lerne.'' - Chaucer
> >  ``Ars longa, vita brevis'' - Hippocrates
> > Chet Ramey, UTech, CWRUc...@case.edu
> http://tiswww.cwru.edu/~chet/
> >
> >
>

Re: multi-threaded compiling

[Mischa Baars]

On Mon, Mar 11, 2024 at 10:26 PM Chet Ramey  wrote:

> On 3/11/24 3:44 PM, Mischa Baars wrote:
> > On Mon, 11 Mar 2024, 20:20 Chet Ramey,  > > wrote:
> >
> > On 3/11/24 2:50 PM, Mischa Baars wrote:
> >  > Which sort of brings us back to the original question I suppose.
> Who
> > does
> >  > that line of code function from a script and why does it fail
> from the
> >  > command line?
> >
> > Job control and when the shell notifies the user about job
> completion,
> > most likely, two of the relevant things that differ between
> interactive
> > and non-interactive shells.
> >
> >
> > Wasn't expecting any better answer, but thanks anyway.
>
> OK, here's the longer answer. When the shell is interactive, and job
> control is enabled, the shell prints job completion notifications to
> stdout at command boundaries. Once the shell notifies the user of a
> job's completion, it is no longer `new' (or the `next', whichever
> mnemonic you prefer), and `wait -n' will not return it.
>
> So if you include the complete output of an interactive shell executing
> the above list (with the obvious corrections already discussed), you'll
> get a series of
>
> 32 messages about job creation
> 32 notifications of job exits, with exit status (they exit immediately,
> after all)
> 32 error messages because the user has already been notified of the
> job's completion and the shell has reaped it (so it is no longer the
> `next' job to exit)
>
> When the shell is not interactive, it only prints job notifications to
> stdout when the job is terminated by a signal, so the jobs are available
> to `wait -n'.
>
> There was quite a long discussion about this aspect of `wait -n' behavior
> a couple of months ago, starting at
>
> https://lists.gnu.org/archive/html/bug-bash/2024-01/msg00104.html
>
>
> > And the script from directory 'two'? How do I pass this string through
> the
> > CFLAGS variable to the compiler? What am I doing wrong in #3 of the
> script?
>
> You're not taking word splitting into account. It seems like you want
> selective word splitting -- for the embedded double quotes in the various
> elements of CFLAGS to have some semantic meaning and somehow inhibit
> word splitting. Quotes are special to the parser, not when they are in
> the results of word expansion. If you want the parser to evaluate the
> results of word expansion, you're going to have to run it through the
> parser again, using something like `eval'.
>
> It's clearer if you replace `gcc' in your command with
>
> printf '<%s> '
>
> and `./main' with `echo'.
>
> Yes. Now I see a difference in all the different lines. I wasn't familiar
with this printf construct, but the results are clear:

Using make:

<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>

Using ./make.sh:

<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=0><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>
<-o><-D__STRINGIZED__=1><-D__STRING__=one and two>

and either for unquoted variable in #3

<-o><-D__STRINGIZED__=0><-D__STRING__="one>
<-o><-D__STRINGIZED__=0><-D__STRING__="one>
<-o><-D__STRINGIZED__=1><-D__STRING__="one>
<-o><-D__STRINGIZED__=1><-D__STRING__="one>

or for double quoted variable in #3

<-o><-D__STRINGIZED__=0 -D__STRING__="one and two">
<-o><-D__STRINGIZED__=0 -D__STRING__="one and two">
<-o><-D__STRINGIZED__=1 -D__STRING__="one and two">
<-o><-D__STRINGIZED__=1 -D__STRING__="one and two">

So... how do I fix it?


> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Mischa Baars]

On Tue, Mar 12, 2024 at 8:16 AM Robert Elz  wrote:

> This whole discussuon has been a total mess.
>
> First the original message didn't give nearly enough info.
> That might have been in the attachment, but no-one should be
> expecting people to do any needless work when you're asking
> for (free) assistance - if you want help you need to make
> it as easy as possible for people to do that.
>
> Second, a very common issue with people having problems, is
> when they have a problem to solve, they work out how they think
> they should do that, and then have difficulty making one
> of the steps work.  At that point they ask how to do that
> thing, which might sometimes be very complex, even impossible.
> If the initial problem had been stated, along with the
> solution proposed, leading to the issue that's hard, then
> instead of (perhaps) wasting needless time dreaming up
> possible solutions to difficult problems, someone might
> just point out a much simpler method to solve the original
> problem, making finding a solution to the difficult issue moot.
>
> Second, the replies here have mostly been hopeless, with
> just a couple addressing real issues.
>
> Many of the initial replies totally missed that the OP
> said that the bash script in 'one' worked, and there
> was this long thread suggesting other ways to do it.
> Waste if everyone's time (even if the advice not to
> use SECONDS was probably useful).
>
> There is however no blanket rule that vars in upper case are
> reserved for the shell - posix contains a list of var
> names scripts should avoid using (unless they want some
> particular feature supported by a particular shell they
> are using - SECONDS & RANDOM are on that list).   Anything
> else is (or should be) OK to use.
>
> The issue with 'one' was that the same code did not work on
> the command line, or in a makefile.
>
> Chet explained the command line problem:
>
> Date:Mon, 11 Mar 2024 17:25:57 -0400
> From:Chet Ramey 
> Message-ID:  <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>
>
>   | OK, here's the longer answer. When the shell is interactive, and job
>   | control is enabled, the shell prints job completion notifications to
>   | stdout at command boundaries.
>
> which is, IMO, yet another bogus bash misfeature.  That should
> happen, with the effect described, but only when PS1 is about
> to be printed - precisely so commands like the one described
> will work.   Sigh.
>
> There aren't many complaints about this misfeature of bash,
> as almost no-one writes interactive command lines where it makes
> a difference.   That doesn't mean they should not be able to.
>
>
> For the problem using make, it was suggested to explicity set
> the SHELL to use in the makefile, which is good advice when
> using shell specific features, as is being done here.
>
> But the real problem is almost certainly that the Makefile
> is attempting to pass scripts containing '$' chars to the
> shell.  '$' is interpreted by make, to send a '$' to the
> shell, it must be written as '$$'.
>
> Uhm... I think you forgot to look at the actual code in the Makefile here.
Every '$' is written as a '$$', exactly as you say it should.

>
> For problem 'two', Greg (I think it was) diagnosed the
> issue with var expansions, and field splitting - but if
> that was unknown to the OP then that OP has no business
> attempting to write any shell script (with any kind of
> variable) - get an introductory shell programming text
> and learn how it works - at least the basics.  Don't
> just guess or attempt to copy other scripts, you'll fail.
>
> kre
>
>

Re: multi-threaded compiling

[Robert Elz]

This whole discussuon has been a total mess.

First the original message didn't give nearly enough info.
That might have been in the attachment, but no-one should be
expecting people to do any needless work when you're asking
for (free) assistance - if you want help you need to make
it as easy as possible for people to do that.

Second, a very common issue with people having problems, is
when they have a problem to solve, they work out how they think
they should do that, and then have difficulty making one
of the steps work.  At that point they ask how to do that
thing, which might sometimes be very complex, even impossible.
If the initial problem had been stated, along with the
solution proposed, leading to the issue that's hard, then
instead of (perhaps) wasting needless time dreaming up
possible solutions to difficult problems, someone might
just point out a much simpler method to solve the original
problem, making finding a solution to the difficult issue moot.

Second, the replies here have mostly been hopeless, with
just a couple addressing real issues.

Many of the initial replies totally missed that the OP
said that the bash script in 'one' worked, and there
was this long thread suggesting other ways to do it.
Waste if everyone's time (even if the advice not to
use SECONDS was probably useful).

There is however no blanket rule that vars in upper case are
reserved for the shell - posix contains a list of var
names scripts should avoid using (unless they want some
particular feature supported by a particular shell they
are using - SECONDS & RANDOM are on that list).   Anything
else is (or should be) OK to use.

The issue with 'one' was that the same code did not work on
the command line, or in a makefile.

Chet explained the command line problem:

Date:Mon, 11 Mar 2024 17:25:57 -0400
From:Chet Ramey 
Message-ID:  <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>

  | OK, here's the longer answer. When the shell is interactive, and job
  | control is enabled, the shell prints job completion notifications to
  | stdout at command boundaries.

which is, IMO, yet another bogus bash misfeature.  That should
happen, with the effect described, but only when PS1 is about
to be printed - precisely so commands like the one described
will work.   Sigh.

There aren't many complaints about this misfeature of bash,
as almost no-one writes interactive command lines where it makes
a difference.   That doesn't mean they should not be able to.


For the problem using make, it was suggested to explicity set
the SHELL to use in the makefile, which is good advice when
using shell specific features, as is being done here.

But the real problem is almost certainly that the Makefile
is attempting to pass scripts containing '$' chars to the
shell.  '$' is interpreted by make, to send a '$' to the
shell, it must be written as '$$'.


For problem 'two', Greg (I think it was) diagnosed the
issue with var expansions, and field splitting - but if
that was unknown to the OP then that OP has no business
attempting to write any shell script (with any kind of
variable) - get an introductory shell programming text
and learn how it works - at least the basics.  Don't
just guess or attempt to copy other scripts, you'll fail.

kre

Re: multi-threaded compiling

[alex xmb sw ratchev]

a version with success or fail handling

~ $ bash xmb.smallt.2
2/2
1/1
~ $ cat xmb.smallt.2
#!/bin/bash

unset -v success fail cmd a e

 run() {
# uaage , $1 successcase cmd , $2 fail cmd , $3 and further actual cmd to
run
local IFS=' '
eval "${*:3} &"
success[$!]=$1 fail[$!]=$2 cmd[$!]=${*:3}
 }

   wa() {
local pid ret
wait -n -p pid
ret=$?
#printf %s\\n "pid $pid cmd ${me[pid]} returned $ret"
  if (( ret )) && e=${fail[pid]} ; then
 [[ $e ]] &&
eval "$e"
  else
e=${success[pid]}
 [[ $e ]] &&
eval "$e"
  fi
   }

 t2() {
sleep .75
return 3
 }

run 'echo 1/1' 'echo 1/2' sleep 1
run 'echo 2/1' 'echo 2/2' t2

wa
wa

On Mon, Mar 11, 2024, 22:45 alex xmb sw ratchev  wrote:

>
>
> On Mon, Mar 11, 2024, 22:40 alex xmb sw ratchev  wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev 
>> wrote:
>>
>>> ~ $ bash xmb.smallt
>>> pid 14333 cmd t2 returned 3
>>> pid 14332 cmd sleep 1 returned 0
>>>
>>> ~ $ cat xmb.smallt
>>> #!/bin/bash
>>>
>>>  run() {
>>> local IFS=' ' run=$*
>>> eval "$run &"
>>> me[$!]=$run
>>>  }
>>>
>>
>> alternativly to mention aliases usage
>>
>> shopt -s expand_aliases
>>
>> then in func
>> IFS=' '
>> alias torun=$*
>>
>
> and with quotes prolly ye =/
>
> then
>> torun &
>>
>>  wa() {
>>> local pid
>>> wait -n -p pid
>>> printf %s\\n "pid $pid cmd ${me[pid]} returned $?"
>>>  }
>>>
>>>  t2() {
>>> sleep .75
>>> return 3
>>>  }
>>>
>>> run sleep 1
>>> run t2
>>>
>>> wa
>>> wa
>>>
>>> On Mon, Mar 11, 2024, 22:24 Greg Wooledge  wrote:
>>>
 On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
 > On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
 > > The pid can be obtained with the -p option, as of 5.1. Below is a
 > > synthetic example of how it might be put into practice.

 I'd forgotten about that one.  A recent addition, and one I've never
 used
 yet.

 > > #!/bin/bash
 > >
 > > declare -A job_by status_by
 > > max_jobs=4
 > > jobs=0
 > >
 > > wait_next() {
 > > local pid
 > > wait -n -p pid
 > > status_by[$pid]=$?
 > >
 >
 > How exactly is this indexing implemented internally? Does a first
 number on
 > index n take m bits (using a linked list) or does it take n * m bits
 (using
 > realloc(3))?

 "declare -A" makes it an associative array (hash table).

 Without that declare -A, it would be a sparsely indexed array.

 In either case, it doesn't just blindly allocate millions of bytes of
 memory.  It uses something slimmer.




xmb.smallt.2
Description: Binary data

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 22:40 alex xmb sw ratchev  wrote:

>
>
> On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev  wrote:
>
>> ~ $ bash xmb.smallt
>> pid 14333 cmd t2 returned 3
>> pid 14332 cmd sleep 1 returned 0
>>
>> ~ $ cat xmb.smallt
>> #!/bin/bash
>>
>>  run() {
>> local IFS=' ' run=$*
>> eval "$run &"
>> me[$!]=$run
>>  }
>>
>
> alternativly to mention aliases usage
>
> shopt -s expand_aliases
>
> then in func
> IFS=' '
> alias torun=$*
>

and with quotes prolly ye =/

then
> torun &
>
>  wa() {
>> local pid
>> wait -n -p pid
>> printf %s\\n "pid $pid cmd ${me[pid]} returned $?"
>>  }
>>
>>  t2() {
>> sleep .75
>> return 3
>>  }
>>
>> run sleep 1
>> run t2
>>
>> wa
>> wa
>>
>> On Mon, Mar 11, 2024, 22:24 Greg Wooledge  wrote:
>>
>>> On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
>>> > On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
>>> > > The pid can be obtained with the -p option, as of 5.1. Below is a
>>> > > synthetic example of how it might be put into practice.
>>>
>>> I'd forgotten about that one.  A recent addition, and one I've never used
>>> yet.
>>>
>>> > > #!/bin/bash
>>> > >
>>> > > declare -A job_by status_by
>>> > > max_jobs=4
>>> > > jobs=0
>>> > >
>>> > > wait_next() {
>>> > > local pid
>>> > > wait -n -p pid
>>> > > status_by[$pid]=$?
>>> > >
>>> >
>>> > How exactly is this indexing implemented internally? Does a first
>>> number on
>>> > index n take m bits (using a linked list) or does it take n * m bits
>>> (using
>>> > realloc(3))?
>>>
>>> "declare -A" makes it an associative array (hash table).
>>>
>>> Without that declare -A, it would be a sparsely indexed array.
>>>
>>> In either case, it doesn't just blindly allocate millions of bytes of
>>> memory.  It uses something slimmer.
>>>
>>>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 22:40 alex xmb sw ratchev  wrote:

>
>
> On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev  wrote:
>
>> ~ $ bash xmb.smallt
>> pid 14333 cmd t2 returned 3
>> pid 14332 cmd sleep 1 returned 0
>>
>> ~ $ cat xmb.smallt
>> #!/bin/bash
>>
>>  run() {
>> local IFS=' ' run=$*
>> eval "$run &"
>> me[$!]=$run
>>  }
>>
>
> alternativly to mention aliases usage
>
> shopt -s expand_aliases
>
> then in func
> IFS=' '
> alias torun=$*
>
> then
> torun &
>

still needs then one more $* expand to set cmds[$!] to it

 wa() {
>> local pid
>> wait -n -p pid
>> printf %s\\n "pid $pid cmd ${me[pid]} returned $?"
>>  }
>>
>>  t2() {
>> sleep .75
>> return 3
>>  }
>>
>> run sleep 1
>> run t2
>>
>> wa
>> wa
>>
>> On Mon, Mar 11, 2024, 22:24 Greg Wooledge  wrote:
>>
>>> On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
>>> > On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
>>> > > The pid can be obtained with the -p option, as of 5.1. Below is a
>>> > > synthetic example of how it might be put into practice.
>>>
>>> I'd forgotten about that one.  A recent addition, and one I've never used
>>> yet.
>>>
>>> > > #!/bin/bash
>>> > >
>>> > > declare -A job_by status_by
>>> > > max_jobs=4
>>> > > jobs=0
>>> > >
>>> > > wait_next() {
>>> > > local pid
>>> > > wait -n -p pid
>>> > > status_by[$pid]=$?
>>> > >
>>> >
>>> > How exactly is this indexing implemented internally? Does a first
>>> number on
>>> > index n take m bits (using a linked list) or does it take n * m bits
>>> (using
>>> > realloc(3))?
>>>
>>> "declare -A" makes it an associative array (hash table).
>>>
>>> Without that declare -A, it would be a sparsely indexed array.
>>>
>>> In either case, it doesn't just blindly allocate millions of bytes of
>>> memory.  It uses something slimmer.
>>>
>>>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev  wrote:

> ~ $ bash xmb.smallt
> pid 14333 cmd t2 returned 3
> pid 14332 cmd sleep 1 returned 0
>
> ~ $ cat xmb.smallt
> #!/bin/bash
>
>  run() {
> local IFS=' ' run=$*
> eval "$run &"
> me[$!]=$run
>  }
>

alternativly to mention aliases usage

shopt -s expand_aliases

then in func
IFS=' '
alias torun=$*

then
torun &

 wa() {
> local pid
> wait -n -p pid
> printf %s\\n "pid $pid cmd ${me[pid]} returned $?"
>  }
>
>  t2() {
> sleep .75
> return 3
>  }
>
> run sleep 1
> run t2
>
> wa
> wa
>
> On Mon, Mar 11, 2024, 22:24 Greg Wooledge  wrote:
>
>> On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
>> > On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
>> > > The pid can be obtained with the -p option, as of 5.1. Below is a
>> > > synthetic example of how it might be put into practice.
>>
>> I'd forgotten about that one.  A recent addition, and one I've never used
>> yet.
>>
>> > > #!/bin/bash
>> > >
>> > > declare -A job_by status_by
>> > > max_jobs=4
>> > > jobs=0
>> > >
>> > > wait_next() {
>> > > local pid
>> > > wait -n -p pid
>> > > status_by[$pid]=$?
>> > >
>> >
>> > How exactly is this indexing implemented internally? Does a first
>> number on
>> > index n take m bits (using a linked list) or does it take n * m bits
>> (using
>> > realloc(3))?
>>
>> "declare -A" makes it an associative array (hash table).
>>
>> Without that declare -A, it would be a sparsely indexed array.
>>
>> In either case, it doesn't just blindly allocate millions of bytes of
>> memory.  It uses something slimmer.
>>
>>

Re: multi-threaded compiling

[alex xmb sw ratchev]

~ $ bash xmb.smallt
pid 14333 cmd t2 returned 3
pid 14332 cmd sleep 1 returned 0

~ $ cat xmb.smallt
#!/bin/bash

 run() {
local IFS=' ' run=$*
eval "$run &"
me[$!]=$run
 }

 wa() {
local pid
wait -n -p pid
printf %s\\n "pid $pid cmd ${me[pid]} returned $?"
 }

 t2() {
sleep .75
return 3
 }

run sleep 1
run t2

wa
wa

On Mon, Mar 11, 2024, 22:24 Greg Wooledge  wrote:

> On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
> > On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
> > > The pid can be obtained with the -p option, as of 5.1. Below is a
> > > synthetic example of how it might be put into practice.
>
> I'd forgotten about that one.  A recent addition, and one I've never used
> yet.
>
> > > #!/bin/bash
> > >
> > > declare -A job_by status_by
> > > max_jobs=4
> > > jobs=0
> > >
> > > wait_next() {
> > > local pid
> > > wait -n -p pid
> > > status_by[$pid]=$?
> > >
> >
> > How exactly is this indexing implemented internally? Does a first number
> on
> > index n take m bits (using a linked list) or does it take n * m bits
> (using
> > realloc(3))?
>
> "declare -A" makes it an associative array (hash table).
>
> Without that declare -A, it would be a sparsely indexed array.
>
> In either case, it doesn't just blindly allocate millions of bytes of
> memory.  It uses something slimmer.
>
>


xmb.smallt
Description: Binary data

Re: multi-threaded compiling

[Chet Ramey]

On 3/11/24 3:44 PM, Mischa Baars wrote:
On Mon, 11 Mar 2024, 20:20 Chet Ramey, > wrote:


On 3/11/24 2:50 PM, Mischa Baars wrote:
 > Which sort of brings us back to the original question I suppose. Who
does
 > that line of code function from a script and why does it fail from the
 > command line?

Job control and when the shell notifies the user about job completion,
most likely, two of the relevant things that differ between interactive
and non-interactive shells.


Wasn't expecting any better answer, but thanks anyway.


OK, here's the longer answer. When the shell is interactive, and job
control is enabled, the shell prints job completion notifications to
stdout at command boundaries. Once the shell notifies the user of a
job's completion, it is no longer `new' (or the `next', whichever
mnemonic you prefer), and `wait -n' will not return it.

So if you include the complete output of an interactive shell executing
the above list (with the obvious corrections already discussed), you'll
get a series of

32 messages about job creation
32 notifications of job exits, with exit status (they exit immediately,
after all)
32 error messages because the user has already been notified of the
job's completion and the shell has reaped it (so it is no longer the
`next' job to exit)

When the shell is not interactive, it only prints job notifications to
stdout when the job is terminated by a signal, so the jobs are available
to `wait -n'.

There was quite a long discussion about this aspect of `wait -n' behavior
a couple of months ago, starting at

https://lists.gnu.org/archive/html/bug-bash/2024-01/msg00104.html


And the script from directory 'two'? How do I pass this string through the 
CFLAGS variable to the compiler? What am I doing wrong in #3 of the script?


You're not taking word splitting into account. It seems like you want
selective word splitting -- for the embedded double quotes in the various
elements of CFLAGS to have some semantic meaning and somehow inhibit
word splitting. Quotes are special to the parser, not when they are in
the results of word expansion. If you want the parser to evaluate the
results of word expansion, you're going to have to run it through the
parser again, using something like `eval'.

It's clearer if you replace `gcc' in your command with

printf '<%s> '

and `./main' with `echo'.


--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote:
> On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:
> > The pid can be obtained with the -p option, as of 5.1. Below is a
> > synthetic example of how it might be put into practice.

I'd forgotten about that one.  A recent addition, and one I've never used
yet.

> > #!/bin/bash
> >
> > declare -A job_by status_by
> > max_jobs=4
> > jobs=0
> >
> > wait_next() {
> > local pid
> > wait -n -p pid
> > status_by[$pid]=$?
> >
> 
> How exactly is this indexing implemented internally? Does a first number on
> index n take m bits (using a linked list) or does it take n * m bits (using
> realloc(3))?

"declare -A" makes it an associative array (hash table).

Without that declare -A, it would be a sparsely indexed array.

In either case, it doesn't just blindly allocate millions of bytes of
memory.  It uses something slimmer.

Re: multi-threaded compiling

[Mischa Baars]

On Mon, 11 Mar 2024, 21:08 Kerin Millar,  wrote:

> On Mon, 11 Mar 2024 15:36:48 -0400
> Greg Wooledge  wrote:
>
> > > On Mon, Mar 11, 2024, 20:13 Mischa Baars  >
> > > wrote:
> > >
> > > > Also I don't think that gives you an exit status for each 'exit $i'
> > > > started. I need that exit status.
> >
> > "wait -n" without a PID won't help you, then.  You don't get the PID or
> > job ID that terminated, and you don't get the exit status.  It's only
>
> It does convey the exit status.
>
> > of interest if you're trying to do something like "run these 100 jobs,
> > 5 at a time" without storing their exit statuses.
>
> The pid can be obtained with the -p option, as of 5.1. Below is a
> synthetic example of how it might be put into practice.
>
> #!/bin/bash
>
> declare -A job_by status_by
> max_jobs=4
> jobs=0
>
> wait_next() {
> local pid
> wait -n -p pid
> status_by[$pid]=$?
>

How exactly is this indexing implemented internally? Does a first number on
index n take m bits (using a linked list) or does it take n * m bits (using
realloc(3))?

unset -v 'job_by[$pid]'
> }
>
> worker() {
> sleep "$(( RANDOM % 5 ))"
> exit "$(( RANDOM % 2 ))"
> }
>
> for (( i = 0; i < 16; ++i )); do
> (( jobs++ < max_jobs )) || wait_next
> worker & job_by[$!]=
> done
>
> while (( ${#job_by[@]} )); do
> wait_next
> done
>
> declare -p status_by
>
> --
> Kerin Millar
>
>

Re: multi-threaded compiling

[Mischa Baars]

Like this:

seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done; sleep ${seconds};
for (( i=0;i<32;i++ )); do wait -np pid; e=${?}; echo "$(printf %3u ${i})
pid ${pid} exit ${e}"; done;

which again only works when called from a script. I have been looking for
the -p option. Guess I overlooked it.

Now you would have to look up the pid in the pid array of started 'exit
${i}''s to lookup the log file corresponding to that index. I suppose
waiting for the processes to end in the same order as they were started
isn't that bad a solution. You then have that index right at hand.

On Mon, Mar 11, 2024 at 8:49 PM Chet Ramey  wrote:

> On 3/11/24 2:46 PM, Mischa Baars wrote:
> > You mean:
>
> > for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> > wait -n; echo $?; done;
> >
> > Because this doesn't and to be honest, I needed the pid and its index to
> > retrieve gcc's output from a log file array afterwards.
>
> If you want the pid, take a look at the -p option to wait.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Kerin Millar]

On Mon, 11 Mar 2024 15:36:48 -0400
Greg Wooledge  wrote:

> > On Mon, Mar 11, 2024, 20:13 Mischa Baars 
> > wrote:
> > 
> > > Also I don't think that gives you an exit status for each 'exit $i'
> > > started. I need that exit status.
> 
> "wait -n" without a PID won't help you, then.  You don't get the PID or
> job ID that terminated, and you don't get the exit status.  It's only

It does convey the exit status.

> of interest if you're trying to do something like "run these 100 jobs,
> 5 at a time" without storing their exit statuses.

The pid can be obtained with the -p option, as of 5.1. Below is a synthetic 
example of how it might be put into practice.

#!/bin/bash

declare -A job_by status_by
max_jobs=4
jobs=0

wait_next() {
local pid
wait -n -p pid
status_by[$pid]=$?
unset -v 'job_by[$pid]'
}

worker() {
sleep "$(( RANDOM % 5 ))"
exit "$(( RANDOM % 2 ))"
}

for (( i = 0; i < 16; ++i )); do
(( jobs++ < max_jobs )) || wait_next
worker & job_by[$!]=
done

while (( ${#job_by[@]} )); do
wait_next
done

declare -p status_by

-- 
Kerin Millar

Re: multi-threaded compiling

[alex xmb sw ratchev]

bash s wait returns the exit code of the proc from the job

On Mon, Mar 11, 2024, 20:52 Mischa Baars 
wrote:

> On Mon, 11 Mar 2024, 20:36 Greg Wooledge,  wrote:
>
> > > On Mon, Mar 11, 2024, 20:13 Mischa Baars  >
> > > wrote:
> > >
> > > > Also I don't think that gives you an exit status for each 'exit $i'
> > > > started. I need that exit status.
> >
> > "wait -n" without a PID won't help you, then.  You don't get the PID or
> > job ID that terminated, and you don't get the exit status.  It's only
> > of interest if you're trying to do something like "run these 100 jobs,
> > 5 at a time" without storing their exit statuses.
> >
> > If you need to reap each individual job and store its exit status indexed
> > by its PID, then you're probably going to need something like:
> >
> >
> > #!/bin/bash
> > i=0 pid=() status=()
> > for job in ...; do
> > longrunner "$job" & pid[i++]=$!
> > done
> >
> > for ((i=0; i < ${#pid[@]}; i++)); do
> > wait "${pid[i]}"; status[i]=$#
> > done
> >
> >
> > You won't be able to take advantage of "wait -n"'s ability to react
> > to the first job that finishes.  You'll end up reaping each job in the
> > order they started, not the order they finished.
> >
>
> Exactly my thought, but does that really matter and do we even have another
> option?
>
> The glibc wait(2) returns a pid to that purpose. This bash wait doesn't.
>
> >
>

Re: multi-threaded compiling

[Mischa Baars]

On Mon, 11 Mar 2024, 20:36 Greg Wooledge,  wrote:

> > On Mon, Mar 11, 2024, 20:13 Mischa Baars 
> > wrote:
> >
> > > Also I don't think that gives you an exit status for each 'exit $i'
> > > started. I need that exit status.
>
> "wait -n" without a PID won't help you, then.  You don't get the PID or
> job ID that terminated, and you don't get the exit status.  It's only
> of interest if you're trying to do something like "run these 100 jobs,
> 5 at a time" without storing their exit statuses.
>
> If you need to reap each individual job and store its exit status indexed
> by its PID, then you're probably going to need something like:
>
>
> #!/bin/bash
> i=0 pid=() status=()
> for job in ...; do
> longrunner "$job" & pid[i++]=$!
> done
>
> for ((i=0; i < ${#pid[@]}; i++)); do
> wait "${pid[i]}"; status[i]=$#
> done
>
>
> You won't be able to take advantage of "wait -n"'s ability to react
> to the first job that finishes.  You'll end up reaping each job in the
> order they started, not the order they finished.
>

Exactly my thought, but does that really matter and do we even have another
option?

The glibc wait(2) returns a pid to that purpose. This bash wait doesn't.

>

Re: multi-threaded compiling

[Chet Ramey]

On 3/11/24 2:46 PM, Mischa Baars wrote:

You mean:



for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
wait -n; echo $?; done;

Because this doesn't and to be honest, I needed the pid and its index to
retrieve gcc's output from a log file array afterwards.


If you want the pid, take a look at the -p option to wait.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: multi-threaded compiling

[Mischa Baars]

On Mon, 11 Mar 2024, 20:20 Chet Ramey,  wrote:

> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>

Wasn't expecting any better answer, but thanks anyway.

And the script from directory 'two'? How do I pass this string through the
CFLAGS variable to the compiler? What am I doing wrong in #3 of the script?


> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>  ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>

Re: multi-threaded compiling

[Paul Smith]

On Mon, 2024-03-11 at 15:36 -0400, Greg Wooledge wrote:
> You won't be able to take advantage of "wait -n"'s ability to react
> to the first job that finishes.  You'll end up reaping each job in
> the order they started, not the order they finished.

It feels to me like you're trying to reproduce make's already-available
parallel build support, in a kind of kludgy way with bash.

Maybe this is an XY problem and you should step back and describe WHY
you're trying to create this behavior using bash instead of just using
make's support for parallel jobs.

What about make's parallel jobs doesn't meet your requirements (or put
another way, what actually are your requirements)?

Re: multi-threaded compiling

[Greg Wooledge]

> On Mon, Mar 11, 2024, 20:13 Mischa Baars 
> wrote:
> 
> > Also I don't think that gives you an exit status for each 'exit $i'
> > started. I need that exit status.

"wait -n" without a PID won't help you, then.  You don't get the PID or
job ID that terminated, and you don't get the exit status.  It's only
of interest if you're trying to do something like "run these 100 jobs,
5 at a time" without storing their exit statuses.

If you need to reap each individual job and store its exit status indexed
by its PID, then you're probably going to need something like:


#!/bin/bash
i=0 pid=() status=()
for job in ...; do
longrunner "$job" & pid[i++]=$!
done

for ((i=0; i < ${#pid[@]}; i++)); do
wait "${pid[i]}"; status[i]=$#
done


You won't be able to take advantage of "wait -n"'s ability to react
to the first job that finishes.  You'll end up reaping each job in the
order they started, not the order they finished.

Re: multi-threaded compiling

[alex xmb sw ratchev]

afik wait will exit with that $? of returned

On Mon, Mar 11, 2024, 20:13 Mischa Baars 
wrote:

> That's not really an answer to the question.
>
> Also I don't think that gives you an exit status for each 'exit $i'
> started. I need that exit status.
>
>
> On Mon, 11 Mar 2024, 20:03 alex xmb sw ratchev,  wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev 
>> wrote:
>>
>>> the logic between my code
>>>
>>> 1 threads_max
>>> 2 loop
>>> 3 inside loop , do if run is > than threads_max then wait -n one
>>> then 4 spawn thread
>>>
>>
>> 3 if run isnt more than max , simply ignore and spawn thread in next cmd
>>
>> i dont get ur points
>>>
>>> On Mon, Mar 11, 2024, 19:55 Mischa Baars 
>>> wrote:
>>>
 Sorry. I mean:

 for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ ));
 do wait -n; echo $?; done;

 doesn't function. With an ampersand instead of a semicolon. Why does it
 function when called from a script and why does it fail when called from
 the command line?

 On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars <
 mjbaars1977.bac...@gmail.com> wrote:

> You mean:
>
> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>
> with one command and one wait in a single loop. And this does execute
> on the command line. How interesting!
>
> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ ));
> do wait -n; echo $?; done;
>
> Because this doesn't and to be honest, I needed the pid and its index
> to retrieve gcc's output from a log file array afterwards.
>
> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 19:22 Mischa Baars <
>> mjbaars1977.bac...@gmail.com> wrote:
>>
>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev <
>>> fxmb...@gmail.com> wrote:
>>>
 i also completly dont get ur issue

 f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=

   while [[ -v f[++i] ]] ; do
  (( ++r > threads )) &&
 wait -n
 gcc -c "${f[i]}" &
   done

>>>
>>> How nice!
>>>
>>> wait -n exit 1 & echo $?
>>>
>>
>> doesnt need a pid
>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>
>> You got me the solution :) Except that wait expects a pid after -n.
>>>
>>> Maybe
>>>
>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>
>>> is what you meant? The equivalence of sequential execution?
>>>
>>> First think, then do magic.
>>>
>>>


>>>

 On Mon, Mar 11, 2024, 18:16 Mischa Baars <
 mjbaars1977.bac...@gmail.com> wrote:

> Hello Paul,
>
> It seems I'm awake a little longer than you are.
>
> The second paragraph as you see it, belongs to 1)
> The fourth paragraph as you see it, belongs to 2)
>
> The actual command invocations (a Makefile, a make.sh script) can
> be found
> in the attachment, as indicated on the first line of the mail. In
> the
> attachment there are two directories, one and two, belonging to 1)
> and 2)
> respectively.
>
> I'm not into Vulcan mindmelds, so I hope everything from the first
> mail
> makes sense to you and everyone on this mailing list now.
>
> Best regards,
> Mischa Baars.
>
> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>
> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> > > I've attached the problem. It consists of two parts:
> > >
> > > 1) multi-threaded bash script and / or multi-threaded Makefile
> > >
> > > Running bash script functions as expected, but executing the
> same
> > > line of code with make and / or the command line, does not
> function.
> > > Perhaps someone could explain to me why?
> > >
> > > 2) passing a string argument from a bash script and / or
> Makefile to
> > > the gcc -D option
> > >
> > > Running the makefile functions as expected, but I have not
> been able
> > > to get similar code to work from a bash script. Can someone
> please
> > > explain to me what I'm doing wrong?
> >
> > I don't understand the problem.  In the third paragraph above
> you say
> > the bash script works as expected and the makefile doesn't work,
> but in
> > the last paragraph you say that the makefile works as expected
> but you
> > can't get it to work in bash.
> >
> > Please provide actual command invocations (cut and pasted)
> showing the
> > output you received and 

Re: multi-threaded compiling

[Chet Ramey]

On 3/11/24 2:50 PM, Mischa Baars wrote:

Which sort of brings us back to the original question I suppose. Who does
that line of code function from a script and why does it fail from the
command line? 


Job control and when the shell notifies the user about job completion,
most likely, two of the relevant things that differ between interactive
and non-interactive shells.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: multi-threaded compiling

[Mischa Baars]

That's not really an answer to the question.

Also I don't think that gives you an exit status for each 'exit $i'
started. I need that exit status.


On Mon, 11 Mar 2024, 20:03 alex xmb sw ratchev,  wrote:

>
>
> On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev  wrote:
>
>> the logic between my code
>>
>> 1 threads_max
>> 2 loop
>> 3 inside loop , do if run is > than threads_max then wait -n one
>> then 4 spawn thread
>>
>
> 3 if run isnt more than max , simply ignore and spawn thread in next cmd
>
> i dont get ur points
>>
>> On Mon, Mar 11, 2024, 19:55 Mischa Baars 
>> wrote:
>>
>>> Sorry. I mean:
>>>
>>> for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ ));
>>> do wait -n; echo $?; done;
>>>
>>> doesn't function. With an ampersand instead of a semicolon. Why does it
>>> function when called from a script and why does it fail when called from
>>> the command line?
>>>
>>> On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars <
>>> mjbaars1977.bac...@gmail.com> wrote:
>>>
 You mean:

 for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;

 with one command and one wait in a single loop. And this does execute
 on the command line. How interesting!

 for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ ));
 do wait -n; echo $?; done;

 Because this doesn't and to be honest, I needed the pid and its index
 to retrieve gcc's output from a log file array afterwards.

 On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
 wrote:

>
>
> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
> wrote:
>
>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev <
>> fxmb...@gmail.com> wrote:
>>
>>> i also completly dont get ur issue
>>>
>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>
>>>   while [[ -v f[++i] ]] ; do
>>>  (( ++r > threads )) &&
>>> wait -n
>>> gcc -c "${f[i]}" &
>>>   done
>>>
>>
>> How nice!
>>
>> wait -n exit 1 & echo $?
>>
>
> doesnt need a pid
> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>
> You got me the solution :) Except that wait expects a pid after -n.
>>
>> Maybe
>>
>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>
>> is what you meant? The equivalence of sequential execution?
>>
>> First think, then do magic.
>>
>>
>>>
>>>
>>
>>>
>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars <
>>> mjbaars1977.bac...@gmail.com> wrote:
>>>
 Hello Paul,

 It seems I'm awake a little longer than you are.

 The second paragraph as you see it, belongs to 1)
 The fourth paragraph as you see it, belongs to 2)

 The actual command invocations (a Makefile, a make.sh script) can
 be found
 in the attachment, as indicated on the first line of the mail. In
 the
 attachment there are two directories, one and two, belonging to 1)
 and 2)
 respectively.

 I'm not into Vulcan mindmelds, so I hope everything from the first
 mail
 makes sense to you and everyone on this mailing list now.

 Best regards,
 Mischa Baars.

 On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:

 > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
 > > I've attached the problem. It consists of two parts:
 > >
 > > 1) multi-threaded bash script and / or multi-threaded Makefile
 > >
 > > Running bash script functions as expected, but executing the
 same
 > > line of code with make and / or the command line, does not
 function.
 > > Perhaps someone could explain to me why?
 > >
 > > 2) passing a string argument from a bash script and / or
 Makefile to
 > > the gcc -D option
 > >
 > > Running the makefile functions as expected, but I have not been
 able
 > > to get similar code to work from a bash script. Can someone
 please
 > > explain to me what I'm doing wrong?
 >
 > I don't understand the problem.  In the third paragraph above you
 say
 > the bash script works as expected and the makefile doesn't work,
 but in
 > the last paragraph you say that the makefile works as expected
 but you
 > can't get it to work in bash.
 >
 > Please provide actual command invocations (cut and pasted)
 showing the
 > output you received and explaining exactly what is wrong with it.
 >
 > But before you do that, be aware that make does NOT invoke
 /bin/bash as
 > its shell.  It invokes /bin/sh.  On some systems /bin/sh is
 actually an
 > alias for bash.  On other systems it isn't.

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev  wrote:

> the logic between my code
>
> 1 threads_max
> 2 loop
> 3 inside loop , do if run is > than threads_max then wait -n one
> then 4 spawn thread
>

3 if run isnt more than max , simply ignore and spawn thread in next cmd

i dont get ur points
>
> On Mon, Mar 11, 2024, 19:55 Mischa Baars 
> wrote:
>
>> Sorry. I mean:
>>
>> for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); do
>> wait -n; echo $?; done;
>>
>> doesn't function. With an ampersand instead of a semicolon. Why does it
>> function when called from a script and why does it fail when called from
>> the command line?
>>
>> On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars <
>> mjbaars1977.bac...@gmail.com> wrote:
>>
>>> You mean:
>>>
>>> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>>>
>>> with one command and one wait in a single loop. And this does execute on
>>> the command line. How interesting!
>>>
>>> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
>>> wait -n; echo $?; done;
>>>
>>> Because this doesn't and to be honest, I needed the pid and its index to
>>> retrieve gcc's output from a log file array afterwards.
>>>
>>> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
>>> wrote:
>>>


 On Mon, Mar 11, 2024, 19:22 Mischa Baars 
 wrote:

> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
> wrote:
>
>> i also completly dont get ur issue
>>
>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>
>>   while [[ -v f[++i] ]] ; do
>>  (( ++r > threads )) &&
>> wait -n
>> gcc -c "${f[i]}" &
>>   done
>>
>
> How nice!
>
> wait -n exit 1 & echo $?
>

 doesnt need a pid
 1 : 1 as i wrote it , excepts add 'wait' as new last line

 You got me the solution :) Except that wait expects a pid after -n.
>
> Maybe
>
> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>
> is what you meant? The equivalence of sequential execution?
>
> First think, then do magic.
>
>
>>
>>
>
>>
>> On Mon, Mar 11, 2024, 18:16 Mischa Baars <
>> mjbaars1977.bac...@gmail.com> wrote:
>>
>>> Hello Paul,
>>>
>>> It seems I'm awake a little longer than you are.
>>>
>>> The second paragraph as you see it, belongs to 1)
>>> The fourth paragraph as you see it, belongs to 2)
>>>
>>> The actual command invocations (a Makefile, a make.sh script) can be
>>> found
>>> in the attachment, as indicated on the first line of the mail. In the
>>> attachment there are two directories, one and two, belonging to 1)
>>> and 2)
>>> respectively.
>>>
>>> I'm not into Vulcan mindmelds, so I hope everything from the first
>>> mail
>>> makes sense to you and everyone on this mailing list now.
>>>
>>> Best regards,
>>> Mischa Baars.
>>>
>>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>>>
>>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>>> > > I've attached the problem. It consists of two parts:
>>> > >
>>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>>> > >
>>> > > Running bash script functions as expected, but executing the same
>>> > > line of code with make and / or the command line, does not
>>> function.
>>> > > Perhaps someone could explain to me why?
>>> > >
>>> > > 2) passing a string argument from a bash script and / or
>>> Makefile to
>>> > > the gcc -D option
>>> > >
>>> > > Running the makefile functions as expected, but I have not been
>>> able
>>> > > to get similar code to work from a bash script. Can someone
>>> please
>>> > > explain to me what I'm doing wrong?
>>> >
>>> > I don't understand the problem.  In the third paragraph above you
>>> say
>>> > the bash script works as expected and the makefile doesn't work,
>>> but in
>>> > the last paragraph you say that the makefile works as expected but
>>> you
>>> > can't get it to work in bash.
>>> >
>>> > Please provide actual command invocations (cut and pasted) showing
>>> the
>>> > output you received and explaining exactly what is wrong with it.
>>> >
>>> > But before you do that, be aware that make does NOT invoke
>>> /bin/bash as
>>> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is
>>> actually an
>>> > alias for bash.  On other systems it isn't.
>>> >
>>> > If you want your makefile to always use bash as its shell, you
>>> should
>>> > add an explicit:
>>> >
>>> > SHELL := /bin/bash
>>> >
>>> > to your makefile to force it.  Maybe that will solve your
>>> problem.  If
>>> > not we'll need details such as I mention above.
>>> >
>>>
>>

Re: multi-threaded compiling

[alex xmb sw ratchev]

the logic between my code

1 threads_max
2 loop
3 inside loop , do if run is > than threads_max then wait -n one
then 4 spawn thread

i dont get ur points

On Mon, Mar 11, 2024, 19:55 Mischa Baars 
wrote:

> Sorry. I mean:
>
> for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); do
> wait -n; echo $?; done;
>
> doesn't function. With an ampersand instead of a semicolon. Why does it
> function when called from a script and why does it fail when called from
> the command line?
>
> On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars 
> wrote:
>
>> You mean:
>>
>> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>>
>> with one command and one wait in a single loop. And this does execute on
>> the command line. How interesting!
>>
>> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
>> wait -n; echo $?; done;
>>
>> Because this doesn't and to be honest, I needed the pid and its index to
>> retrieve gcc's output from a log file array afterwards.
>>
>> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
>> wrote:
>>
>>>
>>>
>>> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
>>> wrote:
>>>
 On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
 wrote:

> i also completly dont get ur issue
>
> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>
>   while [[ -v f[++i] ]] ; do
>  (( ++r > threads )) &&
> wait -n
> gcc -c "${f[i]}" &
>   done
>

 How nice!

 wait -n exit 1 & echo $?

>>>
>>> doesnt need a pid
>>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>>
>>> You got me the solution :) Except that wait expects a pid after -n.

 Maybe

 for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;

 is what you meant? The equivalence of sequential execution?

 First think, then do magic.


>
>

>
> On Mon, Mar 11, 2024, 18:16 Mischa Baars 
> wrote:
>
>> Hello Paul,
>>
>> It seems I'm awake a little longer than you are.
>>
>> The second paragraph as you see it, belongs to 1)
>> The fourth paragraph as you see it, belongs to 2)
>>
>> The actual command invocations (a Makefile, a make.sh script) can be
>> found
>> in the attachment, as indicated on the first line of the mail. In the
>> attachment there are two directories, one and two, belonging to 1)
>> and 2)
>> respectively.
>>
>> I'm not into Vulcan mindmelds, so I hope everything from the first
>> mail
>> makes sense to you and everyone on this mailing list now.
>>
>> Best regards,
>> Mischa Baars.
>>
>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>>
>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>> > > I've attached the problem. It consists of two parts:
>> > >
>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>> > >
>> > > Running bash script functions as expected, but executing the same
>> > > line of code with make and / or the command line, does not
>> function.
>> > > Perhaps someone could explain to me why?
>> > >
>> > > 2) passing a string argument from a bash script and / or Makefile
>> to
>> > > the gcc -D option
>> > >
>> > > Running the makefile functions as expected, but I have not been
>> able
>> > > to get similar code to work from a bash script. Can someone please
>> > > explain to me what I'm doing wrong?
>> >
>> > I don't understand the problem.  In the third paragraph above you
>> say
>> > the bash script works as expected and the makefile doesn't work,
>> but in
>> > the last paragraph you say that the makefile works as expected but
>> you
>> > can't get it to work in bash.
>> >
>> > Please provide actual command invocations (cut and pasted) showing
>> the
>> > output you received and explaining exactly what is wrong with it.
>> >
>> > But before you do that, be aware that make does NOT invoke
>> /bin/bash as
>> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is
>> actually an
>> > alias for bash.  On other systems it isn't.
>> >
>> > If you want your makefile to always use bash as its shell, you
>> should
>> > add an explicit:
>> >
>> > SHELL := /bin/bash
>> >
>> > to your makefile to force it.  Maybe that will solve your problem.
>> If
>> > not we'll need details such as I mention above.
>> >
>>
>

Re: multi-threaded compiling

[Mischa Baars]

Sorry. I mean:

for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); do
wait -n; echo $?; done;

doesn't function. With an ampersand instead of a semicolon. Why does it
function when called from a script and why does it fail when called from
the command line?

On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars 
wrote:

> You mean:
>
> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>
> with one command and one wait in a single loop. And this does execute on
> the command line. How interesting!
>
> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> wait -n; echo $?; done;
>
> Because this doesn't and to be honest, I needed the pid and its index to
> retrieve gcc's output from a log file array afterwards.
>
> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
>> wrote:
>>
>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
>>> wrote:
>>>
 i also completly dont get ur issue

 f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=

   while [[ -v f[++i] ]] ; do
  (( ++r > threads )) &&
 wait -n
 gcc -c "${f[i]}" &
   done

>>>
>>> How nice!
>>>
>>> wait -n exit 1 & echo $?
>>>
>>
>> doesnt need a pid
>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>
>> You got me the solution :) Except that wait expects a pid after -n.
>>>
>>> Maybe
>>>
>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>
>>> is what you meant? The equivalence of sequential execution?
>>>
>>> First think, then do magic.
>>>
>>>


>>>

 On Mon, Mar 11, 2024, 18:16 Mischa Baars 
 wrote:

> Hello Paul,
>
> It seems I'm awake a little longer than you are.
>
> The second paragraph as you see it, belongs to 1)
> The fourth paragraph as you see it, belongs to 2)
>
> The actual command invocations (a Makefile, a make.sh script) can be
> found
> in the attachment, as indicated on the first line of the mail. In the
> attachment there are two directories, one and two, belonging to 1) and
> 2)
> respectively.
>
> I'm not into Vulcan mindmelds, so I hope everything from the first mail
> makes sense to you and everyone on this mailing list now.
>
> Best regards,
> Mischa Baars.
>
> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>
> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> > > I've attached the problem. It consists of two parts:
> > >
> > > 1) multi-threaded bash script and / or multi-threaded Makefile
> > >
> > > Running bash script functions as expected, but executing the same
> > > line of code with make and / or the command line, does not
> function.
> > > Perhaps someone could explain to me why?
> > >
> > > 2) passing a string argument from a bash script and / or Makefile
> to
> > > the gcc -D option
> > >
> > > Running the makefile functions as expected, but I have not been
> able
> > > to get similar code to work from a bash script. Can someone please
> > > explain to me what I'm doing wrong?
> >
> > I don't understand the problem.  In the third paragraph above you say
> > the bash script works as expected and the makefile doesn't work, but
> in
> > the last paragraph you say that the makefile works as expected but
> you
> > can't get it to work in bash.
> >
> > Please provide actual command invocations (cut and pasted) showing
> the
> > output you received and explaining exactly what is wrong with it.
> >
> > But before you do that, be aware that make does NOT invoke /bin/bash
> as
> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually
> an
> > alias for bash.  On other systems it isn't.
> >
> > If you want your makefile to always use bash as its shell, you should
> > add an explicit:
> >
> > SHELL := /bin/bash
> >
> > to your makefile to force it.  Maybe that will solve your problem.
> If
> > not we'll need details such as I mention above.
> >
>

Re: multi-threaded compiling

[Mischa Baars]

Which sort of brings us back to the original question I suppose. Who does
that line of code function from a script and why does it fail from the
command line? My guess was that the same thing makes this line fail from
the Makefile.

On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars 
wrote:

> You mean:
>
> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>
> with one command and one wait in a single loop. And this does execute on
> the command line. How interesting!
>
> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> wait -n; echo $?; done;
>
> Because this doesn't and to be honest, I needed the pid and its index to
> retrieve gcc's output from a log file array afterwards.
>
> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
>> wrote:
>>
>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
>>> wrote:
>>>
 i also completly dont get ur issue

 f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=

   while [[ -v f[++i] ]] ; do
  (( ++r > threads )) &&
 wait -n
 gcc -c "${f[i]}" &
   done

>>>
>>> How nice!
>>>
>>> wait -n exit 1 & echo $?
>>>
>>
>> doesnt need a pid
>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>
>> You got me the solution :) Except that wait expects a pid after -n.
>>>
>>> Maybe
>>>
>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>
>>> is what you meant? The equivalence of sequential execution?
>>>
>>> First think, then do magic.
>>>
>>>


>>>

 On Mon, Mar 11, 2024, 18:16 Mischa Baars 
 wrote:

> Hello Paul,
>
> It seems I'm awake a little longer than you are.
>
> The second paragraph as you see it, belongs to 1)
> The fourth paragraph as you see it, belongs to 2)
>
> The actual command invocations (a Makefile, a make.sh script) can be
> found
> in the attachment, as indicated on the first line of the mail. In the
> attachment there are two directories, one and two, belonging to 1) and
> 2)
> respectively.
>
> I'm not into Vulcan mindmelds, so I hope everything from the first mail
> makes sense to you and everyone on this mailing list now.
>
> Best regards,
> Mischa Baars.
>
> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>
> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> > > I've attached the problem. It consists of two parts:
> > >
> > > 1) multi-threaded bash script and / or multi-threaded Makefile
> > >
> > > Running bash script functions as expected, but executing the same
> > > line of code with make and / or the command line, does not
> function.
> > > Perhaps someone could explain to me why?
> > >
> > > 2) passing a string argument from a bash script and / or Makefile
> to
> > > the gcc -D option
> > >
> > > Running the makefile functions as expected, but I have not been
> able
> > > to get similar code to work from a bash script. Can someone please
> > > explain to me what I'm doing wrong?
> >
> > I don't understand the problem.  In the third paragraph above you say
> > the bash script works as expected and the makefile doesn't work, but
> in
> > the last paragraph you say that the makefile works as expected but
> you
> > can't get it to work in bash.
> >
> > Please provide actual command invocations (cut and pasted) showing
> the
> > output you received and explaining exactly what is wrong with it.
> >
> > But before you do that, be aware that make does NOT invoke /bin/bash
> as
> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually
> an
> > alias for bash.  On other systems it isn't.
> >
> > If you want your makefile to always use bash as its shell, you should
> > add an explicit:
> >
> > SHELL := /bin/bash
> >
> > to your makefile to force it.  Maybe that will solve your problem.
> If
> > not we'll need details such as I mention above.
> >
>

Re: multi-threaded compiling

[Mischa Baars]

You mean:

for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;

with one command and one wait in a single loop. And this does execute on
the command line. How interesting!

for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
wait -n; echo $?; done;

Because this doesn't and to be honest, I needed the pid and its index to
retrieve gcc's output from a log file array afterwards.

On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev 
wrote:

>
>
> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
> wrote:
>
>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
>> wrote:
>>
>>> i also completly dont get ur issue
>>>
>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>
>>>   while [[ -v f[++i] ]] ; do
>>>  (( ++r > threads )) &&
>>> wait -n
>>> gcc -c "${f[i]}" &
>>>   done
>>>
>>
>> How nice!
>>
>> wait -n exit 1 & echo $?
>>
>
> doesnt need a pid
> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>
> You got me the solution :) Except that wait expects a pid after -n.
>>
>> Maybe
>>
>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>
>> is what you meant? The equivalence of sequential execution?
>>
>> First think, then do magic.
>>
>>
>>>
>>>
>>
>>>
>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars 
>>> wrote:
>>>
 Hello Paul,

 It seems I'm awake a little longer than you are.

 The second paragraph as you see it, belongs to 1)
 The fourth paragraph as you see it, belongs to 2)

 The actual command invocations (a Makefile, a make.sh script) can be
 found
 in the attachment, as indicated on the first line of the mail. In the
 attachment there are two directories, one and two, belonging to 1) and
 2)
 respectively.

 I'm not into Vulcan mindmelds, so I hope everything from the first mail
 makes sense to you and everyone on this mailing list now.

 Best regards,
 Mischa Baars.

 On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:

 > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
 > > I've attached the problem. It consists of two parts:
 > >
 > > 1) multi-threaded bash script and / or multi-threaded Makefile
 > >
 > > Running bash script functions as expected, but executing the same
 > > line of code with make and / or the command line, does not function.
 > > Perhaps someone could explain to me why?
 > >
 > > 2) passing a string argument from a bash script and / or Makefile to
 > > the gcc -D option
 > >
 > > Running the makefile functions as expected, but I have not been able
 > > to get similar code to work from a bash script. Can someone please
 > > explain to me what I'm doing wrong?
 >
 > I don't understand the problem.  In the third paragraph above you say
 > the bash script works as expected and the makefile doesn't work, but
 in
 > the last paragraph you say that the makefile works as expected but you
 > can't get it to work in bash.
 >
 > Please provide actual command invocations (cut and pasted) showing the
 > output you received and explaining exactly what is wrong with it.
 >
 > But before you do that, be aware that make does NOT invoke /bin/bash
 as
 > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually
 an
 > alias for bash.  On other systems it isn't.
 >
 > If you want your makefile to always use bash as its shell, you should
 > add an explicit:
 >
 > SHELL := /bin/bash
 >
 > to your makefile to force it.  Maybe that will solve your problem.  If
 > not we'll need details such as I mention above.
 >

>>>

Re: multi-threaded compiling

[Paul Smith]

On Mon, 2024-03-11 at 19:37 +0100, alex xmb sw ratchev wrote:
> /data/data/com.termux/files/usr/bin/sh: 1: Syntax error: Bad for loop
> variable

This is because of the issue I mentioned in my initial reply.  This
invocation is using /bin/sh which is a POSIX shell but is not bash;
probably it's dash or similar (you are running on Debian or one of the
GNU/Linux distros that build on it).

If you set "SHELL := /bin/bash" (or whatever the path is for bash on
your system) in the makefile you won't see this, I expect.

Re: multi-threaded compiling

[alex xmb sw ratchev]

other output

~/2024031100 - gnu questions/one $ make
SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!};
done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]};
e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
/data/data/com.termux/files/usr/bin/sh: 1: Syntax error: Bad for loop
variable
make: *** [Makefile:3: all] Error 2
~/2024031100 - gnu questions/one $ cd ..
~/2024031100 - gnu questions $ cd two
~/2024031100 - gnu questions/two $ make
STRINGIZED: 0, STRING: one and two
STRINGIZED: 0, STRING: one and two
STRINGIZED: 1, STRING: one and two
STRINGIZED: 1, STRING: one and two
STRINGIZED: 0, STRING: one and two
STRINGIZED: 0, STRING: one and two
STRINGIZED: 1, STRING: one and two
STRINGIZED: 1, STRING: one and two
STRINGIZED: 0, STRING: one and two
STRINGIZED: 0, STRING: one and two
STRINGIZED: 1, STRING: one and two
STRINGIZED: 1, STRING: one and two

On Mon, Mar 11, 2024, 18:54 Mischa Baars 
wrote:

> Oh, ok. I have executed the examples for you now.
>
> What I expected was also in the first mail:
>
> 1) The one.make.sh.log has the expected output. The same commands executed
> from a Makefile (one.make.log) do not behave as expected. This also holds
> for this command executed from the command line.
> 2) The two.make.log has the expected. The same commands executed from the
> bash script do not behave as expected. I'm having trouble obtaining the
> desired results with the bash script.
>
> Regards,
> Mischa Baars.
>
> On Mon, Mar 11, 2024 at 6:27 PM Paul Smith  wrote:
>
> > On Mon, 2024-03-11 at 18:14 +0100, Mischa Baars wrote:
> > > The actual command invocations (a Makefile, a make.sh script) can be
> > > found in the attachment, as indicated on the first line of the mail.
> > > In the attachment there are two directories, one and two, belonging
> > > to 1) and 2) respectively.
> >
> > That's not what I asked: I asked YOU to run the commands on YOUR system
> > and cut and paste the results of the run into your email message, with
> > an explicit explanation of exactly what is wrong with the results and
> > what you wanted to see instead.  What errors did you get?  Or what was
> > unexpected about the behavior?
> >
> > I'm not interested in running random scripts on my system that were
> > downloaded from an email, or examining their behavior to make sure
> > they're OK, or setting up a container to run them in, or guessing what
> > you wanted to happen, etc.
> >
>
> A container? These are no viri, these are scripts :)
>
>
> > Or maybe someone else will want to do that work instead.
> >
> > But before putting in more effort on the "misbehaving" makefile, read
> > this part of my response carefully:
> >
> > > But before you do that, be aware that make does NOT invoke /bin/bash
> > > as its shell.  It invokes /bin/sh.  On some systems /bin/sh is
> > > actually an alias for bash.  On other systems it isn't.
> > >
> > > If you want your makefile to always use bash as its shell, you should
> > > add an explicit:
> > >
> > > SHELL := /bin/bash
> > >
> > > to your makefile to force it.  Maybe that will solve your problem.
> > > If not we'll need details such as I mention above.
> >
> > and verify in your response you added the suggested line to your
> > makefile and it didn't help (in the situation where the makefile was
> > behaving in an unexpected way).
> >
>
> No. Doesn't help the tiniest bit.
>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 19:25 alex xmb sw ratchev  wrote:

>
>
> On Mon, Mar 11, 2024, 19:22 Mischa Baars 
> wrote:
>
>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
>> wrote:
>>
>>> i also completly dont get ur issue
>>>
>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>
>>>   while [[ -v f[++i] ]] ; do
>>>  (( ++r > threads )) &&
>>> wait -n
>>> gcc -c "${f[i]}" &
>>>   done
>>>
>>
>> How nice!
>>
>> wait -n exit 1 & echo $?
>>
>
> doesnt need a pid
> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>
> You got me the solution :) Except that wait expects a pid after -n.
>>
>> Maybe
>>
>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>
>
maxt=32
for (( .. )) ; do
(( i > maxt )) && wait -n
"$process_here" &
done

wait

eof

is what you meant? The equivalence of sequential execution?
>>
>> First think, then do magic.
>>
>>
>>>
>>>
>>
>>>
>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars 
>>> wrote:
>>>
 Hello Paul,

 It seems I'm awake a little longer than you are.

 The second paragraph as you see it, belongs to 1)
 The fourth paragraph as you see it, belongs to 2)

 The actual command invocations (a Makefile, a make.sh script) can be
 found
 in the attachment, as indicated on the first line of the mail. In the
 attachment there are two directories, one and two, belonging to 1) and
 2)
 respectively.

 I'm not into Vulcan mindmelds, so I hope everything from the first mail
 makes sense to you and everyone on this mailing list now.

 Best regards,
 Mischa Baars.

 On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:

 > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
 > > I've attached the problem. It consists of two parts:
 > >
 > > 1) multi-threaded bash script and / or multi-threaded Makefile
 > >
 > > Running bash script functions as expected, but executing the same
 > > line of code with make and / or the command line, does not function.
 > > Perhaps someone could explain to me why?
 > >
 > > 2) passing a string argument from a bash script and / or Makefile to
 > > the gcc -D option
 > >
 > > Running the makefile functions as expected, but I have not been able
 > > to get similar code to work from a bash script. Can someone please
 > > explain to me what I'm doing wrong?
 >
 > I don't understand the problem.  In the third paragraph above you say
 > the bash script works as expected and the makefile doesn't work, but
 in
 > the last paragraph you say that the makefile works as expected but you
 > can't get it to work in bash.
 >
 > Please provide actual command invocations (cut and pasted) showing the
 > output you received and explaining exactly what is wrong with it.
 >
 > But before you do that, be aware that make does NOT invoke /bin/bash
 as
 > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually
 an
 > alias for bash.  On other systems it isn't.
 >
 > If you want your makefile to always use bash as its shell, you should
 > add an explicit:
 >
 > SHELL := /bin/bash
 >
 > to your makefile to force it.  Maybe that will solve your problem.  If
 > not we'll need details such as I mention above.
 >

>>>

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 07:22:39PM +0100, Mischa Baars wrote:
> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
> wrote:
> 
> > i also completly dont get ur issue
> >
> > f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
> >
> >   while [[ -v f[++i] ]] ; do
> >  (( ++r > threads )) &&
> > wait -n
> > gcc -c "${f[i]}" &
> >   done
> >
> 
> How nice!
> 
> wait -n exit 1 & echo $?
> 
> You got me the solution :) Except that wait expects a pid after -n.

No, wait -n without a PID waits for any one job to complete.

hobbit:~$ sleep 123 & sleep 234 & sleep 3 & wait -n
[1] 513337
[2] 513338
[3] 513339
[3]+  Donesleep 3
hobbit:~$ ps
PID TTY  TIME CMD
   1138 pts/000:00:00 bash
 513337 pts/000:00:00 sleep
 513338 pts/000:00:00 sleep
 513341 pts/000:00:00 ps
hobbit:~$ 

Re: multi-threaded compiling

[Mischa Baars]

On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge  wrote:

> On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!};
> done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]};
> e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> > /bin/bash: line 1: wait: 1747087: no such job
> >   0 pid 1747087 exit 127
> > /bin/bash: line 1: wait: 1747088: no such job
> >   1 pid 1747088 exit 127
>
> Without analyzing this in depth, one thing struck me immediately:
> you're using the reserved variable SECONDS, which has special semantics
> in bash.  ${SECONDS} is going to expand to an ever-increasing number,
> beginning with 5 (since that's the value you assigned), and going up
> by 1 per second that the script runs.  I'm assuming that was not your
> intention.
>

It wasn't. I modified the script and makefile accordingly.


>
> In general, any variable whose name is all capital letters *may* have
> special meaning to the shell and should not be used for general storage
> purposes in scripts.
>

Completely right again. I will definitely use these recommendations next
time I write a script or a makefile.

Re: multi-threaded compiling

[Mischa Baars]

Hi Greg,

Good point. One for you :)

Cheerz,
Mischa.

On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge  wrote:

> On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!};
> done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]};
> e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> > /bin/bash: line 1: wait: 1747087: no such job
> >   0 pid 1747087 exit 127
> > /bin/bash: line 1: wait: 1747088: no such job
> >   1 pid 1747088 exit 127
>
> Without analyzing this in depth, one thing struck me immediately:
> you're using the reserved variable SECONDS, which has special semantics
> in bash.  ${SECONDS} is going to expand to an ever-increasing number,
> beginning with 5 (since that's the value you assigned), and going up
> by 1 per second that the script runs.  I'm assuming that was not your
> intention.
>
> In general, any variable whose name is all capital letters *may* have
> special meaning to the shell and should not be used for general storage
> purposes in scripts.
>

Re: multi-threaded compiling

[alex xmb sw ratchev]

On Mon, Mar 11, 2024, 19:22 Mischa Baars 
wrote:

> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
> wrote:
>
>> i also completly dont get ur issue
>>
>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>
>>   while [[ -v f[++i] ]] ; do
>>  (( ++r > threads )) &&
>> wait -n
>> gcc -c "${f[i]}" &
>>   done
>>
>
> How nice!
>
> wait -n exit 1 & echo $?
>

doesnt need a pid
1 : 1 as i wrote it , excepts add 'wait' as new last line

You got me the solution :) Except that wait expects a pid after -n.
>
> Maybe
>
> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>
> is what you meant? The equivalence of sequential execution?
>
> First think, then do magic.
>
>
>>
>>
>
>>
>> On Mon, Mar 11, 2024, 18:16 Mischa Baars 
>> wrote:
>>
>>> Hello Paul,
>>>
>>> It seems I'm awake a little longer than you are.
>>>
>>> The second paragraph as you see it, belongs to 1)
>>> The fourth paragraph as you see it, belongs to 2)
>>>
>>> The actual command invocations (a Makefile, a make.sh script) can be
>>> found
>>> in the attachment, as indicated on the first line of the mail. In the
>>> attachment there are two directories, one and two, belonging to 1) and 2)
>>> respectively.
>>>
>>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>>> makes sense to you and everyone on this mailing list now.
>>>
>>> Best regards,
>>> Mischa Baars.
>>>
>>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>>>
>>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>>> > > I've attached the problem. It consists of two parts:
>>> > >
>>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>>> > >
>>> > > Running bash script functions as expected, but executing the same
>>> > > line of code with make and / or the command line, does not function.
>>> > > Perhaps someone could explain to me why?
>>> > >
>>> > > 2) passing a string argument from a bash script and / or Makefile to
>>> > > the gcc -D option
>>> > >
>>> > > Running the makefile functions as expected, but I have not been able
>>> > > to get similar code to work from a bash script. Can someone please
>>> > > explain to me what I'm doing wrong?
>>> >
>>> > I don't understand the problem.  In the third paragraph above you say
>>> > the bash script works as expected and the makefile doesn't work, but in
>>> > the last paragraph you say that the makefile works as expected but you
>>> > can't get it to work in bash.
>>> >
>>> > Please provide actual command invocations (cut and pasted) showing the
>>> > output you received and explaining exactly what is wrong with it.
>>> >
>>> > But before you do that, be aware that make does NOT invoke /bin/bash as
>>> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually an
>>> > alias for bash.  On other systems it isn't.
>>> >
>>> > If you want your makefile to always use bash as its shell, you should
>>> > add an explicit:
>>> >
>>> > SHELL := /bin/bash
>>> >
>>> > to your makefile to force it.  Maybe that will solve your problem.  If
>>> > not we'll need details such as I mention above.
>>> >
>>>
>>

Re: multi-threaded compiling

[Mischa Baars]

On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev 
wrote:

> i also completly dont get ur issue
>
> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>
>   while [[ -v f[++i] ]] ; do
>  (( ++r > threads )) &&
> wait -n
> gcc -c "${f[i]}" &
>   done
>

How nice!

wait -n exit 1 & echo $?

You got me the solution :) Except that wait expects a pid after -n.

Maybe

for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;

is what you meant? The equivalence of sequential execution?

First think, then do magic.


>
>

>
> On Mon, Mar 11, 2024, 18:16 Mischa Baars 
> wrote:
>
>> Hello Paul,
>>
>> It seems I'm awake a little longer than you are.
>>
>> The second paragraph as you see it, belongs to 1)
>> The fourth paragraph as you see it, belongs to 2)
>>
>> The actual command invocations (a Makefile, a make.sh script) can be found
>> in the attachment, as indicated on the first line of the mail. In the
>> attachment there are two directories, one and two, belonging to 1) and 2)
>> respectively.
>>
>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>> makes sense to you and everyone on this mailing list now.
>>
>> Best regards,
>> Mischa Baars.
>>
>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>>
>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>> > > I've attached the problem. It consists of two parts:
>> > >
>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>> > >
>> > > Running bash script functions as expected, but executing the same
>> > > line of code with make and / or the command line, does not function.
>> > > Perhaps someone could explain to me why?
>> > >
>> > > 2) passing a string argument from a bash script and / or Makefile to
>> > > the gcc -D option
>> > >
>> > > Running the makefile functions as expected, but I have not been able
>> > > to get similar code to work from a bash script. Can someone please
>> > > explain to me what I'm doing wrong?
>> >
>> > I don't understand the problem.  In the third paragraph above you say
>> > the bash script works as expected and the makefile doesn't work, but in
>> > the last paragraph you say that the makefile works as expected but you
>> > can't get it to work in bash.
>> >
>> > Please provide actual command invocations (cut and pasted) showing the
>> > output you received and explaining exactly what is wrong with it.
>> >
>> > But before you do that, be aware that make does NOT invoke /bin/bash as
>> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually an
>> > alias for bash.  On other systems it isn't.
>> >
>> > If you want your makefile to always use bash as its shell, you should
>> > add an explicit:
>> >
>> > SHELL := /bin/bash
>> >
>> > to your makefile to force it.  Maybe that will solve your problem.  If
>> > not we'll need details such as I mention above.
>> >
>>
>

Re: multi-threaded compiling

[Greg Wooledge]

On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; 
> sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; 
> echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> /bin/bash: line 1: wait: 1747087: no such job
>   0 pid 1747087 exit 127
> /bin/bash: line 1: wait: 1747088: no such job
>   1 pid 1747088 exit 127

Without analyzing this in depth, one thing struck me immediately:
you're using the reserved variable SECONDS, which has special semantics
in bash.  ${SECONDS} is going to expand to an ever-increasing number,
beginning with 5 (since that's the value you assigned), and going up
by 1 per second that the script runs.  I'm assuming that was not your
intention.

In general, any variable whose name is all capital letters *may* have
special meaning to the shell and should not be used for general storage
purposes in scripts.

Re: multi-threaded compiling

[Mischa Baars]

Oh, ok. I have executed the examples for you now.

What I expected was also in the first mail:

1) The one.make.sh.log has the expected output. The same commands executed
from a Makefile (one.make.log) do not behave as expected. This also holds
for this command executed from the command line.
2) The two.make.log has the expected. The same commands executed from the
bash script do not behave as expected. I'm having trouble obtaining the
desired results with the bash script.

Regards,
Mischa Baars.

On Mon, Mar 11, 2024 at 6:27 PM Paul Smith  wrote:

> On Mon, 2024-03-11 at 18:14 +0100, Mischa Baars wrote:
> > The actual command invocations (a Makefile, a make.sh script) can be
> > found in the attachment, as indicated on the first line of the mail.
> > In the attachment there are two directories, one and two, belonging
> > to 1) and 2) respectively.
>
> That's not what I asked: I asked YOU to run the commands on YOUR system
> and cut and paste the results of the run into your email message, with
> an explicit explanation of exactly what is wrong with the results and
> what you wanted to see instead.  What errors did you get?  Or what was
> unexpected about the behavior?
>
> I'm not interested in running random scripts on my system that were
> downloaded from an email, or examining their behavior to make sure
> they're OK, or setting up a container to run them in, or guessing what
> you wanted to happen, etc.
>

A container? These are no viri, these are scripts :)


> Or maybe someone else will want to do that work instead.
>
> But before putting in more effort on the "misbehaving" makefile, read
> this part of my response carefully:
>
> > But before you do that, be aware that make does NOT invoke /bin/bash
> > as its shell.  It invokes /bin/sh.  On some systems /bin/sh is
> > actually an alias for bash.  On other systems it isn't.
> >
> > If you want your makefile to always use bash as its shell, you should
> > add an explicit:
> >
> > SHELL := /bin/bash
> >
> > to your makefile to force it.  Maybe that will solve your problem.
> > If not we'll need details such as I mention above.
>
> and verify in your response you added the suggested line to your
> makefile and it didn't help (in the situation where the makefile was
> behaving in an unexpected way).
>

No. Doesn't help the tiniest bit.
SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
/bin/bash: line 1: wait: 1747087: no such job
  0 pid 1747087 exit 127
/bin/bash: line 1: wait: 1747088: no such job
  1 pid 1747088 exit 127
/bin/bash: line 1: wait: 1747089: no such job
  2 pid 1747089 exit 127
/bin/bash: line 1: wait: 1747090: no such job
  3 pid 1747090 exit 127
/bin/bash: line 1: wait: 1747091: no such job
  4 pid 1747091 exit 127
/bin/bash: line 1: wait: 1747092: no such job
  5 pid 1747092 exit 127
/bin/bash: line 1: wait: 1747093: no such job
  6 pid 1747093 exit 127
/bin/bash: line 1: wait: 1747094: no such job
  7 pid 1747094 exit 127
/bin/bash: line 1: wait: 1747095: no such job
  8 pid 1747095 exit 127
/bin/bash: line 1: wait: 1747096: no such job
  9 pid 1747096 exit 127
/bin/bash: line 1: wait: 1747097: no such job
 10 pid 1747097 exit 127
/bin/bash: line 1: wait: 1747098: no such job
 11 pid 1747098 exit 127
/bin/bash: line 1: wait: 1747099: no such job
 12 pid 1747099 exit 127
/bin/bash: line 1: wait: 1747100: no such job
 13 pid 1747100 exit 127
/bin/bash: line 1: wait: 1747101: no such job
 14 pid 1747101 exit 127
/bin/bash: line 1: wait: 1747102: no such job
 15 pid 1747102 exit 127
/bin/bash: line 1: wait: 1747103: no such job
 16 pid 1747103 exit 127
/bin/bash: line 1: wait: 1747104: no such job
 17 pid 1747104 exit 127
/bin/bash: line 1: wait: 1747105: no such job
 18 pid 1747105 exit 127
/bin/bash: line 1: wait: 1747106: no such job
 19 pid 1747106 exit 127
/bin/bash: line 1: wait: 1747107: no such job
 20 pid 1747107 exit 127
/bin/bash: line 1: wait: 1747108: no such job
 21 pid 1747108 exit 127
/bin/bash: line 1: wait: 1747109: no such job
 22 pid 1747109 exit 127
/bin/bash: line 1: wait: 1747110: no such job
 23 pid 1747110 exit 127
/bin/bash: line 1: wait: 1747111: no such job
 24 pid 1747111 exit 127
/bin/bash: line 1: wait: 1747112: no such job
 25 pid 1747112 exit 127
/bin/bash: line 1: wait: 1747113: no such job
 26 pid 1747113 exit 127
/bin/bash: line 1: wait: 1747114: no such job
 27 pid 1747114 exit 127
/bin/bash: line 1: wait: 1747115: no such job
 28 pid 1747115 exit 127
/bin/bash: line 1: wait: 1747116: no such job
 29 pid 1747116 exit 127
/bin/bash: line 1: wait: 1747117: no such job
 30 pid 1747117 exit 127
/bin/bash: line 1: wait: 1747118: no such job
 31 pid 1747118 exit 127
  0 pid 1746688 exit 0
  1 pid 1746689 exit 1
  2 pid 1746690 exit 2
  3 pid 1746691 exit 3
  4 pid 1746692 exit 4
  5 pid 1746693 exit 5
  6 pid 1746694 exit 6
  7 pid 1746695 exit 

Re: multi-threaded compiling

[Paul Smith]

On Mon, 2024-03-11 at 18:14 +0100, Mischa Baars wrote:
> The actual command invocations (a Makefile, a make.sh script) can be
> found in the attachment, as indicated on the first line of the mail.
> In the attachment there are two directories, one and two, belonging
> to 1) and 2) respectively.

That's not what I asked: I asked YOU to run the commands on YOUR system
and cut and paste the results of the run into your email message, with
an explicit explanation of exactly what is wrong with the results and
what you wanted to see instead.  What errors did you get?  Or what was
unexpected about the behavior?

I'm not interested in running random scripts on my system that were
downloaded from an email, or examining their behavior to make sure
they're OK, or setting up a container to run them in, or guessing what
you wanted to happen, etc.

Or maybe someone else will want to do that work instead.

But before putting in more effort on the "misbehaving" makefile, read
this part of my response carefully:

> But before you do that, be aware that make does NOT invoke /bin/bash
> as its shell.  It invokes /bin/sh.  On some systems /bin/sh is
> actually an alias for bash.  On other systems it isn't.
> 
> If you want your makefile to always use bash as its shell, you should
> add an explicit:
> 
> SHELL := /bin/bash
> 
> to your makefile to force it.  Maybe that will solve your problem. 
> If not we'll need details such as I mention above.

and verify in your response you added the suggested line to your
makefile and it didn't help (in the situation where the makefile was
behaving in an unexpected way).

Re: multi-threaded compiling

[alex xmb sw ratchev]

i also completly dont get ur issue

f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=

  while [[ -v f[++i] ]] ; do
 (( ++r > threads )) &&
wait -n
gcc -c "${f[i]}" &
  done

On Mon, Mar 11, 2024, 18:16 Mischa Baars 
wrote:

> Hello Paul,
>
> It seems I'm awake a little longer than you are.
>
> The second paragraph as you see it, belongs to 1)
> The fourth paragraph as you see it, belongs to 2)
>
> The actual command invocations (a Makefile, a make.sh script) can be found
> in the attachment, as indicated on the first line of the mail. In the
> attachment there are two directories, one and two, belonging to 1) and 2)
> respectively.
>
> I'm not into Vulcan mindmelds, so I hope everything from the first mail
> makes sense to you and everyone on this mailing list now.
>
> Best regards,
> Mischa Baars.
>
> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:
>
> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> > > I've attached the problem. It consists of two parts:
> > >
> > > 1) multi-threaded bash script and / or multi-threaded Makefile
> > >
> > > Running bash script functions as expected, but executing the same
> > > line of code with make and / or the command line, does not function.
> > > Perhaps someone could explain to me why?
> > >
> > > 2) passing a string argument from a bash script and / or Makefile to
> > > the gcc -D option
> > >
> > > Running the makefile functions as expected, but I have not been able
> > > to get similar code to work from a bash script. Can someone please
> > > explain to me what I'm doing wrong?
> >
> > I don't understand the problem.  In the third paragraph above you say
> > the bash script works as expected and the makefile doesn't work, but in
> > the last paragraph you say that the makefile works as expected but you
> > can't get it to work in bash.
> >
> > Please provide actual command invocations (cut and pasted) showing the
> > output you received and explaining exactly what is wrong with it.
> >
> > But before you do that, be aware that make does NOT invoke /bin/bash as
> > its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually an
> > alias for bash.  On other systems it isn't.
> >
> > If you want your makefile to always use bash as its shell, you should
> > add an explicit:
> >
> > SHELL := /bin/bash
> >
> > to your makefile to force it.  Maybe that will solve your problem.  If
> > not we'll need details such as I mention above.
> >
>

Re: multi-threaded compiling

[Mischa Baars]

Hello Paul,

It seems I'm awake a little longer than you are.

The second paragraph as you see it, belongs to 1)
The fourth paragraph as you see it, belongs to 2)

The actual command invocations (a Makefile, a make.sh script) can be found
in the attachment, as indicated on the first line of the mail. In the
attachment there are two directories, one and two, belonging to 1) and 2)
respectively.

I'm not into Vulcan mindmelds, so I hope everything from the first mail
makes sense to you and everyone on this mailing list now.

Best regards,
Mischa Baars.

On Mon, Mar 11, 2024 at 5:01 PM Paul Smith  wrote:

> On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> > I've attached the problem. It consists of two parts:
> >
> > 1) multi-threaded bash script and / or multi-threaded Makefile
> >
> > Running bash script functions as expected, but executing the same
> > line of code with make and / or the command line, does not function.
> > Perhaps someone could explain to me why?
> >
> > 2) passing a string argument from a bash script and / or Makefile to
> > the gcc -D option
> >
> > Running the makefile functions as expected, but I have not been able
> > to get similar code to work from a bash script. Can someone please
> > explain to me what I'm doing wrong?
>
> I don't understand the problem.  In the third paragraph above you say
> the bash script works as expected and the makefile doesn't work, but in
> the last paragraph you say that the makefile works as expected but you
> can't get it to work in bash.
>
> Please provide actual command invocations (cut and pasted) showing the
> output you received and explaining exactly what is wrong with it.
>
> But before you do that, be aware that make does NOT invoke /bin/bash as
> its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually an
> alias for bash.  On other systems it isn't.
>
> If you want your makefile to always use bash as its shell, you should
> add an explicit:
>
> SHELL := /bin/bash
>
> to your makefile to force it.  Maybe that will solve your problem.  If
> not we'll need details such as I mention above.
>

Re: multi-threaded compiling

[Paul Smith]

On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
> I've attached the problem. It consists of two parts:
> 
> 1) multi-threaded bash script and / or multi-threaded Makefile
> 
> Running bash script functions as expected, but executing the same
> line of code with make and / or the command line, does not function.
> Perhaps someone could explain to me why?
> 
> 2) passing a string argument from a bash script and / or Makefile to
> the gcc -D option
> 
> Running the makefile functions as expected, but I have not been able
> to get similar code to work from a bash script. Can someone please
> explain to me what I'm doing wrong?

I don't understand the problem.  In the third paragraph above you say
the bash script works as expected and the makefile doesn't work, but in
the last paragraph you say that the makefile works as expected but you
can't get it to work in bash.

Please provide actual command invocations (cut and pasted) showing the
output you received and explaining exactly what is wrong with it.

But before you do that, be aware that make does NOT invoke /bin/bash as
its shell.  It invokes /bin/sh.  On some systems /bin/sh is actually an
alias for bash.  On other systems it isn't.

If you want your makefile to always use bash as its shell, you should
add an explicit:

SHELL := /bin/bash

to your makefile to force it.  Maybe that will solve your problem.  If
not we'll need details such as I mention above.

multi-threaded compiling

[Mischa Baars]

Hi,

I'd like to compile a project of mine using multiple logical cores.

I've attached the problem. It consists of two parts:

1) multi-threaded bash script and / or multi-threaded Makefile

Running bash script functions as expected, but executing the same line of
code with make and / or the command line, does not function. Perhaps
someone could explain to me why?

2) passing a string argument from a bash script and / or Makefile to the
gcc -D option

Running the makefile functions as expected, but I have not been able to get
similar code to work from a bash script. Can someone please explain to me
what I'm doing wrong?

Hope to hear from you soon.

Best regards,
Mischa Baars.


2024031100 - gnu questions.tar.xz
Description: application/xz