Re: multi-threaded compiling
No? No further suggestions?
Then I'd like to thank you all for your help and your input.
I'll be off-list now.
Kind regards,
Mischa Baars.
On Thu, Mar 14, 2024 at 9:17 AM Mischa Baars
wrote:
> Ok. Then this is what I was able to make of it. To be honest, I prefer the
> Makefile syntax over the script syntax, because of its direct substitutions.
>
> Other improvements that anyone likes to suggest? Is this the best bash can
> do?
>
>
> On Thu, Mar 14, 2024 at 3:47 AM Martin D Kealey
> wrote:
>
>> On Wed, 13 Mar 2024, Mischa Baars wrote:
>>
>> > Date: Wed, 13 Mar 2024 22:52:16
>> > From: Mischa Baars
>> > To: alex xmb sw ratchev
>> > Cc: psm...@gnu.org, bug-bash , help-m...@gnu.org
>> > Subject: Re: multi-threaded compiling
>> >
>> > I found another mistake of mine:
>> >
>> > #ifndef __STRINGIZED__
>> >
>> > should be
>> >
>> > #if __STRINGIZED__ == 0
>> >
>> > or it will always go for the second statement in the conditional.
>> >
>> > Can someone please correct my script??? Things are starting to itch.
>>
>> The fundamental limitation is that Bash only provides lists of scalars
>> masquerading as arrays, and lists of scalar-scalar pairs masquerading as
>> maps (aka hashes, dicts, or "associative arrays").
>>
>> It does not support any more complex data structures, and in particular,
>> does not have arrays-of-arrays or lists-of-lists.
>>
>> So I would change
>>
>> CFLAGS[0]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[0]}\""
>> CFLAGS[1]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[1]}\""
>> CFLAGS[2]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[0]}\""
>> CFLAGS[3]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[1]}\""
>>
>> to
>>
>> CFLAGS0=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}" )
>> CFLAGS1=( -D__STRINGIZED__=0 -D__STRING__="${STR[1]}" )
>> CFLAGS2=( -D__STRINGIZED__=1 -D__STRING__="${STR[0]}" )
>> CFLAGS3=( -D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>>
>> and then change
>>
>> gcc -o main main.c ${CFLAGS[0]} ; ./main
>> gcc -o main main.c ${CFLAGS[1]} ; ./main
>> gcc -o main main.c ${CFLAGS[2]} ; ./main
>> gcc -o main main.c ${CFLAGS[3]} ; ./main
>>
>> to
>>
>> gcc -o main main.c "${CFLAGS0[@]}" && ./main
>> gcc -o main main.c "${CFLAGS1[@]}" && ./main
>> gcc -o main main.c "${CFLAGS2[@]}" && ./main
>> gcc -o main main.c "${CFLAGS3[@]}" && ./main
>>
>> If you absolutely must have some kind one indexed array for all the
>> permutations of cflags, then you could use a synthetic multidimensional
>> array, like this:
>>
>> CFLAGS=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}"
>>-D__STRINGIZED__=0 -D__STRING__="${STR[1]}"
>>-D__STRINGIZED__=1 -D__STRING__="${STR[0]}"
>>-D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>>
>> gcc -o main main.c "${CFLAGS[@]:0*2:2}" && ./main
>> gcc -o main main.c "${CFLAGS[@]:1*2:2}" && ./main
>> gcc -o main main.c "${CFLAGS[@]:2*2:2}" && ./main
>> gcc -o main main.c "${CFLAGS[@]:3*2:2}" && ./main
>>
>> However, given the pattern, I'd have gone with:
>>
>> for i in 0 1 2 3 ; do
>> gcc -o main main.c -D__STRINGIZED__=$((i/2))
>> -D__STRING__="${STR[i%2]}" &&
>> ./main
>> done
>>
>> -Martin
>>
>
Re: multi-threaded compiling
Ok. Then this is what I was able to make of it. To be honest, I prefer the
Makefile syntax over the script syntax, because of its direct substitutions.
Other improvements that anyone likes to suggest? Is this the best bash can
do?
On Thu, Mar 14, 2024 at 3:47 AM Martin D Kealey
wrote:
> On Wed, 13 Mar 2024, Mischa Baars wrote:
>
> > Date: Wed, 13 Mar 2024 22:52:16
> > From: Mischa Baars
> > To: alex xmb sw ratchev
> > Cc: psm...@gnu.org, bug-bash , help-m...@gnu.org
> > Subject: Re: multi-threaded compiling
> >
> > I found another mistake of mine:
> >
> > #ifndef __STRINGIZED__
> >
> > should be
> >
> > #if __STRINGIZED__ == 0
> >
> > or it will always go for the second statement in the conditional.
> >
> > Can someone please correct my script??? Things are starting to itch.
>
> The fundamental limitation is that Bash only provides lists of scalars
> masquerading as arrays, and lists of scalar-scalar pairs masquerading as
> maps (aka hashes, dicts, or "associative arrays").
>
> It does not support any more complex data structures, and in particular,
> does not have arrays-of-arrays or lists-of-lists.
>
> So I would change
>
> CFLAGS[0]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[0]}\""
> CFLAGS[1]="-D__STRINGIZED__=0 -D__STRING__=\"${STR[1]}\""
> CFLAGS[2]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[0]}\""
> CFLAGS[3]="-D__STRINGIZED__=1 -D__STRING__=\"${STR[1]}\""
>
> to
>
> CFLAGS0=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}" )
> CFLAGS1=( -D__STRINGIZED__=0 -D__STRING__="${STR[1]}" )
> CFLAGS2=( -D__STRINGIZED__=1 -D__STRING__="${STR[0]}" )
> CFLAGS3=( -D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>
> and then change
>
> gcc -o main main.c ${CFLAGS[0]} ; ./main
> gcc -o main main.c ${CFLAGS[1]} ; ./main
> gcc -o main main.c ${CFLAGS[2]} ; ./main
> gcc -o main main.c ${CFLAGS[3]} ; ./main
>
> to
>
> gcc -o main main.c "${CFLAGS0[@]}" && ./main
> gcc -o main main.c "${CFLAGS1[@]}" && ./main
> gcc -o main main.c "${CFLAGS2[@]}" && ./main
> gcc -o main main.c "${CFLAGS3[@]}" && ./main
>
> If you absolutely must have some kind one indexed array for all the
> permutations of cflags, then you could use a synthetic multidimensional
> array, like this:
>
> CFLAGS=( -D__STRINGIZED__=0 -D__STRING__="${STR[0]}"
>-D__STRINGIZED__=0 -D__STRING__="${STR[1]}"
>-D__STRINGIZED__=1 -D__STRING__="${STR[0]}"
>-D__STRINGIZED__=1 -D__STRING__="${STR[1]}" )
>
> gcc -o main main.c "${CFLAGS[@]:0*2:2}" && ./main
> gcc -o main main.c "${CFLAGS[@]:1*2:2}" && ./main
> gcc -o main main.c "${CFLAGS[@]:2*2:2}" && ./main
> gcc -o main main.c "${CFLAGS[@]:3*2:2}" && ./main
>
> However, given the pattern, I'd have gone with:
>
> for i in 0 1 2 3 ; do
> gcc -o main main.c -D__STRINGIZED__=$((i/2))
> -D__STRING__="${STR[i%2]}" &&
> ./main
> done
>
> -Martin
>
make.sh
Description: application/shellscript
/* * * *
* :set tabstop=9
*/
#include
#define str(x) lit(x)
#define lit(x) #x
#if __STRINGIZED__ == 0
const char* s = __STRING__ ;
#else
const char* s = str(__STRING__);
#endif
int main ()
{
printf ("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}
Makefile
Description: Binary data
Re: multi-threaded compiling
I found another mistake of mine:
#ifndef __STRINGIZED__
should be
#if __STRINGIZED__ == 0
or it will always go for the second statement in the conditional.
Can someone please correct my script??? Things are starting to itch.
On Wed, Mar 13, 2024 at 8:44 AM alex xmb sw ratchev
wrote:
>
>
> On Wed, Mar 13, 2024, 08:26 Mischa Baars
> wrote:
>
>> On Tue, Mar 12, 2024 at 10:00 PM Paul Smith wrote:
>>
>> > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote:
>> > > > I'd still like to hear why you aren't simply using "make -j".
>> > >
>> > > That's because I don't want to define static compile and link targets
>> > > for every new project I start. The Makefile in question contains only
>> > > a few lines of code and some environment variables, but works on
>> > > every new project as long as you follow certain guidelines regarding
>> > > the directory structure. It scans, compiles and links on the fly.
>> >
>> > I don't find this argument compelling. It's relatively straightforward
>> > to create a makefile environment that "works on every new project as
>> > long as you follow certain guidelines regarding the directory
>> > structure" while still using parallel builds, that will "scan, compile,
>> > and link" on the fly, using things like the wildcard function, pattern
>> > rules, etc.
>> >
>> > You are merely trading a bit of extra complexity in your makefile for a
>> > whole lot of complexity and tricky hacking in bash, as can be seen by
>> > following this thread.
>> >
>>
>> Only the Makefile is functional right now. The bash script is not working.
>> Good enough to reduce compile time from 1:43 to 0:10. I would have liked
>> to
>> see the script working.
>>
>
> u said u have fs naming conventions
> write em up here , i make the bash ( not sh )
>
> But if you prefer to re-invent make's parallel build capabilities in
>> > bash, certainly that's your prerogative.
>> >
>>
>
Makefile
Description: Binary data
/* * * *
* :set tabstop=9
*/
#include
#define str(x) lit(x)
#define lit(x) #x
int main ()
{
#ifndef __STRINGIZED__
const char* s = __STRING__ ;
#else
const char* s = str(__STRING__);
#endif
printf ("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}
Re: multi-threaded compiling
On Wed, Mar 13, 2024 at 4:14 AM Martin D Kealey wrote: > > On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote: >> > On 3/11/24 2:50 PM, Mischa Baars wrote: >> > > Which sort of brings us back to the original question I suppose. Who >> does >> > > that line of code function from a script and why does it fail from the >> > > command line? >> > >> > Job control and when the shell notifies the user about job completion, >> > most likely, two of the relevant things that differ between interactive >> > and non-interactive shells. >> > > In this case, no. > > I inserted « echo $- ; shopt -p ; shopt -po ; » in front of each case, and > the ONLY difference was that « echo $- » reported “hxBc” vs “hxB”. Not an > “m” in sight. And no “i” or “l” either. (The “c” was expected, given how « > make » invokes the shell.) > > -Martin >
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 10:00 PM Paul Smith wrote: > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote: > > > I'd still like to hear why you aren't simply using "make -j". > > > > That's because I don't want to define static compile and link targets > > for every new project I start. The Makefile in question contains only > > a few lines of code and some environment variables, but works on > > every new project as long as you follow certain guidelines regarding > > the directory structure. It scans, compiles and links on the fly. > > I don't find this argument compelling. It's relatively straightforward > to create a makefile environment that "works on every new project as > long as you follow certain guidelines regarding the directory > structure" while still using parallel builds, that will "scan, compile, > and link" on the fly, using things like the wildcard function, pattern > rules, etc. > > You are merely trading a bit of extra complexity in your makefile for a > whole lot of complexity and tricky hacking in bash, as can be seen by > following this thread. > Only the Makefile is functional right now. The bash script is not working. Good enough to reduce compile time from 1:43 to 0:10. I would have liked to see the script working. But if you prefer to re-invent make's parallel build capabilities in > bash, certainly that's your prerogative. >
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 12:49 PM Greg Wooledge wrote:
> On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote:
> > Here's the script and the Makefile using "printf '<%s>'":
>
> Sadly, your mail user agent chose to attach "Makefile" with content-type
> application/octet-stream, which my MUA (mutt) refuses to show inline,
> or to include in a reply as quoted text.
>
> Here's the top part of it, inline:
>
>
> STR[0]="one and two"
> STR[1]=one\ and\ two
>
> CFLAGS[0]=-D__STRINGIZED__=0 -D__STRING__=${STR[0]}
> CFLAGS[1]=-D__STRINGIZED__=0 -D__STRING__=${STR[1]}
> CFLAGS[2]=-D__STRINGIZED__=1 -D__STRING__=${STR[0]}
> CFLAGS[3]=-D__STRINGIZED__=1 -D__STRING__=${STR[1]}
>
>
> And here's what Martin said about it:
>
> > On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey >
> > wrote:
> > > In section two, the problem is that quote removal is done BEFORE
> variables
> > > are expanded, even though it prevents word splitting from being done
> AFTER
> > > variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> > > absolutely cannot do what you might expect; the embedded quote marks
> will
> > > be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> > > resulting string will be split on any embedded whitespace..
>
>
> As someone said yesterday, you will need eval for this.
>
>
> hobbit:/tmp/x$ cat Makefile
> FOO=-D__x="one two three"
>
> all:
> @bash -c 'eval '\''CFLAGS=${FOO}'\''; declare -p CFLAGS'
> hobbit:/tmp/x$ make
> declare -- CFLAGS="-D__x=one two three"
>
>
> Of course, using eval presents its own set of challenges, so proceed
> with extreme caution.
>
Indeed it does :) The Makefile was already working :) It is the script that
won't.
>
> I'd still like to hear why you aren't simply using "make -j".
>
That's because I don't want to define static compile and link targets for
every new project I start. The Makefile in question contains only a few
lines of code and some environment variables, but works on every new
project as long as you follow certain guidelines regarding the directory
structure. It scans, compiles and links on the fly.
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 12:30 PM Greg Wooledge wrote:
> On Tue, Mar 12, 2024 at 02:15:38PM +0700, Robert Elz wrote:
> > Date:Mon, 11 Mar 2024 17:25:57 -0400
> > From:Chet Ramey
> > Message-ID: <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu>
> >
> > | OK, here's the longer answer. When the shell is interactive, and job
> > | control is enabled, the shell prints job completion notifications to
> > | stdout at command boundaries.
> >
> > which is, IMO, yet another bogus bash misfeature. That should
> > happen, with the effect described, but only when PS1 is about
> > to be printed - precisely so commands like the one described
> > will work. Sigh.
> >
> > There aren't many complaints about this misfeature of bash,
> > as almost no-one writes interactive command lines where it makes
> > a difference. That doesn't mean they should not be able to.
>
> Yeah, it appears you're right. In a script, this works as expected:
>
> hobbit:~$ cat foo
> #!/bin/bash
> for i in {0..3}; do sleep 1 & done
> for i in {0..3}; do
> wait -n -p pid; e=$?
> printf 'pid %s status %s\n' "$pid" "$e"
> done
> hobbit:~$ ./foo
> pid 530359 status 0
> pid 530360 status 0
> pid 530361 status 0
> pid 530362 status 0
>
>
> But interactively, *even with bash -c and set +m*, it just fails:
>
> hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; for i in
> {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n" "$pid" "$e";
> done'
> pid 530407 status 0
> pid 530410 status 0
> pid status 127
> pid status 127
>
> Looks like a race condition, where some of the children get reaped and
> tossed away before "wait -n -p pid" has a chance to grab their status.
>
> If I stick a "sleep 2" in between the two loops, then it's even worse:
>
> hobbit:~$ bash -c 'set +m; for i in {0..3}; do sleep 1 & done; sleep 2;
> for i in {0..3}; do wait -n -p pid; e=$?; printf "pid %s status %s\n"
> "$pid" "$e"; done'pid status 127
> pid status 127
> pid status 127
> pid status 127
>
> ALL of the children are discarded before the second loop has a chance to
> catch a single one of them. * This is clearly not working as expected*.
>
Now we're talking :) To observe different output from script and from
Makefile certainly is somewhat peculiar and unexpected.
>
> Using "bash +i -c" doesn't change anything, either. Or "bash +i +m -c".
> Whatever magic it is that you get by putting the code in an actual
> script, I can't figure out how to replicate it from a prompt.
>
>
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 12:19 PM Greg Wooledge wrote:
> On Tue, Mar 12, 2024 at 10:33:36AM +0100, Mischa Baars wrote:
> > bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
> > sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
> > "$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
>
> "wait -p pid" is not correct here. In that command, pid is an output
> variable, and you have not specified any process IDs to wait for -- so
> wait is going to wait for *all* of the children to finish, and you'll
> get zero as the exit status:
>
These are the two waits I'm now using:
wait ${pid[${i}]};
wait -np pid;
Which gives the following two lines to execute either from script or from
makefile:
script:
seconds=5; for (( i=0;i<32;i++ )); do exit ${i} & pid[${i}]=${!}; done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait ${pid[${i}]}; e=${?};
echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
seconds=5; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -np pid; e=${?};
echo "$(printf %3u ${i}) pid ${pid} exit ${e}"; done; #!!! script
only !!!
makefile:
seconds=5; for (( i=0;i<32;i++ )); do exit $${i} & pid[$${i}]=$${!}; done;
sleep $${seconds}; for (( i=0;i<32;i++ )); do wait $${pid[$${i}]}; e=$${?};
echo "$$(printf %3u $${i}) pid $${pid[$${i}]} exit $${e}"; done;
seconds=5; for (( i=0;i<32;i++ )); do exit $${i} & done;
sleep $${seconds}; for (( i=0;i<32;i++ )); do wait -np pid;e=$${?};
echo "$$(printf %3u $${i}) pid $${pid} exit $${e}";done; #???
script only ???
The first executes correctly from both the script and the Makefile. The
second only executes correctly from script.
> $ help wait
> [...]
>
> Waits for each process identified by an ID, which may be a process ID
> or a
> job specification, and reports its termination status. If ID is not
> given, waits for all currently active child processes, and the return
> status is zero. If ID is a job specification, waits for all processes
> in that job's pipeline.
>
>
Re: multi-threaded compiling
Got it!!!
bash -c 'set +m; seconds=5; for (( i=0;i<32;i++ )); do exit ${i} &
pid[${i}]=${!}; done; sleep ${seconds}; for (( i=0;i<32;i++ )); do wait
${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}";
done;'
One problem solved :) The problem was exactly the -n, and had nothing to do
with interactive / non-interactive shells. Thank you for helping! Now the
Makefile works just as good as the script.
I can now use at least Makefiles for the parallel compilation. Would be
nice if we got #3 in the script working as well. Because I'm still curious
how to do the job right.
Best regards,
Mischa.
On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote:
> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
Re: multi-threaded compiling
Oh, sorry. Without the -n that is.
bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
bash -c 'set -m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
disables / enables the notifications respectively, but doesn't do anything
otherwise.
On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote:
> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
Re: multi-threaded compiling
Hi Chet,
The only thing I can find about this behavior is the set +m / set -m
option, which disables / enables the notifications, but has no effect
otherwise.
The line from the script won't execute from the command line with either
option, neither enclosed with a 'bash -c' as in:
set +m; bash -c 'seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done;
sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -np pid; e=${?}; echo
"$(printf %3u ${i}) pid ${pid} exit ${e}"; done;'
On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote:
> On 3/11/24 2:50 PM, Mischa Baars wrote:
> > Which sort of brings us back to the original question I suppose. Who does
> > that line of code function from a script and why does it fail from the
> > command line?
>
> Job control and when the shell notifies the user about job completion,
> most likely, two of the relevant things that differ between interactive
> and non-interactive shells.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
Re: multi-threaded compiling
Here's the script and the Makefile using "printf '<%s>'":
On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey
wrote:
> In section one, the problem is that "wait -n" does not do what you think
> it does. (Lots of us think this behaviour is broken, and it may be fixed in
> an upcoming version of Bash.) You don't need '-n' when you specify a PID;
> the fix is simply to remove it.
>
> In section two, the problem is that quote removal is done BEFORE variables
> are expanded, even though it prevents word splitting from being done AFTER
> variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> absolutely cannot do what you might expect; the embedded quote marks will
> be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> resulting string will be split on any embedded whitespace..
>
>
> On Mon, 11 Mar 2024 at 18:56, Mischa Baars
> wrote:
>
>> Hi,
>>
>> I'd like to compile a project of mine using multiple logical cores.
>>
>> I've attached the problem. It consists of two parts:
>>
>> 1) multi-threaded bash script and / or multi-threaded Makefile
>>
>> Running bash script functions as expected, but executing the same line of
>> code with make and / or the command line, does not function. Perhaps
>> someone could explain to me why?
>>
>> 2) passing a string argument from a bash script and / or Makefile to the
>> gcc -D option
>>
>> Running the makefile functions as expected, but I have not been able to
>> get
>> similar code to work from a bash script. Can someone please explain to me
>> what I'm doing wrong?
>>
>> Hope to hear from you soon.
>>
>> Best regards,
>> Mischa Baars.
>>
>
Makefile
Description: Binary data
make.sh
Description: application/shellscript
/* * * *
* :set tabstop=9
*/
#include
#define str(x) lit(x)
#define lit(x) #x
int main ()
{
#ifndef __STRINGIZED__
const char* s = __STRING__ ;
#else
const char* s = str(__STRING__);
#endif
printf ("STRINGIZED: %d, STRING: %s\n", __STRINGIZED__, s);
}
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey
wrote:
> In section one, the problem is that "wait -n" does not do what you think
> it does. (Lots of us think this behaviour is broken, and it may be fixed in
> an upcoming version of Bash.) You don't need '-n' when you specify a PID;
> the fix is simply to remove it.
>
Thanks. I already noticed the mistake yesterday and corrected it.
>
> In section two, the problem is that quote removal is done BEFORE variables
> are expanded, even though it prevents word splitting from being done AFTER
> variable expansion. Therefore writing VAR=" \"string 1\" \"string 2\" "
> absolutely cannot do what you might expect; the embedded quote marks will
> be used literally, and then (because ${CFLAGS[0]} is not quoted) the
> resulting string will be split on any embedded whitespace..
>
>
So.. how do I make the script produce the same output as the Makefile?
Using 'printf <%s>' as Chet said I should, does show the difference between
the two, but it doesn't fix the problem.
>
> On Mon, 11 Mar 2024 at 18:56, Mischa Baars
> wrote:
>
>> Hi,
>>
>> I'd like to compile a project of mine using multiple logical cores.
>>
>> I've attached the problem. It consists of two parts:
>>
>> 1) multi-threaded bash script and / or multi-threaded Makefile
>>
>> Running bash script functions as expected, but executing the same line of
>> code with make and / or the command line, does not function. Perhaps
>> someone could explain to me why?
>>
>> 2) passing a string argument from a bash script and / or Makefile to the
>> gcc -D option
>>
>> Running the makefile functions as expected, but I have not been able to
>> get
>> similar code to work from a bash script. Can someone please explain to me
>> what I'm doing wrong?
>>
>> Hope to hear from you soon.
>>
>> Best regards,
>> Mischa Baars.
>>
>
Re: multi-threaded compiling
On Mon, Mar 11, 2024 at 10:26 PM Chet Ramey wrote: > On 3/11/24 3:44 PM, Mischa Baars wrote: > > On Mon, 11 Mar 2024, 20:20 Chet Ramey, > <mailto:chet.ra...@case.edu>> wrote: > > > > On 3/11/24 2:50 PM, Mischa Baars wrote: > > > Which sort of brings us back to the original question I suppose. > Who > > does > > > that line of code function from a script and why does it fail > from the > > > command line? > > > > Job control and when the shell notifies the user about job > completion, > > most likely, two of the relevant things that differ between > interactive > > and non-interactive shells. > > > > > > Wasn't expecting any better answer, but thanks anyway. > > OK, here's the longer answer. When the shell is interactive, and job > control is enabled, the shell prints job completion notifications to > stdout at command boundaries. Once the shell notifies the user of a > job's completion, it is no longer `new' (or the `next', whichever > mnemonic you prefer), and `wait -n' will not return it. > > So if you include the complete output of an interactive shell executing > the above list (with the obvious corrections already discussed), you'll > get a series of > > 32 messages about job creation > 32 notifications of job exits, with exit status (they exit immediately, > after all) > 32 error messages because the user has already been notified of the > job's completion and the shell has reaped it (so it is no longer the > `next' job to exit) > > When the shell is not interactive, it only prints job notifications to > stdout when the job is terminated by a signal, so the jobs are available > to `wait -n'. > > There was quite a long discussion about this aspect of `wait -n' behavior > a couple of months ago, starting at > > https://lists.gnu.org/archive/html/bug-bash/2024-01/msg00104.html > > > > And the script from directory 'two'? How do I pass this string through > the > > CFLAGS variable to the compiler? What am I doing wrong in #3 of the > script? > > You're not taking word splitting into account. It seems like you want > selective word splitting -- for the embedded double quotes in the various > elements of CFLAGS to have some semantic meaning and somehow inhibit > word splitting. Quotes are special to the parser, not when they are in > the results of word expansion. If you want the parser to evaluate the > results of word expansion, you're going to have to run it through the > parser again, using something like `eval'. > > It's clearer if you replace `gcc' in your command with > > printf '<%s> ' > > and `./main' with `echo'. > > Yes. Now I see a difference in all the different lines. I wasn't familiar with this printf construct, but the results are clear: Using make: <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> Using ./make.sh: <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=0><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> <-o><-D__STRINGIZED__=1><-D__STRING__=one and two> and either for unquoted variable in #3 <-o><-D__STRINGIZED__=0><-D__STRING__="one> <-o><-D__STRINGIZED__=0><-D__STRING__="one> <-o><-D__STRINGIZED__=1><-D__STRING__="one> <-o><-D__STRINGIZED__=1><-D__STRING__="one> or for double quoted variable in #3 <-o><-D__STRINGIZED__=0 -D__STRING__="one and two"> <-o><-D__STRINGIZED__=0 -D__STRING__="one and two"> <-o><-D__STRINGIZED__=1 -D__STRING__="one and two"> <-o><-D__STRINGIZED__=1 -D__STRING__="one and two"> So... how do I fix it? > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > >
Re: multi-threaded compiling
On Tue, Mar 12, 2024 at 8:16 AM Robert Elz wrote: > This whole discussuon has been a total mess. > > First the original message didn't give nearly enough info. > That might have been in the attachment, but no-one should be > expecting people to do any needless work when you're asking > for (free) assistance - if you want help you need to make > it as easy as possible for people to do that. > > Second, a very common issue with people having problems, is > when they have a problem to solve, they work out how they think > they should do that, and then have difficulty making one > of the steps work. At that point they ask how to do that > thing, which might sometimes be very complex, even impossible. > If the initial problem had been stated, along with the > solution proposed, leading to the issue that's hard, then > instead of (perhaps) wasting needless time dreaming up > possible solutions to difficult problems, someone might > just point out a much simpler method to solve the original > problem, making finding a solution to the difficult issue moot. > > Second, the replies here have mostly been hopeless, with > just a couple addressing real issues. > > Many of the initial replies totally missed that the OP > said that the bash script in 'one' worked, and there > was this long thread suggesting other ways to do it. > Waste if everyone's time (even if the advice not to > use SECONDS was probably useful). > > There is however no blanket rule that vars in upper case are > reserved for the shell - posix contains a list of var > names scripts should avoid using (unless they want some > particular feature supported by a particular shell they > are using - SECONDS & RANDOM are on that list). Anything > else is (or should be) OK to use. > > The issue with 'one' was that the same code did not work on > the command line, or in a makefile. > > Chet explained the command line problem: > > Date:Mon, 11 Mar 2024 17:25:57 -0400 > From:Chet Ramey > Message-ID: <322e10a6-3808-49be-aa9d-a1d367a90...@case.edu> > > | OK, here's the longer answer. When the shell is interactive, and job > | control is enabled, the shell prints job completion notifications to > | stdout at command boundaries. > > which is, IMO, yet another bogus bash misfeature. That should > happen, with the effect described, but only when PS1 is about > to be printed - precisely so commands like the one described > will work. Sigh. > > There aren't many complaints about this misfeature of bash, > as almost no-one writes interactive command lines where it makes > a difference. That doesn't mean they should not be able to. > > > For the problem using make, it was suggested to explicity set > the SHELL to use in the makefile, which is good advice when > using shell specific features, as is being done here. > > But the real problem is almost certainly that the Makefile > is attempting to pass scripts containing '$' chars to the > shell. '$' is interpreted by make, to send a '$' to the > shell, it must be written as '$$'. > > Uhm... I think you forgot to look at the actual code in the Makefile here. Every '$' is written as a '$$', exactly as you say it should. > > For problem 'two', Greg (I think it was) diagnosed the > issue with var expansions, and field splitting - but if > that was unknown to the OP then that OP has no business > attempting to write any shell script (with any kind of > variable) - get an introductory shell programming text > and learn how it works - at least the basics. Don't > just guess or attempt to copy other scripts, you'll fail. > > kre > >
Re: multi-threaded compiling
On Mon, 11 Mar 2024, 21:08 Kerin Millar, wrote:
> On Mon, 11 Mar 2024 15:36:48 -0400
> Greg Wooledge wrote:
>
> > > On Mon, Mar 11, 2024, 20:13 Mischa Baars >
> > > wrote:
> > >
> > > > Also I don't think that gives you an exit status for each 'exit $i'
> > > > started. I need that exit status.
> >
> > "wait -n" without a PID won't help you, then. You don't get the PID or
> > job ID that terminated, and you don't get the exit status. It's only
>
> It does convey the exit status.
>
> > of interest if you're trying to do something like "run these 100 jobs,
> > 5 at a time" without storing their exit statuses.
>
> The pid can be obtained with the -p option, as of 5.1. Below is a
> synthetic example of how it might be put into practice.
>
> #!/bin/bash
>
> declare -A job_by status_by
> max_jobs=4
> jobs=0
>
> wait_next() {
> local pid
> wait -n -p pid
> status_by[$pid]=$?
>
How exactly is this indexing implemented internally? Does a first number on
index n take m bits (using a linked list) or does it take n * m bits (using
realloc(3))?
unset -v 'job_by[$pid]'
> }
>
> worker() {
> sleep "$(( RANDOM % 5 ))"
> exit "$(( RANDOM % 2 ))"
> }
>
> for (( i = 0; i < 16; ++i )); do
> (( jobs++ < max_jobs )) || wait_next
> worker & job_by[$!]=
> done
>
> while (( ${#job_by[@]} )); do
> wait_next
> done
>
> declare -p status_by
>
> --
> Kerin Millar
>
>
Re: multi-threaded compiling
Like this:
seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done; sleep ${seconds};
for (( i=0;i<32;i++ )); do wait -np pid; e=${?}; echo "$(printf %3u ${i})
pid ${pid} exit ${e}"; done;
which again only works when called from a script. I have been looking for
the -p option. Guess I overlooked it.
Now you would have to look up the pid in the pid array of started 'exit
${i}''s to lookup the log file corresponding to that index. I suppose
waiting for the processes to end in the same order as they were started
isn't that bad a solution. You then have that index right at hand.
On Mon, Mar 11, 2024 at 8:49 PM Chet Ramey wrote:
> On 3/11/24 2:46 PM, Mischa Baars wrote:
> > You mean:
>
> > for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> > wait -n; echo $?; done;
> >
> > Because this doesn't and to be honest, I needed the pid and its index to
> > retrieve gcc's output from a log file array afterwards.
>
> If you want the pid, take a look at the -p option to wait.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
>
Re: multi-threaded compiling
On Mon, 11 Mar 2024, 20:36 Greg Wooledge, wrote:
> > On Mon, Mar 11, 2024, 20:13 Mischa Baars
> > wrote:
> >
> > > Also I don't think that gives you an exit status for each 'exit $i'
> > > started. I need that exit status.
>
> "wait -n" without a PID won't help you, then. You don't get the PID or
> job ID that terminated, and you don't get the exit status. It's only
> of interest if you're trying to do something like "run these 100 jobs,
> 5 at a time" without storing their exit statuses.
>
> If you need to reap each individual job and store its exit status indexed
> by its PID, then you're probably going to need something like:
>
>
> #!/bin/bash
> i=0 pid=() status=()
> for job in ...; do
> longrunner "$job" & pid[i++]=$!
> done
>
> for ((i=0; i < ${#pid[@]}; i++)); do
> wait "${pid[i]}"; status[i]=$#
> done
>
>
> You won't be able to take advantage of "wait -n"'s ability to react
> to the first job that finishes. You'll end up reaping each job in the
> order they started, not the order they finished.
>
Exactly my thought, but does that really matter and do we even have another
option?
The glibc wait(2) returns a pid to that purpose. This bash wait doesn't.
>
Re: multi-threaded compiling
On Mon, 11 Mar 2024, 20:20 Chet Ramey, wrote: > On 3/11/24 2:50 PM, Mischa Baars wrote: > > Which sort of brings us back to the original question I suppose. Who does > > that line of code function from a script and why does it fail from the > > command line? > > Job control and when the shell notifies the user about job completion, > most likely, two of the relevant things that differ between interactive > and non-interactive shells. > Wasn't expecting any better answer, but thanks anyway. And the script from directory 'two'? How do I pass this string through the CFLAGS variable to the compiler? What am I doing wrong in #3 of the script? > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ > >
Re: multi-threaded compiling
That's not really an answer to the question.
Also I don't think that gives you an exit status for each 'exit $i'
started. I need that exit status.
On Mon, 11 Mar 2024, 20:03 alex xmb sw ratchev, wrote:
>
>
> On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev wrote:
>
>> the logic between my code
>>
>> 1 threads_max
>> 2 loop
>> 3 inside loop , do if run is > than threads_max then wait -n one
>> then 4 spawn thread
>>
>
> 3 if run isnt more than max , simply ignore and spawn thread in next cmd
>
> i dont get ur points
>>
>> On Mon, Mar 11, 2024, 19:55 Mischa Baars
>> wrote:
>>
>>> Sorry. I mean:
>>>
>>> for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ ));
>>> do wait -n; echo $?; done;
>>>
>>> doesn't function. With an ampersand instead of a semicolon. Why does it
>>> function when called from a script and why does it fail when called from
>>> the command line?
>>>
>>> On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars <
>>> mjbaars1977.bac...@gmail.com> wrote:
>>>
>>>> You mean:
>>>>
>>>> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>>>>
>>>> with one command and one wait in a single loop. And this does execute
>>>> on the command line. How interesting!
>>>>
>>>> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ ));
>>>> do wait -n; echo $?; done;
>>>>
>>>> Because this doesn't and to be honest, I needed the pid and its index
>>>> to retrieve gcc's output from a log file array afterwards.
>>>>
>>>> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 11, 2024, 19:22 Mischa Baars
>>>>> wrote:
>>>>>
>>>>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev <
>>>>>> fxmb...@gmail.com> wrote:
>>>>>>
>>>>>>> i also completly dont get ur issue
>>>>>>>
>>>>>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>>>>>
>>>>>>> while [[ -v f[++i] ]] ; do
>>>>>>> (( ++r > threads )) &&
>>>>>>> wait -n
>>>>>>> gcc -c "${f[i]}" &
>>>>>>> done
>>>>>>>
>>>>>>
>>>>>> How nice!
>>>>>>
>>>>>> wait -n exit 1 & echo $?
>>>>>>
>>>>>
>>>>> doesnt need a pid
>>>>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>>>>
>>>>> You got me the solution :) Except that wait expects a pid after -n.
>>>>>>
>>>>>> Maybe
>>>>>>
>>>>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>>>>
>>>>>> is what you meant? The equivalence of sequential execution?
>>>>>>
>>>>>> First think, then do magic.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars <
>>>>>>> mjbaars1977.bac...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello Paul,
>>>>>>>>
>>>>>>>> It seems I'm awake a little longer than you are.
>>>>>>>>
>>>>>>>> The second paragraph as you see it, belongs to 1)
>>>>>>>> The fourth paragraph as you see it, belongs to 2)
>>>>>>>>
>>>>>>>> The actual command invocations (a Makefile, a make.sh script) can
>>>>>>>> be found
>>>>>>>> in the attachment, as indicated on the first line of the mail. In
>>>>>>>> the
>>>>>>>> attachment there are two directories, one and two, belonging to 1)
>>>>>>>> and 2)
>>>>>>>> respectively.
>>>>>>>>
>>>>>>>> I'm not into Vulcan mindmelds, so I hope everything from the first
>>>>>>>> mail
>>>>>>>> makes sense to you and everyone on this mailing list now.
Re: multi-threaded compiling
Sorry. I mean:
for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); do
wait -n; echo $?; done;
doesn't function. With an ampersand instead of a semicolon. Why does it
function when called from a script and why does it fail when called from
the command line?
On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars
wrote:
> You mean:
>
> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>
> with one command and one wait in a single loop. And this does execute on
> the command line. How interesting!
>
> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> wait -n; echo $?; done;
>
> Because this doesn't and to be honest, I needed the pid and its index to
> retrieve gcc's output from a log file array afterwards.
>
> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 19:22 Mischa Baars
>> wrote:
>>
>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev
>>> wrote:
>>>
>>>> i also completly dont get ur issue
>>>>
>>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>>
>>>> while [[ -v f[++i] ]] ; do
>>>> (( ++r > threads )) &&
>>>> wait -n
>>>> gcc -c "${f[i]}" &
>>>> done
>>>>
>>>
>>> How nice!
>>>
>>> wait -n exit 1 & echo $?
>>>
>>
>> doesnt need a pid
>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>
>> You got me the solution :) Except that wait expects a pid after -n.
>>>
>>> Maybe
>>>
>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>
>>> is what you meant? The equivalence of sequential execution?
>>>
>>> First think, then do magic.
>>>
>>>
>>>>
>>>>
>>>
>>>>
>>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars
>>>> wrote:
>>>>
>>>>> Hello Paul,
>>>>>
>>>>> It seems I'm awake a little longer than you are.
>>>>>
>>>>> The second paragraph as you see it, belongs to 1)
>>>>> The fourth paragraph as you see it, belongs to 2)
>>>>>
>>>>> The actual command invocations (a Makefile, a make.sh script) can be
>>>>> found
>>>>> in the attachment, as indicated on the first line of the mail. In the
>>>>> attachment there are two directories, one and two, belonging to 1) and
>>>>> 2)
>>>>> respectively.
>>>>>
>>>>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>>>>> makes sense to you and everyone on this mailing list now.
>>>>>
>>>>> Best regards,
>>>>> Mischa Baars.
>>>>>
>>>>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith wrote:
>>>>>
>>>>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>>>>> > > I've attached the problem. It consists of two parts:
>>>>> > >
>>>>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>>>>> > >
>>>>> > > Running bash script functions as expected, but executing the same
>>>>> > > line of code with make and / or the command line, does not
>>>>> function.
>>>>> > > Perhaps someone could explain to me why?
>>>>> > >
>>>>> > > 2) passing a string argument from a bash script and / or Makefile
>>>>> to
>>>>> > > the gcc -D option
>>>>> > >
>>>>> > > Running the makefile functions as expected, but I have not been
>>>>> able
>>>>> > > to get similar code to work from a bash script. Can someone please
>>>>> > > explain to me what I'm doing wrong?
>>>>> >
>>>>> > I don't understand the problem. In the third paragraph above you say
>>>>> > the bash script works as expected and the makefile doesn't work, but
>>>>> in
>>>>> > the last paragraph you say that the makefile works as expected but
>>>>> you
>>>>> > can't get it to work in bash.
>>>>> >
>>>>> > Please provide actual command invocations (cut and pasted) showing
>>>>> the
>>>>> > output you received and explaining exactly what is wrong with it.
>>>>> >
>>>>> > But before you do that, be aware that make does NOT invoke /bin/bash
>>>>> as
>>>>> > its shell. It invokes /bin/sh. On some systems /bin/sh is actually
>>>>> an
>>>>> > alias for bash. On other systems it isn't.
>>>>> >
>>>>> > If you want your makefile to always use bash as its shell, you should
>>>>> > add an explicit:
>>>>> >
>>>>> > SHELL := /bin/bash
>>>>> >
>>>>> > to your makefile to force it. Maybe that will solve your problem.
>>>>> If
>>>>> > not we'll need details such as I mention above.
>>>>> >
>>>>>
>>>>
Re: multi-threaded compiling
Which sort of brings us back to the original question I suppose. Who does
that line of code function from a script and why does it fail from the
command line? My guess was that the same thing makes this line fail from
the Makefile.
On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars
wrote:
> You mean:
>
> for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
>
> with one command and one wait in a single loop. And this does execute on
> the command line. How interesting!
>
> for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
> wait -n; echo $?; done;
>
> Because this doesn't and to be honest, I needed the pid and its index to
> retrieve gcc's output from a log file array afterwards.
>
> On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev
> wrote:
>
>>
>>
>> On Mon, Mar 11, 2024, 19:22 Mischa Baars
>> wrote:
>>
>>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev
>>> wrote:
>>>
>>>> i also completly dont get ur issue
>>>>
>>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>>
>>>> while [[ -v f[++i] ]] ; do
>>>> (( ++r > threads )) &&
>>>> wait -n
>>>> gcc -c "${f[i]}" &
>>>> done
>>>>
>>>
>>> How nice!
>>>
>>> wait -n exit 1 & echo $?
>>>
>>
>> doesnt need a pid
>> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>>
>> You got me the solution :) Except that wait expects a pid after -n.
>>>
>>> Maybe
>>>
>>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>>
>>> is what you meant? The equivalence of sequential execution?
>>>
>>> First think, then do magic.
>>>
>>>
>>>>
>>>>
>>>
>>>>
>>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars
>>>> wrote:
>>>>
>>>>> Hello Paul,
>>>>>
>>>>> It seems I'm awake a little longer than you are.
>>>>>
>>>>> The second paragraph as you see it, belongs to 1)
>>>>> The fourth paragraph as you see it, belongs to 2)
>>>>>
>>>>> The actual command invocations (a Makefile, a make.sh script) can be
>>>>> found
>>>>> in the attachment, as indicated on the first line of the mail. In the
>>>>> attachment there are two directories, one and two, belonging to 1) and
>>>>> 2)
>>>>> respectively.
>>>>>
>>>>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>>>>> makes sense to you and everyone on this mailing list now.
>>>>>
>>>>> Best regards,
>>>>> Mischa Baars.
>>>>>
>>>>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith wrote:
>>>>>
>>>>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>>>>> > > I've attached the problem. It consists of two parts:
>>>>> > >
>>>>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>>>>> > >
>>>>> > > Running bash script functions as expected, but executing the same
>>>>> > > line of code with make and / or the command line, does not
>>>>> function.
>>>>> > > Perhaps someone could explain to me why?
>>>>> > >
>>>>> > > 2) passing a string argument from a bash script and / or Makefile
>>>>> to
>>>>> > > the gcc -D option
>>>>> > >
>>>>> > > Running the makefile functions as expected, but I have not been
>>>>> able
>>>>> > > to get similar code to work from a bash script. Can someone please
>>>>> > > explain to me what I'm doing wrong?
>>>>> >
>>>>> > I don't understand the problem. In the third paragraph above you say
>>>>> > the bash script works as expected and the makefile doesn't work, but
>>>>> in
>>>>> > the last paragraph you say that the makefile works as expected but
>>>>> you
>>>>> > can't get it to work in bash.
>>>>> >
>>>>> > Please provide actual command invocations (cut and pasted) showing
>>>>> the
>>>>> > output you received and explaining exactly what is wrong with it.
>>>>> >
>>>>> > But before you do that, be aware that make does NOT invoke /bin/bash
>>>>> as
>>>>> > its shell. It invokes /bin/sh. On some systems /bin/sh is actually
>>>>> an
>>>>> > alias for bash. On other systems it isn't.
>>>>> >
>>>>> > If you want your makefile to always use bash as its shell, you should
>>>>> > add an explicit:
>>>>> >
>>>>> > SHELL := /bin/bash
>>>>> >
>>>>> > to your makefile to force it. Maybe that will solve your problem.
>>>>> If
>>>>> > not we'll need details such as I mention above.
>>>>> >
>>>>>
>>>>
Re: multi-threaded compiling
You mean:
for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done;
with one command and one wait in a single loop. And this does execute on
the command line. How interesting!
for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do
wait -n; echo $?; done;
Because this doesn't and to be honest, I needed the pid and its index to
retrieve gcc's output from a log file array afterwards.
On Mon, Mar 11, 2024 at 7:25 PM alex xmb sw ratchev
wrote:
>
>
> On Mon, Mar 11, 2024, 19:22 Mischa Baars
> wrote:
>
>> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev
>> wrote:
>>
>>> i also completly dont get ur issue
>>>
>>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>>>
>>> while [[ -v f[++i] ]] ; do
>>> (( ++r > threads )) &&
>>> wait -n
>>> gcc -c "${f[i]}" &
>>> done
>>>
>>
>> How nice!
>>
>> wait -n exit 1 & echo $?
>>
>
> doesnt need a pid
> 1 : 1 as i wrote it , excepts add 'wait' as new last line
>
> You got me the solution :) Except that wait expects a pid after -n.
>>
>> Maybe
>>
>> for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
>>
>> is what you meant? The equivalence of sequential execution?
>>
>> First think, then do magic.
>>
>>
>>>
>>>
>>
>>>
>>> On Mon, Mar 11, 2024, 18:16 Mischa Baars
>>> wrote:
>>>
>>>> Hello Paul,
>>>>
>>>> It seems I'm awake a little longer than you are.
>>>>
>>>> The second paragraph as you see it, belongs to 1)
>>>> The fourth paragraph as you see it, belongs to 2)
>>>>
>>>> The actual command invocations (a Makefile, a make.sh script) can be
>>>> found
>>>> in the attachment, as indicated on the first line of the mail. In the
>>>> attachment there are two directories, one and two, belonging to 1) and
>>>> 2)
>>>> respectively.
>>>>
>>>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>>>> makes sense to you and everyone on this mailing list now.
>>>>
>>>> Best regards,
>>>> Mischa Baars.
>>>>
>>>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith wrote:
>>>>
>>>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>>>> > > I've attached the problem. It consists of two parts:
>>>> > >
>>>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>>>> > >
>>>> > > Running bash script functions as expected, but executing the same
>>>> > > line of code with make and / or the command line, does not function.
>>>> > > Perhaps someone could explain to me why?
>>>> > >
>>>> > > 2) passing a string argument from a bash script and / or Makefile to
>>>> > > the gcc -D option
>>>> > >
>>>> > > Running the makefile functions as expected, but I have not been able
>>>> > > to get similar code to work from a bash script. Can someone please
>>>> > > explain to me what I'm doing wrong?
>>>> >
>>>> > I don't understand the problem. In the third paragraph above you say
>>>> > the bash script works as expected and the makefile doesn't work, but
>>>> in
>>>> > the last paragraph you say that the makefile works as expected but you
>>>> > can't get it to work in bash.
>>>> >
>>>> > Please provide actual command invocations (cut and pasted) showing the
>>>> > output you received and explaining exactly what is wrong with it.
>>>> >
>>>> > But before you do that, be aware that make does NOT invoke /bin/bash
>>>> as
>>>> > its shell. It invokes /bin/sh. On some systems /bin/sh is actually
>>>> an
>>>> > alias for bash. On other systems it isn't.
>>>> >
>>>> > If you want your makefile to always use bash as its shell, you should
>>>> > add an explicit:
>>>> >
>>>> > SHELL := /bin/bash
>>>> >
>>>> > to your makefile to force it. Maybe that will solve your problem. If
>>>> > not we'll need details such as I mention above.
>>>> >
>>>>
>>>
Re: multi-threaded compiling
On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge wrote:
> On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!};
> done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]};
> e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> > /bin/bash: line 1: wait: 1747087: no such job
> > 0 pid 1747087 exit 127
> > /bin/bash: line 1: wait: 1747088: no such job
> > 1 pid 1747088 exit 127
>
> Without analyzing this in depth, one thing struck me immediately:
> you're using the reserved variable SECONDS, which has special semantics
> in bash. ${SECONDS} is going to expand to an ever-increasing number,
> beginning with 5 (since that's the value you assigned), and going up
> by 1 per second that the script runs. I'm assuming that was not your
> intention.
>
It wasn't. I modified the script and makefile accordingly.
>
> In general, any variable whose name is all capital letters *may* have
> special meaning to the shell and should not be used for general storage
> purposes in scripts.
>
Completely right again. I will definitely use these recommendations next
time I write a script or a makefile.
Re: multi-threaded compiling
Hi Greg,
Good point. One for you :)
Cheerz,
Mischa.
On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge wrote:
> On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote:
> > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!};
> done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]};
> e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
> > /bin/bash: line 1: wait: 1747087: no such job
> > 0 pid 1747087 exit 127
> > /bin/bash: line 1: wait: 1747088: no such job
> > 1 pid 1747088 exit 127
>
> Without analyzing this in depth, one thing struck me immediately:
> you're using the reserved variable SECONDS, which has special semantics
> in bash. ${SECONDS} is going to expand to an ever-increasing number,
> beginning with 5 (since that's the value you assigned), and going up
> by 1 per second that the script runs. I'm assuming that was not your
> intention.
>
> In general, any variable whose name is all capital letters *may* have
> special meaning to the shell and should not be used for general storage
> purposes in scripts.
>
Re: multi-threaded compiling
On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev
wrote:
> i also completly dont get ur issue
>
> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r=
>
> while [[ -v f[++i] ]] ; do
> (( ++r > threads )) &&
> wait -n
> gcc -c "${f[i]}" &
> done
>
How nice!
wait -n exit 1 & echo $?
You got me the solution :) Except that wait expects a pid after -n.
Maybe
for (( i=0; i<32; i++ )); do exit 1 & wait -n $!; echo $?; done;
is what you meant? The equivalence of sequential execution?
First think, then do magic.
>
>
>
> On Mon, Mar 11, 2024, 18:16 Mischa Baars
> wrote:
>
>> Hello Paul,
>>
>> It seems I'm awake a little longer than you are.
>>
>> The second paragraph as you see it, belongs to 1)
>> The fourth paragraph as you see it, belongs to 2)
>>
>> The actual command invocations (a Makefile, a make.sh script) can be found
>> in the attachment, as indicated on the first line of the mail. In the
>> attachment there are two directories, one and two, belonging to 1) and 2)
>> respectively.
>>
>> I'm not into Vulcan mindmelds, so I hope everything from the first mail
>> makes sense to you and everyone on this mailing list now.
>>
>> Best regards,
>> Mischa Baars.
>>
>> On Mon, Mar 11, 2024 at 5:01 PM Paul Smith wrote:
>>
>> > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote:
>> > > I've attached the problem. It consists of two parts:
>> > >
>> > > 1) multi-threaded bash script and / or multi-threaded Makefile
>> > >
>> > > Running bash script functions as expected, but executing the same
>> > > line of code with make and / or the command line, does not function.
>> > > Perhaps someone could explain to me why?
>> > >
>> > > 2) passing a string argument from a bash script and / or Makefile to
>> > > the gcc -D option
>> > >
>> > > Running the makefile functions as expected, but I have not been able
>> > > to get similar code to work from a bash script. Can someone please
>> > > explain to me what I'm doing wrong?
>> >
>> > I don't understand the problem. In the third paragraph above you say
>> > the bash script works as expected and the makefile doesn't work, but in
>> > the last paragraph you say that the makefile works as expected but you
>> > can't get it to work in bash.
>> >
>> > Please provide actual command invocations (cut and pasted) showing the
>> > output you received and explaining exactly what is wrong with it.
>> >
>> > But before you do that, be aware that make does NOT invoke /bin/bash as
>> > its shell. It invokes /bin/sh. On some systems /bin/sh is actually an
>> > alias for bash. On other systems it isn't.
>> >
>> > If you want your makefile to always use bash as its shell, you should
>> > add an explicit:
>> >
>> > SHELL := /bin/bash
>> >
>> > to your makefile to force it. Maybe that will solve your problem. If
>> > not we'll need details such as I mention above.
>> >
>>
>
Re: multi-threaded compiling
Oh, ok. I have executed the examples for you now.
What I expected was also in the first mail:
1) The one.make.sh.log has the expected output. The same commands executed
from a Makefile (one.make.log) do not behave as expected. This also holds
for this command executed from the command line.
2) The two.make.log has the expected. The same commands executed from the
bash script do not behave as expected. I'm having trouble obtaining the
desired results with the bash script.
Regards,
Mischa Baars.
On Mon, Mar 11, 2024 at 6:27 PM Paul Smith wrote:
> On Mon, 2024-03-11 at 18:14 +0100, Mischa Baars wrote:
> > The actual command invocations (a Makefile, a make.sh script) can be
> > found in the attachment, as indicated on the first line of the mail.
> > In the attachment there are two directories, one and two, belonging
> > to 1) and 2) respectively.
>
> That's not what I asked: I asked YOU to run the commands on YOUR system
> and cut and paste the results of the run into your email message, with
> an explicit explanation of exactly what is wrong with the results and
> what you wanted to see instead. What errors did you get? Or what was
> unexpected about the behavior?
>
> I'm not interested in running random scripts on my system that were
> downloaded from an email, or examining their behavior to make sure
> they're OK, or setting up a container to run them in, or guessing what
> you wanted to happen, etc.
>
A container? These are no viri, these are scripts :)
> Or maybe someone else will want to do that work instead.
>
> But before putting in more effort on the "misbehaving" makefile, read
> this part of my response carefully:
>
> > But before you do that, be aware that make does NOT invoke /bin/bash
> > as its shell. It invokes /bin/sh. On some systems /bin/sh is
> > actually an alias for bash. On other systems it isn't.
> >
> > If you want your makefile to always use bash as its shell, you should
> > add an explicit:
> >
> > SHELL := /bin/bash
> >
> > to your makefile to force it. Maybe that will solve your problem.
> > If not we'll need details such as I mention above.
>
> and verify in your response you added the suggested line to your
> makefile and it didn't help (in the situation where the makefile was
> behaving in an unexpected way).
>
No. Doesn't help the tiniest bit.
SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;
/bin/bash: line 1: wait: 1747087: no such job
0 pid 1747087 exit 127
/bin/bash: line 1: wait: 1747088: no such job
1 pid 1747088 exit 127
/bin/bash: line 1: wait: 1747089: no such job
2 pid 1747089 exit 127
/bin/bash: line 1: wait: 1747090: no such job
3 pid 1747090 exit 127
/bin/bash: line 1: wait: 1747091: no such job
4 pid 1747091 exit 127
/bin/bash: line 1: wait: 1747092: no such job
5 pid 1747092 exit 127
/bin/bash: line 1: wait: 1747093: no such job
6 pid 1747093 exit 127
/bin/bash: line 1: wait: 1747094: no such job
7 pid 1747094 exit 127
/bin/bash: line 1: wait: 1747095: no such job
8 pid 1747095 exit 127
/bin/bash: line 1: wait: 1747096: no such job
9 pid 1747096 exit 127
/bin/bash: line 1: wait: 1747097: no such job
10 pid 1747097 exit 127
/bin/bash: line 1: wait: 1747098: no such job
11 pid 1747098 exit 127
/bin/bash: line 1: wait: 1747099: no such job
12 pid 1747099 exit 127
/bin/bash: line 1: wait: 1747100: no such job
13 pid 1747100 exit 127
/bin/bash: line 1: wait: 1747101: no such job
14 pid 1747101 exit 127
/bin/bash: line 1: wait: 1747102: no such job
15 pid 1747102 exit 127
/bin/bash: line 1: wait: 1747103: no such job
16 pid 1747103 exit 127
/bin/bash: line 1: wait: 1747104: no such job
17 pid 1747104 exit 127
/bin/bash: line 1: wait: 1747105: no such job
18 pid 1747105 exit 127
/bin/bash: line 1: wait: 1747106: no such job
19 pid 1747106 exit 127
/bin/bash: line 1: wait: 1747107: no such job
20 pid 1747107 exit 127
/bin/bash: line 1: wait: 1747108: no such job
21 pid 1747108 exit 127
/bin/bash: line 1: wait: 1747109: no such job
22 pid 1747109 exit 127
/bin/bash: line 1: wait: 1747110: no such job
23 pid 1747110 exit 127
/bin/bash: line 1: wait: 1747111: no such job
24 pid 1747111 exit 127
/bin/bash: line 1: wait: 1747112: no such job
25 pid 1747112 exit 127
/bin/bash: line 1: wait: 1747113: no such job
26 pid 1747113 exit 127
/bin/bash: line 1: wait: 1747114: no such job
27 pid 1747114 exit 127
/bin/bash: line 1: wait: 1747115: no such job
28 pid 1747115 exit 127
/bin/bash: line 1: wait: 1747116: no such job
29 pid 1747116 exit 127
/bin/bash: line 1: wait: 1747117: no such job
30 pid 1747117 exit 127
/bin/bash: line 1: wait: 1747118: no such job
31 pid 1747118 e
Re: multi-threaded compiling
Hello Paul, It seems I'm awake a little longer than you are. The second paragraph as you see it, belongs to 1) The fourth paragraph as you see it, belongs to 2) The actual command invocations (a Makefile, a make.sh script) can be found in the attachment, as indicated on the first line of the mail. In the attachment there are two directories, one and two, belonging to 1) and 2) respectively. I'm not into Vulcan mindmelds, so I hope everything from the first mail makes sense to you and everyone on this mailing list now. Best regards, Mischa Baars. On Mon, Mar 11, 2024 at 5:01 PM Paul Smith wrote: > On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote: > > I've attached the problem. It consists of two parts: > > > > 1) multi-threaded bash script and / or multi-threaded Makefile > > > > Running bash script functions as expected, but executing the same > > line of code with make and / or the command line, does not function. > > Perhaps someone could explain to me why? > > > > 2) passing a string argument from a bash script and / or Makefile to > > the gcc -D option > > > > Running the makefile functions as expected, but I have not been able > > to get similar code to work from a bash script. Can someone please > > explain to me what I'm doing wrong? > > I don't understand the problem. In the third paragraph above you say > the bash script works as expected and the makefile doesn't work, but in > the last paragraph you say that the makefile works as expected but you > can't get it to work in bash. > > Please provide actual command invocations (cut and pasted) showing the > output you received and explaining exactly what is wrong with it. > > But before you do that, be aware that make does NOT invoke /bin/bash as > its shell. It invokes /bin/sh. On some systems /bin/sh is actually an > alias for bash. On other systems it isn't. > > If you want your makefile to always use bash as its shell, you should > add an explicit: > > SHELL := /bin/bash > > to your makefile to force it. Maybe that will solve your problem. If > not we'll need details such as I mention above. >
multi-threaded compiling
Hi, I'd like to compile a project of mine using multiple logical cores. I've attached the problem. It consists of two parts: 1) multi-threaded bash script and / or multi-threaded Makefile Running bash script functions as expected, but executing the same line of code with make and / or the command line, does not function. Perhaps someone could explain to me why? 2) passing a string argument from a bash script and / or Makefile to the gcc -D option Running the makefile functions as expected, but I have not been able to get similar code to work from a bash script. Can someone please explain to me what I'm doing wrong? Hope to hear from you soon. Best regards, Mischa Baars. 2024031100 - gnu questions.tar.xz Description: application/xz
Re: bash conditional expressions
Hi Greg, It is how my makefiles work :) Do you realize how much time it takes to load the stat executable on all source files and object files? Conditional expressions are already in memory. I prefer using the mtime field, the atime field and conditional expressions. Mischa. On Fri, Nov 12, 2021 at 1:25 PM Greg Wooledge wrote: > On Fri, Nov 12, 2021 at 10:36:01AM +0100, Mischa Baars wrote: > > All of my makefiles only compile source files and link object files that > > are NEW, as in the modification timestamp is newer than OR EQUAL TO the > > access timestamp, > > That's not how Makefiles work. > > Makefiles compare the mtimes of two files against each other. Source > and target. If the source's mtime is greater than the target's mtime, > then the target needs to be rebuilt. > > Most Linux distributions these days are disabling or half-disabling > the atime field at the file system mount-option level, because of the > tremendous inefficiency and wear on disks that it creates. > >
bash conditional expressions
Hi All, All of my makefiles only compile source files and link object files that are NEW, as in the modification timestamp is newer than OR EQUAL TO the access timestamp, such that when I include a new source file into a project or produce a new object file during compilation of the project, it does not take more time than required. I have two different versions of Fedora running here, Fedora 32 and Fedora 35. Using Fedora 32 (bash 5.0.17) this returns a true, while on Fedora 35 (bash 5.1.8) this returns a false: touch test; if [[ -N test ]]; then echo true; else echo false; fi; This means that newly created object files are no longer recognized by the compile scripts as new object files and executables are no longer linked against these new object files. Could you please restore the Fedora 32 behaviour? Someone must have read the bash manual a little too precise, because now the statement only returns true when a 'touch -a test' is given and not when a 'touch -am test' is given. As I understand it, -N stands for NEW and therefore should return a true when either a 'touch -a test' or a 'touch -am test' is given. In my opinion it is the manual page that should have been updated, and not the bash response to a -N conditional expression. Looking forward to a fix! Best regards, Mischa Baars.
Re: Command grouping
On Sun, 2019-10-13 at 10:54 +0100, k...@plushkava.net wrote:
> On Sun, 13 Oct 2019 10:09:26 +0200
> Mischa Baars wrote:
>
> > On Sat, 2019-10-12 at 10:42 -0400, Chet Ramey wrote:
> > > On 10/12/19 9:02 AM, Mischa Baars wrote:
> > > > Hi,
> > > >
> > > > Perhaps to better have a look at this mail, than the previous mail.
> > > >
> > > > In trying to group commands, in this case compiler commands, I found
> > > > some peculiarities while trying different combinations of the 'Internal
> > > > Field
> > > > Separator'
> > > > and the 'Parameter Expansion' operator ${parameter@P}.
> > >
> > > I haven't looked at this in any depth, but you should realize that running
> > >
> > > declare IFS=$(printf ' \t\n')
> > >
> > > will result in IFS being set to space and tab, since command substitution
> > > removes the trailing newline. Maybe that will make a difference.
> > >
> >
> > Hi Chet,
> >
> > The bash manual page tells us that is the default value, nothing special.
> >
> > I was trying to remove the whitespace, such that whitespaces are allowed in
> > project directory names, as was considered the default scenario as far as I
> > know.
>
> You are not removing the whitespace values from IFS. Instead, you are
> re-asserting the default value of IFS, only with \n missing (for the reason
> that Chet
> describes). Given that the output of your find command is newline-delimited,
> doing so will not end well. Writing IFS=$'\n' may function as you expected.
> However, this is an intrinsically broken approach to begin with. See
> https://mywiki.wooledge.org/DontReadLinesWithFor.
>
> Instead, consider any of the following options:-
>
> * find -exec or -execdir
> * a for loop that iterates over a glob
> * a while/read loop that consumes find's output (using -print0, ideally)
>
> Your problems are compounded by failing to quote the expansion of i in the
> course of invoking gcc (lines 4-5), and by the flawed attempt at a workaround
> (lines 2 and 6-7). See https://mywiki.wooledge.org/BashFAQ/050. Not only
> that, but the *.c argument conveyed to find should be quoted, so as to inhibit
> pathname expansion.
>
> Here is an example of how one might use for:-
>
> shopt -s globstar
> for i in ./pace/**/*.c; do
> gcc -o "${i%.c}" "$i"
> done
>
Hi Kerin,
I believe you're missing the point. You were probably distracted by me
brabbling about whitespaces.
Again. When scripting, commands often do not fit on one line. To overcome this
problem, let's try to separate the commands from the loops:
--- emphasis on separation ---
shopt -s globstar; export E0="\${i}"; for i in ./pace/**/*.c; do ls ${E0@P};
done; # functions as supposed to
shopt -s globstar; export E0="ls \${i}"; for i in ./pace/**/*.c; do ${E0@P};
done; # functions as supposed to
--- emphasis on pathname expansion ---
shopt -s globstar; export E0="\${i}"; for i in ./pace/**/*.c; do ls "${E0@P}";
done;# functions as supposed to
shopt -s globstar; export E0="ls \${i}"; for i in ./pace/**/*.c; do "${E0@P}";
done;# does not function as supposed to
Aliases are not defined in makefiles, and I remember there to be something
uncomfortable with functions in the loop too, I would have to look that up. So
we'll
have to do with variables being used to define functions.
Regards,
Mischa.
Re: Command grouping
On Sat, 2019-10-12 at 10:42 -0400, Chet Ramey wrote:
> On 10/12/19 9:02 AM, Mischa Baars wrote:
> > Hi,
> >
> > Perhaps to better have a look at this mail, than the previous mail.
> >
> > In trying to group commands, in this case compiler commands, I found some
> > peculiarities while trying different combinations of the 'Internal Field
> > Separator'
> > and the 'Parameter Expansion' operator ${parameter@P}.
>
> I haven't looked at this in any depth, but you should realize that running
>
> declare IFS=$(printf ' \t\n')
>
> will result in IFS being set to space and tab, since command substitution
> removes the trailing newline. Maybe that will make a difference.
>
Hi Chet,
The bash manual page tells us that is the default value, nothing special.
I was trying to remove the whitespace, such that whitespaces are allowed in
project directory names, as was considered the default scenario as far as I
know.
Mischa.
Command grouping
Hi,
Perhaps to better have a look at this mail, than the previous mail.
In trying to group commands, in this case compiler commands, I found some
peculiarities while trying different combinations of the 'Internal Field
Separator'
and the 'Parameter Expansion' operator ${parameter@P}.
Perhaps someone can tell me if this is the behaviour that should have been
expected or that something is broken.
Thanks,
Mischa.
goodworx.tar.xz
Description: application/xz-compressed-tar
Re: selecting modules for kernel compilation automatically
On Sat, 2019-10-12 at 13:50 +0200, Mischa Baars wrote: > Hi, > > Can someone please tell me why this does work: > > > for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do > echo $(basename $i .ko.xz); done; > > While this doesn't: > > for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do > grep $(basename $i .ko.xz) /proc/modules; done; > for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do > lsmod | grep $(basename $i .ko.xz); done; > > Thanks, > Mischa. Never mind, I've probably been smoking too much. Thanks, Mischa.
selecting modules for kernel compilation automatically
Hi, Can someone please tell me why this does work: for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do echo $(basename $i .ko.xz); done; While this doesn't: for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do grep $(basename $i .ko.xz) /proc/modules; done; for i in $(find /lib/modules/5.3.0-next-20190924/ -type f | grep scsi); do lsmod | grep $(basename $i .ko.xz); done; Thanks, Mischa.
Re: file access time and file modification time
On Tue, 2019-09-24 at 09:29 -0400, Chet Ramey wrote: > On 9/24/19 3:04 AM, Mischa Baars wrote: > > Hi Kerin, > > > > That indeed solves the problem in 'accesstime.sh', although I would > > recommend some sort of reference from the bash manpage to the mount > > manpage. > > > > Did you have a look at the 'conditional.sh' script too? Looks like > > the > > '-N' switch compares only the integer part of the timestamp > > seconds. > > All this was fixed back in July, and the changes are in the bash > devel > git branch. > > Chet > Thank you Chet, you're a real hero :) Mischa.
Re: file access time and file modification time
Strange, it seems the replies did come through. So have you seen them? Let's try the gnome mail client, at least I got a receipt this time. Regards, Mischa. On Tue, 2019-09-24 at 09:04 +0200, Mischa Baars wrote: > Hi Kerin, > > That indeed solves the problem in 'accesstime.sh', although I would > recommend some sort of reference from the bash manpage to the mount > manpage. > > Did you have a look at the 'conditional.sh' script too? Looks like > the > '-N' switch compares only the integer part of the timestamp seconds. > > Regards, > Mischa. > > On Mon, 2019-07-08 at 09:28 +0100, k...@plushkava.net wrote: > > On Mon, 08 Jul 2019 09:29:19 +0200 > > "mjbaars1977.bug-bash" wrote: > > > > > Hi,I'm having some difficulties with the file access time and the > > > file modification time. Scripts are attached for your > > > convenience.Regards,Mischa Baars.Sent from my Samsung Galaxy > > > smartphone. > > > > If you are using Linux, you should be aware that it defaults to > > using > > "relatime", meaning that atimes are only updated under certain > > circumstances. The correct (POSIX compliant) behaviour is achieved > > by > > mounting the filesystem with the "strictatime" option. See > > http://man7.org/linux/man-pages/man8/mount.8.html for further > > information. > > > >
Re: file access time and file modification time
Hi Kerin, That indeed solves the problem in 'accesstime.sh', although I would recommend some sort of reference from the bash manpage to the mount manpage. Did you have a look at the 'conditional.sh' script too? Looks like the '-N' switch compares only the integer part of the timestamp seconds. Regards, Mischa. On Mon, 2019-07-08 at 09:28 +0100, k...@plushkava.net wrote: > On Mon, 08 Jul 2019 09:29:19 +0200 > "mjbaars1977.bug-bash" wrote: > > > Hi,I'm having some difficulties with the file access time and the > > file modification time. Scripts are attached for your > > convenience.Regards,Mischa Baars.Sent from my Samsung Galaxy > > smartphone. > > If you are using Linux, you should be aware that it defaults to using > "relatime", meaning that atimes are only updated under certain > circumstances. The correct (POSIX compliant) behaviour is achieved by > mounting the filesystem with the "strictatime" option. See > http://man7.org/linux/man-pages/man8/mount.8.html for further > information. >
Re: file access time and file modification time
Hi Kerin and Chet,
Testing mailing list. My earlier replies didn't come through.
Regards,
Mischa.
On Tue, 2019-07-09 at 11:16 +0100, k...@plushkava.net wrote:
> Hi Mischa,
>
> On Tue, 09 Jul 2019 09:05:45 +0200
> Mischa Baars wrote:
>
> > Hi Kerin,
> >
> > That indeed solves the problem in 'accesstime.sh', although I would
> > recommend some sort of reference from the bash manpage to the mount
> > manpage.
> >
> > Did you have a look at the 'conditional.sh' script too? Looks like
> > the '-N' switch compares only the integer part of the timestamp
> > seconds.
>
> The stat structure supports timestamp fields of nanosecond
> granularity since POSIX.1-2008, so it should work. I tried a simple
> test case here, and the behaviour of -N seems generally broken:
>
> # f=$(mktemp); [[ -N $f ]]; echo $?; touch -m "$f"; [[ -N $f ]]; echo
> $?
> 0
> 0
>
> I expected the value of $? to be non-zero at first, followed by 0
> after updating the mtime. Using the test command that ships with GNU
> coreutils (v8.31) exhibits the same problem:
>
> # f=$(mktemp); command test -N "$f"; echo $?; touch -m "$f"; command
> test -N "$f"; echo $?
> 0
> 0
>
> That was with ext4 as the filesystem backing the TMPDIR. I also
> tested FreeBSD/UFS, with the same results, even if sleeping one
> second before updating the mtime. Perhaps Chet would be so kind as to
> clarify what's going on here. In the meantime, your technique of
> comparing the timestamps lexicographically seems like a good
> workaround, although you should use [[ instead. For example:
>
> # check() { local at=$(stat -c %x "$1") mt=$(stat -c %y "$1"); [[ $mt
> > $at ]]; }
> # f=$(mktemp); check "$f"; echo $?; touch -m "$f"; check "$f"; echo
> $?
> 1
> 0
>
