[bug #61805] .ONESHELL causes quoting in .SHELLFLAGS to not work as expected.

[Paul D. Smith]

Update of bug #61805 (project make):

  Status:None => Fixed  
 Assigned to:None => psmith 
 Open/Closed:Open => Closed 
Operating System:None => Any
   Fixed Release:None => SCM
   Triage Status:None => Medium Effort  

___

Follow-up Comment #5:

I applied this but disabled the extra tests on Windows systems.

At some point we'll have to work on supporting non-standard SHELL values on
Windows but that's a larger problem.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #61864] incorrect debug output

[Paul D. Smith]

Update of bug #61864 (project make):

  Status:None => Fixed  
 Assigned to:None => psmith 
 Open/Closed:Open => Closed 
Operating System:None => Any
   Fixed Release:None => SCM
   Triage Status:None => Small Effort   


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

Re: Errors in man pages of make

[Paul Smith]

On Sat, 2022-02-05 at 11:24 +0100, Helge Kreutzmann wrote:
> I'm now reporting the errors for your project. If future reports
> should use another channel, please let me know.

This channel is fine, but please always do include the release of GNU
make that you're looking at when you submit changes (run "make
--version").

Note that there are no translations of the man page provided with GNU
make.

> Man page: make.1
> Issue: I doesn't appear in SYNOPSIS, why not format »target«
> instead?

I addressed this, thanks!

> --
> Man page: make.1
> Issue: B -> B;
> 
> "Internal option B uses to pass the jobserver pipe read and write file "
> "descriptor numbers to B see the section B "JOBSERVER> for details"
> --
> Man page: make.1
> Issue: I → I,
> 
> "If the job to be run is not a B then B will close the "
> "jobserver pipe file descriptors before invoking the commands, so that the "
> "command can not interfere with the I and the command does not "
> "find any unusual file descriptors."

I can't find either of the above two texts in the GNU make man page,
neither the current latest nor any other previous version going back to
the first committed version, in 1990.  There has never been any
description of the jobserver in any version of the man page.

All I can assume is that your distribution has provided you with a
version of the man page that is modified from the standard one.

Re: Bug in $(shell ...) I can't understand

[Paul Smith]

On Sun, 2022-02-06 at 17:12 -0500, Dmitry Goncharov wrote:
> This behavior is correct, is not it?
> 
> $ cat makefile
> SHELL:=/badpath/bash
> value:=$(shell echo hello world)
> all: ; $(info $(value))
> $ make >/dev/null
> make: /badpath/bash: No such file or directory
> $
> 
> stdout is redirected to /dev/null, but the user still wants to see
> errors. Is not this scenario what stderr was invented for?

Yes, but that's sent directly to stderr so make's shell function is not
even involved in that.

The weird behavior we're considering is that make was printing the
_stdout_ of the invoked shell to make's stderr, if the shell exited
with code 127 (see the source I mentioned previously).

> On Sun, Feb 6, 2022 at 4:48 PM Paul Smith  wrote:
> > I decided this was a bug and changed the behavior for the next
> > release.
> 
> What is the new behavior?

The above makefile behaves the same, and this makefile:

  $ cat Makefile
  out := $(shell bad-command 2>&1)
  all: ; @echo 'out = $(out)'

gives:

  $ make
  out = /bin/sh: 1: bad-command: not found

instead of what we get with 4.3:

  $ make
  /bin/sh: 1: bad-command: not found
  out =

Re: Bug in $(shell ...) I can't understand

[Dmitry Goncharov]

On Sun, Feb 6, 2022 at 3:28 PM Paul Smith  wrote:
> Oh.  I see the problem.  If the shell exits with an exit code of 127
> then make's shell function assumes that it failed because the sub-
> process was not found, and it actually writes its output to stderr!!

This behavior is correct, is not it?

$ cat makefile
SHELL:=/badpath/bash
value:=$(shell echo hello world)
all: ; $(info $(value))
$ make >/dev/null
make: /badpath/bash: No such file or directory
$

stdout is redirected to /dev/null, but the user still wants to see
errors. Is not this scenario what stderr was invented for?

On Sun, Feb 6, 2022 at 4:48 PM Paul Smith  wrote:
> I decided this was a bug and changed the behavior for the next release.

What is the new behavior?

regards, Dmitry

Re: Bug in $(shell ...) I can't understand

[Paul Smith]

On Sun, 2022-02-06 at 15:21 -0500, Paul Smith wrote:
> I'm not sure this is correct.  I will need to think about it.

I decided this was a bug and changed the behavior for the next release.

Re: Bug in $(shell ...) I can't understand

[Dmitry V. Levin]

On Sun, Feb 06, 2022 at 03:21:39PM -0500, Paul Smith wrote:
> On Sun, 2022-02-06 at 20:18 +0300, Dmitry V. Levin wrote:
> > 4175643 write(2, "/bin/sh: bad-program: command no"..., 
> > 40) = 40
> > 4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
> > 4175640 read(5,  
> > 4175643 +++ exited with 127 +++
> > 4175640 <... read resumed>"", 160)  = 0
> > 4175642 +++ exited with 127 +++
> > 4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
> > 4175640 write(1, "out = ", 6) = 6
> > 4175640 write(1, "\n", 1)    = 1
> > 4175640 read(4, "", 4096) = 0
> > 4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
> > 4175640 +++ exited with 2 +++
> 
> Yes, I saw this as well.  But this doesn't help me understand WHY it's
> happening.  There's something make is doing here that's causing this
> but I don't understand what it is.  Make is reading the output of the
> pipe, like we want, but somehow it's then rewriting the output to its
> own stderr... why is that happening?
> 
> Oh.  I see the problem.  If the shell exits with an exit code of 127
> then make's shell function assumes that it failed because the sub-
> process was not found, and it actually writes its output to stderr!!

Sorry if my examples were not clear enough.
Yes, it's the exit status that makes all the difference.


-- 
ldv

Re: Bug in $(shell ...) I can't understand

[Britton Kerin]

On Sun, Feb 6, 2022 at 8:19 AM Dmitry V. Levin  wrote:
>
> On Sun, Feb 06, 2022 at 11:23:03AM -0500, Paul Smith wrote:
> > OK, someone posted a question to SO and that led me to an hour or more
> > of banging my head against a wall trying to understand what's
> > happening... and I can't.
> >
> > The problem is that the user would like to invoke $(shell ...) and
> > capture errors, even errors that the program being run doesn't exist.
> > The shell function only captures stdout, not stderr.  This simple idea
> > won't work of course:
> >
> >   out := $(shell bad-program 2>&1)
> >   $(info out = $(out))
> >
> >   $ make
> >   /bin/sh: bad-program: not found
> >   out =
> >
> > because this redirects the output of the bad-program, but it's the
> > shell printing the error not the program.
>
> What's actually happening is this:
>
> strace-buildroot/usr/bin/strace -f -y -eexecve,/pipe,/dup,read,write -o'|cat 
> >&3' --signal=none make 3>&2 &>/dev/null
> 4175640 execve("/usr/bin/make", ["make"], 0x7ffdad18ea88 /* 17 vars */) = 0
> 4175640 read(4, 
> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 
> 832
> 4175640 read(4, 
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) 
> = 832
> 4175640 read(4, "out := $(shell bad-program 2>&1)"..., 4096) = 
> 54
> 4175640 pipe2([5, 6], 0) = 0
> 4175642 dup2(6, 1) = 1
> 4175642 execve("/bin/sh", ["/bin/sh", "-c", "bad-program 2>&1"], 
> 0x7ffd69c8a5a8 /* 17 vars */) = 0
> 4175640 read(5,  
> 4175642 read(4, 
> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 
> 832
> 4175642 read(4, 
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) 
> = 832
> 4175642 read(4, "#\n# Please refer to nsswitch.con"..., 
> 4096) = 1808
> 4175642 read(4, "", 4096) = 0
> 4175642 read(4, "root:x:0:0:System Administrator:"..., 4096) = 
> 1326
> 4175643 dup2(1, 2) = 2
> 4175643 write(2, "/bin/sh: bad-program: command no"..., 
> 40) = 40
> 4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
> 4175640 read(5,  
> 4175643 +++ exited with 127 +++
> 4175640 <... read resumed>"", 160)  = 0
> 4175642 +++ exited with 127 +++
> 4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
> 4175640 write(1, "out = ", 6) = 6
> 4175640 write(1, "\n", 1)= 1
> 4175640 read(4, "", 4096) = 0
> 4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
> 4175640 +++ exited with 2 +++
>
> Try the following instead:
>
> out := $(shell bad-program 2>&1 ||:)
> $(info out = $(out))

or

 out := $(shell bad-program 2>&1 || true)
 $(info out = $(out))

which makes it more obvious what's going on.

Britton

Re: Bug in $(shell ...) I can't understand

[Paul Smith]

On Sun, 2022-02-06 at 20:18 +0300, Dmitry V. Levin wrote:
> 4175643 write(2, "/bin/sh: bad-program: command no"..., 
> 40) = 40
> 4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
> 4175640 read(5,  
> 4175643 +++ exited with 127 +++
> 4175640 <... read resumed>"", 160)  = 0
> 4175642 +++ exited with 127 +++
> 4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
> 4175640 write(1, "out = ", 6) = 6
> 4175640 write(1, "\n", 1)    = 1
> 4175640 read(4, "", 4096) = 0
> 4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
> 4175640 +++ exited with 2 +++

Yes, I saw this as well.  But this doesn't help me understand WHY it's
happening.  There's something make is doing here that's causing this
but I don't understand what it is.  Make is reading the output of the
pipe, like we want, but somehow it's then rewriting the output to its
own stderr... why is that happening?

Oh.  I see the problem.  If the shell exits with an exit code of 127
then make's shell function assumes that it failed because the sub-
process was not found, and it actually writes its output to stderr!!

function.c:func_shell_base() contains:

/* shell_completed() will set shell_function_completed to 1 when the
   child dies normally, or to -1 if it dies with status 127, which is
   most likely an exec fail.  */

if (shell_function_completed == -1)
  {
/* This likely means that the execvp failed, so we should just
   write the error message in the pipe from the child.  */
fputs (buffer, stderr);
fflush (stderr);
  }
else
  ...

I'm not sure this is correct.  I will need to think about it.

> Try the following instead:
> 
> out := $(shell bad-program 2>&1 ||:)
> $(info out = $(out))
> 
> $ make
> out = /bin/sh: bad-program: command not found

Sure, there are definitely workarounds: I suggested a different one in
my message.  I was trying to understand why the behavior happened, not
how to work around it.

Re: Bug in $(shell ...) I can't understand

[Martin Dorey]

>> because this redirects the output of the bad-program, but it's the
>> shell printing the error not the program.

Dmitry's reply is more constructive but those strace arguments, all the pids 
and pipe inodes, ow my head, so perhaps it's worth a simple demonstration that, 
despite the undisputed truth of what Paul wrote above, the redirection still 
catches the error message:

mad@shuttle:~/tmp/make-shell-2022-02-06$ bad-program 2>&1 | wc
  1   5  37
mad@shuttle:~/tmp/make-shell-2022-02-06$


From: Bug-make  on behalf of 
Dmitry V. Levin 
Sent: Sunday, February 6, 2022 09:18
To: Paul Smith 
Cc: bug-make@gnu.org 
Subject: Re: Bug in $(shell ...) I can't understand

* EXTERNAL EMAIL *

On Sun, Feb 06, 2022 at 11:23:03AM -0500, Paul Smith wrote:
> OK, someone posted a question to SO and that led me to an hour or more
> of banging my head against a wall trying to understand what's
> happening... and I can't.
>
> The problem is that the user would like to invoke $(shell ...) and
> capture errors, even errors that the program being run doesn't exist.
> The shell function only captures stdout, not stderr.  This simple idea
> won't work of course:
>
>   out := $(shell bad-program 2>&1)
>   $(info out = $(out))
>
>   $ make
>   /bin/sh: bad-program: not found
>   out =
>
> because this redirects the output of the bad-program, but it's the
> shell printing the error not the program.

What's actually happening is this:

strace-buildroot/usr/bin/strace -f -y -eexecve,/pipe,/dup,read,write -o'|cat 
>&3' --signal=none make 3>&2 &>/dev/null
4175640 execve("/usr/bin/make", ["make"], 0x7ffdad18ea88 /* 17 vars */) = 0
4175640 read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 832
4175640 read(4, 
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) = 
832
4175640 read(4, "out := $(shell bad-program 2>&1)"..., 4096) = 54
4175640 pipe2([5, 6], 0) = 0
4175642 dup2(6, 1) = 1
4175642 execve("/bin/sh", ["/bin/sh", "-c", "bad-program 2>&1"], 0x7ffd69c8a5a8 
/* 17 vars */) = 0
4175640 read(5,  
4175642 read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 832
4175642 read(4, 
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) = 
832
4175642 read(4, "#\n# Please refer to nsswitch.con"..., 
4096) = 1808
4175642 read(4, "", 4096) = 0
4175642 read(4, "root:x:0:0:System Administrator:"..., 4096) = 1326
4175643 dup2(1, 2) = 2
4175643 write(2, "/bin/sh: bad-program: command no"..., 40) 
= 40
4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
4175640 read(5,  
4175643 +++ exited with 127 +++
4175640 <... read resumed>"", 160)  = 0
4175642 +++ exited with 127 +++
4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
4175640 write(1, "out = ", 6) = 6
4175640 write(1, "\n", 1)= 1
4175640 read(4, "", 4096) = 0
4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
4175640 +++ exited with 2 +++

Try the following instead:

out := $(shell bad-program 2>&1 ||:)
$(info out = $(out))

$ make
out = /bin/sh: bad-program: command not found

The only difference is the exit status.


--
ldv

Re: Bug in $(shell ...) I can't understand

[Dmitry V. Levin]

On Sun, Feb 06, 2022 at 11:23:03AM -0500, Paul Smith wrote:
> OK, someone posted a question to SO and that led me to an hour or more
> of banging my head against a wall trying to understand what's
> happening... and I can't.
> 
> The problem is that the user would like to invoke $(shell ...) and
> capture errors, even errors that the program being run doesn't exist. 
> The shell function only captures stdout, not stderr.  This simple idea
> won't work of course:
> 
>   out := $(shell bad-program 2>&1)
>   $(info out = $(out))
> 
>   $ make
>   /bin/sh: bad-program: not found
>   out =
> 
> because this redirects the output of the bad-program, but it's the
> shell printing the error not the program.

What's actually happening is this:

strace-buildroot/usr/bin/strace -f -y -eexecve,/pipe,/dup,read,write -o'|cat 
>&3' --signal=none make 3>&2 &>/dev/null 
4175640 execve("/usr/bin/make", ["make"], 0x7ffdad18ea88 /* 17 vars */) = 0
4175640 read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 832
4175640 read(4, 
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) = 
832
4175640 read(4, "out := $(shell bad-program 2>&1)"..., 4096) = 54
4175640 pipe2([5, 6], 0) = 0
4175642 dup2(6, 1) = 1
4175642 execve("/bin/sh", ["/bin/sh", "-c", "bad-program 2>&1"], 0x7ffd69c8a5a8 
/* 17 vars */) = 0
4175640 read(5,  
4175642 read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 832
4175642 read(4, 
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) = 
832
4175642 read(4, "#\n# Please refer to nsswitch.con"..., 
4096) = 1808
4175642 read(4, "", 4096) = 0
4175642 read(4, "root:x:0:0:System Administrator:"..., 4096) = 1326
4175643 dup2(1, 2) = 2
4175643 write(2, "/bin/sh: bad-program: command no"..., 40) 
= 40
4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
4175640 read(5,  
4175643 +++ exited with 127 +++
4175640 <... read resumed>"", 160)  = 0
4175642 +++ exited with 127 +++
4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
4175640 write(1, "out = ", 6) = 6
4175640 write(1, "\n", 1)= 1
4175640 read(4, "", 4096) = 0
4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
4175640 +++ exited with 2 +++

Try the following instead:

out := $(shell bad-program 2>&1 ||:)
$(info out = $(out))

$ make
out = /bin/sh: bad-program: command not found

The only difference is the exit status.


-- 
ldv

Bug in $(shell ...) I can't understand

[Paul Smith]

OK, someone posted a question to SO and that led me to an hour or more
of banging my head against a wall trying to understand what's
happening... and I can't.

The problem is that the user would like to invoke $(shell ...) and
capture errors, even errors that the program being run doesn't exist. 
The shell function only captures stdout, not stderr.  This simple idea
won't work of course:

  out := $(shell bad-program 2>&1)
  $(info out = $(out))

  $ make
  /bin/sh: bad-program: not found
  out =

because this redirects the output of the bad-program, but it's the
shell printing the error not the program.  So I suggested this:

  out := $(shell $(SHELL) -c 'bad-program' 2>&1)
  $(info out = $(out))

This SHOULD work: the outer shell is invoking a sub-shell with the sub-
shell's stderr redirected to its stdout, then the subshell will print
the not found error to its stderr (redirected to stdout).

But it DOES NOT WORK!

  $ make
  /bin/sh: bad-program: not found
  out =

What is happening here?!?!  I wrote a little C program to verify my
thinking:

/* - */
#include 
#include 
#include 

int main(int argc, char** argv)
{
char *args[4] = {"/bin/sh", "-c",
 "/bin/sh -c bad-program 2>&1", NULL};

int pid = fork();
if (pid == 0) {
execv(args[0], args);
} else {
int st;
waitpid(pid, , 0);
}

return 0;
}
/* - */

But, this works as expected.  Something make is doing is causing
problems here.  I tried building both with and without posix_spawn and
both behave the same so it's nothing like that.

If you force the sub-shell to create its own subshell, then it works! 
This works:

  out := $(shell $(SHELL) -c 'bad-program | cat' 2>&1)
  $(info out = $(out))

  $ make
  out = /bin/sh: bad-program: not found

Re: [PATCH] RFC: add --shuffle[=seed] argument support

[Sergei Trofimovich]

On Sat, 05 Feb 2022 18:39:41 -0500
Paul Smith  wrote:

> Nice work!
> 
> On Sat, 2022-02-05 at 22:04 +, Sergei Trofimovich wrote:
> > Some high level TODOs:  
> 
> For this amount of change it's likely that you'll need to provide
> copyright assignment paperwork.  Let me know if you'd like more details
> about this.
> 
> > - No documentation for the optin yet.  
> 
> This feature would also need a set of regression tests.

Sounds good. Will start adding the tests.

> > - No(?) environment passing for recursive make.  I would prefer to
> > share the same see across all calls to get easier reproducers.  
> 
> You explicitly disabled this, though... was there a reason?

Just a bug. I misinterpreted the 'toenv' meaning.

> > - The dependency traversal is naive and uses potentially unbounded
> > stack.
> > - srand() / rand() is used from system libc.  This might make it
> > harder to reproduce a failure on another machine.  But maybe it's
> > fine.  
> 
> There are a few issues here:
> 
> I recommend you try your version of GNU make on a bunch of different
> real codebases and make sure it still works (and of course, create the
> above-mentioned regression tests).

Will do. I tried on 100 simple packages, but most of them are resursive
makefiles (which I missed): 'toenv = 0' effectively disabled the option
very early. Will try a few crafted inputs and handpick projects with
handwritten build systems.

> First, I think it's not correct to shuffle the goaldeps.  The goals
> that are specified on the command line should always be invoked in the
> order that the user requested.  It would be bad to convert:
> 
>   make clean all install
> 
> to:
> 
>   make install clean all

Just to clarify for myself: I agree changing the order of MAKECMDGOALS is
unexpected and harmful as it's observable in rules definitions (like
'echo $MAKECMDGOALS'). But I also think reordering rule execution should
be fine as it would be close to effect of -j in:

make -j clean all install

Does it sound about right? Or there is some subtler difference between goal
order and prerequisite order treatment?

> Second, you cannot rearrange the first prerequisite.  The first
> prerequisite always must be placed in $<.  Consider the simple pattern
> rule:
> 
>   %.o : %.c
>   $(CC) $(CFLAGS) -c -o $@ $<
> 
>   foo.o: foo.c foo.h bar.h baz.h
> 
> If you rearrange the prerequisites to "bar.h foo.h foo.c baz.h" it will
> not work well :).
> 
> Lastly, I'm not entirely sure that this is the best way to do the
> shuffle.  Similar to my concern above about the first prerequisite,
> this change will break makefiles that rely on the order of
> prerequisites in perfectly legitimate ways; for example:
> 
>   foo%: arg%1 arg%2 arg%3 arg%4
>   bld $< $(word 3,$^) $(word 2,$^) $(word 4,$^)
> 
> The concept you want to implement is not the shuffling of the actual
> prerequisites, it's the shuffling of the BUILDING of the prerequisites.
> The list of deps should not be modified but instead when make goes to
> build the deps it should build them in a different order than they
> appear in the prerequisites list.
> 
> Put another way, you don't want to change the structure of make's
> dependency graph; you just want to change the order in which it's
> walked.
> 
> If you do it this way you don't have to worry about any of the
> reordering issues I raise above, because the values of $<, $^, etc.
> won't change, and also you won't have to worry about deep recursions
> etc. because you'll just be rearranging the targets that are being
> built.
> 
> But, I think making it work this way will be trickier to code.

Oh, I did not realize prerequisite reordering completely breaks rule
definitions! That makes sense.

I'll spend some time to understand where I can plug in to reshuffle
schedulable queue instead.

Ideally I would like to preserve the order of execution across
incremental runs of:

make --shuffle=$seed foo
make --shuffle=$seed foo

but was afraid of the change of already satisfied prerequisites. Might
have to abandon the ideal for simplest first implementation.

Thank you for the detailed comment!

-- 

  Sergei