[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Paul D. Smith]

Update of bug #53201 (project make):

  Status:None => Works for me   
 Open/Closed:Open => Closed 

___

Follow-up Comment #5:

I re-examined the code including the code for 3.81 (the reported version) and
can't see any way that make itself could be invoking execvp() multiple times
for the same command.

You mentioned using .ONESHELL to fix the issue but that feature was not
available in GNU make version 3.82 so maybe you tried a newer version as
well?

If you can still reproduce this with the latest GNU make please add a comment
providing more information as requested below, and also information on what
operating system you're using (your use of GNU make 3.81 in 2018 makes me
wonder if you're on Mac OSX and using the version that comes with Xcode, which
has been modified by Apple and we've seen other odd behavior with it).

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Paul D. Smith]

Follow-up Comment #4, bug #53201 (project make):

I don't understand the issue being reported.  It would be greatly illuminating
if you could provide an example of the extra-long shebang line for me to
examine.  I've tried for a bit to reproduce the problem and cannot.  It's not
clear from your report whether it's the path to the interpreter which is too
long, or whether it's an argument to the interpreter which is too long.

The situation cannot be as simple as you suggest.  First, the exec operation
that make invokes is in a forked process (obviously) so even if we were to try
to catch the ENOENT error, we couldn't do anything about it in the main make
process.  It doesn't know anything about failures during exec in its child
process (it only knows when the child process exits, whether it had an error
or not).

Second, make itself never tries to parse the shebang line in a script.  It
merely runs the program as provided to it.  Make uses the execvp() function in
the forked process to invoke the command and this function performs PATH
lookup then invokes execve() (according to the GNU/Linux man page, these
functions are wrappers around execve()).

If the way make is invoking processes and the way the shell invoke them are
different then I can only put it down to differences in the way that the shell
and the execvp() function in GNU libc treat interpreter scripts.  I'm not sure
I can see what GNU make should do about this difference.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Casey McGinty]

Interesting observation but I don’t follow how that is related to this
issue. The interpreter is not yet running in this case.


On March 17, 2018 at 10:12:54 AM, Brian Vandenberg (
phantall+gnum...@gmail.com) wrote:

  I cannot remark on pip specifically, but some script interpreters seem to
ignore the shebang line if the script is passed as an argument:

$ cat /tmp/blah.sh
#!/bin/bash
printf( "%s:%d\n", __FILE__, __LINE__ );
$ /tmp/blah.sh
/tmp/blah.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/blah.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'
$ perl /tmp/blah.sh
/tmp/blah.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/blah.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'

>From the above it's apparent that perl (without extra args) obeys the
shebang line; ditto for ruby (example omitted).  This next example
demonstrates that bash (versions 3.2 and 4.3.33) does not:

$ cat /tmp/stuff.sh
#!/usr/bin/perl
printf( "%s:%d\n", __FILE__, __LINE__ );
$ /tmp/stuff.sh
/tmp/stuff.sh:2
$ bash /tmp/stuff.sh
/tmp/stuff.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/stuff.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'

I've omitted the text, but I ran the same test with sh, csh, tcsh, zsh and
python.

In summary, when the script is passed as an argument to interpreter:

obeyed shebang: perl, ruby
ignored shebang: sh, csh, tcsh, zsh, bash, python

-brian


On Thu, Feb 22, 2018 at 6:16 AM, David Boyce 
wrote:

> On Thu, Feb 22, 2018 at 4:42 AM,  wrote:
>
>> Casey McGinty writes:
>>  > Follow-up Comment #2, bug #53201 (project make):
>>  >
>>  > See http://man7.org/linux/man-pages/man2/execve.2.html
>>  >
>>  > *A maximum line length of 127 characters is allowed for the first line
>> in an
>>  > interpreter script.*
>>
>>  I think this is referring to the '#!' line of a script, not internal
>>  kernel limits.  This limitation should not affect Gnu Make recipes.
>>
>
> Well, the #! limit is a kernel limit too. But your point that it shouldn't
> affect GNU make recipes (directly) is a good one since make invokes recipes
> via "/bin/sh -c recipe" without ever using a shebang. I have to admit I'm a
> little lost on where the shebang limit is coming in here but it does seem
> to be somehow.
>
>
> ___
> Bug-make mailing list
> Bug-make@gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-make
>
>
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Brian Vandenberg]

  I cannot remark on pip specifically, but some script interpreters seem to
ignore the shebang line if the script is passed as an argument:

$ cat /tmp/blah.sh
#!/bin/bash
printf( "%s:%d\n", __FILE__, __LINE__ );
$ /tmp/blah.sh
/tmp/blah.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/blah.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'
$ perl /tmp/blah.sh
/tmp/blah.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/blah.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'

>From the above it's apparent that perl (without extra args) obeys the
shebang line; ditto for ruby (example omitted).  This next example
demonstrates that bash (versions 3.2 and 4.3.33) does not:

$ cat /tmp/stuff.sh
#!/usr/bin/perl
printf( "%s:%d\n", __FILE__, __LINE__ );
$ /tmp/stuff.sh
/tmp/stuff.sh:2
$ bash /tmp/stuff.sh
/tmp/stuff.sh: line 2: syntax error near unexpected token `"%s:%d\n",'
/tmp/stuff.sh: line 2: `printf( "%s:%d\n", __FILE__, __LINE__ );'

I've omitted the text, but I ran the same test with sh, csh, tcsh, zsh and
python.

In summary, when the script is passed as an argument to interpreter:

obeyed shebang: perl, ruby
ignored shebang: sh, csh, tcsh, zsh, bash, python

-brian


On Thu, Feb 22, 2018 at 6:16 AM, David Boyce 
wrote:

> On Thu, Feb 22, 2018 at 4:42 AM,  wrote:
>
>> Casey McGinty writes:
>>  > Follow-up Comment #2, bug #53201 (project make):
>>  >
>>  > See http://man7.org/linux/man-pages/man2/execve.2.html
>>  >
>>  > *A maximum line length of 127 characters is allowed for the first line
>> in an
>>  > interpreter script.*
>>
>>  I think this is referring to the '#!' line of a script, not internal
>>  kernel limits.  This limitation should not affect Gnu Make recipes.
>>
>
> Well, the #! limit is a kernel limit too. But your point that it shouldn't
> affect GNU make recipes (directly) is a good one since make invokes recipes
> via "/bin/sh -c recipe" without ever using a shebang. I have to admit I'm a
> little lost on where the shebang limit is coming in here but it does seem
> to be somehow.
>
>
> ___
> Bug-make mailing list
> Bug-make@gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-make
>
>
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[David Boyce]

Follow-up Comment #3, bug #53201 (project make):

Sorry, I was thinking of a different kernel limit but my second observation,
that the difference observed has to do with fast path optimization, remains. 
Working around it by setting SHELL=//bin/sh in the makefile is probably safer
than .ONESHELL. Again, discussion relating to the fix per se belongs to Paul.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[thutt]

Casey McGinty writes:
 > Follow-up Comment #2, bug #53201 (project make):
 > 
 > See http://man7.org/linux/man-pages/man2/execve.2.html
 > 
 > *A maximum line length of 127 characters is allowed for the first line in an
 > interpreter script.*

 I think this is referring to the '#!' line of a script, not internal
 kernel limits.  This limitation should not affect Gnu Make recipes.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Casey McGinty]

Follow-up Comment #2, bug #53201 (project make):

See http://man7.org/linux/man-pages/man2/execve.2.html

*A maximum line length of 127 characters is allowed for the first line in an
interpreter script.*

My kernel is Linux 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux

This is a production system and can not be updated, however I don't see how
that would help.

The kernel define is still set to 127:
https://github.com/torvalds/linux/blob/master/include/uapi/linux/binfmts.h#L19

If you think this is incorrect, please explain where my misunderstanding is.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[David Boyce]

Follow-up Comment #1, bug #53201 (project make):

I think a couple of facts here may need correcting though the issue is quite
real. I suspect you are not running into a kernel limit since modern Linux
kernels have no such limit (at one time the limit was 131K, then doubled to
262K, then removed altogether). However, the mere fact that the kernel imposes
no limit doesn't mean individual utilities, in particular the shell, have the
same property.

Most likely the limit being exceeded belongs to bash. The so-called fast-path
optimization in make skips the shell and execs the recipe directly if it
contains no shell special characters, thus in your original test case the
shell isn't used which explains the variance. Use of .ONESHELL suppresses
fast-path causing the shell to come back in and fail properly.

There are other ways of suppressing fast path to test this thesis; try
appending a semicolon to the recipe line or overriding the SHELL variable. One
of my favorite tricks is SHELL=//bin/sh which addresses the same executable
while convincing make it's different, thus disabling fast path.

I have no wisdom to offer regarding the fix. Your solution sounds fine but
that belongs to Paul.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #53201] Target runs incorrect command when shebang line exceeds kernel limit

[Casey McGinty]

URL:
  

 Summary: Target runs incorrect command when shebang line
exceeds kernel limit
 Project: make
Submitted by: cmcginty
Submitted on: Wed 21 Feb 2018 07:35:13 AM UTC
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 3.81
Operating System: POSIX-Based
   Fixed Release: None
   Triage Status: None

___

Details:

I have a Python script (in ./env/bin/pip) generated by virtualenv with a very
long shebang line. The virtualenv is active and the PATH env contains
"./env/bin;" before all other paths. When I run "pip" from the system
shell I see the following error:

```
... /bin/python: can't open file 'pip': [Errno 2] No such file or directory
```

This error is expected because the shebang line is beyond the limit allowed by
the kernel.

However when the same command runs in Make, it does NOT fail. Instead, it
succeeds by running /bin/pip. This is very unusual since the local script
`./env/bin/pip` is first in the PATH . The problem with running the system pip
is that this should never happen as the local environment thinks it is inside
of Python virtualenv. By running the system pip, we are tainting the system
Python runtime environment with unwanted packages.

Some debugging with strace revealed exactly what Make is doing:

```
[pid  8578] execve("/var/lib/jenkins/workspace//env/bin/pip", ["pip",
"list"], [/* 26 vars */]) = -1 ENOENT (No such file or directory)
[pid  8578] execve("/sbin/pip", ["pip", "list"], [/* 26 vars */]) = -1 ENOENT
(No such file or directory)
[pid  8578] execve("/bin/pip", ["pip", "list"], [/* 26 vars */] 
```

Note that adding the .ONESHELL directive resolve the issue and causes Make
returns the expected error string above.

I think the proper fix is to stat() the file before accepting the ENOENT error
from execve. If the file exists and execve still returns an ENOENT error, Make
should fail the target and return the error to the user.




___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make