Re: Alpine: useless-if-before-free: Exec format error

[Bruno Haible]

Paul Eggert wrote:
> Perhaps we 
> should just prepend "#!/bin/sh" or "#!/usr/bin/env perl" (not sure 
> which) and let people on weird systems fix things by hand.

"#!/usr/bin/env perl" does not work on GuixSD (where the only program
that has a hardcoded file name is /bin/sh; there is no /usr and no
/bin/env on this distro).

Naively prepending "#!/bin/sh" leads to this (with perl 5.22.1):

$ build-aux/useless-if-before-free
/bin/sh: 0: Illegal option -w

$ strace -f build-aux/useless-if-before-free 2>&1 | grep ^exec
execve("build-aux/useless-if-before-free", 
["build-aux/useless-if-before-free"], [/* 79 vars */]) = 0
execve("/home/bruno/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/home/bruno/local/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/darch/x86_64-linux-gnu/gnu/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/arch/local/x86_64-linux-gnu/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/arch/local/x86_64-linux/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/usr/local/sbin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/usr/local/bin/perl", ["perl", "-wST", 
"build-aux/useless-if-before-free"], [/* 79 vars */]) = -1 ENOENT (No such file 
or directory)
execve("/usr/sbin/perl", ["perl", "-wST", "build-aux/useless-if-before-free"], 
[/* 79 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/bin/perl", ["perl", "-wST", "build-aux/useless-if-before-free"], 
[/* 79 vars */]) = 0
execve("/bin/sh", ["/bin/sh", "-wST", "build-aux/useless-if-before-free"], [/* 
79 vars */]) = 0

So, what you can see is that the redirect from /bin/sh to perl works right.
But perl then sees the first line and redirects back to /bin/sh. In other
words, there is logic in perl that amounts to "I refuse to be executed
through scripts that start with '#!/bin/sh'".

This finding is confirmed by the following comment in the perl source code:
/* ALTERNATE_SHEBANG:
 *  This symbol, if defined, contains a "magic" string which may be used
 *  as the first line of a Perl program designed to be executed directly
 *  by name, instead of the standard Unix #!.  If ALTERNATE_SHEBANG
 *  begins with a character other then #, then Perl will only treat
 *  it as a command line if it finds the string "perl" in the first
 *  word; otherwise it's treated as the first line of code in the script.
 *  (IOW, Perl won't hand off to another interpreter via an alternate
 *  shebang sequence that might be legal Perl code.)
 */

I propose this solution:

  - Rename 'useless-if-before-free' to 'useless-if-before-free.pl',
removing the first 3 lines, and removing the execute permission.
Also update the reference to ME in the --help output.

  - Add a shell script useless-if-before-free that merely does this:

 #!/bin/sh
 exec perl "$0".pl "$@"

  - Update the module description, to list both
build-aux/useless-if-before-free and build-aux/useless-if-before-free.pl.

And likewise for the other programs:

  build-aux/gitlog-to-changelog
  build-aux/update-copyright
  build-aux/announce-gen
  build-aux/prefix-gnulib-mk
  tests/test-update-copyright.sh

Bruno

Re: Alpine: useless-if-before-free: Exec format error

[Paul Eggert]

On 6/14/19 4:51 PM, Bruno Haible wrote:

But POSIX does not specify that the
command interpreter for scripts without shebang is /bin/sh; therefore IMHO
it would be good not to make this assumption.


Perhaps the code in those scripts was taken from an earlier version of 
the output of "perldoc perlrun". The current version 
 gives several alternative script 
headers (including at least one that appears to be a joke) and say that 
none work everywhere. It is a bit of a portability mess. Perhaps we 
should just prepend "#!/bin/sh" or "#!/usr/bin/env perl" (not sure 
which) and let people on weird systems fix things by hand.

Re: Alpine: useless-if-before-free: Exec format error

[Bruno Haible]

>Since xargs happens to use execvp(), the executable gets executed by "a
>known command interpreter". This command interpreter might be sh, python,
>emacs, or whatever.
> 
> So, the *omission* of #! renders a script non-portable, when that script is
> invoked like an executable.

Anyway, there seems to be an agreement that execvp() needs to execute scripts
through a command interpreter, unlike execv().

  * The glibc source code does so, see glibc/posix/execvpe.c.
Also the description of this commit:

https://sourceware.org/git/?p=glibc.git;a=commit;h=283d98512272a12cb84e7798c23edbdf1adb287d

  * The musl author agrees as well:
https://www.openwall.com/lists/musl/2018/03/09/1
https://www.openwall.com/lists/musl/2018/03/09/2
https://www.openwall.com/lists/musl/2018/03/11/1

So, the ENOEXEC error is a bug in musl. But POSIX does not specify that the
command interpreter for scripts without shebang is /bin/sh; therefore IMHO
it would be good not to make this assumption.

Bruno

Re: Alpine: useless-if-before-free: Exec format error

[Bruno Haible]

Eric Blake wrote:
> Conversely, POSIX requires that execution by 'sh' is the expected
> behavior when #! is missing, and that use of #! renders a script
> non-portable:
> 
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_01

This paragraph specifies what happens when you run
  sh some-file

Here, maint.mk invokes xargs, which invokes 'useless-if-before-free'.
xargs is not a shell. Therefore the portions of POSIX that matter here are

1) The description of "Executable File" in section 3.154 of
   https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html
   "The internal format of an executable file is unspecified"

2) The description of the exec() system call:
   https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html
   "Historically, there have been two ways that implementations can exec shell
scripts.
One common historical implementation is that the execl(), execv(), execle(),
and execve() functions return an [ENOEXEC] error for any file not
recognizable as executable, including a shell script. When the execlp() and
execvp() functions encounter such a file, they assume the file to be a shell
script and invoke a known command interpreter to interpret such files. This
is now required by POSIX.1-2017. ..."

   Since xargs happens to use execvp(), the executable gets executed by "a
   known command interpreter". This command interpreter might be sh, python,
   emacs, or whatever.

So, the *omission* of #! renders a script non-portable, when that script is
invoked like an executable.

Bruno

Re: Alpine: useless-if-before-free: Exec format error

[Eric Blake]

On 6/14/19 4:28 PM, Bruno Haible wrote:
> Tim Rühsen wrote:
>> Do you think it makes more sense to open a bug at busybox then ?
> 
> I don't think so. Execution of scripts without shebang is considered
> legacy. Quoting  :
> 
>   "Note that, even in systems with full kernel support for the #! magic 
> number,
>some scripts lacking interpreter directives (although usually still
>requiring execute permission) are still runnable by virtue of the legacy
>script handling of the Bourne shell, still present in many of its modern
>descendants. Scripts are then interpreted by the user's default shell."
> 

Conversely, POSIX requires that execution by 'sh' is the expected
behavior when #! is missing, and that use of #! renders a script
non-portable:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_01
"The shell reads its input from a file (see sh), from the -c option or
from the system() and popen() functions defined in the System Interfaces
volume of POSIX.1-2017. If the first line of a file of shell commands
starts with the characters "#!", the results are unspecified."

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Alpine: useless-if-before-free: Exec format error

[Bruno Haible]

Tim Rühsen wrote:
> Do you think it makes more sense to open a bug at busybox then ?

I don't think so. Execution of scripts without shebang is considered
legacy. Quoting  :

  "Note that, even in systems with full kernel support for the #! magic number,
   some scripts lacking interpreter directives (although usually still
   requiring execute permission) are still runnable by virtue of the legacy
   script handling of the Bourne shell, still present in many of its modern
   descendants. Scripts are then interpreted by the user's default shell."

Bruno

Re: Alpine: useless-if-before-free: Exec format error

[Tim Rühsen]

On 14.06.19 19:50, Paul Eggert wrote:
> On 6/14/19 10:04 AM, Tim Rühsen wrote:
>> Adding a shebang to build-aux/useless-if-before-free fixes the issue.
> 
> What shebang should that be, exactly? The current style is used in
> several scripts (build-aux/announce-gen, build-aux/gitlog-to-changelog,
> build-aux/prefix-gnulib-mk, build-aux/update-copyright,
> build-aux/useless-if-before-free) and presumably works in a wide variety
> of systems. Adding a shebang might break one of those systems unless
> it's done carefully.

I am not an expert in this area, just mentioned the shebang to give a
hint. Alpine / busybox is a pretty common system and it would be nice to
see 'make syntax-check' work there as well.
Do you think it makes more sense to open a bug at busybox then ?

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: Alpine: useless-if-before-free: Exec format error

[Paul Eggert]

On 6/14/19 10:04 AM, Tim Rühsen wrote:

Adding a shebang to build-aux/useless-if-before-free fixes the issue.


What shebang should that be, exactly? The current style is used in 
several scripts (build-aux/announce-gen, build-aux/gitlog-to-changelog, 
build-aux/prefix-gnulib-mk, build-aux/update-copyright, 
build-aux/useless-if-before-free) and presumably works in a wide variety 
of systems. Adding a shebang might break one of those systems unless 
it's done carefully.

Alpine: useless-if-before-free: Exec format error

[Tim Rühsen]

Hi,

on Alpine, with pretty much latest gnulib commit
5905d8ca9945f0d60ff40eb6cfa42afc0199ab8f, 'make syntax-check' throws out

xargs: ./build-aux/useless-if-before-free: Exec format error

Adding a shebang to build-aux/useless-if-before-free fixes the issue.

Regards, Tim



signature.asc
Description: OpenPGP digital signature