Re: [Reproducible-builds] Bug#806331: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

[Thorsten Glaser]

Ximin Luo dixit:

>needs to more clearly distinguish between the build and the host
>environment - like how compilers do. So for example, here the "most
>correct" solution would be to add a HOST_POSIX_SHELL and default this

No, this is outside of the scope of autotools and a common misuse
of them actually.

The real bug here is that the configure.ac script of the package
hardcodes POSIX_SHELL in the output while your reproducible builds
effort treats this as a(n unsupported, at least in this scenario)
cross compilation.

The same issue would happen when you set CONFIG_SHELL=/bin/bash
and then cross-compile xz-utils from GNU/Linux to, say, MirBSD,
where There Is No Such Thing As /bin/bash.

So the root of the problem is that xz-utils is not cross-compilable
(well, not completely, and enough for Helmut’s bootstrapping effort).

bye,
//mirabilos
-- 
Yay for having to rewrite other people's Bash scripts because bash
suddenly stopped supporting the bash extensions they make use of
-- Tonnerre Lombard in #nosec

Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

[Thorsten Glaser]

Ximin Luo dixit:

>(Did you mean to add debian-bugs-dist and Jonathan Nieder on purpose
>or by accident? I removed them, but feel free to add them back in.)

I didn’t, maybe debbugs did.

>In other words: in your scenario, one would not use $POSIX_SHELL but
>some other specific test for those "other bugs".

No, you don’t understand what I tried to write. The thing is, this
is about the *user* (the person calling the configure script) being
able to *override* the automatic mechanism to determine POSIX_SHELL
by setting CONFIG_SHELL to some value, forcing it to use it (if it
passes the tests).

Removing this ability will be A Bad Thing™.

bye,
//mirabilos
-- 
18:47⎜ well channels… you see, I see everything in the
same window anyway  18:48⎜ i know, you have some kind of
telnet with automatic pong 18:48⎜ haha, yes :D
18:49⎜ though that's more tinyirc – sirc is more comfy

Re: [Reproducible-builds] Bug#806331: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

[Ximin Luo]

Paul Eggert:
> On 06/15/2016 01:44 PM, Ximin Luo wrote:
>> In such a case, it is a bug to be using $POSIX_SHELL - which only tests for 
>> conformance with POSIX and not these "other bugs that make it unusable".
> Gnulib can't test for all POSIX violations, only for the ones it knows about. 
> CONFIG_SHELL lets the user override Gnulib's guess in environments where the 
> guess is wrong. This sort of thing has been in Gnulib (and Autoconf) for 
> ages, I expect many people have grown used to it, and I'm leery of changing 
> this just for the purpose of reproducible builds. For reproducible builds, I 
> suggest configuring with CONFIG_SHELL=/bin/sh as that should make the build 
> reproducible without having to change Autoconf or Gnulib.
> 

OK, thanks for the explanation. I was under the impression that, since POSIX is 
a finite set of specified behaviours, posix-shell would be or could be a 
complete test. But I guess it's harder than I thought, to write that sort of 
test.

> More generally, 'configure' and reproducible builds are competing objectives. 
> 'configure' aims to guess characteristics of the target environment by 
> depending on details of the build environment; in contrast, reproducible 
> builds want to suppress details of the build environment whenever possible. 
> Probably the best way to marry these two is for the reproducible build to 
> start with a reproducible environment, and setting CONFIG_SHELL to a known 
> value is one step in that direction.

I wouldn't say they're necessarily competing, just that the build process (e.g. 
'configure' in this case) needs to more clearly distinguish between the build 
and the host environment - like how compilers do. So for example, here the 
"most correct" solution would be to add a HOST_POSIX_SHELL and default this to 
POSIX_SHELL (or something). But this is also more complex, so I can understand 
if you don't want to do this.

Thanks for the tip, we'll try the other options.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

[Thorsten Glaser]

Ximin Luo dixit:

>I'm not sure if you understood what was being discussed.

I understand it extremely well.

>proposed patch affect your scenario? This is not about CONFIG_SHELL,

It is. Straight from your diff:

>  for gl_cv_posix_shell in \
>-"$CONFIG_SHELL" "$SHELL" /bin/sh /bin/bash /bin/ksh /bin/sh5 no; do
>+/bin/sh /bin/bash /bin/ksh /bin/sh5 "$CONFIG_SHELL" "$SHELL" no; do

This makes $CONFIG_SHELL no longer the preferred POSIX shell.

This breaks when a system has a /bin/sh that passes all gnulib
tests (and thus is used as POSIX_SHELL) but has other bugs that
make it unusable, for example. The user can then no longer over‐
ride this bad choice of the configury.

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

[Ximin Luo]

+bugs-gnulib, reproducible-builds

Lasse Collin:
> On 2016-06-07 Ximin Luo wrote:
>> I've attached a patch that makes m4/posix-shell.m4 try constant paths
>> first. This should fix the issue.
>>
>> Upstream should also apply it - see more-stable-shell.patch.
> 
> Thanks!
> 
> posix-shell.m4 comes from gnulib so it would be nice if you could send
> the patch there and discuss the issue with gnulib developers. Perhaps
> there is a reason why the shells are tested in that order, although the
> gnulib commit messages don't have any reasoning. A guess is that
> someone might prefer if the same shell is used for running
> configure and the test suite scripts. Anyway, getting it fixed in gnulib
> would get it fixed in projects other than XZ Utils too.
> 

Thanks, yes this wasn't clear to me. We'd still need to contact projects that 
have already copied posix-shell.m4 into their source tree, but I suppose at 
least future projects will benefit.

bugs-gnulib, do you see any issue with this patch? The context is that some 
projects embed POSIX_SHELL into build products, so for build reproducibility it 
is better to have this detection script first try constant paths.

(Yes, if /bin/sh is not POSIX for one of the build machines, then we still have 
unreproducibility. But that's much less likely to happen. This is an "effort vs 
correctness" trade-off that I'm making.)

Regarding "someone might prefer if the same shell" as Lasse theorised, I'm not 
sure this will ever be an issue: if they prefer "the same shell", they would 
just use $SHELL or $CONFIG_SHELL instead of $POSIX_SHELL, so the only useful 
scenario is if somehow they wanted a shell that was POSIX-compatible but for 
some reason (I can't imagine why) needed this selection to favour 
$SHELL/$CONFIG_SHELL but still for it to be OK to fall back to the other 
constant values.

> One can force the POSIX shell to a specific value on the configure
> command line by passing, for example, "gl_cv_posix_shell=/bin/sh" as an
> argument. It's not documented in the --help output but it's mentioned
> in INSTALL section 3.1. That is an alternative to patching to get
> reproducible builds.
> 

Yeah, I saw that too, but thought this approach was a bit cleaner.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git
Description: More stable shell in build output
 When detecting a POSIX shell, try constant systems paths first because that is
 more likely to give a reproducible build output.
Author: Ximin Luo 
Bug: https://www.mail-archive.com/xz-devel@tukaani.org/
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806331
---
This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
--- a/m4/posix-shell.m4
+++ b/m4/posix-shell.m4
@@ -43,7 +43,7 @@
subshell_umask_sanity
  '
  for gl_cv_posix_shell in \
-"$CONFIG_SHELL" "$SHELL" /bin/sh /bin/bash /bin/ksh /bin/sh5 no; do
+/bin/sh /bin/bash /bin/ksh /bin/sh5 "$CONFIG_SHELL" "$SHELL" no; do
case $gl_cv_posix_shell in
  /*)
   "$gl_cv_posix_shell" -c "$gl_test_posix_shell_script" 2>/dev/null \