Re: GCC build-path patch getting blocked
Vagrant Cascadian: > On 2017-08-16, Ximin Luo wrote: >> It looks like the GCC reviewer that looked at my patch this time >> around, really doesn't like environment variables. They seem to be >> happy to support the variable (including the syntax) as a command-line >> flag however. > >> The original patch fixed ~1800 packages, which were unreproducible due >> to a combination of (a) __FILE__, (b) CFLAGS et al being embedded into >> the output, and (c) packages/upstreams not honoring CFLAGS in the >> first place, and (d) possibly other reasons. > ... >> For these reasons, I have the following proposal, as a work around for the >> time being: >> >> 1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this >> will at least fix packages affected by (a). > > What about a compromise, patching GCC to support a commandline flag > "--respect-build-path-prefix-map-variable" that tells it to respect the > BUILD_PATH_PREFIX_MAP as an environment variable? > > The commandline flag could essentially be a boolean, and would fix (a) > as well as (b). > > Or maybe GCC is just fundamentally opposed to environment variables at > all? > > > This is perhaps a correlary to the additional variable added to some of > the latex toolchain that basically said "really, use SOURCE_DATE_EPOCH > for the current time please, please, really" > Thanks for the suggestion. I think it's unlikely but I'll try it and if it doesn't work, do what I said originally. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
Dan Kegel: > On Mon, Sep 4, 2017 at 9:26 PM, Vagrant Cascadian wrote: >> Or maybe GCC is just fundamentally opposed to environment variables at >> all? > > Not completely: > https://gcc.gnu.org/onlinedocs/cpp/Environment-Variables.html > does mention SOURCE_DATE_EPOCH :-) > - Dan > We got lucky with the particular reviewer on that one. The current reviewers for this patch say they didn't like that that one (SOURCE_DATE_EPOCH) was accepted either. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
On Mon, Sep 4, 2017 at 9:26 PM, Vagrant Cascadian wrote: > Or maybe GCC is just fundamentally opposed to environment variables at > all? Not completely: https://gcc.gnu.org/onlinedocs/cpp/Environment-Variables.html does mention SOURCE_DATE_EPOCH :-) - Dan ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: GCC build-path patch getting blocked
On 2017-08-16, Ximin Luo wrote: > It looks like the GCC reviewer that looked at my patch this time > around, really doesn't like environment variables. They seem to be > happy to support the variable (including the syntax) as a command-line > flag however. > The original patch fixed ~1800 packages, which were unreproducible due > to a combination of (a) __FILE__, (b) CFLAGS et al being embedded into > the output, and (c) packages/upstreams not honoring CFLAGS in the > first place, and (d) possibly other reasons. ... > For these reasons, I have the following proposal, as a work around for the > time being: > > 1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this > will at least fix packages affected by (a). What about a compromise, patching GCC to support a commandline flag "--respect-build-path-prefix-map-variable" that tells it to respect the BUILD_PATH_PREFIX_MAP as an environment variable? The commandline flag could essentially be a boolean, and would fix (a) as well as (b). Or maybe GCC is just fundamentally opposed to environment variables at all? This is perhaps a correlary to the additional variable added to some of the latex toolchain that basically said "really, use SOURCE_DATE_EPOCH for the current time please, please, really" live well, vagrant signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
Ximin, > I don't want to have to patch 1800 packages with specific logic (Obviously...) Have we truly exhausted the possibility of getting this patch carried by Debian's GCC? Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
Chris Lamb: > Hi Holger et. al., > >>> This idea is similar to hardening-wrapper. That is now deprecated, but was >>> useful as a stepping-stone to more "proper" fixes. Likewise, this shouldn't >>> be thought of as "the proper fix", [...] >> >> when reading this at first, I didnt like the idea of working it around as you >> described > > Same.. :( > >> […] but then you have a valid point with hardening-wrapper > > Hm, I'm not entirely convinced. Whilst h-w *did* exist, it not only stuck > around far too long, I fear that a parallel reproducibile-wrapper will stick > around even longer and be even harder still to eventually deprecate as it > will be used a lot more due to Policy pushes and/or general buzz around > getting one's packages reproducible. > > It's also — and this should go without saying — unspeakably ugly! > Obviously, my ideal clean solution was the original GCC patch using environment variables. Do you have some alternative suggestions to what I proposed? I don't want to have to patch 1800 packages with specific logic, or rewrite existing working build scripts simply to get reproducibility. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
Hi Holger et. al., > > This idea is similar to hardening-wrapper. That is now deprecated, but was > > useful as a stepping-stone to more "proper" fixes. Likewise, this shouldn't > > be thought of as "the proper fix", [...] > > when reading this at first, I didnt like the idea of working it around as you > described Same.. :( > […] but then you have a valid point with hardening-wrapper Hm, I'm not entirely convinced. Whilst h-w *did* exist, it not only stuck around far too long, I fear that a parallel reproducibile-wrapper will stick around even longer and be even harder still to eventually deprecate as it will be used a lot more due to Policy pushes and/or general buzz around getting one's packages reproducible. It's also — and this should go without saying — unspeakably ugly! Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
Hi Ximin, On Wed, Aug 16, 2017 at 03:40:00PM +, Ximin Luo wrote: > It looks like the GCC reviewer that looked at my patch this time around, > really doesn't like environment variables. They seem to be happy to > support the variable (including the syntax) as a command-line flag however. that sounds more good than bad to me. > For these reasons, I have the following proposal, as a work around for the > time being: > 1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this > will at least fix packages affected by (a). > 2. Add `gcc` (et al) wrappers to strip-nondeterminism: > 3. Add a Makefile snippet to strip-nondeterminism: [... ] > This idea is similar to hardening-wrapper. That is now deprecated, but was > useful as a stepping-stone to more "proper" fixes. Likewise, this shouldn't > be thought of as "the proper fix", [...] when reading this at first, I didnt like the idea of working it around as you described, but then you have a valid point with hardening-wrapper, so I'm now inclined to accept this. What do other think? also: should we have this discussion in/cc:ed to some Debian bug? -- cheers, Holger signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
On Wed 2017-08-16 15:40:00 +, Ximin Luo wrote: > This idea is similar to hardening-wrapper. That is now deprecated, but > was useful as a stepping-stone to more "proper" fixes. Likewise, this > shouldn't be thought of as "the proper fix", but will give us some > useful data on how many packages are affected by which combinations of > (a), (b), (c) or (d). Depending on the number of packages we will have > to patch (with that 2-line patch), it will maybe give weight to future > attempts to have GCC support this env var - and then it would be easy, > since the core functionality would already be in there - or else > highlight the issue so that maintainers of those packages fix things > "properly". This sounds like a frustrating situation, Ximin, but i think your proposal is a good path forward. Thanks for sticking with it. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
GCC build-path patch getting blocked
Hi list,
It looks like the GCC reviewer that looked at my patch this time around, really
doesn't like environment variables. They seem to be happy to support the
variable (including the syntax) as a command-line flag however.
The original patch fixed ~1800 packages, which were unreproducible due to a
combination of (a) __FILE__, (b) CFLAGS et al being embedded into the output,
and (c) packages/upstreams not honoring CFLAGS in the first place, and (d)
possibly other reasons.
Upstream are essentially arguing that fixing (c) is the proper way forward, but
I don't think this is realistic, and unnecessarily couples two separate
problems together - IMO reproducibility is not a CFLAGS issue, see e.g. a voice
of support at [1]. It also doesn't fix (b); the only way to fix (b) is to add
logic to build scripts to strip out a particular command-line flag, in the same
way that we patched GCC to strip it out from DW_AT_producer. I don't think this
is a clean solution either, reproducibility should come "by default" and most
people shouldn't have to add complex logic to their own build scripts.
For these reasons, I have the following proposal, as a work around for the time
being:
1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this will
at least fix packages affected by (a).
2. Add `gcc` (et al) wrappers to strip-nondeterminism:
/usr/share/strip-nondeterminism/bin/gcc
/usr/share/strip-nondeterminism/bin/g++
etc, with contents
#!/bin/sh
exec "$0" --path-prefix-map="${BUILD_PATH_PREFIX_MAP:-}" "$@"
3. Add a Makefile snippet to strip-nondeterminism:
/usr/share/strip-nondeterminism/mk/build-path.mk
with contents
DEB_BUILD_MAINT_OPTIONS += reproducible=-fixdebugpath
export DEB_BUILD_MAINT_OPTIONS
PATH := /usr/share/strip-nondeterminism/bin:$(PATH)
export PATH
Then, fixing a package affected by (b), (c) or (d) will simply consist of:
d/control:
+Build-Depends: strip-nondeterminism
d/rules:
+include /usr/share/strip-nondeterminism/mk/build-path.mk
which will be much much quicker than going in and doing more invasive patches.
This idea is similar to hardening-wrapper. That is now deprecated, but was
useful as a stepping-stone to more "proper" fixes. Likewise, this shouldn't be
thought of as "the proper fix", but will give us some useful data on how many
packages are affected by which combinations of (a), (b), (c) or (d). Depending
on the number of packages we will have to patch (with that 2-line patch), it
will maybe give weight to future attempts to have GCC support this env var -
and then it would be easy, since the core functionality would already be in
there - or else highlight the issue so that maintainers of those packages fix
things "properly".
X
[1] https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00770.html
--
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
