Re: Build path variation and __FILE__
Hi all, > HW42: >> Sascha Steinbiss: >>> a) Introducing something like -ffile-prefix-map similarly to >>> -fdebug-prefix-map to address this issue. There is already an >>> unreviewed patch for gcc submitted by an unrelated party [3] and I >>> could try and raise awareness for this bug with the GCC devs. However, >>> this option would be incompatible with old compilers and hence >>> probably not easily enabled in the default dpkg-buildflags. IMHO it >>> would be a good thing to have though. >> >> If I understood Guillem correctly he is fine with dpkg-buildflags in >> unstable to export compiler flags which are only supported by the >> compilers in unstable. So a new flag would be ok. > > This is my preferred approach as well. On top of [3] we should add a > patch for GCC to use SOURCE_ROOT_DIR[*] for -ffile-prefix-map if it is > not set on the command line. Agreed. I have posted a follow-up messge on the bug tracker, but no response yet. Will also ping the gcc-patches mailing list. The original author submitted a patch [1] which seemed to have been overlooked and/or abandoned. If I can get that discussion going again then we can suggest this addition. Cheers Sascha >>> >>> [1] http://public.kitware.com/pipermail/cmake/2013-January/053114.html >>> [2] https://paste.debian.net/806611 >>> [3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268 >>> [4] http://ohnopub.net/~ohnobinki/bin_replace_string/ >> > -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Build path variation and __FILE__
HW42: > Sascha Steinbiss: >> a) Introducing something like -ffile-prefix-map similarly to >> -fdebug-prefix-map to address this issue. There is already an >> unreviewed patch for gcc submitted by an unrelated party [3] and I >> could try and raise awareness for this bug with the GCC devs. However, >> this option would be incompatible with old compilers and hence >> probably not easily enabled in the default dpkg-buildflags. IMHO it >> would be a good thing to have though. > > If I understood Guillem correctly he is fine with dpkg-buildflags in > unstable to export compiler flags which are only supported by the > compilers in unstable. So a new flag would be ok. > This is my preferred approach as well. On top of [3] we should add a patch for GCC to use SOURCE_ROOT_DIR[*] for -ffile-prefix-map if it is not set on the command line. This would be consistent with our other approaches, and this consistency would help to strengthen the legitimacy of this approach more broadly too. X [*] name TBD, see the other thread "Minimising work needed for this build-path issue". >> >> [1] http://public.kitware.com/pipermail/cmake/2013-January/053114.html >> [2] https://paste.debian.net/806611 >> [3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268 >> [4] http://ohnopub.net/~ohnobinki/bin_replace_string/ > -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Build path variation and __FILE__
Sascha Steinbiss: > Hi all, > > I’d like to contribute to the build path discussion as well with a few > observations I made when trying to fix reproducibility problems > related to that in my packages. In addition to the problem already > raised by Ximin concerning upstream storing CFLAGS in artifacts, most > of the issues I noticed were related to build path capture via > __FILE__. __FILE__ is typically set to the exact source file path > passed to the compiler, and in quite common cases where it is used in > debugging statements etc. it is included as a string in the final > binaries’ rodata section. So if upstream’s build system passes > absolute source paths and __FILE__ is used, the binaries will be > different even with -fdebug-prefix-map set :( This problem > particularly shows in build systems that run builds outside the source > tree like CMake, which makes use of absolute paths a lot. There is > apparently an option CMAKE_USE_RELATIVE_PATHS but it is apparently > unreliable [1] and did not help in a single case for me. I’ll take a > look at the list of reproducible packages build-depending on CMake [2] > later and try to learn something. > > Any ideas what to do about that in general? I can imagine that patch > complexity would be increased a lot by trying to address this solely > at the build system level, which also probably isn't always possible. > One might think of a solution closer to the compiler, such as: I agree that fixing this in CMake and co. is not a great solution. > a) Introducing something like -ffile-prefix-map similarly to > -fdebug-prefix-map to address this issue. There is already an > unreviewed patch for gcc submitted by an unrelated party [3] and I > could try and raise awareness for this bug with the GCC devs. However, > this option would be incompatible with old compilers and hence > probably not easily enabled in the default dpkg-buildflags. IMHO it > would be a good thing to have though. If I understood Guillem correctly he is fine with dpkg-buildflags in unstable to export compiler flags which are only supported by the compilers in unstable. So a new flag would be ok. > b) Using something like libelf or bin_replace_string [4] to remove or > replace the build path in the resulting binary, e.g. as part of > strip-nondeterminism. Not a permanent solution but would do the job. I wold recommend against this approach. On the one hand side the concept of patching the binaries after the build is not nice. On the practical side I would expect quite some failures. The other post-processing of strip-nondeterminism already created some problems. So I would be careful with patching binaries in a large portion of packages. > c) Any other ideas? I think we should try to convince GCC upstream of a solution (new flag or debug-prefix-map or something else). And if we can convince GCC upstream of a SOURCE_ROOT environment variable or something similar the flags can be automatically set. > Cheers > Sascha > > [1] http://public.kitware.com/pipermail/cmake/2013-January/053114.html > [2] https://paste.debian.net/806611 > [3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268 > [4] http://ohnopub.net/~ohnobinki/bin_replace_string/ signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Build path variation and __FILE__
Hi all, I’d like to contribute to the build path discussion as well with a few observations I made when trying to fix reproducibility problems related to that in my packages. In addition to the problem already raised by Ximin concerning upstream storing CFLAGS in artifacts, most of the issues I noticed were related to build path capture via __FILE__. __FILE__ is typically set to the exact source file path passed to the compiler, and in quite common cases where it is used in debugging statements etc. it is included as a string in the final binaries’ rodata section. So if upstream’s build system passes absolute source paths and __FILE__ is used, the binaries will be different even with -fdebug-prefix-map set :( This problem particularly shows in build systems that run builds outside the source tree like CMake, which makes use of absolute paths a lot. There is apparently an option CMAKE_USE_RELATIVE_PATHS but it is apparently unreliable [1] and did not help in a single case for me. I’ll take a look at the list of reproducible packages build-depending on CMake [2] later and try to learn something. Any ideas what to do about that in general? I can imagine that patch complexity would be increased a lot by trying to address this solely at the build system level, which also probably isn't always possible. One might think of a solution closer to the compiler, such as: a) Introducing something like -ffile-prefix-map similarly to -fdebug-prefix-map to address this issue. There is already an unreviewed patch for gcc submitted by an unrelated party [3] and I could try and raise awareness for this bug with the GCC devs. However, this option would be incompatible with old compilers and hence probably not easily enabled in the default dpkg-buildflags. IMHO it would be a good thing to have though. b) Using something like libelf or bin_replace_string [4] to remove or replace the build path in the resulting binary, e.g. as part of strip-nondeterminism. Not a permanent solution but would do the job. c) Any other ideas? Cheers Sascha [1] http://public.kitware.com/pipermail/cmake/2013-January/053114.html [2] https://paste.debian.net/806611 [3] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70268 [4] http://ohnopub.net/~ohnobinki/bin_replace_string/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
