Re: [Reproducible-builds] faketime by default?
Hans-Christoph Steiner: > Thanks for the explanation, that's a bummer that it was so problematic, but I > suppose not surprising. Would it make sense to have something like a "dh > --use-faketime" flag so that maintainers could opt-in to use it where it might > be helpful like when upstream refuses to remove build stamps, etc? The idea > there would be that dh would handle setting up faketime before the upstream > build system was called. In that case, a flag to dh would be a lot less work > to maintain then a set of patches. I am not aware of anyone experimenting the introduction of libfaketime as part of debian/rules. I would assume they would be tricky repercussions (debian/rules targets can be called independently), but that's just a hunch. I don't believe we should forbid using libfaketime in debian/rules if that works out for the maintainers of a given package. I don't think it should be a general solution. It's really not that of a big deal to carry small patches removing or nullifying a couple of timestamps: Git does wonders. I'm basing my thoughts on what I have experimented so far. With the current toolchain modifications [1], out of the 70 core packages that are already using dh, 74% have reproducible builds. For the 18 remaining packages, several could be fixed at once by fixing the documentation generators. For one of the patch I've sent, the issue was pure randomness (so no timestamps), and for the other the build system also captured hostname, username, and uname. This really makes me feel like timestamps are not the hardest issue I'm totally fine to be proven wrong. Everybody, please make more experiments. [1]: dh_strip_nondeterminism, dh_fixmtimes, patches against dpkg for stable file ordering and timestamp based on changelog -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] faketime by default?
Jérémy Bobbio wrote: > Hans-Christoph Steiner: >> I've been using faketime in my work on reproducible builds on Android (both >> "native" C code and Java), and it has been working well. It seems to me that >> the current approach in Debian does not use faketime. >> >> Since so much of the little issues are due to timestamps, it could make a lot >> of sense if faketime was a default part of the process. For the make+C >> builds >> I've done, faketime just froze the clock entirely and that worked. For >> ant+java builds, it would hang forever with a frozen clock. So I set >> faketime >> to run the clock at 5% speed, and that seems to work well. > > Before getting into any technical details, let's start by the social > problem. We are talking about Debian (and derivatives!). There is no > canonical way a Debian package is built. Most developers will, for > example, use `fakeroot`, but it's optional and `sudo` is also supported. > To reproduce builds using libfaketime, you need both the original build > and the later builds to use libfaketime. Requiring that every Debian > developers uses libfaketime when building a package is a change that > would be very hard, if not impossible, to achieve. > > For the most important technical problem: libfaketime cannot make the > build process entirely deterministic. According to my tests, parallel > execution, variations in file order, and randomness can all participate > in making the faked times different from one build to the next. > > While experimenting in my little corner, I also saw several build > systems just fail, or fail to be reproducible with libfaketime. That's > what actually led me to remove the time fixing feature of the > `test-reproducibility` script entirely: > https://anonscm.debian.org/cgit/reproducible/misc.git/commit/?id=f4146ee73d > I really would love to have it back, but it simply could not be made to > work reliably under my fingers. > > There's a consensus that libfaketime is not the good approach for > reproducible builds since the initial BoF during DebConf13. Patching > timestamps is a harder path, but we know there's not going to be a > hidden cliff in the middle of the journey. Thanks for the explanation, that's a bummer that it was so problematic, but I suppose not surprising. Would it make sense to have something like a "dh --use-faketime" flag so that maintainers could opt-in to use it where it might be helpful like when upstream refuses to remove build stamps, etc? The idea there would be that dh would handle setting up faketime before the upstream build system was called. In that case, a flag to dh would be a lot less work to maintain then a set of patches. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] faketime by default?
Hans-Christoph Steiner: > I've been using faketime in my work on reproducible builds on Android (both > "native" C code and Java), and it has been working well. It seems to me that > the current approach in Debian does not use faketime. > > Since so much of the little issues are due to timestamps, it could make a lot > of sense if faketime was a default part of the process. For the make+C builds > I've done, faketime just froze the clock entirely and that worked. For > ant+java builds, it would hang forever with a frozen clock. So I set faketime > to run the clock at 5% speed, and that seems to work well. Before getting into any technical details, let's start by the social problem. We are talking about Debian (and derivatives!). There is no canonical way a Debian package is built. Most developers will, for example, use `fakeroot`, but it's optional and `sudo` is also supported. To reproduce builds using libfaketime, you need both the original build and the later builds to use libfaketime. Requiring that every Debian developers uses libfaketime when building a package is a change that would be very hard, if not impossible, to achieve. For the most important technical problem: libfaketime cannot make the build process entirely deterministic. According to my tests, parallel execution, variations in file order, and randomness can all participate in making the faked times different from one build to the next. While experimenting in my little corner, I also saw several build systems just fail, or fail to be reproducible with libfaketime. That's what actually led me to remove the time fixing feature of the `test-reproducibility` script entirely: https://anonscm.debian.org/cgit/reproducible/misc.git/commit/?id=f4146ee73d I really would love to have it back, but it simply could not be made to work reliably under my fingers. There's a consensus that libfaketime is not the good approach for reproducible builds since the initial BoF during DebConf13. Patching timestamps is a harder path, but we know there's not going to be a hidden cliff in the middle of the journey. -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
