Re: [Reproducible-builds] using disorderfs to build packages
Hi, thanks to everybody for clarifying and especially to Andrew for fixing the remaining concerns in the code! On Donnerstag, 27. August 2015, HW42 wrote: What about doing only the second build in disorderfs with some random sorting. Yes, that's my idea. For that we'll need a trivial backport to jessie for amd64 and armhf. Then we'll just need to find a way to neatly hook this into pbuilder - patches as usual much welcome! ;-) cheers, Holger (it might take some time until I get implementing this on my own... ) signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] using disorderfs to build packages
Chris Lamb: After reading the disorderfs manpage I wonder if it's really already suitable to build everything This is a little awkward for me as I raised these concerns a couple of times during the BoF but I did not want to temper anyone's enthusiasm in the room. I therefore did not labour the point, especially as I was curious to have the tool existing for private tests, ie. even if we did not -- or could not -- introduce it into our regular variations. As it happens, I'm not especially concerned that it will /break/ builds per-se, but rather that any FUSE/LD_PRELOAD/$FOO-based hack makes it more difficult to file bugs against maintainers' packages; the results are either immediately suspect or can be blamed on such hackery. Putting it another way, I would certainly feel justified in replying to such a bug report asking the reporter whether it's reproducible outside of such a tool. What about doing only the second build in disorderfs with some random sorting. When everything works order you have successfully tested reproducibility (including file ordering). When the first build fails it can't be disorderfs' fault. And builds where only the second build fails needs more investigation. signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] using disorderfs to build packages
Andrew Ayer: On Wed, 26 Aug 2015 10:58:08 +0200 HW42 h...@ipsumj.de wrote: Errors from close are not properly reported. Probably don't hurt. Most programs ignore them anyway. I think close() only fails if there's a disk error or you run out of disk space/quota. Neither of these should happen during a normal build. fsync is not implemented. Beside of timing this should not make a difference as long the maschine don't crash. (I think that's what libeatmydata is doing via LD_PRELOAD). And we are building on tmpfs anyway (I don't know if fsync does anything on tmpfs). Right. To be clear, fsync() and fdatasync() still report success with disorderfs - they just don't do anything. xattrs are not implemented. Probably only hurts for packages which need to handle those and have test cases for it. (e.g. rsync, rdiff-backup, ...) I'm guessing that only a minority of packages would be affected. But, implementing xattrs turned out to be trivial, so I just released 0.2.0 with xattr support. The Debian package is ready for someone to upload. nice. Can packages contain files with xattrs and if yes does this anybody? Locks are not propagated to the underlying filesystem. This will hurt when a build system relies on them. I have no idea how many are affected by this. Locks still work within the overlay filesystem itself. However, if one process is accessing the underlying filesystem directly, and another process is accessing through disorderfs, they won't see each others' locks. Since the build will only access through disorderfs, the lack of lock propagation shouldn't affect builds. Ah ok. Then I think nothing _should_ break builds (with the updated version). signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] using disorderfs to build packages
Hi, Holger Levsen: Hi, (repeating here as it seems to have been lost on irc.) After reading the disorderfs manpage I wonder if it's really already suitable to build everything, as the Bugs sections reads: BUGS Errors from close are not properly reported. Probably don't hurt. Most programs ignore them anyway. fsync is not implemented. Beside of timing this should not make a difference as long the maschine don't crash. (I think that's what libeatmydata is doing via LD_PRELOAD). And we are building on tmpfs anyway (I don't know if fsync does anything on tmpfs). xattrs are not implemented. Probably only hurts for packages which need to handle those and have test cases for it. (e.g. rsync, rdiff-backup, ...) Can packages contain files with xattrs and if yes does this anybody? Locks are not propagated to the underlying filesystem. This will hurt when a build system relies on them. I have no idea how many are affected by this. I *believe* this will break several builds, won't it? I think the locks should be fixed; the xattrs on the long run; fsync and close can probably be broken for ever without (much) damage. Thats obviously only my personal speculation (the only proper thing is probably a (partial) archive rebuild to see what happens) HW42 signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] using disorderfs to build packages
On Wed, 26 Aug 2015 10:58:08 +0200 HW42 h...@ipsumj.de wrote: Errors from close are not properly reported. Probably don't hurt. Most programs ignore them anyway. I think close() only fails if there's a disk error or you run out of disk space/quota. Neither of these should happen during a normal build. fsync is not implemented. Beside of timing this should not make a difference as long the maschine don't crash. (I think that's what libeatmydata is doing via LD_PRELOAD). And we are building on tmpfs anyway (I don't know if fsync does anything on tmpfs). Right. To be clear, fsync() and fdatasync() still report success with disorderfs - they just don't do anything. xattrs are not implemented. Probably only hurts for packages which need to handle those and have test cases for it. (e.g. rsync, rdiff-backup, ...) I'm guessing that only a minority of packages would be affected. But, implementing xattrs turned out to be trivial, so I just released 0.2.0 with xattr support. The Debian package is ready for someone to upload. Can packages contain files with xattrs and if yes does this anybody? Locks are not propagated to the underlying filesystem. This will hurt when a build system relies on them. I have no idea how many are affected by this. Locks still work within the overlay filesystem itself. However, if one process is accessing the underlying filesystem directly, and another process is accessing through disorderfs, they won't see each others' locks. Since the build will only access through disorderfs, the lack of lock propagation shouldn't affect builds. -- Andrew ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
