Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, i wrote: > > As it is now, it is the clearest no-brainer: > > Export SOURCE_DATE_EPOCH and forget about any other time issues which > > you don't create yourself by own xorriso arguments. Chris Lamb wrote: > I'm sorry but I don't quite follow and I don't want to read what I want > to read. Currently it is (hopefully) this way: If the same SOURCE_DATE_EPOCH value is in effect with each xorriso run then the user may expect that timestamps in the ISO are not an obstacle for reproducibility. Regardless of input. So it is only about names, ownership, permissions, and data content. With popular option -r one can make ownership and permissions flatly reproducible. This can be changed by own xorriso arguments to the three xorrisofs options named in the man page text: --modification-date=, --gpt_disk_guid, --set_all_file_dates. > trying to avoid the case of — heaven forbid we broke > the Debian CD creation! — that we would get the blame as ACKing on their > behalf. :) The popular debian-cd ones i can ACK quite well myself. (And then i'd first need to convince Steve McIntyre of using Sid's xorriso.) I test with debian-8.4.0-amd64-netinst.iso debian-8.5.0-i386-lxde-CD-1.iso debian-live-8.4.0-amd64-standard.iso debian-8.1.0-arm64-netinst.iso debian-7.9.0-kfreebsd-amd64-netinst.iso debian-7.4.0-mips-netinst.iso debian-7.4.0-sparc-netinst.iso debian-7.0-hppa-NETINST-1.iso debian-8.3.0-ppc64el-netinst.iso debian-9.0-sparc64-NETINST-1.iso and mini.iso for amd64 and i386. I mount them, let xorriso pack up the files and replay the boot related -as mkisofs options which it deduced from inspecting the unmounted ISO. Then i let xorriso report both boot equipments and let diff compare the reports. After mounting both ISOs, two find runs over both trees make sure that differences in file content, ownership, permissions, or time get reported. The few reported differences come from automatic timestamps and differing block addresses which get patched into boot images. (The originally used xorriso versions still produced the reproducibility-unfriendly sequence of file data extents.) My scruples are more towards adventurous distros which immediately use my newest releases. Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
> > *iff* SOURCE_DATE_EPOCH is set: > > > > - mtimes are taken from stat(2). [The FS creator is responsible for > > ensuring they are reproducible, possibly by some "clamping" call > > to `find -newermt | xargs touch …`.] > > > > - atime is copied from mtime [as FS creator "can't" set that]. > > > > - ctime is copied from mtime [ditto]. > > man 2 utime promises to set atime and xorriso tries to restore it when > extracting files to disk. I put "can't" in quotes as there is no pervasive/widespread shell-level tool for this (cf. touch(1)) so I stand by this. > But for ctime i know no portable method. Mm. > Export SOURCE_DATE_EPOCH and forget about any other time issues which > you don't create yourself by own xorriso arguments. I'm sorry but I don't quite follow and I don't want to read what I want to read. Please could you clarify exactly the behaviour in this case? > Looks like --set_all_file_dates is losing its only user even before it > gets released. (Which is good; the less options the better IMHO.) > > I am therefore not testing them from > > the "Debian" point of view and ACKs are simply my own testcases. > > Any ACK or NAK is highly welcome. I'm sure; I was just trying to avoid the case of — heaven forbid we broke the Debian CD creation! — that we would get the blame as ACKing on their behalf. :) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, Chris Lamb wrote: > May I propose the following behaviour *iff* SOURCE_DATE_EPOCH > is set: > > - mtimes are taken from stat(2). [The FS creator is responsible for > ensuring they are reproducible, possibly by some "clamping" call > to `find -newermt | xargs touch …`.] > > - atime is copied from mtime [as FS creator "can't" set that]. > > - ctime is copied from mtime [ditto]. man 2 utime promises to set atime and xorriso tries to restore it when extracting files to disk. But for ctime i know no portable method. One needs to be filesystem ultra user to influence that one. (E.g. operator of xorriso inside the emerging ISO.) Looks like --set_all_file_dates is losing its only user even before it gets released. Consider: Your new FS creator responsibilities dilute the radical decision to which the crowd pushed me a few days ago. As it is now, it is the clearest no-brainer: Export SOURCE_DATE_EPOCH and forget about any other time issues which you don't create yourself by own xorriso arguments. We can justify this behavior by pointing to program cp which, too, updates all file times by default. Just that we set and hold the clock as we desire. Ok. It is definitely too early for release. :)) (I understand we have time until end of september for Stretch ...) > This seems less like a giant hammer of setting them all to target->now. Until release we can reshape it to a screwdriver. Afterwards i am restricted by general compatibility commitments. I need to think where and how to implement the new proposal. Probably it is worth regardless how SOURCE_DATE_EPOCH will finally be interpreted. If it becomes an alternative to --set_all_file_dates then the user is free to override a hammer SOURCE_DATE_EPOCH by the new optiom --set_file_times_to_mtime (is this a good name ?). > I am therefore not testing them from > the "Debian" point of view and ACKs are simply my own testcases. Any ACK or NAK is highly welcome. Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi Thomas, > - Verify that SOURCE_DATE_EPOCH works as promised and expected. > (Promised is to set the defaults of three xorriso settings. >Expected is that these defaults suffice for reliable reproducibility.) My current schedule is to test this and report back in the next 4 or 5 days. As a heads-up, I think I am likely to relax one change that might be of interest. May I propose the following behaviour *iff* SOURCE_DATE_EPOCH is set: - mtimes are taken from stat(2). [The FS creator is responsible for ensuring they are reproducible, possibly by some "clamping" call to `find -newermt | xargs touch …`.] - atime is copied from mtime [as FS creator "can't" set that]. - ctime is copied from mtime [ditto]. This seems less like a giant hammer of setting them all to target->now. > - Verify that bootable Debian ISOs made with xorriso-1.4.6 still work > as good or bad as those currently made with 1.3.6 (or maybe 1.4.4). To clear up any ambiguity here, whilst I am making ISOs based on Debian, they are not "Debian ISOs" in any sense of being connected with debian-cd or being official in any sense. I am therefore not testing them from the "Debian" point of view and ACKs are simply my own testcases. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, i am pondering when to release xorriso-1.4.6 et.al. 1.4.5 passed my rebuild-ISO regression tests by reporting no differences of file content or boot equipment as far as xorriso can replay the latter. Normally i would wait a few weeks whether my daily backups or some courageous user find any other problems. But my backups hardly touch the code parts which underwent the dangerous changes. Courageous bootability users are sparse. So, if i release 1.4.6 now and get it into Debian Sid a few days later, do you already have test cases where the new code would be challenged ? - Verify that SOURCE_DATE_EPOCH works as promised and expected. (Promised is to set the defaults of three xorriso settings. Expected is that these defaults suffice for reliable reproducibility.) - Verify that bootable Debian ISOs made with xorriso-1.4.6 still work as good or bad as those currently made with 1.3.6 (or maybe 1.4.4). Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi Thomas, > uploaded is > http://www.gnu.org/software/xorriso/xorriso-1.4.5.tar.gz [..] > which obeys SOURCE_DATE_EPOCH by stating in its man pages Awesome! :) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, uploaded is http://www.gnu.org/software/xorriso/xorriso-1.4.5.tar.gz MD5 2e64a46b50c5192a0797e63dd3182b6a Version timestamp : 2016.08.16.131907 which obeys SOURCE_DATE_EPOCH by stating in its man pages ENVIRONMENT ... SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It is supposed to be either undefined or to contain a decimal number which tells the seconds since january 1st 1970. If it contains a number, then it is used as time value to set the default of --modification-date=, --gpt_disk_guid, and --set_all_file_dates. Startup files and program options can override the effect of SOURCE_DATE_EPOCH. Changeset: http://libburnia-project.org/changeset/5747 If you start xorriso in an environment with SOURCE_DATE_EPOCH it will react as follows. Issue a NOTE message about acceptable values and DEBUG messages about the consequences: $ export SOURCE_DATE_EPOCH=1471290111 $ xorriso -report_about all -as mkisofs ... ... xorriso : NOTE : Environment variable SOURCE_DATE_EPOCH encountered with value 1471290111 xorriso : DEBUG : SOURCE_DATE_EPOCH : -volume_date uuid 2016081519415100 xorriso : DEBUG : SOURCE_DATE_EPOCH : -volume_date all_file_dates 2016081519415100 xorriso : DEBUG : SOURCE_DATE_EPOCH : -boot_image any volume_date_uuid ... The DEBUG messages will not be visible by default. Issue a SORRY message (mild error severity) on unacceptable values, go on with ISO production, and end with non-zero exit value: $ export SOURCE_DATE_EPOCH=xyz1471290111 $ xorriso -as mkisofs ... ... xorriso : SORRY : Malformed environment variable SOURCE_DATE_EPOCH encountered xorriso : HINT : Unset SOURCE_DATE_EPOCH before starting xorriso or see https://reproducible-builds.org/specs/source-date-epoch/ ... $ echo $? 32 The continued work and the exit value are consequences of default settings -abort_on FAILURE -return_with SORRY 32 It can be influenced by the user of xorriso in startup files or program arguments: $ xorriso -abort_on sorry -as mkisofs ... ... xorriso : SORRY : Malformed environment variable SOURCE_DATE_EPOCH encountered xorriso : HINT : Unset SOURCE_DATE_EPOCH before starting xorriso or see https://reproducible-builds.org/specs/source-date-epoch/ xorriso : aborting : -abort_on 'SORRY' encountered 'SORRY' $ echo $? 5 I compared again the MD5s of two rebuild runs of my ISO collection. All identical. All disk GUIDs the same. (Shudder.) In another comparison the modification date of SOURCE_DATE_EPOCH was overridden by program arguments. All identical. At least slight differences between disk GUIDs. So this mixing is reproducible as expected. Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
> If file "$HOME/project_42_iso_ids" exists, then the lengthy parameter > is taken from there. It defines a timestamp, a disk GUID, and what else > might turn out to be in need of being reproduced. I also understand and appreciate that some use-cases need more entropy than a 32-bit integer can provide. However, I feel like the parameter you suggest could quite easily fallback to SOURCE_DATE_EPOCH with the extra bits -- I use the term hand-wavingly for now -- masked to zero. Anyone needing to provide more "secure" entropy could then use the interface you suggest. Otherwise, I sincerely worry that your efforts to make the output reproducible will be neglected as the user interface is simply too complicated. I agree that an environment variable is not the perfect channel to communicate values to a process, but this particular now has such widespread adoption that its becoming somewhat quixotic not to make use of it. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
On Mon, Aug 15, 2016 at 02:43:49PM +0200, Thomas Schmitt wrote: > Control via environment is unpopular on my side because it would be > a novelty in the user interface of xorriso. are you sure? I mean, usually stuff like locales and language settings are already taken from the environment… SOURCE_DATE_EPOCH is quite special, and the general expectation is that it's used if it's set. (And it will never be set if it's not ment to be used.) -- cheers, Holger signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Thomas Schmitt: > A new source of irreproducibility appeared: Future xorriso versions. > [...] > > I will of course try to keep such changes as rare as possible. But it > cannot be totally guaranteed that the same input data and options will > yield the same ISO with future versions of xorriso. > In case of a bug, i'd have few justification to preserve its consequences. > > The most obvious reason is the Preparer Id which xorriso writes into > the ISO by default. Current debian-cd ISOs have: > > XORRISO-1.3.6 2014.04.08.18, LIBISOBURN-1.3.6, LIBISOFS-1.3.6, > LIBBURN-1.3.6 > > whereas currently uploaded GNU xorriso writes by default: > > XORRISO-1.4.5 2016.08.12.185822, LIBISOBURN-1.4.5, LIBISOFS-1.4.5, > LIBBURN-1.4.5 > > This can be overridden by classical mkisofs option -p. > E.g.: > > -p "Yoyodyne Reproducible ISO Maker and xorriso" It obviously depends on the context in which xorriso is used but in general you need to fixate the version of your build depends anyway to archive reproducible builds (for example an updated compiler should be able to output better optimized code). So this should be no reproducibility problem. For Debian packages we are going to record the packages used to satisfy the build depends in so called .buildinfo files (see [0] for an example). [0]: https://tests.reproducible-builds.org/debian/buildinfo/unstable/amd64/libisoburn_1.4.4-1_amd64.buildinfo signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, i wrote: > > This becomes lengthy. Wiki article size. Chris Lamb wrote: > I can't try and convince you -- one last time -- that a single global > argument, perhaps set via the environment, would be one way to simplify > this? I am open to the idea of a bundle-it-all option when all issues and solutions are explored. Control via environment is unpopular on my side because it would be a novelty in the user interface of xorriso. In general the individual parameter which the user carries from production run to production run must contain more entropy than a time stamp of centiseconds granularity. That's inconvenient to handle manually. How about this: xorriso -as mkisofs ... --reproducible_ids "$HOME/project_42_iso_ids" ... If file "$HOME/project_42_iso_ids" exists, then the lengthy parameter is taken from there. It defines a timestamp, a disk GUID, and what else might turn out to be in need of being reproduced. If the file does not exist, the user sees a loud warning and the file is created with the "now" timestamp, a good random GUID, and everything else. This option would also trigger options -r, --set_all_file_date, -p. - There remains the issue of future xorriso versions with bug fixes or other changes of the ISO result. We need some quick check whether the new result matches the old one. Record an MD5 in the ids file ? Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
On Fri 2016-08-12 15:27:40 -0400, Thomas Schmitt wrote: > Although grub-mkrescue probably can live with poor GPT GUIDs, i meanwhile > found a use case in xorriso where user defined modification-date does not > express the desire for reproducibile GUIDs: xorriso command > -boot_image "any" "replay". > If xorriso modifies a bootable ISO made by grub-mkrescue, then it has > to maintain the modification date so that GRUB2 after waking up finds > the ISO. It is then inappropriate to keep GPT GUIDs, because the ISOs > are nevertheless not meant to be identical. > > So the default of new option --gpt_disk_guid is old behavior "random". Would it possible to generate the GPT GUID based on a digest of the contents of the ISO itself? I don't understand well enough how GPT interacts with ISOs to be able to sketch out the details, but if there is a way to look at the rest of the generated filesystem *aside* from the GUID, then you could push all that data through a simple hash function, and then deterministically derive the GUID from the hash function. (what hash function to use? it probably doesn't even need to be cryptographically secure, but sha256 is cheap these days and it avoids any risk that someone could come up with a plausible attack based on forcing GUID collisions) That would give you identical GUIDs for identical ISOs, and distinct GUIDs for ISOs that vary in any way, without having to include any randomness or asking the user to do the work to select a non-random GUID (which they're probably not likely to do responsibly). Thanks for your work on this, Thomas! Let me know if this idea doesn't make sense for some reason, like if there are other bits in the ISO that themselves depend on the GUID. I'd be happy to brainstorm other approaches. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, i wrote: > > It would be quite cumbersome to produce upstream 1.4.5 releases of the > > libraries for Sid. Chris Lamb wrote: > It was just a general and friendly offer, I didn't mean for it to come > across as a request or require a time-consuming justification if you did > not want to proceed. I felt the need to explain why Sid has to wait for 1.4.6 and why potential testers don't have to wait but should use GNU xorriso-1.4.5 which is easy to build and runs without system-wide installation. Newly uploaded: http://www.gnu.org/software/xorriso/xorriso-1.4.5.tar.gz MD5 e89f717787749a1331e8213c0684cda0 Version timestamp : 2016.08.12.185822 Changesets for reproducible GPT GUIDs: http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/revision/1331 http://libburnia-project.org/changeset/5741 http://libburnia-project.org/changeset/5742 Are there proposals where to publicly expose the advise for reproducibility via xorriso -as mkisofs emulation as it is now ? -- - Use xorriso-1.4.5 snapshot 2016.08.12.185822 or newer. (Will after release become: Use xorriso-1.4.6 or newer.) - Use option --modification-date=MMDDhhmmsscc to control the timestamps of the filesystem superblocks and other global components of the ISO filesystem. - If you let xorriso produce GPT, use option --gpt_disk_guid modification-date to produce GUIDs of low quality, or obtain once a better GUID string (e.g. 2303cd2a-73c7-424a-a298-25632da7f446) and use it with each reproducible ISO production run by --gpt_disk_guid 2303cd2a-73c7-424a-a298-25632da7f446 - Consider to use option --set_all_file_dates MMDDhhmmsscc to override the timestamps of the input files and directories. - Consider to use option -r to override POSIX ownership and access permissions. -- Although grub-mkrescue probably can live with poor GPT GUIDs, i meanwhile found a use case in xorriso where user defined modification-date does not express the desire for reproducibile GUIDs: xorriso command -boot_image "any" "replay". If xorriso modifies a bootable ISO made by grub-mkrescue, then it has to maintain the modification date so that GRUB2 after waking up finds the ISO. It is then inappropriate to keep GPT GUIDs, because the ISOs are nevertheless not meant to be identical. So the default of new option --gpt_disk_guid is old behavior "random". -- What to test: - If you produce bootable ISOs, then please do with current xorriso-1.4.5 and try whether the resulting ISOs play as well with boot firmware and partition editors as do the ISOs from previous xorriso versions. - If you are interested in reproducible ISOs, then please try whether above advise suffices to get identical ISOs from equivalent file trees on different computers, in different timezones, and on different days. Use as many xorrisofs options as you can find in scripts or the web. Tested so far: I reproducibly rebuilt my ISO image collection of 65 GB in 126 files by two passes, one of them under valgrind supervision. The MD5s of the resulting ISOs were recorded. No differences were found between the passes. Among the ISOs were 35 bootable Debian ISOs, mostly version 7 or newer. So it seems that the exotic boot sectors impose no new instability. (I'm just not sure whether i got an ISO of every arch. And the powerpc ISO will not work due to lack of HFS ...) Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
> It would be quite cumbersome to produce upstream 1.4.5 releases of the > libraries for Sid. It was just a general and friendly offer, I didn't mean for it to come across as a request or require a time-consuming justification if you did not want to proceed. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, i wrote: > > Release 1.4.6 might come soon. [after enough testing] > > Then it depends on whether my Debian sponsor is on holiday. Chris Lamb wrote: > For this, or any other packages that you wish to see in Debian, I would > be very happy to sponsor. Please let me know when appropriate with a link > to a .dsc. Thank you for the offer. Currently i am very well sponsored by Dominique Dumont. I prepare the ./debian directories in my Sid VM, test by debuild, lintian, cme, commit to SVN, and notify Dominique that it's ready. debian-mentors answers my technical questions. See https://anonscm.debian.org/viewvc/pkg-libburnia/trunk/ (The debian/changelog files show versions 1.4.4-2 which are not uploaded. That's intentional. I am just collecting little improvements for the next release.) It would be quite cumbersome to produce upstream 1.4.5 releases of the libraries for Sid. They'd need a different .so number to indicate that they do not fulfill the promise of future compatible ABI. Released symbols and their semantics are guaranteed to be supported in future releases. But during development i reserve the right to correct freshly introduced mistakes. So for testing the changes, GNU xorriso-1.4.5 is exactly the thing to use. Even debian-cd is still using GNU xorriso 1.3.6 because 1.3.2 is too old and Steve McIntyre obviously found no time to test the 1.4.X packages since they are in Debian. debian-8.5.0-i386-lxde-CD-1.iso bears in its superblock as preparer id: "XORRISO-1.3.6 2014.04.08.18". Have a nice day :) Thomas ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Dear Thomas, > i now implemented the new -as mkisofs option: > > --set_all_file_dates timestring >Set mtime, atime, and ctime of all files and directories to the >given time. Thank you so much for your efforts on this and the other changes. :) > - Use xorriso-1.4.5 snapshot 2016.08.07.19 or newer. > > - Use option > --modification-date=MMDDhhmmsscc > to control the timestamps of the filesystem superblocks and other global > components of the ISO filesystem. > > - Consider to use option > --set_all_file_dates MMDDhhmmsscc > to override the timestamps of the input files and directories. > > - Consider to use option > -r > to override POSIX ownership and access permissions. Great summary. > Uploaded is the first GNU xorriso 1.4.5 which hopefully fulfills the promise > of above advise: > > http://www.gnu.org/software/xorriso/xorriso-1.4.5.tar.gz Can we get this version upload to sid? :) > Make sure that not only isoinfo and Linix mount show no differences > but that the test ISOs are really identical byte by byte. Oh, of course! Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Thomas Schmitt wrote: > This discussion yielded that your xorriso-1.3.2 with SOURCE_DATE_EPOCH > will not work reproducibly if you present the same input file tree on > different hard disks or after copying it to another location in the > global tree. Just to confirm, this is due to non-determinstic extent numbering due to sorting? > If you need reliable effect of SOURCE_DATE_EPOCH right now, then you have > to port your changeset to xorriso-1.4.2 or 1.4.4. I'm actually targetting stable/jessie for my specific use-case. When I checked before, my patches ported almost trivially to the version in sid at least. No need to make this a blocker or worry about it further; let's just consider git/svn HEAD. > One single suggested switch, to be exacting. :)) > And only if really file timestamps shall not matter. For now, sure. But extent ordering, etc. etc.. In my experience of hunting down reproducible issues, it *can* turn into a bit of a rabbit hole.. > I do not deem the overriding of file timestamps a normal goal of > reproducibility. In general the POSIX attributes of a file are as significant > as is the file content or the softlink target string. Well, if you are using any tool that generates files (for example, the tool that's actually building the files to put on your ISO!), then these files will have varying timestamps between builds, no? > Of course, if your use case makes it desirable to override the file timestamps > then "filesystem manipulator" comes in handy. So i currently strive to put > your use case on that track. Can you briefly elaborate/confirm what you mean by that? Post-process the ISO with the manipulators rather than do this during build? > > developer [...] will need to hunt down and learn [...] all the > > various command-line arguments > > A valid point, indeed. Further to this, someone once did a survey of all the Debian-area ISO/chroot build tools and found there were at least ten. All of these would need separate patching to achieve reproducible ISOs, alas. Just another data point really.. > (I plan to derive the isohybrid MBR id from the --modification-date= > setting if present.) Nice! > > As I understand the code in Xorriso_make_iso_write_opts, > > isoburn_igopt_set_scdbackup_tag appears to be called unconditionally > > It copies to libisoburn the scdbackup_tag_name, which is empty by default. > This causes libisoburn not to call iso_write_opts_set_scdbackup_tag(). > Else you'd get a message at the end of the run: > xorriso : NOTE : scdbackup tag written : x_y B60805.122358 3111051 > 397e84253510b0dc3d8f17178585e226 Ahh, okay. You can now see why I had not sent my patches before you requested them as they contained -- at least -- unnecessary things like this. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, meanwhile i remember that xorriso ISO production already went through a Debian Reproducible Builds discussion: http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg6.html This discussion yielded that your xorriso-1.3.2 with SOURCE_DATE_EPOCH will not work reproducibly if you present the same input file tree on different hard disks or after copying it to another location in the global tree. Moving the tree root should be ok, though. If you need reliable effect of SOURCE_DATE_EPOCH right now, then you have to port your changeset to xorriso-1.4.2 or 1.4.4. 1.4.4 is in Debian testing or available as standalone tarball at https://www.gnu.org/software/xorriso/#download This tarball contains the libraries which will get linked statically to form a xorriso binary that depends only on vanilla system libraries. No interference with installed libburn.so, libisofs.so, libisoburn.so. - i wrote: > > --set_all_file_dates "$timestring" Chris Lamb wrote: > Assuming it also sets iso_write_opts_set_always_gmt, yes. It is default of xorriso $ xorriso -no_rc -status -compliance ... -compliance clear:...:always_gmt:... because timezones are a swamp on their own. But yes, --set_all_file_dates would include xorriso command -compliance always_gmt > I'm not 100% convinced though. I'm a little loathed to drop the > SOURCE_DATE_EPOCH handling in general It is a quick and global solution. But it does not fit into the libraries' traditions. xorriso is controlled by commands, the libraries by API. The reason for my approach is that xorriso-1.4.4 can already do nearly everything you want to achieve. > zooming out a bit from these suggested new switches, One single suggested switch, to be exacting. :)) And only if really file timestamps shall not matter. > it does make it somewhat difficult for the fly-by developer to build > a reproducible ISO I do not deem the overriding of file timestamps a normal goal of reproducibility. In general the POSIX attributes of a file are as significant as is the file content or the softlink target string. It's rather the automatically generated timestamps of an ISO which i would call an obstacle to reproducibility of ISO production. (Or random sequences of file content extents. See below.) Of course, if your use case makes it desirable to override the file timestamps then "filesystem manipulator" comes in handy. So i currently strive to put your use case on that track. > developer [...] will need to hunt down and learn [...] all the > various command-line arguments A valid point, indeed. Currently there is the need to use command -volume_date "uuid" or -as mkisofs option --modification-date=, with the bug-like problem that isohybrid gets an MBR id from current time. (I plan to derive the isohybrid MBR id from the --modification-date= setting if present.) > As I understand the code in Xorriso_make_iso_write_opts, > isoburn_igopt_set_scdbackup_tag appears to be called unconditionally It copies to libisoburn the scdbackup_tag_name, which is empty by default. This causes libisoburn not to call iso_write_opts_set_scdbackup_tag(). Else you'd get a message at the end of the run: xorriso : NOTE : scdbackup tag written : x_y B60805.122358 3111051 397e84253510b0dc3d8f17178585e226 The documentation indeed does not state that empty name means no tag. I changed this now. This reminded me of the Debian Reproducible Builds discussion: http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg6.html where the scdbackup tag would have been noticed during image comparison. That comparison yielded that xorriso-1.3.2 from Debian "stable" is quite hopeless in respect to reproducibility, anyways: A change in sequence of directory traversal on hard disk could cause a change in the sequence of data file content extents. (See mentioning of Red-Black tree in the discussion.) Hopefully fixed by release 1.4.2. No need to manipulate file sorting weights as discussed in the thread. > Sure, but you would you really be backing up with SOURCE_DATE_EPOCH defined? You did not convince me of SOURCE_DATE_EPOCH yet. (Might change if i stumble over a target->now problem which --modification-date= does not solve.) Anyways, if i want a scdbackup tag, then one with real date. On the other hand without special option no scdbackup tag emerges. So i think your use case and mine are wide enough apart already. > Would it matter if their MBR ID clashed? ;-) I really don't know. hpa was not overly verbous wbout the meaning of this field in his MBR producer isohybrid.pl. So i decided to be cowardish and scramble it. -- I am now testing a MBR id solution based on --modification-date=. Then i finish my assessment of target->now whether it can endanger reproducibility despite --modification-date=. Have a nice day :) Thomas
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi, (this time i hopefully managed to exempt you from Pkg-libburnia-devel list moderation) Requests for agreement: - Do you agree that a new -as mkisofs option --set_all_file_dates "$timestring" which sets atime, mtime, and ctime of all files, covers your patches 0001-source_date_epoch.patch 0002-set-default-timestamp.patch ? - Do you agree that i omit 0003-set-scdbackup_tag_time-from-source_date_epoch.patch and rather expect that option --scdbackup_tag is not to be used when reproducibly building an ISO ? (Reason is the meaning of the tag in the context of my backup tool scdbackup. A fake timestamp would be flatly wrong.) - Do you agree with http://libburnia-project.org/changeset/5733 which i derived from your 0004-normalize-wday-yday.patch Details: i wrote: > > Is it really desirable/correct to override the timestamps of files when > > they get put into the ISO image ? Chris Lamb wrote: > At least for modification times, I think we *could* go > with asking users to run something like: > $ find sourcedir/ -newermt '$SOURCE_DATE_EPOCH' -print0 | >xargs -0r touch --no-dereference --date='$SOURCE_DATE_EPOCH' > .. prior to building, but AIUI one cannot reliably set atime and > ctime so I ended up just overridding if-and-only-if SOURCE_DATE_EPOCH > is set. Well, if date manipulation is desired, then we can already do it in xorriso. "RockRidge filesystem manipulator" ... yada yada ... :)) The only problem is the inconvenience to leave -as mkisofs emulation mode and to issue two -alter_date_r commands: xorriso -as mkisofs ...mkisofs.options... -- \ -alter_date_r b "$timestring" / -- \ -alter_date_r c "$timestring" / -- A convenience option for -as mkisofs emulation could do the trick (and make the libisoburn patches obsolete). How about this: xorriso -as mkisofs mkisofs.options... --set_all_file_dates "$timestring" ... > > What about other file attributes which might change without > > file content change ? > Such as? :) I didn't think that the ISO format really supported "much". Rock Ridge records the attributes from stat(2): owner, group, permissions, file type. Linux mount(8) then shows them. People often use option -r to make all files read-only and owned by user id 0. Device files and Unix pipes will hardly appear. But softlinks are not uncommon. > > > +time_t Xorriso__current_time() > > > +return now; > To be honest I now feel it should probably exit(EXIT_FAILURE), otherwise > we're just confusing users Well, on the level of libisoburn patches this would now be a FAILURE event if xorriso cannot decode the timestamp argument of the new option --set_all_file_dates. By default, FAILURE in batch mode of xorriso yields abort and non-zero exit value. Probably there will be no new function iso_current_time() either. But if it emerges, then it cannot call exit(), but rather would put a FAILURE event into the message queue, so that the application above libisofs can decide what to do. (I see that i did not read all your info while diving into the patches. Organisational blindness combined with programmer's enthusiasm.) > > Where did you get negative effects from not setting erg->wday , erg->yday > Only in the "Understanding timestring" log messages. Good catch. xorriso : DEBUG : Understanding ECMA-119 timestring '2016080323051000' as: Sun Aug 3 23:05:10 2016 Re-reading man 3 mktime i now understand why your patch corrects the day name. The line normalized = mktime(erg); does the trick by modifying members of erg, whereas the following line is without lasting effect: erg = gmtime(); mktime() is quite aggressive. It manipulates not only the day numbers if it deems it necessary. So i only let it operate on a copy of erg and rather risk to get a wrong weekday number in case of hour rollover due to timezone or DST oddities. (--modification-date is intended as interface to GRUB command "search --fs-uuid") See http://libburnia-project.org/changeset/5733 > > > 0003-isohybrid-mbr.patch > > This weakens the weak random number even in the case of non-constant time: > its a terrible PRNG in either case so it can hardly matter. ;) It shall prevent collisions. Within a day, i'd say it gives at least 24 bits of entropy. Considering the birthday paradox, this should be good enough for about 4000 ISOs per day. But actually nobody could tell me what this MBR id is good for. I got it from the isohybrid script. Now i ponder whether it is worth to make
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
> Is it really desirable/correct to override the timestamps of files when > they get put into the ISO image ? I.e. isn't reproducible build supposed to > be identical only if the input is identical ? Good question. At least for modification times, I think we *could* go with asking users to run something like: $ find sourcedir/ -newermt '$SOURCE_DATE_EPOCH' -print0 | xargs -0r touch --no-dereference --date='$SOURCE_DATE_EPOCH' .. prior to building, but AIUI one cannot reliably set atime and ctime so I ended up just overridding if-and-only-if SOURCE_DATE_EPOCH is set. > If desirable: What about other file attributes which might change without > file content change ? Such as? :) I didn't think that the ISO format really supported "much". I mean, I doubt it supports extended attributes or selinux blah.. > Currently it seems to me that xorriso lacks only two features: [..] > This does not mean that i would not see the convenience of a single > option which overrides all variable time stamps. I would agree, but on the other hand I may be missing some other features outside of my somewhat narrow usecase. > With which xorriso commands (or xorriso -as mkisofs options) did you test > whether the proposed changes suffice for reproducible ISOs ? Nothing too fancy. $ xorriso -as mkisofs \ --modification-date XXX \ [boot catalog stuff; am using syslinux] \ -isohybrid-mbr [..] \ -V blah -A blah > > +time_t Xorriso__current_time() > > ... > > +if ((errno == ERANGE && (epoch == ULLONG_MAX || epoch == 0)) > > +|| (errno != 0 && epoch == 0)) > > +return now; > > Since the application wants a constant time, shouldn't we return some > constant default value in this case ? (Proposal for a value ? 1970 ?) To be honest I now feel it should probably exit(EXIT_FAILURE), otherwise we're just confusing users > - > > 0004-normalize-wday-yday.patch > - > > Where did you get negative effects from not setting erg->wday , erg->yday ? Only in the "Understanding timestring" log messages. (parse_exec.c & opts_p_z.c). It's otherwise harmless as previously mentioned, > The Unix time functions are tricky and the purpose of the timestamp might > be the job of an id string when converted back to some time format. > So it seems best to keep all significant struct members as they are now. Ah, I didn't know. Thanks. > > 0003-isohybrid-mbr.patch [..] > This weakens the weak random number even in the case of non-constant time: Indeed I mentioned this in my notes, going on to say that its a terrible PRNG in either case so it can hardly matter. ;) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `- ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
Hi,
> +++ libisoburn-1.3.2/libisoburn/isoburn.c
Oh nostalgy. :))
-
General questions:
Is it really desirable/correct to override the timestamps of files when
they get put into the ISO image ? I.e. isn't reproducible build supposed to
be identical only if the input is identical ?
If desirable: What about other file attributes which might change without
file content change ?
The question arises whether we need a general overrider for file dates.
xorriso generic commands could be used to set attrbutes and times after
grafting the files into the upcoming ISO. debian-cd normally uses only
one generic xorriso command: "-as" with first parameter "mkisofs".
The parameter list of this command "-as" would end at the first "--"
parameter. Afterwards one could perform commands like
-alter_date_r , -chmod_r , -chown_r , -chgrp_r in order to force file
attributes.
Currently it seems to me that xorriso lacks only two features:
- User defined target->now in libisofs.
(Its use seems to be overrideable by other libisofs features.
I'm still verifying whether this is true and whether they are
reachable via xorriso ...)
- User defined MBR "Disc signature" at bytes 440 to 443.
This does not mean that i would not see the convenience of a single
option which overrides all variable time stamps. But if we decide that
input file dates shall not be overridden, then we need a new plan anyway.
With which xorriso commands (or xorriso -as mkisofs options) did you test
whether the proposed changes suffice for reproducible ISOs ?
("A normal debian-cd run on arch XYZ" would be sufficient as answer.
I just wonder whether there are more features like isohybrid pseudo-random
which might not have been touched by your test run.)
Some detail thoughts on the patches:
-
> 0001-source_date_epoch.patch (libisoburn)
-
> +source_date_epoch = getenv("SOURCE_DATE_EPOCH");
This would be the first env variable to influence libisoburn behavior.
Very economic shortcut but probably inappropriate on the long run.
I will have to think about an API call and a xorriso command to control
this.
> +epoch = strtoull(source_date_epoch, , 10);
That would be the first usage of strtoull() in the libraries.
The man page looks like it is not always available. (More retro would
be to read the string as type double and then convert to some int type.)
> +time_t Xorriso__current_time()
> ...
> +if ((errno == ERANGE && (epoch == ULLONG_MAX || epoch == 0))
> +|| (errno != 0 && epoch == 0))
> +return now;
Since the application wants a constant time, shouldn't we return some
constant default value in this case ? (Proposal for a value ? 1970 ?)
-
> 0002-set-default-timestamp.patch
-
Looks ok, except the pending decision about getenv().
-
> 0003-set-scdbackup_tag_time-from-source_date_epoch.patch
-
Ok.
-
> 0004-normalize-wday-yday.patch
-
Where did you get negative effects from not setting erg->wday , erg->yday ?
The patch would look dangerous because of the volatility of the gmtime()
result but actually seems to be without lasting effect because erg is only
overwritten on the stack but not in the caller's variable space:
int Decode_ecma119_format(struct tm *erg, char *text, int flag)
...
+ erg = gmtime();
If the two members are really needed then i would not regenerate the whole
erg by gmtime() but only the two missing struct members:
normalized_tm = gmtime();
erg->wday = normalized_tm->wday;
erg->yday = normalized_tm->yday;
The Unix time functions are tricky and the purpose of the timestamp might
be the job of an id string when converted back to some time format.
So it seems best to keep all significant struct members as they are now.
-
> 0001-source_date_epoch.patch (for libisofs)
-
Like with 0001-source_date_epoch.patch(libisoburn) i am not happy
with the environment variable.
Maybe one should make the time producer function of libisofs public
in the API and use it from libisoburn too, instead of having a
separate Xorriso__current_time().
-
> 0002-set-target-now-from-source_date_epoch.patch
