Hello, On 01/03/2023 22:59, Thomas Schmitt wrote:
Mattia Rizzolo wrote:
...
... The file tree and the data files' content is only a part of a bootable ISO 9660 image. There's executable code in the blind spots of such a view: MBR legacy BIOS boot code, EFI programs in the EFI partition, "hidden" El Torito boot images, ...I know that thing has been used to work on reproducible bootable image,Possibly because typical pitfalls aren't tested. Like FAT timestamps in the EFI partition.
In live-build, the FAT timestamps in the EFI partition are carefully ironed out [1][2]. Before the fixes, they did show up in the diffoscope, but only in the hex dump. After I realised that the differences in the hex dump were actually timestamps (or timestamp related), I could remove the differences.
From the original mail:** 2023-02-27 14:09:12 D: diffoscope.comparators.utils.command: Executing xxd {}
** KilledI've seen this too. You'll need loads of memory and temporary disk space, especially in the initial stages of finding the root cause of the differences (comparing 2 xxd outputs of 6GB files can produce lots of output) The original mail also hint at a possible root cause: the timestamp inside each partition have not been clamped to SOURCE_DATE_EPOCH. See e.g. [3] how it is done in live-build.
> It is possible to make reproducible ISOs by xorriso and SOURCE_DATE_EPOCH.
> But the input files for the ISO production have to be identical, too.Indeed, before the meta-content of ISO files can be considered, its payload should have been harmonized. Live-build does it in [3].
> Both, the use of SOURCE_DATE_EPOCH and care for identical input are not tradition with bootable ISO production.
>> So another reason for no protests might be that bootable ISOs aren't often challenged by reproducibility tests which use diffoscope.
I agree that having a specialised comparator would be nice, but even the hex dump has fulfilled its purpose. The live ISO images are now 100% reproducible (the header is containing a lot of the magic in the previous mails). The ISO images are tested with diffoscope on a daily basis in Jenkins [4].
With kind regards, Roland[1] https://sources.debian.org/src/live-build/1:20230131/scripts/build/binary_grub-efi/?hl=262#L262 [2] https://sources.debian.org/src/live-build/1:20230131/scripts/build/efi-image/?hl=79#L79 [3] https://sources.debian.org/src/live-build/1:20230131/scripts/build/binary_iso/?hl=180#L180
[4] https://jenkins.debian.net/view/live/
OpenPGP_signature
Description: OpenPGP digital signature
