Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
Bernhard M. Wiedemann wrote: > There are ways to read files off a remote iso without downloading the > whole thing: > https://github.com/bmwiedemann/curlwwwfs + fuseiso > or maybe > https://github.com/higlass/simple-httpfs I am sure there are many neat hacks for doing that sort of thing, but IMHO all of them seem at odds with making the process of reproducing an artifact as simple, straightforward and (most importantly) as transparent as possible. Regards, -- o ⬋ ⬊ Chris Lamb o o reproducible-builds.org ⬊ ⬋ o
Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
Hey John, > I'm very glad to see reproducible Debian releases being built! > Congratulations! Is this a feature that Debian values enough to put it > in its major releases, or is it an offshoot? 11.0.0 just came out this > month: was it reproducible? Or if not, perhaps 12 will be? Just to jump in here on behalf of Roland... This is actually all about various *live-build* images. That is, builds of the handful of "live" ISO images that can be built using a tool of that name. It has little to do with the official Debian installation media (ie. ISOs that use the Debian Installer, also known as "d-i"), and nor does it refer to the Debian distribution as a whole. Your queries about whether the recent release of bullseye 11.0.0 was reproducible or not made that bit worthy of a quick clarification. (As a technical aside, Debian live images are constructed in such a way that means that the underlying packages do not need to be reproducible in the usual "source → binary" sense for the build of the *live* image to be reproducible/nondeterministic. This is what makes a statement such as "live image X of Debian Y is reproducible" potentially misleading about whether Debian Y is itself reproducible — alas, it's likely not.) Finally, this is still all an unofficial effort at least at the moment... Still, really admire your keenness and am heartened by the progress in the right direction. Hope the above helps. Best wishes, -- o ⬋ ⬊ Chris Lamb o o reproducible-builds.org ⬊ ⬋ o
Re: Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
I'm very glad to see reproducible Debian releases being built! Congratulations! Is this a feature that Debian values enough to put it in its major releases, or is it an offshoot? 11.0.0 just came out this month: was it reproducible? Or if not, perhaps 12 will be? Chris Lamb wrote: > Including the full config inside the ISO definitely seems like a good > idea, especially as this config is both small and will be compressed. Please include the config seeds for each binary release, in the *source* media. And the bootstrap script in both the source and binary media. I have long had this idea that it should be possible to take an OS release's live install media, and its matching source code media, and take those two offline to any compatible computer, with no Internet access. Boot the machine from the live install media, and run an included script that will rebuild a copy of the live install media from its own source code, using its own binaries to bootstrap it. (With this setup, you would also be able to rebuild a copy of *slightly modified* source media, e.g. with a few critical bug fixes patched. This would let you, or anyone, determine exactly which bits in which files in the resulting binary live install media were changed by those patches, by diffoscoping with the original binary live install media.) With such a bit of infrastructure, you could then make one of its target builds a very minimal binary live medium (netinst sized or much smaller) that would contain just the binaries needed to bootstrap all builds creatable from that source tree. Those bootstrap binaries could certainly rebuild their own bootstrap binary media (as above). But they could also rebuild the main live x86 or x86 release install media, from the config seed included in the source code media, without having any copy of the target binary media accessible to it! The size of this bootstrap ISO medium would show just how small the must-be-trusted bootstrap binary code is for any given OS release. For example, isn't the OS kernel the biggest remaining binary dependency? Does the GNU Mes bootstrap-reducing team have a plan to replace Grub and the Linux kernel and init (and perhaps a BIOS?) with something tiny that runs on bare metal and implements a file system, the mount command, and processes? Many realtime OS's are much smaller than Linux or BSD and yet have those capabilities. eCos might be a great start, and is free, highly portable, and includes a POSIX layer (and TCP/IP for debugging), though it currently lacks fork/exec/wait. The original V7 UNIX kernel would work, if process sizes and filename sizes are patched, and a few device drivers written for modern disk and CDROM drives. Such a bootstrap kernel would enable the Scheme bootstrap programs to run well enough to build gcc, then use gcc to build the Linux kernel, then boot it, and continue building. John
Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
On 31/08/2021 15.53, Chris Lamb wrote: > Indeed, needing to > extract parts of the ISO to recreate it is slightly sub-optimal, if > only because it would require someone to download it first before > attempting to recreate it (rather than just possessing the minuscule > .buildinfo file containing the inputs and output hashes). There are ways to read files off a remote iso without downloading the whole thing: https://github.com/bmwiedemann/curlwwwfs + fuseiso or maybe https://github.com/higlass/simple-httpfs However, it would also be possible to place them as tarball next to it, but then you add other challenges in toolchains and workflows, if you think about the separate .buildinfo vs ArchLinux embeeded one. How do you find the right buildinfo? What if someone only fetches the binary? OpenPGP_signature Description: OpenPGP digital signature
Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
Hi Roland, > > One question actually — how might a third-party reproduce these > > images? Or putting the same question in more technical terms — are you > > generating some kind of .buildinfo file that contains (just for > > example's sake) the value of SOURCE_DATE_EPOCH and any other relevant > > inputs, as well as the resulting checksums? > > Good question. I was initially interested in getting a reproducible > image, the next step would be record the required steps. > > The .buildinfo manpage [1] looks really tightly coupled to packages, so > (in its present form) it cannot record the information needed for > rebuilding a live-build ISO image. Yes, you are absolutely right that the .buildinfo spec outlined in the manpage you link is oriented towards packages and Debian packages in particular. But perhaps I should have been clearer: I was hypothesising about a file that is *analogous* to that Debian .buildinfo format (aka. the deb-buildinfo(5) spec), rather than using _that_ particular specification. As in, some new file that encodes the inputs (that you later list — thanks!) as well as the checksums of the outputs. (I don't think the deb-buildinfo(5) spec could be hacked to fit here tbh, although many things could obviously be inspired from it.) > If desired, the full configuration for the lb commands could be > embedded into the ISO image. Then you can, after obtaining a live > image, use the config provided there to rebuild exactly the same > image. Including the full config inside the ISO definitely seems like a good idea, especially as this config is both small and will be compressed. Still, an external build attestation document will always be needed to store the output checksums, so I wouldn't worry too much about trying to include everything within the ISO itself. Indeed, needing to extract parts of the ISO to recreate it is slightly sub-optimal, if only because it would require someone to download it first before attempting to recreate it (rather than just possessing the minuscule .buildinfo file containing the inputs and output hashes). Anyway, I *totally* ACK that you were getting this all working first before moving on... and I hope didn't come across as "Never Satisfied Mailing List Guy". Looking forward to seeing what you come up with. :) Best wishes, -- o ⬋ ⬊ Chris Lamb o o reproducible-builds.org ⬊ ⬋ o
Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
On 31/08/2021 12.11, Roland Clobus wrote: > There are a few things required to reproduce the images: > 1) The git hash for the live-config repository > 2) The value of SOURCE_DATE_EPOCH > 3) The command line of 'lb config' (which contains the path with > timestamp to the snapshot server) > 4) The list of additional packages (e.g. live-task-gnome) > 5) The additional 'hacks' required > > 3, 4 and 5 can also be obtained from the config directory, which is > generated by the 'lb config' command. What stops you from also adding 1 and 2 in the the config directory? One could then patch the tools to also pick it up from there. It could make life very convenient for a rebuilder. Just take the original iso, take its config dir and run a build with it to see if you get identical output. OpenPGP_signature Description: OpenPGP digital signature
Re: Recoding the configuration for live-build images (Was: Third status update about reproducible live-build ISO images in Jenkins)
Hello Chris, On 30/08/2021 18:58, Chris Lamb wrote: > One question actually — how might a third-party reproduce these > images? Or putting the same question in more technical terms — are you > generating some kind of .buildinfo file that contains (just for > example's sake) the value of SOURCE_DATE_EPOCH and any other relevant > inputs, as well as the resulting checksums? Good question. I was initially interested in getting a reproducible image, the next step would be record the required steps. The .buildinfo manpage [1] looks really tightly coupled to packages, so (in its present form) it cannot record the information needed for rebuilding a live-build ISO image. There are a few things required to reproduce the images: 1) The git hash for the live-config repository 2) The value of SOURCE_DATE_EPOCH 3) The command line of 'lb config' (which contains the path with timestamp to the snapshot server) 4) The list of additional packages (e.g. live-task-gnome) 5) The additional 'hacks' required 3, 4 and 5 can also be obtained from the config directory, which is generated by the 'lb config' command. If desired, the full configuration for the lb commands could be embedded into the ISO image. Then you can, after obtaining a live image, use the config provided there to rebuild exactly the same image. With kind regards, Roland Clobus [1] https://manpages.debian.org/bullseye/dpkg-dev/deb-buildinfo.5.en.html OpenPGP_signature Description: OpenPGP digital signature
