Context for folks I've added to CC: The topic of reproducible ext4 filesystem image generation came up on the reproducible builds mailing list. I was aware of your prior art, and figured I would bring it to the attention of those working on reproducibility, both to assist the OP in their pursuit, and moreover in the hopes of further reproducibility validation / testing / any fixes to the approach you all have taken.
On Wed, Jan 22, 2025 at 3:49 PM David A. Wheeler via rb-general <[email protected]> wrote: > > > On Jan 9, 2025, at 12:22 AM, [email protected] wrote: > > > > Hi All, > > > > I am working towards reproducible builds for a project that I am involved > > in. We use a few ext4 partitions in our disk images and I am trying to make > > the ext4 filesystems reproducible. > > I'm a little concerned with this as a goal, at least if your goal is to > detect & counter builds that don't do what they claim to do. > > Different versions of operating systems will generate different bit images > for a given partition, and of course, not everyone uses ext4. If you must, I > imagine you should userspace tools (over which you have complete control & > can fix version numbers), then run them to make changes in a sequential order. > > For most cases I can think of, they should be compared file-by-file and > dir-by-dir, ignoring the filesystem. But your use case may be VERY different > from what I have in mind. > > --- David A. Wheeler Along exactly these lines, NixOS uses tooling to construct ext filesystem images entirely in userspace, which, to quote a comment from their integration thereof [1]: > `make-disk-image` has a bit of magic to minimize the amount of work to do in > a virtual machine. > > It relies on the LKL (Linux Kernel Library) project [2] which provides Linux > kernel as userspace library. Specifically, they use the `cptofs` tool [3] from that project, which is a convenient single-command "just manipulate the filesystem image in this file to add the contents from this directory", in a manner straightforward for use from build scripts, etc. I'll note that LKL also includes a FUSE filesystem that calls into the same linux-as-library in userspace filesystem implementation, for more fine-grained interactive manipulation of filesystem images in a manner which, one would suppose, ought to be able to be more controllable than "whatever the kernel you're currently running on will do". In the NixOS use case, I believe the primary motivation was to be able to build full filesystem images without needing to expose authority to the build sandbox to do things like mount filesystems, and without the authority of needing to expose KVM, and without the overhead of needing to run unaccelerated [tcg] qemu to achieve similar. Even if not directly for reproducible builds (though the authors may well have had that in mind as well), it still aligns the dependencies of filesystem manipulation to be normal userspace tools that can be controlled and versioned just like any other tools, in the manner David rightly recommended above. I have not validated myself whether the tooling as implemented in NixOS unconditionally produces reproducible results today, but it's a much more sound starting point than implicitly relying on the behavior of the filesystem implementation of whatever kernel you happen to be running on, which is unlikely to be within the control of your build system, and which is IMO unwise to make assumptions about with respect to forward format stability. None of the above ideas are mine, I'm just pointing you at prior art by other fine folks I happen to be aware of. Hope that sets you on a useful path. If you do end up finding reproducibility issues, particularly with the above approach (which, as far as I'm aware, would be the most promising approach if you want general reproducible ext filesystem images), then fixing them in the aforementioned tooling would also benefit Nix, and be most appreciated :) Adding some folks who have worked on the above to CC, for awareness & if they should wish to add anything: - Dan Peebles, who was the one who did the initial work to make use of use LKL in NixOS' image-building infrastructure, in [4] with discussion at [5]. - Ryan Lahfa, who documented the approach and made the process more deterministic [reproducible], by doing things like using fixed (instead of random) UUIDs/GUIDs for parititions, filesystems, etc. in [6] with discussion at [7]. - Jörg Thalheim, who provided review of the above, worked on LKL's cptofs tool in a way which also serves reproducible builds (explicit resulting file ownership within the disk image independent of wherever they're being supplied from for copying), and was incidentally the last to touch the cptofs tool in the LKL project. - lassulus, who maintains disk image generation tooling for NixOS. Regards, Jean-Philippe [1]: https://github.com/NixOS/nixpkgs/blob/936f4e016d49cbc8086961732927a4c297ab7c49/nixos/lib/make-disk-image.nix#L2-L24 [2]: https://github.com/lkl/linux [3]: https://github.com/lkl/linux/blob/master/tools/lkl/cptofs.c [4]: https://github.com/NixOS/nixpkgs/commit/f1708a9d7d79e2bf2961fc648625578b23b3460f [5]: https://github.com/NixOS/nixpkgs/pull/24964 [6]: https://github.com/NixOS/nixpkgs/commit/22adcaa4491dde18442a234252e1d7ed8c098672 [7]: https://github.com/NixOS/nixpkgs/pull/207038
