Bug#942098: Wish for "mmdebstrap unstable unstable-chroot.squashfs"
Hi Trent, Quoting Trent W. Buck (2019-10-16 10:58:15) > tar2sqfs exists in squashfs-tools-ng. > I haven't gotten it working with mmdebstrap yet, but > this (below) is enough to show that the approach is sound. > Further work can happen in squashfs-land, not mmdebstrap-land. > > [snip awesome research] > > tar2sqfs can definitely see the files in the tar, because it prints their > names to stderr as they go past. maybe this is because mmdebstrap is doing something wrong with the device nodes? Maybe comment out that mmdebstrap prepends a tarball with /dev in front of the actual file content and see if it works then? > When I have more time, I'll try to debug this further. Awesome!! Feel free to keep me posted. :) cheers, josch signature.asc Description: signature
Bug#942098: Wish for "mmdebstrap unstable unstable-chroot.squashfs"
Just FYI / FTR, tar2sqfs exists in squashfs-tools-ng. I haven't gotten it working with mmdebstrap yet, but this (below) is enough to show that the approach is sound. Further work can happen in squashfs-land, not mmdebstrap-land. Trent W. Buck wrote: > 3. tar2squashfs is useful for this & other things. > I don't think it's possible, but I'll ask the squashfs people. squashfs-tools isn't the only squashfs-tools anymore, * squashfs-tools-ng https://lkml.org/lkml/2019/8/1/1154 https://github.com/AgentD/squashfs-tools-ng https://bugs.debian.org/932971 (wnpp ITP) * squashfskit https://bugs.debian.org/918480 Neither are in Debian yet. I grabbed squashfs-tools-ng and spent twenty minutes with it. * claims to solve tar2squashfs https://bugs.debian.org/942288 https://github.com/ plougher/squashfs-tools/issues/25 * claims to solve --one-file-system, too - yay! https://bugs.debian.org/586757 * This is working: tar c tmp.txt | ./tar2sqfs tmp.sq # generate from plain tar unsquashfs -ll tmp.sq # squashfs-tools can read it mount -o loop tmp.sq /mnt # linux 5.2 can mount it * This is *not* working: mmdebstrap ["go faster" options...] buster | ./tar2sqfs tmp2.sq # generate from mmdebstrap unsquashfs -ll tmp2.sq # squashfs-tools can't read it mount -o loop tmp2.sq /mnt # linux 5.2 can mount it, but it's empty file tmp2.sq # header looks OK ./rdsquashfs --describe tmp2.sq # even squashfs-tools-ng can't read it ==> tmp2.sq: reading filesystem tree: location out of bounds. tar2sqfs can definitely see the files in the tar, because it prints their names to stderr as they go past. When I have more time, I'll try to debug this further.
Bug#942098: Wish for "mmdebstrap unstable unstable-chroot.squashfs"
Hi, Quoting Trent W. Buck (2019-10-10 11:31:37) > It would cool if I could do an unprivileged mmdebstrap and get a .squashfs > instead of a tarball. I agree. That would be very cool! Unfortunately, I think there are some complications: 1. mmdebstrap can work completely without superuser privileges because in the end the output is just a file containing a certain arrangement of bytes and there is no sensible reason why one should need root privileges to figure out the right arrangement of bytes. This is easily possible with the tar output format because how the right arrangement of bytes for device nodes for example has to look like can easily be produced without being root. This is not possible with mksquashfs as it requires the files it packs to actually exists beforehand. 2. mmdebstrap allows to produce its output on stdout but mksquasfs can only output to either a block device or a regular file. 3. as you noted, squashfs is read-only, so mmdebstrap would need yet another command line argument so that the user can specify which paths they do not want in the final image. More CLI options mean yet more complexity and longer documentation. With tar as output format we don't need such an option because we can just do: $ mmdebstrap unstable - | tar --exclude=./dev > out.tar 4. mksquashfs has a ton of options which somehow have to be exposed via mmdebstrap which means yet even more CLI options and more complexity What we could do is to add squashfs support for all the cases where the above does not apply, so namely when mmdebstrap is run as root and the output is to be placed into a file. Namely when doing this: $ sudo mmdebstrap [...] unstable-chroot.squashfs But this also adds a bunch of complexity to an already very complex codebase because now we will need even more exception handling, error printing and documentation which exactly details what conditions have to be met for direct squashfs output. And then there is also the question of about all the additional options we will need to change compression type block size etc... Would it not help many more users if there was a tar2squashfs command which could also read from stdin? That command could then be used by an even wider audience for even more purposes. I agree squashfs is great but with the current tooling around it forbidding piping and root-less operation, it is not quite the perfect output format for mmdebstrap as tar is. So maybe you should rather approach mksquashfs upstream and talk about allowing tarballs as input? Sales pitch: you could create a squashfs with files owned by root without ever having needing root privileges. :) Thanks! cheers, josch signature.asc Description: signature
Bug#942098: Wish for "mmdebstrap unstable unstable-chroot.squashfs"
Package: mmdebstrap
Version: 0.5.0-1
Severity: wishlist
File: /usr/bin/mmdebstrap
It would cool if I could do an unprivileged mmdebstrap and get a .squashfs
instead of a tarball.
Sales pitch:
squashfs is a file format that can act both like an archive (replaces
.tar.xz):
unsquashfs -d foo/ foo.sq
and like a block device (replaces .iso):
sudo mount -o loop foo.sq foo/
squashfs-tools 4.4 understands SOURCE_DATE_EPOCH (reproducible builds).
Unlike .tar.xz, it is possible to list of extract a subtree without
decompressing the whole thing.
Unlike (some versions of) tar, it has sensible defaults, so you do
not get burned by forgetting --numeric-owner.
Debian Live images are squashfs-based for about a decade.
I can't read perl very well anymore, so it's not clear to me how to actually
patch this support in.
Looking at references to "@taropts", it seems like tar gets called about 4
times, and
you'd need an equivalent "mksquashfs" in each one of those cases? Yuk.
For reference, a basic usage is:
mksquashfs foo/ foo.squashfs
And a complicated horrible usage is (in bash):
# NB: with -regex, each path component is matched as a separate,
*unanchored* ERE.
# If you just pass 'usr/share/info/', it will also exclude
usr/share/zoneinfo.
exclusions=(
# Since boot/* is needed outside the squashfs, don't duplicate it
inside.
'^boot$/.'
# Filesystems created at boot time.
'^(dev|tmp|run)$/.'
'^var$/^(lock|run|tmp)$/.'
# Cryptographic keys MUST NOT be baked in.
'^etc$/^dropbear$/^dropbear_.*_host_key$'
'^etc$/^ssh$/^ssh_host_.*_key(.pub)?$'
'^etc$/^ssl$/^certs$/^ssl-cert-snakeoil.pem$'
'^etc$/^ssl$/^private$/^ssl-cert-snakeoil.key$'
# Build-time configuration and cache.
'^etc$/^machine-id$'
'^etc$/^(debian_chroot|hostname|hosts|motd(\.tail)?|resolv.conf)$'
'^etc$/^apt$/^apt.conf.d$/^10bootstrap$'
'^etc$/^network$/^interfaces$'
'^usr$/^sbin$/^policy-rc\.d$'
'^var$/^cache$/^apt$/^(src)?pkgcache\.bin$'
'^var$/^cache$/^apt$/^archives$/\.deb$'
'^var$/^cache$/^bootstrap$'
'^var$/^lib$/^apt$/^lists$/.'
'^var$/^log$/.'
# No documentation (safety net for dpkg.cfg.d/75no-docs).
'^usr$/^share$/^(doc|info|man|omf|help)$/.'
'^usr$/^share$/^gnome$/^help$/.'
)
mksquashfs target/ target/boot/filesystem.squashfs -b 1M -comp xz -Xbcj x86
-Xdict-size 100% -no-recovery -regex -e "${exclusions[@]}"
-- System Information:
Debian Release: 10.1
APT prefers stable
APT policy: (990, 'stable'), (500, 'stable-updates'), (500,
'proposed-updates'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8),
LANGUAGE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mmdebstrap depends on:
ii apt 1.8.2
ii perl 5.28.1-6
ii perl-doc 5.28.1-6
Versions of packages mmdebstrap recommends:
ii arch-test 0.15-2
ii fakechroot 2.19-3.2
ii fakeroot 1.23-1
ii mount 2.33.1-0.1
ii pseudo [fakeroot] 1.9.0+git20180920-1
ii uidmap 1:4.5-1.1
Versions of packages mmdebstrap suggests:
pn binfmt-support
ii dpkg-dev 1.19.7
ii proot 5.1.0-1.3
pn qemu-user
pn qemu-user-static
-- no debconf information
