Firstly, many thanks for Julian for starting this thread; I've had far
too much fun delving back into some SQL+jupyter notebooks today while
gathering a bit more info on this...
If you just want to go see or play with the data (and possibly expand it
with some more samples; I'd love to see some riscv64 data in there!):
https://github.com/waveform80/compression/blob/master/analysis.ipynb
Otherwise...
On Wed, Dec 08, 2021 at 06:12:43PM +0100, Julian Andres Klode wrote:
[snip]
# Way(s) forward
To remedy the issue the proposal is to build with
- zstd -1 on hardware with 512 MB or less memory
- zstd between -1 and -19 on other hardware
- zstd -19 during image building
Finding the right level between -1 and -19 is hard. The more
cores you have, the less penalty you pay for higher level.
Looking at my results, I'm reasonably convinced that -19 is overkill in
pretty much every case (perhaps even image building). The gains in
compression are minimal beyond about -7. That said, I'm absolutely
convinced that zstd -1 is a win (in almost all circumstances).
Going for adaptive compression would remove the guess work, but
will result in larger images on faster machines. Maybe that's
fine, though - they probably have more space on /boot anyway?
If we want to aim for 5% of total memory, we should probably
aim for something like:
-1 on <= 512MB
-6 on <= 2 GB (or --adapt=min=1,max=6)
-12 on the rest (or --adapt=min=12)
On Pi Zero 2 (and 3A+) with 512MB total RAM, it's worth bearing in mind
that the GPU firmware defaults to grabbing 64MB of that, and with
everything else loaded there's typically only 200-250MB left at runtime
(the arm64 architecture is particularly memory hungry). As a result,
zstd even at level -1 is already well above 5% of "idle" memory usage
(and pushing for 10%). However, it's probably still a viable (and
reasonable) option.
It's clear that in all cases, zstd -1 is at least better than the
lz4 -9 we used before; both in terms of space used, and time spent.
Yup, confirmed in all cases here too.
# Concerns
Lowering the compression level will reduce the boot speed by fractions
of a second on hardware with fast I/O.
Looking at the figures in my database, I'm not concerned about this. On
the platforms where zstd -19 was actually viable (big fat PC platforms)
the difference in output size would lead to I/O differences in the
tenths of seconds. Meanwhile on the slower platforms (i.e. the Pi), the
largest difference was ~10MB in output size and even at SD card speeds,
that's only 1 second extra on boot (and for those platforms that's a
tiny fraction of the overall boot time).
Cheers,
Dave.
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
