Re: [xz-devel] Memory requirement for linux kernel compression/decompression
On 2012-02-21 Gilles Espinasse wrote: > On 2.6.32, only lzma option is available and by default, kernel use > lzma -9 and this require much more memory than needed. As one > compiler of my distrib reported a compilation breakage on a 512 MB VM > during kernel compression, I started hacking scripts/Makefile.lib, > removed -9 and added -vv. I then played with information displayed > during compression to adjust xz memory requirement. lzma -9 from LZMA Utils uses 32 MiB dictionary and requires 311 MiB of memory. xz -9 uses 64 MiB dictionary and requires 674 MiB of memory. The lzma emulation in xz uses the same presets as xz, so lzma -9 from XZ Utils needs 674 MiB of memory. So the emulation isn't very good although by default both XZ Utils and LZMA Utils use 8 MiB dictionary. Using a dictionary bigger than the uncompressed file is waste of memory. So if the kernel image is small, switching to a much smaller dictionary doesn't affect compression ratio. > Should not a patch be pushed on LKLM to at least remove the -9 part? I don't know. If -9 is removed, then a kernel bigger than 8 MiB may compress worse than it does now. The -9 probably was put there when XZ Utils hadn't taken over LZMA Utils, so the memory usage was much lower. Using a high setting is fine from decompression point of view, because in the specific case of kernel decompression the dictionary size doesn't affect the decompressor memory usage. So from that point of view it is fine to use a high setting "just in case". scripts/xz_wrap.sh uses 32 MiB dictionary (370 MiB memory) to compress a kernel image with xz. Maybe that would work on 512 MiB VMs but it can still be a bit annoying on them. An alternative to local patching is to set a memory usage limit for xz when compiling the kernel: $ XZ_OPT=-M100MiB make xz will then scale down the dictionary size. It does it also when emulating lzma. > Secondly, could I trust the decompression memory requirement > displayed by xz? It can be trusted but: - It's rounded *up* to the nearest MiB, so it's not very precise when memory requirements are low. This could be fixed since a more accurate number is known internally already. - The number assumes that the decompressor needs to allocate a separate dictionary buffer. This isn't always the case. Linux kernel decompression doesn't need a dictionary buffer but initramfs and initrd decompression does. > Is the kernel decompressor really requiring the same > memory size that xz display during compression? No. Kernel decompression with a XZ-compressed kernel requires about 30 KiB of memory. The dictionary size doesn't matter because the output buffer is used as the dictionary buffer. This is done even when a BCJ filter is used. I think with a LZMA-compressed kernel the memory usage is very similar to XZ. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
[xz-devel] Memory requirement for linux kernel compression/decompression
Since some time I am the happy user of xz to compress linux kernel (2.6.27 patched, 2.6.32 and now 3.0). All tests below were made with xz-5.0.3. Machine has 1 GB memory, x86 32 bits. On 2.6.32, only lzma option is available and by default, kernel use lzma -9 and this require much more memory than needed. As one compiler of my distrib reported a compilation breakage on a 512 MB VM during kernel compression, I started hacking scripts/Makefile.lib, removed -9 and added -vv. I then played with information displayed during compression to adjust xz memory requirement. Using the value from linux kernel (lzma -9), adding -vv show as result lzma: Filter chain: --lzma1=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 lzma: 674 MiB of memory is required. The limit is 17592186044416 MiB. lzma: Decompression will need 65 MiB of memory. (stdin): 1413.3 KiB / 3437.4 KiB = 0.411, 741 KiB/s, 0:04 Removing -9 show as result lzma: Filter chain: --lzma1=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 lzma: 94 MiB of memory is required. The limit is 17592186044416 MiB. lzma: Decompression will need 9 MiB of memory. (stdin): 1413.3 KiB / 3437.4 KiB = 0.411, 747 KiB/s, 0:04 So still same size result with less memory requirement. What I elected to use as the data size to be compressed is know and near 3 MiB was sed -i 's/lzma -9/lzma -vv --lzma1=dict=3MiB/' scripts/Makefile.lib showing lzma: Filter chain: --lzma1=dict=3MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 lzma: 38 MiB of memory is required. The limit is 17592186044416 MiB. lzma: Decompression will need 4 MiB of memory. (stdin): 1413.0 KiB / 3437.4 KiB = 0.411, 752 KiB/s, 0:04 Still same size and even less memory requirement. Should not a patch be pushed on LKLM to at least remove the -9 part? I understand it could be not easy to do more tuning like adding --lzma1=dict= setting as that will not work with every lzma versions. Or a test will be required to identify lzma when xz in disguise, capable to support that option and adjust dictionary size. Secondly, could I trust the decompression memory requirement displayed by xz? Is the kernel decompressor really requiring the same memory size that xz display during compression? Gilles