Hi Brandon, On Tue, 27 Dec 2022 at 09:16, Brandon Maier <[email protected]> wrote: > > Update the zstd implementation to match Linux zstd 1.5.2 from commit > 2aa14b1ab2. > > This was motivated by running into decompression corruption issues when > trying to uncompress files compressed with newer versions of zstd. zstd > users also claim significantly improved decompression times with newer > zstd versions which is a side benefit. > > Original zstd code was copied from Linux commit 2aa14b1ab2 which is a > custom-built implementation based on zstd 1.3.1. Linux switched to an > implementation that is a copy of the upstream zstd code in Linux commit > e0c1b49f5b, this results in a large code diff. However this should make > future updates easier along with other benefits[1]. > > This commit is a straight mirror of the Linux zstd code, except to: > - update a few #include that do not translate cleanly > - linux/swab.h -> asm/byteorder.h > - linux/limits.h -> linux/kernel.h > - linux/module.h -> linux/compat.h > - remove assert() from debug.h so it doesn't conflict with u-boot's > assert() > - strip out the compressor code as was done in the previous u-boot zstd > - update existing zstd users to the new Linux zstd API > - add a new KConfig option that sets zstd build options to minify code > based on zstd's ZSTD_LIB_MINIFY[2]. > > These changes were tested by booting a zstd 1.5.2 compressed kernel inside a > FIT. And the squashfs changes by loading a file from zstd compressed squashfs > with sqfsload. buildman was used to compile test other boards and check for > binary bloat, as follows: > > > $ buildman -b zstd2 --boards > > dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia > > -sS > > Summary of 3 commits for 9 boards (8 threads, 1 job per thread) > > 01: Merge tag 'tpm-23122022' of > > https://source.denx.de/u-boot/custodians/u-boot-tpm > > arm: w+ m53menlo dh_imx6 > > 02: lib: zstd: update to latest Linux zstd 1.5.2 > > aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0 > > arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)? > > sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0 > > [1] > https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8 > [2] > https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31 > > Signed-off-by: Brandon Maier <[email protected]> > --- > v2: > - zstd_decompress(): Handle junk at the end of the decompression buffer > by checking actual size with zstd_find_frame_compressed_size(). > --- > fs/squashfs/sqfs_decompressor.c | 15 +- > include/linux/zstd.h | 1244 ++------ > include/linux/zstd_errors.h | 77 + > include/linux/zstd_lib.h | 2551 +++++++++++++++++ > lib/Kconfig | 15 + > lib/zstd/Makefile | 35 +- > lib/zstd/bitstream.h | 344 --- > lib/zstd/common/bitstream.h | 446 +++ > lib/zstd/common/compiler.h | 184 ++ > lib/zstd/common/cpu.h | 194 ++ > lib/zstd/common/debug.c | 24 + > lib/zstd/common/debug.h | 92 + > lib/zstd/common/entropy_common.c | 357 +++ > lib/zstd/common/error_private.c | 56 + > lib/zstd/common/error_private.h | 145 + > lib/zstd/common/fse.h | 711 +++++ > lib/zstd/common/fse_decompress.c | 390 +++ > lib/zstd/common/huf.h | 358 +++ > lib/zstd/common/mem.h | 261 ++ > lib/zstd/common/portability_macros.h | 93 + > lib/zstd/common/zstd_common.c | 83 + > lib/zstd/common/zstd_deps.h | 125 + > lib/zstd/common/zstd_internal.h | 443 +++ > lib/zstd/decompress.c | 2516 ---------------- > lib/zstd/decompress/huf_decompress.c | 1740 +++++++++++ > lib/zstd/decompress/zstd_ddict.c | 241 ++ > lib/zstd/decompress/zstd_ddict.h | 44 + > lib/zstd/decompress/zstd_decompress.c | 2131 ++++++++++++++ > lib/zstd/decompress/zstd_decompress_block.c | 2072 +++++++++++++ > lib/zstd/decompress/zstd_decompress_block.h | 68 + > .../decompress/zstd_decompress_internal.h | 228 ++ > lib/zstd/decompress_sources.h | 34 + > lib/zstd/entropy_common.c | 213 -- > lib/zstd/error_private.h | 43 - > lib/zstd/fse.h | 545 ---- > lib/zstd/fse_decompress.c | 302 -- > lib/zstd/huf.h | 182 -- > lib/zstd/huf_decompress.c | 930 ------ > lib/zstd/mem.h | 142 - > lib/zstd/zstd.c | 52 +- > lib/zstd/zstd_common.c | 66 - > lib/zstd/zstd_common_module.c | 32 + > lib/zstd/zstd_decompress_module.c | 105 + > lib/zstd/zstd_internal.h | 253 -- > lib/zstd/zstd_opt.h | 1004 ------- > 45 files changed, 13638 insertions(+), 7548 deletions(-) > create mode 100644 include/linux/zstd_errors.h > create mode 100644 include/linux/zstd_lib.h > delete mode 100644 lib/zstd/bitstream.h > create mode 100644 lib/zstd/common/bitstream.h > create mode 100644 lib/zstd/common/compiler.h > create mode 100644 lib/zstd/common/cpu.h > create mode 100644 lib/zstd/common/debug.c > create mode 100644 lib/zstd/common/debug.h > create mode 100644 lib/zstd/common/entropy_common.c > create mode 100644 lib/zstd/common/error_private.c > create mode 100644 lib/zstd/common/error_private.h > create mode 100644 lib/zstd/common/fse.h > create mode 100644 lib/zstd/common/fse_decompress.c > create mode 100644 lib/zstd/common/huf.h > create mode 100644 lib/zstd/common/mem.h > create mode 100644 lib/zstd/common/portability_macros.h > create mode 100644 lib/zstd/common/zstd_common.c > create mode 100644 lib/zstd/common/zstd_deps.h > create mode 100644 lib/zstd/common/zstd_internal.h > delete mode 100644 lib/zstd/decompress.c > create mode 100644 lib/zstd/decompress/huf_decompress.c > create mode 100644 lib/zstd/decompress/zstd_ddict.c > create mode 100644 lib/zstd/decompress/zstd_ddict.h > create mode 100644 lib/zstd/decompress/zstd_decompress.c > create mode 100644 lib/zstd/decompress/zstd_decompress_block.c > create mode 100644 lib/zstd/decompress/zstd_decompress_block.h > create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h > create mode 100644 lib/zstd/decompress_sources.h > delete mode 100644 lib/zstd/entropy_common.c > delete mode 100644 lib/zstd/error_private.h > delete mode 100644 lib/zstd/fse.h > delete mode 100644 lib/zstd/fse_decompress.c > delete mode 100644 lib/zstd/huf.h > delete mode 100644 lib/zstd/huf_decompress.c > delete mode 100644 lib/zstd/mem.h > delete mode 100644 lib/zstd/zstd_common.c > create mode 100644 lib/zstd/zstd_common_module.c > create mode 100644 lib/zstd/zstd_decompress_module.c > delete mode 100644 lib/zstd/zstd_internal.h > delete mode 100644 lib/zstd/zstd_opt.h >
