Hi folks, This is the EROFS DEFLATE support which I've been working on these months, mainly implementing DEFLATE fixed-sized output approach.
DEFLATE is a popular generic-purpose compression algorithm for a quite long time (many advanced formats like zlib, gzip, zip, png are all based on that), which is made of LZ77 as wells as Huffman code, fully documented as RFC1951 and quite easy to understand. DEFLATE encoder and decoder are also easy to be implemented and they generally have fairly small code size and runtime memory consumption. In addition, there are several hardware on-market DEFLATE accelerators as well, such as (s390) DFLTCC, (Intel) IAA/QAT, (HiSilicon) ZIP accelerator, etc. Therefore, it's useful to support DEFLATE in order to use these for async I/Os and get benefits from these later. As usual, some numbers of enwik8 and silesia.tar are available below: Commandline option: -zdeflate,9 (to match mksquashfs default level) Sliding window size: 32768 enwik8: ____________________________ | file system | size | |_______________|____________| |___ enwik8 ____|_100000000__| |_ squashfs_4k _|__46325760__| |_ squashfs_8k _|__43618304__| |__ erofs_4k ___|__43073536__| (erofs default) | squashfs_16k _|__41406464__| |__ erofs_8k ___|__40873984__| |__ erofs_16k __|__38998016__| | squashfs_64k _|__38109184__| | squashfs_128k |__37306368__| (squashfs default) |__ erofs_64k __|__37142528__| silesia.tar (much similar to enwik8 so I omit some): ____________________________ | file system | size | |_______________|____________| |_ silesia.tar _|_211957760__| |_ squashfs_4k _|__86884352__| |_ squashfs_8k _|__81125376__| |__ erofs_4k ___|__78970880__| (erofs default) |__ erofs_8k ___|__74768384__| | squashfs_64k _|__70447104__| | squashfs_128k |__69079040__| (squashfs default) |__ erofs_64k __|__68640768__| Thanks, Gao Xiang Gao Xiang (4): erofs-utils: switch to effective unaligned access erofs-utils: add a built-in DEFLATE compressor erofs-utils: fuse: add DEFLATE algorithm support erofs-utils: mkfs: add DEFLATE algorithm support configure.ac | 45 ++ dump/Makefile.am | 3 +- fsck/Makefile.am | 6 +- fuse/Makefile.am | 2 +- include/erofs/defs.h | 24 +- include/erofs/internal.h | 2 - include/erofs_fs.h | 7 + lib/Makefile.am | 2 + lib/compress.c | 24 + lib/compressor.c | 1 + lib/compressor.h | 1 + lib/compressor_deflate.c | 77 +++ lib/decompress.c | 147 +++++ lib/dedupe.c | 23 +- lib/kite_deflate.c | 1267 ++++++++++++++++++++++++++++++++++++++ 15 files changed, 1618 insertions(+), 13 deletions(-) create mode 100644 lib/compressor_deflate.c create mode 100644 lib/kite_deflate.c -- 2.24.4
