In order to achieve greater compression ratio, let's introduce compressed fragments feature which can merge tail of per-file or the whole files into one special inode to reach the target.
And we can also set pcluster size to fragments inode for different compression requirments. In this patchset, we also improve the uncompressed data layout of compressed files. Just write it from 'clusterofs' instead of 0 since it can benefit from in-place I/O. For now, it only goes with fragments. The main idea above is from Xiang. Here is some test data of Linux 5.10.87 source code under Ubuntu 18.04: linux-5.10.87 (erofs, uncompressed) 1.1G linux-5.10.87 (erofs, lz4hc,12 4k fragments,4k) 301M linux-5.10.87 (erofs, lz4hc,12 8k fragments,8k) 268M linux-5.10.87 (erofs, lz4hc,12 16k fragments,16k) 242M linux-5.10.87 (erofs, lz4hc,12 32k fragments,32k) 225M linux-5.10.87 (erofs, lz4hc,12 64k fragments,64k) 217M linux-5.10.87 (erofs, lz4hc,12 4k vanilla) 396M linux-5.10.87 (erofs, lz4hc,12 8k vanilla) 376M linux-5.10.87 (erofs, lz4hc,12 16k vanilla) 364M linux-5.10.87 (erofs, lz4hc,12 32k vanilla) 359M linux-5.10.87 (erofs, lz4hc,12 64k vanilla) 358M Usage: mkfs.erofs -zlz4hc,12 -C65536 -Efragments,65536 foo.erofs.img foo/ Changes since v2: - mainly reimplment the decompression logic for fragment inode due to kernel side; - fix compatibility issue to old image with ztailpacking feature; - move code of super.c in patch 3/3 to patch 1/3; - minor naming change. Changes since v1: - mainly optimize index space for fragment inode; - add merging tail with len <= pclustersize into fragments directly; - use a inode instead of nid to avoid multiple load fragments; - fix memory leak of building fragments; - minor change to diff special fragments with normal inode. - rebase to commit cb058526 with patch [1]; - code cleanup. Note that inode will be extended version (64 bytes) due to mtime, may use 'force-inode-compact' option to reduce the size if mtime careless. [1] https://lore.kernel.org/linux-erofs/[email protected]/ Yue Hu (3): erofs-utils: lib: add support for fragments data decompression erofs-utils: lib: support on-disk offset for shifted decompression erofs-utils: introduce compressed fragments support include/erofs/compress.h | 3 +- include/erofs/config.h | 3 +- include/erofs/decompress.h | 3 ++ include/erofs/fragments.h | 25 +++++++++ include/erofs/inode.h | 2 + include/erofs/internal.h | 9 ++++ include/erofs_fs.h | 27 +++++++--- lib/Makefile.am | 4 +- lib/compress.c | 108 +++++++++++++++++++++++++++---------- lib/data.c | 28 +++++++++- lib/decompress.c | 10 +++- lib/fragments.c | 76 ++++++++++++++++++++++++++ lib/inode.c | 43 ++++++++++----- lib/super.c | 24 ++++++++- lib/zmap.c | 26 +++++++++ mkfs/main.c | 64 +++++++++++++++++++--- 16 files changed, 393 insertions(+), 62 deletions(-) create mode 100644 include/erofs/fragments.h create mode 100644 lib/fragments.c -- 2.17.1
