Hi Yue, On Wed, Aug 03, 2022 at 11:51:27AM +0800, Yue Hu wrote: > In order to achieve greater compression ratio, let's introduce > compressed fragments feature which can merge tail of per-file or the > whole files into one special inode to reach the target. > > And we can also set pcluster size to fragments inode for different > compression requirments. > > In this patchset, we also improve the uncompressed data layout of > compressed files. Just write it from 'clusterofs' instead of 0 since it > can benefit from in-place I/O. For now, it only goes with fragments. > > The main idea above is from Xiang.
Thanks for your hard work! I will take a deep try this weekend, Also I'd like to enable logical cluster size != 4k for big pcluster with large pclustersize in order to reduce the size of compression indexes. In such cases, I think compact indexes are unnecessary. I think it's already supported on the kernel side, so we just need to implement the userspace side. Thanks, Gao Xiang > > Here is some test data of Linux 5.10.87 source code under Ubuntu 18.04: > > linux-5.10.87 (erofs, uncompressed) 1.1G > > linux-5.10.87 (erofs, lz4hc,12 4k fragments,4k) 301M > linux-5.10.87 (erofs, lz4hc,12 8k fragments,8k) 268M > linux-5.10.87 (erofs, lz4hc,12 16k fragments,16k) 242M > linux-5.10.87 (erofs, lz4hc,12 32k fragments,32k) 225M > linux-5.10.87 (erofs, lz4hc,12 64k fragments,64k) 217M > > linux-5.10.87 (erofs, lz4hc,12 4k vanilla) 396M > linux-5.10.87 (erofs, lz4hc,12 8k vanilla) 376M > linux-5.10.87 (erofs, lz4hc,12 16k vanilla) 364M > linux-5.10.87 (erofs, lz4hc,12 32k vanilla) 359M > linux-5.10.87 (erofs, lz4hc,12 64k vanilla) 358M > > Usage: > mkfs.erofs -zlz4hc,12 -C65536 -Efragments,65536 foo.erofs.img foo/ > > Changes since v2: > - mainly reimplment the decompression logic for fragment inode due to > kernel side; > - fix compatibility issue to old image with ztailpacking feature; > - move code of super.c in patch 3/3 to patch 1/3; > - minor naming change. > > Changes since v1: > - mainly optimize index space for fragment inode; > - add merging tail with len <= pclustersize into fragments directly; > - use a inode instead of nid to avoid multiple load fragments; > - fix memory leak of building fragments; > - minor change to diff special fragments with normal inode. > - rebase to commit cb058526 with patch [1]; > - code cleanup. > > Note that inode will be extended version (64 bytes) due to mtime, may > use 'force-inode-compact' option to reduce the size if mtime careless. > > [1] > https://lore.kernel.org/linux-erofs/[email protected]/ > > Yue Hu (3): > erofs-utils: lib: add support for fragments data decompression > erofs-utils: lib: support on-disk offset for shifted decompression > erofs-utils: introduce compressed fragments support > > include/erofs/compress.h | 3 +- > include/erofs/config.h | 3 +- > include/erofs/decompress.h | 3 ++ > include/erofs/fragments.h | 25 +++++++++ > include/erofs/inode.h | 2 + > include/erofs/internal.h | 9 ++++ > include/erofs_fs.h | 27 +++++++--- > lib/Makefile.am | 4 +- > lib/compress.c | 108 +++++++++++++++++++++++++++---------- > lib/data.c | 28 +++++++++- > lib/decompress.c | 10 +++- > lib/fragments.c | 76 ++++++++++++++++++++++++++ > lib/inode.c | 43 ++++++++++----- > lib/super.c | 24 ++++++++- > lib/zmap.c | 26 +++++++++ > mkfs/main.c | 64 +++++++++++++++++++--- > 16 files changed, 393 insertions(+), 62 deletions(-) > create mode 100644 include/erofs/fragments.h > create mode 100644 lib/fragments.c > > -- > 2.17.1
