Hi Xiang, On Wed, 3 Aug 2022 14:07:24 +0800 Gao Xiang <[email protected]> wrote:
> Hi Yue, > > On Wed, Aug 03, 2022 at 11:51:27AM +0800, Yue Hu wrote: > > In order to achieve greater compression ratio, let's introduce > > compressed fragments feature which can merge tail of per-file or the > > whole files into one special inode to reach the target. > > > > And we can also set pcluster size to fragments inode for different > > compression requirments. > > > > In this patchset, we also improve the uncompressed data layout of > > compressed files. Just write it from 'clusterofs' instead of 0 since it > > can benefit from in-place I/O. For now, it only goes with fragments. > > > > The main idea above is from Xiang. > > Thanks for your hard work! I will take a deep try this weekend, Got it. > > Also I'd like to enable logical cluster size != 4k for big pcluster with > large pclustersize in order to reduce the size of compression indexes. Let me think about this first. Thanks. > > In such cases, I think compact indexes are unnecessary. I think it's > already supported on the kernel side, so we just need to implement the > userspace side. > > Thanks, > Gao Xiang > > > > > Here is some test data of Linux 5.10.87 source code under Ubuntu 18.04: > > > > linux-5.10.87 (erofs, uncompressed) 1.1G > > > > linux-5.10.87 (erofs, lz4hc,12 4k fragments,4k) 301M > > linux-5.10.87 (erofs, lz4hc,12 8k fragments,8k) 268M > > linux-5.10.87 (erofs, lz4hc,12 16k fragments,16k) 242M > > linux-5.10.87 (erofs, lz4hc,12 32k fragments,32k) 225M > > linux-5.10.87 (erofs, lz4hc,12 64k fragments,64k) 217M > > > > linux-5.10.87 (erofs, lz4hc,12 4k vanilla) 396M > > linux-5.10.87 (erofs, lz4hc,12 8k vanilla) 376M > > linux-5.10.87 (erofs, lz4hc,12 16k vanilla) 364M > > linux-5.10.87 (erofs, lz4hc,12 32k vanilla) 359M > > linux-5.10.87 (erofs, lz4hc,12 64k vanilla) 358M > > > > Usage: > > mkfs.erofs -zlz4hc,12 -C65536 -Efragments,65536 foo.erofs.img foo/ > > > > Changes since v2: > > - mainly reimplment the decompression logic for fragment inode due to > > kernel side; > > - fix compatibility issue to old image with ztailpacking feature; > > - move code of super.c in patch 3/3 to patch 1/3; > > - minor naming change. > > > > Changes since v1: > > - mainly optimize index space for fragment inode; > > - add merging tail with len <= pclustersize into fragments directly; > > - use a inode instead of nid to avoid multiple load fragments; > > - fix memory leak of building fragments; > > - minor change to diff special fragments with normal inode. > > - rebase to commit cb058526 with patch [1]; > > - code cleanup. > > > > Note that inode will be extended version (64 bytes) due to mtime, may > > use 'force-inode-compact' option to reduce the size if mtime careless. > > > > [1] > > https://lore.kernel.org/linux-erofs/[email protected]/ > > > > Yue Hu (3): > > erofs-utils: lib: add support for fragments data decompression > > erofs-utils: lib: support on-disk offset for shifted decompression > > erofs-utils: introduce compressed fragments support > > > > include/erofs/compress.h | 3 +- > > include/erofs/config.h | 3 +- > > include/erofs/decompress.h | 3 ++ > > include/erofs/fragments.h | 25 +++++++++ > > include/erofs/inode.h | 2 + > > include/erofs/internal.h | 9 ++++ > > include/erofs_fs.h | 27 +++++++--- > > lib/Makefile.am | 4 +- > > lib/compress.c | 108 +++++++++++++++++++++++++++---------- > > lib/data.c | 28 +++++++++- > > lib/decompress.c | 10 +++- > > lib/fragments.c | 76 ++++++++++++++++++++++++++ > > lib/inode.c | 43 ++++++++++----- > > lib/super.c | 24 ++++++++- > > lib/zmap.c | 26 +++++++++ > > mkfs/main.c | 64 +++++++++++++++++++--- > > 16 files changed, 393 insertions(+), 62 deletions(-) > > create mode 100644 include/erofs/fragments.h > > create mode 100644 lib/fragments.c > > > > -- > > 2.17.1
