[PATCH] erofs: fix ztailpacking for subpage compressed blocks

2023-12-14 Thread Gao Xiang
`pageofs_in` should be the compressed data offset of the page rather than of the block. Signed-off-by: Gao Xiang --- fs/erofs/zdata.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index a2c3e87d2f81..8264936b8612 100644 --- a/fs/erofs

Re: [PATCH v1] erofs-utils: fuse: support FUSE 2/3 multi-threading

2023-12-12 Thread Gao Xiang
} else { +#if FUSE_MINOR_VERSION < 2 Can we check FUSE_USE_VERSION directly? if FUSE_USE_VERSION is neither 30 or 32, let's raise a build error with #error unsupported FUSE_USE_VERSION instead. Thanks, Gao Xiang +

Re: [RFC KERNEL] initoverlayfs - a scalable initial filesystem

2023-12-11 Thread Gao Xiang
On 2023/12/12 15:35, Christoph Hellwig wrote: On Tue, Dec 12, 2023 at 08:50:56AM +0800, Gao Xiang wrote: For non-virtualization cases, I guess you could try to use `memmap` kernel option [2] to specify a memory region by bootloaders which contains an EROFS rootfs and a customized init

Re: [PATCH RFC v2 for-6.8/block 11/18] erofs: use bdev api

2023-12-11 Thread Gao Xiang
_metabuf() is not always called before erofs_bread() is used. For example, buf->inode can be one of directory inodes other than initialized by erofs_init_metabuf(). Thanks, Gao Xiang pgoff_t index = offset >> PAGE_SHIFT; struct page *page = buf->page; struct

Re: [RFC KERNEL] initoverlayfs - a scalable initial filesystem

2023-12-11 Thread Gao Xiang
ndividual filesystem anyway. [1] https://www.qemu.org/docs/master/system/devices/virtio-pmem.html [2] https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap Thanks, Gao Xiang Echo'ing Lennart we must also "keep in mi

Re: [PATCH 1/5] erofs: support I/O submission for sub-page compressed blocks

2023-12-11 Thread Gao Xiang
On 2023/12/6 17:10, Gao Xiang wrote: Add a basic I/O submission path first to support sub-page blocks: - Temporary short-lived pages will be used entirely; - In-place I/O pages can be used partially, but compressed pages need to be able to be mapped in contiguous virtual memory

Re: [PATCH 4/5] erofs: refine z_erofs_transform_plain() for sub-page block support

2023-12-07 Thread Gao Xiang
On 2023/12/8 15:34, Gao Xiang wrote: ... min_t(unsigned int, ,)? ../include/linux/minmax.h:21:28: error: comparison of distinct pointer types lacks a cast [-Werror]    (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1))) What compiler version are you using? I didn't find any error

Re: [PATCH 4/5] erofs: refine z_erofs_transform_plain() for sub-page block support

2023-12-07 Thread Gao Xiang
On 2023/12/8 13:20, Yue Hu wrote: On Wed, 6 Dec 2023 17:10:56 +0800 Gao Xiang wrote: Sub-page block support is still unusable even with previous commits if interlaced PLAIN pclusters exist. Such pclusters can be found if the fragment feature is enabled. This commit tries to handle

[PATCH 5/5] erofs: enable sub-page compressed block support

2023-12-06 Thread Gao Xiang
Let's just disable cached decompression and inplace I/Os for partial pages as a first step in order to enable sub-page block initial support. In other words, currently it works primarily based on temporary short-lived pages. Don't expect too much in terms of performance. Signed-off-by: Gao

[PATCH 4/5] erofs: refine z_erofs_transform_plain() for sub-page block support

2023-12-06 Thread Gao Xiang
ffc091e6f9 ("erofs: support interlaced uncompressed data for compressed files"). It uses a unique way for both shifted and interlaced PLAIN pclusters. As an added bonus, PLAIN pclusters larger than the block size is also supported now for the upcoming large lclusters. Signed-off-by: Gao

[PATCH 1/5] erofs: support I/O submission for sub-page compressed blocks

2023-12-06 Thread Gao Xiang
is explicitly disabled for sub-page blocks, which will be supported in the future. Signed-off-by: Gao Xiang --- fs/erofs/zdata.c | 156 ++- 1 file changed, 74 insertions(+), 82 deletions(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index a33cd6757f98

[PATCH 3/5] erofs: fix up compacted indexes for block size < 4096

2023-12-06 Thread Gao Xiang
`Z_EROFS_LI_D0_CBLKCNT`. To support smaller block sizes, `lobits` for each compacted lcluster is now calculated as: lobits = max(lclusterbits, ilog2(Z_EROFS_LI_D0_CBLKCNT) + 1) Signed-off-by: Gao Xiang --- fs/erofs/zmap.c | 32 ++-- 1 file changed, 14 insertions

[PATCH 0/5] erofs: basic sub-page compressed data support

2023-12-06 Thread Gao Xiang
temporary buffers all the time to keep compressed data if sub-page blocks are identified. In other words, no inplace/cache decompression for the preliminary support. This patchset survives EROFS stress test on my own testfarms. Thanks, Gao Xiang Gao Xiang (5): erofs: support I/O submission

[PATCH 2/5] erofs: record `pclustersize` in bytes instead of pages

2023-12-06 Thread Gao Xiang
-by: Gao Xiang --- fs/erofs/zdata.c | 64 1 file changed, 26 insertions(+), 38 deletions(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index 421c0a88a0ca..d02989466711 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -56,6 +56,9 @@ struct

[PATCH v2] erofs: fix lz4 inplace decompression

2023-12-05 Thread Gao Xiang
ompress big pcluster for lz4 backend") Cc: stable # 5.4+ Signed-off-by: Gao Xiang --- changes since v1: - address some nits pointed out by Juhyung. fs/erofs/decompressor.c | 31 --- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/fs/erofs/decompr

Re: [PATCH] erofs: fix lz4 inplace decompression

2023-12-05 Thread Gao Xiang
On 2023/12/6 12:39, Gao Xiang wrote: On 2023/12/6 12:28, Juhyung Park wrote: Hi Gao, Thanks for the timely fix. This also fixes the issue. Reported-and-tested-by: Juhyung Park Might wanna add Cc: sta...@vger.kernel.org # 5.4+ but it seems like this doesn't apply cleanly for 5.4, 5.10

Re: [PATCH] erofs: fix lz4 inplace decompression

2023-12-05 Thread Gao Xiang
conflict resolution. :)) I will handle these LTSes manually, so don't worry about that :) Wrote some nits below: On Wed, Dec 6, 2023 at 12:08 PM Gao Xiang wrote: Currently EROFS can map another compressed buffer for inplace decompression, which was used to handle the cases that some pages

Re: Weird EROFS data corruption

2023-12-05 Thread Gao Xiang
Hi Juhyung, On 2023/12/5 22:43, Juhyung Park wrote: On Tue, Dec 5, 2023 at 11:34 PM Gao Xiang wrote: ... I'm still analyzing this behavior as well as the root cause and I will also try to get a recent cloud server with FSRM myself to find more clues. Down the rabbit hole we go... Let

Re: Weird EROFS data corruption

2023-12-05 Thread Gao Xiang
On 2023/12/5 22:23, Juhyung Park wrote: Hi Gao, On Tue, Dec 5, 2023 at 4:32 PM Gao Xiang wrote: Hi Juhyung, On 2023/12/4 11:41, Juhyung Park wrote: ... - Could you share the full message about the output of `lscpu`? Sure: Architecture:x86_64 CPU op-mode(s

Re: Weird EROFS data corruption

2023-12-04 Thread Gao Xiang
startup latency Thanks, Gao Xiang

Re: Weird EROFS data corruption

2023-12-03 Thread Gao Xiang
t of `lscpu`? Thanks, Gao Xiang

Re: Weird EROFS data corruption

2023-12-03 Thread Gao Xiang
On 2023/12/4 01:01, Juhyung Park wrote: Hi Gao, On Mon, Dec 4, 2023 at 1:52 AM Gao Xiang wrote: Hi Juhyung, On 2023/12/4 00:22, Juhyung Park wrote: (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a while ago, which may mean that this is not specific to EROFS: https

Re: Weird EROFS data corruption

2023-12-03 Thread Gao Xiang
nce like different compliers? Thanks, Gao Xiang

[PATCH v2] erofs: fix memory leak on short-lived bounced pages

2023-11-28 Thread Gao Xiang
k like this, so it won't be impacted. Fixes: 67139e36d970 ("erofs: introduce `z_erofs_parse_in_bvecs'") Signed-off-by: Gao Xiang --- v2: - should be `be->compressed_pages`. fs/erofs/zdata.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/erofs/zdata.c b/fs/

[PATCH] erofs: fix memory leak on short-lived bounced pages

2023-11-28 Thread Gao Xiang
k like this, so it won't be impacted. Fixes: 67139e36d970 ("erofs: introduce `z_erofs_parse_in_bvecs'") Signed-off-by: Gao Xiang --- fs/erofs/zdata.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index a7e6847f6f8f..e5e

Re: [PATCH] erofs-utils: update .gitignore for test results and tools

2023-11-23 Thread Gao Xiang
On 2023/11/24 13:30, Yue Hu wrote: From: Yue Hu No need to track these generated test files. Signed-off-by: Yue Hu --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 33e5d30..5b9ecd0 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3

Re: [PATCH v2 2/2] erofs-utils: tests: fix build warning in test_LZ4_compress_HC_destSize

2023-11-23 Thread Gao Xiang
t; %d)\n", ~^ %ld Signed-off-by: Yue Hu Thanks, folded this with your SoB. Thanks, Gao Xiang

Re: [PATCH 2/2] erofs-utils: tests: fix build warning in test_LZ4_compress_HC_destSize

2023-11-23 Thread Gao Xiang
On 2023/11/24 11:50, Yue Hu wrote: On Fri, 24 Nov 2023 11:46:53 +0800 Gao Xiang wrote: Hi Yue, On 2023/11/24 11:36, Yue Hu wrote: From: Yue Hu badlz4.c:72:58: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘long unsigned int’ [-Wformat=] printf("

Re: [PATCH 2/2] erofs-utils: tests: fix build warning in test_LZ4_compress_HC_destSize

2023-11-23 Thread Gao Xiang
cSize <= DstSize) { printf("test LZ4_compress_HC_destSize(%d) error (%d < %d)\n", - inlen, SrcSize, sizeof(dst)); Could we just use printf(, (int)sizeof(dst)); instead? Thanks, Gao Xiang + inlen, SrcSize, DstSize); re

Re: [PATCH 1/2] erofs-utils: tests: double-quote entries assignment in _check_xattrs

2023-11-23 Thread Gao Xiang
Hu Thanks, folded this with your SoB. Thanks, Gao Xiang

Re: [PATCH v2] erofs-utils: mkfs: generate on-disk indexes after compression

2023-11-23 Thread Gao Xiang
indexes after the compression for the file is completed. Signed-off-by: Yifan Zhao --- Sorry for the late reply. There are some issues on my side. In order to make this work efficient. Let me to refine this commit as well. Thanks, Gao Xiang

[PATCH 1/2] erofs-utils: lib: fix up compact indexes for block size < 4096

2023-11-22 Thread Gao Xiang
, there is no enough room for the special value `Z_EROFS_LI_D0_CBLKCNT`. To support smaller block sizes, `lobits` for each compacted lcluster is now calculated as: lobits = max(lclusterbits, ilog2(Z_EROFS_LI_D0_CBLKCNT) + 1) Signed-off-by: Gao Xiang --- lib/zmap.c | 32 ++-- 1 file

[PATCH 2/2] erofs-utils: mkfs: support compact indexes for smaller block sizes

2023-11-22 Thread Gao Xiang
This commit also adds mkfs support of compact indexes for smaller block sizes (less than 4096). Signed-off-by: Gao Xiang --- lib/compress.c | 15 --- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index 47f1c1d..61328ed 100644 --- a/lib

[GIT PULL] erofs fixes for 6.7-rc3

2023-11-19 Thread Gao Xiang
s have been in -next and no potential merge conflict is observed. Thanks, Gao Xiang The following changes since commit b85ea95d086471afb4ad062012a4d73cd328fa86: Linux 6.7-rc1 (2023-11-12 16:19:07 -0800) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git

Re: [PATCH] MAINTAINERS: erofs: add EROFS webpage

2023-11-19 Thread Gao Xiang
On 2023/11/20 11:31, Chao Yu wrote: On 2023/11/20 11:23, Gao Xiang wrote: On 2023/11/20 11:18, Chao Yu wrote: On 2023/11/17 16:53, Gao Xiang wrote: Add a new `W:` field of the EROFS entry points to the documentation site at <https://erofs.docs.kernel.org>. In addition,

Re: [PATCH] MAINTAINERS: erofs: add EROFS webpage

2023-11-19 Thread Gao Xiang
On 2023/11/20 11:18, Chao Yu wrote: On 2023/11/17 16:53, Gao Xiang wrote: Add a new `W:` field of the EROFS entry points to the documentation site at <https://erofs.docs.kernel.org>. In addition, update the in-tree documentation and Kconfig too. Signed-off-by: Gao Xiang Nic

[PATCH] MAINTAINERS: erofs: add EROFS webpage

2023-11-17 Thread Gao Xiang
Add a new `W:` field of the EROFS entry points to the documentation site at <https://erofs.docs.kernel.org>. In addition, update the in-tree documentation and Kconfig too. Signed-off-by: Gao Xiang --- Documentation/filesystems/erofs.rst | 4 MAINTAINERS | 1

[PATCH] erofs-utils: lib: `fragment_size` should be 64 bits

2023-11-14 Thread Gao Xiang
`-Eall-fragments` is broken if i_size is more than 32 bits. Fixes: fcaa988a6ef6 ("erofs-utils: add `-Eall-fragments` option") Signed-off-by: Gao Xiang --- include/erofs/internal.h | 12 lib/compress.c | 9 + 2 files changed, 13 insertions(+), 8 deletion

Re: [PATCH] erofs: fix NULL dereference of dif->bdev_handle in fscache mode

2023-11-13 Thread Gao Xiang
2/0xf0 entry_SYSCALL_64_after_hwframe+0x6e/0x76 Reported-by: Yiqun Leng Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=7245 Fixes: 49845720080d ("erofs: Convert to use bdev_open_by_path()") Signed-off-by: Jingbo Xu Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH 2/3] mm: Add folio_fill_tail() and use it in iomap

2023-11-10 Thread Gao Xiang
his up. Assuming that is for the next cycle (no rush), I will also test this patch and feedback later since I'm now working on other stuffs. Thanks, Gao Xiang

Re: [PATCH v2] erofs: simplify erofs_read_inode()

2023-11-09 Thread Gao Xiang
y: Ferry Meng Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH -next V2] erofs: code clean up for function erofs_read_inode()

2023-11-09 Thread Gao Xiang
On 2023/11/9 21:45, Zizhi Wo wrote: 在 2023/11/9 21:14, Gao Xiang 写道: Hi, On 2023/11/10 03:48, WoZ1zh1 wrote: Because variables "die" and "copied" only appear in case EROFS_INODE_LAYOUT_EXTENDED, move them from the outer space into this case. Also, call "kfree(

Re: [PATCH -next V2] erofs: code clean up for function erofs_read_inode()

2023-11-09 Thread Gao Xiang
the code 2) We'd like to avoid defining variables like this (in the switch block), and I even don't think this patch can compile. 3) The logic itself is also broken... Thanks, Gao Xiang + vi->inode_isize = sizeof(struct erofs_inode_extended); /* check if

Re: [PATCH] erofs: simplify erofs_read_inode()

2023-11-08 Thread Gao Xiang
UPP; goto err_out; } - if (vi->datalayout == EROFS_INODE_CHUNK_BASED) { + switch (inode->i_mode & S_IFMT) { + case S_IFREG: + case S_IFDIR: + case S_IFLNK: + vi->raw_blkaddr = le32_to_cpu(iu->raw_blkaddr); may cause use-after-free for acrossing block boundary case due to kfree(copied) before? Yeah, good catch, I think we could just make a copy instead. Thanks, Gao Xiang

[PATCH v2] erofs-utils: mkfs,fsck,dump: support `--offset` option

2023-11-06 Thread Gao Xiang
Add `--offset` option to allows users to specify an offset in the file where the filesystem will begin. Suggested-by: Pavel Otchertsov Closes: https://lore.kernel.org/r/caaxntogtd2nkknbphz+ver7nvnwvt0u02e+c8pn8zvfcxp5...@mail.gmail.com Signed-off-by: Gao Xiang --- I plan to apply this version

[PATCH] erofs-utils: mkfs,fsck,dump: support `--offset` option

2023-11-06 Thread Gao Xiang
Add `--offset` option to allows users to specify an offset in the file where the filesystem will begin. Suggested-by: Pavel Otchertsov Closes: https://lore.kernel.org/r/caaxntogtd2nkknbphz+ver7nvnwvt0u02e+c8pn8zvfcxp5...@mail.gmail.com Signed-off-by: Gao Xiang --- Hi Pavel, Could you check

Re: Feature Request: Add --offset option to mkfs.erofs

2023-11-06 Thread Gao Xiang
mation or assist in any way. Thanks for the reference! Let's implement this feature. Thanks, Gao Xiang Best regards, Pavel Otchertsov mailto:pavel.otchert...@gmail.com>>

Re: [PATCH 0/3] erofs-utils: Address doc and flag-parsing snags

2023-11-05 Thread Gao Xiang
. Thanks, applied for testing. Thanks, Gao Xiang Luke T. Shumaker (3): erofs-utils: have each non-fuse command take -h, --help, -V, and --version erofs-utils: improve the usage and version text of non-fuse commands erofs-utils: fsck: Add -a, -A, and -y flags dump/main.c

Re: [PATCH] erofs-utils: mkfs: fix potential memory leak

2023-11-05 Thread Gao Xiang
() and realpath(). Signed-off-by: Yifan Zhao Reviewed-by: Gao Xiang BTW, would you mind adding a configuration to enable Valgrind? Thanks, Gao Xiang

[GIT PULL] erofs updates for 6.7-rc1

2023-10-31 Thread Gao Xiang
, and there are minor cleanups around documentation and codebase. In addition, it also fixes incorrect lockref usage in erofs_insert_workgroup(). All commits have been in -next and no potential merge conflict is observed. Thanks, Gao Xiang The following changes since commit

Re: [PATCH] erofs: fix erofs_insert_workgroup() lockref usage

2023-10-31 Thread Gao Xiang
On 2023/10/31 14:20, Linus Torvalds wrote: On Mon, 30 Oct 2023 at 20:08, Gao Xiang wrote: As Linus pointed out [1], lockref_put_return() is fundamentally designed to be something that can fail. It behaves as a fastpath-only thing, and the failure case needs to be handled anyway. Actually

[PATCH] erofs: fix erofs_insert_workgroup() lockref usage

2023-10-31 Thread Gao Xiang
place handcrafted approach") Signed-off-by: Gao Xiang --- fs/erofs/utils.c | 8 +--- fs/erofs/zdata.c | 1 + 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/fs/erofs/utils.c b/fs/erofs/utils.c index cc6fb9e98899..4256a85719a1 100644 --- a/fs/erofs/utils.c +++ b/fs/ero

Re: [PATCH] erofs-utils: lib: tidy up erofs_compress_destsize()

2023-10-29 Thread Gao Xiang
On 2023/10/27 15:06, Gao Xiang wrote: Drop the old workaround logic to prepare for the following development. (I've checked the Linux 6.1.53 source code and an AOSP rootfs without any image size change and strange behavior.) Signed-off-by: Gao Xiang I found that It could cause

[PATCH] erofs-utils: lib: tidy up erofs_compress_destsize()

2023-10-27 Thread Gao Xiang
Drop the old workaround logic to prepare for the following development. (I've checked the Linux 6.1.53 source code and an AOSP rootfs without any image size change and strange behavior.) Signed-off-by: Gao Xiang --- lib/compress.c | 71 +++- lib

Re: [PATCH 2/2] erofs: tidy up redundant includes

2023-10-25 Thread Gao Xiang
On 2023/10/26 10:16, Ferry Meng wrote: - Remove unused includes like and ; - Move common includes into "internal.h". Signed-off-by: Ferry Meng Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH 1/2] erofs: get rid of ROOT_NID()

2023-10-25 Thread Gao Xiang
On 2023/10/26 10:16, Ferry Meng wrote: Let's open code this helper for simplicity. Signed-off-by: Ferry Meng Reviewed-by: Gao Xiang Thanks, Gao Xiang

[PATCH v2 3/3] erofs-utils: lib: propagate return code for erofs_bflush()

2023-10-24 Thread Gao Xiang
Instead of just using a boolean. Signed-off-by: Gao Xiang --- include/erofs/cache.h | 8 lib/cache.c | 21 ++--- lib/compress.c| 1 + lib/inode.c | 18 +- mkfs/main.c | 14 +++--- 5 files changed, 35

[PATCH 2/3] erofs-utils: get rid of .preflush()

2023-10-23 Thread Gao Xiang
It's actually never used. Signed-off-by: Gao Xiang --- include/erofs/cache.h | 1 - lib/cache.c | 5 - 2 files changed, 6 deletions(-) diff --git a/include/erofs/cache.h b/include/erofs/cache.h index de5584e..31f54a4 100644 --- a/include/erofs/cache.h +++ b/include/erofs/cache.h

[PATCH 3/3] erofs-utils: lib: propagate return code for erofs_bflush()

2023-10-23 Thread Gao Xiang
Instead of just using a boolean. Signed-off-by: Gao Xiang --- include/erofs/cache.h | 8 lib/cache.c | 21 ++--- lib/inode.c | 18 +- mkfs/main.c | 14 +++--- 4 files changed, 34 insertions(+), 27 deletions(-) diff

[PATCH 1/3] erofs-utils: lib: use BLK_ROUND_UP() for __erofs_battach()

2023-10-23 Thread Gao Xiang
Also avoid division in BLK_ROUND_UP(). Signed-off-by: Gao Xiang --- include/erofs/internal.h | 3 ++- lib/cache.c | 15 --- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/include/erofs/internal.h b/include/erofs/internal.h index c1ff582..4d794ae 100644

[PATCH v2] erofs: simplify compression configuration parser

2023-10-22 Thread Gao Xiang
From: Gao Xiang Move erofs_load_compr_cfgs() into decompressor.c as well as introduce a callback instead of a hard-coded switch for each algorithm for simplicity. Signed-off-by: Gao Xiang --- v2: - fix build warning of `-Wmissing-declarations` https://lore.kernel.org/r

[PATCH] erofs: don't warn MicroLZMA format anymore

2023-10-20 Thread Gao Xiang
The LZMA algorithm support has been landed for more than one year since Linux 5.16. Besides, the new XZ Utils 5.4 has been available in most Linux distributions. Signed-off-by: Gao Xiang --- fs/erofs/Kconfig | 7 ++- fs/erofs/decompressor_lzma.c | 2 -- 2 files changed, 2

[PATCH] erofs: simplify compression configuration parser

2023-10-20 Thread Gao Xiang
Move erofs_load_compr_cfgs() into decompressor.c as well as introduce a callback instead of a hard-coded switch for each algorithm for simplicity. Signed-off-by: Gao Xiang --- fs/erofs/compress.h | 6 +++ fs/erofs/decompressor.c | 60 ++- fs/erofs

[PATCH] erofs-utils: fix reference leak in erofs_mkfs_build_tree_from_path()

2023-10-19 Thread Gao Xiang
commit 8cbc205185a1 ("erofs-utils: mkfs: fix corrupted directories with hardlinks") introduced a reference leak although it has no real impact to users. Fix it now. Signed-off-by: Gao Xiang --- lib/inode.c | 13 + 1 file changed, 9 insertions(+), 4 deletions(-) diff -

[PATCH] erofs-utils: mkfs: fix corrupted directories with hardlinks

2023-10-17 Thread Gao Xiang
adata") Signed-off-by: Gao Xiang --- include/erofs/internal.h | 5 +++-- lib/inode.c | 29 - 2 files changed, 15 insertions(+), 19 deletions(-) diff --git a/include/erofs/internal.h b/include/erofs/internal.h index d859905..c1ff582 100644 --- a/inc

Re: [PATCH] erofs: fix inode metadata space layout description in documentation

2023-10-11 Thread Gao Xiang
Hi Chao, On 2023/10/12 09:10, Chao Yu wrote: On 2023/10/10 21:06, Gao Xiang wrote: Signed-off-by: Tiwei Bie Reviewed-by: Gao Xiang Looks fine to me for the version in dev-test branch. Reviewed-by: Chao Yu Yeah, thanks! I will add the tag when applying to -next. Thanks, Gao Xiang

Re: [PATCH] erofs: fix inode metadata space layout description in documentation

2023-10-10 Thread Gao Xiang
ument") I'm not sure if it's necessary to tag document fixes anyway since docs.kernel.org already uses the latest version and `.rst` format is adapted much later.. I will drop this tag for the next merge window if not urgent. Signed-off-by: Tiwei Bie Reviewed-by: Gao Xiang Thanks,

Re: [PATCH v10] erofs-utils: add support for fuse 2/3 lowlevel API

2023-10-09 Thread Gao Xiang
_VERSION >= 3 + fputs("\nFUSE options:\n", stderr); fuse_cmdline_help(); #else + struct fuse_args args = FUSE_ARGS_INIT(0, NULL); It's not a good idea to mix definitions and the rest code. I will fix manually. Thanks, Gao Xiang

Re: errno is set to a negative value in lib/tar.c

2023-10-07 Thread Gao Xiang
On 2023/10/6 02:43, Gao Xiang wrote: Hi Erik, On Mon, Oct 02, 2023 at 07:36:08PM +0200, Erik Sjölund wrote: Hi, Does this patch make sense? (I thought errno should be set to a non-negative value) Best regards, Erik Sjölund Thanks for the patch. I'm on vacation, sorry for late reply

Re: [PATCH v1] erofs-utils: Fix cross compile with autoconf

2023-10-07 Thread Gao Xiang
On 2023/10/7 15:16, Sandeep Dhavale wrote: On Fri, Oct 6, 2023 at 8:27 PM Gao Xiang wrote: Hi Sandeep! On 2023/10/6 06:40, Sandeep Dhavale wrote: AC_RUN_IFELSE expects the action if cross compiling. If not provided cross compilation fails with error "configure: error: cannot run

Re: [PATCH 08/29] erofs: move erofs_xattr_handlers and xattr_handler_map to .rodata

2023-10-06 Thread Gao Xiang
On 2023/9/30 13:00, Wedson Almeida Filho wrote: From: Wedson Almeida Filho This makes it harder for accidental or malicious changes to erofs_xattr_handlers or xattr_handler_map at runtime. Cc: Gao Xiang Cc: Chao Yu Cc: Yue Hu Cc: Jeffle Xu Cc: linux-erofs@lists.ozlabs.org Signed-off

Re: [PATCH v1] erofs-utils: Fix cross compile with autoconf

2023-10-06 Thread Gao Xiang
_block_size=$MAX_BLOCK_SIZE]) Actually the following check will reset erofs_cv_max_block_size to 4096 if needed. But it seems that it has syntax errors. I wonder if the following diff could fix the issue too? Thanks, Gao Xiang diff --git a/configure.ac b/configure.ac index 13ee616..94eec01 1006

[GIT PULL] erofs fixes for 6.6-rc5

2023-10-05 Thread Gao Xiang
for a while and no potential merge conflict is observed. Thanks, Gao Xiang The following changes since commit 0bb80ecc33a8fb5a682236443c1e740d5c917d1d: Linux 6.6-rc1 (2023-09-10 16:28:41 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git

Re: errno is set to a negative value in lib/tar.c

2023-10-05 Thread Gao Xiang
I will address it when I'm back. Thanks, Gao Xiang > > diff --git a/lib/tar.c b/lib/tar.c > index 0744972..8204939 100644 > --- a/lib/tar.c > +++ b/lib/tar.c > @@ -241,7 +241,7 @@ static long long tarerofs_otoi(const char *ptr, int len) > val = strtol(ptr, , 8)

Re: [PATCH v2] erofs: update documentation

2023-09-28 Thread Gao Xiang
On 2023/9/28 21:48, Jingbo Xu wrote: - update new features like bloom filter and DEFLATE. - add documentation for the long xattr name prefixes, which was landed upstream since v6.4. Signed-off-by: Jingbo Xu Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH] erofs: update documentation

2023-09-28 Thread Gao Xiang
efix is stored in the format: ALIGN({__le16 len, data}, 4), > +where len represents the total size of the data part. The data part is > actually > +represented by 'struct erofs_xattr_long_prefix', where base_index represents > the > +index of the predefined xattr name prefix, e.g. EROFS_XATTR_INDEX_TRUSTED for > +"trusted.overlay." long name prefix, while the infix string kepts the string ^ keeps Thanks, Gao Xiang

[PATCH 3/3] erofs-utils: lib: drop prefix_sha256 digests

2023-09-22 Thread Gao Xiang
People care more about speed when building. Later, we could also try to use file-backed memory to avoid memory pressure. As a result, it could decrease time by 50% (0m58.255s -> 0m29.120s). Signed-off-by: Gao Xiang --- lib/dedupe.c | 15 +-- 1 file changed, 5 insertions(+),

[PATCH 1/3] erofs-utils: lib: use xxh64() for faster filtering first for dedupe

2023-09-22 Thread Gao Xiang
Let's check if xxh64 equals when rolling back on global compressed deduplication. As a result, it could decrease time by 26.4% (6m57.990s -> 5m7.755s) on a dataset with "-Ededupe -C8192". Signed-off-by: Gao Xiang --- include/erofs/defs.h | 5 +++ include/erofs/

[PATCH 2/3] erofs-utils: switch dedupe_{sub,}tree[] to a hash table

2023-09-22 Thread Gao Xiang
This rb-tree implementation is too slow and there is no benefits of it. As a result, it could decrease time by 81.1% (5m7.755s -> 0m58.255s) with the same dataset, sigh. Signed-off-by: Gao Xiang --- lib/Makefile.am | 2 +- lib/dedupe.c| 116 +-- lib/rb_tree.c |

[ANNOUNCE] erofs-utils: release 1.7

2023-09-20 Thread Gao Xiang
rnel jobs).  It's always worth trying, and any contribution is much appreciated. Thanks, Gao Xiang [1] https://lore.kernel.org/r/20230918090306.2524624-1-lyy0...@sjtu.edu.cn

[PATCH] erofs-utils: fix the previous pcluster CBLKCNT missing for big pcluster dedupe

2023-09-20 Thread Gao Xiang
ed. Fixes: f3f9a2ce3137 ("erofs-utils: mkfs: introduce global compressed data deduplication") Signed-off-by: Gao Xiang --- lib/compress.c | 25 ++--- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index 81f277a..f6dc12a 100644

[PATCH] erofs-utils: fix build error when `-Waddress-of-temporary` is on

2023-09-20 Thread Gao Xiang via Linux-erofs
From: Gao Xiang Actually, it's false positive and only used for build assertion. Reported-by: Kelvin Zhang Signed-off-by: Gao Xiang --- include/erofs_fs.h | 14 -- 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/include/erofs_fs.h b/include/erofs_fs.h index

[PATCH RESEND] erofs-utils: mkfs: limit total shared xattrs of a single inode

2023-09-20 Thread Gao Xiang
Don't output more than 255 shared xattrs for a single inode due to the EROFS on-disk format limitation. Fixes: 116ac0a254fc ("erofs-utils: introduce shared xattr support") Signed-off-by: Gao Xiang --- lib/xattr.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff -

[PATCH] erofs-utils: mkfs: limit total shared xattrs of a single inode

2023-09-20 Thread Gao Xiang
Don't output more than 255 shared xattrs for a single inode due to the EROFS on-disk format limitation. Fixes: 116ac0a254fc ("erofs-utils: introduce shared xattr support") Signed-off-by: Gao Xiang --- lib/xattr.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff -

[PATCH] erofs-utils: manpages: update new options of mkfs.erofs

2023-09-20 Thread Gao Xiang
-b block-size -E ^xattr-name-filter --gzip --tar=[fi] --xattr-prefix=X Signed-off-by: Gao Xiang --- man/mkfs.erofs.1 | 23 +++ 1 file changed, 23 insertions(+) diff --git a/man/mkfs.erofs.1 b/man/mkfs.erofs.1 index 1cfde28..5dd718d 100644 --- a/man/mkfs.erofs.1 +++ b

Re: [PATCH] erofs-utils: mkfs: support tgz streams for tarerofs

2023-09-20 Thread Gao Xiang
On Wed, Sep 20, 2023 at 02:59:47AM +0800, Gao Xiang wrote: > Introduce iostream to wrap up the input tarball stream for tarerofs. > > Besides, add bultin tgz support if zlib is linked to mkfs. > > Signed-off-by: Gao Xiang Apply the following diff to fix pax header regression: d

Re: Optimizing write_uncompressed_file_from_fd()

2023-09-19 Thread Gao Xiang
Hi Daan, On 2023/8/31 17:48, Gao Xiang wrote: Hi Daan, On 2023/8/31 17:33, Daan De Meyer wrote: Hi, For hacking on systemd, we build disk images using mkosi, and use an erofs filesystem for the /usr directory. When hacking on systemd, we would like to be able to rebuild the disk image

[PATCH] erofs-utils: lib: fix --force-{g,u}id support for tarerofs

2023-09-19 Thread Gao Xiang
Temporarily move the common part into __erofs_fill_inode() for tarerofs. Signed-off-by: Gao Xiang --- include/erofs/inode.h | 2 ++ lib/inode.c | 19 --- lib/tar.c | 16 ++-- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git

[PATCH] erofs-utils: mkfs: support exporting GNU tar archive labels

2023-09-19 Thread Gao Xiang
GNU tar volume labels (by using `-V`) will be applied to EROFS. Signed-off-by: Gao Xiang --- lib/tar.c | 24 ++-- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/lib/tar.c b/lib/tar.c index 8e71f11..1cd820f 100644 --- a/lib/tar.c +++ b/lib/tar.c @@ -17,6 +17,9

[PATCH v2] erofs-utils: lib: introduce diskbuf

2023-09-19 Thread Gao Xiang
From: Gao Xiang Previously, each tar data file will be kept as a temporary file before landing to the target image since the input stream may be non-seekable. It's somewhat ineffective. Let's introduce a new diskbuf approach to manage those buffers. Laterly, each stream can be redirected

Re: [PATCH] erofs-utils: lib: introduce diskbuf

2023-09-19 Thread Gao Xiang
On 2023/9/20 11:35, Gao Xiang wrote: On 2023/9/20 11:32, Jingbo Xu wrote: On 9/15/23 11:29 PM, Gao Xiang wrote: Previously, each tar data file will be kept as a temporary file before landing to the target image since the input stream may be non-seekable. It's somewhat ineffective

Re: [PATCH] erofs-utils: lib: introduce diskbuf

2023-09-19 Thread Gao Xiang
On 2023/9/20 11:32, Jingbo Xu wrote: On 9/15/23 11:29 PM, Gao Xiang wrote: Previously, each tar data file will be kept as a temporary file before landing to the target image since the input stream may be non-seekable. It's somewhat ineffective. Let's introduce a new diskbuf approach

Re: [PATCH v1] erofs-utils: lib: Restore memory address before free()

2023-09-19 Thread Gao Xiang
ply like this directly. Thanks, Gao Xiang inode->chunkindexes = NULL; return ret;

[PATCH] erofs-utils: mkfs: support tgz streams for tarerofs

2023-09-19 Thread Gao Xiang
Introduce iostream to wrap up the input tarball stream for tarerofs. Besides, add bultin tgz support if zlib is linked to mkfs. Signed-off-by: Gao Xiang --- include/erofs/tar.h | 21 +++ lib/tar.c | 309 mkfs/main.c | 34 - 3

Re: [PATCH v9] erofs-utils: add support for fuse 2/3 lowlevel API

2023-09-18 Thread Gao Xiang
to apply it after erofs-utils v1.7 is released. Thanks, Gao Xiang

Re: [PATCH v9] erofs-utils: add support for fuse 2/3 lowlevel API

2023-09-16 Thread Gao Xiang
: Structure needs cleaning [xize.gx@e69b19392.et15sqa /home/xize.gx/erofs-utils] $fssum -MACUG mnt stat failed for mnt//Documentation/.gitignore: Structure needs cleaning [xize.gx@e69b19392.et15sqa /home/xize.gx/erofs-utils] $fssum -MACUG mnt stat failed for mnt//LICENSES/deprecated: Structure needs cleaning [xize.gx@e69b19392.et15sqa /home/xize.gx/erofs-utils] $fssum -MACUG mnt stat failed for mnt//drivers/staging/most: Structure needs cleaning [xize.gx@e69b19392.et15sqa /home/xize.gx/erofs-utils] $fssum -MACUG mnt stat failed for mnt//drivers/staging/most: Structure needs cleaning [xize.gx@e69b19392.et15sqa /home/xize.gx/erofs-utils] $fssum -MACUG mnt stat failed for mnt//tools/testing/selftests/drivers/net/bonding/settings: Structure needs cleaning I don't think it's a correct sign. I try to debug more but it seems more issues here, I gave up eveuatually. Thanks, Gao Xiang

[PATCH] erofs-utils: lib: introduce diskbuf

2023-09-15 Thread Gao Xiang
for external reference. Signed-off-by: Gao Xiang --- include/erofs/diskbuf.h | 30 include/erofs/internal.h | 8 ++- lib/Makefile.am | 3 +- lib/diskbuf.c| 147 +++ lib/inode.c | 33 + lib/io.c

Re: [PATCH v2] erofs-utils: mkfs: support flatdev for multi-blob images

2023-09-15 Thread Gao Xiang
vs[src->dev - 1].blocks = nblocks; + idx = src->dev - 1; + sbi.devs[idx].blocks = nblocks; + memcpy(sbi.devs[idx].tag, src->uuid, sizeof(src->uuid)); Should we use string conversion instead? Should we use uuid only if sbi.devs[idx].tag is empty? Thanks, Gao Xiang

Re: [RESEND PATCH] erofs: relax the constraint of non-empty device tag in flatdev mode

2023-09-15 Thread Gao Xiang
actually required in flatdev mode, thus fix mount failure due to empty device tags in flatdev mode. Fixes: 8b465fecc35a ("erofs: support flattened block device for multi-blob images") Reviewed-by: Gao Xiang Thanks, Gao Xiang

[PATCH] erofs-utils: mkfs: pop up most recently used dentries for tarerofs

2023-09-15 Thread Gao Xiang
Each tar header keeps the full file path. It's useful to move most recently used intermediate dirs to list heads. User time of tarerofs index mode can be reduced by 19%. Signed-off-by: Gao Xiang --- include/erofs/rebuild.h | 2 +- lib/rebuild.c | 9 +++-- lib/tar.c

<    1   2   3   4   5   6   7   8   9   10   >