Re: [PATCH v2 4/5] cachefiles: cyclic allocation of msg_id to avoid reuse

2024-05-20 Thread Gao Xiang
Hi Baokun, On 2024/5/20 21:24, Baokun Li wrote: On 2024/5/20 20:54, Gao Xiang wrote: On 2024/5/20 20:42, Baokun Li wrote: On 2024/5/20 18:04, Jeff Layton wrote: On Mon, 2024-05-20 at 12:06 +0800, Baokun Li wrote: Hi Jeff, Thank you very much for your review! On 2024/5/19 19:11, Jeff

Re: [PATCH v2 4/5] cachefiles: cyclic allocation of msg_id to avoid reuse

2024-05-20 Thread Gao Xiang
ill the mount instance if there is a malicious user daemon. So in that case, hung task will not be triggered anymore, and you don't need to care about cyclic allocation too. Thanks, Gao Xiang Regards, Baokun +    xas_clear_mark(, XA_FREE_MARK); +    xas_set_mark(, CACHEFILES_REQ_NEW)

Re: [PATCH v2 08/12] cachefiles: never get a new anonymous fd if ondemand_id is valid

2024-05-20 Thread Gao Xiang
u'd better to switch wait_for_completion() to wait_for_completion_killable() at least IMHO anyway. Thanks, Gao Xiang

[PATCH] erofs: avoid allocating DEFLATE streams before mounting

2024-05-20 Thread Gao Xiang
. Fixes: ffa09b3bd024 ("erofs: DEFLATE compression support") Cc: # 6.6+ Signed-off-by: Gao Xiang --- fs/erofs/decompressor_deflate.c | 55 + 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/fs/erofs/decompressor_deflate.c b/fs/erofs/decompressor

Re: [PATCH v2 03/12] cachefiles: fix slab-use-after-free in cachefiles_ondemand_get_fd()

2024-05-20 Thread Gao Xiang
later patches modify the logic here as well. Or add Fixes tag for patch 2? I think we might better to avoid unnecessary dependencies since it relies on some "AI" magic and often mis-backportes real dependencies. I tend to leave real bugfixes first, and do cleanup next. But please don't leave cleanup patches with "Fixes:" tags anyway since it just misleads people. Thanks, Gao Xiang

[PATCH v2] erofs-utils: unify the tree traversal for the rebuild mode

2024-05-20 Thread Gao Xiang
Let's drop the legacy approach and `tarerofs` will be applied too. Signed-off-by: Gao Xiang --- change since v1: - fix incorrect unencoded data reported by: https://github.com/erofs/erofsnightly/actions/runs/9153196829 include/erofs/internal.h | 7 +- lib/diskbuf.c| 14

Re: [PATCH v2 01/12] cachefiles: remove request from xarry during flush requests

2024-05-19 Thread Gao Xiang
avoid accessing freed requests. Fixes: c8383054506c ("cachefiles: notify the user daemon when looking up cookie") Signed-off-by: Baokun Li Reviewed-by: Jia Zhu Reviewed-by: Gao Xiang Thanks, Gao Xiang --- fs/cachefiles/daemon.c | 1 + 1 file changed, 1 insertion(+) diff --git

Re: [PATCH v2 02/12] cachefiles: remove err_put_fd tag in cachefiles_ondemand_daemon_read()

2024-05-19 Thread Gao Xiang
ed-by: Jia Zhu Reviewed-by: Gao Xiang Thanks, Gao Xiang --- fs/cachefiles/ondemand.c | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/fs/cachefiles/ondemand.c b/fs/cachefiles/ondemand.c index 4ba42f1fa3b4..fd49728d8bae 100644 --- a/fs/cachefiles/ondemand.c +++ b/fs/

Re: [PATCH] erofs: clean up erofs_show_options()

2024-05-17 Thread Gao Xiang
On 2024/5/17 17:56, Hongzhen Luo wrote: Avoid unnecessary #ifdefs and simplify the code a bit. Signed-off-by: Hongzhen Luo Reviewed-by: Gao Xiang Thanks, Gao Xiang

[PATCH] erofs-utils: mkfs: add `--zfeature-bits` option

2024-05-17 Thread Gao Xiang
Thus, we could traverse all compression features with a number easily in the testcases. Signed-off-by: Gao Xiang --- mkfs/main.c | 160 1 file changed, 124 insertions(+), 36 deletions(-) diff --git a/mkfs/main.c b/mkfs/main.c index c26cb56

[PATCH 2/2] erofs-utils: pretty root directory progressinfo

2024-05-15 Thread Gao Xiang
Avoid `Processing ...` or `file dumped (mode 40755)`.. Signed-off-by: Gao Xiang --- lib/inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/inode.c b/lib/inode.c index 8ec87e6..67a572d 100644 --- a/lib/inode.c +++ b/lib/inode.c @@ -1405,10 +1405,11 @@ static

[PATCH 1/2] erofs-utils: correct the default number of workers in the usage

2024-05-15 Thread Gao Xiang
Fixes: 59c36e7a4008 ("erofs-utils: mkfs: use all available processors by default") Signed-off-by: Gao Xiang --- mkfs/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mkfs/main.c b/mkfs/main.c index 4c3620d..455c152 100644 --- a/mkfs/main.c +++ b/m

[PATCH] erofs-utils: unify the tree traversal for the rebuild mode

2024-05-15 Thread Gao Xiang
Let's drop the legacy approach and `tarerofs` will be applied too. Signed-off-by: Gao Xiang --- include/erofs/internal.h | 7 +- lib/diskbuf.c| 14 +- lib/inode.c | 384 ++- 3 files changed, 232 insertions(+), 173 deletions

[PATCH v3] erofs-utils: add preliminary zstd support [x]

2024-05-14 Thread Gao Xiang
it will take more efforts. [ TODO: Later I tend to add another way to generate fixed-sized input pclusters temporarily for relatively large pcluster sizes as an option since it will have minor impacts to the results. ] Signed-off-by: Gao Xiang --- changes since v2: - use

[GIT PULL] erofs updates for 6.10-rc1

2024-05-13 Thread Gao Xiang
merge conflict is observed. Thanks, Gao Xiang The following changes since commit dd5a440a31fae6e459c0d627162825505361: Linux 6.9-rc7 (2024-05-05 14:06:01 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git tags/erofs-for-6.10-rc1

[PATCH v3] erofs-utils: add preliminary zstd support [x]

2024-05-09 Thread Gao Xiang
it will take more efforts. [ TODO: Later I tend to add another way to generate fixed-sized input pclusters temporarily for relatively large pcluster sizes as an option since it will have minor impacts to the results. ] Signed-off-by: Gao Xiang --- v3: - Fix incorrect assumption

Re: [PATCH] erofs-utils: add preliminary zstd support

2024-05-09 Thread Gao Xiang
I will look into that, thanks! Thanks, Gao Xiang

[PATCH v2] erofs-utils: add preliminary zstd support [x]

2024-05-08 Thread Gao Xiang
it will take more efforts. [ TODO: Later I tend to add another way to generate fixed-sized input pclusters temporarily for relatively large pcluster sizes as an option since it will have minor impacts to the results. ] Signed-off-by: Gao Xiang --- v2: Use a newer helper

[PATCH v2] erofs: Zstandard compression support

2024-05-08 Thread Gao Xiang
From: Gao Xiang Add Zstandard compression as the 4th supported algorithm since it becomes more popular now and some end users have asked this for quite a while [1][2]. Each EROFS physical cluster contains only one valid standard Zstandard frame as described in [3] so that decompression can

[PATCH] erofs: clean up z_erofs_load_full_lcluster()

2024-05-08 Thread Gao Xiang
Only four lcluster types here, remove redundant code. No real logic changes. Signed-off-by: Gao Xiang --- Some random cleanup out of the upcoming big lclusters.. fs/erofs/erofs_fs.h | 5 + fs/erofs/zmap.c | 21 + 2 files changed, 6 insertions(+), 20 deletions

[PATCH] erofs-utils: add preliminary zstd support

2024-05-08 Thread Gao Xiang
it will take more efforts. [ TODO: Later I tend to add another way to generate fixed-sized input pclusters temporarily for relatively large pcluster sizes as an option since it will have minor impacts to the results. ] Signed-off-by: Gao Xiang --- configure.ac | 34

[PATCH] erofs: Zstandard compression support

2024-05-08 Thread Gao Xiang
://lore.kernel.org/r/y08h+z6czdns1...@b-p7tqmd6m-0146.lan [3] https://www.rfc-editor.org/rfc/rfc8478.txt Signed-off-by: Gao Xiang --- fs/erofs/Kconfig | 15 ++ fs/erofs/Makefile| 1 + fs/erofs/compress.h | 4 + fs/erofs/decompressor.c | 7 + fs/erofs

Re: [PATCH AUTOSEL 6.8 26/52] erofs: reliably distinguish block based and fscache mode

2024-05-07 Thread Gao Xiang
[...] Now when erofs_kill_sb() is called, erofs_sb_info must have been initialised, so use sbi->fsid to distinguish between the two modes. Signed-off-by: Christian Brauner Signed-off-by: Baokun Li Reviewed-by: Jingbo Xu Reviewed-by: Gao Xiang Reviewed-by: Chao Yu Link: https://lore.kernel.or

Re: [PATCH AUTOSEL 6.6 21/43] erofs: reliably distinguish block based and fscache mode

2024-05-07 Thread Gao Xiang
[...] Now when erofs_kill_sb() is called, erofs_sb_info must have been initialised, so use sbi->fsid to distinguish between the two modes. Signed-off-by: Christian Brauner Signed-off-by: Baokun Li Reviewed-by: Jingbo Xu Reviewed-by: Gao Xiang Reviewed-by: Chao Yu Link: https://lore.kernel.or

[PATCH v2 1/2] erofs-utils: record pclustersize in bytes instead of blocks

2024-04-30 Thread Gao Xiang
So that we don't need to handle blocksizes everywhere. Signed-off-by: Gao Xiang --- v1: fix CI failures: https://github.com/erofs/erofsnightly/actions/runs/8896341493/job/24428809079 include/erofs/config.h | 4 +++- lib/compress.c | 38

Re: [PATCH] erofs-utils: optimize pthread_cond_signal calling

2024-04-30 Thread Gao Xiang
On 2024/5/1 10:24, Noboru Asai wrote: Call pthread_cond_signal once per file. Signed-off-by: Noboru Asai Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH] erofs-utils: simplify file handling

2024-04-30 Thread Gao Xiang
ncoded_file(inode, ctx->fd, 0); -out: - close(ctx->fd); + fd = open(inode->i_srcpath, O_RDONLY | O_BINARY); At a quick glance, here we need to open i_srcpath again, I tend to avoid it so I use dup instead. Thanks, Gao Xiang

[PATCH 1/2] erofs-utils: record pclustersize in bytes instead of blocks

2024-04-30 Thread Gao Xiang
So that we don't need to handle blocksizes everywhere. Signed-off-by: Gao Xiang --- include/erofs/config.h | 4 +++- lib/compress.c | 30 -- lib/compress_hints.c | 11 ++- lib/config.c | 2 -- mkfs/main.c| 16

[PATCH 2/2] erofs-utils: lib: adjust MicroLZMA default dictionary size

2024-04-30 Thread Gao Xiang
If dict_size is not given, it will be set as max(32k, pclustersize * 8) but no more than Z_EROFS_LZMA_MAX_DICT_SIZE. Also kill an obsolete warning since multi-threaded support is landed. Signed-off-by: Gao Xiang --- lib/compressor_liblzma.c | 19 +++ mkfs/main.c

[GIT PULL] erofs fixes for 6.9-rc7

2024-04-29 Thread Gao Xiang
have been in -next and no potential merge conflict is observed. [1] https://lore.kernel.org/r/15ab9875-5123-7bc2-bb25-fc683129a...@huawei.com Thanks, Gao Xiang The following changes since commit ed30a4a51bb196781c8058073ea720133a65596f: Linux 6.9-rc5 (2024-04-21 12:35:54 -0700) are available

Re: [PATCH -next v3 2/2] erofs: reliably distinguish block based and fscache mode

2024-04-28 Thread Gao Xiang
initialised, so use sbi->fsid to distinguish between the two modes. Signed-off-by: Christian Brauner Signed-off-by: Baokun Li Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH -next v3 1/2] erofs: get rid of erofs_fs_context

2024-04-28 Thread Gao Xiang
, no functional changes. Suggested-by: Jingbo Xu Signed-off-by: Baokun Li Reviewed-by: Gao Xiang Thanks, Gao Xiang

[PATCH] erofs-utils: mkfs: use all available processors by default

2024-04-27 Thread Gao Xiang
Fulfill the needs of most users. Signed-off-by: Gao Xiang --- include/erofs/config.h | 3 +-- lib/compress.c | 16 lib/config.c | 5 - mkfs/main.c| 10 -- 4 files changed, 17 insertions(+), 17 deletions(-) diff --git a/include/erofs

Re: [PATCH v2] erofs-utils: add missing block counting

2024-04-25 Thread Gao Xiang
On 2024/4/25 10:48, Noboru Asai wrote: Hi Gao, Oh, sorry. I knew to access i_blkaddr on uncompressed file, but it didn't occur on the file system for testing, so I overlooked it. I needed to be careful. np, I've fixed it :) Thanks, Gao Xiang

Re: Trying to work with the tests

2024-04-24 Thread Gao Xiang
On 2024/4/24 18:37, Ian Kent wrote: On 24/4/24 17:47, Ian Kent wrote: On 23/4/24 18:34, Gao Xiang wrote: Hi Ian, On 2024/4/22 21:10, Ian Kent wrote: On 22/4/24 17:12, Gao Xiang wrote: Hi Ian, (+Cc Jingbo here). On 2024/4/22 16:31, Ian Kent wrote: I'm new to the list so Hi to all

Re: [PATCH v2] erofs-utils: add missing block counting

2024-04-24 Thread Gao Xiang
On Wed, Apr 24, 2024 at 02:15:58PM +0800, Gao Xiang wrote: > > > On 2024/4/24 13:59, Noboru Asai wrote: > > Add missing block counting when the data to be inlined is not inlined. > > > > --- > > v2: > > - move from erofs_write_tail_end() to erofs_p

Re: Trying to work with the tests

2024-04-24 Thread Gao Xiang
On 2024/4/24 17:47, Ian Kent wrote: On 23/4/24 18:34, Gao Xiang wrote: Hi Ian, On 2024/4/22 21:10, Ian Kent wrote: On 22/4/24 17:12, Gao Xiang wrote: Hi Ian, (+Cc Jingbo here). On 2024/4/22 16:31, Ian Kent wrote: I'm new to the list so Hi to all, I'm working with a heavily patched

Re: [PATCH v2] erofs-utils: add missing block counting

2024-04-24 Thread Gao Xiang
On 2024/4/24 13:59, Noboru Asai wrote: Add missing block counting when the data to be inlined is not inlined. --- v2: - move from erofs_write_tail_end() to erofs_prepare_tail_block() Signed-off-by: Noboru Asai Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH v2] erofs-utils: fsck: extract chunk-based file with hole correctly

2024-04-24 Thread Gao Xiang
t EROFS_MAP_MAPPED is unset, which is not the case for chunk-based files with holes. This patch handles the corner case correctly. Signed-off-by: Yifan Zhao Signed-off-by: Gao Xiang --- fsck/main.c | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/fsck/main.c b

Re: [PATCH] erofs-utils: add missing block counting

2024-04-23 Thread Gao Xiang
to revisit it. erofs_prepare_tail_block() is the original place to decide if a fallback tail block is needed, that is also true for old versions. Thanks, Gao Xiang

Re: [PATCH] erofs-utils: add missing block counting

2024-04-23 Thread Gao Xiang
for this. Thanks, Gao Xiang --- lib/inode.c | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/inode.c b/lib/inode.c index cf22bbe..727dcee 100644 --- a/lib/inode.c +++ b/lib/inode.c @@ -840,6 +840,7 @@ static int erofs_write_tail_end(struct erofs_inode *inode) inode->idata_size

Re: [PATCH 5/5] cachefiles: add missing lock protection when polling

2024-04-23 Thread Gao Xiang
OLLIN events in ondemand mode") Signed-off-by: Jingbo Xu Reviewed-by: Joseph Qi Reviewed-by: Gao Xiang I'm not sure why this patch didn't send upstream, https://gitee.com/anolis/cloud-kernel/commit/324ecaaa10fefb0e3d94b547e3170e40b90cda1f But since we're now working on upstreaming, so

Re: [PATCH -next] erofs: modify the error message when prepare_ondemand_read failed

2024-04-23 Thread Gao Xiang
(+cc linux-erofs & LKML) On 2024/4/24 10:39, Hongbo Li wrote: When prepare_ondemand_read failed, wrong error message is printed. The prepare_read is also implemented in cachefiles, so we amend it. Signed-off-by: Hongbo Li Reviewed-by: Gao Xiang Could you resend the patch with pr

Re: [syzbot] [erofs?] BUG: using smp_processor_id() in preemptible code in z_erofs_get_gbuf

2024-04-23 Thread Gao Xiang
On 2024/4/10 13:19, syzbot wrote: Hello, syzbot tried to test the proposed patch but the build/boot failed: #syz invalid Since https://lore.kernel.org/r/20240408215231.3376659-1-dhav...@google.com has been folded into the original patch and this issue is only in -next. Thanks, Gao Xiang

Re: [PATCH -next v3 1/2] erofs: get rid of erofs_fs_context

2024-04-23 Thread Gao Xiang
, no functional changes. Suggested-by: Jingbo Xu Signed-off-by: Baokun Li Thanks, it looks good to me, let's see how it behaves after applying to -next. Thanks, Gao Xiang

Re: Trying to work with the tests

2024-04-23 Thread Gao Xiang
Hi Ian, On 2024/4/22 21:10, Ian Kent wrote: On 22/4/24 17:12, Gao Xiang wrote: Hi Ian, (+Cc Jingbo here). On 2024/4/22 16:31, Ian Kent wrote: I'm new to the list so Hi to all, I'm working with a heavily patched 5.14 kernel and I've gathered together patches to bring erofs up to 5.19

Re: [PATCH] erofs-utils: fsck: extract chunk-based file with hole correctly

2024-04-22 Thread Gao Xiang
if (ret) - goto out; + if (!(map.m_flags & EROFS_MAP_MAPPED)) { + memset(raw, 0, count); I think we could use lseek instead of write zeros explicitly.. Thanks, Gao Xiang

Re: Trying to work with the tests

2024-04-22 Thread Gao Xiang
quite sure why "fscache/005" fails, hopefully Jingbo could help you on this. Thanks! Gao Xiang erofs/018 does not run due to "lzma compression is disabled, skipped." message which I think is too old a version of xz. Any insight into cases that could cause these tests to f

[PATCH v2 8/8] erofs-utils: mkfs: enable inter-file multi-threaded compression

2024-04-21 Thread Gao Xiang
From: Gao Xiang Dispatch deferred ops in another per-sb worker thread. Note that deferred ops are strictly FIFOed. Signed-off-by: Gao Xiang --- include/erofs/internal.h | 6 ++ lib/inode.c | 119 ++- 2 files changed, 123 insertions(+), 2

[PATCH v2 7/8] erofs-utils: lib: introduce non-directory jobitem context

2024-04-21 Thread Gao Xiang
From: Gao Xiang It will describe EROFS_MKFS_JOB_NDIR defer work. Also, start compression before queueing EROFS_MKFS_JOB_NDIR. Signed-off-by: Gao Xiang --- lib/inode.c | 62 +++-- 1 file changed, 51 insertions(+), 11 deletions(-) diff --git

[PATCH v2 6/8] erofs-utils: mkfs: prepare inter-file multi-threaded compression

2024-04-21 Thread Gao Xiang
that erofs_begin_compressed_file() and erofs_write_compressed_file() run with different threads even the global inode context is used, thus add another synchronization point. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin Signed-off-by: Gao Xiang --- include/erofs/compress.h | 5 +- lib/compress.c

[PATCH v2 5/8] erofs-utils: lib: split up z_erofs_mt_compress()

2024-04-21 Thread Gao Xiang
From: Gao Xiang The on-disk compressed data write will be moved into a new function erofs_mt_write_compressed_file(). Signed-off-by: Gao Xiang --- lib/compress.c | 162 - 1 file changed, 93 insertions(+), 69 deletions(-) diff --git a/lib

[PATCH v2 4/8] erofs-utils: rearrange several fields for multi-threaded mkfs

2024-04-21 Thread Gao Xiang
From: Gao Xiang They should be located in `struct z_erofs_compress_ictx`. Signed-off-by: Gao Xiang --- lib/compress.c | 57 +++--- 1 file changed, 31 insertions(+), 26 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index 8ca4033..0bc5426

[PATCH v2 3/8] erofs-utils: lib: split out erofs_commit_compressed_file()

2024-04-21 Thread Gao Xiang
From: Gao Xiang Just split out on-disk compressed metadata commit logic. Signed-off-by: Gao Xiang --- lib/compress.c | 191 +++-- 1 file changed, 105 insertions(+), 86 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index b084446..8ca4033

[PATCH v2 2/8] erofs-utils: lib: prepare for later deferred work

2024-04-21 Thread Gao Xiang
From: Gao Xiang Split out ordered metadata operations and add the following helpers: - erofs_mkfs_jobfn() - erofs_mkfs_go() to handle these mkfs job items for multi-threadding support. Signed-off-by: Gao Xiang --- lib/inode.c | 69 - 1

[PATCH v2 1/8] erofs-utils: use erofs_atomic_t for inode->i_count

2024-04-21 Thread Gao Xiang
From: Gao Xiang Since `inode->i_count` can be touched for more than one thread if multi-threading is enabled. Signed-off-by: Gao Xiang --- patchset v1->v2: - Fix `--all-fragments` functionality; - Fix issues pointed out by Yifan. include/erofs/atomic.h | 10 ++ include

[PATCH v2] erofs-utils: mkfs: skip the redundant write for ztailpacking block

2024-04-18 Thread Gao Xiang
mkfs: introduce inner-file multi-threaded compression") Signed-off-by: Yifan Zhao [ Gao Xiang: simplify a bit. ] Signed-off-by: Gao Xiang --- v2: Yifan's patch is almost correct, it just has minor changes. include/erofs/dedupe.h | 2 +- lib/compress.c | 14 +++--- li

Re: [PATCH v2] erofs: reliably distinguish block based and fscache mode

2024-04-18 Thread Gao Xiang
>   } > > > > >   -static int erofs_fc_get_tree(struct fs_context *fc) > > > > > +static void erofs_ctx_to_info(struct fs_context *fc) > > > > >   { > > > > >   struct erofs_fs_context *ctx = fc->fs_private; > > > > > +    struct erofs_sb_info *sbi = fc->s_fs_info; > > > > > + > > > > > +    sbi->opt = ctx->opt; > > > > > +    sbi->devs = ctx->devs; > > > > > +    ctx->devs = NULL; > > > > > +    sbi->fsid = ctx->fsid; > > > > > +    ctx->fsid = NULL; > > > > > +    sbi->domain_id = ctx->domain_id; > > > > > +    ctx->domain_id = NULL; > > > > > +} > > > > I'm not sure if abstracting this logic into a seperate helper really > > > > helps understanding the code as the logic itself is quite simple and > > > > easy to be understood. Usually it's a hint of over-abstraction when a > > > > simple helper has only one caller. > > > > > > > Static functions that have only one caller are compiled inline, so we > > > don't have to worry about how that affects the code. > > > > > > The reason these codes are encapsulated in a separate function is so > > > that the code reader understands that these codes are integrated > > > as a whole, and that we shouldn't have to move one or two of these > > > lines individually. > > > > > > But after we get rid of erofs_fs_context, those won't be needed > > > anymore. > > Yeah, I understand. It's only coding style concerns. > > > > > > > Okay, thanks! I'm fine to get rid of those (erofs_fs_context) as long as the codebase is more clearer and simple. BTW, for the current codebase, I also think it's unneeded to have a separate helper called once without extra actual meaning... Thanks, Gao Xiang > > -- > With Best Regards, > Baokun Li

Re: [PATCH 1/3] erofs-utils: determine the [un]compressed data block is inline or not early

2024-04-18 Thread Gao Xiang
our effort on this issue again! Thanks, Gao Xiang

Re: [PATCH v3] erofs-utils: lib: treat data blocks filled with 0s as a hole

2024-04-17 Thread Gao Xiang
o detect blocks filled with zeros during chunking, we insert block > filled with zeros (zerochunk) in the hashmap. If we detect a possible > dedupe, we map it to the hole so there is no physical block assigned. > > Signed-off-by: Sandeep Dhavale Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH] erofs-utils: mkfs: skip the redundant write for ztailpacking block

2024-04-17 Thread Gao Xiang
lled for inline fallback instead. Let me think more about it and update the patch. Thanks, Gao Xiang

Re: [PATCH v3] erofs-utils: dump: print filesystem blocksize

2024-04-17 Thread Gao Xiang
Wed Apr 17 16:53:10 2024 > Filesystem features: sb_csum mtime 0padding > Filesystem UUID: > e66f6dd1-6882-48c3-9770-fee7c4841a93 > > Signed-off-by: Sandeep Dhavale Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH] erofs: set SB_NODEV sb_flags when mounting with fsid

2024-04-16 Thread Gao Xiang
On Wed, Apr 17, 2024 at 10:59:53AM +0800, Baokun Li wrote: > On 2024/4/16 22:49, Gao Xiang wrote: > > On Tue, Apr 16, 2024 at 02:35:08PM +0200, Christian Brauner wrote: > > > > > I'm not sure how to resolve it in EROFS itself, anyway... > > > Instead of allocatin

Re: [PATCH] fs/erofs: add DEFLATE algorithm support

2024-04-16 Thread Gao Xiang
les since EROFS > supports multiple compression algorithms in one image. > > Signed-off-by: Jianan Huang Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH 4/8] erofs-utils: rearrange several fields for multi-threaded mkfs

2024-04-16 Thread Gao Xiang
On Tue, Apr 16, 2024 at 07:55:05PM +0800, Yifan Zhao wrote: > > On 4/16/24 4:04 PM, Gao Xiang wrote: > > From: Gao Xiang > > > > They should be located in `struct z_erofs_compress_ictx`. > > > > Signed-off-by: Gao Xian

Re: [PATCH 2/8] erofs-utils: lib: prepare for later deferred work

2024-04-16 Thread Gao Xiang
Hi Yifan, On Tue, Apr 16, 2024 at 07:58:30PM +0800, Yifan Zhao wrote: > > On 4/16/24 4:04 PM, Gao Xiang wrote: > > From: Gao Xiang > > > > Split out ordered metadata operations and add the following helpers: > > > > - erofs_mkfs_jobfn() > >

Re: [PATCH] erofs: set SB_NODEV sb_flags when mounting with fsid

2024-04-16 Thread Gao Xiang
n, Could you also check this on your side? Thanks, Gao Xiang > From e4f586a41748b6edc05aca36d49b7b39e55def81 Mon Sep 17 00:00:00 2001 > From: Christian Brauner > Date: Mon, 15 Apr 2024 20:17:46 +0800 > Subject: [PATCH] erofs: reliably distinguish block based and fscache mode > > W

[PATCH 7/8] erofs-utils: lib: introduce non-directory jobitem context

2024-04-16 Thread Gao Xiang
From: Gao Xiang It will describe EROFS_MKFS_JOB_NDIR defer work. Also start compression before queueing EROFS_MKFS_JOB_NDIR. Signed-off-by: Gao Xiang --- lib/inode.c | 62 +++-- 1 file changed, 51 insertions(+), 11 deletions(-) diff --git

[PATCH 6/8] erofs-utils: mkfs: prepare inter-file multi-threaded compression

2024-04-16 Thread Gao Xiang
Signed-off-by: Gao Xiang --- include/erofs/compress.h | 5 +- lib/compress.c | 115 ++- lib/inode.c | 17 +- 3 files changed, 95 insertions(+), 42 deletions(-) diff --git a/include/erofs/compress.h b/include/erofs/compress.h index

[PATCH 4/8] erofs-utils: rearrange several fields for multi-threaded mkfs

2024-04-16 Thread Gao Xiang
From: Gao Xiang They should be located in `struct z_erofs_compress_ictx`. Signed-off-by: Gao Xiang --- lib/compress.c | 55 -- 1 file changed, 31 insertions(+), 24 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index a2e0d0f..72f33d2

[PATCH 8/8] erofs-utils: mkfs: enable inter-file multi-threaded compression

2024-04-16 Thread Gao Xiang
From: Gao Xiang Dispatch deferred ops in another per-sb worker thread. Note that deferred ops are strictly FIFOed. Signed-off-by: Gao Xiang --- include/erofs/internal.h | 6 ++ lib/inode.c | 121 ++- 2 files changed, 124 insertions(+), 3

[PATCH 3/8] erofs-utils: lib: split out erofs_commit_compressed_file()

2024-04-16 Thread Gao Xiang
From: Gao Xiang Just split out on-disk compressed metadata commit logic. Signed-off-by: Gao Xiang --- lib/compress.c | 191 +++-- 1 file changed, 105 insertions(+), 86 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index 74c5707..a2e0d0f

[PATCH 5/8] erofs-utils: lib: split up z_erofs_mt_compress()

2024-04-16 Thread Gao Xiang
From: Gao Xiang The on-disk compressed data write will be moved into a new function erofs_mt_write_compressed_file(). Signed-off-by: Gao Xiang --- lib/compress.c | 172 - 1 file changed, 99 insertions(+), 73 deletions(-) diff --git a/lib

[PATCH 2/8] erofs-utils: lib: prepare for later deferred work

2024-04-16 Thread Gao Xiang
From: Gao Xiang Split out ordered metadata operations and add the following helpers: - erofs_mkfs_jobfn() - erofs_mkfs_go() to handle these mkfs job items for multi-threadding support. Signed-off-by: Gao Xiang --- lib/inode.c | 68 + 1

[PATCH 1/8] erofs-utils: use erofs_atomic_t for inode->i_count

2024-04-16 Thread Gao Xiang
From: Gao Xiang Since it can be touched for more than one thread if multi-threading is enabled. Signed-off-by: Gao Xiang --- include/erofs/atomic.h | 10 ++ include/erofs/inode.h| 2 +- include/erofs/internal.h | 3 ++- lib/inode.c | 5 +++-- 4 files changed, 16

Re: [PATCH v2] erofs-utils: dump: print filesystem blocksize

2024-04-15 Thread Gao Xiang via Linux-erofs
%u` for `erofs_blksiz()`? since currently EROFS block size isn't possible to be larger than PAGE_SIZE. Even if block size > page size is supported, I think we should not consider too large blocksizes. Otherwise it looks good to me. Thanks, Gao Xiang

Re: [PATCH] erofs: set SB_NODEV sb_flags when mounting with fsid

2024-04-15 Thread Gao Xiang
se super_block to determine whether it is currently in nodev > fscahe mode or block device based mode. So if it is recorded in > erofs_fs_context (aka fc->fs_private), we can't access the recorded data > unless we pass fc into erofs_kill_sb() as well. > If I understand correctly, from the discussion above, I think there exists a gap between alloc_super() and sb->s_bdev is set. But .kill_sb() can be called between them and fc is not passed into .kill_sb(). I'm not sure how to resolve it in EROFS itself, anyway... Thanks, Gao Xiang

Re: [PATCH] erofs: Consider NUMA affinity when allocating memory for per-CPU pcpubuf

2024-04-15 Thread Gao Xiang
s needed anyway.) Thanks, Gao Xiang

Re: [PATCH v1] erofs-utils: dump: print filesystem blocksize

2024-04-12 Thread Gao Xiang
On 2024/4/13 09:00, Sandeep Dhavale wrote: On Fri, Apr 12, 2024 at 5:09 PM Gao Xiang wrote: Hi Sandeep, On 2024/4/13 06:51, Sandeep Dhavale wrote: mkfs.erofs supports creating filesystem images with different blocksizes. Add filesystem blocksize in super block dump so its easier

Re: [PATCH v1] erofs-utils: dump: print filesystem blocksize

2024-04-12 Thread Gao Xiang
-b811a815fa07 Filesystem blocksize: 65536 Signed-off-by: Sandeep Dhavale Just a minor nit: Could we move "Filesystem blocksize:" between the line of "Filesystem magic number:" and "Filesystem blocks:" ? Otherwise it looks good to me, thanks for this! Thanks, Gao Xiang

Re: [PATCH v2] erofs-utils: lib: treat data blocks filled with 0s as a hole

2024-04-12 Thread Gao Xiang
if (!zeros) + return -ENOMEM; + + erofs_sha256(zeros, chunksize, sha256); + hash = memhash(sha256, sizeof(sha256)); `zeros` needs to be freed here I guess: free(zeros); Thanks, Gao Xiang + chunk = malloc(sizeof(struct erofs_blobchunk)); +

[PATCH] erofs-utils: lib: fix tarerofs 32-bit overflows

2024-04-11 Thread Gao Xiang
Otherwise, large files won't be imported properly. Fixes: e3dfe4b8db26 ("erofs-utils: mkfs: support tgz streams for tarerofs") Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs") Signed-off-by: Gao Xiang --- lib/tar.c | 5 ++--- 1 file changed, 2 insertions(+), 3 dele

Re: [PATCH] erofs: derive fsid from on-disk UUID for .statfs() if possible

2024-04-09 Thread Gao Xiang
On 2024/4/9 16:11, Hongzhen Luo wrote: Use the superblock's UUID to generate the fsid when it's non-null. Signed-off-by: Hongzhen Luo Reviewed-by: Gao Xiang Thanks, Gao Xiang --- fs/erofs/super.c | 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/fs

Re: [syzbot] [erofs?] BUG: using smp_processor_id() in preemptible code in z_erofs_get_gbuf

2024-04-08 Thread Gao Xiang
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git dev

Re: [PATCH v1] erofs: use raw_smp_processor_id() to get buffer from global buffer pool

2024-04-08 Thread Gao Xiang
. Thanks, Gao Xiang

Re: [PATCH v1] erofs: use raw_smp_processor_id() to get buffer from global buffer pool

2024-04-08 Thread Gao Xiang
patch? I will add your credit into the original patch. Thanks, Gao Xiang --- fs/erofs/zutil.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/erofs/zutil.c b/fs/erofs/zutil.c index b9b99158bb4e..036024bce9f7 100644 --- a/fs/erofs/zutil.c +++ b/fs/erofs/zutil.c @@ -

Re: [PATCH] erofs-utils: change temporal buffer to non static

2024-04-08 Thread Gao Xiang
enable this. So I will apply this first :) Thanks, Gao Xiang Thanks, Yifan Zhao On 4/8/24 5:16 PM, Noboru Asai wrote: In multi-threaded mode, each thread must use a different buffer in tryrecompress_trailing function, so change this buffer to non static. Signed-off-by: Noboru Asai

Re: [PATCH] erofs-utils: change temporal buffer to non static

2024-04-08 Thread Gao Xiang
On 2024/4/8 17:16, Noboru Asai wrote: In multi-threaded mode, each thread must use a different buffer in tryrecompress_trailing function, so change this buffer to non static. Signed-off-by: Noboru Asai Thanks for catching this! Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: 您好,我在2024年看到了社区的开源项目

2024-04-06 Thread Gao Xiang
] https://summer-ospp.ac.cn/org/orgdetail/192db666-dfa0-46d9-9f58-9d8d165e2914 [2] https://erofs.docs.kernel.org Thanks, Gao Xiang

[PATCH 2/2] erofs-utils: lib: refine on-disk meta arrangement again

2024-04-05 Thread Gao Xiang
% (33.040s->2.861s) as well as the new image size is decreased by 0.0094% (120 KiB), which is minor. Signed-off-by: Gao Xiang --- lib/inode.c | 192 +--- 1 file changed, 91 insertions(+), 101 deletions(-) diff --git a/lib/inode.c b/lib/inode.c index f419

[PATCH 1/2] erofs-utils: lib: split out several helpers in inode.c

2024-04-05 Thread Gao Xiang
From: Yifan Zhao The following new helpers are added to prepare for the upcoming multi-threaded inter-file compression: - erofs_mkfs_handle_{non,}directory; - erofs_write_unencoded_file. Signed-off-by: Yifan Zhao Signed-off-by: Gao Xiang --- lib/inode.c | 106

Re: [PATCH 1/1] erofs-utils: lib: treat data blocks filled with 0s as a hole

2024-04-04 Thread Gao Xiang
unk returns erofs_holechunk. Does it sound a good idea to you? Thanks, Gao Xiang if (IS_ERR(chunk)) { ret = PTR_ERR(chunk);

Re: [PATCH v2] erofs-utils: lib: Fix calculation of minextblks when working with sparse files

2024-04-03 Thread Gao Xiang
multiple places, put the logic in helper function for avoiding repetition and easier reading. Fixes: 7b46f7a0160a (erofs-utils: lib: merge consecutive chunks if possible) Signed-off-by: Sandeep Dhavale Reviewed-by: Gao Xiang Thanks, Gao Xiang

Re: [PATCH v1 1/1] erofs-utils: lib: Fix calculation of minextblks when working with sparse files

2024-04-03 Thread Gao Xiang
*minextblks = lb; } could be better (or just my own perference) since no input + output on a same variable, otherwise it looks good to me. If you agree on this minor nit, I could revise it directly or you could send another version. Both are fine with me :) Thanks, Gao Xiang + int erofs_bl

Re: [PATCH v2] erofs: add a reserved buffer pool for lz4 decompression

2024-04-02 Thread Gao Xiang
on the benchmark logs, 64 pages are sufficient for 95% of scenarios. This value can be adjusted from the module parameter. The default value is 0. This pool is currently only used for the LZ4 decompressor, but it can be applied to more decompressors if needed. Signed-off-by: Chunhai Guo Reviewed-by: Gao Xiang

Re: [PATCH] erofs: add a reserved buffer pool for lz4 decompression

2024-04-02 Thread Gao Xiang
erofs_rsv_nrpages = 0; } Thanks, Gao Xiang

Re: [PATCH v2] erofs: do not use pagepool in z_erofs_gbuf_growsize()

2024-04-02 Thread Gao Xiang
On 2024/4/2 17:27, Chunhai Guo wrote: Let's use alloc_pages_bulk_array() for simplicity and get rid of unnecessary pagepool. Signed-off-by: Chunhai Guo --- Would you mind adding a changelog here next time? no matter how short the text is. Reviewed-by: Gao Xiang Thanks, Gao Xiang fs

Re: [PATCH] erofs: do not use pagepool in z_erofs_gbuf_growsize()

2024-04-02 Thread Gao Xiang
_pages[j]); + kfree(tmp_pages); + } mutex_unlock(_resize_mutex); return ret; return i < z_erofs_gbuf_count ? -ENOMEM: 0; Otherwise it looks good to me. Thanks, Gao Xiang }

Re: [PATCH 2/2] erofs: rename per CPU buffer to global buffer pool and make it configurable

2024-04-01 Thread Gao Xiang
with hundreds of CPUs, and only a small number of CPUs concurrently decompress most of the time. This patch renames it as 'global buffer pool' and makes it configurable. This allows two or more CPUs to share a common buffer to reduce memory occupation. Suggested-by: Gao Xiang Reviewed-by: Gao

[PATCH] erofs-utils: set opaque flag for directories in tarerofs mode

2024-04-01 Thread Gao Xiang
Opaque dir flag is needed if the tar tree is used immediately for the upcoming append mode. Signed-off-by: Gao Xiang --- lib/tar.c | 5 + 1 file changed, 5 insertions(+) diff --git a/lib/tar.c b/lib/tar.c index 7c271f6..b45657d 100644 --- a/lib/tar.c +++ b/lib/tar.c @@ -909,6 +909,11

  1   2   3   4   5   6   7   8   9   10   >