[PATCH] erofs-utils: mkfs: error return if meets an unknown extended option

2023-09-09 Thread Yifan Zhao
Currently mkfs would ignore any unknown extended option, which keeps silent if a mistyped option is given. Return failure in this case allows users to catch their errors more easily. --- mkfs/main.c | 27 --- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git

[PATCH] erofs-utils: mkfs: fix potential memory leak

2023-11-04 Thread Yifan Zhao
: realpath_stk (canonicalize.c:409) by 0x48FFE2B: realpath@@GLIBC_2.3 (canonicalize.c:431) by 0x10B7EB: mkfs_parse_options_cfg (main.c:587) by 0x10B7EB: main (main.c:936) Fix it by freeing the memory allocated by strdup() and realpath(). Signed-off-by: Yifan Zhao --- lib

[PATCH v2] erofs-utils: mkfs: generate on-disk indexes after compression

2023-11-07 Thread Yifan Zhao
. Signed-off-by: Yifan Zhao --- Change since v1: 1. Polish commit message. 2. Iterate the extent list in a wrapper function to avoid massive change of indentation. 3. Remove the redundant reset_clusterofs field in struct z_erofs_inmem_extent. Instead, we check that if an uncompressed

[PATCH] erofs-utils: mkfs: write lcluster index after compression

2023-11-06 Thread Yifan Zhao
, this patch moves the writing of the lcluster indexes to take place after the file's compression is completed. Signed-off-by: Yifan Zhao --- include/erofs/dedupe.h | 4 +- lib/compress.c | 366 - lib/dedupe.c | 1 + 3 files changed

[RFC PATCH] erofs-utils: mkfs: introduce multi-thread compression

2023-08-19 Thread Yifan Zhao
This patch introduce multi-thread compression to accelerate image packaging. --- Hi all: This is a very imperfect patch not ready for merging, and any suggestions would be appreciated! If it's on track, I'd like to follow up on that. The inefficiency of EROFS compressed image creation is a much

[PATCH] erofs-utils: lib: fallback for copy_file_range

2023-06-25 Thread Yifan Zhao
If tmpfs is used for /tmp and blob device is not specified, we need to copy data between two different file systems during mkfs, which is not supported by the copy_file_range() syscall. In this case, let's give it a second chance by fallback to __erofs_copy_file_range(). Signed-off-by: Yifan Zhao

[PATCH] erofs-utils: mkfs: reorganize logic in erofs_compressor_init()

2024-01-21 Thread Yifan Zhao
last, allowing it to use the compression level and dictsize already set so that the behavior of the functions matches their names. Signed-off-by: Yifan Zhao --- lib/compressor.c| 8 lib/compressor_deflate.c| 30 +++--- lib/compressor_libdeflate.c

Re: [PATCH] erofs-utils: lib: reset hc to avoid 32-bit overflow of kite-deflate

2024-01-24 Thread Yifan Zhao
On 1/24/24 17:16, Gao Xiang wrote: Yifan reported a "segmentation fault (core dumped)" error days ago with a large dataset (enwik9 x 5). Let's fix it. Reported-by: Yifan Zhao Fixes: 861037f4fc15 ("erofs-utils: add a built-in DEFLATE compressor") Signed-off-by: Gao

Re: [PATCH] erofs-utils: lib: reset hc to avoid 32-bit overflow of kite-deflate

2024-01-24 Thread Yifan Zhao
On 2024/1/24 17:53, Gao Xiang wrote: Hi Yifan, On 2024/1/24 17:47, Yifan Zhao wrote: On 1/24/24 17:16, Gao Xiang wrote: Yifan reported a "segmentation fault (core dumped)" error days ago with a large dataset (enwik9 x 5).   Let's fix it. Reported-by: Yifan Zhao Fixes: 86

[PATCH v2 0/7] erofs-utils: mkfs: introduce multi-threaded compression

2024-02-19 Thread Yifan Zhao
Change log since v1: - Re-implement workqueue API instead of using xfsprogs' workqueue - Use per-worker tmpfile for multi-threaded mkfs Gao Xiang (1): erofs-utils: add a helper to get available processors Yifan Zhao (6): erofs-utils: introduce multi-threading framework erofs-utils: mkfs

[PATCH v2 3/7] erofs-utils: mkfs: add --worker=# parameter

2024-02-19 Thread Yifan Zhao
This patch introduces a --worker=# parameter for the incoming multi-threaded compression support. It also introduces a segment size used in multi-threaded compression, which has the default value 16MB and cannot be modified. Signed-off-by: Yifan Zhao --- include/erofs/config.h | 4 lib

[PATCH v2 2/7] erofs-utils: add a helper to get available processors

2024-02-19 Thread Yifan Zhao
From: Gao Xiang In order to prepare for multi-threaded decompression. Signed-off-by: Gao Xiang --- configure.ac | 1 + include/erofs/config.h | 1 + lib/config.c | 12 3 files changed, 14 insertions(+) diff --git a/configure.ac b/configure.ac index

[PATCH v2 1/7] erofs-utils: introduce multi-threading framework

2024-02-19 Thread Yifan Zhao
Add a workqueue implementation for multi-threading support inspired by xfsprogs. Signed-off-by: Yifan Zhao Suggested-by: Gao Xiang --- configure.ac | 16 + include/erofs/internal.h | 3 + include/erofs/workqueue.h | 38 +++ lib/Makefile.am | 4 ++ lib

[PATCH v2 6/7] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-02-19 Thread Yifan Zhao
of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 17 ++ include/erofs/internal.h | 3 + include/erofs/list.h | 8 + include/erofs/queue.h| 22 ++ lib

[PATCH v2 5/7] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-02-19 Thread Yifan Zhao
compresses a segment as if it were a separate file. Finally, the main thread merges all the compressed segments. Multi-threaded compression is not compatible with -Ededupe, -E(all-)fragments and -Eztailpacking for now. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h

[PATCH v2 4/7] erofs-utils: mkfs: optionally print warning in erofs_compressor_init

2024-02-19 Thread Yifan Zhao
() interface. Signed-off-by: Yifan Zhao --- lib/compress.c | 3 ++- lib/compressor.c| 5 +++-- lib/compressor.h| 5 +++-- lib/compressor_deflate.c| 4 +++- lib/compressor_libdeflate.c | 4 +++- lib/compressor_liblzma.c| 5 - lib/compressor_lz4.c| 2

[PATCH v2 7/7] erofs-utils: mkfs: use per-worker tmpfile for multi-threaded mkfs

2024-02-19 Thread Yifan Zhao
the fallocate() syscall and FALLOC_FL_PUNCH_HOLE flag. Signed-off-by: Yifan Zhao --- lib/compress.c | 68 +++--- 1 file changed, 54 insertions(+), 14 deletions(-) diff --git a/lib/compress.c b/lib/compress.c index d5a5f16..3fae260 100644 --- a/lib/compress.c

Re: [PATCH v2] erofs: fix lz4 inplace decompression

2023-12-06 Thread Yifan Zhao
by the same reason. Gao's patch successfully fixes the problem that both Juhyung and I encountered in my machine. Tested-by: Yifan Zhao Thanks, Yifan Zhao On 2023/12/6 12:55, Gao Xiang wrote: Currently EROFS can map another compressed buffer for inplace decompression, that was used

[PATCH 2/3] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-06 Thread Yifan Zhao
be specified together with dictsize. The old -zX, way is still supported for compatibility. Suggested-by: Gao Xiang Signed-off-by: Yifan Zhao --- include/erofs/config.h | 10 ++-- include/erofs_fs.h | 4 ++ lib/compress.c | 45 + lib/compress_hints.c | 2 +- lib

[PATCH 3/3] erofs-utils: mkfs: reorganize logic in erofs_compressor_init()

2024-01-06 Thread Yifan Zhao
last, allowing it to use the compression level and dictsize already set so that the behavior of the functions matches their names. Signed-off-by: Yifan Zhao --- lib/compressor.c| 8 lib/compressor_deflate.c| 32 lib/compressor_libdeflate.c

[PATCH 1/3] erofs-utils: mkfs: merge erofs_compressor_setlevel() into erofs_compressor_init()

2024-01-06 Thread Yifan Zhao
it, let's report a warning rather than early aborting. Besides, we do not need to assign the {default,best}_level for such an algorithm in erofs_compressor struct. Signed-off-by: Yifan Zhao --- lib/compress.c | 6 +- lib/compressor.c | 28 ++-- lib

[PATCH] erofs-utils: mkfs: fix a misspelling

2024-01-13 Thread Yifan Zhao
Fix a misspelling in the version() function of mkfs.erofs. Signed-off-by: Yifan Zhao --- mkfs/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkfs/main.c b/mkfs/main.c index 0517849..13fea41 100644 --- a/mkfs/main.c +++ b/mkfs/main.c @@ -175,7 +175,7 @@ static void

[PATCH v2 2/3] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-07 Thread Yifan Zhao
be specified together with dictsize. The old -zX, way is still supported for compatibility. Suggested-by: Gao Xiang Signed-off-by: Yifan Zhao --- change since v1: - fix a bug in mkfs_parse_compress_algs() that "-zlzma,dictsize=" is not correctly parsed as error. include/erofs/config

[PATCH 7/7] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-02-04 Thread Yifan Zhao
of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 17 ++ include/erofs/internal.h | 3 + include/erofs/list.h | 8 + include/erofs/queue.h| 22 ++ lib

[PATCH 6/7] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-02-04 Thread Yifan Zhao
compresses a segment as if it were a separate file. Finally, the main thread merges all the compressed segments. Multi-threaded compression is not compatible with -Ededupe, -E(all-)fragments and -Eztailpacking for now. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h

[PATCH 1/7] erofs-utils: introduce multi-threading framework

2024-02-04 Thread Yifan Zhao
From: Gao Xiang Add a workqueue implementation based on xfsprogs for multi-threading support. Signed-off-by: Gao Xiang --- configure.ac | 16 +++ include/erofs/internal.h | 3 + include/erofs/workqueue.h | 44 lib/Makefile.am | 4 + lib/workqueue.c

[PATCH 4/7] erofs-utils: mkfs: optionally print warning in erofs_compressor_init

2024-02-04 Thread Yifan Zhao
() interface. Signed-off-by: Yifan Zhao --- lib/compress.c | 3 ++- lib/compressor.c| 5 +++-- lib/compressor.h| 5 +++-- lib/compressor_deflate.c| 10 ++ lib/compressor_libdeflate.c | 6 -- lib/compressor_liblzma.c| 9 ++--- lib

[PATCH 5/7] erofs-utils: extend multi-threading framework for per-thread fields

2024-02-04 Thread Yifan Zhao
Currently each worker thread is stateless. This patch allows us to bind some per-worker fields to each worker thread, which could be reused accross different tasks. One of the examples is the compressor and buffer used in multi-threaded compression. Signed-off-by: Yifan Zhao --- include/erofs

[PATCH 3/7] erofs-utils: mkfs: add --worker=# parameter

2024-02-04 Thread Yifan Zhao
This patch introduces a --worker=# parameter for the incoming multi-threaded compression support. It also introduces a segment size used in multi-threaded compression, which has the default value 16MB and cannot be modified. Signed-off-by: Yifan Zhao --- include/erofs/config.h | 4 lib

[PATCH 2/7] erofs-utils: add a helper to get available processors

2024-02-04 Thread Yifan Zhao
From: Gao Xiang In order to prepare for multi-threaded decompression. Signed-off-by: Gao Xiang --- configure.ac | 1 + include/erofs/config.h | 1 + lib/config.c | 12 3 files changed, 14 insertions(+) diff --git a/configure.ac b/configure.ac index

[PATCH 0/7] erofs-utils: mkfs: introduce multi-threaded compression

2024-02-04 Thread Yifan Zhao
| 183.26 | 4.55x | Gao Xiang (2): erofs-utils: introduce multi-threading framework erofs-utils: add a helper to get available processors Yifan Zhao (5): erofs-utils: mkfs: add --worker=# parameter erofs-utils: mkfs: optionally print warning in erofs_compressor_init erofs

[PATCH v3 0/3] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-19 Thread Yifan Zhao
, reorganize and code style fix Yifan Zhao (3): erofs-utils: mkfs: merge erofs_compressor_setlevel() into erofs_compressor_init() erofs-utils: mkfs: allow to specify dictionary size for compression algorithms erofs-utils: mkfs: reorganize logic in erofs_compressor_init() include/erofs/config.h

[PATCH v3 1/3] erofs-utils: mkfs: merge erofs_compressor_setlevel() into erofs_compressor_init()

2024-01-19 Thread Yifan Zhao
does not support the compression level in its erofs_compressor struct. Signed-off-by: Yifan Zhao --- lib/compress.c | 6 +- lib/compressor.c | 31 +-- lib/compressor.h | 5 ++--- lib/compressor_lz4.c | 2 -- 4 files changed, 20 insertions(+), 24

[PATCH v3 2/3] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-19 Thread Yifan Zhao
together with dictsize. The old -zX, form is still supported for compatibility. Suggested-by: Gao Xiang Signed-off-by: Yifan Zhao --- include/erofs/config.h | 10 ++-- lib/compress.c | 33 - lib/compress_hints.c | 2 +- lib/compressor.c | 16 +- lib

[PATCH v3 3/3] erofs-utils: mkfs: reorganize logic in erofs_compressor_init()

2024-01-19 Thread Yifan Zhao
last, allowing it to use the compression level and dictsize already set so that the behavior of the functions matches their names. Signed-off-by: Yifan Zhao --- lib/compressor.c| 8 lib/compressor_deflate.c| 30 +++--- lib/compressor_libdeflate.c

[PATCH] erofs-utils: introduce GitHub Actions CI

2024-01-20 Thread Yifan Zhao
and compares the sha256sum of the extracted files with the original ones. Signed-off-by: Yifan Zhao --- Hi forks, I believe it's time to introduce a CI for erofs-utils, which could help us to ensure the correctness of the code and avoid regressions. Currently this simple CI covers only a limited

[PATCH v4 2/2] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-20 Thread Yifan Zhao
together with dictsize. The old -zX, form is still supported for compatibility. Suggested-by: Gao Xiang Signed-off-by: Yifan Zhao --- include/erofs/config.h | 10 ++-- lib/compress.c | 33 - lib/compress_hints.c | 2 +- lib/compressor.c | 16 +- lib

[PATCH v4 0/2] erofs-utils: mkfs: allow to specify dictionary size for compression algorithms

2024-01-20 Thread Yifan Zhao
This patchset allows to specify dictionary size for compression algorithms. change since v3: - remove the third patch as is unrelated to this patchset, will send as a separate patch later - Now patch 1 and 2 are able to be `git am` without any conflict Yifan Zhao (2): erofs-utils: mkfs

[PATCH v4 1/2] erofs-utils: mkfs: merge erofs_compressor_setlevel() into erofs_compressor_init()

2024-01-20 Thread Yifan Zhao
does not support the compression level in its erofs_compressor struct. Signed-off-by: Yifan Zhao --- lib/compress.c | 6 +- lib/compressor.c | 31 +-- lib/compressor.h | 5 ++--- lib/compressor_lz4.c | 2 -- 4 files changed, 20 insertions(+), 24

[PATCH v3 0/4] erofs-utils: mkfs: introduce multi-threaded compression

2024-02-25 Thread Yifan Zhao
the patch related to pring warning from this patchset, which may be supported later with atomic variables Gao Xiang (1): erofs-utils: add a helper to get available processors Yifan Zhao (3): erofs-utils: introduce multi-threading framework erofs-utils: mkfs: add --worker=# parameter erofs

[PATCH v3 1/4] erofs-utils: introduce multi-threading framework

2024-02-25 Thread Yifan Zhao
Add a workqueue implementation for multi-threading support inspired by xfsprogs. Signed-off-by: Yifan Zhao Suggested-by: Gao Xiang --- configure.ac | 16 + include/erofs/internal.h | 3 + include/erofs/workqueue.h | 37 +++ lib/Makefile.am | 4 ++ lib

[PATCH v3 2/4] erofs-utils: add a helper to get available processors

2024-02-25 Thread Yifan Zhao
From: Gao Xiang In order to prepare for multi-threaded decompression. Signed-off-by: Gao Xiang --- configure.ac | 1 + include/erofs/config.h | 1 + lib/config.c | 12 3 files changed, 14 insertions(+) diff --git a/configure.ac b/configure.ac index

[PATCH v3 3/4] erofs-utils: mkfs: add --worker=# parameter

2024-02-25 Thread Yifan Zhao
for multi-threading, which has the default value 16MB for now. Signed-off-by: Yifan Zhao --- include/erofs/config.h | 4 lib/config.c | 4 mkfs/main.c| 38 ++ 3 files changed, 46 insertions(+) diff --git a/include/erofs

[PATCH v3 4/4] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-02-25 Thread Yifan Zhao
compresses a segment as if it were a separate file. Finally, the main thread merges all the compressed segments. Multi-threaded compression is not compatible with -Ededupe, -E(all-)fragments and -Eztailpacking for now. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h

Re: [PATCH] erofs-utils: lib: introduce atomic operations

2024-02-28 Thread Yifan Zhao
will be printed only once by adding `__warnonce` atomic booleans to each erofs_compressor_init(). Cc: Yifan Zhao Signed-off-by: Gao Xiang --- include/erofs/atomic.h | 27 +++ lib/compressor_deflate.c| 11 --- lib/compressor_libdeflate.c | 6 +- lib

Re: [PATCH v3 4/4] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-02-28 Thread Yifan Zhao
On 2/28/24 17:57, Gao Xiang wrote: On 2024/2/25 22:27, Yifan Zhao wrote: Currently, the creation of EROFS compressed image creation is single-threaded, which suffers from performance issues. This patch attempts to address it by compressing the large file in parallel. Specifically, each

[PATCH v4 5/5] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-02-28 Thread Yifan Zhao
compresses a segment as if it were a separate file. Finally, the main thread merges all the compressed segments. Multi-threaded compression is not compatible with -Ededupe, -E(all-)fragments and -Eztailpacking for now. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h

[PATCH v4 1/5] erofs-utils: introduce multi-threading framework

2024-02-28 Thread Yifan Zhao
Add a workqueue implementation for multi-threading support inspired by xfsprogs. Signed-off-by: Yifan Zhao Suggested-by: Gao Xiang --- configure.ac | 16 + include/erofs/internal.h | 3 + include/erofs/workqueue.h | 37 +++ lib/Makefile.am | 4 ++ lib

[PATCH v4 4/5] erofs-utils: lib: introduce atomic operations

2024-02-28 Thread Yifan Zhao
once by adding `__warnonce` atomic booleans to each erofs_compressor_init(). Cc: Yifan Zhao Signed-off-by: Gao Xiang Signed-off-by: Yifan Zhao --- include/erofs/atomic.h | 28 lib/compressor_deflate.c| 11 --- lib/compressor_libdeflate.c | 6

[PATCH v4 0/5] erofs-utils: mkfs: introduce multi-threaded compression

2024-02-28 Thread Yifan Zhao
inode_ctx - leave tryrecompress_trailing() unchanged as it's only used in ztailpacking scenario which is not supported yet Gao Xiang (2): erofs-utils: add a helper to get available processors erofs-utils: lib: introduce atomic operations Yifan Zhao (3): erofs-utils: introduce multi-

[PATCH v4 2/5] erofs-utils: add a helper to get available processors

2024-02-28 Thread Yifan Zhao
From: Gao Xiang In order to prepare for multi-threaded decompression. Signed-off-by: Gao Xiang --- configure.ac | 1 + include/erofs/config.h | 1 + lib/config.c | 12 3 files changed, 14 insertions(+) diff --git a/configure.ac b/configure.ac index

[PATCH v4 3/5] erofs-utils: mkfs: add --worker=# parameter

2024-02-28 Thread Yifan Zhao
for multi-threading, which has the default value 16MB for now. Signed-off-by: Yifan Zhao --- include/erofs/config.h | 4 lib/config.c | 4 mkfs/main.c| 38 ++ 3 files changed, 46 insertions(+) diff --git a/include/erofs

Re: [PATCH v4 1/5] erofs-utils: introduce multi-threading framework

2024-02-29 Thread Yifan Zhao
On 2/29/24 17:43, Gao Xiang wrote: Hi Yifan, On 2024/2/29 00:16, Yifan Zhao wrote: Add a workqueue implementation for multi-threading support inspired by xfsprogs. Signed-off-by: Yifan Zhao Suggested-by: Gao Xiang ---   configure.ac  |  16 +   include/erofs/internal.h

Re: [PATCH v3 0/4] erofs-utils: mkfs: introduce multi-threaded compression

2024-02-28 Thread Yifan Zhao
images for now. (I couldn't apply v2 patch set without rejects.) 2024年2月25日(日) 23:28 Yifan Zhao : We split the inter-file compression functionality as a separate patch set for ease of review. It will be re-sent shortly (maybe a few days) after we finish polishing the inner-file patch set. I

[PATCH 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-17 Thread Yifan Zhao
, the logic of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 16 ++ include/erofs/internal.h | 3 + include/erofs/list.h | 8 + include

[PATCH 1/2] erofs-utils: lib: split function logic in inode.c

2024-03-17 Thread Yifan Zhao
This patch splits part of the logic in function erofs_mkfs_build_tree() and erofs_mkfs_build_tree_from_path() into several new functions. This is in preparation for the upcoming inter-file multi-threaded compression feature. Signed-off-by: Yifan Zhao --- lib/inode.c | 161

[PATCH 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-17 Thread Yifan Zhao
This patchset introduces inter-file multi-threaded compression. Yifan Zhao (2): erofs-utils: lib: split function logic in inode.c erofs-utils: mkfs: introduce inter-file multi-threaded compression include/erofs/compress.h | 16 ++ include/erofs/internal.h | 3 + include/erofs/list.h

[PATCH v6 5/5] erofs-utils: mkfs: introduce inner-file multi-threaded compression

2024-03-14 Thread Yifan Zhao
compresses a segment as if it were a separate file. Finally, the main thread merges all the compressed segments. Multi-threaded compression is not compatible with -Ededupe, -E(all-)fragments and -Eztailpacking for now. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h

[PATCH v6 3/5] erofs-utils: mkfs: add --worker=# parameter

2024-03-14 Thread Yifan Zhao
for multi-threading, which has the default value 16MB for now. Signed-off-by: Yifan Zhao --- include/erofs/config.h | 4 lib/config.c | 4 mkfs/main.c| 28 3 files changed, 36 insertions(+) diff --git a/include/erofs/config.h b

[PATCH v6 1/5] erofs-utils: introduce multi-threading framework

2024-03-14 Thread Yifan Zhao
Add a workqueue implementation for multi-threading support inspired by xfsprogs. Signed-off-by: Yifan Zhao Suggested-by: Gao Xiang --- configure.ac | 16 + include/erofs/internal.h | 3 + include/erofs/workqueue.h | 35 +++ lib/Makefile.am | 4 ++ lib

[PATCH v6 2/5] erofs-utils: add a helper to get available processors

2024-03-14 Thread Yifan Zhao
From: Gao Xiang In order to prepare for multi-threaded decompression. Signed-off-by: Gao Xiang --- configure.ac | 1 + include/erofs/config.h | 1 + lib/config.c | 12 3 files changed, 14 insertions(+) diff --git a/configure.ac b/configure.ac index

[PATCH v6 4/5] erofs-utils: lib: introduce atomic operations

2024-03-14 Thread Yifan Zhao
once by adding `__warnonce` atomic booleans to each erofs_compressor_init(). Cc: Yifan Zhao Signed-off-by: Gao Xiang Signed-off-by: Yifan Zhao --- include/erofs/atomic.h | 28 lib/compressor_deflate.c| 11 --- lib/compressor_libdeflate.c | 6

[PATCH v6 0/5] erofs-utils: mkfs: introduce multi-threaded compression

2024-03-14 Thread Yifan Zhao
changelog since v5: - use memory buffer instead of tmpfile to store intermediate compressed data - re-strucutre the 5th patch Gao Xiang (2): erofs-utils: add a helper to get available processors erofs-utils: lib: introduce atomic operations Yifan Zhao (3): erofs-utils: introduce multi

[PATCH v2 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-21 Thread Yifan Zhao
changelog since v1: - apply fixes from Jianan's review - remove misincluded changes in list.h Yifan Zhao (2): erofs-utils: lib: split function logic in inode.c erofs-utils: mkfs: introduce inter-file multi-threaded compression include/erofs/compress.h | 16 ++ include/erofs/internal.h

[PATCH v2 1/2] erofs-utils: lib: split function logic in inode.c

2024-03-21 Thread Yifan Zhao
This patch splits part of the logic in function erofs_mkfs_build_tree() and erofs_mkfs_build_tree_from_path() into several new functions. This is in preparation for the upcoming inter-file multi-threaded compression feature. Signed-off-by: Yifan Zhao --- lib/inode.c | 161

[PATCH v2 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-21 Thread Yifan Zhao
, the logic of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 16 ++ include/erofs/internal.h | 3 + include/erofs/queue.h| 22 +++ lib

Re: [PATCH 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-21 Thread Yifan Zhao
Thanks for the review! On 2024/3/21 2:15, Huang Jianan wrote: On 2024/3/17 22:41, Yifan Zhao wrote: This patch allows parallelizing the compression process of different files in mkfs. Specifically, a traverser thread traverses the files and issues the compression task, which is handled

Re: [PATCH 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-21 Thread Yifan Zhao
On 2024/3/21 10:07, Gao Xiang wrote: On 2024/3/21 02:15, Huang Jianan wrote: On 2024/3/17 22:41, Yifan Zhao wrote: This patch allows parallelizing the compression process of different files in mkfs. Specifically, a traverser thread traverses the files and issues the compression task

[PATCH v3 1/2] erofs-utils: lib: split function logic in inode.c

2024-03-22 Thread Yifan Zhao
This patch splits part of the logic in function erofs_mkfs_build_tree() and erofs_mkfs_build_tree_from_path() into several new functions. This is in preparation for the upcoming inter-file multi-threaded compression feature. Signed-off-by: Yifan Zhao --- lib/inode.c | 161

[PATCH v3 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-22 Thread Yifan Zhao
, the logic of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 16 ++ include/erofs/inode.h| 17 ++ include/erofs/internal.h | 3 + lib

[PATCH v3 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-22 Thread Yifan Zhao
change log since v2: - erofs_queue => erofs_inode_fifo, defined in inode.c Yifan Zhao (2): erofs-utils: lib: split function logic in inode.c erofs-utils: mkfs: introduce inter-file multi-threaded compression include/erofs/compress.h | 16 ++ include/erofs/inode.h| 17 ++ include/er

[PATCH] erofs-utils: mkfs: fix out-of-bounds memory access in mt-mkfs

2024-03-17 Thread Yifan Zhao
If a segment is smaller than the block size, sizeof(sctx->membuf) should be at least as large as the block size, as memory write into the buffer is done in block size. Signed-off-by: Yifan Zhao --- lib/compress.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/

[PATCH v5 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-25 Thread Yifan Zhao
changelog since v4: - Fix more format issues in inode.c reported by checkpatch.pl - Add missing cfg.c_chunkbits in erofs_mkfs_issue_compress Yifan Zhao (2): erofs-utils: lib: split function logic in inode.c erofs-utils: mkfs: introduce inter-file multi-threaded compression include/erofs

[PATCH v5 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-25 Thread Yifan Zhao
, the logic of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 16 ++ include/erofs/inode.h| 30 include/erofs/internal.h | 3 + lib

[PATCH v5 1/2] erofs-utils: lib: split function logic in inode.c

2024-03-25 Thread Yifan Zhao
This patch splits part of the logic in function erofs_mkfs_build_tree() and erofs_mkfs_build_tree_from_path() into several new functions. This is in preparation for the upcoming inter-file multi-threaded compression feature. Signed-off-by: Yifan Zhao --- lib/inode.c | 161

Re: [PATCH v3 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-23 Thread Yifan Zhao
On 2024/3/23 10:51, Huang Jianan wrote: Hi, Yifan, I got some warnings and errors via checkpatch.pl on this patchset. You can check and fix them since erofs-utils follows kernel coding style. Roger that. Will do it in next patches. Thanks, Yifan Zhao Thanks, Jianan Yifan Zhao 于2024年

[PATCH v4 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-23 Thread Yifan Zhao
, the logic of erofs_write_compressed_file() has been modified to split the creation and completion logic of the compression task. Signed-off-by: Yifan Zhao Co-authored-by: Tong Xin --- include/erofs/compress.h | 16 ++ include/erofs/inode.h| 30 include/erofs/internal.h | 3 + lib

[PATCH v4 1/2] erofs-utils: lib: split function logic in inode.c

2024-03-23 Thread Yifan Zhao
This patch splits part of the logic in function erofs_mkfs_build_tree() and erofs_mkfs_build_tree_from_path() into several new functions. This is in preparation for the upcoming inter-file multi-threaded compression feature. Signed-off-by: Yifan Zhao --- lib/inode.c | 161

Re: [PATCH v3 2/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-23 Thread Yifan Zhao
On 2024/3/23 11:46, Huang Jianan wrote: Yifan Zhao 于2024年3月22日周五 18:25写道: This patch allows parallelizing the compression process of different files in mkfs. Specifically, a traverser thread traverses the files and issues the compression task, which is handled by the workers. Then, the main

[PATCH v4 0/2] erofs-utils: mkfs: introduce inter-file multi-threaded compression

2024-03-23 Thread Yifan Zhao
change log since v2: - Give erofs_*_inode_fifo functions empty implementation if !EROFS_MT_ENABLED - Fix format issue reported by checkpatch.pl - rename: z_erofs_mt_queue => z_erofs_mt_inode_fifo Yifan Zhao (2): erofs-utils: lib: split function logic in inode.c erofs-utils: mkfs: introd

Re: [PATCH] erofs-utils: change temporal buffer to non static

2024-04-08 Thread Yifan Zhao
` modifier is removed in the first version of the patchset, but the change is reversed during the review. I think Xiang may share his opinion about this. Thanks, Yifan Zhao On 4/8/24 5:16 PM, Noboru Asai wrote: In multi-threaded mode, each thread must use a different buffer

[PATCH] erofs-utils: mkfs: skip the redundant write for ztailpacking block

2024-04-17 Thread Yifan Zhao
z_erofs_merge_segment() doesn't consider the ztailpacking block in the extent list and unnecessarily writes it back to the disk. This patch fixes this issue by introducing a new `inlined` field in the struct `z_erofs_inmem_extent`. Signed-off-by: Yifan Zhao --- include/erofs/dedupe.h | 2

Re: [PATCH 8/8] erofs-utils: mkfs: enable inter-file multi-threaded compression

2024-04-16 Thread Yifan Zhao
int err; + + q = malloc(sizeof(*q)); + if (!q) + return ERR_PTR(-ENOMEM); + + q->queue = malloc(q->size * sizeof(*q->queue)); q->size is not initialized here. Should move `q->size = EROFS_MT_QUEUE_SIZE;` before it. Th

[PATCH] erofs-utils: fsck: extract chunk-based file with hole correctly

2024-04-22 Thread Yifan Zhao
Currently fsck skips file extraction if it finds that EROFS_MAP_MAPPED is unset, which is not the case for chunk-based files with hole. This patch handles the corner case correctly. Signed-off-by: Yifan Zhao --- fsck/main.c | 13 + 1 file changed, 9 insertions(+), 4 deletions

[PATCH v2] erofs-utils: fsck: extract chunk-based file with hole correctly

2024-04-22 Thread Yifan Zhao
Currently fsck skips file extraction if it finds that EROFS_MAP_MAPPED is unset, which is not the case for chunk-based files with hole. This patch handles the corner case correctly. Signed-off-by: Yifan Zhao --- changelog since v1: - use lseek instead of write zero to the hole fsck/main.c | 11

Re: [PATCH 4/8] erofs-utils: rearrange several fields for multi-threaded mkfs

2024-04-16 Thread Yifan Zhao
_init(struct erofs_sb_info *sbi, struct erofs_buffer_head *s #ifdef EROFS_MT_ENABLED if (cfg.c_mt_workers > 1) { pthread_mutex_init(_erofs_mt_ctrl.mutex, NULL); Remove this line too. Thanks, Yifan Zhao - pthread_cond_init(_erofs_mt_ctrl.cond, NULL); ret = erofs_alloc_workqueue(_erofs_mt_ctrl.wq, cfg.c_mt_workers, cfg.c_mt_workers << 2,

Re: [PATCH 2/8] erofs-utils: lib: prepare for later deferred work

2024-04-16 Thread Yifan Zhao
line above does not exist in the logic replaced by `erofs_mkfs_jobfn`, should it appear in this patch, or need further explanation in the commit msg? Thanks, Yifan Zhao + return 0; + } + + if (item->type == EROFS_MKFS_JOB_DIR_BH) { + erofs_wr

Re: [PATCH] erofs-utils: mkfs: skip the redundant write for ztailpacking block

2024-04-17 Thread Yifan Zhao
is irrelevant and we don't need to care about it. For the mkfs side I think things could be polished according to Gao's review opinion. Thanks, Yifan Zhao 2024年4月17日(水) 23:43 Yifan Zhao : z_erofs_merge_segment() doesn't consider the ztailpacking block in the extent list and unnecessarily

Re: [PATCH] erofs-utils: add preliminary zstd support

2024-05-09 Thread Yifan Zhao
e based on current + * compression ratio. +*/ + m = (dstsize * m) / csize; + } else { +doesnt_fit: + /* Doesn't fit */ + r = m; +