Hi all, Currently, mkfs.erofs is single-threaded and the time-consuming process of creating EROFS images has emerged as a critical factor. Let's introduce multi-threaded compression to speed up the process. This patchset is based on the previous work of Gao Xiang and the multi-threading framework has been introduced in the first two patches. The following patches introduce inner-file and inter-file multi-threaded compression.
The inner-file multi-threaded compression is implemented by dividing the input file into multiple segments and compressing them in parallel. The inter-file multi-threaded compression is implemented by compressing multiple files in parallel, and we let the main thread reaps the compressed files and writes them to the output image to ensure the reproducibility of the output image. This patchset is only a basic implementation without support for dedupe, (all-)fragments and ztailpacking features. We will add support for them in the future. The performance of the multi-threaded compression is tested on a Dell r740 server with 2 Intel Xeon Gold 5215M CPUs (10 cores per CPU). We test the compression time of serveral workloads with lzma algorithm. The results are as follows: workload: enwik9 (1GB) | # of threads | Time (s) | Speedup | |--------------|----------|---------| | 1 | 216.08 | 1.00x | | 2 | 110.15 | 1.96x | | 4 | 57.22 | 3.78x | | 8 | 31.4 | 6.88x | | 16 | 16.99 | 12.72x | workload: Linux-6.7-rc4 source (1.55GB) | # of threads | Time (s) | Speedup | |--------------|----------|---------| | 1 | 288.65 | 1.00x | | 2 | 140.66 | 2.05x | | 4 | 79.64 | 3.62x | | 8 | 54.74 | 5.27x | | 16 | 47.08 | 6.13x | workload: Arch container with Xfce desktop (4.19GB) | # of threads | Time (s) | Speedup | |--------------|----------|---------| | 1 | 834.26 | 1.00x | | 2 | 430.73 | 1.94x | | 4 | 267.96 | 3.11x | | 8 | 205.67 | 4.06x | | 16 | 183.26 | 4.55x | Gao Xiang (2): erofs-utils: introduce multi-threading framework erofs-utils: add a helper to get available processors Yifan Zhao (5): erofs-utils: mkfs: add --worker=# parameter erofs-utils: mkfs: optionally print warning in erofs_compressor_init erofs-utils: extend multi-threading framework for per-thread fields erofs-utils: mkfs: introduce inner-file multi-threaded compression erofs-utils: mkfs: introduce inter-file multi-threaded compression configure.ac | 17 + include/erofs/compress.h | 18 + include/erofs/config.h | 5 + include/erofs/internal.h | 6 + include/erofs/list.h | 8 + include/erofs/queue.h | 22 + include/erofs/workqueue.h | 48 +++ lib/Makefile.am | 4 + lib/compress.c | 819 +++++++++++++++++++++++++++++------- lib/compressor.c | 7 +- lib/compressor.h | 5 +- lib/compressor_deflate.c | 10 +- lib/compressor_libdeflate.c | 6 +- lib/compressor_liblzma.c | 9 +- lib/compressor_lz4.c | 2 +- lib/compressor_lz4hc.c | 2 +- lib/config.c | 16 + lib/inode.c | 346 ++++++++++++--- lib/queue.c | 64 +++ lib/workqueue.c | 222 ++++++++++ mkfs/main.c | 38 ++ 21 files changed, 1442 insertions(+), 232 deletions(-) create mode 100644 include/erofs/queue.h create mode 100644 include/erofs/workqueue.h create mode 100644 lib/queue.c create mode 100644 lib/workqueue.c -- 2.43.0
