Hi Xiang, Please find the updated multi-threaded decompression implementation for fsck.erofs. This version introduces an algorithm-aware asynchronous worker pool, dynamically sized based on system CPUs, to significantly accelerate the extraction of computationally expensive images.
Benchmarks were performed on an ARM64 environment, extracting the 8.2 GB Linux repository downloaded from Github. Extraction Time (Seconds): | Algorithm | 4k | 8k | 16k | 32k | 64k | | --- | --- | --- | --- | --- | --- | | lz4hc(MT) | 9.36 | 8.31 | 8.27 | 8.54 | 6.94 | | lz4hc(ST) | 4.40 | 5.77 | 3.65 | 5.45 | 3.36 | | zstd(MT) | 9.26 | 8.68 | 8.89 | 8.11 | 7.94 | | zstd(ST) | 5.23 | 4.52 | 4.62 | 4.11 | 4.02 | | lzma(MT) | 23.72 | 24.79 | 25.90 | 26.92 | 27.77 | | lzma(ST) | 56.37 | 65.06 | 71.07 | 74.69 | 81.63 | Performance Analysis: The implementation provides a significant speedup for LZMA (up to 2.9x) as the heavy decompression workload effectively amortizes the thread synchronization overhead. However, for fast algorithms like LZ4 and ZSTD, the current MT overhead (futex contention and scheduling) leads to slower performance compared to the synchronous baseline. I have tried various batch sizes for fast algorithms, but the time did not improve. (I tried from 32 to 256 batch sizes) Can we fall back to synchronous extraction here? Verification: * Deadlocks/Race Conditions: Verified via GDB backtrace analysis and stress testing. * Memory Leaks: Verified via rigorous buffer ownership tracking and ensuring all task resources are cleaned up upon worker completion. * Integrity: All configurations passed bit-for-bit integrity checks between the extracted and original directory for all algorithms and chunk sizes. All concurrency primitives and memory paths have been verified to the best of my knowledge. Thanks, Nithurshen
