Discussions on LZ4 compression algorithm & open source implementations https://lz4.org/
LZ4 1.10 Lossless Compression Algorithm Released LZ4 1.10 compression algorithm released with multithreading, dramatically improving compression speeds by up to 8 times. LZ4, a widely used cross-platform open-source compression algorithm known for its speed and efficiency, has just reached version 1.10. By Bobby Borisov July 22, 2024 https://linuxiac.com/lz4-1-10-lossless-compression-algorithm-released/ This latest update introduces significant enhancements, particularly multithreading support, which leverages modern multi-core processors to accelerate compression and decompression tasks in high-throughput environments. For those who might not know, LZ4 is especially favored when speed is prioritized over compression ratio, such as streaming or real-time applications. What’s New in LZ4 1.10 Until now, LZ4 has operated primarily as a single-threaded process. The 1.10 release, however, completely changes this. LZ4 Multithreading: A Game Changer The headline feature of LZ4 1.10 is its multithreading capability. By allowing multiple threads to process data simultaneously, LZ4 significantly reduces the time it takes to compress and decompress large files. This is particularly beneficial when using high compression modes, which are generally more CPU-intensive. The latest benchmarks vividly illustrate the improvements across different operating systems: On Windows 11, using an AMD 7840HS CPU, compression time has improved from 13.4 seconds to just 1.8 seconds—a 7.4 times speed increase. macOS users with the M1 Pro chip will see a reduction from 16.6 seconds to 2.55 seconds, a 6.5 times faster performance. For Linux users on an i7-9700k, the compression time has been reduced from 16.2 seconds to 3.05 seconds, achieving a 5.4 times speed boost. These improvements are not just limited to compression. Decompression also benefits from multithreading, albeit to a lesser extent, as modern nvme drives can still be saturated with a single thread. However, the overlapping of I/O operations with decompression processes further enhances performance. Enhanced Dictionary Compression Another significant upgrade in LZ4 1.10 is the official support for dictionary compression and decompression, which has transitioned from experimental to fully supported. This allows developers to use LZ4 more reliably in applications that benefit from dictionary techniques, such as small data transfers where initialization can be a bottleneck. New Compression Level and Other Improvements LZ4 1.10 introduces a new “Level 2” compression option, which balances performance and compression ratio, providing an optimal solution for many applications. Moreover, the update includes numerous enhancements, such as better platform support, improved error handling, and more flexible configuration via environment variables. Lastly, the release supports lesser-known architectures such as LoongArch, RISC-V, and others, ensuring LZ4’s portability across various platforms. For an in-depth look at all novelties in LZ4 1.10, refer to the full changelog. Bobby Borisov ..an editor-in-chief at Linuxiac, is a Linux professional with over 20 years of experience. With a strong focus on Linux and open-source software, he has worked as a Senior Linux System Administrator, Software Developer, and DevOps Engineer for small and large multinational companies. _______________________________________________ Link mailing list [email protected] https://mailman.anu.edu.au/mailman/listinfo/link
