pan3793 commented on PR #53453: URL: https://github.com/apache/spark/pull/53453#issuecomment-3645530618
TL;DR - my test results show lz4-java 1.10.1 is about 10~15% slower on lz4 compression than 1.8.0, and is about ~5% slower on lz4 decompression even with migrating to suggested safeDecompressor (#53453) Here is my local test result: lz4-java 1.8.0 with fastDecompressor (state before aa65bda31b955fa9c7e5232e776ec99704799ccc) ``` [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Compression 4 times [info] Stopped after 2 iterations, 4143 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Compression 4 times 2064 2072 11 0.0 515881622.3 1.0X [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Decompression 4 times [info] Stopped after 3 iterations, 2659 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Decompression 4 times 843 886 45 0.0 210840272.0 1.0X ``` lz4-java 1.10.1 with fastDecompressor (current master state) ``` [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Compression 4 times [info] Stopped after 2 iterations, 4740 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Compression 4 times 2351 2370 27 0.0 587850958.3 1.0X [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Decompression 4 times [info] Stopped after 2 iterations, 4147 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Decompression 4 times 2070 2074 5 0.0 517477273.3 1.0X ``` lz4-java 1.10.1 with safeDecompressor (#53453) ``` [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Compression 4 times [info] Stopped after 2 iterations, 4729 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Compression 4 times 2360 2365 7 0.0 589977281.7 1.0X [info] Running benchmark: Benchmark LZ4CompressionCodec [info] Running case: Decompression 4 times [info] Stopped after 3 iterations, 2742 ms [info] OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.17.9-76061709-generic [info] Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz [info] Benchmark LZ4CompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------ [info] Decompression 4 times 886 914 43 0.0 221537120.3 1.0X ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
