This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0774ba09daf9 [SPARK-55803][BUILD] Bump lz4-java 1.10.4 to bring back
performance
0774ba09daf9 is described below
commit 0774ba09daf9aa41364849cfbb9096dad968aa7b
Author: Cheng Pan <[email protected]>
AuthorDate: Tue Mar 3 11:14:34 2026 +0800
[SPARK-55803][BUILD] Bump lz4-java 1.10.4 to bring back performance
### What changes were proposed in this pull request?
Upgrade lz4-java 1.10.4 to bring performance back.
### Why are the changes needed?
https://github.com/yawkat/lz4-java/releases/tag/v1.10.4
> These changes attempt to fix the native performance regression in 1.9+.
They should have no functional or security impact.
### Does this PR introduce _any_ user-facing change?
Performance regression has gone.
### How was this patch tested?
Benchmark reports are updated
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54585 from pan3793/SPARK-55803.
Lead-authored-by: Cheng Pan <[email protected]>
Co-authored-by: pan3793 <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt | 4 ++--
core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt | 4 ++--
core/benchmarks/LZ4TPCDSDataBenchmark-results.txt | 4 ++--
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
5 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
b/core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
index a066ced506b5..031c00ec96bd 100644
--- a/core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
+++ b/core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux
6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Compression 4 times 2611 2619
11 0.0 652760335.3 1.0X
+Compression 4 times 2376 2382
9 0.0 593993111.8 1.0X
OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Decompression 4 times 896 912
20 0.0 224050201.0 1.0X
+Decompression 4 times 731 757
29 0.0 182787977.2 1.0X
diff --git a/core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
b/core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
index 0a03fd3e12f1..05898aa539e8 100644
--- a/core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
+++ b/core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux
6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Compression 4 times 2611 2620
12 0.0 652865216.0 1.0X
+Compression 4 times 2373 2380
10 0.0 593284150.8 1.0X
OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Decompression 4 times 905 916
14 0.0 226166738.0 1.0X
+Decompression 4 times 694 710
23 0.0 173453930.5 1.0X
diff --git a/core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
b/core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
index 8d909dfc7f49..94d8848325e0 100644
--- a/core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
+++ b/core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux
6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Compression 4 times 2602 2609
10 0.0 650438397.0 1.0X
+Compression 4 times 2376 2383
10 0.0 593917596.8 1.0X
OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-Decompression 4 times 938 966
33 0.0 234424499.5 1.0X
+Decompression 4 times 759 761
2 0.0 189753707.8 1.0X
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index ee5d39fdd696..efc079f35cdf 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -187,7 +187,7 @@ log4j-api/2.25.3//log4j-api-2.25.3.jar
log4j-core/2.25.3//log4j-core-2.25.3.jar
log4j-layout-template-json/2.25.3//log4j-layout-template-json-2.25.3.jar
log4j-slf4j2-impl/2.25.3//log4j-slf4j2-impl-2.25.3.jar
-lz4-java/1.10.3//lz4-java-1.10.3.jar
+lz4-java/1.10.4//lz4-java-1.10.4.jar
metrics-core/4.2.37//metrics-core-4.2.37.jar
metrics-graphite/4.2.37//metrics-graphite-4.2.37.jar
metrics-jmx/4.2.37//metrics-jmx-4.2.37.jar
diff --git a/pom.xml b/pom.xml
index 82c9e196968d..d38bc0a9fb5b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -888,7 +888,7 @@
<dependency>
<groupId>at.yawk.lz4</groupId>
<artifactId>lz4-java</artifactId>
- <version>1.10.3</version>
+ <version>1.10.4</version>
</dependency>
<dependency>
<groupId>com.github.luben</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
