This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3cac0df1f5f8 [SPARK-45327][BUILD] Upgrade zstd-jni to 1.5.5-7
3cac0df1f5f8 is described below
commit 3cac0df1f5f818e5e4722a435fb83c44ca155883
Author: panbingkun <[email protected]>
AuthorDate: Wed Nov 1 23:45:52 2023 +0800
[SPARK-45327][BUILD] Upgrade zstd-jni to 1.5.5-7
### What changes were proposed in this pull request?
The pr aims to upgrade zstd-jni from 1.5.5-5 to 1.5.5-7.
### Why are the changes needed?
1.Version compare:
- v1.5.5-6 VS v1.5.5-7:
https://github.com/luben/zstd-jni/compare/v1.5.5-6...v1.5.5-7
- v1.5.5-5 VS v1.5.5-6:
https://github.com/luben/zstd-jni/compare/v.1.5.5-5...v1.5.5-6
2.Some changes include the following:
- Add new method `getFrameContentSize` that will return also the error.
https://github.com/luben/zstd-jni/commit/3f6c55e87a38255bdbeeca3e773f99bf9e8a7b7f
- Fix error.
https://github.com/luben/zstd-jni/commit/e0c66f1230377d4980ad6abc0767b36c860af538
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43113 from panbingkun/SPARK-45327.
Authored-by: panbingkun <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.../ZStandardBenchmark-jdk21-results.txt | 32 +++++++++++-----------
core/benchmarks/ZStandardBenchmark-results.txt | 28 +++++++++----------
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 6 +++-
4 files changed, 36 insertions(+), 32 deletions(-)
diff --git a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
index 0915e7338568..19bd2faa9a5a 100644
--- a/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
+++ b/core/benchmarks/ZStandardBenchmark-jdk21-results.txt
@@ -2,26 +2,26 @@
Benchmark ZStandardCompressionCodec
================================================================================================
-OpenJDK 64-Bit Server VM 21+35 on Linux 5.15.0-1046-azure
-Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
+OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1050-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
-Compression 10000 times at level 1 without buffer pool 2649
2653 6 0.0 264919.3 1.0X
-Compression 10000 times at level 2 without buffer pool 2785
2785 1 0.0 278455.9 1.0X
-Compression 10000 times at level 3 without buffer pool 3078
3082 6 0.0 307845.7 0.9X
-Compression 10000 times at level 1 with buffer pool 2353
2378 35 0.0 235340.6 1.1X
-Compression 10000 times at level 2 with buffer pool 2462
2466 6 0.0 246194.4 1.1X
-Compression 10000 times at level 3 with buffer pool 2761
2765 6 0.0 276095.6 1.0X
+Compression 10000 times at level 1 without buffer pool 1817
1819 3 0.0 181709.6 1.0X
+Compression 10000 times at level 2 without buffer pool 2081
2083 3 0.0 208053.7 0.9X
+Compression 10000 times at level 3 without buffer pool 2288
2290 3 0.0 228795.4 0.8X
+Compression 10000 times at level 1 with buffer pool 1997
1998 1 0.0 199686.9 0.9X
+Compression 10000 times at level 2 with buffer pool 2062
2063 1 0.0 206209.3 0.9X
+Compression 10000 times at level 3 with buffer pool 2243
2243 1 0.0 224271.8 0.8X
-OpenJDK 64-Bit Server VM 21+35 on Linux 5.15.0-1046-azure
-Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
+OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1050-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Benchmark ZStandardCompressionCodec: Best Time(ms)
Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
-Decompression 10000 times from level 1 without buffer pool 2644
2644 0 0.0 264430.7 1.0X
-Decompression 10000 times from level 2 without buffer pool 2597
2608 15 0.0 259715.2 1.0X
-Decompression 10000 times from level 3 without buffer pool 2586
2600 20 0.0 258633.7 1.0X
-Decompression 10000 times from level 1 with buffer pool 2290
2296 9 0.0 228957.6 1.2X
-Decompression 10000 times from level 2 with buffer pool 2315
2319 6 0.0 231535.4 1.1X
-Decompression 10000 times from level 3 with buffer pool 2283
2302 27 0.0 228308.3 1.2X
+Decompression 10000 times from level 1 without buffer pool 1980
1981 1 0.0 197970.8 1.0X
+Decompression 10000 times from level 2 without buffer pool 1978
1979 1 0.0 197813.4 1.0X
+Decompression 10000 times from level 3 without buffer pool 1981
1983 2 0.0 198141.7 1.0X
+Decompression 10000 times from level 1 with buffer pool 1825
1827 3 0.0 182475.2 1.1X
+Decompression 10000 times from level 2 with buffer pool 1827
1827 0 0.0 182667.1 1.1X
+Decompression 10000 times from level 3 with buffer pool 1826
1826 0 0.0 182579.8 1.1X
diff --git a/core/benchmarks/ZStandardBenchmark-results.txt
b/core/benchmarks/ZStandardBenchmark-results.txt
index 5299a52dc7b8..6f7efdd6c94c 100644
--- a/core/benchmarks/ZStandardBenchmark-results.txt
+++ b/core/benchmarks/ZStandardBenchmark-results.txt
@@ -2,26 +2,26 @@
Benchmark ZStandardCompressionCodec
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.8+7-LTS on Linux 5.15.0-1046-azure
+OpenJDK 64-Bit Server VM 17.0.9+8-LTS on Linux 5.15.0-1050-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
-Compression 10000 times at level 1 without buffer pool 2800
2801 2 0.0 279995.2 1.0X
-Compression 10000 times at level 2 without buffer pool 2832
2846 19 0.0 283227.4 1.0X
-Compression 10000 times at level 3 without buffer pool 2978
3003 35 0.0 297782.6 0.9X
-Compression 10000 times at level 1 with buffer pool 2650
2652 2 0.0 265042.4 1.1X
-Compression 10000 times at level 2 with buffer pool 2684
2688 5 0.0 268419.4 1.0X
-Compression 10000 times at level 3 with buffer pool 2811
2816 8 0.0 281069.3 1.0X
+Compression 10000 times at level 1 without buffer pool 2786
2787 2 0.0 278560.1 1.0X
+Compression 10000 times at level 2 without buffer pool 2831
2833 3 0.0 283091.0 1.0X
+Compression 10000 times at level 3 without buffer pool 2958
2959 2 0.0 295806.3 0.9X
+Compression 10000 times at level 1 with buffer pool 211
214 4 0.0 21145.3 13.2X
+Compression 10000 times at level 2 with buffer pool 253
255 1 0.0 25328.1 11.0X
+Compression 10000 times at level 3 with buffer pool 370
371 1 0.0 37046.1 7.5X
-OpenJDK 64-Bit Server VM 17.0.8+7-LTS on Linux 5.15.0-1046-azure
+OpenJDK 64-Bit Server VM 17.0.9+8-LTS on Linux 5.15.0-1050-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Benchmark ZStandardCompressionCodec: Best Time(ms)
Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
-Decompression 10000 times from level 1 without buffer pool 2747
2752 7 0.0 274669.2 1.0X
-Decompression 10000 times from level 2 without buffer pool 2743
2748 8 0.0 274265.3 1.0X
-Decompression 10000 times from level 3 without buffer pool 2743
2750 10 0.0 274344.4 1.0X
-Decompression 10000 times from level 1 with buffer pool 2608
2608 0 0.0 260803.0 1.1X
-Decompression 10000 times from level 2 with buffer pool 2608
2608 0 0.0 260804.2 1.1X
-Decompression 10000 times from level 3 with buffer pool 2605
2607 3 0.0 260514.9 1.1X
+Decompression 10000 times from level 1 without buffer pool 2745
2748 5 0.0 274454.0 1.0X
+Decompression 10000 times from level 2 without buffer pool 2744
2745 1 0.0 274438.1 1.0X
+Decompression 10000 times from level 3 without buffer pool 2746
2746 1 0.0 274586.0 1.0X
+Decompression 10000 times from level 1 with buffer pool 2587
2588 1 0.0 258707.4 1.1X
+Decompression 10000 times from level 2 with buffer pool 2586
2586 1 0.0 258566.8 1.1X
+Decompression 10000 times from level 3 with buffer pool 2589
2589 0 0.0 258870.6 1.1X
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index b1c1ed8d81da..6364ec48fb66 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -264,4 +264,4 @@ xz/1.9//xz-1.9.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.6.3//zookeeper-jute-3.6.3.jar
zookeeper/3.6.3//zookeeper-3.6.3.jar
-zstd-jni/1.5.5-5//zstd-jni-1.5.5-5.jar
+zstd-jni/1.5.5-7//zstd-jni-1.5.5-7.jar
diff --git a/pom.xml b/pom.xml
index 811d075e203a..2e0c95516c17 100644
--- a/pom.xml
+++ b/pom.xml
@@ -799,7 +799,7 @@
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
- <version>1.5.5-5</version>
+ <version>1.5.5-7</version>
</dependency>
<dependency>
<groupId>com.clearspring.analytics</groupId>
@@ -2662,6 +2662,10 @@
<groupId>javax.annotation</groupId>
<artifactId>javax.annotation-api</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>com.github.luben</groupId>
+ <artifactId>zstd-jni</artifactId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
