This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a3f84acd7f3 [SPARK-39124][BUILD] Upgrade `rocksdbjni` to 7.1.2
a3f84acd7f3 is described below
commit a3f84acd7f38f748be06ee53f8d1e84b9101e1e6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun May 8 16:01:07 2022 -0700
[SPARK-39124][BUILD] Upgrade `rocksdbjni` to 7.1.2
### What changes were proposed in this pull request?
This PR aims to upgrade RocksDB JNI library from 7.0.3 to 7.1.2.
### Why are the changes needed?
This will bring improvements. Note that 7.2.0 JNI is not released yet.
- https://github.com/facebook/rocksdb/releases/tag/v7.1.1
- https://github.com/facebook/rocksdb/releases/tag/v7.1.2
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
The benchmark result is here.
**Before (7.0.3)**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.500 0.356
0.856 0.856
dbCreation 4 433.785 195.266
1159.003 1159.003
naturalIndexCreateIterator 1024 0.019 0.002
6.221 0.055
naturalIndexDescendingCreateIterator 1024 0.009 0.006
1.259 0.008
naturalIndexDescendingIteration 1024 0.021 0.005
1.786 0.097
naturalIndexIteration 1024 0.017 0.006
0.765 0.033
randomDeleteIndexed 1024 0.050 0.025
3.664 0.079
randomDeletesNoIndex 1024 0.019 0.016
0.118 0.021
randomUpdatesIndexed 1024 0.118 0.035
26.061 0.357
randomUpdatesNoIndex 1024 0.031 0.026
0.799 0.033
randomWritesIndexed 1024 0.169 0.041
59.242 0.346
randomWritesNoIndex 1024 0.055 0.032
9.539 0.052
refIndexCreateIterator 1024 0.009 0.007
0.200 0.010
refIndexDescendingCreateIterator 1024 0.008 0.003
0.669 0.016
refIndexDescendingIteration 1024 0.019 0.006
0.432 0.061
refIndexIteration 1024 0.018 0.007
1.134 0.044
sequentialDeleteIndexed 1024 0.039 0.023
1.110 0.038
sequentialDeleteNoIndex 1024 0.019 0.016
0.382 0.021
sequentialUpdatesIndexed 1024 0.063 0.032
2.218 0.102
sequentialUpdatesNoIndex 1024 0.045 0.029
1.737 0.064
sequentialWritesIndexed 1024 0.075 0.037
16.837 0.115
sequentialWritesNoIndex 1024 0.065 0.031
16.540 0.055
```
**After (7.1.2)**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.891 0.380
1.286 1.286
dbCreation 4 411.247 195.950
1058.678 1058.678
naturalIndexCreateIterator 1024 0.017 0.002
6.792 0.011
naturalIndexDescendingCreateIterator 1024 0.011 0.008
0.882 0.010
naturalIndexDescendingIteration 1024 0.014 0.005
1.541 0.009
naturalIndexIteration 1024 0.020 0.006
1.347 0.036
randomDeleteIndexed 1024 0.075 0.026
4.560 0.256
randomDeletesNoIndex 1024 0.020 0.017
0.054 0.022
randomUpdatesIndexed 1024 0.117 0.036
24.286 0.363
randomUpdatesNoIndex 1024 0.034 0.026
2.126 0.033
randomWritesIndexed 1024 0.168 0.041
59.470 0.371
randomWritesNoIndex 1024 0.057 0.030
11.681 0.049
refIndexCreateIterator 1024 0.006 0.004
0.201 0.006
refIndexDescendingCreateIterator 1024 0.005 0.003
0.407 0.004
refIndexDescendingIteration 1024 0.015 0.006
1.498 0.017
refIndexIteration 1024 0.020 0.007
0.993 0.021
sequentialDeleteIndexed 1024 0.035 0.023
1.605 0.038
sequentialDeleteNoIndex 1024 0.025 0.017
1.706 0.023
sequentialUpdatesIndexed 1024 0.058 0.031
2.765 0.051
sequentialUpdatesNoIndex 1024 0.052 0.031
3.404 0.058
sequentialWritesIndexed 1024 0.075 0.035
17.668 0.093
sequentialWritesNoIndex 1024 0.065 0.031
16.220 0.054
```
Closes #36480 from dongjoon-hyun/SPARK-39124.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3
b/dev/deps/spark-deps-hadoop-2-hive-2.3
index b6df3ea5ce2..935340a890e 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
+rocksdbjni/7.1.2//rocksdbjni-7.1.2.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
scala-library/2.12.15//scala-library-2.12.15.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index ad0700db8c8..e8c7a25281b 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -224,7 +224,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
+rocksdbjni/7.1.2//rocksdbjni-7.1.2.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
scala-library/2.12.15//scala-library-2.12.15.jar
diff --git a/pom.xml b/pom.xml
index 67efb7f7cb0..b808f37bdae 100644
--- a/pom.xml
+++ b/pom.xml
@@ -675,7 +675,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.0.3</version>
+ <version>7.1.2</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
