This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 83e3b06656f [SPARK-42129][BUILD] Upgrade rocksdbjni to 7.9.2
83e3b06656f is described below
commit 83e3b06656ff30a89717fd3650a43cd07ca12c2b
Author: yangjie01 <[email protected]>
AuthorDate: Fri Jan 20 04:35:33 2023 -0800
[SPARK-42129][BUILD] Upgrade rocksdbjni to 7.9.2
### What changes were proposed in this pull request?
This pr aims to upgrade rocksdbjni from 7.8.3 to 7.9.2.
### Why are the changes needed?
This version bring the performance related to scan iterator and bug fix
related to async io, the release notes as follows:
- https://github.com/facebook/rocksdb/releases/tag/v7.9.2
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual test `RocksDBBenchmark`:
**7.8.3**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.366 0.303
0.530 0.530
dbCreation 4 70.476 3.979
268.270 268.270
naturalIndexCreateIterator 1024 0.005 0.002
1.584 0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.005
0.069 0.007
naturalIndexDescendingIteration 1024 0.006 0.004
0.319 0.009
naturalIndexIteration 1024 0.006 0.004
0.054 0.010
randomDeleteIndexed 1024 0.026 0.019
0.364 0.036
randomDeletesNoIndex 1024 0.015 0.012
0.043 0.017
randomUpdatesIndexed 1024 0.085 0.031
32.553 0.091
randomUpdatesNoIndex 1024 0.032 0.029
0.689 0.036
randomWritesIndexed 1024 0.119 0.035
54.784 0.119
randomWritesNoIndex 1024 0.038 0.032
1.784 0.042
refIndexCreateIterator 1024 0.004 0.003
0.024 0.005
refIndexDescendingCreateIterator 1024 0.003 0.003
0.038 0.004
refIndexDescendingIteration 1024 0.007 0.005
0.048 0.009
refIndexIteration 1024 0.007 0.005
0.070 0.011
sequentialDeleteIndexed 1024 0.021 0.017
0.101 0.026
sequentialDeleteNoIndex 1024 0.015 0.012
0.050 0.019
sequentialUpdatesIndexed 1024 0.042 0.035
0.901 0.052
sequentialUpdatesNoIndex 1024 0.038 0.029
0.815 0.046
sequentialWritesIndexed 1024 0.047 0.040
2.141 0.054
sequentialWritesNoIndex 1024 0.036 0.029
2.780 0.040
```
**7.9.2**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.368 0.295
0.548 0.548
dbCreation 4 70.781 4.029
269.393 269.393
naturalIndexCreateIterator 1024 0.005 0.002
1.640 0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.004
0.068 0.006
naturalIndexDescendingIteration 1024 0.006 0.004
0.031 0.008
naturalIndexIteration 1024 0.006 0.004
0.056 0.010
randomDeleteIndexed 1024 0.026 0.019
0.208 0.037
randomDeletesNoIndex 1024 0.015 0.013
0.043 0.017
randomUpdatesIndexed 1024 0.086 0.033
30.766 0.094
randomUpdatesNoIndex 1024 0.033 0.030
0.673 0.038
randomWritesIndexed 1024 0.120 0.034
54.610 0.121
randomWritesNoIndex 1024 0.038 0.033
1.767 0.042
refIndexCreateIterator 1024 0.004 0.004
0.020 0.006
refIndexDescendingCreateIterator 1024 0.003 0.002
0.026 0.004
refIndexDescendingIteration 1024 0.007 0.005
0.047 0.009
refIndexIteration 1024 0.008 0.005
0.370 0.011
sequentialDeleteIndexed 1024 0.021 0.017
0.102 0.026
sequentialDeleteNoIndex 1024 0.015 0.012
0.046 0.019
sequentialUpdatesIndexed 1024 0.043 0.035
0.943 0.053
sequentialUpdatesNoIndex 1024 0.038 0.029
0.832 0.048
sequentialWritesIndexed 1024 0.047 0.040
1.995 0.054
sequentialWritesNoIndex 1024 0.037 0.029
2.765 0.041
```
Closes #39663 from LuciferYang/SPARK-42129.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3
b/dev/deps/spark-deps-hadoop-2-hive-2.3
index c2489c204b0..8868d8e8d2d 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -238,7 +238,7 @@ pickle/1.3//pickle-1.3.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.8.3//rocksdbjni-7.8.3.jar
+rocksdbjni/7.9.2//rocksdbjni-7.9.2.jar
scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar
scala-compiler/2.12.17//scala-compiler-2.12.17.jar
scala-library/2.12.17//scala-library-2.12.17.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 857daf3026e..fd034c054f8 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -225,7 +225,7 @@ pickle/1.3//pickle-1.3.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/7.8.3//rocksdbjni-7.8.3.jar
+rocksdbjni/7.9.2//rocksdbjni-7.9.2.jar
scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar
scala-compiler/2.12.17//scala-compiler-2.12.17.jar
scala-library/2.12.17//scala-library-2.12.17.jar
diff --git a/pom.xml b/pom.xml
index f31fc31ad41..0e01211fc83 100644
--- a/pom.xml
+++ b/pom.xml
@@ -686,7 +686,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>7.8.3</version>
+ <version>7.9.2</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
