This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9987d17 [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
9987d17 is described below
commit 9987d1756615e22dd71184545409f321b18b66c3
Author: yangjie01 <[email protected]>
AuthorDate: Mon Mar 28 09:14:04 2022 -0700
[SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3
### What changes were proposed in this pull request?
The main change of this pr is upgrade `rockdbjni` from 6.20.3 to 7.0.3.
### Why are the changes needed?
`rockdbjni` 7.x is a Java 8 compiled version, while `rockdbjni` 6.x is
still a Java 7 compiled version, At the same time `rockdbjni` 7.x supports
MacOS universal binary (ARM+x86):
- https://github.com/facebook/rocksdb/releases/tag/v6.22.1
- https://github.com/facebook/rocksdb/releases/tag/v6.23.3
- https://github.com/facebook/rocksdb/releases/tag/v6.24.2
- https://github.com/facebook/rocksdb/releases/tag/v6.25.1
- https://github.com/facebook/rocksdb/releases/tag/v6.25.3
- https://github.com/facebook/rocksdb/releases/tag/v6.26.0
- https://github.com/facebook/rocksdb/releases/tag/v6.26.1
- https://github.com/facebook/rocksdb/releases/tag/v6.27.3
- https://github.com/facebook/rocksdb/releases/tag/v6.28.2
- https://github.com/facebook/rocksdb/releases/tag/v6.29.3
- https://github.com/facebook/rocksdb/releases/tag/v6.29.4
- https://github.com/facebook/rocksdb/releases/tag/v7.0.1
- https://github.com/facebook/rocksdb/releases/tag/v7.0.2
- https://github.com/facebook/rocksdb/releases/tag/v7.0.3
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual performance test using
`org.apache.spark.util.kvstore.RocksDBBenchmark`
environment
```
OS: CentOS release 6.3 (Final)
Kernel: 4.14
CPU: Intel(R) Xeon(R) Gold 2.60GHz
```
**Before**
6.20.3
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.275 0.174 0.484
0.484
dbCreation 4 74.612 3.318 289.818
289.818
naturalIndexCreateIterator 1024 0.005 0.002 1.470
0.007
naturalIndexDescendingCreateIterator 1024 0.004 0.003 0.069
0.006
naturalIndexDescendingIteration 1024 0.006 0.004 0.039
0.009
naturalIndexIteration 1024 0.006 0.004 0.052
0.010
randomDeleteIndexed 1024 0.025 0.019 0.228
0.034
randomDeletesNoIndex 1024 0.015 0.012 0.037
0.017
randomUpdatesIndexed 1024 0.091 0.039 30.743
0.101
randomUpdatesNoIndex 1024 0.063 0.059 0.492
0.068
randomWritesIndexed 1024 0.117 0.033 52.323
0.121
randomWritesNoIndex 1024 0.068 0.022 1.346
0.074
refIndexCreateIterator 1024 0.004 0.004 0.017
0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.027
0.004
refIndexDescendingIteration 1024 0.006 0.005 0.047
0.008
refIndexIteration 1024 0.008 0.005 0.279
0.011
sequentialDeleteIndexed 1024 0.020 0.017 0.125
0.025
sequentialDeleteNoIndex 1024 0.014 0.012 0.040
0.017
sequentialUpdatesIndexed 1024 0.077 0.065 1.297
0.101
sequentialUpdatesNoIndex 1024 0.069 0.049 0.722
0.084
sequentialWritesIndexed 1024 0.076 0.059 2.006
0.087
sequentialWritesNoIndex 1024 0.063 0.025 2.529
0.074
```
**After**
7.0.3
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.332 0.235 0.557
0.557
dbCreation 4 75.354 3.301 290.771
290.771
naturalIndexCreateIterator 1024 0.005 0.002 1.418
0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.006 0.068
0.008
naturalIndexDescendingIteration 1024 0.006 0.004 0.261
0.007
naturalIndexIteration 1024 0.006 0.004 0.061
0.009
randomDeleteIndexed 1024 0.026 0.020 0.231
0.035
randomDeletesNoIndex 1024 0.014 0.012 0.040
0.017
randomUpdatesIndexed 1024 0.081 0.032 31.135
0.085
randomUpdatesNoIndex 1024 0.037 0.033 0.528
0.042
randomWritesIndexed 1024 0.119 0.034 54.771
0.122
randomWritesNoIndex 1024 0.042 0.036 1.448
0.048
refIndexCreateIterator 1024 0.005 0.004 0.020
0.006
refIndexDescendingCreateIterator 1024 0.003 0.003 0.029
0.005
refIndexDescendingIteration 1024 0.006 0.005 0.046
0.008
refIndexIteration 1024 0.007 0.005 0.072
0.009
sequentialDeleteIndexed 1024 0.021 0.017 0.100
0.026
sequentialDeleteNoIndex 1024 0.014 0.012 0.039
0.017
sequentialUpdatesIndexed 1024 0.047 0.040 0.747
0.058
sequentialUpdatesNoIndex 1024 0.040 0.029 0.730
0.049
sequentialWritesIndexed 1024 0.050 0.044 1.842
0.057
sequentialWritesNoIndex 1024 0.041 0.031 2.213
0.044
```
Additional 7.0.3 on Apple silicon/M1
```
OS: Big Sur 11.4
Kernel: 20.5.0
CPU: Apple M1
```
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.445 0.400 0.500
0.500
dbCreation 4 359.993 169.192 886.616
886.616
naturalIndexCreateIterator 1024 0.003 0.001 0.902
0.003
naturalIndexDescendingCreateIterator 1024 0.003 0.003 0.052
0.003
naturalIndexDescendingIteration 1024 0.004 0.003 0.085
0.010
naturalIndexIteration 1024 0.004 0.003 0.175
0.005
randomDeleteIndexed 1024 0.020 0.012 0.335
0.053
randomDeletesNoIndex 1024 0.011 0.009 0.034
0.013
randomUpdatesIndexed 1024 0.043 0.020 14.436
0.040
randomUpdatesNoIndex 1024 0.022 0.015 0.480
0.037
randomWritesIndexed 1024 0.091 0.022 50.383
0.088
randomWritesNoIndex 1024 0.042 0.021 7.483
0.057
refIndexCreateIterator 1024 0.002 0.002 0.317
0.002
refIndexDescendingCreateIterator 1024 0.003 0.001 0.624
0.002
refIndexDescendingIteration 1024 0.009 0.003 4.596
0.010
refIndexIteration 1024 0.009 0.003 1.080
0.009
sequentialDeleteIndexed 1024 0.015 0.013 0.046
0.018
sequentialDeleteNoIndex 1024 0.010 0.009 0.237
0.011
sequentialUpdatesIndexed 1024 0.026 0.020 0.622
0.032
sequentialUpdatesNoIndex 1024 0.022 0.017 0.586
0.026
sequentialWritesIndexed 1024 0.048 0.027 3.646
0.063
sequentialWritesNoIndex 1024 0.035 0.023 2.910
0.043
```
Closes #35580 from LuciferYang/SPARK-38257.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +-
pom.xml | 2 +-
.../scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3
b/dev/deps/spark-deps-hadoop-2-hive-2.3
index cd1af5e..82cb644 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
@@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/6.20.3//rocksdbjni-6.20.3.jar
+rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
scala-library/2.12.15//scala-library-2.12.15.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 7752853..27b0ec5 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -223,7 +223,7 @@ pickle/1.2//pickle-1.2.jar
protobuf-java/2.5.0//protobuf-java-2.5.0.jar
py4j/0.10.9.5//py4j-0.10.9.5.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
-rocksdbjni/6.20.3//rocksdbjni-6.20.3.jar
+rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar
scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
scala-compiler/2.12.15//scala-compiler-2.12.15.jar
scala-library/2.12.15//scala-library-2.12.15.jar
diff --git a/pom.xml b/pom.xml
index 82eda7d..6c5b321 100644
--- a/pom.xml
+++ b/pom.xml
@@ -663,7 +663,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
- <version>6.20.3</version>
+ <version>7.0.3</version>
</dependency>
<dependency>
<groupId>${leveldbjni.group}</groupId>
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index a5bd489..048db5b 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -177,7 +177,7 @@ class RocksDB(
numKeysOnWritingVersion -= 1
}
}
- writeBatch.remove(key)
+ writeBatch.delete(key)
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
