This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 9987d17 [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3 9987d17 is described below commit 9987d1756615e22dd71184545409f321b18b66c3 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Mon Mar 28 09:14:04 2022 -0700 [SPARK-38257][BUILD] Upgrade `rockdbjni` to 7.0.3 ### What changes were proposed in this pull request? The main change of this pr is upgrade `rockdbjni` from 6.20.3 to 7.0.3. ### Why are the changes needed? `rockdbjni` 7.x is a Java 8 compiled version, while `rockdbjni` 6.x is still a Java 7 compiled version, At the same time `rockdbjni` 7.x supports MacOS universal binary (ARM+x86): - https://github.com/facebook/rocksdb/releases/tag/v6.22.1 - https://github.com/facebook/rocksdb/releases/tag/v6.23.3 - https://github.com/facebook/rocksdb/releases/tag/v6.24.2 - https://github.com/facebook/rocksdb/releases/tag/v6.25.1 - https://github.com/facebook/rocksdb/releases/tag/v6.25.3 - https://github.com/facebook/rocksdb/releases/tag/v6.26.0 - https://github.com/facebook/rocksdb/releases/tag/v6.26.1 - https://github.com/facebook/rocksdb/releases/tag/v6.27.3 - https://github.com/facebook/rocksdb/releases/tag/v6.28.2 - https://github.com/facebook/rocksdb/releases/tag/v6.29.3 - https://github.com/facebook/rocksdb/releases/tag/v6.29.4 - https://github.com/facebook/rocksdb/releases/tag/v7.0.1 - https://github.com/facebook/rocksdb/releases/tag/v7.0.2 - https://github.com/facebook/rocksdb/releases/tag/v7.0.3 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - Manual performance test using `org.apache.spark.util.kvstore.RocksDBBenchmark` environment ``` OS: CentOS release 6.3 (Final) Kernel: 4.14 CPU: Intel(R) Xeon(R) Gold 2.60GHz ``` **Before** 6.20.3 ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.275 0.174 0.484 0.484 dbCreation 4 74.612 3.318 289.818 289.818 naturalIndexCreateIterator 1024 0.005 0.002 1.470 0.007 naturalIndexDescendingCreateIterator 1024 0.004 0.003 0.069 0.006 naturalIndexDescendingIteration 1024 0.006 0.004 0.039 0.009 naturalIndexIteration 1024 0.006 0.004 0.052 0.010 randomDeleteIndexed 1024 0.025 0.019 0.228 0.034 randomDeletesNoIndex 1024 0.015 0.012 0.037 0.017 randomUpdatesIndexed 1024 0.091 0.039 30.743 0.101 randomUpdatesNoIndex 1024 0.063 0.059 0.492 0.068 randomWritesIndexed 1024 0.117 0.033 52.323 0.121 randomWritesNoIndex 1024 0.068 0.022 1.346 0.074 refIndexCreateIterator 1024 0.004 0.004 0.017 0.006 refIndexDescendingCreateIterator 1024 0.003 0.003 0.027 0.004 refIndexDescendingIteration 1024 0.006 0.005 0.047 0.008 refIndexIteration 1024 0.008 0.005 0.279 0.011 sequentialDeleteIndexed 1024 0.020 0.017 0.125 0.025 sequentialDeleteNoIndex 1024 0.014 0.012 0.040 0.017 sequentialUpdatesIndexed 1024 0.077 0.065 1.297 0.101 sequentialUpdatesNoIndex 1024 0.069 0.049 0.722 0.084 sequentialWritesIndexed 1024 0.076 0.059 2.006 0.087 sequentialWritesNoIndex 1024 0.063 0.025 2.529 0.074 ``` **After** 7.0.3 ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.332 0.235 0.557 0.557 dbCreation 4 75.354 3.301 290.771 290.771 naturalIndexCreateIterator 1024 0.005 0.002 1.418 0.007 naturalIndexDescendingCreateIterator 1024 0.006 0.006 0.068 0.008 naturalIndexDescendingIteration 1024 0.006 0.004 0.261 0.007 naturalIndexIteration 1024 0.006 0.004 0.061 0.009 randomDeleteIndexed 1024 0.026 0.020 0.231 0.035 randomDeletesNoIndex 1024 0.014 0.012 0.040 0.017 randomUpdatesIndexed 1024 0.081 0.032 31.135 0.085 randomUpdatesNoIndex 1024 0.037 0.033 0.528 0.042 randomWritesIndexed 1024 0.119 0.034 54.771 0.122 randomWritesNoIndex 1024 0.042 0.036 1.448 0.048 refIndexCreateIterator 1024 0.005 0.004 0.020 0.006 refIndexDescendingCreateIterator 1024 0.003 0.003 0.029 0.005 refIndexDescendingIteration 1024 0.006 0.005 0.046 0.008 refIndexIteration 1024 0.007 0.005 0.072 0.009 sequentialDeleteIndexed 1024 0.021 0.017 0.100 0.026 sequentialDeleteNoIndex 1024 0.014 0.012 0.039 0.017 sequentialUpdatesIndexed 1024 0.047 0.040 0.747 0.058 sequentialUpdatesNoIndex 1024 0.040 0.029 0.730 0.049 sequentialWritesIndexed 1024 0.050 0.044 1.842 0.057 sequentialWritesNoIndex 1024 0.041 0.031 2.213 0.044 ``` Additional 7.0.3 on Apple silicon/M1 ``` OS: Big Sur 11.4 Kernel: 20.5.0 CPU: Apple M1 ``` ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.445 0.400 0.500 0.500 dbCreation 4 359.993 169.192 886.616 886.616 naturalIndexCreateIterator 1024 0.003 0.001 0.902 0.003 naturalIndexDescendingCreateIterator 1024 0.003 0.003 0.052 0.003 naturalIndexDescendingIteration 1024 0.004 0.003 0.085 0.010 naturalIndexIteration 1024 0.004 0.003 0.175 0.005 randomDeleteIndexed 1024 0.020 0.012 0.335 0.053 randomDeletesNoIndex 1024 0.011 0.009 0.034 0.013 randomUpdatesIndexed 1024 0.043 0.020 14.436 0.040 randomUpdatesNoIndex 1024 0.022 0.015 0.480 0.037 randomWritesIndexed 1024 0.091 0.022 50.383 0.088 randomWritesNoIndex 1024 0.042 0.021 7.483 0.057 refIndexCreateIterator 1024 0.002 0.002 0.317 0.002 refIndexDescendingCreateIterator 1024 0.003 0.001 0.624 0.002 refIndexDescendingIteration 1024 0.009 0.003 4.596 0.010 refIndexIteration 1024 0.009 0.003 1.080 0.009 sequentialDeleteIndexed 1024 0.015 0.013 0.046 0.018 sequentialDeleteNoIndex 1024 0.010 0.009 0.237 0.011 sequentialUpdatesIndexed 1024 0.026 0.020 0.622 0.032 sequentialUpdatesNoIndex 1024 0.022 0.017 0.586 0.026 sequentialWritesIndexed 1024 0.048 0.027 3.646 0.063 sequentialWritesNoIndex 1024 0.035 0.023 2.910 0.043 ``` Closes #35580 from LuciferYang/SPARK-38257. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- .../scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3 index cd1af5e..82cb644 100644 --- a/dev/deps/spark-deps-hadoop-2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-2-hive-2.3 @@ -235,7 +235,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.5//py4j-0.10.9.5.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/6.20.3//rocksdbjni-6.20.3.jar +rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar scala-compiler/2.12.15//scala-compiler-2.12.15.jar scala-library/2.12.15//scala-library-2.12.15.jar diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 7752853..27b0ec5 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -223,7 +223,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.5//py4j-0.10.9.5.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/6.20.3//rocksdbjni-6.20.3.jar +rocksdbjni/7.0.3//rocksdbjni-7.0.3.jar scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar scala-compiler/2.12.15//scala-compiler-2.12.15.jar scala-library/2.12.15//scala-library-2.12.15.jar diff --git a/pom.xml b/pom.xml index 82eda7d..6c5b321 100644 --- a/pom.xml +++ b/pom.xml @@ -663,7 +663,7 @@ <dependency> <groupId>org.rocksdb</groupId> <artifactId>rocksdbjni</artifactId> - <version>6.20.3</version> + <version>7.0.3</version> </dependency> <dependency> <groupId>${leveldbjni.group}</groupId> diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala index a5bd489..048db5b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala @@ -177,7 +177,7 @@ class RocksDB( numKeysOnWritingVersion -= 1 } } - writeBatch.remove(key) + writeBatch.delete(key) } /** --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org