This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 285d8dd [SPARK-37755][CORE] Optimize RocksDB KVStore configurations
285d8dd is described below
commit 285d8dd9f0b8ee8b85603b4cc0f656c452bff629
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Dec 27 17:30:22 2021 -0800
[SPARK-37755][CORE] Optimize RocksDB KVStore configurations
### What changes were proposed in this pull request?
This PR aims to optimize RocksDB KVStore configurations to improve the
performance up to 4 times.
### Why are the changes needed?
Currently, `RocksDB` KVStore performance is not optimized.
**REFERENCE (LevelDB) **
```
[INFO] Running org.apache.spark.util.kvstore.LevelDBBenchmark
count mean min max
95th
dbClose 4 0.802 0.777 0.824
0.824
dbCreation 4 71.029 2.528 278.049
278.049
naturalIndexCreateIterator 1024 0.006 0.002 1.559
0.007
naturalIndexDescendingCreateIterator 1024 0.003 0.003 0.043
0.004
naturalIndexDescendingIteration 1024 0.007 0.005 0.053
0.008
naturalIndexIteration 1024 0.007 0.005 0.080
0.009
randomDeleteIndexed 1024 0.020 0.017 0.148
0.024
randomDeletesNoIndex 1024 0.025 0.008 14.739
0.013
randomUpdatesIndexed 1024 0.079 0.033 23.527
0.075
randomUpdatesNoIndex 1024 0.078 0.072 0.584
0.082
randomWritesIndexed 1024 0.121 0.043 42.030
0.141
randomWritesNoIndex 1024 0.086 0.059 1.204
0.090
refIndexCreateIterator 1024 0.003 0.002 0.033
0.004
refIndexDescendingCreateIterator 1024 0.002 0.002 0.018
0.003
refIndexDescendingIteration 1024 0.007 0.006 0.045
0.008
refIndexIteration 1024 0.009 0.006 0.261
0.010
sequentialDeleteIndexed 1024 0.019 0.016 0.090
0.024
sequentialDeleteNoIndex 1024 0.013 0.010 0.083
0.014
sequentialUpdatesIndexed 1024 0.031 0.027 0.560
0.035
sequentialUpdatesNoIndex 1024 0.028 0.024 0.618
0.032
sequentialWritesIndexed 1024 0.037 0.031 1.301
0.041
sequentialWritesNoIndex 1024 0.038 0.026 1.699
0.065
```
**AS-IS (RocksDB)**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.621 0.455 0.830
0.830
dbCreation 4 93.812 4.566 365.368
365.368
naturalIndexCreateIterator 1024 0.005 0.002 1.349
0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.054
0.007
naturalIndexDescendingIteration 1024 0.007 0.005 0.034
0.009
naturalIndexIteration 1024 0.007 0.005 0.062
0.009
randomDeleteIndexed 1024 0.138 0.079 0.636
0.186
randomDeletesNoIndex 1024 0.104 0.058 0.223
0.143
randomUpdatesIndexed 1024 0.203 0.145 23.849
0.243
randomUpdatesNoIndex 1024 0.161 0.122 0.509
0.196
randomWritesIndexed 1024 0.239 0.149 42.780
0.279
randomWritesNoIndex 1024 0.143 0.080 1.593
0.221
refIndexCreateIterator 1024 0.003 0.003 0.032
0.005
refIndexDescendingCreateIterator 1024 0.003 0.003 0.028
0.005
refIndexDescendingIteration 1024 0.008 0.006 0.038
0.009
refIndexIteration 1024 0.009 0.007 0.265
0.011
sequentialDeleteIndexed 1024 0.122 0.106 0.251
0.150
sequentialDeleteNoIndex 1024 0.101 0.059 0.315
0.121
sequentialUpdatesIndexed 1024 0.147 0.109 0.812
0.197
sequentialUpdatesNoIndex 1024 0.143 0.112 0.896
0.184
sequentialWritesIndexed 1024 0.170 0.130 1.715
0.235
sequentialWritesNoIndex 1024 0.140 0.097 2.607
0.163
```
**AFTER THIS PR (RocksDB)**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.669 0.633 0.700
0.700
dbCreation 4 94.437 4.312 367.823
367.823
naturalIndexCreateIterator 1024 0.005 0.002 1.371
0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.004 0.052
0.006
naturalIndexDescendingIteration 1024 0.007 0.005 0.060
0.008
naturalIndexIteration 1024 0.007 0.005 0.061
0.009
randomDeleteIndexed 1024 0.032 0.026 0.411
0.040
randomDeletesNoIndex 1024 0.021 0.017 0.138
0.022
randomUpdatesIndexed 1024 0.087 0.042 23.607
0.089
randomUpdatesNoIndex 1024 0.035 0.031 0.421
0.041
randomWritesIndexed 1024 0.123 0.054 41.442
0.128
randomWritesNoIndex 1024 0.040 0.033 1.473
0.045
refIndexCreateIterator 1024 0.003 0.003 0.021
0.005
refIndexDescendingCreateIterator 1024 0.003 0.002 0.036
0.004
refIndexDescendingIteration 1024 0.008 0.006 0.053
0.009
refIndexIteration 1024 0.009 0.006 0.288
0.011
sequentialDeleteIndexed 1024 0.026 0.022 0.116
0.031
sequentialDeleteNoIndex 1024 0.022 0.017 0.146
0.040
sequentialUpdatesIndexed 1024 0.041 0.036 0.720
0.048
sequentialUpdatesNoIndex 1024 0.039 0.034 0.590
0.046
sequentialWritesIndexed 1024 0.051 0.041 1.427
0.088
sequentialWritesNoIndex 1024 0.042 0.034 2.019
0.045
```
### Does this PR introduce _any_ user-facing change?
No because `RocksDB` KVStore backend is a new feature.
### How was this patch tested?
Manually.
Closes #35034 from dongjoon-hyun/SPARK-37755.
Lead-authored-by: Dongjoon Hyun <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/util/kvstore/RocksDB.java | 36 +++++++++++++++++-----
1 file changed, 28 insertions(+), 8 deletions(-)
diff --git
a/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
b/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
index ccc7c5f..7674bc5 100644
--- a/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
+++ b/common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java
@@ -33,11 +33,7 @@ import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.base.Throwables;
-import org.rocksdb.BlockBasedTableConfig;
-import org.rocksdb.Options;
-import org.rocksdb.Statistics;
-import org.rocksdb.WriteBatch;
-import org.rocksdb.WriteOptions;
+import org.rocksdb.*;
import org.apache.spark.annotation.Private;
@@ -63,15 +59,39 @@ public class RocksDB implements KVStore {
/** DB key where type aliases are stored. */
private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8);
+ /**
+ * Use full filter.
+ *
+ *
https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format
+ */
+ private static final BloomFilter fullFilter =
+ new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false);
+
+ /** Disable compression in index data. */
private static final BlockBasedTableConfig tableFormatConfig = new
BlockBasedTableConfig()
+ .setFilterPolicy(fullFilter)
+ .setEnableIndexCompression(false)
+ .setIndexBlockRestartInterval(8)
.setFormatVersion(5);
+ /**
+ * - Use ZSTD at the bottom most level to reduce the disk space
+ * - Use LZ4 at the other levels because it's better than Snappy in general.
+ *
+ * https://github.com/facebook/rocksdb/wiki/Compression#configuration
+ */
private static final Options dbOptions = new Options()
.setCreateIfMissing(true)
- .setTableFormatConfig(tableFormatConfig)
- .setStatistics(new Statistics());
+ .setBottommostCompressionType(CompressionType.ZSTD_COMPRESSION)
+ .setCompressionType(CompressionType.LZ4_COMPRESSION)
+ .setTableFormatConfig(tableFormatConfig);
- private static final WriteOptions writeOptions = new
WriteOptions().setSync(true);
+ /**
+ * - Use explicitly 'sync = false' like LevelDB KVStore implementation.
+ *
+ * https://github.com/google/leveldb/blob/1.23/include/leveldb/options.h#L182
+ */
+ private static final WriteOptions writeOptions = new
WriteOptions().setSync(false);
private final AtomicReference<org.rocksdb.RocksDB> _db;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
