dongjoon-hyun opened a new pull request #35034:
URL: https://github.com/apache/spark/pull/35034
### What changes were proposed in this pull request?
This PR aims to optimize RocksDB KVStore configuration.
### Why are the changes needed?
Currently, `RocksDB` KVStore performance is not optimized.
```
[INFO] Running org.apache.spark.util.kvstore.LevelDBBenchmark
count mean min max
95th
dbClose 4 0.802 0.777 0.824
0.824
dbCreation 4 71.029 2.528 278.049
278.049
naturalIndexCreateIterator 1024 0.006 0.002 1.559
0.007
naturalIndexDescendingCreateIterator 1024 0.003 0.003 0.043
0.004
naturalIndexDescendingIteration 1024 0.007 0.005 0.053
0.008
naturalIndexIteration 1024 0.007 0.005 0.080
0.009
randomDeleteIndexed 1024 0.020 0.017 0.148
0.024
randomDeletesNoIndex 1024 0.025 0.008 14.739
0.013
randomUpdatesIndexed 1024 0.079 0.033 23.527
0.075
randomUpdatesNoIndex 1024 0.078 0.072 0.584
0.082
randomWritesIndexed 1024 0.121 0.043 42.030
0.141
randomWritesNoIndex 1024 0.086 0.059 1.204
0.090
refIndexCreateIterator 1024 0.003 0.002 0.033
0.004
refIndexDescendingCreateIterator 1024 0.002 0.002 0.018
0.003
refIndexDescendingIteration 1024 0.007 0.006 0.045
0.008
refIndexIteration 1024 0.009 0.006 0.261
0.010
sequentialDeleteIndexed 1024 0.019 0.016 0.090
0.024
sequentialDeleteNoIndex 1024 0.013 0.010 0.083
0.014
sequentialUpdatesIndexed 1024 0.031 0.027 0.560
0.035
sequentialUpdatesNoIndex 1024 0.028 0.024 0.618
0.032
sequentialWritesIndexed 1024 0.037 0.031 1.301
0.041
sequentialWritesNoIndex 1024 0.038 0.026 1.699
0.065
```
**AS-IS**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.621 0.455 0.830
0.830
dbCreation 4 93.812 4.566 365.368
365.368
naturalIndexCreateIterator 1024 0.005 0.002 1.349
0.007
naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.054
0.007
naturalIndexDescendingIteration 1024 0.007 0.005 0.034
0.009
naturalIndexIteration 1024 0.007 0.005 0.062
0.009
randomDeleteIndexed 1024 0.138 0.079 0.636
0.186
randomDeletesNoIndex 1024 0.104 0.058 0.223
0.143
randomUpdatesIndexed 1024 0.203 0.145 23.849
0.243
randomUpdatesNoIndex 1024 0.161 0.122 0.509
0.196
randomWritesIndexed 1024 0.239 0.149 42.780
0.279
randomWritesNoIndex 1024 0.143 0.080 1.593
0.221
refIndexCreateIterator 1024 0.003 0.003 0.032
0.005
refIndexDescendingCreateIterator 1024 0.003 0.003 0.028
0.005
refIndexDescendingIteration 1024 0.008 0.006 0.038
0.009
refIndexIteration 1024 0.009 0.007 0.265
0.011
sequentialDeleteIndexed 1024 0.122 0.106 0.251
0.150
sequentialDeleteNoIndex 1024 0.101 0.059 0.315
0.121
sequentialUpdatesIndexed 1024 0.147 0.109 0.812
0.197
sequentialUpdatesNoIndex 1024 0.143 0.112 0.896
0.184
sequentialWritesIndexed 1024 0.170 0.130 1.715
0.235
sequentialWritesNoIndex 1024 0.140 0.097 2.607
0.163
```
**AFTER THIS PR**
```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
count mean min max
95th
dbClose 4 0.673 0.472 0.921
0.921
dbCreation 4 97.226 4.324 376.853
376.853
naturalIndexCreateIterator 1024 0.005 0.002 1.364
0.006
naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.054
0.006
naturalIndexDescendingIteration 1024 0.007 0.005 0.039
0.008
naturalIndexIteration 1024 0.008 0.006 0.061
0.009
randomDeleteIndexed 1024 0.031 0.025 0.224
0.039
randomDeletesNoIndex 1024 0.019 0.016 0.096
0.021
randomUpdatesIndexed 1024 0.091 0.045 24.200
0.092
randomUpdatesNoIndex 1024 0.051 0.031 0.509
0.102
randomWritesIndexed 1024 0.134 0.055 41.833
0.144
randomWritesNoIndex 1024 0.047 0.034 1.077
0.085
refIndexCreateIterator 1024 0.003 0.003 0.019
0.005
refIndexDescendingCreateIterator 1024 0.002 0.002 0.017
0.003
refIndexDescendingIteration 1024 0.007 0.006 0.035
0.008
refIndexIteration 1024 0.009 0.006 0.270
0.010
sequentialDeleteIndexed 1024 0.026 0.022 0.098
0.034
sequentialDeleteNoIndex 1024 0.022 0.016 0.165
0.040
sequentialUpdatesIndexed 1024 0.041 0.037 0.663
0.047
sequentialUpdatesNoIndex 1024 0.037 0.033 0.571
0.043
sequentialWritesIndexed 1024 0.046 0.040 1.249
0.052
sequentialWritesNoIndex 1024 0.044 0.034 1.694
0.098
```
### Does this PR introduce _any_ user-facing change?
No because `RocksDB` KVStore backend is a new feature.
### How was this patch tested?
Manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
