dongjoon-hyun commented on a change in pull request #35034: URL: https://github.com/apache/spark/pull/35034#discussion_r775665367
########## File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java ########## @@ -63,15 +59,38 @@ /** DB key where type aliases are stored. */ private static final byte[] TYPE_ALIASES_KEY = "__types__".getBytes(UTF_8); + /** + * Use full filter. + * + * https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter#full-filters-new-format + */ + private static final BloomFilter fullFilter = + new BloomFilter(10.0D /* BloomFilter.DEFAULT_BITS_PER_KEY */, false); + + /** Disable compression in index data. */ private static final BlockBasedTableConfig tableFormatConfig = new BlockBasedTableConfig() + .setFilterPolicy(fullFilter) + .setEnableIndexCompression(false) Review comment: Thank you. According to the test result, `8` could be used. Let me add it. - `.setIndexBlockRestartInterval(8)` ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.669 0.633 0.700 0.700 dbCreation 4 94.437 4.312 367.823 367.823 naturalIndexCreateIterator 1024 0.005 0.002 1.371 0.006 naturalIndexDescendingCreateIterator 1024 0.005 0.004 0.052 0.006 naturalIndexDescendingIteration 1024 0.007 0.005 0.060 0.008 naturalIndexIteration 1024 0.007 0.005 0.061 0.009 randomDeleteIndexed 1024 0.032 0.026 0.411 0.040 randomDeletesNoIndex 1024 0.021 0.017 0.138 0.022 randomUpdatesIndexed 1024 0.087 0.042 23.607 0.089 randomUpdatesNoIndex 1024 0.035 0.031 0.421 0.041 randomWritesIndexed 1024 0.123 0.054 41.442 0.128 randomWritesNoIndex 1024 0.040 0.033 1.473 0.045 refIndexCreateIterator 1024 0.003 0.003 0.021 0.005 refIndexDescendingCreateIterator 1024 0.003 0.002 0.036 0.004 refIndexDescendingIteration 1024 0.008 0.006 0.053 0.009 refIndexIteration 1024 0.009 0.006 0.288 0.011 sequentialDeleteIndexed 1024 0.026 0.022 0.116 0.031 sequentialDeleteNoIndex 1024 0.022 0.017 0.146 0.040 sequentialUpdatesIndexed 1024 0.041 0.036 0.720 0.048 sequentialUpdatesNoIndex 1024 0.039 0.034 0.590 0.046 sequentialWritesIndexed 1024 0.051 0.041 1.427 0.088 sequentialWritesNoIndex 1024 0.042 0.034 2.019 0.045 ``` - `.setIndexBlockRestartInterval(16)` ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.709 0.546 0.898 0.898 dbCreation 4 93.111 4.301 358.929 358.929 naturalIndexCreateIterator 1024 0.005 0.002 1.389 0.006 naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.063 0.007 naturalIndexDescendingIteration 1024 0.007 0.006 0.055 0.008 naturalIndexIteration 1024 0.007 0.005 0.061 0.009 randomDeleteIndexed 1024 0.033 0.024 0.214 0.049 randomDeletesNoIndex 1024 0.018 0.015 0.082 0.019 randomUpdatesIndexed 1024 0.089 0.045 23.830 0.086 randomUpdatesNoIndex 1024 0.076 0.031 0.496 0.085 randomWritesIndexed 1024 0.147 0.053 41.833 0.165 randomWritesNoIndex 1024 0.039 0.033 1.265 0.045 refIndexCreateIterator 1024 0.003 0.002 0.024 0.004 refIndexDescendingCreateIterator 1024 0.002 0.002 0.026 0.003 refIndexDescendingIteration 1024 0.008 0.006 0.050 0.009 refIndexIteration 1024 0.009 0.006 0.278 0.012 sequentialDeleteIndexed 1024 0.027 0.022 0.107 0.033 sequentialDeleteNoIndex 1024 0.019 0.016 0.068 0.020 sequentialUpdatesIndexed 1024 0.042 0.038 0.569 0.050 sequentialUpdatesNoIndex 1024 0.070 0.066 0.557 0.077 sequentialWritesIndexed 1024 0.047 0.041 1.381 0.053 sequentialWritesNoIndex 1024 0.074 0.035 1.690 0.120 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
