JingsongLi opened a new pull request, #8857: URL: https://github.com/apache/paimon/pull/8857
## What changed - enable dynamically sized per-SST Bloom filters for Simple LSM flush, bulk-load, and compaction output - reduce the default local KV block size from 32 KB to 4 KB and expose `local-kv-db.block-size` - use Caffeine for the primary-key clustering local KV cache while preserving Guava elsewhere - reduce hot-path allocations by caching cache-key hash codes and avoiding entry materialization during block seeks - add focused correctness tests and a configurable Simple LSM versus RocksDB benchmark ## Why Random point lookups in a local KV database larger than its cache were dominated by block read amplification and cache lookup overhead. Missing-key lookups also visited SST files unnecessarily when no Bloom filter was available. ## Impact With 3 million records, 1 million operations, a 128 MB cache, 4 KB blocks, LZ4, and Bloom-filter FPP 0.1: | Workload | Simple LSM | RocksDB | | --- | ---: | ---: | | point lookup hit (best) | 873 K ops/s | 523 K ops/s | | point lookup miss (best) | 6002 K ops/s | 2773 K ops/s | | 50/50 mixed read/write (best) | 482 K ops/s | 356 K ops/s | | database size | 214.55 MB | 445.34 MB | The mixed workload still shows periodic compaction latency, so its average throughput remains roughly level with RocksDB. ## Validation - `SimpleLsmKvDbTest`: 51 tests passed - `CoreOptionsTest`, `MemoryFileStoreWriteTest`, and `ClusteringTableTest`: 60 tests passed - strict compile and test-compile passed for the 15-module micro-benchmark reactor - Simple LSM versus RocksDB point-lookup and mixed read/write benchmarks passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
