JingsongLi opened a new pull request, #8881: URL: https://github.com/apache/paimon/pull/8881
### Purpose Replace the RocksDB-backed state used by full lookup caches and the cross-partition global index with `LocalKvStateFactory`. This lets these paths use the LocalKvDb capabilities added by the preceding PRs, while preserving the existing `StateFactory` contracts and TTL behavior. ### Changes - use `LocalKvStateFactory` for Flink full lookup caches and the cross-partition global index - use Flink's shared async operations pool for cross-partition LocalKvDb compaction - reuse SST readers for range scans while bounding the cache to one reader per LSM level - batch incremental ListState values in the MemTable, with at most 32 values per fragment - add LocalKvDb versus RocksDB state benchmarks and regression coverage ### Performance Local benchmark with ZSTD, 4 KB blocks, off-heap cache, and 10,000 keys with 64 values per key. The table shows the median of three runs: | Incremental ListState | LocalKvDb | RocksDB | | --- | ---: | ---: | | add | 237.7 ms | 420.4 ms | | first get | 8.6 ms | 23.1 ms | | cached get | 2.7 ms | 2.9 ms | | directory size | 0.35 MB | 1.85 MB | Before MemTable batching, the LocalKvDb first get took about 73.2 ms. The bounded batching reduces this to 8.6 ms without making repeated appends quadratic. Initial ListState bulk load remains a separate optimization opportunity: LocalKvDb first get is about 23.8 ms versus 12.9 ms for RocksDB in this benchmark. ### Validation - `LocalKvDbTest` - `LocalKvStateFactoryTest` - `GlobalIndexAssignerTest` - `LookupTableTest` with the Flink 2 profile - non-fast-build compile and formatting checks for `paimon-common` and `paimon-core` - LocalKvDb versus RocksDB state fan-out benchmarks with ZSTD -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
