JeonDaehong opened a new pull request, #1456: URL: https://github.com/apache/paimon-rust/pull/1456
### Purpose Linked issue: none Lookup compaction looks up every changed key in the Lookup File of the older levels. Each `LookupFileCache::read` reads the key's SST data block from the local cache, checks its CRC and decompresses it again, even though the parsed index block and the decoded IPC payload are already kept. Compaction looks keys up in sorted order, so consecutive keys almost always fall in the same data block. On a deletion-vector table, compacting 667k updated keys against one 2M-row file takes about 19 s, with zstd decompression at the top of the profile. Java's `SortLookupStoreReader` keeps decompressed blocks in its `BlockCache`. ### Brief change log - `LookupFileCache` keeps the decoded data block of the previous lookup, keyed by Lookup File path and block offset, next to the parsed index and the decoded IPC payload. A lookup in the same block reuses it, and any other block replaces it. The block is checked against the same memory budget and dropped with the others when a read fails. PK table with `deletion-vectors.enabled`, 2M keys compacted to one file, then every third key updated and compacted with lookups against that file, in-memory FileIO, release build. Measured in one binary with the cache switched off by an env var, over 3 alternating rounds: the second compaction goes from 19268–19337 to 5480–5509 ms. Full reads are unchanged. ### Tests - `lookups_across_data_blocks_switch_the_cached_block`: 20,000 keys over many 8 KiB blocks, looked up ascending, descending and outside the key range. Every key resolves, and an ascending sweep reads far fewer blocks than keys. Caching by path alone, ignoring the block offset, fails it along with `interleaved_payloads_and_sst_blocks_support_point_lookups`. - `parsed_index_is_reused_and_missing_data_still_rebuilds_its_generation` now expects one read per key (the head), and drops the resident block before checking that missing data rebuilds the generation. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
