JeonDaehong opened a new pull request, #1456:
URL: https://github.com/apache/paimon-rust/pull/1456

   ### Purpose
   
   Linked issue: none
   
   Lookup compaction looks up every changed key in the Lookup File of the older 
levels. Each `LookupFileCache::read` reads the key's SST data block from the 
local cache, checks its CRC and decompresses it again, even though the parsed 
index block and the decoded IPC payload are already kept. Compaction looks keys 
up in sorted order, so consecutive keys almost always fall in the same data 
block. On a deletion-vector table, compacting 667k updated keys against one 
2M-row file takes about 19 s, with zstd decompression at the top of the 
profile. Java's `SortLookupStoreReader` keeps decompressed blocks in its 
`BlockCache`.
   
   ### Brief change log
   
   - `LookupFileCache` keeps the decoded data block of the previous lookup, 
keyed by Lookup File path and block offset, next to the parsed index and the 
decoded IPC payload. A lookup in the same block reuses it, and any other block 
replaces it. The block is checked against the same memory budget and dropped 
with the others when a read fails.
   
   PK table with `deletion-vectors.enabled`, 2M keys compacted to one file, 
then every third key updated and compacted with lookups against that file, 
in-memory FileIO, release build. Measured in one binary with the cache switched 
off by an env var, over 3 alternating rounds: the second compaction goes from 
19268–19337 to 5480–5509 ms. Full reads are unchanged.
   
   ### Tests
   
   - `lookups_across_data_blocks_switch_the_cached_block`: 20,000 keys over 
many 8 KiB blocks, looked up ascending, descending and outside the key range. 
Every key resolves, and an ascending sweep reads far fewer blocks than keys. 
Caching by path alone, ignoring the block offset, fails it along with 
`interleaved_payloads_and_sst_blocks_support_point_lookups`.
   - `parsed_index_is_reused_and_missing_data_still_rebuilds_its_generation` 
now expects one read per key (the head), and drops the resident block before 
checking that missing data rebuilds the generation.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to