JeonDaehong opened a new pull request, #1474:
URL: https://github.com/apache/paimon-rust/pull/1474

   ### Purpose
   
   Linked issue: none
   
   With `changelog-producer = lookup`, `sort_merge_stream` yields each key's 
changelog as its own `RecordBatch` and rebuilds the changelog schema for it. 
Every layer after it then handles one-row batches: `KeyValueFileReader` 
reorders and evolves the columns per batch, and compaction writes each one to 
the changelog writer separately. Compacting 2M new keys of a primary-key table 
takes about 10 s with `changelog-producer = lookup`, against 0.46 s without a 
changelog producer.
   
   ### Brief change log
   
   - `sort_merge_stream` collects changelog rows and yields them in batches of 
up to the read batch size, under a changelog schema built once. Rows stay in 
key order, and the buffer stays bounded by the batch size, as the comment on 
`merge_with_changelog` now says.
   
   PK table, 2M keys written and compacted, then every third key updated and 
compacted again, in-memory FileIO, release build, 3 alternating rounds against 
main:
   - first compaction: 10160–10200 → 5337–5365 ms
   - second compaction: 11429–11465 → 9370–9429 ms
   
   ### Tests
   
   - `test_changelog_is_emitted_in_batches`: five keys that each produce 
changelog, with batch size 2. Changelog comes out as `[1, 2]`, `[3, 4]`, `[5]` 
and the data rows are unchanged. Yielding one batch per key again fails it.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to