JeonDaehong opened a new pull request, #1480:
URL: https://github.com/apache/paimon-rust/pull/1480

   ### Purpose
   
   Linked issue: none, follows #1474
   
   With `changelog-producer = lookup`, every changelog row is built with 
`take`, which copies all of its columns into a new one-row batch, and a key 
with several changes is concatenated once more. Since #1474 the reader 
concatenates changelog rows into output batches anyway, so these copies are not 
needed.
   
   ### Brief change log
   
   - `source_row` returns a one-row slice of the source batch instead of a 
`take` copy.
   - A key with a single change returns it as is, without `concat_batches`.
   
   The slices are dropped when the reader concatenates the pending changelog, 
at most once per `batch_size` rows.
   
   PK table, `changelog-producer = lookup`, compaction time, in-memory FileIO, 
release build. Measured in one binary with `take` switched back on by an env 
var, over 3 alternating rounds:
   - 2M keys, first compaction (every key emits a changelog row): 4158–4183 → 
2789–2800 ms
   - 2M keys, a third of them updated, second compaction: 9010–9017 → 8175–8244 
ms
   - 64 more BIGINT columns, 500k keys, first compaction: 12947–13257 → 
8333–8429 ms
   - deletion vectors without a changelog producer, which does not use this 
path: 433–454 vs 433 ms, 5176–5345 vs 5170–5193 ms
   
   ### Tests
   
   No new test. The lookup changelog tests in 
`tests/pk_compact/lookup_changelog.rs` already check single `+I` changes and 
`-U`/`+U` pairs across engines.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to