JeonDaehong opened a new pull request, #1480: URL: https://github.com/apache/paimon-rust/pull/1480
### Purpose Linked issue: none, follows #1474 With `changelog-producer = lookup`, every changelog row is built with `take`, which copies all of its columns into a new one-row batch, and a key with several changes is concatenated once more. Since #1474 the reader concatenates changelog rows into output batches anyway, so these copies are not needed. ### Brief change log - `source_row` returns a one-row slice of the source batch instead of a `take` copy. - A key with a single change returns it as is, without `concat_batches`. The slices are dropped when the reader concatenates the pending changelog, at most once per `batch_size` rows. PK table, `changelog-producer = lookup`, compaction time, in-memory FileIO, release build. Measured in one binary with `take` switched back on by an env var, over 3 alternating rounds: - 2M keys, first compaction (every key emits a changelog row): 4158–4183 → 2789–2800 ms - 2M keys, a third of them updated, second compaction: 9010–9017 → 8175–8244 ms - 64 more BIGINT columns, 500k keys, first compaction: 12947–13257 → 8333–8429 ms - deletion vectors without a changelog producer, which does not use this path: 433–454 vs 433 ms, 5176–5345 vs 5170–5193 ms ### Tests No new test. The lookup changelog tests in `tests/pk_compact/lookup_changelog.rs` already check single `+I` changes and `-U`/`+U` pairs across engines. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
