JeonDaehong opened a new pull request, #1474: URL: https://github.com/apache/paimon-rust/pull/1474
### Purpose Linked issue: none With `changelog-producer = lookup`, `sort_merge_stream` yields each key's changelog as its own `RecordBatch` and rebuilds the changelog schema for it. Every layer after it then handles one-row batches: `KeyValueFileReader` reorders and evolves the columns per batch, and compaction writes each one to the changelog writer separately. Compacting 2M new keys of a primary-key table takes about 10 s with `changelog-producer = lookup`, against 0.46 s without a changelog producer. ### Brief change log - `sort_merge_stream` collects changelog rows and yields them in batches of up to the read batch size, under a changelog schema built once. Rows stay in key order, and the buffer stays bounded by the batch size, as the comment on `merge_with_changelog` now says. PK table, 2M keys written and compacted, then every third key updated and compacted again, in-memory FileIO, release build, 3 alternating rounds against main: - first compaction: 10160–10200 → 5337–5365 ms - second compaction: 11429–11465 → 9370–9429 ms ### Tests - `test_changelog_is_emitted_in_batches`: five keys that each produce changelog, with batch size 2. Changelog comes out as `[1, 2]`, `[3, 4]`, `[5]` and the data rows are unchanged. Yielding one batch per key again fails it. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
