JeonDaehong opened a new pull request, #1395: URL: https://github.com/apache/paimon-rust/pull/1395
### Purpose Linked issue: part of #1336 At flush, the aggregation and partial-update write buffers turn every key with a single version into a one-row `RecordBatch` (`singleton_merge_row`) and concatenate them at the end. Writing 2M new keys takes about 2.1 s with either engine against 0.5 s for deduplicate, which flushes with one `take`. ### Brief change log - A lone key is recorded as a row of the sorted buffer, a merged key as its one-row result, and the output is built once: `take` when no key merged, `interleave` otherwise. Rows and order are unchanged, and a lone retract still keeps its kind and payload. PK table, 2M new keys per write, in-memory FileIO, release build, 3 alternating rounds against main: aggregation 2103–2205 → 622–637 ms, partial-update 2103–2148 → 628–643 ms, deduplicate unchanged. ### Tests - `test_flush_merge_engines_keep_lone_rows_around_merged_keys`: unsorted input with a merged key between lone keys and a string column, for both engines. It also passes on main. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
