JeonDaehong opened a new pull request, #1395:
URL: https://github.com/apache/paimon-rust/pull/1395

   ### Purpose
   
   Linked issue: part of #1336
   
   At flush, the aggregation and partial-update write buffers turn every key 
with a single version into a one-row `RecordBatch` (`singleton_merge_row`) and 
concatenate them at the end. Writing 2M new keys takes about 2.1 s with either 
engine against 0.5 s for deduplicate, which flushes with one `take`.
   
   ### Brief change log
   
   - A lone key is recorded as a row of the sorted buffer, a merged key as its 
one-row result, and the output is built once: `take` when no key merged, 
`interleave` otherwise. Rows and order are unchanged, and a lone retract still 
keeps its kind and payload.
   
   PK table, 2M new keys per write, in-memory FileIO, release build, 3 
alternating rounds against main: aggregation 2103–2205 → 622–637 ms, 
partial-update 2103–2148 → 628–643 ms, deduplicate unchanged.
   
   ### Tests
   
   - `test_flush_merge_engines_keep_lone_rows_around_merged_keys`: unsorted 
input with a merged key between lone keys and a string column, for both 
engines. It also passes on main.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to