JeonDaehong opened a new pull request, #1440:
URL: https://github.com/apache/paimon-rust/pull/1440

   ### Purpose
   
   Linked issue: part of #1336, follows #1420
   
   Aggregation reads still build a one-row `RecordBatch` for every merged key: 
`merge_ordered` slices the key columns, adds one-row metadata arrays and wraps 
them with the aggregator results, and the reader then interleaves those 
batches. #1420 removed this for partial-update.
   
   ### Brief change log
   
   - `AggregateMergeFunction` implements `merge_picked`. A live merged row is 
returned as its key row plus each aggregator's result, without building a 
batch. The reader gathers key columns with `interleave` as for partial-update, 
and concatenates each aggregated column once per output batch.
   - Deleted rows, lone versions and other merge functions keep the existing 
`merge` path, as does the write buffer.
   
   PK table, 2M keys written 3 times, `sum` and `last_non_null_value`, full 
read, in-memory FileIO, release build. Measured in one binary with 
`merge_picked` switched off by an env var, over 3 alternating rounds:
   - every round writes both value columns: 1955–1988 → 1588–1626 ms
   - rounds write different columns: 1921–1936 → 1562–1579 ms
   
   What remains is mostly the aggregators' one-row results. Avoiding those 
would need the builder change discussed in #1336.
   
   ### Tests
   
   - `test_aggregate_picked_rows_match_materialized_rows`: three overlapping 
runs with `sum` and `last_non_null_value`, a key removed by DELETE under 
`aggregation.remove-record-on-delete`, an all-NULL sum, lone keys and batch 
size 2. It reads them once through `merge_picked` and once through a wrapper 
that only has `merge`, and checks that the batches are equal and the values are 
as expected. Dropping the aggregator results from the picked row fails it, 
along with four existing aggregation tests.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to