JeonDaehong opened a new pull request, #1440: URL: https://github.com/apache/paimon-rust/pull/1440
### Purpose Linked issue: part of #1336, follows #1420 Aggregation reads still build a one-row `RecordBatch` for every merged key: `merge_ordered` slices the key columns, adds one-row metadata arrays and wraps them with the aggregator results, and the reader then interleaves those batches. #1420 removed this for partial-update. ### Brief change log - `AggregateMergeFunction` implements `merge_picked`. A live merged row is returned as its key row plus each aggregator's result, without building a batch. The reader gathers key columns with `interleave` as for partial-update, and concatenates each aggregated column once per output batch. - Deleted rows, lone versions and other merge functions keep the existing `merge` path, as does the write buffer. PK table, 2M keys written 3 times, `sum` and `last_non_null_value`, full read, in-memory FileIO, release build. Measured in one binary with `merge_picked` switched off by an env var, over 3 alternating rounds: - every round writes both value columns: 1955–1988 → 1588–1626 ms - rounds write different columns: 1921–1936 → 1562–1579 ms What remains is mostly the aggregators' one-row results. Avoiding those would need the builder change discussed in #1336. ### Tests - `test_aggregate_picked_rows_match_materialized_rows`: three overlapping runs with `sum` and `last_non_null_value`, a key removed by DELETE under `aggregation.remove-record-on-delete`, an all-NULL sum, lone keys and batch size 2. It reads them once through `merge_picked` and once through a wrapper that only has `merge`, and checks that the batches are equal and the values are as expected. Dropping the aggregator results from the picked row fails it, along with four existing aggregation tests. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
