JeonDaehong opened a new issue, #1336: URL: https://github.com/apache/paimon-rust/issues/1336
Reading a primary-key table with 2M keys written in 3 overlapping rounds takes about 1.97 s with `merge-engine=aggregation` (sum) and 1.42 s with `partial-update`, against 0.44 s for deduplicate (release build, in-memory FileIO, medians of 8 reads). Writes show the same gap since the write buffer uses the same merge. In the profile, `malloc`, `GenericByteArray::slice` and `Arc<dyn Array>` drops are near the top: `merge_ordered` and `merge_unreduced` return a one-row `RecordBatch` per key, built from `slice(row, 1)` for key columns and a one-row array from each aggregator's `result()`. Batching those one-row batches before `interleave` made no difference, so the cost is in building them. One option is letting aggregators append their result to a column builder, so a merged key adds a row to per-column builders instead of creating arrays. That touches every aggregator, so I wanted to check whether this direction makes sense before opening a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
