JeonDaehong opened a new issue, #1336:
URL: https://github.com/apache/paimon-rust/issues/1336

   Reading a primary-key table with 2M keys written in 3 overlapping rounds 
takes about 1.97 s with `merge-engine=aggregation` (sum) and 1.42 s with 
`partial-update`, against 0.44 s for deduplicate (release build, in-memory 
FileIO, medians of 8 reads). Writes show the same gap since the write buffer 
uses the same merge. In the profile, `malloc`, `GenericByteArray::slice` and 
`Arc<dyn Array>` drops are near the top: `merge_ordered` and `merge_unreduced` 
return a one-row `RecordBatch` per key, built from `slice(row, 1)` for key 
columns and a one-row array from each aggregator's `result()`.
   
   Batching those one-row batches before `interleave` made no difference, so 
the cost is in building them. One option is letting aggregators append their 
result to a column builder, so a merged key adds a row to per-column builders 
instead of creating arrays. That touches every aggregator, so I wanted to check 
whether this direction makes sense before opening a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to