ranflarion opened a new pull request, #27576:
URL: https://github.com/apache/datafusion/pull/27576

   ## Which issue does this PR close?
   
   - Closes #27575.
   
   ## Rationale for this change
   
   A partial aggregate that skips aggregation calls `convert_to_state` on every 
batch, and for `sum` `PrimitiveGroupsAccumulator` allocates a zeroed array per 
batch and reads it before writing it. That is an extra pass with any allocator 
and 34 page faults per call with jemalloc. Details and numbers per allocator 
are in the issue.
   
   ## What changes are included in this PR?
   
   `PrimitiveGroupsAccumulator::convert_to_state` computes 
`prim_fn(starting_value, value)` per row with `compute::unary` instead of 
`PrimitiveArray::from_value` followed by `compute::binary_mut`. The filter 
handling is unchanged, and the error for a shared buffer goes away with the 
scratch array it guarded.
   
   `cargo bench -p datafusion-functions-aggregate --bench sum`, the five `sum 
i64 convert state` cases, per 8,192-row batch:
   
   | | main | this PR | change |
   |---|---|---|---|
   | Apple M4 Pro, macOS system allocator | 2.68 to 2.85 µs | 0.80 to 0.98 µs | 
-66% to -71% |
   | AMD EPYC, Linux, glibc | 2.33 to 2.62 µs | 1.04 to 1.25 µs | -52% to -56% |
   
   ## What is the testing strategy for this PR?
   
   Existing tests cover it. `aggregate_skip_partial.slt` sets both probe 
thresholds to 0, so its `sum`, `min` and `max` queries, with nulls and 
`FILTER`, run through this function on every batch (dropping the null mask here 
fails six of them), and the aggregation fuzzer enables skip-partial at random 
and compares against the non-skipping result.
   
   ## Are there any user-facing changes?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to