andygrove opened a new issue, #339:
URL: https://github.com/apache/datafusion-spark/issues/339

   ### Is your feature request related to a problem or challenge?
   
   `SparkCollectList` and `SparkCollectSet` 
(`crates/spark/src/function/aggregate/collect.rs`) implement only 
`accumulator()`, so grouped aggregation falls back to 
`GroupsAccumulatorAdapter`: one boxed `Accumulator` per group. At high grouping 
cardinality this dominates. On 2M rows grouped by a string key with 200k 
distinct values, `collect_list` ran at 0.49x of Spark's JVM implementation, 
while `count(*)` with the same grouping ran at 2.88x.
   
   Comet works around this with its own `GroupsAccumulator`s 
(apache/datafusion-comet#5803), which were 85–99% faster than the adapter at 
16k groups.
   
   Moved from apache/datafusion#25118, which keeps the `array_agg` accumulator 
work.
   
   ### Describe the solution you'd like
   
   - `SparkCollectList`: implement `create_groups_accumulator` by reusing 
`ArrayAggGroupsAccumulator` with `ignore_nulls = true`, turning the lists it 
leaves null into empty lists (Spark returns `[]`, not `NULL`, for a group whose 
inputs were all NULL).
   - `SparkCollectSet`: implement a grouped accumulator. 
apache/datafusion#26357 added a `GroupsAccumulator` for `array_agg(DISTINCT)` 
that may be reusable here.
   
   Comet's implementations in `native/spark-expr/src/agg_funcs/collect.rs` 
(apache/datafusion-comet#5803) could be upstreamed more or less as they are.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to