AnuragRaut08 opened a new pull request, #348: URL: https://github.com/apache/datafusion-spark/pull/348
## Which issue does this PR close? Closes #339 ## Rationale for this change Spark-compatible `collect_list` and `collect_set` currently use the scalar accumulator implementation without native grouped accumulator support. As a result, grouped aggregation falls back to DataFusion's generic `GroupsAccumulatorAdapter`, which can introduce significant overhead, particularly when aggregating high-cardinality data with many groups. This change provides native grouped accumulator implementations while preserving Spark's expected behavior: - `collect_list` retains duplicate values. - `collect_set` removes duplicate values. - NULL input values are ignored. - Groups with no retained values, including groups containing only NULL values or rows excluded by a `FILTER` clause, return an empty list (`[]`) rather than `NULL`. ## What changes are included in this PR? - Implement `groups_accumulator_supported()` and `create_groups_accumulator()` for both `SparkCollectList` and `SparkCollectSet`. - Reuse DataFusion's native grouped `array_agg` implementation, configuring NULL handling and distinct semantics to match Spark. - Add a grouped accumulator wrapper that converts NULL evaluation results for empty groups into valid empty lists without modifying the intermediate aggregation state. - Preserve state handling through `state()`, `merge_batch()`, and `convert_to_state()` to support partial aggregation and merging. - Add Rust unit tests covering duplicate handling, NULL-only and unseen groups, filtered aggregation, partial-state merging, state conversion, incremental batches, group growth, partial emission, string values, and non-nullable inputs. - Add SQLLogicTests covering grouped `collect_list` and filtered grouped `collect_list`/`collect_set`. - Add a Criterion benchmark for both functions using 2 million rows and workloads with 1,000 and 200,000 groups. ## Are these changes tested? Yes. - `cargo test -p datafusion-spark` — passed (431 tests). - `cargo test -p datafusion-spark-sqllogictest` — passed (252 test files). - `cargo clippy --all-targets --all-features -- -D warnings` — passed. - `cargo fmt --all -- --check` — passed. - `cargo bench -p datafusion-spark --bench collect --no-run` — passed. The added SQLLogicTests verify grouped NULL handling, duplicate preservation, and `FILTER` behavior. Rust unit tests additionally exercise the grouped accumulator's state and emission paths, which are difficult to cover directly through SQLLogicTests. Expected behavior is based on Spark's documented `collect_list` and `collect_set` semantics. No separate PySpark version comparison was performed. ## Are there any user-facing changes? No SQL syntax or function signatures change. This improves the execution path for grouped `collect_list` and `collect_set` and ensures groups with no retained values evaluate to empty lists rather than NULL, consistent with Spark behavior. ## Benchmark Benchmarked through SQL `GROUP BY` with 2 million rows, 10% NULL values, a single-threaded Tokio runtime, and 10 Criterion samples. Before and after measurements were taken on the same machine. **Results:** - `collect_list` (1,000 groups): 425 ms → 44 ms (~9.7x faster) - `collect_set` (1,000 groups): 339 ms → 101 ms (~3.4x faster) - `collect_list` (200,000 groups): 4.82 s → 110 ms (~43.8x faster) - `collect_set` (200,000 groups): 7.03 s → 270 ms (~26x faster) These results compare the previous and new implementations only; there is no Spark baseline. The grouped `array_agg` implementation emits NULL for groups with no retained values, with empty offset ranges for those entries. Removing the null buffer therefore produces valid empty lists without orphaned values. A `debug_assert!` checks this invariant, and a unit test covers the failure case. ## Known Limitations - Nested-type element behavior is not explicitly covered by the added tests. - No SQL test explicitly forces a multi-phase aggregation plan or the skip-partial path. Rust unit tests exercise `state()`, `merge_batch()`, and `convert_to_state()` directly. ## AI Assistance I used an AI assistant during development. I reviewed the implementation and ran the tests and benchmarks myself manually -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
