goutamadwant opened a new pull request, #24767: URL: https://github.com/apache/datafusion/pull/24767
## Which issue does this PR close? - Closes #24741. ## Rationale for this change Spark's `collect_list` and `collect_set` drop null inputs and declare arrays with `containsNull = false`. The DataFusion Spark implementations declared nullable list elements, and their accumulator results could derive nested fields from runtime arrays. The resulting schema mismatch requires downstream casts; when runtime nested fields drift from the declared type, aggregate output validation can fail. ## What changes are included in this PR? - Declare non-nullable list elements in the return and partial-state types for both aggregates. - Normalize partial state and final accumulator output to the declared list field, including nested child types. - Reuse matching child arrays without casting, while reconciling a differing nested runtime type only when needed. - Preserve the same exact type for empty and all-null results. ## Are these changes tested? Yes. Regression tests cover return and state fields, empty results, primitive values, nested struct nullability, partial-state merging, and both `collect_list` and `collect_set`. The existing Spark aggregate and window SQLLogicTest suites also pass. The full required workspace test suite and workspace-wide clippy with all targets and features pass. ## Are there any user-facing changes? `collect_list` and `collect_set` now expose Arrow list schemas with non-nullable elements, matching Spark. There are no Rust API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
