LinSimon-901101 opened a new pull request, #7220: URL: https://github.com/apache/datafusion-comet/pull/7220
## Which issue does this PR close? Closes #7203. ## Rationale for this change Before Spark 4.2, `collect_set` over FLOAT or DOUBLE collapses signed zeros and retains repeated NaNs in Spark, whereas native execution distinguishes the zeros and deduplicates the NaNs. Enabling native PartialMerge chains exposed this difference when `collect_set` is combined with a DISTINCT aggregate. Comet 1.1.0 kept that shape in Spark. ## What changes are included in this PR? Decline FLOAT/DOUBLE `collect_set` in PartialMerge mode before Spark 4.2. The existing unsafe-partial pass and buffer compatibility checks keep the aggregate chain in Spark, restoring the previous results. Document this restriction in the CollectSet compatibility notes. The restriction is specific to scalar FLOAT/DOUBLE PartialMerge on older Spark versions. Ordinary `collect_set`, `collect_list`, other element types, and Spark 4.2 retain their existing native support. ## How are these changes tested? - Add 16 regressions covering FLOAT/DOUBLE, Spark partial versus Parquet input, AQE on/off, and object-hash versus sort aggregation. Inputs include repeated NaNs, signed zeros, nulls, and an all-null group. Assertions check both results and the same executed plan: the aggregate chain stays in Spark before 4.2, while merge/final remain native on 4.2. - Add a control that checks native ordinary float `collect_set`, float `collect_list` with DISTINCT, and struct-valued `collect_set` with DISTINCT. All 16 regressions fail with wrong results on the unmodified baseline; the control passes. - All 17 cases pass in the full [fork CI run](https://github.com/LinSimon-901101/datafusion-comet/actions/runs/40192838751) at commit `6029ad2`, on five Linux Spark profiles (3.4 through 4.2) and macOS. All 20 Linux Comet matrix rows, all 45 Spark SQL matrix rows, and Required Checks pass. The related SPARK-17616 integration cases also pass on each Spark version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
