peterxcli opened a new pull request, #5166: URL: https://github.com/apache/datafusion-comet/pull/5166
## Which issue does this PR close? Closes #4966. ## Rationale for this change Spark 4.2 changed `collect_set` to normalize NaN values and signed zero before deduplication. Comet passed the raw input to the native partial aggregate, whose set distinguishes floating-point bit patterns, so its result could differ from Spark for NaN and `-0.0` values. ## What changes are included in this PR? - Reuse Spark's recursive floating-point normalizer for Spark 4.2+ `collect_set` partial aggregation. - Keep final aggregation state merging unchanged and preserve the strict floating-point fallback on older Spark versions. - Add Spark-version-specific SQL coverage for scalar, array, and struct values across partial aggregation buffers. ## How are these changes tested? - `make core` - `./mvnw test -Pspark-4.2 -Dtest=none -Dsuites='org.apache.comet.CometSqlFileTestSuite collect_set' -Dscalastyle.skip=true` (4 tests passed) - `./mvnw test -Pspark-4.1 -Dtest=none -Dsuites='org.apache.comet.CometSqlFileTestSuite collect_set_floating_fallback' -Dscalastyle.skip=true` (1 test passed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
