andygrove opened a new pull request, #7489:
URL: https://github.com/apache/datafusion-comet/pull/7489
## Which issue does this PR close?
Backport of #7220 to `branch-1.2`. The source pull request closed #7203.
## Rationale for this change
Before Spark 4.2, `collect_set` over FLOAT or DOUBLE collapses signed zeros
and retains repeated NaNs in Spark, whereas native execution distinguishes the
zeros and deduplicates the NaNs. Enabling native PartialMerge chains exposed
this difference when `collect_set` is combined with a DISTINCT aggregate, so
`branch-1.2` returns different results from Comet 1.1.0 and Spark for that
query shape. `branch-1.2` is the newest release branch and the only one
affected: `branch-1.1` kept that shape in Spark.
## What changes are included in this PR?
Cherry-pick of 4e6b12290f (`git cherry-pick -x`), which declines
FLOAT/DOUBLE `collect_set` in PartialMerge mode before Spark 4.2 so the
aggregate chain stays in Spark. It also adds the SQL file regressions and a
nested-type control.
Adapted for `branch-1.2`:
- `aggregates.scala`: add `PartialMerge` to the import list. Main already
imports it, but this branch does not, so the clean cherry-pick failed to
compile.
-
`docs/source/user-guide/latest/compatibility/expressions/spark-{3.4,3.5,4.0,4.1}/aggregate.md`:
add the new "always falls back" bullet under `CollectSet`. Release branches
commit the generated compatibility pages, so the source change would otherwise
leave them stale. The text matches what `GenerateDocs` emits; Spark 4.2 is
unchanged.
Not included: the `collect_set` item under "Known Differences From Spark
That Are New in 1.2.0" in `migration-guide.md` is now inaccurate (this change
makes that query shape run in Spark again, as in 1.1.0). Main has the same
stale text, so I'd fix it on `main` first and backport it separately.
## How are these changes tested?
- `./mvnw test-compile` passes, as does `cargo build` in `native/`.
- `CometAggregateSuite` and `CometSqlFileTestSuite` pass on the default
Spark profile (909 passed, 0 failed). One test is canceled because it only runs
on Spark 4.2.
- With the fix disabled, the 4 `collect_set_distinct_floating_fallback.sql`
cases fail and everything else passes, so the branch has the bug and the new
tests catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
