peterxcli opened a new pull request, #5750:
URL: https://github.com/apache/datafusion-comet/pull/5750
## Which issue does this PR close?
Closes #5701.
## Rationale for this change
DataFusion deduplicates positive and negative floating-point zero in
`array_distinct` and `array_union`. Spark versions without SPARK-54918
distinguish them, but Comet currently enables both expressions natively by
default. A plain projection such as `array_distinct(array(0.0, double('-0.0'),
1.0))` therefore returns `[0.0, 1.0]` with Comet instead of Spark's `[0.0,
-0.0, 1.0]`. This affects literal and column inputs; `NormalizeFloatingNumbers`
does not normalize arbitrary projection expressions.
SPARK-54918 aligns Spark's behavior starting with 4.0.5, 4.1.4, and 4.2.0.
Comet also supports earlier patch releases and Spark 3.4/3.5, so changing the
build's Spark dependency or using a minor-version shim would not protect
applications running against those versions. The decision must use the Spark
version at runtime, with numeric patch comparison so versions such as 4.0.10
are handled correctly.
## What changes are included in this PR?
- Register a dedicated `ArrayDistinct` serde and add a shared compatibility
check for `ArrayDistinct` and `ArrayUnion`. Arrays containing `FloatType` or
`DoubleType`, including nested element types, report `Incompatible` on Spark
versions without SPARK-54918 and fall back by default. Other element types and
fixed Spark versions retain native execution.
- Reuse Spark's numeric version parser through Comet's existing Spark
utility wrapper. Recognize the individual 4.0.5 and 4.1.4 backport boundaries
as well as Spark 4.2 and later.
- Preserve the existing per-expression `allowIncompatible` opt-in and
provide a fallback explanation identifying SPARK-54918 and the relevant
versions.
- Restore the ignored signed-zero SQL cases. Floating-point distinct/union
SQL queries compare results while allowing the version-dependent fallback;
Scala tests separately assert fallback on older versions and native execution
on fixed versions. Except/intersect signed-zero cases use their existing safe
default dispatch, with incompatible native opt-in retained for the other cases
in those files.
- Document the runtime compatibility boundary and opt-in behavior.
## How are these changes tested?
- 10 targeted tests passed on Spark 4.1.3 and 10 on Spark 4.2.0, covering
version boundaries, nested types, fallback/native plans, opt-in, and the
affected SQL fixtures. The Spark 4.2.0 run was repeated successfully after
rebasing onto current upstream main.
- Native build, formatting, and style checks passed after rebase; JVM
package build also passed before rebase.
- Optional documentation generation is blocked by the checkout's missing
`compatibility/expressions/cast.md` input; partial generated output was
reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
