sunchao opened a new issue, #5401:
URL: https://github.com/apache/datafusion-comet/issues/5401
## Describe the bug
`array_min` and `array_max` do not preserve Spark's choice of the first
element when `+0.0` and `-0.0` compare equal. The corrected fixtures in #5393
expose this for `array_min`. The reverse zero order also exposes it for
`array_max`.
At upstream commit `91f9fbed24ee28f99b23ed559e80e818c9696044`, both
expressions are classified as compatible without checking
`spark.comet.exec.strictFloatingPoint`. They therefore use the differing native
implementation even when strict floating-point behavior is requested.
## Reproduction
Use a Parquet column so the extrema are evaluated at runtime:
```sql
SET
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SET spark.comet.exec.strictFloatingPoint=true;
CREATE TABLE comet_extrema_zero_repro (id INT, arr ARRAY<DOUBLE>) USING
parquet;
INSERT INTO comet_extrema_zero_repro VALUES
(1, array(double('0.0'), double('-0.0'))),
(2, array(double('-0.0'), double('0.0')));
SELECT id, array_min(arr), array_max(arr)
FROM comet_extrema_zero_repro ORDER BY id;
```
The same issue occurs for `ARRAY<FLOAT>`.
| Input | Spark min | Native min | Spark max | Native max |
| --- | --- | --- | --- | --- |
| `[+0.0, -0.0]` | `+0.0` | `-0.0` | `+0.0` | `+0.0` |
| `[-0.0, +0.0]` | `-0.0` | `-0.0` | `-0.0` | `+0.0` |
Spark 3.5.8 and 4.0.1 preserve the first zero. A direct probe of the pinned
DataFusion 54.1.0 / Arrow 58.4.0 UDFs confirms the native results for both
floating-point widths and on both sides of the array-length-32 kernel boundary.
## Expected behavior
Strict floating-point mode should use Spark-compatible evaluation. The
native implementation should eventually preserve Spark's first-equal-element
behavior as well. A narrow first fix can route strict-mode extrema through the
existing Spark-codegen fallback, while keeping this issue open for native
parity.
Relevant code: `CometArrayMin` and `CometArrayMax` in
`spark/src/main/scala/org/apache/comet/serde/arrays.scala`. This is a follow-up
to #5271 and #5393, not a regression introduced by those fixture changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
