sunchao opened a new issue, #5506:
URL: https://github.com/apache/datafusion-comet/issues/5506
### What is the problem the feature request solves?
After #5469, native Sort, Window, WindowGroupLimit, and range-partition
comparison keys normalize scalar `FLOAT` and `DOUBLE` NaNs and signed zeros
consistently with Spark. Admission is still more conservative:
`CometSortOrder.getSupportLevel` calls the shared `strictFloatingPointReason`
helper, which rejects both scalar and nested floating types when
`spark.comet.exec.strictFloatingPoint=true`.
Strict-mode users therefore still lose native scalar floating sorts even
though #5469 fixes their comparison semantics. This follow-up should evaluate
and test narrowing that specific admission decision, without removing the
remaining nested-type protections or changing other callers of the shared
helper.
### Describe the potential solution
Use a Spark session with Comet enabled and `CometShuffleManager` configured.
Create a Parquet input so constant folding cannot remove the sort:
```sql
CREATE TABLE comet_scalar_sort_case (id INT, v DOUBLE) USING parquet;
INSERT INTO comet_scalar_sort_case VALUES
(1, CAST('-0.0' AS DOUBLE)), (2, CAST('0.0' AS DOUBLE)), (3, 1.0);
SET spark.sql.adaptive.enabled=false;
SET spark.sql.shuffle.partitions=1;
SET spark.comet.exec.strictFloatingPoint=true;
EXPLAIN FORMATTED
SELECT id, v FROM comet_scalar_sort_case ORDER BY v, id DESC;
SET spark.comet.exec.strictFloatingPoint=false;
EXPLAIN FORMATTED
SELECT id, v FROM comet_scalar_sort_case ORDER BY v, id DESC;
```
Require the strict-mode plan to use native `CometSortExec` only after its
compatibility is established; the current strict-mode plan falls back. Check
results against Comet disabled, not only against the non-strict native path.
The zero peers must be ordered by the secondary key (`id` 2 before 1), and
returned zero signs must remain unchanged.
The acceptance tests should cover `FLOAT` and `DOUBLE`, both sort directions
and null orders, compound keys, signed zeros, and independently constructed
positive/negative NaN payloads. Cover ordinary window ordering and
`RANK`/`DENSE_RANK` cutoffs with actual native operator assertions. Retain
strict-mode fallback controls for arrays/structs containing floats, and test
all supported Spark versions before narrowing the policy.
### Additional context
The current [SortOrder
gate](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/spark/src/main/scala/org/apache/comet/serde/CometSortOrder.scala#L33-L39)
delegates to a [recursive shared
policy](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/spark/src/main/scala/org/apache/comet/serde/SupportLevel.scala#L87-L104).
The native [scalar key
normalization](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/native/core/src/execution/planner.rs#L927-L963)
is intentionally narrower. Keep this admission follow-up separate from #5468's
scalar wrong-result fix and from the nested ordering/rank limitation. #2626 is
closed and does not track this remaining admission work.
A fresh Spark 4.1.3 planning probe confirmed native scalar sorts with strict
mode disabled and Spark sorts with it enabled for both FLOAT and DOUBLE. The
reused JVM build's `CometSortOrder`, `SupportLevel`, and `CometWindowExec`
sources are byte-identical to the reviewed #5469 head. This checks admission;
it does not claim a new full-binary runtime validation of the scalar
normalization fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
