0lai0 opened a new pull request, #5981:
URL: https://github.com/apache/datafusion-comet/pull/5981
## Which issue does this PR close?
Closes #5506.
## Rationale for this change
#5469 normalized scalar floating-point comparison keys natively, so they
match Spark's `SQLOrderingUtil`. Admission did not follow. Under
`spark.comet.exec.strictFloatingPoint=true`, two gates still rejected scalar
`FLOAT`/`DOUBLE` keys, so the whole stage fell back to Spark:
```
before *(2) Sort
+- Exchange rangepartitioning(...)
+- *(1) CometColumnarToRow
+- CometNativeScan parquet
after CometSort
+- CometExchange rangepartitioning(...), CometNativeShuffle
+- CometNativeScan parquet
```
On 4M `DOUBLE` rows with negative zeros and NaNs, `ORDER BY v, id` went from
a median of 4400 ms to 2551 ms (1.7x). The "before" plan is reproduced on the
same build with `spark.comet.expression.SortOrder.enabled=false`, and the
result held with the run order reversed.
## What changes are included in this PR?
- `CometSortOrder.getSupportLevel` reports scalar `FloatType`/`DoubleType`
as `Compatible`. Floats nested in arrays, structs and maps still fall back
(#5507). The other callers of `strictFloatingPointReason`, including
`SortArray`, are unchanged.
- `supportedRangePartitioningDataType` accepts scalar float and double. The
native range partitioner already normalizes both its keys and its sampled
bounds.
- The `strictFloatingPoint` config description, `tuning.md` and
`floating-point.md` now say which sorts are still affected, including
`sort_array`.
## How are these changes tested?
- A matrix over both types, directions, null orders and compound keys
asserts `CometSortExec` and equality with Spark. Separate tests cover TopK,
raw-bit fidelity of returned values, window ordering on every Spark version,
sort-merge join, and columnar shuffle.
- Every strict-mode test pins `SortOrder.allowIncompatible=false`, because
`CometTestBase` defaults it to true.
- Local: 622 passed on Spark 4.1, 374 on Spark 3.5, and 540 in
`CometSqlFileTestSuite`. Spark 3.4/4.0 and the TPC-DS plan stability suites,
which need `SPARK_HOME`, are left to CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
