andygrove opened a new issue, #6768: URL: https://github.com/apache/datafusion-comet/issues/6768
### Describe the bug From Spark 4.0.5, 4.1.4 and 4.2.1, `arrays_overlap` treats `-0.0` and `0.0` as one value in flat float and double arrays. SPARK-59602 handles "primitive floating-point values locally in `arrays_overlap`". Comet's native `spark_arrays_overlap` keeps them apart in its flat path, as earlier Spark releases do (see the note on `arrays_overlap_list_generic` in `arrays_overlap.rs`), so on those releases it returns `false` where Spark returns `true`. Spark 4.2.0 normalized these arguments in the plan (SPARK-54918), so Comet received normalized input there. SPARK-59602 replaced that rewrite in 4.2.1. ### Steps to reproduce Build Comet against the staged `v4.1.4-rc2` (`orgapachespark-1533`), `v4.0.5-rc1` (`orgapachespark-1534`) or `v4.2.1-rc1` (`orgapachespark-1531`) artifacts and run `expressions/array/arrays_overlap.sql`: ```sql SELECT a, b, arrays_overlap(a, b) FROM test_overlap_dbl ``` Spark returns `[List(0.0),List(-0.0),true]` and Comet `[List(0.0),List(-0.0),false]`. The `arrays_overlap [double]` and `[float]` cases of `CometFloatSemanticsSuite` fail on the same zero pairs. `main` at f80042f78 fails the same way. ### Expected behavior Comet matches each version: flat signed zeros stay apart before 4.0.5, 4.1.4 and 4.2, and merge from 4.0.5, 4.1.4 and 4.2.1 (4.2.0 receives them already normalized). #6563 adds the same version check for `array_distinct` and `array_union` (`ArraySetSupport.normalizesFloats`), and the kernel can take the result as a flag. ### Additional context Found by running the array suites against the Spark release candidates for #6563. Part of #6385. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
