mrhard9090 opened a new pull request, #26011: URL: https://github.com/apache/datafusion/pull/26011
## Which issue does this PR close? - Closes #25981. ## Rationale for this change `SparkArrayContains` delegates the comparison to `array_has`, which compares floats by their bits. Spark's `array_contains` treats `-0.0` as equal to `0.0` and all NaNs as equal, so `array_contains(array(-0.0D), 0.0D)` returned `false` here (Spark: `true`), and two NaNs with different bit patterns did not match. ## What changes are included in this PR? - `array_contains.rs` rewrites the float values before calling `array_has`: `-0.0` becomes `0.0` and every NaN becomes the canonical NaN. It applies to a float needle (scalar or array) and to the elements of the list haystack, including `List`, `LargeList` and `FixedSizeList` and lists nested in lists. The existing null handling is unchanged. - Cases in `spark/array/array_contains.slt`: `-0.0` vs `0.0` in both directions for `double`, `-0.0` for `float`, NaN vs `-NaN`, NaN next to another value, a nested list, and a plain miss that stays `false`. Not covered: floats inside struct elements (the issue also mentions them). I did not look at `arrays_overlap`, which the issue says probably has the same equality. ## What is the testing strategy for this PR? `cargo test -p datafusion-sqllogictest --test sqllogictests -- array_contains.slt` passes. With `array_contains.rs` from `main`, the first four new queries fail. `cargo fmt` and `cargo clippy -p datafusion-spark --all-targets --all-features --no-deps -- -D warnings` pass. ## Are there any user-facing changes? Spark `array_contains` on float or double elements now matches Spark for `-0.0` and NaN. No API change. AI assistance: written with Claude Code; I reviewed the diff and ran the tests above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
