divyankshah opened a new pull request, #5235: URL: https://github.com/apache/datafusion-comet/pull/5235
## Which issue does this PR close? Closes #5191. ## Rationale for this change Arrow's nested comparator (`make_comparator`, used by `arrays_overlap`'s nested path and `array_position`'s nested fallback) orders floats by total order, where `-0.0` and `0.0` are distinct. Spark's `ordering.equiv` (used for structural equality of nested elements) checks numeric equality first, so `-0.0 == 0.0` there, while `NaN == NaN` still holds in both. ## What changes are included in this PR? - Add `nested_float_normalize.rs`: recursively rebuilds nested (List/LargeList/FixedSizeList/Struct) arrays with `-0.0` normalized to `0.0` in Float32/Float64 leaves, leaving NaN untouched. - `arrays_overlap.rs`: normalize both sides before building the nested comparator; update `test_nested_float_total_order` to assert `-0.0` and `0.0` now overlap; add `test_struct_float_field_signed_zero_overlap` covering a struct field. - `array_position.rs`: normalize both sides before building the fallback comparator; update `test_nested_float_and_null_position` (result changes from `[2, 2, 1]` to `[2, 1, 1]` since row 1's `-0.0` vs `0.0` now matches at position 1); add `test_struct_float_field_signed_zero_position` covering a struct field. Note: #5194 is a separate issue (#5101, comparator-hoisting for perf) but touches the same comparator-construction code path. Happy to rebase on top of whichever lands first. ## How are these changes tested? - `cargo test -p datafusion-comet-spark-expr` — all 600+ tests pass, including the new/updated ones above. - `cargo clippy -p datafusion-comet-spark-expr --lib -- -D warnings` — clean. - `cargo fmt -p datafusion-comet-spark-expr -- --check` — clean. - `cargo check --workspace` — clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
