viirya opened a new pull request, #6761: URL: https://github.com/apache/datafusion-comet/pull/6761
## Which issue does this PR close? Closes #6759. ## Rationale for this change Spark orders a null inside a struct or array before every other value, in both `greatest` and `least`. Comet sent nested types without a float leaf to DataFusion's `least`, which orders such a null after every other value (`nulls_first: false`), so `least` could return a different value from Spark. ## What changes are included in this PR? `SparkGreatestLeast::handles` now accepts every nested type, not only those with a float leaf. For those types, `SparkGreatestLeast` compares with `spark_comparator`, which already orders nested nulls first. Scalar non-float types still use DataFusion's `greatest` and `least`. ## How are these changes tested? - New SQL file `expressions/math/greatest_least_nested_nulls.sql`, covering structs, arrays, and arrays of structs with a null nested value. It fails without this change: for `id = 3`, Spark returns `[null, 3]` and Comet returned `[3, 1]`. - New Rust unit test `nested_nulls_sort_first`; `handles_floats_and_nested_types_only` replaces the previous `handles` test. - The existing `greatest`, `least`, and float SQL files still pass. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
