advitrocks9 opened a new pull request, #24827: URL: https://github.com/apache/datafusion/pull/24827
## Which issue does this PR close? No issue filed. Part of #15914, which tracks the Spark function library. ## Rationale for this change `equal_null` is missing, and the `misc` module it belongs in had no functions in it at all. In Spark it is an alias: `EqualNull(l, r)` is rewritten to `EqualNullSafe`, the `<=>` operator, so it is null-safe equality. Both NULL is true, and the result is never NULL: ```sql SELECT equal_null(NULL, NULL); -- true SELECT equal_null(NULL::int, 1::int); -- false ``` DataFusion already has that operator as `IS NOT DISTINCT FROM`, so this mirrors Spark's own structure instead of writing a second comparison kernel. ## What changes are included in this PR? `misc/equal_null.rs` and its registration. `simplify()` rewrites to the operator, which is Spark's `replacement` field, and `invoke_with_args` calls `apply_cmp` with the same operator so the function also works with the logical optimizer disabled, as the crate README requires for Comet. The `.slt` stub is filled in with 27 assertions. Two of its commented-out queries were malformed: the porting script wrote one cast per distinct `typeof()` key, so a call with two identical literals lost an argument. The same artifact affects 10 more pairs in 6 other spark files, left alone here. Two divergences from Spark are left alone because they belong to the operator, not to this function. Spark treats `-NaN` and `NaN` as equal and DataFusion does not, which is true of `IS NOT DISTINCT FROM` generally. Spark also rejects maps at analysis since `MapType` is not orderable, while `comparison_coercion` here accepts them. ## Are these changes tested? Yes, `spark/misc/equal_null.slt` goes from a skipped stub to 27 assertions covering the truth table, float ordering, columns, arrays, structs, decimals and the arity errors. Reverting `simplify()` to plain `Eq` fails 9 of them, so the file is not passing on constant folding. Putting `invoke_with_args` back to a stub fails the two queries that run with the optimizer off, and nothing else. ## Are there any user-facing changes? Yes, `equal_null` is a new function in `datafusion-spark`. Nothing existing changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
