andygrove opened a new pull request, #389:
URL: https://github.com/apache/datafusion-spark/pull/389

   ## Which issue does this PR close?
   
   - Closes #315.
   - Part of #383.
   
   ## Rationale for this change
   
   Spark declares two nullability properties on each expression: `nullable` 
(whether the result can be NULL) and `nullIntolerant` (a NULL input always 
gives a NULL output, which `NullPropagation` uses to fold `f(col, NULL)` to 
NULL). Here each function worked out its nullability by hand, so:
   
   - Functions that only implemented `return_type` were always nullable, even 
where Spark takes nullability from the children.
   - Some followed their arguments where Spark is always nullable (`floor`, 
`make_valid_utf8`).
   - `make_dt_interval` declared a non-nullable result but returns NULL on 
overflow, so a batch failed validation when the call was not constant-folded 
(#315).
   - A call with a NULL literal argument and a column argument, such as 
`pmod(col, NULL)`, was not folded to NULL at plan time.
   
   ## What changes are included in this PR?
   
   - `function/nullability.rs` adds `SparkNullability`. A function declares 
whether it is null-intolerant and whether its result is nullable from its 
children or always, and the helper provides:
     - the return field, for `return_field_from_args`
     - a `simplify` that replaces a call with a NULL literal argument with a 
typed NULL literal, for a null-intolerant function. `add_months`, `date_diff` 
and `unix_date` already had a `simplify`; the fold runs before their rewrite.
   - About 60 functions that Spark marks as null-intolerant now use it. Each 
constant names the Spark expression and its rule, so the declaration can be 
checked against Spark's source.
   - These are now non-nullable for non-nullable arguments, as in Spark: 
`array_position`, `array_remove`, `atan2`, `contains`, `hex`, `hour`, `minute`, 
`second`, `hypot`, `negative`, `pow`, `soundex` and `space`. I checked that 
each only returns NULL for a NULL argument.
   - `floor`, `make_valid_utf8` and `make_dt_interval` are now always nullable. 
`make_dt_interval` differs from Spark here, because an overflow returns NULL 
instead of raising an error (#20).
   - `make_date` and `make_interval` stay always nullable. With ANSI mode on, 
Spark makes them nullable from their arguments, but `return_field_from_args` 
cannot see ANSI mode.
   - `array_contains` is also nullable when the array can hold NULL elements, 
and `map_from_entries` when an entry can be NULL, as in Spark.
   
   Not included: `date_part` and the functions that Spark replaces with a 
null-intolerant `StaticInvoke` (`base64`, `url_encode`, ...), which belong with 
#386, and `spark_cast` (#88).
   
   ## Are these changes tested?
   
   - Unit tests for `SparkNullability` in `nullability.rs`.
   - New `.slt` records for a column combined with a NULL argument, in 
`string/substring.slt`, `array/slice.slt`, `datetime/date_add.slt`, 
`datetime/date_sub.slt`, `bitwise/bit_get.slt`, `string/contains.slt`, 
`predicate/like.slt`, `math/pmod.slt` and `math/atan2.slt`. They run in both 
`simplify` and `evaluate` mode, and an `EXPLAIN` record in `substring.slt` 
shows the fold.
   - The `make_dt_interval.slt` records that were skipped in `evaluate` mode 
for #315 now run in both modes.
   - I checked the changed `.slt` files against Spark 4.2.0 with the PySpark 
runner, which found no differences.
   
   ## Are there any user-facing changes?
   
   - A call to a null-intolerant function with a NULL literal argument is 
folded to NULL during planning.
   - The nullability of the result fields above changes to match Spark. 
`make_dt_interval` no longer fails in `evaluate` mode when it overflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to