andygrove opened a new pull request, #389:
URL: https://github.com/apache/datafusion-spark/pull/389
## Which issue does this PR close?
- Closes #315.
- Part of #383.
## Rationale for this change
Spark declares two nullability properties on each expression: `nullable`
(whether the result can be NULL) and `nullIntolerant` (a NULL input always
gives a NULL output, which `NullPropagation` uses to fold `f(col, NULL)` to
NULL). Here each function worked out its nullability by hand, so:
- Functions that only implemented `return_type` were always nullable, even
where Spark takes nullability from the children.
- Some followed their arguments where Spark is always nullable (`floor`,
`make_valid_utf8`).
- `make_dt_interval` declared a non-nullable result but returns NULL on
overflow, so a batch failed validation when the call was not constant-folded
(#315).
- A call with a NULL literal argument and a column argument, such as
`pmod(col, NULL)`, was not folded to NULL at plan time.
## What changes are included in this PR?
- `function/nullability.rs` adds `SparkNullability`. A function declares
whether it is null-intolerant and whether its result is nullable from its
children or always, and the helper provides:
- the return field, for `return_field_from_args`
- a `simplify` that replaces a call with a NULL literal argument with a
typed NULL literal, for a null-intolerant function. `add_months`, `date_diff`
and `unix_date` already had a `simplify`; the fold runs before their rewrite.
- About 60 functions that Spark marks as null-intolerant now use it. Each
constant names the Spark expression and its rule, so the declaration can be
checked against Spark's source.
- These are now non-nullable for non-nullable arguments, as in Spark:
`array_position`, `array_remove`, `atan2`, `contains`, `hex`, `hour`, `minute`,
`second`, `hypot`, `negative`, `pow`, `soundex` and `space`. I checked that
each only returns NULL for a NULL argument.
- `floor`, `make_valid_utf8` and `make_dt_interval` are now always nullable.
`make_dt_interval` differs from Spark here, because an overflow returns NULL
instead of raising an error (#20).
- `make_date` and `make_interval` stay always nullable. With ANSI mode on,
Spark makes them nullable from their arguments, but `return_field_from_args`
cannot see ANSI mode.
- `array_contains` is also nullable when the array can hold NULL elements,
and `map_from_entries` when an entry can be NULL, as in Spark.
Not included: `date_part` and the functions that Spark replaces with a
null-intolerant `StaticInvoke` (`base64`, `url_encode`, ...), which belong with
#386, and `spark_cast` (#88).
## Are these changes tested?
- Unit tests for `SparkNullability` in `nullability.rs`.
- New `.slt` records for a column combined with a NULL argument, in
`string/substring.slt`, `array/slice.slt`, `datetime/date_add.slt`,
`datetime/date_sub.slt`, `bitwise/bit_get.slt`, `string/contains.slt`,
`predicate/like.slt`, `math/pmod.slt` and `math/atan2.slt`. They run in both
`simplify` and `evaluate` mode, and an `EXPLAIN` record in `substring.slt`
shows the fold.
- The `make_dt_interval.slt` records that were skipped in `evaluate` mode
for #315 now run in both modes.
- I checked the changed `.slt` files against Spark 4.2.0 with the PySpark
runner, which found no differences.
## Are there any user-facing changes?
- A call to a null-intolerant function with a NULL literal argument is
folded to NULL during planning.
- The nullability of the result fields above changes to match Spark.
`make_dt_interval` no longer fails in `evaluate` mode when it overflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]