andygrove opened a new issue, #383: URL: https://github.com/apache/datafusion-spark/issues/383
## Is your feature request related to a problem or challenge? In Spark, nullability is described by two separate members of `Expression`: - `nullIntolerant` (a trait before Spark 4.0): a NULL input always gives a NULL output. The optimizer uses it in `NullPropagation` to replace `f(..., NULL, ...)` with a NULL literal, and in `InferFiltersFromConstraints` to add `IsNotNull` filters on the inputs. - `nullable`: whether the result can be NULL. `UnaryExpression`, `BinaryExpression` and `TernaryExpression` default it to `children.exists(_.nullable)`. Some expressions return NULL for non-NULL inputs and override it, such as `next_day` (`true`) and `make_date` (`true` unless ANSI mode is on). As #2 notes, being null-intolerant does not make an expression's nullability depend only on its children. Each function here works this out for itself: - 28 `return_field_from_args` implementations repeat `args.arg_fields.iter().any(|f| f.is_nullable())`. - Others return `Field::new(.., true)` without saying whether that is Spark's rule. Sometimes it is (`next_day`, `bround`, `is_valid_utf8`), but a reader has to check. The opposite mistake, declaring a field non-nullable when it can be NULL (#1, #315), is a correctness bug, because DataFusion folds `IS NULL` on a non-nullable expression to a constant. - Only 3 files use `compute_null_mask`/`apply_null_mask` in `null_utils.rs` to propagate NULLs at evaluation time. - No function folds `f(col, NULL)` to a NULL literal at plan time as Spark does, unless every argument is a literal. Spark marks 54 of the functions in this crate as null-intolerant (for example `abs`, `ascii`, `crc32`, `date_add`, `hex`, `make_date`, `next_day`, `sha2`, `substring`, `unhex`, `width_bucket`). Others are null-intolerant through the expression they are replaced with (`StaticInvoke` with `propagateNull`), such as `base64` and `url_encode`. ## Describe the solution you'd like A shared helper that a function declares these two properties with, so each one can be checked against Spark's source line by line: - `null_intolerant: bool` - the nullability rule: from the children (the default), always nullable, or never nullable The helper would provide: 1. the return field's nullability, for `return_field_from_args` 2. for a null-intolerant function, a `simplify` that replaces the call with a typed NULL literal when any argument is a NULL literal 3. for a null-intolerant function, a scalar-NULL short-circuit and a combined NULL mask in `invoke_with_args`, built on `null_utils.rs` Where nullability depends on ANSI mode (`make_date`), the helper needs the session config, which `return_field_from_args` does not receive. It can be captured in `with_updated_config`, as `SparkInputTypes` does. I suggest adding the helper with one or two functions first, then moving the rest across one area at a time. ## Describe alternatives you've considered Keep fixing nullability one function at a time under #2. That works, but the same three lines keep being copied, and nothing records which functions Spark treats as null-intolerant. ## Additional context Related: #2 (nullability epic), #1, #315, #21 (bulk NULL builders). Spark 4.1 references: `Expression.nullIntolerant`, `NullPropagation` and `NullDownPropagation` in `optimizer/expressions.scala`, and `QueryPlanConstraints.scanNullIntolerantAttribute`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
