holdenk opened a new pull request, #59978:
URL: https://github.com/apache/spark/pull/59978
Backport-of: #59108
Three related improvements to the Python UDF transpiler:
1. **Null narrowing.** When a parameter is guarded by `x is not None` (or `x
is None`) in an `if`/ternary test, the transpiler tracks that the parameter is
proven non-null in the appropriate branch. Ordering comparisons (`<`, `<=`,
`>`, `>=`) and `len()` calls on such parameters skip the `raise_error` null
guard entirely.
2. **2-statement early-return normalization.** A common guard idiom:
```python
if x is None:
return default
return x > 0
is normalized into a single if/else before lowering, so null narrowing
applies to the main body.
3. UserWarning on null guard emission. When a null guard is still emitted
(non-null cannot be proven), the transpiler warns at UDF construction time.
Co-authored with jing.zhansnowflake.com
Any ordering comparison in a transpiled UDF previously always emitted a
raise_error guard, even when the code already checked x is not None. This
produced unnecessary plan nodes and blocked clean transpilation of the most
common Python null-guard idiom.
Yes. UDFs with is not None-guarded comparisons produce leaner plans (no
raise_error node in the guarded branch). The UserWarning at construction time
is new when unguarded comparisons remain. Behavior on non-null inputs is
unchanged.
Four new test cases in test_udf_transpile_unit.py:
- test_udf_transpile_null_narrowing_skips_null_guard — if/ternary with is
not None guard: plan has no raise_error, results match Python
- test_udf_transpile_early_return_null_guard — 2-stmt early-return pattern
transpiles with no raise_error
- test_udf_transpile_null_guard_warning — unguarded comparison emits
UserWarning
- test_udf_transpile_null_narrowing_len — guarded len() has no raise_error
in plan
Co-authored by: Claude Sonnet 4.6
Closes #59108 from jzhan-2026/spark-58628-non-nullable-null-guard.
Authored-by: jzhan-2026 <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Signed-off-by: Holden Karau <[email protected]>
(cherry picked from commit dc73d212ae6213f0079ffd01dce7dad99e7db827)
Signed-off-by: Holden Karau <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
