holdenk opened a new pull request, #59216: URL: https://github.com/apache/spark/pull/59216
### What changes were proposed in this pull request? Adds chained comparisons (`0 <= x <= 100`, `a < b < c`) to the experimental Python UDF transpiler. A chain lowers one adjacent pair at a time through `_lower_compare_pair` and the links fold with `&`, which short-circuits at evaluation time (`And.eval` / `And.doGenCode` skip the right operand when the left is exactly false), so the `raise_error` guard that mirrors Python's `TypeError` on a NULL operand cannot fire on a row Python never reached. Interior operands are restricted to basic-type expressions (names, constants, arithmetic), so the duplication a chain requires is value-identical to Python's evaluate-once. Plan growth is bounded twice: `_MAX_COMPARISON_OPS` (64) caps the operators in one chain, and `_MAX_LOWERED_COMPARISONS` (128) budgets the comparisons one body lowering can emit across all its comparisons, chains included. Anything past either falls back to interpreted Python. `and` / `or` keep the existing `&` / `|` fold, but their operand gate tightens from `_is_definitely_boolean` to the new `_is_never_null_boolean`: Spark's three-valued logic collapses `NULL AND false` to `false` where Python's `None and False` returns `None`, so operands that can evaluate to NULL (a literal `None`, a ternary with a `None` branch) now fall back instead of diverging. Also fixes `_category` mislabeling an `ast.Compare` whose operands are not basic types: the numeric catch-all let a boolean body satisfy a numeric declared return type, emitting `cast(<boolean> as bigint)` (1/0) where interpreted Python returns NULL. A `Compare` now categorizes as "bool" by node kind. The module docstring records four measured optimizer divergences with the fold, none specific to chains (the `and` form of each repro predates this lowering, so a chain is new syntax reaching existing holes): predicate pushdown through a JOIN (CLOSED by SPARK-58627 -- `RaiseError` now overrides `throwable`), pushdown through an AGGREGATE or WINDOW (OPEN), `BooleanSimplification` common-factor hoisting and absorption (OPEN -- absorption drops a guard and is a wrong-answers bug), and subexpression elimination (OPEN). Those are optimizer fixes, tracked separately; the NaN ordering divergence is SPARK-58781. `in` / `not in` are explicitly refused and fall back; membership needs its own NULL analysis and is deferred to its own ticket. ### Why are the changes needed? Chained comparisons are common in real UDF bodies and previously forced the whole UDF back to interpreted Python. The `and` / `or` NULL divergence is a pre-existing correctness bug on the transpiled path. ### Does this PR introduce _any_ user-facing change? Only under the experimental, off-by-default `spark.sql.experimental.optimizer.transpilePyUDFs` (which also requires ANSI). More UDF bodies transpile to native Catalyst expressions. Bodies with a possibly-NULL `and` / `or` operand, such as `(True if c else None) and (y > 0)`, that previously transpiled now fall back to interpreted Python, whose answer is the correct one. ### How was this patch tested? `test_udf_transpile_unit.py`: chain lowering and plan elision, short-circuit tests run with whole-stage codegen both on and off, all NULL combinations for `x == y == z`, cap boundaries at N and N+1 for both caps, a must-not-transpile set that also asserts each query still returns Python's answer, and a plan-shape regression test asserting the raise guard is not pushed below a join. `test_udf_transpile_hypothesis.py`: four new chained-comparison UDF templates (single-parameter bounds, three-column ordering, all-equality, mixed operators) with property tests comparing transpiled against interpreted results, seeded with the NULL edge combinations for the three-operand chains. Passed at 100 examples per test, and previously at 1000. The non-determinism RED tests this PR originally carried were dropped after SPARK-58626 landed: a call-site argument is now computed once per row, so `50 < x < 60` over `rand(7) * 100` no longer desynchronizes into independent draws. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5; Kimi K3 (rebase onto current master, conflict resolution, commit trailer hygiene, this description refresh); GLM 5.2 Max (code review pass, chained-comparison seed-edge simplification, pre-send cleanup, opening this PR). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
