holdenk opened a new pull request, #59730: URL: https://github.com/apache/spark/pull/59730
### What changes were proposed in this pull request? Two related changes to the experimental Python UDF transpiler's option resolution, the second folded in from SPARK-59090: 1. **SPARK-55210 -- numeric type promotion.** Match Python's type promotion of transpiled operations. The transpiler tags options with narrower numeric categories (`integral`, `integral32`, `fractional`) where an exact lowering depends on the *kind* of number, and `ResolveTranspiledPythonUDFOptions` matches them against an allow-list of column types with a faithful Python counterpart (Byte/Short/Integer/Long/Double for `numeric`; the narrow ones as documented in the rule). Decimal and Float are deliberately excluded -- Python's `decimal.Decimal` and the absence of single-precision float make their Catalyst arithmetic diverge. Also lets array-of-atomic return types through the transpiler's outer gate (the per-variant body-category check still refuses every array-typed body, so only a NULL lowering survives, which casts cleanly to any array type). `PythonNumericPromotion` computes the worst-case magnitude of an operation from the input types alone and widens the operands to a type t hat holds it (`tinyint * 100` -> short arithmetic, `int + int` -> long arithmetic), through internal `python_promoting_add/subtract/multiply/abs/negate` expressions; bigint-range worst cases stay put and raise exactly as before. 2. **SPARK-59090 -- drop transpiled options that fail to resolve (folded in).** A category match does not make an option resolvable -- `cast(binary as bigint)` matches "binary" and never resolves -- and an option is a child of `TranspiledPythonUDF`, so one left unresolved when analysis finishes reached `CheckAnalysis` and failed the whole query with a type mismatch naming a cast the transpiler invented. Adds `DropUnresolvedTranspiledPythonUDFOptions`, a `Once` batch after the Resolution batch converges, which drops any still-unresolved option so the call falls back to interpreted Python instead of failing. The pruning body is shared between the two rules via `pruneByCategoryAndTypeParameters` so they cannot drift. Because the drop resolves the UDF *after* the Resolution batch converged, whatever was waiting on it -- the enclosing `UnresolvedAlias` of `SELECT udf(a)`, or a by-name parent reference like the `r` of `SELECT r + 1 FROM (SELECT udf(a) AS r ...)` -- is left behind and `C heckAnalysis` reports the fall-back itself as an error. The drop rule therefore also re-runs `ResolveReferences`, `ResolveAliases` and `ResolveAggregateFunctions` on exactly the plans it changed, in Resolution-batch order, with the batch placed before "Remove TempResolvedColumn" so the `TempResolvedColumn` machinery that re-run may create is still cleaned up by that batch. ### Why are the changes needed? 1. Python and Catalyst promote numeric types differently, so transpiled operations could raise `ARITHMETIC_OVERFLOW` on intermediates the interpreted UDF carries without noticing (`udf(lambda x: x * 100)` on a tinyint column, `abs` of a width's minimum), or silently diverge from the interpreted UDF (e.g. `x ** 3` rounds twice on a double; `a // b` needs integral input). The narrower categories pick the exact lowering per operand kind, and the promotion expressions make the *result* match Python where the lowering already exists. 2. An option that matches its category but never resolves killed the query at `CheckAnalysis` instead of falling back to Python -- for the most common shape, `df.select(udf(col))`, as an `INTERNAL_ERROR` on the unresolved alias once the option was gone. The failure mode the SPARK-59090 half removes, including for the enclosing aliases and parent references above. ### Does this PR introduce _any_ user-facing change? Transpilation (experimental, off by default) is closer to Python's semantics, and an unresolvable transpiled option now falls back to interpreted Python instead of failing the query. No change when transpilation is disabled. ### How was this patch tested? New hypothesis and unit tests for the numeric promotion (`PythonNumericPromotionSuite`, extended `ResolveTranspiledPythonUDFOptionsSuite`), a full-analysis test pinning the SPARK-59090 drop behavior, and three more for the parent resolution: the unaliased call, the by-name parent reference, and a corner case pinning that an invalid `ORDER BY` in the same query as a dropped option still fails with `UNRESOLVED_COLUMN` rather than being resolved through the re-run. Because the drop rule shares batches with the aggregate machinery, the analysis suites that pin that machinery were re-run locally (JDK 21): | Suite | Result | |---|---| | `ResolveTranspiledPythonUDFOptionsSuite` | 27/27 | | `AnalysisErrorSuite` + `AnalysisSuite` | 205/205 | | `LateralColumnAliasSuite`, `AnalyzerExtensionPropagationSuite` | all pass | | `SQLQueryTestSuite` (golden files incl. group-by / order-by / having / postgreSQL) | 847/847 | | `PythonNumericPromotionSuite` | all pass | | `test_udf_transpile_unit` / `_parity` / `_hypothesis` (RUN_HYPOTHESIS=1, 200 examples) | all pass | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude 5 Generated-by: Kimi (kimi-k3, via Cursor) -- 2026-09-29 rebase onto upstream/master, conflict resolution, array-of-atomic return-type gate Generated-by: GLM 5.2 (Cursor) -- folding SPARK-59090 (drop-unresolved-options) into this PR Generated-by: GLM 5.2 (Cursor) -- 2026-10-04 rebase onto upstream/master, preflight, finishing for review Generated-by: GLM 5.3 (Cursor) -- 2026-10-22 review fixes: scoped post-drop parent resolution, repeat narrowing, custom-transpiler compat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
