holdenk opened a new pull request, #59731:
URL: https://github.com/apache/spark/pull/59731
### What changes were proposed in this pull request?
A second, opt-in Python UDF transpilation target that lowers a UDF to
generated Java source rather than to Catalyst expressions. Enable it by naming
it after `catalyst` in the existing transpiler list:
```
spark.conf.set("spark.sql.experimental.optimizer.pyTranspilers",
"catalyst,java")
```
The default stays `catalyst`, so nothing changes unless asked for. New:
- `python/pyspark/sql/transpile_java.py` -- the lowering, in its own file,
on the `AbstractTranspiler` plugin seam that already exists.
- `TranspiledJavaUDF` -- an expression carrying the generated method source.
`doGenCode` splices it into whole-stage codegen via `ctx.addNewFunction`;
`eval` compiles the same source for the interpreted path, so the two cannot
disagree.
- `TranspiledJavaUDFHelpers` -- every Python/Java/ANSI disagreement lives
here: `//` and `%` signs, `/` on two ints (correctly rounded, not widened
twice), overflow, null propagation, null-safe equality, exact mixed int/float
comparison, `str * int` with its overflow bounds.
- `TranspiledJavaUDFBuilder` -- the Py4J entry; types cross as JSON like
every other Python-to-JVM hand-off.
- `ResolveTranspiledPythonUDFOptions` prunes options tagged with the new
`integral`/`fractional` categories -- the `java` target names one Java type per
parameter, so it splits `numeric`.
What it lowers that the Catalyst target cannot: locals, several statements,
early returns, `//`, `%`. Each argument is a method parameter, so a body
reading it N times still evaluates it once (SPARK-58626); unread arguments are
still evaluated, matching the interpreted UDF. Bodies it cannot reproduce
decline and fall back to interpreted Python (`and`/`or`, truthiness, loops, `==
None`, string methods, ...).
### Why are the changes needed?
The Catalyst target refuses any body with more than one top-level statement,
so most real-world UDFs -- anything with a local or an early return -- only
ever run interpreted, paying the Python worker round trip on every row. This
target buys reach over that boundary. It is not a speed play relative to
Catalyst: a Catalyst lowering stays preferable wherever it exists (the
optimizer can see into it), which is why `catalyst,java` tries Catalyst first.
### Does this PR introduce _any_ user-facing change?
Yes, opt-in only: `spark.sql.experimental.optimizer.pyTranspilers` gains a
`java` value. The default and all existing behaviour are unchanged; nothing
lowers differently unless the conf names the new target.
### How was this patch tested?
- `python/pyspark/sql/tests/test_udf_transpile_java_unit.py` -- lowering
tests asserting on the generated source, and end-to-end tests comparing against
interpreted Python (signs, zero, null, NaN/inf, the 2^53 boundary, error
classes).
- `TranspiledJavaUDFSuite` (both eval paths over hand-written bodies),
`TranspiledJavaUDFHelpersSuite` (CPython-checked operator pins), and new
`ResolveTranspiledPythonUDFOptionsSuite` cases for integral/fractional pruning.
- `TranspiledJavaUDFParityTests` re-runs the shared UDF mixin with
`catalyst,java` enabled.
- `dev/lint-scala`, scalafmt, the license check, and the diff-derived suites
(including the transpile hypothesis suite at 200 examples) pass locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (GLM 5.3)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
