jzhan-2026 opened a new pull request, #59814:
URL: https://github.com/apache/spark/pull/59814
### What changes were proposed in this pull request?
With `spark.sql.crossJoin.enabled=false`, `ConvertToCatalyst` now keeps a
transpiled Python UDF interpreted when it is the only condition tying a join's
two sides together. `CheckCartesianProducts` then fails the query exactly as it
does with transpilation off.
The new `keepCartesianGuard` step runs on each operator before conversion:
- It looks at a `Join` condition, and at a `Filter` above a join
(`join(...).where(udf(...))`, or a comma join in SQL). Predicate pushdown later
moves that Filter into the join.
- It only looks at the join types `CheckCartesianProducts` checks: `Inner`,
`LeftOuter`, `RightOuter`, `FullOuter`. An explicit `Cross` join is left alone.
- A join counts as unkeyed when no conjunct spans both sides, apart from
conjuncts that hold a cross-side Python UDF. Interpreted, those are pulled out
of the join by `ExtractPythonUDFFromJoinCondition`, so they are no key either
way.
- For an unkeyed join, only the cross-side call goes back to Python. A
nested call that reads one side is still lowered.
When something else keys the join (e.g. `a == b AND udf(a, b)`), or cross
joins are allowed, the call is lowered as before.
### Why are the changes needed?
Repro (from the JIRA, with Arrow UDFs off so the UDF is batched):
```python
spark.conf.set("spark.sql.crossJoin.enabled", "false")
same = lambda a, b: a == b
eq = udf(same, BooleanType())
left.join(right, eq("a", "b")).collect()
```
- Transpilation off: `AnalysisException: Detected implicit cartesian
product`.
- Transpilation on (before this PR): no error. The plan is a
`CartesianProduct`.
`CheckCartesianProducts` only asks whether some conjunct references both
sides. Interpreted, the UDF is pulled out of the join into a Filter, so the
join has no condition and the check fires. Lowered, the UDF becomes a native
`CASE WHEN ...` that references both sides, so the check passes. But no join
key can be extracted from it, so the planner still picks a `CartesianProduct`.
A user who disabled cross joins to catch accidental quadratic joins silently
lost that protection when they enabled transpilation.
### Does this PR introduce _any_ user-facing change?
Yes, only for the experimental
`spark.sql.experimental.optimizer.transpilePyUDFs`, which is unreleased. With
cross joins disabled, a Python UDF that is the only cross-side join condition
fails with the cartesian product error again, matching transpilation off.
Nothing changes when `spark.sql.crossJoin.enabled=true` (the default).
### How was this patch tested?
- New tests in `ConvertToCatalystSuite`:
- The UDF stays Python for all four checked join types, and for a Filter
above a join.
- It is still lowered when the join has another key, when cross joins are
allowed, for an explicit `Cross` join, and for a call that reads one side.
- A one-side call nested in a cross-side call is still lowered.
- New `test_udf_transpile_keeps_the_cross_join_guard` in
`test_udf_transpile_unit.py`, end to end:
- The join-condition and `where` forms both raise the cartesian product
error.
- A keyed join is still fully lowered, with no `EvalPython`.
- `build/sbt 'catalyst/testOnly *ConvertToCatalystSuite'`: 42 passed.
- `python/run-tests --testnames "pyspark.sql.tests.test_udf_transpile_unit
UDFTranspileUnitTests.test_udf_transpile_keeps_the_cross_join_guard"`: passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
