Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r220437721
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1308,6 +1312,16 @@ object CheckCartesianProducts extends
Rule[LogicalPlan] with PredicateHelper {
if (SQLConf.get.crossJoinEnabled) {
plan
} else plan transform {
+ case j @ Join(_, _, _, condition)
+ if condition.isDefined &&
PullOutPythonUDFInJoinCondition.hasPythonUDF(condition.get) =>
+ // if the crossJoinEnabled is false, a RuntimeException will be
thrown later while
+ // the PythonUDF need to access both side of join, we throw
firstly here for better
+ // readable information.
+ throw new AnalysisException(s"Detected the join
condition:${j.condition} of this join " +
--- End diff --
The reason for can't be pulled out in the case
https://github.com/apache/spark/pull/22326#discussion_r220418201 is not because
of join type problem. The screenshot is I do a test to do the check by
`havePythonUDFInAllConditions`, and the test result shows it returned by
getting a RuntimeException even though its a inner join.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
