mgaido91 commented on a change in pull request #23769: [SPARK-26864][SQL] Query
may return incorrect result when python udf is used as a join condition and the
udf uses attributes from both legs of left semi join.
URL: https://github.com/apache/spark/pull/23769#discussion_r256284033
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
##########
@@ -180,8 +180,8 @@ object PullOutPythonUDFInJoinCondition extends
Rule[LogicalPlan] with PredicateH
override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case j @ Join(_, _, joinType, Some(cond), _) if
hasUnevaluablePythonUDF(cond, j) =>
- if (!joinType.isInstanceOf[InnerLike] && joinType != LeftSemi) {
- // The current strategy only support InnerLike and LeftSemi join
because for other type,
+ if (!joinType.isInstanceOf[InnerLike]) {
+ // The current strategy is to only support InnerLike join because for
other type,
Review comment:
nit: `The current strategy supports only InnerLike join because for other
types`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
