Github user gatorsmile commented on a diff in the pull request:
    --- Diff: 
    @@ -131,8 +131,20 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] 
with PredicateHelper {
    -  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
    -    case plan: LogicalPlan => extract(plan)
    +  def apply(plan: LogicalPlan): LogicalPlan = plan match {
    +    // SPARK-26293: A subquery will be rewritten into join later, and will 
go through this rule
    +    // eventually. Here we skip subquery, as Python UDF only needs to be 
extracted once.
    +    case _: Subquery => plan
    --- End diff --
    Basically, we want to ensure this rule is running once and only once. In 
the future, if we have another rule/function that calls 
Optimizer.this.execute(plan), this rule needs to be fixed again... We have a 
very strong hidden assumption in the implementation. This looks risky in the 
long term.
    The current fix is fine for backporting to 2.4. 


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to