Github user cloud-fan commented on a diff in the pull request:
    --- Diff: 
    @@ -131,8 +131,20 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] 
with PredicateHelper {
    -  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
    -    case plan: LogicalPlan => extract(plan)
    +  def apply(plan: LogicalPlan): LogicalPlan = plan match {
    +    // SPARK-26293: A subquery will be rewritten into join later, and will 
go through this rule
    +    // eventually. Here we skip subquery, as Python UDF only needs to be 
extracted once.
    +    case _: Subquery => plan
    --- End diff --
    I think you have a point here. If subquery will be converted to join, why 
do we need to optimize subquery ahead?
    Anyway, that's something we need to discuss later. cc @dilipbiswal for the 
subquery question.


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to