L. C. Hsieh created SPARK-61445:
-----------------------------------

             Summary: Pass through only the required child columns in Python 
UDF evaluation and return Python results directly in columnar mode
                 Key: SPARK-61445
                 URL: https://issues.apache.org/jira/browse/SPARK-61445
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: L. C. Hsieh


EvalPythonExec (BatchEvalPython / ArrowEvalPython) always outputs child.output 
++ results, so every column of its child, including the UDF arguments, is 
copied into a HybridRowQueue and joined with the results, even when the parent 
drops it, e.g. df.select(udf(a)). With columnar input 
(spark.sql.execution.arrow.pythonUDF.columnarInput.enabled), the joined rows 
are also converted back to columnar batches.

Like the Python UDTF nodes, the evaluation should pass through only the child 
columns that its parent uses (requiredChildOutput, narrowed by ColumnPruning). 
When no column is passed through, the row path needs no queue, and the columnar 
path can return the result batches of Python directly.

This is a follow-up of SPARK-60325 (https://github.com/apache/spark/pull/59582).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to