[
https://issues.apache.org/jira/browse/SPARK-61445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-61445:
-----------------------------------
Assignee: L. C. Hsieh
> Pass through only the required child columns in Python UDF evaluation and
> return Python results directly in columnar mode
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61445
> URL: https://issues.apache.org/jira/browse/SPARK-61445
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> EvalPythonExec (BatchEvalPython / ArrowEvalPython) always outputs
> child.output ++ results, so every column of its child, including the UDF
> arguments, is copied into a HybridRowQueue and joined with the results, even
> when the parent drops it, e.g. df.select(udf(a)). With columnar input
> (spark.sql.execution.arrow.pythonUDF.columnarInput.enabled), the joined rows
> are also converted back to columnar batches.
> Like the Python UDTF nodes, the evaluation should pass through only the child
> columns that its parent uses (requiredChildOutput, narrowed by
> ColumnPruning). When no column is passed through, the row path needs no
> queue, and the columnar path can return the result batches of Python directly.
> This is a follow-up of SPARK-60325
> (https://github.com/apache/spark/pull/59582).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]