L. C. Hsieh created SPARK-61445:
-----------------------------------
Summary: Pass through only the required child columns in Python
UDF evaluation and return Python results directly in columnar mode
Key: SPARK-61445
URL: https://issues.apache.org/jira/browse/SPARK-61445
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: L. C. Hsieh
EvalPythonExec (BatchEvalPython / ArrowEvalPython) always outputs child.output
++ results, so every column of its child, including the UDF arguments, is
copied into a HybridRowQueue and joined with the results, even when the parent
drops it, e.g. df.select(udf(a)). With columnar input
(spark.sql.execution.arrow.pythonUDF.columnarInput.enabled), the joined rows
are also converted back to columnar batches.
Like the Python UDTF nodes, the evaluation should pass through only the child
columns that its parent uses (requiredChildOutput, narrowed by ColumnPruning).
When no column is passed through, the row path needs no queue, and the columnar
path can return the result batches of Python directly.
This is a follow-up of SPARK-60325 (https://github.com/apache/spark/pull/59582).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]