[
https://issues.apache.org/jira/browse/SPARK-59792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59792:
-----------------------------------
Labels: pull-request-available (was: )
> Skip ColumnarToRow for Arrow-backed input to Python UDTFs
> ---------------------------------------------------------
>
> Key: SPARK-59792
> URL: https://issues.apache.org/jira/browse/SPARK-59792
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 5.0.0
> Reporter: binwei yang
> Priority: Major
> Labels: pull-request-available
>
> SPARK-56350 lets ArrowEvalPythonExec accept Arrow-backed ColumnarBatch input,
> so Arrow Python UDFs no longer pay a columnar -> row -> columnar round-trip
> (ColumnarToRowExec, then ArrowWriter) when the child already produces Arrow
> data.
> Arrow-optimized Python UDTFs (ArrowEvalPythonUDTFExec, used for
> SQL_ARROW_TABLE_UDF and SQL_ARROW_UDTF) still always go through
> ColumnarToRowExec and re-encode their arguments row by row with ArrowWriter,
> even when the child is an Arrow-backed DataSource V2 scan.
> Proposal: apply the same approach to ArrowEvalPythonUDTFExec.
> * When the child supports columnar output and all the UDTF arguments are
> columns of the child, read the columnar child directly and serialize the
> argument columns to the Python worker from their Arrow vectors (reusing
> ColumnarArrowPythonInput from SPARK-56350).
> * A UDTF returns any number of rows per input row (plus the rows of
> terminate()), so its output is still joined row by row with the input rows;
> only the child columns in the output need to be buffered.
> * UDTF arguments that are not child columns (literals, expressions) keep the
> existing row-based path.
> * Guarded by a new conf,
> spark.sql.execution.arrow.pythonUDTF.columnarInput.enabled (default true).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
