[ 
https://issues.apache.org/jira/browse/SPARK-60325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh reassigned SPARK-60325:
-----------------------------------

    Assignee: L. C. Hsieh

> Remove the holding of input batches from columnar Python UDFs
> -------------------------------------------------------------
>
>                 Key: SPARK-60325
>                 URL: https://issues.apache.org/jira/browse/SPARK-60325
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps 
> the pass-through columns of each input batch until its result comes back from 
> Python, many batches later. SPARK-60294 disabled 
> spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because 
> a producer may release or overwrite the vectors it returned before then, e.g. 
> a reader that refills the same vectors for every batch, the Arrow cache, or 
> the Python output of a stacked ArrowEvalPythonExec.
> ColumnVector does not promise that a vector stays valid after the next batch 
> is requested, and only the holding of the pass-through columns needs that: 
> the UDF inputs of a batch are written to Python before the next batch is 
> requested. This ticket removes the Arrow columnar path, so that columnar 
> input always copies the pass-through columns into the row queue and joins the 
> results by row. The results of Python can then be sliced with the output 
> batch limits, and the conversion of the joined rows back to batches supports 
> NullType and UDT columns, a non-positive maxRecordsPerBatch, and 
> maxBytesPerBatch. The configuration stays disabled by default.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to