L. C. Hsieh created SPARK-61767:
-----------------------------------
Summary: Extract the batch export of the in-process Python UDF
evaluator into InProcessArrowBatchExporter
Key: SPARK-61767
URL: https://issues.apache.org/jira/browse/SPARK-61767
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches (SPARK-59718)
interleaves three things in one method of about 300 lines: pulling input rows
under the iterator's lock, building and exporting Arrow batches to Python, and
releasing both at task completion.
This moves the second into a class of its own, InProcessArrowBatchExporter,
which owns the Arrow writer of the current batch, the imported result vectors,
and the registrations of the UDFs. evaluateBatches keeps the orchestration: it
pulls rows into the queue and the exporter until the batch is full, evaluates
it, and joins the results.
This is a refactoring without behavior change. The exporter can be tested
without a Python interpreter.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]