Hyukjin Kwon created SPARK-60725:
------------------------------------
Summary: Write the columnar batches of a query to native data
sources without converting them to rows
Key: SPARK-60725
URL: https://issues.apache.org/jira/browse/SPARK-60725
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A follow-up to SPARK-60101 (https://github.com/apache/spark/pull/59424). When
the query of a batch or micro-batch write to a native data source produces
columnar batches, such as a scan of a native data source, Spark converts them
to rows with ColumnarToRowExec, and the data writer converts the rows back to
Arrow. Instead, the data writers should take the batches, copying their Arrow
buffers, or converting their other columns to Arrow value by value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
