Hyukjin Kwon created SPARK-60726:
------------------------------------
Summary: Add a developer API to read a partition of a native data
source as an Arrow C stream
Key: SPARK-60726
URL: https://issues.apache.org/jira/browse/SPARK-60726
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A follow-up to SPARK-60101 (https://github.com/apache/spark/pull/59424), as
discussed in https://github.com/apache/spark/pull/59424#issuecomment-6196064644
and https://github.com/apache/spark/pull/59424#issuecomment-6204557249. An
engine that processes Arrow data in native code, such as Gluten or Comet,
should be able to read a partition of a batch scan of a native data source as
an Arrow C stream, through a developer API on the reader factory, instead of
the data that Spark imports on the JVM and converts to rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
