HyukjinKwon commented on PR #59424: URL: https://github.com/apache/spark/pull/59424#issuecomment-6204557249
@zhouyuan Thank you, running a native data source under Gluten or Comet without copies is a good use of the C ABI. 2. **Threading contract**: agreed, and done in 534dfca15723. The header now says that the callbacks of the stream that `read` initializes can be called from any thread, but never concurrently, like the functions of `SparkDsHost` and `SparkDsTaskContext`, and the docs say so too. Spark itself still calls them from the thread of the task. 1. **A developer API to open a partition as an Arrow C stream**: this fits version 1 without changing the C ABI, since `read` already initializes a stream at an address that the caller allocates. I would rather add it in a follow-up, as a `@DeveloperApi` with its own review, than make it public in this PR, because a few things need deciding first: which object an engine gets it from; who owns the `SparkDsTaskContext`, and when it is freed; and what the engine must check before importing the arrays, which Spark checks today (sliced arrays, lengths shorter than the parent, the types it cannot read). After the planner split in 534dfca15723, from @gengliangwang's review, the reader factory of a batch scan only holds the location of the library, the name of the data source, the read schema and the state of the reader, which is what your `SupportsArrowStreamRead` needs. **Columnar write entry point**: also a follow-up. The data writers of the adapter already take Arrow-backed `ColumnarBatch`es, and an engine could pass an Arrow C stream to `write` directly in the same way. I'd be glad to have your review on both follow-ups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
