HyukjinKwon commented on PR #59424: URL: https://github.com/apache/spark/pull/59424#issuecomment-6207026653
@dongjoon-hyun @sarutak @gerashegalov You approved #59313, which this PR replaces, so the approvals there do not carry over here. Could you take another look? Since #59313, which this PR includes up to e3ff80c, the changes are: - The interface is a C header, `spark_datasource.h`, called through the FFM API on Java 22 or later instead of JNI (b9ad01ac93be), and shipped in the `include` directory of a distribution (4f032d5f1b9c). - Following the review here, a library exports only `spark_ds_init`, which returns a function table, with size-prefixed structs from Spark and the capabilities of each data source (d2df236bd234, b36ec5e73e27). - A data writer that committed is no longer aborted, and the native data source suites run in the `Java 25 build with Maven` job of pull requests (e5352e191c14). The docs say which sources native data sources fit (51af65c63a93). - Batch scans are planned on the driver, with a serializable reader factory for the executors, and the Data Source V2 table reports the batch writes that a data source supports, so an unsupported write fails while the query is analyzed (534dfca15723). The "Differences from #59313" table at the top of the description summarizes the interface. CI is green on 534dfca15723. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
