HyukjinKwon opened a new pull request, #59932: URL: https://github.com/apache/spark/pull/59932
### What changes were proposed in this pull request? Two follow-ups to SPARK-60101 (#59424) for `spark_datasource.h`, from @gengliangwang's review in https://github.com/apache/spark/pull/59424#issuecomment-6480975990. This PR is based on #59931 (version 1 of the interface), whose commit it includes until that PR is merged. 1. **The order of the calls on a handle**, as a convention of the header: - After a function on a handle fails, Spark calls no other function on the handle than its close function. The exceptions: it aborts a writer whose commit failed, and a data writer whose `write` or commit failed. - Spark calls each function of a batch reader at most once. - Spark calls commit and abort at most once each for a writer, and `commit_data_writer` and `abort_data_writer` at most once each for a data writer, after all the calls to `write`. This documents what Spark already does, so that SDKs do not need to guard against other orders. 2. **`SparkDsHost.is_log_enabled`**, appended to the struct, tells whether Spark logs a level. A library can skip formatting the messages that Spark would not write, and follow the log level of Spark, for example to log at debug only when Spark does. `NativeLibrary` implements it with an upcall to the logger of the native data sources. The docs mention it. The third item of that review, column pruning for streaming reads, is left out. Micro-batch execution asks every streaming scan for all its columns (`MicroBatchExecution`: "TODO: operator pushdown"), so the adapter of native data sources has no pruned schema to pass. ### Why are the changes needed? SDKs such as the Rust prototype in #59520 have to know which call orders they can rely on. Without `is_log_enabled`, a library formats every message, even those that Spark drops, and cannot follow the log level of Spark. ### Does this PR introduce _any_ user-facing change? No for released versions: native data sources are new in the unreleased 4.4.0. `is_log_enabled` is appended to `SparkDsHost`, so it does not change version 1 for libraries that do not use it. ### How was this patch tested? "pass the version of Spark, and log with the log of Spark" in `NativeDataSourceSuite` now also checks `is_log_enabled`. The test library writes whether Spark logs each level, which the test checks with the log of `NativeLibrary` at the levels DEBUG and ERROR. The test library also skips the levels that Spark does not log. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
