lisancao commented on PR #59683: URL: https://github.com/apache/spark/pull/59683#issuecomment-6328723562
> I'd keep "Building a Connector": a task-oriented walkthrough adds value beyond the interface reference. Could we shorten it around four steps? > > 1. Minimal batch read: provider -> table -> scan -> partitions -> reader, including internal value types. > > 2. Optional projection and filter pushdown, emphasizing the residual-filter contract. > > 3. Batch writes: required capabilities, builder, task writers, and driver commit. > > 4. Optional catalog integration for SQL identifiers and DDL. > > > To keep that focused, move V1 migration and IDE/JDK troubleshooting to separate pages or appendices, and link to the full capability reference instead of repeating it. The migration section also repeats partition planning, the commit protocol, and filter handling. > > Could we shorten the snippets by linking to Spark's existing test connectors: [JavaSimpleDataSourceV2](https://github.com/apache/spark/blob/master/sql/core/src/test/java/test/org/apache/spark/sql/connector/JavaSimpleDataSourceV2.java) for reads, [JavaAdvancedDataSourceV2WithV2Filter](https://github.com/apache/spark/blob/master/sql/core/src/test/java/test/org/apache/spark/sql/connector/JavaAdvancedDataSourceV2WithV2Filter.java) for pushdown, and [JavaSimpleWritableDataSource](https://github.com/apache/spark/blob/master/sql/core/src/test/java/test/org/apache/spark/sql/connector/JavaSimpleWritableDataSource.java) for writes, alongside their tests in [DataSourceV2Suite](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala)? These use shared test helpers, but a short explanation would let this page focus on the DSV2 contracts and avoid introducing another partial connector with undefined helpers. The CSV I/O boiler plate, exception text, and physical-plan output could also be shortened. I made the relevant changes, could you take another pass to make sure it fits what you were thinking? Thanks, really appreciate the corrections and you looking this over so quickly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
