lisancao opened a new pull request, #59683: URL: https://github.com/apache/spark/pull/59683
### What changes were proposed in this pull request? Two things to `docs/sql-v2-data-sources.md`, the page on writing a Data Source V2 connector. **Fixes three statements that no longer match the API** - The `TableCatalog` method table lists `createTable(ident, columns, partitions, properties)`, deprecated since 4.1.0. The current contract is `createTable(Identifier, TableInfo)`, which is the overload the create, replace and CTAS execs call. - The `ViewCatalog` section links `TableViewCatalog`, which does not exist. The combined interface is `RelationCatalog` (4.2.0), and `Catalogs.load` rejects a plugin that implements `TableCatalog` and `ViewCatalog` without it. - The `ScanBuilder` section says each pushdown mix-in "returns the operators it can handle". `pushFilters` returns the filters Spark must still evaluate after the scan, and `pushedFilters()` reports what was accepted; the limit, top-N, offset and aggregate mix-ins return a boolean. `SupportsPushDownV2Filters` is now presented as the preferred form. **Adds the walkthrough the page stopped short of** The page describes the interfaces one by one and ends there. The new "Building a Connector" section walks a small CSV connector in the order an author writes it: declaring the table and its capabilities (and what happens when one is missing, since Spark checks the set before calling any builder), the read path with `InputPartition` and a reader producing `InternalRow`, pushdown and how to confirm it reached the connector in `EXPLAIN`, the write path and the commit protocol, registering through a catalog with the `spark.sql.catalog.<name>.<key>` passthrough, and running outside `spark-submit` on JDK 17. A "Migrating from Data Source V1" section maps each V1 interface to its V2 replacement and covers the two structural changes: parallelism moves to `InputPartition`, and a single `insert` call becomes a commit protocol. ### Why are the changes needed? The three statements send a connector author down the wrong path (a deprecated overload, a class that does not exist, pushdown described backwards). Beyond that, the runtime contracts an author hits first are not written down anywhere: which `TableCapability` unlocks which operation, `InternalRow` wanting `UTF8String` rather than `String`, commit messages being collected on the driver, how to see pushdown in the plan. ### Does this PR introduce _any_ user-facing change? Docs only. ### How was this patch tested? Every Java block is code from a connector that was compiled with `-Xlint:all` against the Spark 4.3.0 distribution on JDK 17 and run through its own suite; the quoted errors and the `EXPLAIN` output are what it produced. Each claim in the new sections was reproduced with a small probe connector (table without `BATCH_READ` through a provider and through a catalog, reader emitting `java.lang.String`, non-serializable `InputPartition`, overwrite with and without `TRUNCATE`, catalog option passthrough, plain `main()` with and without the launcher's module options). Docs built with `SKIP_API=1 SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 SKIP_ERRORDOC=1 bundle exec jekyll build` and the page checked in `_site/`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code. All examples were compiled and run by hand as described above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
