Lisa Cao created SPARK-60437:
--------------------------------
Summary: Fix stale API references in the Data Source V2 docs page
and add connector walkthrough and migration sections
Key: SPARK-60437
URL: https://issues.apache.org/jira/browse/SPARK-60437
Project: Spark
Issue Type: Improvement
Components: Documentation
Affects Versions: 4.3.0
Reporter: Lisa Cao
{{docs/sql-v2-data-sources.md}} (added for 4.2.0) has three statements that no
longer match the API on master:
# The {{TableCatalog}} method table lists {{createTable(ident, columns,
partitions, properties)}}, which is {{@Deprecated(since = "4.1.0")}}. The
current form is {{createTable(Identifier, TableInfo)}} ({{@since 4.1.0}}), and
it is the overload {{CreateTableExec}}, {{ReplaceTableExec}} and the CTAS/RTAS
execs call.
# The {{ViewCatalog}} section links {{TableViewCatalog}}, which does not exist.
The combined interface is {{RelationCatalog}} ({{@since 4.2.0}}), and
{{Catalogs.load}} rejects a plugin that implements {{TableCatalog}} and
{{ViewCatalog}} without it.
# The {{ScanBuilder}} section says each pushdown mix-in "returns the operators
it can handle". {{SupportsPushDownFilters.pushFilters}} returns the filters
Spark must still evaluate after the scan, and {{pushedFilters()}} reports what
was accepted; the limit/top-N/offset/aggregate mix-ins return a boolean.
{{SupportsPushDownV2Filters}} is not presented as the preferred form.
The page also stops at the interface level. There is no walkthrough of what a
connector author actually implements, so the same runtime contracts come up
repeatedly and none are documented: which {{TableCapability}} value unlocks
which operation and what happens when one is missing, {{InternalRow}} holding
{{UTF8String}} rather than {{String}}, {{WriterCommitMessage}} being collected
on the driver, {{Scan.description()}} as the way to confirm pushdown in
{{EXPLAIN}}, building {{Column[]}} from a {{StructType}} with
{{Column.create}}, and the {{spark.sql.catalog.<name>.<key>}} passthrough with
{{spark.sql.defaultCatalog}}.
This issue fixes the three statements and adds two sections: "Building a
Connector", which walks through a small CSV connector in the order an author
writes it (declaring the table and its capabilities, the read path, pushdown
and how to confirm it, the write path and the commit protocol, registering
through a catalog, running outside {{spark-submit}}), and "Migrating from Data
Source V1", with a V1 to V2 interface mapping and the structural changes
(parallelism moves to {{InputPartition}}, writes become a commit protocol,
{{unhandledFilters}} becomes the return value of {{pushFilters}}).
Every example in the new sections was compiled and run against the Spark 4.3.0
binary distribution on JDK 17; the output and error messages shown are the ones
produced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
