[
https://issues.apache.org/jira/browse/SPARK-60437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned SPARK-60437:
---------------------------------
Assignee: Lisa Cao
> Fix stale API references in the Data Source V2 docs page and add connector
> walkthrough and migration sections
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60437
> URL: https://issues.apache.org/jira/browse/SPARK-60437
> Project: Spark
> Issue Type: Improvement
> Components: Documentation
> Affects Versions: 4.3.0
> Reporter: Lisa Cao
> Assignee: Lisa Cao
> Priority: Major
> Labels: pull-request-available
>
> {{docs/sql-v2-data-sources.md}} (added for 4.2.0) has three statements that
> no longer match the API on master:
> # The {{TableCatalog}} method table lists {{createTable(ident, columns,
> partitions, properties)}}, which is {{@Deprecated(since = "4.1.0")}}. The
> current form is {{createTable(Identifier, TableInfo)}} ({{@since 4.1.0}}),
> and it is the overload {{CreateTableExec}}, {{ReplaceTableExec}} and the
> CTAS/RTAS execs call.
> # The {{ViewCatalog}} section links {{TableViewCatalog}}, which does not
> exist. The combined interface is {{RelationCatalog}} ({{@since 4.2.0}}), and
> {{Catalogs.load}} rejects a plugin that implements {{TableCatalog}} and
> {{ViewCatalog}} without it.
> # The {{ScanBuilder}} section says each pushdown mix-in "returns the
> operators it can handle". {{SupportsPushDownFilters.pushFilters}} returns the
> filters Spark must still evaluate after the scan, and {{pushedFilters()}}
> reports what was accepted; the limit/top-N/offset/aggregate mix-ins return a
> boolean. {{SupportsPushDownV2Filters}} is not presented as the preferred form.
> The page also stops at the interface level. There is no walkthrough of what a
> connector author actually implements, so the same runtime contracts come up
> repeatedly and none are documented: which {{TableCapability}} value unlocks
> which operation and what happens when one is missing, {{InternalRow}} holding
> {{UTF8String}} rather than {{String}}, {{WriterCommitMessage}} being
> collected on the driver, {{Scan.description()}} as the way to confirm
> pushdown in {{EXPLAIN}}, building {{Column[]}} from a {{StructType}} with
> {{Column.create}}, and the {{spark.sql.catalog.<name>.<key>}} passthrough
> with {{spark.sql.defaultCatalog}}.
> This issue fixes the three statements and adds two sections: "Building a
> Connector", which walks through a small CSV connector in the order an author
> writes it (declaring the table and its capabilities, the read path, pushdown
> and how to confirm it, the write path and the commit protocol, registering
> through a catalog, running outside {{spark-submit}}), and "Migrating from
> Data Source V1", with a V1 to V2 interface mapping and the structural changes
> (parallelism moves to {{InputPartition}}, writes become a commit protocol,
> {{unhandledFilters}} becomes the return value of {{pushFilters}}).
> Every example in the new sections was compiled and run against the Spark
> 4.3.0 binary distribution on JDK 17; the output and error messages shown are
> the ones produced.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
