[
https://issues.apache.org/jira/browse/SPARK-57616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57616:
-----------------------------------
Labels: pull-request-available (was: )
> Extend SupportsCatalogOptions to let DSv2 sources configure path-based write
> behavior
> -------------------------------------------------------------------------------------
>
> Key: SPARK-57616
> URL: https://issues.apache.org/jira/browse/SPARK-57616
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Johan Lasperas
> Priority: Minor
> Labels: pull-request-available
>
> [{{SupportsCatalogOptions}}|https://github.com/apache/spark/blob/8c51482a325774451c2e9c0134efb8254b3ee9fc/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsCatalogOptions.java#L35]
> allows DSv1 sources to identify a catalog to be used to resolve tables
> accessed by path, and route read and write operations through DSv2.
> Delta will be migrating from a file-based data source (DSv1) for path based
> accesses to DSv2 using {{{}SupportsCatalogOption{}}}.
> Two blockers are present:
> * The migration can't be a static on/off based on the delta
> {{TableProvider}} extending {{SupportsCatalogOptions}} or not. it will be a
> dynamic decision based on a flag to provide a migration and possibility to
> disable the new behavior if needed.
> * Delta, as a file-based source, allows
> {{df.write.format(<format>).mode("append"/"overwrite").save()}} when the
> table doesn't exist yet and will create it. This behavior must be preserved.
> {{{}{}}}The SupportsCatalogOptions should be extended to allow these use cases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
