Johan Lasperas created SPARK-57616:
--------------------------------------
Summary: Extend SupportsCatalogOptions to let DSv2 sources
configure path-based write behavior
Key: SPARK-57616
URL: https://issues.apache.org/jira/browse/SPARK-57616
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Johan Lasperas
[{{SupportsCatalogOptions}}|https://github.com/apache/spark/blob/8c51482a325774451c2e9c0134efb8254b3ee9fc/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsCatalogOptions.java#L35]
allows DSv1 sources to identify a catalog to be used to resolve tables
accessed by path, and route read and write operations through DSv2.
Delta will be migrating from a file-based data source (DSv1) for path based
accesses to DSv2 using {{{}SupportsCatalogOption{}}}.
Two blockers are present:
* The migration can't be a static on/off based on the delta {{TableProvider}}
extending {{SupportsCatalogOptions}} or not. it will be a dynamic decision
based on a flag to provide a migration and possibility to disable the new
behavior if needed.
* Delta, as a file-based source, allows
{{df.write.format(<format>).mode("append"/"overwrite").save()}} when the table
doesn't exist yet and will create it. This behavior must be preserved.
{{{}{}}}The SupportsCatalogOptions should be extended to allow these use cases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
