johanl-db opened a new pull request, #56617:
URL: https://github.com/apache/spark/pull/56617
### What changes were proposed in this pull request?
Extend `SupportsCatalogOption` DSv2 interface with:
- `useCatalogResolution`: dynamically toggles using the behavior offered by
`SupportsCatalogOption`. Needed to have a migration path from file-based source
(DSv1) to DSv2 + SupportsCatalogOption that's not just a static on/off
- `failIfTableDoesNotExist`: whether
`df.write.format(<format>).mode("append"/"overwrite").save()` should succeed
when the table doesn't exist.
### Why are the changes needed?
Delta will be migrating from a file-based data source (DSv1) for path based
accesses to DSv2 using `SupportsCatalogOption`.
Two blockers are present:
- The migration can't be a static on/off based on the delta `TableProvider`
extending `SupportsCatalogOptions` or not. it will be a dynamic decision based
on a flag to provide a migration and possibility to disable the new behavior if
needed.
- Delta, as a file-based source, allows
`df.write.format(<format>).mode("append"/"overwrite").save()` when the table
doesn't exist yet and will create it. This behavior must be preserved.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added tests in `SupportsCatalogOptionSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
