aokolnychyi commented on code in PR #56039:
URL: https://github.com/apache/spark/pull/56039#discussion_r3320619249
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala:
##########
@@ -135,7 +135,18 @@ private[sql] trait LookupCatalog extends Logging {
Some((catalog, ident))
} catch {
case _: CatalogNotFoundException =>
- Some((currentCatalog, nameParts.asIdentifier))
+ // No catalog matched. As a fallback, try path-based data sources:
+ // formats implementing SupportsCatalogOptions (e.g.
`pathformat.`/path/to/t``)
+ // route to the catalog the connector designates. If no SCO format
claims the
+ // identifier head, fall through to currentCatalog and let later
analysis raise
+ // table-not-found. This matches the v1 file-format precedence
(catalog first,
+ // path-based as fallback).
+
Option(catalogManager.catalogAndIdentForDataSource(nameParts)).flatten match {
Review Comment:
I think this introduces a slight but problematic behavior change.
Say a user has a database named `delta` in their session catalog with a
table `orders`.
```
SELECT * FROM delta.orders
```
**Before:** resolves to (session_catalog, ns=["delta"], name="orders") →
finds the table correctly.
**After**: if the delta data source implements SupportsCatalogOptions, the
SCO resolver intercepts, calls extractIdentifier with options derived from
{"path": "orders"}, and routes to whatever catalog extractCatalog returns.
It seem like we should only pick the SQL on file approach if we KNOW / TEST
that the current catalog can't resolve the requested namespace / identifier
pair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
