aokolnychyi commented on PR #56039: URL: https://github.com/apache/spark/pull/56039#issuecomment-4608471188
Let me share my thoughts: 1. Using I am worried about using `CatalogAndIdentifier` in the current way as it is too broad. It applies to functions, procedures, etc. We only care about tables in this case. 2. Our `SupportsNamespaces` guard is not reliable. Even its doc says the class is optional. This means table catalogs can load tables within namespaces without implementing this trait. 3. The `SupportsCatalogOptions` was designed for `DataFrame` writer/reader cases, we adapt it for use in SQL on path. Do we actually have to teach Spark to resolve SQL on path in DSv2? In theory, connectors like Delta and already override the session catalog (`spark_catalog`), so can't we simply support path identifiers within the existing catalog APIs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
