aokolnychyi commented on pull request #1783: URL: https://github.com/apache/iceberg/pull/1783#issuecomment-732384116
A couple of questions before we try to come up with a solution. 1. How important or even valid is it to load non-path based tables through our source in Spark 3? We decided to interpret `path` option as a table identifier in 2.4 to avoid limitations of the DS V2 API at that point. Since we have proper catalog API in Spark 3, shall we assume all catalog-based tables should be loaded through it and not through source? We've replicated the `LookupCatalog` logic in Spark3Util but I am not sure it is a good idea to use that in the source. 2. It is a bit unclear to me how we are going to produce the catalog name from `SupportsCatalogOptions`. The catalog manager does not provide a list of catalogs that were registered. Will we require a catalog name in options? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
