imback82 commented on issue #26684: [WIP][SPARK-30001][SQL] ResolveRelations should handle both V1 and V2 tables. URL: https://github.com/apache/spark/pull/26684#issuecomment-559282995 Thanks @cloud-fan and @rdblue for detailed explanation and suggestion! I will try to minimize changes in v1 code path in the next iteration. The fact that `V1Table` has the V1 provider should help a bit. There are few things we need to follow up: 1) _"If we make those changes, then we no longer need to run `ResolveTables` in `ResolveRelations`, which doesn't entirely work anyway because of the custom recursion."._ I can address this in this PR. 2) _"For Spark 3.0, I suggest we disallow view creation when any catalog other than the session catalog is the current catalog, or ensure that all tables are fully-qualified."_ I agree, and I can do a follow up PR to disallow this scenario. 3) _"Whether to return a V1 table is up to the session catalog implementation. That's where the TableProvider check should happen, so it can either return a V1Table for a v1 provider, or a v2 table for a v2 provider."_ The current implementation of `V2SessionCatalog` always returns `V1Table`. Should we update it to handle v2 providers?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
