cloud-fan commented on issue #24768: [SPARK-27919][SQL] Add v2 session catalog URL: https://github.com/apache/spark/pull/24768#issuecomment-504859851 > After this PR, the default catalog -- if set -- it is responsible for all table names without an explicit catalog. I'm OK with it as a path to the final shape: default catalog is applied everywhere, including tables, permanent views, functions, etc. The thing I'm worried about is how to approach it. The current catalog stuff(without DS v2) in Spark is organized as: 1. every session has a singleton `SessionCatalog`, which deals with some internal management(like temp view management, table relation cache, etc.). It also tracks the current database of this session, so that different session can have different default database. 2. `SessionCatalog` forwards all other requests to a global singleton `ExternalCatalog`. In my understanding, the new v2 catalog(`TableCatalog`) is to replace `ExternalCatalog` only, not `SessionCatalog` + `ExternalCatalog`. We still want a place in Spark to deal with internal management like temp view, and that is `SessionCatalog`. After DS v2 is completed, I expect the catalog stuff to be organized as: 1. every session has a singleton `SessionCatalog`, which deals with some internal management, like temp view management, and track the current default catalog of this session. 2. `SessionCatalog` forwards other requests to different v2 catalogs, according to the identifier. If the identifier has a catalog part, forward requests to that registered v2 catalog. Otherwise, forward to the default catalog. The `SessionCatalog` is not a federal catalog, we can rename it to `CatalogProxy` if we want.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
