rdblue commented on issue #24768: [SPARK-27919][SQL] Add v2 session catalog URL: https://github.com/apache/spark/pull/24768#issuecomment-504134829 @cloud-fan, we need a v2 catalog that delegates to the session catalog. Long term, we want to move operations to v2 and remove paths that call directly to a singleton session catalog. In the short term, we hit a problem with the way CTAS works for v2 sources when there is no v2 catalog specified in the identifier. Here's a case that demonstrates the problem: ``` CREATE TABLE db.dest USING com.example.V2DataSource AS SELECT id, data FROM db.source; SELECT * FROM db.dest; -- FAIL: table not found! ``` Before this change, a v2 DataSource implementation would use the [default catalog for the table create](https://github.com/apache/spark/pull/24768). But there is no guarantee that the default catalog shares the table namespace with the session catalog. That's what causes the problem above: the default catalog is used for the CTAS, but the session catalog is used for the SELECT. The solution is to add a v2 catalog that delegates to the session catalog. That way, the v2 catalog used **is** the session catalog, just wrapped to implement the v2 API.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
