kbendick commented on issue #3541: URL: https://github.com/apache/iceberg/issues/3541#issuecomment-972279217
Hi @pan3793 - Thanks for reporting this! I spent some time looking into this, and it doesn't seem as though Spark exposes `CASCADE` to the interfaces that external catalogs are supposed to implement. In fact, their javadocs _do_ say that the default behavior is to `CASCADE`, but `SupportsNamespace` doesn't capture any of the information related to cascading from `DropNamespaceExec`. Even in their tests for external catalogs, they only have a test that throws if the namespace isn't empty: https://github.com/apache/spark/blob/5c0762b5d2004464c36df353a4d483f8f57eebeb/sql/[…]/spark/sql/execution/datasources/v2/V2SessionCatalogSuite.scala That's sort of what they're implying in the comment: ``` // The default behavior of `SupportsNamespace.dropNamespace()` is cascading, // so make sure the namespace to drop is empty. ``` Even their implementation `DropNamespaceExec` (which you posted here) doesn't seem to handle `CASCADE` behavior at all which is why they actually check and throw vs list and drop. And poking around the query planner, it doesn't seem to be the case that any instantiated `DropNamespace` node actually causes cascading either, at least for V2Catalog interfaces. I'm going to poke around a bit more to see if there is an easy way to get at the `CASCADE` information - but I think that their documentation is a little off, specifically for external catalogs. I will get back and confirm that. Since it's not exposed in Spark's `SupportsNamespace` interface presently, it will probably a little more difficult to get this behavior until it can be fixed upstream. But hopefully I'm wrong and it's a simple change! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
