My case is DROP TABLE and DROP TABLE supports both v1 and v2 (as it simply works when I use custom catalog without replacing the default catalog).
It just fails on v2 when the "default catalog" is replaced (say I replace 'spark_catalog'), because TempViewOrV1Table is providing value even with v2 table, and then the catalyst goes with v1 exec. I guess all commands leveraging TempViewOrV1Table to determine whether the table is v1 vs v2 would all suffer from this issue. On Wed, Oct 7, 2020 at 5:45 PM Wenchen Fan <cloud0...@gmail.com> wrote: > Not all the DDL commands support v2 catalog APIs (e.g. CREATE TABLE LIKE), > so it's possible that some commands still go through the v1 session catalog > although you configured a custom v2 session catalog. > > Can you create JIRA tickets if you hit any DDL commands that don't support > v2 catalog? We should fix them. > > On Wed, Oct 7, 2020 at 9:15 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> The logical plan for the parsed statement is getting converted either for >> old one or v2, and for the former one it keeps using an external catalog >> (Hive) - so replacing default session catalog with custom one and trying to >> use it like it is in external catalog doesn't work, which destroys the >> purpose of replacing the default session catalog. >> >> Btw I see one approach: in TempViewOrV1Table, if it matches >> with SessionCatalogAndIdentifier where the catalog is TableCatalog, call >> loadTable in catalog and see whether it's V1 table or not. Not sure it's a >> viable approach though, as it requires loading a table during resolution of >> the table identifier. >> >> On Wed, Oct 7, 2020 at 10:04 AM Ryan Blue <rb...@netflix.com> wrote: >> >>> I've hit this with `DROP TABLE` commands that should be passed to a >>> registered v2 session catalog, but are handled by v1. I think that's the >>> only case we hit in our downstream test suites, but we haven't been >>> exploring the use of a session catalog for fallback. We use v2 for >>> everything now, which avoids the problem and comes with multi-catalog >>> support. >>> >>> On Tue, Oct 6, 2020 at 5:55 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>>> Hi devs, >>>> >>>> I'm not sure whether it's addressed in Spark 3.1, but at least from >>>> Spark 3.0.1, many SQL DDL statements don't seem to go through the custom >>>> catalog when I replace default catalog with custom catalog and only provide >>>> 'dbName.tableName' as table identifier. >>>> >>>> I'm not an expert in this area, but after skimming the code I feel >>>> TempViewOrV1Table looks to be broken for the case, as it can still be a V2 >>>> table. Classifying the table identifier to either V2 table or "temp view or >>>> v1 table" looks to be mandatory, as former and latter have different code >>>> paths and different catalog interfaces. >>>> >>>> That sounds to me as being stuck and the only "clear" approach seems to >>>> disallow default catalog with custom one. Am I missing something? >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >>
