Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 )
Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients ...................................................................... Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/20367/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20367/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1006 PS2, Line 1006: msTbl > How do we deal with the case when table is renamed through external source( If the rename event happened outside of metastore, how do you expect it sync to catalogD? CatalogD can deal with events present in the metastore. http://gerrit.cloudera.org:8080/#/c/20367/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1007 PS2, Line 1007: tbl = catalog_.updateMetastoreTable(msTbl, tbl); > We don't seem to reload table ? Probably table schema, partitions would hav We don't have to reload entire hdfs table for 2 reasons. 1) It's a very costly operation to reload hdfs tables. (even during alter table query from impala, we only reload what is really required in the table like table schema or file schema). 2) We always access the metastore table from the cached impala representation of the table, so we would apply the current change on top of the latest metastore's table and reload the part that is really required. All in all, I believe reloading the table without knowing what changed is too aggressive and it would have complications in the performance. http://gerrit.cloudera.org:8080/#/c/20367/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1011 PS2, Line 1011: long oldCatalogVersion = tbl.getCatalogVersion(); > I believe, we need to get the oldCatalogVersion before updating tbl. Becaus Ack http://gerrit.cloudera.org:8080/#/c/20367/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2398 PS2, Line 2398: long newCatalogVersion = catalog_.incrementAndGetCatalogVersion(); > It is good to get the newCatalogVersion after catalog_.updateMetastoreTable Ack -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Mon, 16 Oct 2023 18:49:42 +0000 Gerrit-HasComments: Yes
