Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 )
Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients ...................................................................... Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG@8 PS1, Line 8: Impala clien > I think that impala shell is misleading here, as any client initiate DDLs. Ack http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG@10 PS1, Line 10: by Impala, it al > Same as for the title, IMO "performed by Impala" would be better. Ack http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG@14 PS1, Line 14: Impala clien > same as for the title Ack http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG@20 PS1, Line 20: HIVE-27499 > Do you think that this feature can practically work before HIVE-27499? My Before HIVE-27499, we'll always fetch the HMS metadata no matter the event processor is lagging or not. Unfortunately, It does add some overhead. But this overhead ensures consistency. Yeah, any thing can happen between RPC call and processing of DDL, as it happens today and we cannot do anything about it. http://gerrit.cloudera.org:8080/#/c/20367/1//COMMIT_MSG@24 PS1, Line 24: Set enable_sync_to_latest_event_on_ddls to true to use this feature. > This flag is not exposed yet, we can only set it in tests. This is global flag https://github.com/apache/impala/blob/master/be/src/catalog/catalog-server.cc#L121 and by default it is disabled. http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1369 PS1, Line 1369: the table with the given metastore table > This function seems to be about tables, not databases. Ack http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1012 PS1, Line 1012: // Get a new catalog version to assign to the table being altered. > Calling an HMS RPC when the global catalog version lock is locked is proble Valid point. We should always do RPC call before locking the catalog version. http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1488 PS1, Line 1488: impala shell > We are not using the shell here (or any Impala client) here, right? My unde Ack. I'm just simulating what would happen once the request lands on Impala's catalog OpExecutor. http://gerrit.cloudera.org:8080/#/c/20367/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1498 PS1, Line 1498: BackendConfig.INSTANCE.setHMSPollingIntervalInSeconds(10); > Will this have any effect? We seem check this only once, when the event pro Ah!! I thought this would take into effect. Is there any other way to make the event processor lagging in the Java unit tests? We would want to this in Java tests because we need to verify loaded table in java. There is no way we could verify this in end-to-end python tests. -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Tue, 05 Sep 2023 19:46:27 +0000 Gerrit-HasComments: Yes
