Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20648 )
Change subject: IMPALA-10987: Changing impala.disableHmsSync in Hive should not break event processing ...................................................................... Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/20648/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20648/1//COMMIT_MSG@22 PS1, Line 22: te l > Ack jumping back to this topic: My impression is that this goes against what disabling event sync tries to achieve. I see 2 reasons for disabling it: 1. performance (avoiding event processing to be blocked due to reloads triggered by events) 2. wanting old behavior (table metadata snapshots can be manually controlled by IM/REFRESH) Invalidating the table goes against both 1 and 2. If the table is actively queried by Impala and modified by some other component at the same time, then this will lead to constantly invalidating and reloading it. >Now, if we were to access this table from Impala, we would access the older >table I think that this is the feature here, not a bug :) So while I agree with invalidating the table in the reenable case, if hms sync is disabled, then we should totally ignore the event. If the user wants to have fresh metadata, then they should keep HMS sync enabled, while if they http://gerrit.cloudera.org:8080/#/c/20648/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20648/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1556 PS3, Line 1556: wasEventSyncTurnedOn > In the above scenario, first (2) is executed, and while reloading table met ah, I see, we will see 1 anyway as: - 2's DDL leads to reloading the table - if the table is invalidated + loaded after 2, the new load will see the fresh state http://gerrit.cloudera.org:8080/#/c/20648/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20648/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@808 PS4, Line 808: instanceof HdfsTable What should happen with HBase/Kudu tables? We just want to ignore incomplete tables, right? -- To view, visit http://gerrit.cloudera.org:8080/20648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37055990be49e91462ebc98aa97009ca768a0072 Gerrit-Change-Number: 20648 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Fri, 08 Dec 2023 10:09:15 +0000 Gerrit-HasComments: Yes
