[email protected] has posted comments on this change. ( http://gerrit.cloudera.org:8080/21031 )
Change subject: IMPALA-12709: Add support for hierarchical metastore event processing ...................................................................... Patch Set 20: (3 comments) http://gerrit.cloudera.org:8080/#/c/21031/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/21031/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@808 PS20, Line 808: if (dbCount < minDbCount) { why we do not just get a random index of the list dbEventExecutors_ ,then we can get a random DBEventExecutor to process the event http://gerrit.cloudera.org:8080/#/c/21031/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1250 PS20, Line 1250: if (eventProcessingOnHold_) { return; } It is not very reasonable to use the total number of processes in db plus table to judge whether to perform event processing. If the blocking number of a single thread exceeds 1000, but other thread pools are very idle, the entire processing thread will degrade to a single thread. However, if at this time, there are new requests coming in, and new requests can enter the idle thread pool. If the thread pool is used, then the multi-thread processing effect we want will not be achieved in the end. http://gerrit.cloudera.org:8080/#/c/21031/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1251 PS20, Line 1251: long eventCount = getOutstandingEventCount(); I think ,we should keep to value , one is for outstandingEventCount another is outstandingEventCountPerTable. if all table 's pending number is more than outstandingEventCountPerTable and total number is bigger than outstandingEventCount ,then we should block -- To view, visit http://gerrit.cloudera.org:8080/21031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 Gerrit-Change-Number: 21031 Gerrit-PatchSet: 20 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Tue, 09 Jul 2024 09:20:02 +0000 Gerrit-HasComments: Yes
