[email protected] has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24975
Change subject: IMPALA-15451: Make hierarchical event executors event-driven ...................................................................... IMPALA-15451: Make hierarchical event executors event-driven When hierarchical event processing is enabled, catalogd consumes noticeable CPU on an idle cluster. Each DbEventExecutor and TableEventExecutor thread is scheduled with scheduleAtFixedRate() every 10 ms, regardless of whether it had any work. With the default configuration of 5 db executors, each having 5 table executors, 30 threads wake up 100 times per second each, i.e. ~3000 wakeups per second. Although each pass finds no events and does little work, the cost of scheduling and context switches adds up. This patch replaces the fixed-rate polling with event-driven wakeups. Each executor thread now runs a process loop and blocks in a new EventExecutorWaiter when it has nothing to do. It is woken up in the following cases: - DbEventExecutor is signalled when an event is enqueued to one of its DbProcessors, and when all the TableProcessors waiting on a DbBarrierEvent have reached it. - TableEventExecutor is signalled when an event is enqueued to one of its TableProcessors, and when a DbBarrierEvent its TableProcessor waits on is processed. EventExecutorWaiter records a signal raised while the executor thread is busy, so the next wait returns immediately and no wakeup is lost. An executor still wakes up on a timeout in the following cases: - TableEventExecutor retries every 10 ms while it has outstanding events that could not be processed yet, e.g. events waiting on a RenameTableBarrierEvent. - DbEventExecutor wakes up every 1 second while it has DbProcessors, to remove the TableProcessors that have been idle for min_event_processor_idle_ms. Otherwise, the executor waits indefinitely until it is signalled. Since events are processed as soon as they are enqueued, instead of waiting up to 10 ms for the next tick, this also removes the scheduling latency from event processing. Testing: - Ran existing tests. - Verified with top -H that DbEventExecutor and TableEventExecutor threads do not consume CPU on an idle catalogd. Assisted-by: Claude Opus 4.8 (Claude Code) Change-Id: I97107e99ead7907023a936d0017f33fcdba23185 --- M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java 7 files changed, 228 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/1 -- To view, visit http://gerrit.cloudera.org:8080/24975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185 Gerrit-Change-Number: 24975 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <[email protected]>
