[
https://issues.apache.org/jira/browse/IMPALA-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18065078#comment-18065078
]
ASF subversion and git services commented on IMPALA-14535:
----------------------------------------------------------
Commit e2aeb0a6bda7012f8aba083f74c61cbaaa16d635 in impala's branch
refs/heads/master from Venu Reddy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e2aeb0a6b ]
IMPALA-14535: Improve wait for HMS events sync with hierarchical event
processing
With hierarchical event processing, MetastoreEventsProcessor's
lastSyncedEventId_ represents the last event dispatched to the
EventExecutorService. Events are queued to appropriate
DbEventExecutors and/or TableEventExecutor for processing.
And events are removed from these queues only after they
are processed by the respective executor threads.
When a wait for HMS events sync request is issued, the
following sequence is executed:
1. The current notification event id is fetched from HMS.
2. MetastoreEventsProcessor is ensured to fetch and dispatch
all events up to and including this notification event id
to the EventExecutorService.
3. The list of databases and tables requiring synchronization
is extracted from the request.
4. To determine whether synchronization for a required
database and or table is complete, relevant DbEventExecutor
and TableEventExecutor instances are checked directly.
Synchronization is considered complete only when the
event-processing progress of the required DbProcessor and
TableProcessor instances has surpassed the captured
notification event id.
This approach guarantees that all HMS events that existed at
the time of sync request have been fully processed for the
requested databases and tables.
Testing:
- Added an FE test and ran the existing tests.
Change-Id: I55cea4cb8e04860202e56e1b1bf2596613b4946c
Reviewed-on: http://gerrit.cloudera.org:8080/23789
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
> Wait for events in the table queue instead of global lastSycnedEventId
> ----------------------------------------------------------------------
>
> Key: IMPALA-14535
> URL: https://issues.apache.org/jira/browse/IMPALA-14535
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Venugopal Reddy K
> Priority: Major
>
> When Hierarchical Event Processing is enabled, we can improve the
> implementation of sync_hms_events_wait_time_s (IMPALA-12152) to check events
> in the table queue instead of the global lastSycnedEventId. If the
> table-level lastSycnedEventId reaches the max event id in the queue, catalogd
> can return the waitForHmsEvent request.
> CC [~VenuReddy]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
