[
https://issues.apache.org/jira/browse/IMPALA-12399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764431#comment-17764431
]
ASF subversion and git services commented on IMPALA-12399:
----------------------------------------------------------
Commit 73430a2cdcb4be876cd73303a664414c3f94f9fd in impala's branch
refs/heads/master from Venu Reddy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=73430a2cd ]
IMPALA-12399: Add filter to skip OPEN_TXN events from HMS
Notification events like OPEN_TXN are ignored on catalogd
MetastoreEventsProcessor. So, we can pass eventTypeSkipList with OPEN_TXN
in NotificationEventRequest while invoking get_next_notification() to
avoid reading such notification messages from metastore and then ignoring
on catalogd. OPEN_TXN event being more frequent(received even upon
describe table operation from beeline), we can significantly reduce
unwanted processing on both hive metastore and catalogd. Catalogd reads
events in batches of EVENTS_BATCH_SIZE_PER_RPC, skipping such unnecessary
events can help catchup the events faster.
Testing:
- Manually tested in cluster and added a testcase
Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf
Reviewed-on: http://gerrit.cloudera.org:8080/20427
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Sai Hemanth Gantasala <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
> Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid
> receiving OPEN_TXN events from HMS
> ------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12399
> URL: https://issues.apache.org/jira/browse/IMPALA-12399
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Venugopal Reddy K
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Notification events like OPEN_TXN are ignored on catalogd
> {{{}MetastoreEventsProcessor{}}}. So, we can pass eventTypeSkipList with
> OPEN_TXN in NotificationEventRequest while invoking get_next_notification()
> to avoid reading such notification messages from HMS and then ignoring on
> catalogd. OPEN_TXN event being more frequent(received even upon describe
> table operation from beeline), we can significantly reduce unwanted
> processing on both HMS and catalogd. Catalogd reads events in batches of
> EVENTS_BATCH_SIZE_PER_RPC, skipping such unnecessary events can help catchup
> the events faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
