[
https://issues.apache.org/jira/browse/IMPALA-12933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-12933.
-------------------------------------
Fix Version/s: Impala 4.4.0
Resolution: Fixed
> Catalogd should set eventTypeSkipList when fetching specifit events for a
> table
> -------------------------------------------------------------------------------
>
> Key: IMPALA-12933
> URL: https://issues.apache.org/jira/browse/IMPALA-12933
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 4.4.0
>
>
> There are several places that catalogd will fetch all events of a specifit
> type on a table. E.g. in TableLoader#load(), if the table has an old
> createEventId, catalogd will fetch all CREATE_TABLE events after that
> createEventId on the table.
> Fetching the list of events is expensive since the filtering is done on
> client side, i.e. catalogd fetch all events and filter them locally based on
> the event type and table name:
> [https://github.com/apache/impala/blob/148888e3ed4f97292499b2e6ee8d5a756dc648d9/fe/src/main/java/org/apache/impala/catalog/TableLoader.java#L98-L102]
> [https://github.com/apache/impala/blob/b7ddbcad0dd6accb559a3f391a897a8c442d1728/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L336]
> This could take hours if there are lots of events (e.g 1M) in HMS. In fact,
> NotificationEventRequest can specify an eventTypeSkipList. Catalogd can do
> the filtering of event type in HMS side. On higher Hive versions that have
> HIVE-27499, catalogd can also specify the table name in the request
> (IMPALA-12607).
> This Jira focus on specifying the eventTypeSkipList when fetching events of a
> particular type on a table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
