[
https://issues.apache.org/jira/browse/IMPALA-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved IMPALA-11028.
------------------------------------------
Fix Version/s: Impala 4.1.0
Resolution: Fixed
> Table loading could fail if metastore cleans up old events
> ----------------------------------------------------------
>
> Key: IMPALA-11028
> URL: https://issues.apache.org/jira/browse/IMPALA-11028
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Major
> Fix For: Impala 4.1.0
>
>
> After IMPALA-10502, Catalogd tracks the table's create event id. When the
> table is loaded for the first time, it updates the create event id of the
> table. But if the table is loaded for the first time after a long delay
> (after 24 hrs) it is possible the metastore cleans up old notification logs
> entries which are required by catalogd during the table load.
> See this snippet from TableLoader.java
> {noformat}
> if (eventId != -1 && catalog_.isEventProcessingActive()) {
> // If the eventId is not -1 it means this table was likely created by
> Impala.
> // However, since the load operation of the table can happen much
> later, it is
> // possible that the table was recreated outside Impala and hence the
> eventId
> // which is stored in the loaded table needs to be updated to the
> latest.
> // we are only interested in fetching the events if we have a valid
> eventId
> // for a table. For tables where eventId is unknown are not created by
> // this catalogd and hence the self-event detection logic does not
> apply.
> events = MetastoreEventsProcessor.getNextMetastoreEvents(catalog_,
> eventId,
> notificationEvent -> CreateTableEvent.CREATE_TABLE_EVENT_TYPE
> .equals(notificationEvent.getEventType())
> &&
> notificationEvent.getDbName().equalsIgnoreCase(db.getName())
> &&
> notificationEvent.getTableName().equalsIgnoreCase(tblName));
> }
> {noformat}
> {{getNextMetastoreEvents}} method can throw the following exception if the
> metastore has cleaned up older entries (by default 24 hrs). This is
> controlled by configuration {{hive.metastore.event.db.listener.timetolive}}
> on the metastore side.
> I could reproduce the problem setting the following metastore configs.
> {noformat}
> hive.metastore.event.db.listener.clean.interval=10s
> hive.metastore.event.db.listener.timetolive=120s
> {noformat}
> Now run the following Impala script
> {noformat}
> create table t1 (c1 int);
> create table t2 (c1 int);
> select sleep(240000);
> create table t3 (c1 int);
> select * from t1;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
