Vihang Karajgaonkar created IMPALA-8266: -------------------------------------------
Summary: Event filtering logic may not filter all the events Key: IMPALA-8266 URL: https://issues.apache.org/jira/browse/IMPALA-8266 Project: IMPALA Issue Type: Sub-task Reporter: Vihang Karajgaonkar Assignee: Vihang Karajgaonkar Here is a the logic to filter out create events from a given batch of events. {code} Iterator<MetastoreEvent> it = metastoreEvents.iterator(); // filter out the create events which has a corresponding drop event later int fromIndex = 0; int numFilteredEvents = 0; int inputSize = metastoreEvents.size(); while (it.hasNext()) { MetastoreEvent current = it.next(); if (fromIndex < metastoreEvents.size() && current.isRemovedAfter( metastoreEvents.subList(fromIndex + 1, metastoreEvents.size()))) { LOG.info(current.debugString("Filtering out this event since the object is " + "either removed or renamed later in the event stream")); it.remove(); numFilteredEvents++; } fromIndex++; } {code} If the event list contains CREATE_DATABASE, CREATE_TABLE, DROP_TABLE, DROP_DATABASE events, it is possible that we only filter out CREATE_DATABASE event and not CREATE_TABLE event. This is because the {{fromIndex}} above gets incremented event iteration of the {{while}} loop and hence when it is evaluating if there is a inverse event later on in the stream, it starts to look from DROP_DATABASE event onwards. The fix is simple, the fromIndex needs to be incremented only when the item from the list is not removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)