Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18038 )
Change subject: IMPALA-11028: Table loading can fail when events are cleaned up ...................................................................... IMPALA-11028: Table loading can fail when events are cleaned up IMPALA-10502 introduces a createEventId field of a table which is updated when Impala creates a table. This is used by the events processor to determine if the subsequent CREATE_TABLE event which is received should be skipped or not. When the table is loaded for the first time, in order to avoid race conditions, TableLoader updates the createEventId to the last CREATE_TABLE event id from the metastore. In order to fetch the latest CREATE_TABLE event id, it fetches all the events from metastore since the last known createEventId of the table. However, if there is a significant delay between (more than 24hrs) between the time a table is created or invalidated, and the table is queried, it is possible that the metastore cleanup thread deletes the events which are generated since the table's createEventId. In such a case, the HMS Client method getNextNotification() throws an IllegalStateException due to the missing events. This exception causes the Table load to fail and query to error out. The fix is to not rely on the HMS Client method which throws the IllegalStateException. Instead we use the backing thrift API directly. Testing: 1. Introduced a custom cluster test which can reproduce this issue. 2. Test works after the patch. 3. Core tests. Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a --- M bin/create-test-configuration.sh M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/resources/hive-site.xml.py A tests/custom_cluster/test_metastore_events_cleanup.py 8 files changed, 107 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/18038/11 -- To view, visit http://gerrit.cloudera.org:8080/18038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a Gerrit-Change-Number: 18038 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sourabh Goyal <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
