Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12428
Change subject: IMPALA-7961: Avoid adding unmodified objects to DDL response ...................................................................... IMPALA-7961: Avoid adding unmodified objects to DDL response When a DDL is processed, we typically add the affected (added/removed) objects to the response TCatalogUpdateResult struct. This response is processed on the coordinator and the changes are applied locally. When SYNC_DDL is enabled, the Catalog server also includes a topic version number that should include all the affected objects so that the coordinator can wait for that miniumum topic version to be applied on all other coordinators before returning the control back to the user. This covering topic version is calculated by looking at the topic update log, which contains all the in-flight updates (and to an extent past updates) that are perodically GC'ed. Bug: In certain cases like CREATE TBL IF NOT EXISTS, we could end up adding objects to the DDL response which haven't been modified in a while (> TOPIC_UPDATE_LOG_GC_FREQUENCY) and hence could be potentially GC'ed from the TopicUpdateLog. This means that the Catalog server wouldn't be able to find a covering topic update version and eventually gives up throwing an error as described in the jira. Fix: The idea here is to avoid any additions that include objects that haven't been modified by this DDL. Also added some additional diagnostic logging that could've simplified debugging an issue like this. Testing: Since this is a racy bug, I could only repro it by forcing frequent topic update log GCs along with a specific sequence of actions. Couldn't reproduce it with the patch. Change-Id: If3e914b70ba796c9b224e9dea559b8c40aa25d83 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/TopicUpdateLog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 4 files changed, 32 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/12428/3 -- To view, visit http://gerrit.cloudera.org:8080/12428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If3e914b70ba796c9b224e9dea559b8c40aa25d83 Gerrit-Change-Number: 12428 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Vissapragada <[email protected]>
