Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9921 )
Change subject: IMPALA-6793: Fix empty metadata after statestore restarts ...................................................................... Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/9921/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/9921/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1157 PS3, Line 1157: oldDbNames.remove(BUILTINS_DB); > agreed that there should not be a reason for this static db to be involved I kind of understand it now. There is another bug behind it indeed. When statestore sends the first catalog update to impalad, the update is not a delta, so delete operations are not included. TTopicDelta has an optional field to_version. When catalogd sends the catalog update, to_version is not set and it's initialized to 0. After statestore gets the catalog update, it will call SetLastTopicVersionProcessed(to_version), without checking whether to_version is set. So LastTopicVersionProcessed remains to be 0. When statestore prepares the next catalogd update RPC, the from_version in the request is set to LastTopicVersionProcessed (0). Here is how this patch comes in: without this patch, catalog checks catalog_objects_min_version_, which is 0, and decides not to send anything. With this patch catalog checks last_sent_version_, which is not 0, and will send the catalog update a second time, which contains the removing builtins_db operation. This time the statestore will send the update to impalads as a delta, and it will include the delete operation, so impalad is screwed. The conclusion is that the to_version bug has to be fixed before or together with this patch. -- To view, visit http://gerrit.cloudera.org:8080/9921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I332a60e172af84b93b3544373fe363cdced5e8d0 Gerrit-Change-Number: 9921 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tianyi Wang <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Vuk Ercegovac <[email protected]> Gerrit-Comment-Date: Thu, 05 Apr 2018 00:31:07 +0000 Gerrit-HasComments: Yes
