liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )
Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync ...................................................................... Patch Set 24: (7 comments) http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@9 PS23, Line 9: The change of catalogServiceId did not trigger a full topic update : request. : : ImpaladCatalog#updateCatalog() doesn't trigger a full topic update : request when detecting catalogServiceId changes. It just updates the : local catalogServiceId and throws an exception to abort applying the : DDL/DML results. This causes a problem when catalogd is restarted and : the DDL/DML is executed on the restarted instance. In this case, only : the local catalogServiceId is updated to the latest. The local catalog : remains stale. Then when dealing with the following > nit: ImpaladCatalog#updateCatalog() doesn't trigger a full topic update req Done http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@20 PS23, Line 20: applied without exceptions. However, the catalog objects usually won't : be updated since they have higher versions (from the old catalogd : instance) than those in the update. This brings the local catalog out : of sync until the catalog version of the new catalogd grows larger : enough. > nit: Note that in dealing with the catalog updates from statestore, if the Done http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@25 PS23, Line 25: > Let's add some description about how we fix the bug: Done http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@28 PS23, Line 28: more in ImpalaServer::CatalogUpda > nit: several tests Done http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@29 PS23, Line 29: > nit: tests Done http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc@2146 PS23, Line 2146: // TODO Restart catalogd twice. When restarting for the second time, ddl executes : // successfully, but we may receive the service id of the first restart, which will : // cause the local cache to not be updated in time > nit: could you move these into the else-block at line 2164 and change them Done http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py@230 PS23, Line 230: execute_query(query > Currently execute_query() and execute_query_async() are the same for DDLs. Done -- To view, visit http://gerrit.cloudera.org:8080/17645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c Gerrit-Change-Number: 17645 Gerrit-PatchSet: 24 Gerrit-Owner: liuyao <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: liuyao <[email protected]> Gerrit-Comment-Date: Fri, 20 Aug 2021 08:50:13 +0000 Gerrit-HasComments: Yes
