Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15561 )
Change subject: IMPALA-9549: Handle catalogd startup delays when using local catalog ...................................................................... IMPALA-9549: Handle catalogd startup delays when using local catalog Impalads should be tolerant of delays in catalogd startup. Currently, when running with the local catalog (use_local_catalog=true), impalad startup can fail when catalogd startup is delayed. What happens is that ImpalaServer's constructor calls ImpalaServer::UpdateCatalogMetrics(), which maintains two metrics counting the number of tables and databases. This is before the code in ImpalaServer::Start() that waits for the catalogd to start (added by IMPALA-4704), so there is no guarantee that catalogd is running. The UpdateCatalogMetrics() call ends up calling getDbs() in the frontend catalog. LocalCatalog::getDbs() tries to load the databases (and thus contact catalogd), and this call will fail if catalogd is not running. This fails startup. use_local_catalog=false is immune to this only because it does not contact catalogd in Catalog::getDbs(). This moves the UpdateCatalogMetrics() call from the ImpalaServer constructor to ImpalaServer::Start() after the impalad has already waited for the catalogd to start up. It also limits the call to run only in coordinators. Prior to this change, when using local catalog, the executors would have catalog.num-databases and catalog.num-tables set to the right values at startup. These values would not be kept up to date. With this change, the executors do not have these values set. Without local catalog, both before and after this change, executors do not have accurate counts for catalog.num-databases or catalog.num-tables. Testing: - Added a test to custom_cluster.test_catalog_wait to delay catalogd start up by 60 seconds and verify that the impalads successfully start up. This test fails prior to this change. - Hand tested to verify that the metrics that are maintained by UpdateCatalogMetrics() are not meaningfully changed for coordinators and that executors do not have metrics set. Change-Id: I1b5a94c59faaaa25927a169dcb58f310ce6b1044 Reviewed-on: http://gerrit.cloudera.org:8080/15561 Reviewed-by: Vihang Karajgaonkar <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/service/impala-server.cc M tests/custom_cluster/test_catalog_wait.py 4 files changed, 70 insertions(+), 5 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1b5a94c59faaaa25927a169dcb58f310ce6b1044 Gerrit-Change-Number: 15561 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Anurag Mantripragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
