Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20192 )
Change subject: IMPALA-12267: DMLs/DDLs can hang as a result of catalogd restart ...................................................................... IMPALA-12267: DMLs/DDLs can hang as a result of catalogd restart IMPALA-5476 added handling for changes in catalogd service ID during the lifetime of a DML/DDL query, but the loop that waits for a new catalogd service ID can wait indefinitely in case the DML/DDL was handled by the previous catalogd and restart happened just after replying to the coordinator: https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/be/src/service/impala-server.cc#L2204 This change adds a new startup flag, '--wait_for_catalog_update_timeout_sec', which can set an upper limit on the waiting time. Negative values and zero mean no timeout, positive values are interpreted as seconds. If the query times out, the local catalog cache is not updated and a warning is logged. TODO: { Now we also stop waiting if - a timeout is set AND - we have received 10 catalog updates from the statestore but none have a new catalog service ID. We should decide whether both the timeout and the limit on the number of updates should be kept, whether both should only be effective if a timeout is set, and when we stop waiting: if both timeout and number limit are hit OR as soon as one of them is hit. } Testing: - added a custom cluster test that reproduces the above condition using a debug action and verifies that the waiting loop times out. Change-Id: Ib71bec8f67f80b0bdfe0a6cc46a16ef624163d8b --- M be/src/service/client-request-state.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_restart_services.py 6 files changed, 171 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/20192/3 -- To view, visit http://gerrit.cloudera.org:8080/20192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib71bec8f67f80b0bdfe0a6cc46a16ef624163d8b Gerrit-Change-Number: 20192 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
