Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13061 )
Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart ...................................................................... IMPALA-7665: Fix unwarranted query cancellation on statestore restart Currently, if the statestore restarts and disseminates an inconsistent view of cluster membership to the coordinators, then they might believe that the backends no longer in the membership update are down and would start canceling queries that are running or scheduled to run on those allegedly failed backends. This patch adds a grace period after statestore recovery/successful registration that give it enough time to gather a consistent state of the cluster. Testing: - Added an e2e test. - Did manual stress testing using concurrent_select.py with statestore_subscriber_timeout_seconds set to 2 secs and failed_backends_query_cancellation_grace_period_ms set to 5 seconds, and the statestore being restarted every 15 seconds. To avoid other effects of statestore restarts cropping up, I used a local catalog (catalog v2) and ignored query errors caused due to scheduler having an incomplete view of the cluster(no backends). Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Reviewed-on: http://gerrit.cloudera.org:8080/13061 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_restart_services.py 4 files changed, 91 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
