Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9038 )
Change subject: IMPALA-2642: Fix a potential deadlock in statestore ...................................................................... IMPALA-2642: Fix a potential deadlock in statestore The statestored can deadlock if the number of subscribers has reached STATESTORE_MAX_SUBSCRIBERS, because the DoSubscriberUpdate() method calls OfferUpdate(), while holding subscribers_lock_, which also tries to take the same lock in this situation. Fix the issue by moving out the call to acquire subscribers_lock_ from OfferUpdate(), and depend on the callers to take it. We also make the maximum number of statestore subscribers a start-up time tuneable, to allow us to test the limit more easily. Testing: The problem is easily reproduced by lowering the value of STATESTORE_MAX_SUBSCRIBERS to 3, and then launching a mini cluster with 3 impalads. Without the fix, the statestored becomes completely deadlocked. A new EE test has been added to exercise this scenario. The test verifies that statestored correctly rejects new subscription requests when the limit it reached. Change-Id: I5d49dede221ce1f50ec299643b5532c61f93f0c6 Reviewed-on: http://gerrit.cloudera.org:8080/9038 Reviewed-by: Sailesh Mukil <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_custom_statestore.py 3 files changed, 108 insertions(+), 15 deletions(-) Approvals: Sailesh Mukil: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5d49dede221ce1f50ec299643b5532c61f93f0c6 Gerrit-Change-Number: 9038 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Zoram Thanga <[email protected]>
