Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5749: coordinator race hits DCHECK 'num_remaining_backends_ > 0' ......................................................................
IMPALA-5749: coordinator race hits DCHECK 'num_remaining_backends_ > 0' In Coordinator::UpdateBackendExecStatus(), we check if the backend has already completed with BackendState::IsDone() and return without applying the update if so to avoid updating num_remaining_backends_ twice for the same completed backend. The problem is that the value of BackendState::IsDone() is updated by the call to BackendState::ApplyExecStatusReport() that comes after it, but these operations are not performed atomically, so if there are two simultaneous calls to UpdateBackendExecStatus(), they can both call IsDone(), both get 'false', and then proceed to erroneously both update num_remaining_backends_, hitting a DCHECK. This patch modifies ApplyExecStatusReport to return true iff this report transitioned the backend to a done status, and then only updates num_remaining_backends_ in this case, ensuring it is only updated once per backend. Testing: - Ran test_finst_cancel_when_query_complete 10,000 times without hitting the DCHECK (previously, it would hit about once per 300 runs). Change-Id: I1528661e5df6d9732ebfeb414576c82ec5c92241 Reviewed-on: http://gerrit.cloudera.org:8080/7577 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc 3 files changed, 52 insertions(+), 52 deletions(-) Approvals: Impala Public Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7577 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1528661e5df6d9732ebfeb414576c82ec5c92241 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]>
