Yida Wu has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21896 )
Change subject: IMPALA-12146: Fix incorrect host memory reserved when the executor quits abnormally ...................................................................... IMPALA-12146: Fix incorrect host memory reserved when the executor quits abnormally Currently there is an issue where if an executor quits abnormally while running a query, its reserved memory may still remain in the coordinator's host stats. The remote aggregated memory reserved uses all available remote pool stats for calculation. The problem happens when the statestore sends a topic update to update the pool stats. Although the coordinator removes the remote stats from the pool for the terminated executor during the update, in UpdateClusterAggregates(), it fails to reset for the corresponding aggregated memory reserved for that host if all the remote stats for that host have been removed. This can lead to stale memory reserved value remaining. To fix this, added a logic to ensure that the stats of memory reserved of that host are reset in the aggregated host stats when a delete topic for the host is detected and the host no longer exists in any remote pool stats. Tests: Passed exhaustive tests. Added testcase AdmissionControllerTest::EraseHostStats. Manually verified that the coordinator web ui correctly showed the reserved memory after the crashed executor recovered and rejoined. Change-Id: Ic6f6edd28c55904d63d0c494230ee2bf7a0f6cce --- M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h 3 files changed, 117 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/21896/4 -- To view, visit http://gerrit.cloudera.org:8080/21896 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic6f6edd28c55904d63d0c494230ee2bf7a0f6cce Gerrit-Change-Number: 21896 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
