Huaisi Xu has submitted this change and it was merged. Change subject: IMPALA-2378: check proc mem limit before preparing fragment ......................................................................
IMPALA-2378: check proc mem limit before preparing fragment Preparing a query fragment for execution uses a significant amount of untracked memory. Currently, if a query is admitted, it will consume a significant amount of memory regardless of how much process memory is available on each impalad. This is a problem since the untracked memory is included in the Tcmalloc stats used towards thhe process memory limit. We saw stress tests get into a state where the vast majority of cluster memory was taken up by this untracked memory, causing allocations of tracked memory to fail. This patch partially addresses the problem by checking whether the process memory limit is exceeded before preparing a query fragment. If there is no process memory available, fragment execution is cancelled before any additional memory is consumed for LLVM code, descriptors, execution nodes, and other untracked structures. This gives already-running queries a better chance of succeeding and does not waste resources preparing fragments that will immediately hit a memory limit. Testing: ran stress tests with the two TPCDS queries known to exhibit this problem especially badly. The memory limit was not exceeded by more than 0.1GB. Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc Reviewed-on: http://gerrit.cloudera.org:8080/1215 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Internal Jenkins (cherry picked from commit b54b6930be3fb902ae8425a3e170332aa7762996) Reviewed-on: http://gerrit.cloudera.org:8080/2639 Reviewed-by: anujphadke <[email protected]> Tested-by: Huaisi Xu <[email protected]> --- M be/src/service/fragment-mgr.cc 1 file changed, 13 insertions(+), 0 deletions(-) Approvals: Huaisi Xu: Verified anujphadke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/2639 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie7743834a57a044657f6fcac4623a609c2eb07dc Gerrit-PatchSet: 2 Gerrit-Project: Impala Gerrit-Branch: cdh5-2.2.0_5.4.x Gerrit-Owner: Huaisi Xu <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Huaisi Xu <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: anujphadke <[email protected]>
