Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5598: Fix excessive dumping in MemLimitExceeded ......................................................................
IMPALA-5598: Fix excessive dumping in MemLimitExceeded ExecQueryFInstances RPC timeouts in stress tests were traced to excessive dumping in MemLimitExceeded() and LogUsage(). The QueryState::Init() hits the process memory limit, and this causes MemLimitExceeded to dump the process memory tracker. On the stress test, this involves dumping hundreds of queries and all the structures underneath. This is expensive and the contention between threads accessing these structures causes RPC timeouts. This adds the ability to the limit the recursive depth when dumping memory trackers. It also modifies the dumping in MemLimitExceeded() to have the following semantics: 1. The query memory tracker is always logged in full if it exists. 2. The process memory tracker is logged if the query memory tracker doesn't exist or if the process memory limit is being hit. The process memory tracker is limited to dumping only two levels of children. Other uses of LogUsage() are not limited (e.g. /memz and the query memory page on the web UI). A stress test run with the process memory tracker limited to two levels showed no RPC timeouts. Change-Id: Ib151c3894d4c43b508779e6809d77e2f8af89cf2 Reviewed-on: http://gerrit.cloudera.org:8080/7597 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/exec/exec-node.cc M be/src/exec/hash-table-test.cc M be/src/runtime/bufferpool/reservation-tracker-test.cc M be/src/runtime/initial-reservations.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/runtime-state.cc M be/src/service/impala-http-handler.cc M be/src/util/default-path-handlers.cc 9 files changed, 61 insertions(+), 31 deletions(-) Approvals: Impala Public Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7597 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib151c3894d4c43b508779e6809d77e2f8af89cf2 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Matthew Jacobs <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
