Hello Bikramjeet Vig, Dan Hecht, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11350
to look at the new patch set (#3).
Change subject: IMPALA-7446: enable missing buffer pool GC
......................................................................
IMPALA-7446: enable missing buffer pool GC
This was not hooked up as an oversight, and means that sometimes queries
can spill or OOM because of accumulated free buffers and pages in the
buffer pool. This patch ensures that we try to free up unused memory
from the buffer pool before we start failing memory allocations.
I was somewhat concerned about an interaction between the soft
mem_limit and the GC. If the system is near or over the soft process
memory limit, it's possible that many spilling operators or HDFS scans
could be repeatedly bumping up against the soft memory limit and doing
failed GCs. With hard memory limits, failed GCs lead to many queries
failing and the situation resolving itself, but with soft memory limits
the queries will likely be able to continue running while doing many
GCs.
My solution was to back off soft GCs for a 100ms period after a
failure.
Testing:
Added unit tests.
Looped the query from IMPALA-7442 to confirm it didn't fail.
Change-Id: I32400bda18a36f60cbe315fae715748d33677c10
---
M be/src/common/init.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
5 files changed, 140 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/11350/3
--
To view, visit http://gerrit.cloudera.org:8080/11350
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32400bda18a36f60cbe315fae715748d33677c10
Gerrit-Change-Number: 11350
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
