Tim Armstrong has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10988 )
Change subject: IMPALA-7297: soft memory limit for reservation increases ...................................................................... IMPALA-7297: soft memory limit for reservation increases https://goo.gl/N9LgQt summarises the memory problems I'm trying to solve here. Reject reservation increases that would leave < 10% of the memory available under a query or process mem_limit, which put the query or impalad into a state with a heightened chance of OOM. This avoids OOM in some scenarios where reserved memory is increased too aggressively and starves non-reserved memory. One way of thinking about this that it's extending the 80% limit heuristic for reserved memory so that there's a soft limit of min(80%, 90% - non-reserved memory consumption) A follow-on will use this for scanner threads to prevent starting scanner threads that may exceed the soft limit. Testing: Added unit tests for MemTracker and for ReservationTracker. Change-Id: I39dcf2dd870a59c8b8cda4488fe41ce936d715ac --- M be/src/exec/exchange-node.cc M be/src/rpc/impala-service-pool.cc M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/reservation-tracker-test.cc M be/src/runtime/bufferpool/reservation-tracker.cc M be/src/runtime/bufferpool/reservation-tracker.h M be/src/runtime/initial-reservations.cc M be/src/runtime/mem-pool-test.cc M be/src/runtime/mem-tracker-test.cc A be/src/runtime/mem-tracker-types.h M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-state.cc M be/src/runtime/runtime-state.cc M be/src/udf/udf.cc 17 files changed, 313 insertions(+), 151 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/10988/5 -- To view, visit http://gerrit.cloudera.org:8080/10988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39dcf2dd870a59c8b8cda4488fe41ce936d715ac Gerrit-Change-Number: 10988 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
