Todd Lipcon has submitted this change and it was merged.
Change subject: KUDU-1949. Maintenance Manager should trigger flushes earlier
......................................................................
KUDU-1949. Maintenance Manager should trigger flushes earlier
This changes the maintenance manager to start triggering flushes at the
60% heap usage threshold, and adjusts the memory limiting to only start
rejecting writes at the 80% threshold.
This is based on analysis in [1] ("After tweaking memory backpressure
rejection") in which we found that earlier flushing could keep the
system from making too many rejections.
I ran tpch_real_world with SF=300 as described in the doc below before
and after the patch. Some key metrics:
Before After
Mem rejections 1,162,527 62,790 (18.5x fewer)
Wall time 3156s 2490s (1.26x faster)
Server CPU 37,348s 72,873s (1.95x less efficient)
This matches the results from the original experiment. Because the
writes come in faster, we don't get enough time to run compactions, and
so each write ends up taking more CPU. Some future optimizations are
still pending which will reduce the CPU time per bloom lookup.
Regardless, this new behavior is configurable such that, if overall CPU
efficiency is more important than wall time, we could bump the "memory
pressure threshold" up higher than the "memory rejection threshold".
[1]
https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit#heading=h.gw87heq4fq6l
Change-Id: I8b7e703c82ac14fbce3a699bdf6a2f0fb4ed93a1
Reviewed-on: http://gerrit.cloudera.org:8080/6802
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <[email protected]>
Reviewed-by: David Ribeiro Alves <[email protected]>
---
M src/kudu/util/maintenance_manager.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
3 files changed, 31 insertions(+), 2 deletions(-)
Approvals:
David Ribeiro Alves: Looks good to me, approved
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/6802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I8b7e703c82ac14fbce3a699bdf6a2f0fb4ed93a1
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: David Ribeiro Alves <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <[email protected]>
