Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14910 )
Change subject: KUDU-3002: prioritize WAL unanchoring when under memory pressure ...................................................................... KUDU-3002: prioritize WAL unanchoring when under memory pressure When under memory pressure, we currently prioritize performing the op that will free the most memory. In theory this seems reasonable -- if Kudu's using way too much memory, we should try to use less memory as quickly as possible. In practice, this meant that, while under memory pressure and under a constant, memory-consuming workload (e.g. inserts to the MRS), Kudu would starve operations that anchor relatively little memory (e.g. DMS flushes). This patch updates the behavior so that we prioritize operations that unanchor the most WAL bytes, breaking ties by prioritizing the ops that use more memory. This seems reasonable because: - Ops that anchor WALs also anchor memory. In performing an op that unanchor WALs, we are performing an op that frees memory, so we'll still prefer MRS and DMS flushing over compactions when under memory pressure. - We already use this heuristic when _not_ under memory pressure, but it is only used when the ops under consideration anchor too many WAL bytes (per --log_target_replay_size_mb), lending some credibility to it when used conditionally. - It becomes much more difficult to think of a scenario in which we're "stuck" using too much space for WALs or too much memory. Change-Id: Ibd85e8f2904a36b74cd6a3038c9ec49bb1ff9844 Reviewed-on: http://gerrit.cloudera.org:8080/14910 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo <[email protected]> --- M src/kudu/tserver/tablet_server-test.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h 4 files changed, 117 insertions(+), 18 deletions(-) Approvals: Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14910 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibd85e8f2904a36b74cd6a3038c9ec49bb1ff9844 Gerrit-Change-Number: 14910 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Wong <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Bankim Bhavsar <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon <[email protected]>
