Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12048
Change subject: KUDU-2618 Factor the amount of data into time-based flush decisions ...................................................................... KUDU-2618 Factor the amount of data into time-based flush decisions Previously, the perf improvement score for flush operations was determined first by assigning a large score if the amount of RAM anchored was over a threshold, and second by assigning a modest score based on how long since the last flush. There was a small problem with this, which was that if the rate of inserts was very slow, and the cluster was generally not very busy writing, even the modest scores for flushes were enough to cause fairly frequent (~2min) flushes of replicas. Over time, this would add up to KUDU-1400-style problems with too many DRS. The changes to fix KUDU-1400 address those problems, but at the cost of write amplification based in part on the number of DRS that need to be compacted. So, there's still some gains to be had from increasing the average sizes of flushes. This patch adjusts how the score increases with time: flush candidates with RAM anchored below the target rowset size must wait for half of the time it takes to get the maximum score before they get a positive score. So if data is coming in very slowly Kudu will wait longer before flushing, flushing less overall, but only for smaller amounts of data where the total RAM held across many tablet replicas shouldn't be too large (memory-pressure-bassed flushing would kick in then) and the amount of data held in WALs shouldn't be very big. Change-Id: I27fbd3ec34c5615d07deaf47b3400ca7c4ea426a --- M src/kudu/tablet/tablet_replica-test.cc M src/kudu/tablet/tablet_replica_mm_ops.cc 2 files changed, 63 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/12048/1 -- To view, visit http://gerrit.cloudera.org:8080/12048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I27fbd3ec34c5615d07deaf47b3400ca7c4ea426a Gerrit-Change-Number: 12048 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley <[email protected]>
