If you can afford the disk space, and potential latency overheads, you can make your bitcask merges less frequent. You can find a window when your databases are going to be in low-usage, and have each node have a separate window for bitcask merges, which (A) will latency-level across your cluster, and (B) ensure that latency-leveling doesn't when you don't want it to.
On Thu, Mar 13, 2014 at 10:49 AM, Matthew MacClary <[email protected]> wrote: > I thought I would share part of one of our post install scripts with the > community to get feedback. Our goal is to avoid big merges that affect the > whole cluster at once. We set the merge trigger and merge threshold to the > same value which should result in more frequent but smaller scoped merges. > The trigger for each riak node is set randomly between 50% and 85% for dead > bytes and "fragmentation". We also chose to use largish values for the > threshold because that dictates how much of our disk bandwidth we will > "waste", if you merge after 10% dead bytes - then your are "wasting" your > time copying 90% good bytes and only recovering 10% of the slab's disk > space. > > What we were seeing in testing is that soon after the cluster was installed, > all the nodes were doing heavy merges at the same time. This created a very > noticeable bad performance window. Over time the merges would start to > spread out a little bit due to random factors. I should also point out that > our work mix is basically 1:1:1:0 for write:read:delete:update so merging > happens a lot. > > We decided to make the merges smaller and start the cluster out with > randomized merges. If you read with a consistency value of 1, then there is > a good chance that if one replica is merging, then another replica vnode > would not be part of a merge and could reply with normal latency. > > Let me know what you think! Is any of this reasoning faulty? > > # values used to compute values for app.config file > NUMERATOR=2147483648 # matches max_file_size > DENOMINATOR=100 > > THRESH=`shuf -i 50-85 -n 1` > #BYTES=$(expr $THRESH \* 2147483648 / 100 ) > BYTES=$(expr $THRESH \* $NUMERATOR / $DENOMINATOR ) > > sed -ri "s/(.*dead_bytes_merge_trigger,)\d*.*$/\1${BYTES}},/" > ${RIAK_CONFIG_STAGING}/app.config > sed -ri "s/(.*dead_bytes_threshold,)\d*.*$/\1${BYTES}},/" > ${RIAK_CONFIG_STAGING}/app.config > > sed -ri "s/(.*frag_merge_trigger,)\d*.*$/\1${THRESH}},/" > ${RIAK_CONFIG_STAGING}/app.config > sed -ri "s/(.*frag_threshold,)\d*.*$/\1${THRESH}},/" > ${RIAK_CONFIG_STAGING}/app.config > > > -Matt > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
