Looking over the changes from 1.4.7 -> 1.4.8 I don't see any changes that would be likely to affect merges.
It it possible that it's been happening for longer? Can you give a rough description of your workload so I can make a start on reproducing the issue tomorrow? On Wed, Feb 26, 2014 at 7:52 PM, John Welter <[email protected]> wrote: > Hi everyone, > > Upgraded to 1.4.8 and got into some issues with AAE overloading our vnodes. > We've worked around that. > > But now we're seeing a merge issue manifest itself since we are into our > nightly merge window. > > I see several vnodes on multiple nodes with similar in the logs: > > 2014-02-26 20:27:39.736 [info] <0.1040.0> > "/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552" > needs_merge: [] > 2014-02-26 20:27:39.736 [info] <0.29858.20> Merged {[],[]} in 1.0e-6 seconds. > 2014-02-26 20:30:39.739 [info] <0.1040.0> > "/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552" > needs_merge: [] > 2014-02-26 20:30:39.739 [info] <0.32391.20> Merged {[],[]} in 2.0e-6 seconds. > 2014-02-26 20:33:39.742 [info] <0.1040.0> > "/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552" > needs_merge: [] > 2014-02-26 20:33:39.742 [info] <0.2113.21> Merged {[],[]} in 1.0e-6 seconds. > 2014-02-26 20:36:39.744 [info] <0.1040.0> > "/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552" > needs_merge: [] > 2014-02-26 20:36:39.744 [info] <0.4756.21> Merged {[],[]} in 1.0e-6 seconds. > 2014-02-26 20:39:39.746 [info] <0.1040.0> > "/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552" > needs_merge: [] > 2014-02-26 20:39:39.746 [info] <0.7288.21> Merged {[],[]} in 1.0e-6 seconds. > > Looking at that vnodes bit cask folder: > > root@app25u:/var/lib/riak/bitcask/639406966332270026714112114313373821099470487552# > ls -ltrh > total 7.3G > -rw-rw-r-- 1 riak riak 499M Jun 22 2013 2.bitcask.data > -rw-rw-r-- 1 riak riak 2.4M Jun 22 2013 2.bitcask.hint > -rw-rw-r-- 1 riak riak 43K Oct 3 11:25 9.bitcask.hint > -rw-rw-r-- 1 riak riak 15M Oct 3 11:25 9.bitcask.data > -rw-rw-r-- 1 riak riak 51M Oct 6 19:35 12.bitcask.data > -rw-rw-r-- 1 riak riak 129K Oct 6 20:49 12.bitcask.hint > -rw-r--r-- 1 riak riak 121M Dec 23 20:56 15.bitcask.data > -rw-r--r-- 1 riak riak 301K Dec 23 21:19 15.bitcask.hint > -rw-r--r-- 1 riak riak 804K Feb 5 12:41 17.bitcask.hint > -rw-r--r-- 1 riak riak 304M Feb 5 12:41 17.bitcask.data > -rw-r--r-- 1 riak riak 2.0G Feb 5 12:42 18.bitcask.data > -rw-r--r-- 1 riak riak 5.0M Feb 5 12:42 18.bitcask.hint > -rw-r--r-- 1 riak riak 2.0G Feb 5 12:43 20.bitcask.data > -rw-r--r-- 1 riak riak 5.1M Feb 5 12:43 20.bitcask.hint > -rw-r--r-- 1 riak riak 1.7G Feb 5 12:43 22.bitcask.data > -rw-r--r-- 1 riak riak 8.7M Feb 5 12:43 22.bitcask.hint > -rw-r--r-- 1 riak riak 495M Feb 23 17:30 77.bitcask.data > -rw-r--r-- 1 riak riak 1.5M Feb 23 17:34 77.bitcask.hint > -rw-r--r-- 1 riak riak 71M Feb 26 09:05 78.bitcask.data > -rw-r--r-- 1 riak riak 194K Feb 26 09:06 78.bitcask.hint > -rw-r--r-- 1 riak riak 13M Feb 26 18:05 80.bitcask.data > -rw-r--r-- 1 riak riak 33K Feb 26 18:05 80.bitcask.hint > -rw-r--r-- 1 riak riak 75K Feb 26 18:32 79.bitcask.hint > -rw-r--r-- 1 riak riak 28M Feb 26 18:32 79.bitcask.data > -rw------- 1 riak riak 93 Feb 26 18:32 bitcask.write.lock > -rw-r--r-- 1 riak riak 1.7K Feb 26 20:45 81.bitcask.hint > -rw-r--r-- 1 riak riak 667K Feb 26 20:45 81.bitcask.data > > We're starting riak via the /etc/init.d/riak script and have a > /etc/default/riak file with ulimit -n 65535 in it so open files should not be > an issue (and have not been previously with 1.4.7 and earlier). > > Any suggestions? > > Thanks, > > John > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
