Another thing that I have seen cause riak_kv to stall like this is LevelDB corruption preventing vnodes from starting properly. I use this command to look for LOG files with the word error in them. It is also worth checking the anti_entropy levelDB LOG files as well.
find . -name "LOG" -exec grep -l -i 'error' {} \; -cv > On Nov 11, 2015, at 7:05 PM, Steven Sylvester <ssylves...@basho.com> wrote: > > Hi Markus, > > One thing that stands out in what you pasted is the following line: > > 2015-11-09 11:31:45.263 [info] <0.73.0> alarm_handler: > {set,{{disk_almost_full,"/"},[]}} > > Check the output of `df` and add nodes/disk as necessary to keep your disk > usage at a manageable level. Our docs recommend keeping disk use below 60%: > http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk > <http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk> > > Disk capacity issues can sneak up on you due to some backend operations that > require a lot of space all at once, such as Bitcask merging or AAE. *Bad > things* tend to happen to a Riak cluster with full disks, including data > loss/corruption. > > Other than that, are there any changes that have been made to the environment > lately? Have you performed any upgrades recently? > > Check the output of `ps aux` and kill any processes related to Riak, such as > any instances of `beam.smp` before attempting to start the node again. > > If the node won't start after these troubleshooting steps, run `riak console` > which will attempt to start the node and put you into an Erlang attach > shell. Monitor the messages that are printed to stdout and if/when it seems > to hang, paste the output of the whole thing here and we can take a look. > > -- > Steven Sylvester, CISSP > Client Services Engineer > Basho Technologies > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com