> 02:59:35.392 [error] CRASH REPORT Process <0.149.0> with 0 neighbours exited > with reason: bad argument in call to erlang:binary_to_term(<<>>) in > riak_core_ring_manager:read_ringfile/1 line 154 in gen_server2:init_it/6 > line 384
There are several cases where Riak doesn't start cleanly after running out of disk space. Many of cases are "on the list" to fix. Luckily, most of these cases don't compromise your data and can be corrected. Your particular error is a truncated ring file. This is your cluster metadata that is gossiped between nodes and stored in Riak's data/ring directory. There should be multiple files in that directory. You should be able to delete the most recent file and try running "riak console" again. If that doesn't work, you can always delete the entire "data/ring" directory and start the node up fresh. It'll have all it's data, but no longer be a member of your Riak cluster -- no big deal, just "riak-admin join" back to cluster and everything will be sorted. Part of the joining process is replacing the local node's ring with the ring from the cluster it's joining. As for absolute guarantees, an individual Riak node doesn't strictly guarantee to be out-of-space proof. Most of the time things are fine, but there are cases where you end up with corrupted backend files that cannot be recovered. You can delete the appropriate bad files, and Riak will start just fine, but some of your data may be missing. Of course, you have replicas on other nodes, so your missing replicas can be recovered from theirs siblings. Replicas are repaired anytime data is read through Riak's read-repair mechanism. We also have the manual repair feature we introduced in Riak 1.2: http://docs.basho.com/riak/latest/cookbooks/Repairing-KV-Indexes The next major release of Riak will also include continuous/incremental active anti-entropy that will automatically repair all divergent/missing replicas without any user intervention at all. Obviously, all of these solutions assume you didn't somehow corrupt the exact same data on all nodes in your clusters. You're right that the the real world, having a node here or there run out of disk space is possible. Have multiple nodes run out of disk space at the same time is much less common, and something that Riak assumes doesn't happen. This is something that you will naturally aim to avoid regardless of the crash-recovery scenario. You don't want nodes running out of disk-space and shutting down. Your service isn't available when nodes are offline, and Riak is intended to be used for highly available, always running services. Part of this is proper capacity planning and monitoring, and making sure to add new nodes to your Riak cluster (or upgrade existing nodes) in order to grow capacity as needed. Hope this answers some of your questions. Regards, Joe On Thu, Nov 15, 2012 at 7:35 PM, Mike Ellis <[email protected]> wrote: > Hi fellow Riak users – a special shout-out to those I got a chance to meet > at Ricon 2012. > > > > I learned some new things, interacted with really smart people, and decided > to do a Riak project to learn about this tech. > > > > 5 node cluster, all proud of myself that I got JSON searching etc. working > (some speed/other things to clean up), haproxy, but overall, things are > looking pretty good. > > > > I’m pumping in loads of JSON data into these KV pairs, and things are > looking good… > > UNTIL…. > > I let the nodes run out of disk-space… OUCH rookie mistake. > > > > But I’m thinking – this shouldn’t (but can) happen in real life, lets see > how well Riak recovers from such a thing. Learning opportunity! > > > > -- > > > > The boxes were wedged HARD – no login possible, so they had to be rebooted. > > Riak wasn’t running after reboot (just the epm daemon), rest couldn’t start > since disk was still 100% full. > > > > I brought some new filesystems online, (properly) copied the content of > leveldb, and pointed the config-files to the new location of leveldb, > cleaned some disk-space. Did this on all 5 nodes. > > > > On 3 out of 5 nodes, things came back to life… Not bad. But on 2 nodes, > there is no joy. (riak console error below). > > > > I found some utilities that check the merge_index for consistency, and that > didn’t point to any issues. > > > > -- > > > > So since 3/5 nodes are up, I THINK all my data is safe. Is there some sort > of scrubbing operation I can run to basically fsck the bajezus out of this > thing to ensure the data is correct/consistent/fully available across these > 3 surviving nodes? Its taking writes again, so it appears (on the surface) > to be in reasonable shape, eventhough 2/5 nodes are MIA. > > > > I guess I would have been up the creek if 2/5 came back to life, not 3/5? > > Is there any value (or possibility) to bring one of the failed nodes back to > life, or just nuke the data and re-join? > > This sort of thing is bound to happen in real life to some unsuspecting > person out there. Is this a bug? > > Should Riak have handled an out of space condition a bit cleaner? (if not > from crashing from a recovery perspective?) I would have expected some sort > of atomic recovery point. > > > > Any general thoughts/guidance/education (other than “never let your riak > cluster run out of disk-space”) are much appreciated. > > > > Thanks, > > > > n MikeE > > > > > > -bash-4.1$ riak console > > Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot > /usr/lib64/riak/releases/1.2.0/riak -embedded -config > /etc/riak/app.config -pa /usr/lib64/riak/basho-patches > -args_file /etc/riak/vm.args -- console > > Root: /usr/lib64/riak > > Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:64] > [kernel-poll:true] > > > > > > =INFO REPORT==== 16-Nov-2012::02:59:34 === > > alarm_handler: {set,{system_memory_high_watermark,[]}} > > ** /usr/lib64/riak/lib/observer-1.1/ebin/etop_txt.beam hides > /usr/lib64/riak/lib/basho-patches/etop_txt.beam > > ** Found 1 name clashes in code paths > > 02:59:35.273 [info] Application lager started on node 'riak@xxxxxxxxxxx' > > 02:59:35.392 [error] CRASH REPORT Process <0.149.0> with 0 neighbours exited > with reason: bad argument in call to erlang:binary_to_term(<<>>) in > riak_core_ring_manager:read_ringfile/1 line 154 in gen_server2:init_it/6 > line 384 > > /usr/lib64/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed. > > > > =INFO REPORT==== 16-Nov-2012::02:59:35 === > > alarm_handler: {clear,system_memory_high_watermark} > > Erlang has closed > > {"Kernel pid > terminated",application_controller,"{application_start_failure,riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}}"} > > > > Crash dump was written to: /var/log/riak/erl_crash.dump > > Kernel pid terminated (application_controller) > ({application_start_failure,riak_core,{shutdown,{riak_core_app,start,[normal,[]]}}}) > > -bash-4.1$ > > > > > > > > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- Joseph Blomstedt <[email protected]> Senior Software Engineer Basho Technologies, Inc. http://www.basho.com/ _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
