Hey David, So there are two likely causes here: either your ran out of disk space and/or some sort of drive corruption.
When you have a moment: 1. grep LOG files to find 'error\ '. (You may have already figured this out, but it looks like you have a drive issue) 2. save the LOG files before doing recovery ... sort /var/lib/riak/leveldb/*/LOG >LOG.save 3. Tar them up and send them our way so we can take a look at them. We do have some documentation on how to repair level [1] (which will soon be part of our official docs), but hold off on running this until we can get a look at the logs. Mark [1] https://gist.github.com/b88aee6da7fee81dc036 On Tue, Dec 4, 2012 at 7:34 PM, David Montgomery <[email protected]> wrote: > Hi, > > I am trying to add a new nodes. I get node un reachable when handing off. > I restart the new node and dies quick. Below is the output of the error > log. So....how do I resolve? > > > 2012-12-05 02:50:35.335 [error] > <0.14969.101>@riak_core_handoff_sender:start_fold:210 ownership_handoff > transfer of riak_kv_vnode from '[email protected]' > 296867520082839655260123481645494988367611297792 to '[email protected]' > 296867520082839655260123481645494988367611297792 failed because of closed > 2012-12-05 02:50:35.348 [error] > <0.152.0>@riak_core_handoff_manager:handle_info:274 An outbound handoff of > partition riak_kv_vnode 296867520082839655260123481645494988367611297792 was > terminated for reason: {shutdown,{error,closed}} > 2012-12-05 02:50:35.688 [error] > <0.14445.101>@riak_core_handoff_sender:start_fold:210 ownership_handoff > transfer of riak_kv_vnode from '[email protected]' > 388211372416021087647853783690262677096107081728 to '[email protected]' > 388211372416021087647853783690262677096107081728 failed because of closed > 2012-12-05 02:50:35.694 [error] > <0.152.0>@riak_core_handoff_manager:handle_info:274 An outbound handoff of > partition riak_kv_vnode 388211372416021087647853783690262677096107081728 was > terminated for reason: {shutdown,{error,closed}} > 2012-12-05 02:50:53.487 [error] > <0.15966.101>@riak_core_handoff_sender:start_fold:210 ownership_handoff > transfer of riak_kv_vnode from '[email protected]' > 296867520082839655260123481645494988367611297792 to '[email protected]' > 296867520082839655260123481645494988367611297792 failed because of closed > 2012-12-05 02:50:53.494 [error] > <0.152.0>@riak_core_handoff_manager:handle_info:274 An outbound handoff of > partition riak_kv_vnode 296867520082839655260123481645494988367611297792 was > terminated for reason: {shutdown,{error,closed}} > 2012-12-05 03:29:23.686 [error] <0.549.0>@riak_kv_vnode:init:265 Failed to > start riak_kv_eleveldb_backend Reason: {db_open,"Corruption: CURRENT file > does not end with newline"} > 2012-12-05 03:29:23.751 [error] <0.169.0> gen_server riak_core_vnode_manager > terminated with reason: no match of right hand value > {error,{db_open,"Corruption: CURRENT file does not end with newline"}} in > riak_core_vnode_manager:get_vnode/3 line 489 > 2012-12-05 03:29:23.767 [error] <0.169.0> CRASH REPORT Process > riak_core_vnode_manager with 0 neighbours exited with reason: no match of > right hand value {error,{db_open,"Corruption: CURRENT file does not end with > newline"}} in riak_core_vnode_manager:get_vnode/3 line 489 in > gen_server:terminate/6 line 747 > 2012-12-05 03:29:23.775 [error] <0.549.0> CRASH REPORT Process <0.549.0> > with 0 neighbours exited with reason: {db_open,"Corruption: CURRENT file > does not end with newline"} in gen_fsm:init_it/6 line 371 > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
