Hi Jeppe, As you suspected, this looks like index corruption in Search that's preventing handoff from finishing. Specifically, you'll need to delete the segment files for the two partitions' indexes and rebuild those indexes post-transfer.
Here's the full process: - Stop each node that owns the partitions in question. - Delete the data directory for each partition (which contains the segment files). It should be something like: "rm -rf /var/lib/riak/merge_index/<p>" - Restart each node - Wait for the transfers to complete - Rebuild the indexes in question [1] Let us know if you run into any further issues. Mark [1] http://docs.basho.com/riak/latest/ops/running/recovery/repairing-indexes/ On Tue, Nov 19, 2013 at 4:26 AM, Jeppe Toustrup <[email protected]>wrote: > Hi > > I have recently added two extra nodes to the now seven node Riak > cluster. The rebalancing following the expansion worked fine, except > for two partitions which seem to not being able to go through. Running > "riak-admin ring-status" shows the following: > > ============================== Ownership Handoff > ============================== > Owner: [email protected] > Next Owner: [email protected] > > Index: 239777612374601260017792042867515182912301432832 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > Index: 696496874040508421956443553091353626554780352512 > Waiting on: [] > Complete: [riak_kv_vnode,riak_pipe_vnode] > > > ------------------------------------------------------------------------------- > > I can see from the log file on the source node (10.0.0.96) that it has > made numerous attempt to transfer the partitions, but it ends up > failing all the time. Here's an except of the log file showing the > lines from when the transfer attempt ends up failing: > > 2013-11-18 12:29:03.694 [error] emulator Error in process <0.5745.8> > on node '[email protected]' with exit value: > {badarg,[{erlang,binary_to_term,[<<29942 > > bytes>>],[]},{mi_segment,iterate_all_bytes,2,[{file,"src/mi_segment.erl"},{line,167}]},{mi_server,'-group_iterator/2-fun-1-',2,[{file,"src/mi_server.erl"},{line,725}]},{mi_server,'-group_iterator/2-fun-0-'... > 2013-11-18 12:29:03.885 [error] <0.3269.0>@mi_server:handle_info:524 > lookup/range failure: > > {badarg,[{erlang,binary_to_term,[<<131,109,0,0,244,240,108,109,102,97,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,...>>],...},...]} > 2013-11-18 12:29:03.889 [error] > <0.30353.0>@merge_index_backend:async_fold_fun:116 failed to iterate > the index with reason > > {badarg,[{erlang,binary_to_term,[<<131,109,0,0,244,240,108,109,102,97,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,...>>],...},...]} > and partial acc > > {{ho_acc,1,{error,closed},#Fun<riak_core_handoff_sender.15.9980475>,riak_search_vnode,<0.3268.0>,#Port<0.133923>,{696496874040508421956443553091353626554780352512,696496874040508421956443553091353626554780352512},{ho_stats,{1384,775537,708230},undefined,249,1050473},gen_tcp,246,1053542,[<<1,131,104,4,109,0,0,0,8,109,111,114,101,111,118,101,114,109,0,0,0,13,99,97,110,111,110,105,99,97,108,68,97,116,101,109,0,0,0,14,50,48,49,51,49,48,48,55,49,50,49,49,53,52,108,0,0,0,57,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,102,98,56,55,98,102,50,51,55,51,98,51,98,56,56,100,52,48,51,52,51,50,100,97,97,55,51,99,55,101,98,99,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,154,61,194,52,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,102,51,101,54,55,54,57,56,99,55,52,49,102,57,56,50,101,50,102,48,55,54,101,57,57,101,97,56,100,100,102,57,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,137,181,162,60,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,101,98,57,51,102,52,55,48,53,52,55,52,50,56,98,49,49,50,101,57,98,50,98,49,52,51,52,57,52,52,98,50,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,31,143,101,69,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,101,55,49,52,57,49,53,52,49,102,48,55,102,102,97,99,51,48,98,51,52,51,56,52,49,98,56,50,54,55,56,48,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,214,30,30,54,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,101,53,56,53,97,52,48,49,98,55,97,52,56,57,56,48,53,55,54,53,101,99,98,98,98,98,101,52,97,101,48,100,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,41,65,120,56,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,100,101,56,99,99,99,51,56,53,56,52,48,53,97,51,49,57,49,48,97,51,55,57,57,98,49,54,52,53,100,55,50,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,222,186,251,66,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,...>>,...],...},...} > 2013-11-18 12:29:03.889 [error] > <0.15384.7>@riak_core_handoff_sender:start_fold:269 ownership_transfer > transfer of riak_search_vnode from '[email protected]' > 696496874040508421956443553091353626554780352512 to '[email protected]' > 696496874040508421956443553091353626554780352512 failed because of > error:{badrecord,ho_acc} > > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,193}]}] > > > I've got an idea that it me be caused by corrupt data, but the > question is then how I can get this corrected, enabling the transfer > to complete. Any suggestions? > > We are using Riak 1.4.2 with LevelDB as the backend on Ubuntu 12.04. > Riak was installed through the apt.basho.com repository. Let me know > if you need any more information. > > -- > Jeppe Fihl Toustrup > Operations Engineer > Falcon Social > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
