I'm not receiving any errors actually. I'm far from an expert in riak logs, but the console.log messages seem to indicate the handoff is working… just extremely slowly. Here's a snippet from the console.log of the node that is attempting to leave…
2013-09-24 20:59:43.991 [info] <0.7140.0>@riak_core_handoff_receiver:process_message:99 Receiving handoff data for partition riak_kv_vnode:685078892498860742907977265335757665463718379520 2013-09-24 20:59:43.994 [info] <0.7140.0>@riak_core_handoff_receiver:handle_info:69 Handoff receiver for partition 685078892498860742907977265335757665463718379520 exited after processing 0 objects 2013-09-24 20:59:44.035 [info] <0.7142.0>@riak_core_handoff_receiver:process_message:99 Receiving handoff data for partition riak_kv_vnode:45671926166590716193865151022383844364247891968 2013-09-24 20:59:44.038 [info] <0.7142.0>@riak_core_handoff_receiver:handle_info:69 Handoff receiver for partition 45671926166590716193865151022383844364247891968 exited after processing 0 objects 2013-09-24 20:59:44.068 [info] <0.7144.0>@riak_core_handoff_receiver:process_message:99 Receiving handoff data for partition riak_kv_vnode:1415829711164312202009819681693899175291684651008 2013-09-24 20:59:44.073 [info] <0.7144.0>@riak_core_handoff_receiver:handle_info:69 Handoff receiver for partition 1415829711164312202009819681693899175291684651008 exited after processing 0 objects The percentages in member_status still have not changed though. Thank you again for any help!!! Dave On Sep 24, 2013, at 3:42 PM, Dmitry Demeshchuk <[email protected]> wrote: > Seems like a potential problem with handoff. We had similar problems > upgrading from 1.2.1 to 1.4.0. Check the logs for handoff errors (something > like <<"unknown_msg">> or similar). > > If that's the case, leave that node be, and do in-place upgrade for the rest > of the nodes, without making them leave the cluster. The third node will > probably leave after that, so you'll be able to re-join it. > > > On Tue, Sep 24, 2013 at 12:33 PM, David Greenstein <[email protected]> > wrote: > > I'm performing a rolling upgrade to 1.4.2 (from 1.3.1). The first two nodes > that I replaced left the cluster without an issue and the new nodes joined > without an issue. Now, the next node seams to be in a state where it won't > leave the cluster. The status is leaving but it has been pending for several > hours. Perhaps it is due to the pending ownership handoff from ring_status > that also doesn't seem to be completing. Any insight or help on how to > "kickstart" the leave would be greatly appreciated! > > Dave > > [user@ip-10-0-1-12 user]# /db/riak/bin/riak-admin ring_status > ================================== Claimant > =================================== > Claimant: '[email protected]' > Status: up > Ring Ready: true > > ============================== Ownership Handoff > ============================== > Owner: [email protected] > Next Owner: [email protected] > > Index: 456719261665907161938651510223838443642478919680 > Waiting on: [riak_kv_vnode] > Complete: [riak_pipe_vnode] > > ------------------------------------------------------------------------------- > > ============================== Unreachable Nodes > ============================== > All nodes are up and reachable > > > > > > [user@ip-10-0-1-12 user]# /db/riak/bin/riak-admin member_status > ================================= Membership > ================================== > Status Ring Pending Node > ------------------------------------------------------------------------------- > leaving 14.1% 14.1% '[email protected]' > valid 14.1% 14.1% '[email protected]' > valid 14.1% 14.1% '[email protected]' > valid 17.2% 15.6% '[email protected]' > valid 12.5% 14.1% '[email protected]' > valid 14.1% 14.1% '[email protected]' > valid 14.1% 14.1% '[email protected]' > ------------------------------------------------------------------------------- > Valid:6 / Leaving:1 / Exiting:0 / Joining:0 / Down:0 > > > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > -- > Best regards, > Dmitry Demeshchuk
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
