To answer my own question… I did a `stop node`, and `mark down` through the Riak Control interface of the one stuck at 0.2% data (235) and trying to send files to 234.
leaving 0.2% 0.0% '[email protected]<mailto:[email protected]>' Then restarted 234: leaving 15.6% 15.8% '[email protected]<mailto:[email protected]>' And joined it back to the cluster. (it was thinking it was a cluster of 1, though not reported as this way through Control or member_status. Now it thinks it is back and ready for duty. ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- down 0.2% 0.2% '[email protected]<mailto:[email protected]>' down 16.6% 16.6% '[email protected]<mailto:[email protected]>' valid 2.7% 9.0% '[email protected]<mailto:[email protected]>' valid 2.9% 8.2% '[email protected]<mailto:[email protected]>' valid 2.1% 6.8% '[email protected]<mailto:[email protected]>' valid 17.8% 11.3% '[email protected]<mailto:[email protected]>' valid 17.0% 14.5% '[email protected]<mailto:[email protected]>' valid 13.1% 12.9% '[email protected]<mailto:[email protected]>' valid 13.5% 10.5% '[email protected]<mailto:[email protected]>' valid 14.1% 10.0% '[email protected]<mailto:[email protected]>' ------------------------------------------------------------------------------- Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:2 I'm pretty sure I had done both of these things before (marking as down, restarting, etc), but I guess the time wasn't right. That is fine. At least the cluster is blowing chunks back and forth again like a good little party animal. Best Regards, Austin On Jun 4, 2012, at 2:16 PM, Swinney, Austin wrote: Hi All, The following is about leveldb, riak (1.1.1 2012-03-07) RedHat x86_64, and one riak newbie known as me! I had this problem over the weekend whereby two nodes are leaving and they are both stuck trying to send transfers from one to the other. I had backed them up with tar, and after they became stuck, I tried launching new instances with those levedb tar file backups. But those new hosts, although listed in connected_nodes, are not in the ring_members. Are there any work arounds to resolving the stuck ownership handoff between the two leaving nodes? I tried different scenarios of marking them as down. That didn't seem to help. ring_status indicated it wanted them down, then it wanted them back online. etc. I don't really need either one. I'd like to eject them both from the cluster and have it rebalance onto the new nodes. Both these were asked to leave: Owner: [email protected]<mailto:[email protected]> Next Owner: [email protected]<mailto:[email protected]> ring_status output: [root@ip-10-0-0-171 riak]# riak-admin ring_status Attempting to restart script through sudo -u riak ================================== Claimant =================================== Claimant: '[email protected]<mailto:[email protected]>' Status: up Ring Ready: false ============================== Ownership Handoff ============================== Owner: [email protected]<mailto:[email protected]> Next Owner: [email protected]<mailto:[email protected]> Index: 727896323280039539339725844419242519555200778240 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode,riak_search_vnode] Index: 876330083321459366969787585241990013739006427136 Waiting on: [riak_kv_vnode] Complete: [riak_pipe_vnode,riak_search_vnode] ------------------------------------------------------------------------------- ============================== Unreachable Nodes ============================== All nodes are up and reachable And member_status output: [root@ip-10-0-0-168 ~]# riak-admin member_status Attempting to restart script through sudo -u riak ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- down 16.6% 16.6% '[email protected]<mailto:[email protected]>' leaving 0.2% 0.0% '[email protected]<mailto:[email protected]>' leaving 15.6% 15.8% '[email protected]<mailto:[email protected]>' valid 0.0% 0.0% '[email protected]<mailto:[email protected]>' valid 0.0% 0.0% '[email protected]<mailto:[email protected]>' valid 18.0% 18.0% '[email protected]<mailto:[email protected]>' valid 17.4% 17.4% '[email protected]<mailto:[email protected]>' valid 15.4% 15.4% '[email protected]<mailto:[email protected]>' valid 16.8% 16.8% '[email protected]<mailto:[email protected]>' ------------------------------------------------------------------------------- Valid:6 / Leaving:1 / Exiting:0 / Joining:1 / Down:1 Thanks for your input! Austin _______________________________________________ riak-users mailing list [email protected]<mailto:[email protected]> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
