> What may be causing high IO wait on 2.new and 3.new? Do you have RF=3?
The problem with 'nodetool ring' in terms of interpreting those '0%' is that it does not take RF and replication strategy into account. If you have RF=3, whatever data has its primary range assigned to node N will also be on N+1 and N+2. Additionally, whatever data has it's primary range assigned to N-1 and N-2 will also have a copy on N. So despite all the old nodes showing up as 0% in nodetool ring, they are in fact responsible for data. And you would expect to see nodes N+2 and N-2 be the two "old" nodes affected by decommissioning node N. (Unless I'm tripping myself up somewhere now...) -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)