Re: unreachable nodes mystery in describecluster output

2016-08-03 Thread Aleksandr Ivanov
orks. > Is there a VPN between DCs? Is there room for improvement at the network > level? TCP tuning, etc. I'm not saying you won't have unreachable nodes but > it's worth it if you can. > > Romain > > Le Mercredi 3 août 2016 15h02, Aleksandr Ivanov <ale...@gm

Re: unreachable nodes mystery in describecluster output

2016-08-03 Thread Romain Hardouin
That's a good news if describecluster shows the same version on each node. Try with a high timeout like 120 seconds to see if it works. Is there a VPN between DCs? Is there room for improvement at the network level? TCP tuning, etc. I'm not saying you won't have unreachable nodes but it's worth

Re: unreachable nodes mystery in describecluster output

2016-08-03 Thread Aleksandr Ivanov
> I use 25sec timeout (--request-timeout 25) Apart the unreachable nodes, do you know if all nodes have the same schema > version? > "nodetool gossipinfo" shows same scheme version on all nodes. Best, > > Romain >

Re: unreachable nodes mystery in describecluster output

2016-08-03 Thread Romain Hardouin
Hi, The latency is high... Regarding the ALTER, did you try to increase the timeout with "cqlsh --request-timeout=REQUEST_TIMEOUT"? Because the default is 10 seconds. Apart the unreachable nodes, do you know if all nodes have the same schema version?  Best, Romain

unreachable nodes mystery in describecluster output

2016-08-03 Thread Aleksandr Ivanov
Hello, I'm running v3.0.8 in multi-data center deployment (6 DCs, 6 nodes per DC, maximum latency between some nodes ~200ms). After clean cluster start I run into issue when "nodetool descibecluster" shows that some random nodes from deployment are UNREACHABLE however in "nodetool status" or

Unreachable Nodes

2013-05-22 Thread Vasileios Vlachos
Hello All, A while ago we had 3 cassandra nodes on Amazon. At some point we decided to buy some servers and deploy cassandra there. The problem is that since then we have a list of dead IPs listed as UNREACHABLE nodes when we run describe cluster on cassandra-cli. I have seen other posts which

Re: Unreachable Nodes

2013-05-22 Thread Alain RODRIGUEZ
the CLI? That's a good thing. Gossiper just keep this information for a while (7 or 10 days by default off the top off my head), but this doesn't harm your cluster in any ways, but having UNREACHABLE nodes could have been annoying. By the way gossipinfo shows you those nodes as STATUS:LEFT which

Re: Unreachable Nodes

2013-05-22 Thread Vasileios Vlachos
off the top off my head), but this doesn't harm your cluster in any ways, but having UNREACHABLE nodes could have been annoying. By the way gossipinfo shows you those nodes as STATUS:LEFT which is good. I am quite sure that this status changed when you used the jmx unsafeAssassinateEndpoint

Re: Unreachable Nodes

2013-05-22 Thread Brian Tarbox
UNREACHABLE nodes could have been annoying. By the way gossipinfo shows you those nodes as STATUS:LEFT which is good. I am quite sure that this status changed when you used the jmx unsafeAssassinateEndpoint. do a full cluster restart (I presume that means a rolling restart - not shut-down the entire

Re: Unreachable Nodes

2013-05-22 Thread Alain RODRIGUEZ
I had to face this too, but precisely the unsafeAssassinateEndpoint removed the UNREACHABLE nodes (from describe cluster - CLI). After that, I had these ghost host marked as STATUS:LEFT on gossipinfo (nodetool) and my truncate could run properly. But this is only my own experience, and you might