"Connection reset by peer" is almost certainly network issues. Same error:
https://github.com/netty/netty/issues/5993 mtr - ping/trace tool to find possible flaky switch/router tcpdump and/or wireshark - tools to gather and observe network packets -- Michael On 09/29/2017 10:38 AM, Thakrar, Jayesh wrote: > Other things you can do to test network issues (as a non-sysadmin user) > is to use the following commands - > > > > "netstat -i" > > This will show all the network interfaces (including any bonded > interfaces) and the in/out traffic (as rx/tx) and any errors in them. > > Note that these are cumulative counters, so you need to do a number of > samplings to ensure that the errors, if any, are not old ones. > > > > "netstat -s" > > This show more extended network level stats - and the info varies by the > OS flavor and version. > > But many of the stats are kind of self-explanatory. > > > > *From: *Jeff Jirsa <[email protected]> > *Date: *Friday, September 29, 2017 at 10:32 AM > *To: *cassandra <[email protected]> > *Subject: *Re: Issue with New Production Cluster > > > > I don't know what logging is available driver side, I'd probably be > writing a shell script to ping all three servers to see if I can prove > there's a network problem outside of cassandra first. > > > > On Fri, Sep 29, 2017 at 8:19 AM, Jonathan Baynes > <[email protected] <mailto:[email protected]>> wrote: > > Thank you Jeff, you are a credit to this community.. very helpful > > > > Last question Is there any logging I can turn on (especially with > the cassandra drive 2.5.0) that would assist in gaining more of an > insight as to what’s going on? If I’m getting any failed connections > at the cassandra layer? > > > > Thanks > > J > > > > *From:*Jeff Jirsa [mailto:[email protected] <mailto:[email protected]>] > *Sent:* 29 September 2017 16:07 > > > *To:* cassandra > *Subject:* Re: Issue with New Production Cluster > > > > The failure detector is seeing updates every 2.1-2.5 seconds, which > it will ignore because it's over the 2 second default failure > detector interval. > > > > If there's no load, there's no reason it shouldn't be seeing it far > more frequently. > > > > If you're not seeing any signs of GC pauses (like GCInspector log > lines), then it seems like you've got a network interface flapping - > maybe STP is triggering on a port, or a bad cable/interface, or > something along those lines. > > > > > > > > > > > > On Fri, Sep 29, 2017 at 7:32 AM, Jonathan Baynes > <[email protected] <mailto:[email protected]>> > wrote: > > Hi Jeff, > > > > This is version 3.0.11. Being run on Oracle Red Hat Linux. > > > > If I retry immediately it fails, leave it for 20 minutes, like I > have just now and retry it and it has worked. (?!?!) > > Ive checked all the logs (system and Debug ) and in the logs I have > this: > > > > DEBUG [GossipStage:1] 2017-09-29 15:28:04,390 > FailureDetector.java:456 - Ignoring interval time of 2102011879 > <tel:(210)%20201-1879> for /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,382 Message.java:615 > - Unexpected exception during request; channel = [id: 0x963c334d, > L:/10.172.117.61:9042 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > - R:/10.172.117.21:51853 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51853&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=NwQs5pv53MTu7RXO4tUrDSVZAtsEdXTsObmf9GWbNKE&e=>] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat > io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown > Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > DEBUG [GossipStage:1] 2017-09-29 15:28:07,389 > FailureDetector.java:456 - Ignoring interval time of 2085147992 > <tel:(208)%20514-7992> for /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > INFO [SharedPool-Worker-2] 2017-09-29 15:28:07,400 Message.java:615 > - Unexpected exception during request; channel = [id: 0xa7633234, > L:/10.172.117.61:9042 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > ! R:/10.172.117.21:51818 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51818&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=Zd17m05nbPXla0hA2GWhNKHRRo-N_nFqOfrN2_c6meY&e=>] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat > io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown > Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > INFO [SharedPool-Worker-1] 2017-09-29 15:28:07,418 Message.java:615 > - Unexpected exception during request; channel = [id: 0x8c685c2c, > L:/10.172.117.61:9042 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.61-3A9042&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=N1yysCzQbMJfPonp7tTVzQ5V6P_NGmCRCIGYb-JJS0w&e=> > ! R:/10.172.117.21:51838 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.21-3A51838&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=P-JajFY4WPK7GbEnPsK2RAoRmzPCztQnE8RT5atNnhY&e=>] > > io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() > failed: Connection reset by peerat > io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown > Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > > INFO [HANDSHAKE-/10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=>] > 2017-09-29 15:28:10,140 OutboundTcpConnection.java:523 - Handshaking > version with /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:12,288 > FailureDetector.java:456 - Ignoring interval time of 2153356364 > <tel:(215)%20335-6364> for /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:15,289 > FailureDetector.java:456 - Ignoring interval time of 2000150143 for > /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:16,305 > FailureDetector.java:456 - Ignoring interval time of 2790916098 for > /10.172.117.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:17,289 > FailureDetector.java:456 - Ignoring interval time of 2000209617 for > /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:18,391 > FailureDetector.java:456 - Ignoring interval time of 2085345347 > <tel:(208)%20534-5347> for /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:19,391 > FailureDetector.java:456 - Ignoring interval time of 2102054326 > <tel:(210)%20205-4326> for /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 > FailureDetector.java:456 - Ignoring interval time of 2000754004 for > /10.172.181.61 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:20,391 > FailureDetector.java:456 - Ignoring interval time of 2000787648 for > /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,391 > FailureDetector.java:456 - Ignoring interval time of 2254752613 > <tel:(225)%20475-2613> for /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 > FailureDetector.java:456 - Ignoring interval time of 2086206045 > <tel:(208)%20620-6045> for /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:23,392 > FailureDetector.java:456 - Ignoring interval time of 2086239333 > <tel:(208)%20623-9333> for /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:26,392 > FailureDetector.java:456 - Ignoring interval time of 2000751862 for > /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:26,516 > FailureDetector.java:456 - Ignoring interval time of 2124952635 > <tel:(212)%20495-2635> for /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:29,392 > FailureDetector.java:456 - Ignoring interval time of 2254658974 > <tel:(225)%20465-8974> for /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:30,306 > FailureDetector.java:456 - Ignoring interval time of 2000326847 for > /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 > FailureDetector.java:456 - Ignoring interval time of 2086445998 > <tel:(208)%20644-5998> for /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:32,393 > FailureDetector.java:456 - Ignoring interval time of 2086481727 > <tel:(208)%20648-1727> for /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 > FailureDetector.java:456 - Ignoring interval time of 2363746015 > <tel:(236)%20374-6015> for /10.172.117.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:35,290 > FailureDetector.java:456 - Ignoring interval time of 2151813348 for > /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:37,393 > FailureDetector.java:456 - Ignoring interval time of 2103432914 > <tel:(210)%20343-2914> for /10.172.181.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=F0Xvn8VrY2v_GJioteQGl2Xl9NrlI4T532IaGBwQL-U&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:40,394 > FailureDetector.java:456 - Ignoring interval time of 2255155834 > <tel:(225)%20515-5834> for /10.172.117.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=kcNQ4izrbn-35U5vWnv5cBKG6XfN3nmgkKozyHEfaWA&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 > FailureDetector.java:456 - Ignoring interval time of 2086966679 > <tel:(208)%20696-6679> for /10.172.117.62 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.117.62&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=S4jvfMu8gpqeNX_WcphxcT67g_pCf41sOwHuiM4LWRg&e=> > > DEBUG [GossipStage:1] 2017-09-29 15:28:41,394 > FailureDetector.java:456 - Ignoring interval time of 2087037617 > <tel:(208)%20703-7617> for /10.172.181.63 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.63&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=C4UhWVbSaGt0q4hKIko0f7TmUwYlgXU39lZ3xvAZrRc&e=> > > INFO [HANDSHAKE-/10.172.181.61 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=>] > 2017-09-29 15:28:47,132 OutboundTcpConnection.java:523 - Handshaking > version with /10.172.181.61 > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.172.181.61&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=-lefWC2Mo3ksHhmGfeD7BgS4uynKiolxK6_cLxaySOM&e=> > > > > > > The Handshake @15:28 is about the same time as I just tried it and > it working. So it has an inability to handshake with the nodes?? > > > > Any suggestions would be great, I have to go to the Network team > with this, but I’m not sure where to start… > > > > *From:*Jeff Jirsa [mailto:[email protected] <mailto:[email protected]>] > *Sent:* 29 September 2017 14:54 > *To:* cassandra > *Subject:* Re: Issue with New Production Cluster > > > > What version? > > If you retry immediately, does it reconnect? > > Anything in the logs? > > > > What you describe is atypical - timeouts on queries can (and will) > happen occasionally under load, but timeout on connect is atypical. > Any sign of networking issues/slowness/dns problems/etc? > > > > > > On Fri, Sep 29, 2017 at 6:43 AM, Jonathan Baynes > <[email protected] <mailto:[email protected]>> > wrote: > > Hi Community, > > > > I have a 6 node ring, covering 2 DC’s, the ring isn’t being used yet > and we are just in the connectivity and testing phase. So the boxes > are NOT under any load. > > > > I’ve gone to connect to CQLSH this afternoon and I’ve had this returned: > > > > cqlsh xx.xxx.xxx.xx -u cassandra -p cassandra > > Connection error: ('Unable to connect to any servers', { > xx.xxx.xxx.xx’: OperationTimedOut('errors=Timed out creating > connection (5 seconds), last_host=None',)}) > > > > Status of the service is “Running” > > Nodetool status is UP for all nodes. > > Tested telnet on all ports and they are all up and connecting. > > Checked the PID is up for cassandra (it is) > > > > Any idea how I can debug the cause of this further? > > > > > > Thanks > > J > > > > > > > > *Jonathan Baynes* > > DBA* > *Tradeweb Europe Limited > > Moor Place • 1 Fore Street Avenue > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> > • > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=a46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo&e=> > London > EC2Y 9DT > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w&e=> > P +44 (0)20 77760988 <tel:+44%2020%207776%200988> • > > <https://maps.google.com/?q=1%0D+Fore+Street+Avenue%C2%A0%C2%A0%E2%80%A2+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-250D-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-250D-2BEC2Y-2B9DT-2B-253Chttps-3A__urldefense.proofpoint.com_v2_url-3Fu-253Dhttps-2D3A-5F-5Fmaps.google.com-5F-2D3Fq-2D3D1-2D2BFore-2D2BStreet-2D2BAvenue-2D25C2-2D25A0-2D25C2-2D25A0-2D25E2-2D2580-2D25A2-2D25C2-2D25A0-2D25C2-2D25A0London-2D2BEC2Y-2D2B9DT-2D26entry-2D3Dgmail-2D26source-2D3Dg-2526d-253DDwMFaQ-2526c-253DsA0VaJZJFLZREu2pbPeqjXHJ-2DWd9NNzgHW3gpUOLSSk-2526r-253DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-2DdiQ8-2526m-253DynRqj6nQ2Rc0PoXhoOv-5F0HAaz8G8xuGO8wNbuHB-2D3pU-2526s-253D46cq-2D66-2DQUqnj-2DtaNMRFYe7YK3eNez6ogHJCv014X9w-2526e-253D-253E-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DLHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA%26s%3Da46k9sWzI3NKA2ZlZh8o78iGyheFPgXJFCmwuhYy-Qo%26e%3D%3E%C2%A0%C2%A0London%0D+EC2Y+9DT+%3Chttps://urldefense.proofpoint.com/v2/url?u%3Dhttps-3A__maps.google.com_-3Fq-3D1-2BFore-2BStreet-2BAvenue-25C2-25A0-25C2-25A0-25E2-2580-25A2-25C2-25A0-25C2-25A0London-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg%26d%3DDwMFaQ%26c%3DsA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk%26r%3DCNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8%26m%3DynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU%26s%3D46cq-66-QUqnj-taNMRFYe7YK3eNez6ogHJCv014X9w%26e%3D%3E&entry=gmail&source=g> > F > +44 (0)20 7776 3201 <tel:+44%2020%207776%203201> • M +44 (0)xxxx > xxxxxx > > [email protected] <mailto:[email protected]> > > > > id:[email protected] <http://www.tradeweb.com/> > follow us: *id:[email protected]* > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=saD3ZbF7LhTn3IHCHWSzhGp8Ns6SIRO0s0KXCnEbqLU&e=> > > id:[email protected] > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=Duw_ud1x7HIX3h-0UEtKorCrlKn8C5tUoH6tkrvXDy0&e=> > > ————————————————————————————————————————— > > A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for > electronic fixed income, derivatives and ETF trading > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. > If you are not the intended recipient (or have received this e-mail > in error) please notify the sender immediately and destroy it. Any > unauthorized copying, disclosure or distribution of the material in > this e-mail is strictly forbidden. Tradeweb reserves the right to > monitor all e-mail communications through its networks. If you do > not wish to receive marketing emails about our products / services, > please let us know by contacting us, either by email at > [email protected] <mailto:[email protected]> or by writing > to us at the registered office of Tradeweb in the UK, which is: > Tradeweb Europe Limited (company number 3912826), 1 Fore Street > Avenue London EC2Y 9DT > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=ynRqj6nQ2Rc0PoXhoOv_0HAaz8G8xuGO8wNbuHB-3pU&s=VSn9NuhaGsP_VCpKoBZFV2hz_46iJ3VhVk5rYBKHeN8&e=>. > To see our privacy policy, visit our website @ www.tradeweb.com > <http://www.tradeweb.com>. > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. > If you are not the intended recipient (or have received this e-mail > in error) please notify the sender immediately and destroy it. Any > unauthorized copying, disclosure or distribution of the material in > this e-mail is strictly forbidden. Tradeweb reserves the right to > monitor all e-mail communications through its networks. If you do > not wish to receive marketing emails about our products / services, > please let us know by contacting us, either by email at > [email protected] <mailto:[email protected]> or by writing > to us at the registered office of Tradeweb in the UK, which is: > Tradeweb Europe Limited (company number 3912826), 1 Fore Street > Avenue London EC2Y 9DT > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3Dnumber-2B3912826-29-2C-2B1-2BFore-2BStreet-2BAvenue-2BLondon-2BEC2Y-2B9DT-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=LHuC2PiAE_qEnK8nzWN-xEQVrqsbc2SCkDvN8UV7FvA&s=m5TdlyiKgTyluGY2ssC5qtvdFJw9hAnGpfbDAKEHzxA&e=>. > To see our privacy policy, visit our website @ www.tradeweb.com > <http://www.tradeweb.com>. > > > > ________________________________________________________________________ > > This e-mail may contain confidential and/or privileged information. > If you are not the intended recipient (or have received this e-mail > in error) please notify the sender immediately and destroy it. Any > unauthorized copying, disclosure or distribution of the material in > this e-mail is strictly forbidden. Tradeweb reserves the right to > monitor all e-mail communications through its networks. If you do > not wish to receive marketing emails about our products / services, > please let us know by contacting us, either by email at > [email protected] <mailto:[email protected]> or by writing > to us at the registered office of Tradeweb in the UK, which is: > Tradeweb Europe Limited (company number 3912826), 1 Fore Street > Avenue London EC2Y 9DT > > <https://maps.google.com/?q=number+3912826),+1+Fore+Street+Avenue+London+EC2Y+9DT&entry=gmail&source=g>. > To see our privacy policy, visit our website @ www.tradeweb.com > <http://www.tradeweb.com>. > > >
