Re: Connection refused - 127.0.0.1-Gossip
Besides seed nodes (which are resolved correctly) I could not find anything that would require DNS resolution in Cassandra config. On 12/10/17, 12:42 AM, "Jeff Jirsa" <jji...@apache.org> wrote: Does everything have valid forward/reverse DNS? Nothing that's going to reverse to a domain that will then resolve to 127.0.0.1? On 2017-12-07 01:37, "Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco)" <mka...@cisco.com> wrote: > I tried setting it up with pod ip, but it did not help. > > From: "ZAIDI, ASAD A" <az1...@att.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Wednesday, December 6, 2017 at 8:45 PM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: RE: Connection refused - 127.0.0.1-Gossip > > Throwing my $ .002 > > Rpc_address defaults to localhosts that if not set, picks value from dns/hostname file. May be you can try setting rpc_address: - see if this helps! > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Connection refused - 127.0.0.1-Gossip
Does everything have valid forward/reverse DNS? Nothing that's going to reverse to a domain that will then resolve to 127.0.0.1? On 2017-12-07 01:37, "Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco)" <mka...@cisco.com> wrote: > I tried setting it up with pod ip, but it did not help. > > From: "ZAIDI, ASAD A" <az1...@att.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Wednesday, December 6, 2017 at 8:45 PM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: RE: Connection refused - 127.0.0.1-Gossip > > Throwing my $ .002 > > Rpc_address defaults to localhosts that if not set, picks value from > dns/hostname file. May be you can try setting rpc_address: ip> - see if this helps! > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Connection refused - 127.0.0.1-Gossip
I tried setting it up with pod ip, but it did not help. From: "ZAIDI, ASAD A" <az1...@att.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Wednesday, December 6, 2017 at 8:45 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: RE: Connection refused - 127.0.0.1-Gossip Throwing my $ .002 Rpc_address defaults to localhosts that if not set, picks value from dns/hostname file. May be you can try setting rpc_address: - see if this helps! From: Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) [mailto:mka...@cisco.com] Sent: Wednesday, December 06, 2017 2:19 AM To: user@cassandra.apache.org Subject: Re: Connection refused - 127.0.0.1-Gossip Thanks for any ideas/hints, any straw is worth checking at this point ☺ Well, the clusters “work”, data is correctly stored and queries. I’m interested in why it tries to open a gossip to localhost, and what kind of (performance) impact could this have on clusters. The env vars are correctly passed, and cassandra yaml seems to be correctly set. We are using Cassandra docker image. listen_address: 100.110.253.6 (correct pod ip) # listen_interface: eth0 # listen_interface_prefer_ipv6: false broadcast_rpc_address: 100.110.253.6 It’s also observable with minikube and single C* node on local machine. From: Lerh Chuan Low <l...@instaclustr.com<mailto:l...@instaclustr.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, December 5, 2017 at 11:14 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Connection refused - 127.0.0.1-Gossip I think as Jeff mentioned it sounds like a configuration issue, are you sure you are using the same configmap/however it's being passed in and just throwing out ideas, maybe the pods are behind a http proxy and you may have forgotten to pass in the env vars? On 6 December 2017 at 08:45, Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: I don't have any k8 clusters to test with, but do you know how your yaml translates to cassandra.yaml ? What are the listen/broadcast addresses being set? On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) <mka...@cisco.com<mailto:mka...@cisco.com>> wrote: We are experiencing following issues with Cassandra on our kubernetes clusters: ``` @ kubectl exec -it cassandra-cassandra-0 -- tail /var/log/cassandra/debug.log DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to localhost/127.0.0.1<http://127.0.0.1> java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_131] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnection.connect(OutboundTcpConnection.java:433) [apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.0.jar:3.11.0] ``` Basically, it’s tons and tons of the same message over and over (on all clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp connection to localhost (?) for gossiping. What we know: -
RE: Connection refused - 127.0.0.1-Gossip
Throwing my $ .002 Rpc_address defaults to localhosts that if not set, picks value from dns/hostname file. May be you can try setting rpc_address: - see if this helps! From: Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) [mailto:mka...@cisco.com] Sent: Wednesday, December 06, 2017 2:19 AM To: user@cassandra.apache.org Subject: Re: Connection refused - 127.0.0.1-Gossip Thanks for any ideas/hints, any straw is worth checking at this point ☺ Well, the clusters “work”, data is correctly stored and queries. I’m interested in why it tries to open a gossip to localhost, and what kind of (performance) impact could this have on clusters. The env vars are correctly passed, and cassandra yaml seems to be correctly set. We are using Cassandra docker image. listen_address: 100.110.253.6 (correct pod ip) # listen_interface: eth0 # listen_interface_prefer_ipv6: false broadcast_rpc_address: 100.110.253.6 It’s also observable with minikube and single C* node on local machine. From: Lerh Chuan Low <l...@instaclustr.com<mailto:l...@instaclustr.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, December 5, 2017 at 11:14 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Connection refused - 127.0.0.1-Gossip I think as Jeff mentioned it sounds like a configuration issue, are you sure you are using the same configmap/however it's being passed in and just throwing out ideas, maybe the pods are behind a http proxy and you may have forgotten to pass in the env vars? On 6 December 2017 at 08:45, Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: I don't have any k8 clusters to test with, but do you know how your yaml translates to cassandra.yaml ? What are the listen/broadcast addresses being set? On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) <mka...@cisco.com<mailto:mka...@cisco.com>> wrote: We are experiencing following issues with Cassandra on our kubernetes clusters: ``` @ kubectl exec -it cassandra-cassandra-0 -- tail /var/log/cassandra/debug.log DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to localhost/127.0.0.1<http://127.0.0.1> java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_131] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnection.connect(OutboundTcpConnection.java:433) [apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=O8QyTZVJIRUNcvKDLpX2IcSY35f_-yEoGJt0ELBa2xE=Lqf0BleAj9UkULmHnPY_AdTK0vXmP--1Yh1oSWSbjz0=>.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.0.jar:3.11.0] ``` Basically, it’s tons and tons of the same message over and over (on all clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp connection to localhost (?) for gossiping. What we know: - does not happen on Cassandra 3.0.15, but happen on 3.11.1 (same configuration). - does happen even on minikube-single-Cassandra “cluster”. - does not happen on docker-compose Cassandra cluster, only on kubernetes one. Our configuration is pretty much this helm chart: https://github.com/kubernetes/charts/blob/master/incubator/cassandra/values.yaml<https://urld
Re: Connection refused - 127.0.0.1-Gossip
Thanks for any ideas/hints, any straw is worth checking at this point ☺ Well, the clusters “work”, data is correctly stored and queries. I’m interested in why it tries to open a gossip to localhost, and what kind of (performance) impact could this have on clusters. The env vars are correctly passed, and cassandra yaml seems to be correctly set. We are using Cassandra docker image. listen_address: 100.110.253.6 (correct pod ip) # listen_interface: eth0 # listen_interface_prefer_ipv6: false broadcast_rpc_address: 100.110.253.6 It’s also observable with minikube and single C* node on local machine. From: Lerh Chuan Low <l...@instaclustr.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, December 5, 2017 at 11:14 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: Connection refused - 127.0.0.1-Gossip I think as Jeff mentioned it sounds like a configuration issue, are you sure you are using the same configmap/however it's being passed in and just throwing out ideas, maybe the pods are behind a http proxy and you may have forgotten to pass in the env vars? On 6 December 2017 at 08:45, Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: I don't have any k8 clusters to test with, but do you know how your yaml translates to cassandra.yaml ? What are the listen/broadcast addresses being set? On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco) <mka...@cisco.com<mailto:mka...@cisco.com>> wrote: We are experiencing following issues with Cassandra on our kubernetes clusters: ``` @ kubectl exec -it cassandra-cassandra-0 -- tail /var/log/cassandra/debug.log DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to localhost/127.0.0.1<http://127.0.0.1> java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131] at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_131] at org.apache.cassandra.net<http://org.apache.cassandra.net>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<http://org.apache.cassandra.net>.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<http://org.apache.cassandra.net>.OutboundTcpConnection.connect(OutboundTcpConnection.java:433) [apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.net<http://org.apache.cassandra.net>.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.0.jar:3.11.0] ``` Basically, it’s tons and tons of the same message over and over (on all clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp connection to localhost (?) for gossiping. What we know: - does not happen on Cassandra 3.0.15, but happen on 3.11.1 (same configuration). - does happen even on minikube-single-Cassandra “cluster”. - does not happen on docker-compose Cassandra cluster, only on kubernetes one. Our configuration is pretty much this helm chart: https://github.com/kubernetes/charts/blob/master/incubator/cassandra/values.yaml Do you have any idea what it could be related to?
Re: Connection refused - 127.0.0.1-Gossip
I think as Jeff mentioned it sounds like a configuration issue, are you sure you are using the same configmap/however it's being passed in and just throwing out ideas, maybe the pods are behind a http proxy and you may have forgotten to pass in the env vars? On 6 December 2017 at 08:45, Jeff Jirsawrote: > I don't have any k8 clusters to test with, but do you know how your yaml > translates to cassandra.yaml ? What are the listen/broadcast addresses > being set? > > > On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS > LTD at Cisco) wrote: > >> We are experiencing following issues with Cassandra on our kubernetes >> clusters: >> >> ``` >> >> @ kubectl exec -it cassandra-cassandra-0 -- tail >> /var/log/cassandra/debug.log >> >> DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 >> 09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to >> localhost/127.0.0.1 >> >> java.net.ConnectException: Connection refused >> >> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131] >> >> at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131] >> >> at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131] >> >> at sun.nio.ch.SocketChannelImpl.c >> onnect(SocketChannelImpl.java:648) ~[na:1.8.0_131] >> >> at org.apache.cassandra.net.Outbo >> undTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) >> ~[apache-cassandra-3.11.0.jar:3.11.0] >> >> at org.apache.cassandra.net.Outbo >> undTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) >> ~[apache-cassandra-3.11.0.jar:3.11.0] >> >> at org.apache.cassandra.net.Outbo >> undTcpConnection.connect(OutboundTcpConnection.java:433) >> [apache-cassandra-3.11.0.jar:3.11.0] >> >> at org.apache.cassandra.net.Outbo >> undTcpConnection.run(OutboundTcpConnection.java:262) >> [apache-cassandra-3.11.0.jar:3.11.0] >> >> ``` >> >> >> >> Basically, it’s tons and tons of the same message over and over (on all >> clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp >> connection to localhost (?) for gossiping. >> >> >> >> What we know: >> >> - does not happen on Cassandra 3.0.15, but happen on 3.11.1 (same >> configuration). >> >> - does happen even on minikube-single-Cassandra “cluster”. >> >> - does not happen on docker-compose Cassandra cluster, only on kubernetes >> one. >> >> >> >> Our configuration is pretty much this helm chart: >> https://github.com/kubernetes/charts/blob/master/incubator/c >> assandra/values.yaml >> >> >> >> Do you have any idea what it could be related to? >> >> >> > >
Re: Connection refused - 127.0.0.1-Gossip
I don't have any k8 clusters to test with, but do you know how your yaml translates to cassandra.yaml ? What are the listen/broadcast addresses being set? On Tue, Dec 5, 2017 at 6:09 AM, Marek Kadek -T (mkadek - CONSOL PARTNERS LTD at Cisco)wrote: > We are experiencing following issues with Cassandra on our kubernetes > clusters: > > ``` > > @ kubectl exec -it cassandra-cassandra-0 -- tail > /var/log/cassandra/debug.log > > DEBUG [MessagingService-Outgoing-localhost/127.0.0.1-Gossip] 2017-12-05 > 09:02:06,560 OutboundTcpConnection.java:545 - Unable to connect to > localhost/127.0.0.1 > > java.net.ConnectException: Connection refused > > at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_131] > > at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_131] > > at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_131] > > at > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) > ~[na:1.8.0_131] > > at org.apache.cassandra.net.OutboundTcpConnectionPool. > newSocket(OutboundTcpConnectionPool.java:146) > ~[apache-cassandra-3.11.0.jar:3.11.0] > > at org.apache.cassandra.net.OutboundTcpConnectionPool. > newSocket(OutboundTcpConnectionPool.java:132) > ~[apache-cassandra-3.11.0.jar:3.11.0] > > at org.apache.cassandra.net.OutboundTcpConnection.connect( > OutboundTcpConnection.java:433) [apache-cassandra-3.11.0.jar:3.11.0] > > at org.apache.cassandra.net.OutboundTcpConnection.run( > OutboundTcpConnection.java:262) [apache-cassandra-3.11.0.jar:3.11.0] > > ``` > > > > Basically, it’s tons and tons of the same message over and over (on all > clusters, all C* nodes). It tries roughly 4-5 times a second to open a tcp > connection to localhost (?) for gossiping. > > > > What we know: > > - does not happen on Cassandra 3.0.15, but happen on 3.11.1 (same > configuration). > > - does happen even on minikube-single-Cassandra “cluster”. > > - does not happen on docker-compose Cassandra cluster, only on kubernetes > one. > > > > Our configuration is pretty much this helm chart: > https://github.com/kubernetes/charts/blob/master/incubator/ > cassandra/values.yaml > > > > Do you have any idea what it could be related to? > > >