I’ve resolved the issue. These hosts were initially created in a different
VLAN. There was a holdover entry in /etc/hosts pointing to the old IP Addr for
this host.
So, the return back to the client never happened because it was sending the
connection to the old IP to a host that did not exist.
Thanks for looking!
James
From: James Lovato <james.lov...@iacapps.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Thursday, January 25, 2018 at 10:44 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
It starts up, I can telnet to the port. Also the log states so:
^C[root@dfprdbcdlpcs5 logs]# grep JMX cassandra.log
INFO [main] 2018-01-25 10:52:54,176 JMXServerUtils.java:249 - Configured JMX
server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:52:54,379 StartupChecks.java:164 - JMX is enabled to
receive remote connections on port: 7199
INFO [main] 2018-01-25 10:55:25,751 JMXServerUtils.java:249 - Configured JMX
server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:55:25,957 StartupChecks.java:164 - JMX is enabled to
receive remote connections on port: 7199
From: Roger Brown <roger.br...@perfectsearchcorp.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Thursday, January 25, 2018 at 9:51 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
My first thought is restart cassandra and monitor its log to make sure it
starts up.
On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa
<jji...@gmail.com<mailto:jji...@gmail.com>> wrote:
Can you telnet to the 7199 port?
--
Jeff Jirsa
On Jan 25, 2018, at 8:00 AM, James Lovato
<james.lov...@iacapps.com<mailto:james.lov...@iacapps.com>> wrote:
I have a 15 node cluster, 5 in each of 3 dcs. One host just recently started
hanging doing any `nodetool` requests. I’ve enabled remote JMX on all these
hosts and it was working fine until this week. No other changes have been made.
All the Cassandra-env.sh files are the same among the nodes, and only seeds and
listen_address are different for Cassandra.yaml files.
Nothing in debug log about why the Connection timed out. This is what I get
running nodetool from the same host:
[root@host5 conf]# nodetool status
nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException:
'Connection timed out (Connection timed out)'.
Any thoughts or how to log nodetool’s actions? Or see why it’s not accepting
the request? Telnet to the port from localhost as well as another node work
fine.
Thanks,
James
