I’m not sure it’s going to help you a lot as the issue is still open and there is not a lot of info in the ticket but your issue looks a lot like this JIRA to me: https://issues.apache.org/jira/browse/CASSANDRA-13363
Cheers Ben On Tue, 18 Jul 2017 at 17:46 Shashikant Kulkarni (शशिकांत कुलकर्णी) < shashikant.kulka...@gmail.com> wrote: > Hello All, > > I have Cassandra v3.9, with 3 nodes in cluster. Replication factor 2 > > 10.0.0.11,10.0.0.12,10.0.0.13 > > 10.0.0.11,10.0.0.12 are seed nodes > > What could be the possible reasons for following error in > /etc/cassandra/conf/debug.log > > The error is > > DEBUG [RMI TCP Connection(174)-127.0.0.1] 2017-07-12 04:47:49,002 > StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from > everybody: 10.0.0.13 > > At the time of the error here are some statistics from all the servers > > [user1@ip-10-0-0-11 ~]$ nodetool status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.0.0.12 2.55 MiB 256 67.2% > 83a68750-2238-4a6e-87be-03a3d7246824 rack1 > UN 10.0.0.11 1.78 GiB 256 70.6% > 052fda9d-0474-4dfb-b2f8-0c5cbec15266 rack1 > UN 10.0.0.13 1.78 GiB 256 62.2% > 86438dc9-77e0-43b2-a672-5b2e7cf216bf rack1 > > [user1@ip-10-0-0-11 ~]$ nodetool describecluster > Cluster Information: > Name: PiedmontCluster > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.11] > > UNREACHABLE: [10.0.0.13] > > > [user1@ip-10-0-0-13 ~]$ nodetool status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.0.0.12 2.55 MiB 256 67.2% > 83a68750-2238-4a6e-87be-03a3d7246824 rack1 > UN 10.0.0.11 1.78 GiB 256 70.6% > 052fda9d-0474-4dfb-b2f8-0c5cbec15266 rack1 > UN 10.0.0.13 1.78 GiB 256 62.2% > 86438dc9-77e0-43b2-a672-5b2e7cf216bf rack1 > > [user1@ip-10-0-0-13 ~]$ nodetool describecluster > Cluster Information: > Name: PiedmontCluster > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.13] > > UNREACHABLE: [10.0.0.11] > > > [user1@ip-10-0-0-12 ~]$ nodetool status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.0.0.12 2.55 MiB 256 67.2% > 83a68750-2238-4a6e-87be-03a3d7246824 rack1 > UN 10.0.0.11 1.78 GiB 256 70.6% > 052fda9d-0474-4dfb-b2f8-0c5cbec15266 rack1 > UN 10.0.0.13 1.78 GiB 256 62.2% > 86438dc9-77e0-43b2-a672-5b2e7cf216bf rack1 > > [user1@ip-10-0-0-12 ~]$ nodetool describecluster > Cluster Information: > Name: PiedmontCluster > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, > 10.0.0.11, 10.0.0.13] > The above mentioned error is in the /var/log/cassandra/debug.log on > 10.0.0.11 > > Error in /var/log/cassandra/debug.php on 10.0.0.13 is > > DEBUG [RMI TCP Connection(4)-127.0.0.1] 2017-07-13 02:31:23,846 > StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from > everybody: 10.0.0.11 > ERROR [MessagingService-Incoming-/10.0.0.11] 2017-07-13 02:35:04,982 > CassandraDaemon.java:226 - Exception in thread > Thread[MessagingService-Incoming-/10.0.0.11,5,main] > java.lang.ArrayIndexOutOfBoundsException: 4 > at > > org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1041) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:696) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:626) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.9.0.jar:3.9.0] > No error in /var/log/cassandra/debug.php on 10.0.0.12 > > Remember 10.0.0.11 & 10.0.0.12 are seed nodes > > Please give me some direction to fix this problem. > Thanks > > Regards, > Shashikant > -- *Ben Slater* *Chief Product Officer <https://www.instaclustr.com/>* <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> <https://www.linkedin.com/company/instaclustr> Read our latest technical blog posts here <https://www.instaclustr.com/blog/>. This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
