Hi, All,
We have a 12 nodes cluster with 2.1.9 version for near one month. Last
week it have an exception . Cluster’s write and read latency will go up to
4 seconds from 0.4ms average after exception.
I suspect OutboundTcpConnection is broken .I try to disablegossip then
enablegossip to rescue OutboundTcpConnction ,but fail .
I have to restart every node to restore cluster to normal . Cluster’s
load is low. The client use datastax java driver 2.1.7.1 . CF with RF 3.
Question:
Which situation to trigger this AssertionError? I read
OutboundTcpConnection.java source code, line 228 comments
“writeConnected() is reasonably robust”.
Thanks
Attachment :
ERROR [MessagingService-Outgoing-/172.20.114.13] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.13,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
ERROR [MessagingService-Outgoing-/172.20.114.19] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.19,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
INFO [NativePoolCleaner] 2015-11-08 10:36:47,414
ColumnFamilyStore.java:1231 - Flushing largest CFS(Keyspace='qinglong',
ColumnFamily='package_state') to free up room. Used total: 0.04/0.14, live:
0.04/0.14, flushing: 0.00/0.00, this: 0.02/0.02
Best Regards
