[
https://issues.apache.org/jira/browse/CASSANDRA-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Moos updated CASSANDRA-10005:
-----------------------------------
Description:
I'm adding a new node to the cluster and I'm seeing a bunch of the errors
below. The node never joins the cluster. This causes a deadlock, (see deadlock
below):
{code}
WARN [STREAM-IN-/10.220.0.160] 2015-08-06 16:16:42,640 StreamSession.java:638
- [Stream #4be6d7c0-3c53-11e5-b5bc-dbbae7f19873] Retrying for following error
java.lang.IllegalArgumentException: Not enough bytes
at
org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:162)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
{code}
{code}
ERROR 06:28:26 [Stream #059b7cc0-3c04-11e5-8c56-dbbae7f19873] Streaming error
occurred
java.lang.IllegalArgumentException: Unknown type 0
at
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:90)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
{code}
Found one Java-level deadlock:
=============================
"Thread-869":
waiting to lock monitor 0x00007f2ef8003f08 (object 0x000000062b8a9a28, a
java.lang.Object),
which is held by "STREAM-IN-/10.220.0.147"
"STREAM-IN-/10.220.0.147":
waiting to lock monitor 0x00007f2ed00436a8 (object 0x000000062bc96d68, a
java.lang.Object),
which is held by "Thread-869"
Java stack information for the threads listed above:
===================================================
"Thread-869":
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
- waiting to lock <0x000000062b8a9a28> (a java.lang.Object)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
- locked <0x000000062bc96d68> (a java.lang.Object)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
- locked <0x00000006340e3f30> (a sun.nio.ch.ChannelInputStream)
at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:161)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:745)
"STREAM-IN-/10.220.0.147":
at
java.nio.channels.spi.AbstractSelectableChannel.isBlocking(AbstractSelectableChannel.java:261)
- waiting to lock <0x000000062bc96d68> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:299)
- locked <0x000000062b8a9a28> (a java.lang.Object)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:52)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
at java.lang.Thread.run(Thread.java:745)
was:
I'm adding a new node to the cluster and I'm seeing a bunch of the errors
below. The node never joins the cluster. Does anyone know anything about this
error? I don't know if its related to why its stalling..
{code}
WARN [STREAM-IN-/10.220.0.160] 2015-08-06 16:16:42,640 StreamSession.java:638
- [Stream #4be6d7c0-3c53-11e5-b5bc-dbbae7f19873] Retrying for following error
java.lang.IllegalArgumentException: Not enough bytes
at
org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:162)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
{code}
{code}
ERROR 06:28:26 [Stream #059b7cc0-3c04-11e5-8c56-dbbae7f19873] Streaming error
occurred
java.lang.IllegalArgumentException: Unknown type 0
at
org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:90)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[apache-cassandra-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
{code}
> Streaming not enough bytes error
> --------------------------------
>
> Key: CASSANDRA-10005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10005
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Chris Moos
> Priority: Minor
>
> I'm adding a new node to the cluster and I'm seeing a bunch of the errors
> below. The node never joins the cluster. This causes a deadlock, (see
> deadlock below):
> {code}
> WARN [STREAM-IN-/10.220.0.160] 2015-08-06 16:16:42,640
> StreamSession.java:638 - [Stream #4be6d7c0-3c53-11e5-b5bc-dbbae7f19873]
> Retrying for following error
> java.lang.IllegalArgumentException: Not enough bytes
> at
> org.apache.cassandra.db.composites.AbstractCType.checkRemaining(AbstractCType.java:362)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.composites.AbstractCompoundCellNameType.fromByteBuffer(AbstractCompoundCellNameType.java:98)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:365)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.appendFromStream(BigTableWriter.java:243)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:162)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:95)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {code}
> {code}
> ERROR 06:28:26 [Stream #059b7cc0-3c04-11e5-8c56-dbbae7f19873] Streaming error
> occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:90)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {code}
> Found one Java-level deadlock:
> =============================
> "Thread-869":
> waiting to lock monitor 0x00007f2ef8003f08 (object 0x000000062b8a9a28, a
> java.lang.Object),
> which is held by "STREAM-IN-/10.220.0.147"
> "STREAM-IN-/10.220.0.147":
> waiting to lock monitor 0x00007f2ed00436a8 (object 0x000000062bc96d68, a
> java.lang.Object),
> which is held by "Thread-869"
> Java stack information for the threads listed above:
> ===================================================
> "Thread-869":
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> - waiting to lock <0x000000062b8a9a28> (a java.lang.Object)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
> - locked <0x000000062bc96d68> (a java.lang.Object)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> - locked <0x00000006340e3f30> (a sun.nio.ch.ChannelInputStream)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:161)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Thread.java:745)
> "STREAM-IN-/10.220.0.147":
> at
> java.nio.channels.spi.AbstractSelectableChannel.isBlocking(AbstractSelectableChannel.java:261)
> - waiting to lock <0x000000062bc96d68> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:299)
> - locked <0x000000062b8a9a28> (a java.lang.Object)
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:52)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
