I hit some more errors while running repair. Repair starts streaming,
and then hangs. On 2 of the 3 nodes involved in the repair, "nodetool
netstats" show no streams. On the 3rd node there is an error in the
logs and "nodetool netstats" show several streams that appear hung --
no progress in about 1 hour. The error message from the log is below,
also viewable here: http://pastebin.com/JTcXgkNR. Any ideas how to
fix this?
ERROR [Thread-233] 2013-04-01 22:22:21,759 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-233,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:270)
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
at
org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 1 more
ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
at
org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at
org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:128)
at
org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
at java.util.TreeMap.compare(TreeMap.java:1173)
at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1243)
at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1677)
at java.util.TreeMap.tailMap(TreeMap.java:890)
at java.util.TreeSet.tailSet(TreeSet.java:333)
at java.util.TreeSet.tailSet(TreeSet.java:366)
at
org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:206)
at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:180)
at
org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:276)
at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Details:
cassandra 1.2.2 - single token per node
Ubuntu 10.10 x86_64
EC2 m1.large
Cassandra max heap: 4G
Dane
