[
https://issues.apache.org/jira/browse/CASSANDRA-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623692#comment-13623692
]
Igor Ivanov edited comment on CASSANDRA-5418 at 4/5/13 3:05 PM:
----------------------------------------------------------------
Since that I've tried to run offline sstablescrub, it didn't help. As was
suggested on IRC, I've tried sstable2json on data file and it worked. But when
joining node tries to stream file - it fails repeatedly on the same file.
was (Author: radev):
Since that I've tried to run offline sstablescrub, it didn't help. As was
suggested on IRC, I've tried sstable2json on data file and it worked. But when
joining node tries to stream file - it fails repeatedly on that file.
> repair freezing
> ---------------
>
> Key: CASSANDRA-5418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5418
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.2
> Environment: 5 nodes, RackInferring snitch, OpenVZ VMs, Centos 6,
> Oracle JVM with JNA enabled.
> Reporter: Igor Ivanov
>
> When I run *nodetool repair* on cas01 node it get's stuck at some point.
> I see following exceptions in cas01 system.log:
> {quote}
> ERROR [Streaming to /10.10.45.60:28] 2013-04-02 09:03:55,353
> CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to
> /10.10.45.60:28,5,main]
> java.lang.RuntimeException: java.io.EOFException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(Unknown Source)
> at
> org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> at
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> at
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 3 more
> ERROR [Thread-2076] 2013-04-02 09:07:12,261 CassandraDaemon.java (line 132)
> Exception in thread Thread[Thread-2076,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to
> /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-3660-Data.db;
> correct is 131074
> at
> org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> at
> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> {quote}
> On other machines there are some exceptions too:
> {quote}
> ERROR [Thread-1424] 2013-04-02 09:07:12,248 CassandraDaemon.java (line 132)
> Exception in thread Thread[Thread-1424,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to
> /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-2268-Data.db;
> correct is 131074
> at
> org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> at
> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> ERROR [Streaming to /10.10.45.58:55] 2013-04-02 09:07:12,263
> CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to
> /10.10.45.58:55,5,main]
> java.lang.RuntimeException: java.io.EOFException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(Unknown Source)
> at
> org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> at
> org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> at
> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 3 more
> {quote}
> Then I see frozen status in *nodetool netstats* and repair never completes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
