[
https://issues.apache.org/jira/browse/CASSANDRA-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033182#comment-15033182
]
Omri Iluz commented on CASSANDRA-10448:
---------------------------------------
+1
any chance that this can be resolve in time for 2.2.4? this is blocking us from
expanding the cluster as well. happy to test patches that increase log level or
dump more information.
> "Unknown type 0" Stream failure on Repair
> -----------------------------------------
>
> Key: CASSANDRA-10448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10448
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.2.2
> 5 Nodes in Google Compute Engine
> Java 1.8.0_60
> Reporter: Omri Iluz
> Assignee: Yuki Morishita
> Fix For: 2.2.x
>
> Attachments: casslogs.txt, receiversystem.log, sendersystem.log
>
>
> While running repair after upgrading to 2.2.2 I am getting many stream fail
> errors:
> {noformat}
> [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330
> for range (59694553044959221,86389982480621619] failed with error [repair
> #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti
> vities, (59694553044959221,86389982480621619]] Sync failed between
> /10.240.81.104 and /10.240.134.221 (progress: 4%)
> {noformat}
> Logs from both sides of the stream:
> Sides 1 -
> {noformat}
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Creating new streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10
> files(469491729 bytes)
> ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.81.104 is complete
> WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
> Side 2 -
> {noformat}
> INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86
> - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061
> StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Starting streaming to /10.240.134.221
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063
> StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Beginning stream session with /10.240.134.221
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13
> files(517391317 bytes)
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.134.221 is complete
> ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> ... 11 common frames omitted
> WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
