[
https://issues.apache.org/jira/browse/CASSANDRA-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954278#comment-14954278
]
Yuki Morishita commented on CASSANDRA-10448:
--------------------------------------------
Looking at debug.log again, it seems /10.240.127.67 thought it received file
right before throwing "Unknown type 0":
{code}
DEBUG [STREAM-IN-/10.240.187.13] 2015-10-06 20:55:24,238
ConnectionHandler.java:266 - [Stream #8d0faa30-6c6c-11e5-b01c-e95897ba4d20]
Received File (Header (cfId: 63f58440-4d5d-11e5-871c-e95897ba4d20, #4, version:
la, format: BIG, estimated keys: 640, transfer size: 643153318, compressed?:
true, repairedAt: 1444153390653, level: 0), file:
/cassandra/data/px/activities-63f584404d5d11e5871ce95897ba4d20/tmp-la-53903-big-Data.db)
{code}
but streaming node /10.240.187.13 seems still it was transferring the file:
{code}
ERROR [STREAM-OUT-/10.240.127.67] 2015-10-06 20:55:24,563
StreamSession.java:524 - [Stream #8d0faa30-6c6c-11e5-b01c-e95897ba4d20]
Streaming error occurred
org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
at
org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
~[apache-cassandra-2.2.2.jar:2.2.2]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
~[apache-cassandra-2.2.2.jar:2.2.2]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
stack trace is from {{OutgoingFileMessage}} which is used to transfer file.
So it may be either 1) sender streamed extra bytes or 2) receiver thought file
was received too early.
If this is happening only for 2.2+, then this can be related to the change in
CASSANDRA-8670?
> "Unknown type 0" Stream failure on Repair
> -----------------------------------------
>
> Key: CASSANDRA-10448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10448
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.2
> 5 Nodes in Google Compute Engine
> Java 1.8.0_60
> Reporter: Omri Iluz
> Assignee: Yuki Morishita
> Fix For: 2.2.x
>
> Attachments: casslogs.txt
>
>
> While running repair after upgrading to 2.2.2 I am getting many stream fail
> errors:
> {noformat}
> [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330
> for range (59694553044959221,86389982480621619] failed with error [repair
> #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti
> vities, (59694553044959221,86389982480621619]] Sync failed between
> /10.240.81.104 and /10.240.134.221 (progress: 4%)
> {noformat}
> Logs from both sides of the stream:
> Sides 1 -
> {noformat}
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Creating new streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10
> files(469491729 bytes)
> ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.81.104 is complete
> WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
> Side 2 -
> {noformat}
> INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86
> - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061
> StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Starting streaming to /10.240.134.221
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063
> StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Beginning stream session with /10.240.134.221
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13
> files(517391317 bytes)
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.134.221 is complete
> ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> ... 11 common frames omitted
> WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
