[
https://issues.apache.org/jira/browse/CASSANDRA-10448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044962#comment-15044962
]
Piotr Westfalewicz commented on CASSANDRA-10448:
------------------------------------------------
[~mroi], I've figured out workaround how to expand the cluster:
1. Setup new node, without bootstraping it.
2. Take the snapshot of the data on one of the old nodes.
3. scp between old node and the new node. From snapshot to the
cassandra/data/kayspace/tablename folder.
4. Load the data without restarting the server - nodetool refresh.
5. Now triggering nodetool repair works. Run it.
6. Run nodetool cleanup.
> "Unknown type 0" Stream failure on Repair
> -----------------------------------------
>
> Key: CASSANDRA-10448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10448
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.2.2
> 5 Nodes in Google Compute Engine
> Java 1.8.0_60
> Reporter: Omri Iluz
> Assignee: Yuki Morishita
> Fix For: 2.2.x
>
> Attachments: casslogs.txt, receiversystem.log, sendersystem.log
>
>
> While running repair after upgrading to 2.2.2 I am getting many stream fail
> errors:
> {noformat}
> [2015-10-05 23:52:30,353] Repair session 4c181051-6bbb-11e5-acdb-d9a8bbd39330
> for range (59694553044959221,86389982480621619] failed with error [repair
> #4c181051-6bbb-11e5-acdb-d9a8bbd39330 on px/acti
> vities, (59694553044959221,86389982480621619]] Sync failed between
> /10.240.81.104 and /10.240.134.221 (progress: 4%)
> {noformat}
> Logs from both sides of the stream:
> Sides 1 -
> {noformat}
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:111 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Creating new streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52722] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-INIT-/10.240.81.104:52723] 2015-10-05 23:52:30,063
> StreamResultFuture.java:118 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Received streaming plan for Repair
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 13 files(517391317 bytes), sending 10
> files(469491729 bytes)
> ERROR [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,234
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:57)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> INFO [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.81.104 is complete
> WARN [STREAM-IN-/10.240.81.104] 2015-10-05 23:52:30,302
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
> Side 2 -
> {noformat}
> INFO [AntiEntropyStage:1] 2015-10-05 23:52:30,060 StreamResultFuture.java:86
> - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,061
> StreamSession.java:232 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Starting streaming to /10.240.134.221
> INFO [StreamConnectionEstablisher:6] 2015-10-05 23:52:30,063
> StreamCoordinator.java:213 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550,
> ID#0] Beginning stream session with /10.240.134.221
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,098
> StreamResultFuture.java:168 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550
> ID#0] Prepare completed. Receiving 10 files(469491729 bytes), sending 13
> files(517391317 bytes)
> INFO [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:182 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Session with /10.240.134.221 is complete
> ERROR [STREAM-OUT-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamSession.java:524 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_60]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_60]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.2.jar:2.2.2]
> ... 11 common frames omitted
> WARN [STREAM-IN-/10.240.134.221] 2015-10-05 23:52:30,349
> StreamResultFuture.java:209 - [Stream #239d8e60-6bbc-11e5-93ac-31bdef2dc550]
> Stream failed
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
