If you're on 3.9 it's likely unrelated as streaming_socket_timeout_in_ms is
48 hours. Appears rebuild is trying to stream the same file twice. Are
there other exceptions in the logs related to the file, or can you find out
if it's previously been sent by the same session? Search the logs for the
file that failed and post back any exceptions.

On 29 December 2017 at 10:18, Martin Mačura <m.mac...@gmail.com> wrote:

> Is this something that can be resolved by CASSANDRA-11841 ?
>
> Thanks,
>
> Martin
>
> On Thu, Dec 21, 2017 at 3:02 PM, Martin Mačura <m.mac...@gmail.com> wrote:
> > Hi all,
> > we are trying to add a new datacenter to the existing cluster, but the
> > 'nodetool rebuild' command always fails after a couple of hours.
> >
> > We're on Cassandra 3.9.
> >
> > Example 1:
> >
> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:55735] 2017-12-13
> > 23:55:38,840 StreamResultFuture.java:174 - [Stream
> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
> > Receiving 0 files(0.000KiB), sending 9844 files(885.587GiB)
> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-13
> > 23:55:38,858 StreamResultFuture.java:174 - [Stream
> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
> > Receiving 9844 files(885.587GiB), sending 0 files(0.000KiB)
> >
> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:55735] 2017-12-14
> > 04:28:09,064 StreamSession.java:533 - [Stream
> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
> > session with peer 172.25.16.125
> > 172.24.16.169 java.io.IOException: Connection reset by peer
> >
> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:49412] 2017-12-14
> > 07:26:26,832 StreamSession.java:533 - [Stream
> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
> > session with peer 172.25.16.125
> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
> > <redacted>-13d700008e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
> > already completed or aborted (perhaps session failed?).
> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-14
> > 07:26:50,004 StreamSession.java:533 - [Stream
> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
> > session with peer 172.24.16.169
> > 172.25.16.125 java.io.IOException: Connection reset by peer
> >
> > Example 2:
> >
> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:35202] 2017-12-18
> > 03:24:31,423 StreamResultFuture.java:174 - [Stream
> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
> > Receiving 0 files(0.000KiB), sending 12312 files(895.973GiB)
> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-18
> > 03:24:31,441 StreamResultFuture.java:174 - [Stream
> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
> > Receiving 12312 files(895.973GiB), sending 0 files(0.000KiB)
> >
> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:35202] 2017-12-18
> > 06:39:42,049 StreamSession.java:533 - [Stream
> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
> > session with peer 172.25.16.125
> > 172.24.16.169 java.io.IOException: Connection reset by peer
> >
> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:42744] 2017-12-18
> > 09:25:36,188 StreamSession.java:533 - [Stream
> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
> > session with peer 172.25.16.125
> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
> > <redacted>-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
> > already completed or aborted (perhaps session failed?).
> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-18
> > 09:25:59,447 StreamSession.java:533 - [Stream
> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
> > session with peer 172.24.16.169
> > 172.25.16.125 java.io.IOException: Connection timed out
> >
> > Datacenter: PRIMARY
> > ===================
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > --  Address        Load       Tokens       Owns (effective)  Host ID
> >                             Rack
> > UN  172.24.16.169  918.31 GiB  256          100.0%
> > bc4a980b-cca6-4ca2-b32f-f8206d48e14c  RAC1
> > UN  172.24.16.170  908.76 GiB  256          100.0%
> > 37b2742e-c83a-4341-896f-09d244810e69  RAC1
> > UN  172.24.16.171  908.44 GiB  256          100.0%
> > 6dc2b9d8-75dd-48f8-858c-53b1af42e8fb  RAC1
> > Datacenter: SECONDARY
> > =====================
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > --  Address        Load       Tokens       Owns (effective)  Host ID
> >                             Rack
> > UN  172.25.16.125  27.48 GiB  256          100.0%
> > 1e1669eb-cfd2-4718-a073-558946a8c947  RAC2
> > UN  172.25.16.124  28.24 GiB  256          100.0%
> > 896d9894-10c8-4269-9476-5ddab3c8abe9  RAC2
> >
> > Any ideas?
> >
> > Thanks,
> >
> > Martin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Reply via email to