None of the files is listed more than once in the logs:

java.lang.RuntimeException: Transfer of file
/fs3/cassandra/data/<redacted>/event_group-3b5782d08e4411e6842917253f111990/mc-116042-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs0/cassandra/data/<redacted>/event_group-3b5782d08e4411e6842917253f111990/mc-111370-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs3/cassandra/data/<redacted>/event_alert-13d700008e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs4/cassandra/data/<redacted>/event_alert-13d700008e3f11e6a6cbe1698349da4d/mc-9133-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs2/cassandra/data/<redacted>/event_alert-13d700008e3f11e6a6cbe1698349da4d/mc-3997-big-Data.db
already completed or aborted (perhaps session failed?).
java.lang.RuntimeException: Transfer of file
/fs1/cassandra/data/<redacted>//event_group-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
already completed or aborted (perhaps session failed?).




On Mon, Jan 8, 2018 at 2:21 AM, kurt greaves <k...@instaclustr.com> wrote:
> If you're on 3.9 it's likely unrelated as streaming_socket_timeout_in_ms is
> 48 hours. Appears rebuild is trying to stream the same file twice. Are there
> other exceptions in the logs related to the file, or can you find out if
> it's previously been sent by the same session? Search the logs for the file
> that failed and post back any exceptions.
>
> On 29 December 2017 at 10:18, Martin Mačura <m.mac...@gmail.com> wrote:
>>
>> Is this something that can be resolved by CASSANDRA-11841 ?
>>
>> Thanks,
>>
>> Martin
>>
>> On Thu, Dec 21, 2017 at 3:02 PM, Martin Mačura <m.mac...@gmail.com> wrote:
>> > Hi all,
>> > we are trying to add a new datacenter to the existing cluster, but the
>> > 'nodetool rebuild' command always fails after a couple of hours.
>> >
>> > We're on Cassandra 3.9.
>> >
>> > Example 1:
>> >
>> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:55735] 2017-12-13
>> > 23:55:38,840 StreamResultFuture.java:174 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
>> > Receiving 0 files(0.000KiB), sending 9844 files(885.587GiB)
>> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-13
>> > 23:55:38,858 StreamResultFuture.java:174 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
>> > Receiving 9844 files(885.587GiB), sending 0 files(0.000KiB)
>> >
>> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:55735] 2017-12-14
>> > 04:28:09,064 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.io.IOException: Connection reset by peer
>> >
>> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:49412] 2017-12-14
>> > 07:26:26,832 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
>> > <redacted>-13d700008e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
>> > already completed or aborted (perhaps session failed?).
>> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-14
>> > 07:26:50,004 StreamSession.java:533 - [Stream
>> > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
>> > session with peer 172.24.16.169
>> > 172.25.16.125 java.io.IOException: Connection reset by peer
>> >
>> > Example 2:
>> >
>> > 172.24.16.169 INFO  [STREAM-IN-/172.25.16.125:35202] 2017-12-18
>> > 03:24:31,423 StreamResultFuture.java:174 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
>> > Receiving 0 files(0.000KiB), sending 12312 files(895.973GiB)
>> > 172.25.16.125 INFO  [STREAM-IN-/172.24.16.169:7000] 2017-12-18
>> > 03:24:31,441 StreamResultFuture.java:174 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
>> > Receiving 12312 files(895.973GiB), sending 0 files(0.000KiB)
>> >
>> > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:35202] 2017-12-18
>> > 06:39:42,049 StreamSession.java:533 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.io.IOException: Connection reset by peer
>> >
>> > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:42744] 2017-12-18
>> > 09:25:36,188 StreamSession.java:533 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
>> > session with peer 172.25.16.125
>> > 172.24.16.169 java.lang.RuntimeException: Transfer of file
>> > <redacted>-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
>> > already completed or aborted (perhaps session failed?).
>> > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-18
>> > 09:25:59,447 StreamSession.java:533 - [Stream
>> > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
>> > session with peer 172.24.16.169
>> > 172.25.16.125 java.io.IOException: Connection timed out
>> >
>> > Datacenter: PRIMARY
>> > ===================
>> > Status=Up/Down
>> > |/ State=Normal/Leaving/Joining/Moving
>> > --  Address        Load       Tokens       Owns (effective)  Host ID
>> >                             Rack
>> > UN  172.24.16.169  918.31 GiB  256          100.0%
>> > bc4a980b-cca6-4ca2-b32f-f8206d48e14c  RAC1
>> > UN  172.24.16.170  908.76 GiB  256          100.0%
>> > 37b2742e-c83a-4341-896f-09d244810e69  RAC1
>> > UN  172.24.16.171  908.44 GiB  256          100.0%
>> > 6dc2b9d8-75dd-48f8-858c-53b1af42e8fb  RAC1
>> > Datacenter: SECONDARY
>> > =====================
>> > Status=Up/Down
>> > |/ State=Normal/Leaving/Joining/Moving
>> > --  Address        Load       Tokens       Owns (effective)  Host ID
>> >                             Rack
>> > UN  172.25.16.125  27.48 GiB  256          100.0%
>> > 1e1669eb-cfd2-4718-a073-558946a8c947  RAC2
>> > UN  172.25.16.124  28.24 GiB  256          100.0%
>> > 896d9894-10c8-4269-9476-5ddab3c8abe9  RAC2
>> >
>> > Any ideas?
>> >
>> > Thanks,
>> >
>> > Martin
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Reply via email to