Hi all,
we are trying to add a new datacenter to the existing cluster, but the
'nodetool rebuild' command always fails after a couple of hours.
We're on Cassandra 3.9.
Example 1:
172.24.16.169 INFO [STREAM-IN-/172.25.16.125:55735] 2017-12-13
23:55:38,840 StreamResultFuture.java:174 - [Stream
#b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
Receiving 0 files(0.000KiB), sending 9844 files(885.587GiB)
172.25.16.125 INFO [STREAM-IN-/172.24.16.169:7000] 2017-12-13
23:55:38,858 StreamResultFuture.java:174 - [Stream
#b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed.
Receiving 9844 files(885.587GiB), sending 0 files(0.000KiB)
172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:55735] 2017-12-14
04:28:09,064 StreamSession.java:533 - [Stream
#b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
session with peer 172.25.16.125
172.24.16.169 java.io.IOException: Connection reset by peer
172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:49412] 2017-12-14
07:26:26,832 StreamSession.java:533 - [Stream
#b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
session with peer 172.25.16.125
172.24.16.169 java.lang.RuntimeException: Transfer of file
<redacted>-13d700008e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db
already completed or aborted (perhaps session failed?).
172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-14
07:26:50,004 StreamSession.java:533 - [Stream
#b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on
session with peer 172.24.16.169
172.25.16.125 java.io.IOException: Connection reset by peer
Example 2:
172.24.16.169 INFO [STREAM-IN-/172.25.16.125:35202] 2017-12-18
03:24:31,423 StreamResultFuture.java:174 - [Stream
#95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
Receiving 0 files(0.000KiB), sending 12312 files(895.973GiB)
172.25.16.125 INFO [STREAM-IN-/172.24.16.169:7000] 2017-12-18
03:24:31,441 StreamResultFuture.java:174 - [Stream
#95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed.
Receiving 12312 files(895.973GiB), sending 0 files(0.000KiB)
172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:35202] 2017-12-18
06:39:42,049 StreamSession.java:533 - [Stream
#95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
session with peer 172.25.16.125
172.24.16.169 java.io.IOException: Connection reset by peer
172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:42744] 2017-12-18
09:25:36,188 StreamSession.java:533 - [Stream
#95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
session with peer 172.25.16.125
172.24.16.169 java.lang.RuntimeException: Transfer of file
<redacted>-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db
already completed or aborted (perhaps session failed?).
172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-18
09:25:59,447 StreamSession.java:533 - [Stream
#95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on
session with peer 172.24.16.169
172.25.16.125 java.io.IOException: Connection timed out
Datacenter: PRIMARY
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.24.16.169 918.31 GiB 256 100.0%
bc4a980b-cca6-4ca2-b32f-f8206d48e14c RAC1
UN 172.24.16.170 908.76 GiB 256 100.0%
37b2742e-c83a-4341-896f-09d244810e69 RAC1
UN 172.24.16.171 908.44 GiB 256 100.0%
6dc2b9d8-75dd-48f8-858c-53b1af42e8fb RAC1
Datacenter: SECONDARY
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.25.16.125 27.48 GiB 256 100.0%
1e1669eb-cfd2-4718-a073-558946a8c947 RAC2
UN 172.25.16.124 28.24 GiB 256 100.0%
896d9894-10c8-4269-9476-5ddab3c8abe9 RAC2
Any ideas?
Thanks,
Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
