Did you look at the logs on the source DC as well? How big is the dataset? -- Jacob Shadix
On Fri, Apr 7, 2017 at 7:16 AM, Roland Otta <roland.o...@willhaben.at> wrote: > Hi! > > we are on 3.7. > > we have some debug messages ... but i guess they are not related to that > issue > DEBUG [GossipStage:1] 2017-04-07 13:11:00,440 FailureDetector.java:456 - > Ignoring interval time of 2002469610 for /192.168.0.27 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2598593732 for /10.192.116.4 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002612298 for /10.192.116.5 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002660534 for /10.192.116.9 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027212880 for /10.192.116.3 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027279042 for /192.168.0.188 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027313992 for /10.192.116.10 > > beside that the debug.log is clean > > all the mentioned cassandra.yml parameters are the shipped defaults ( > streaming_socket_timeout_in_ms does not exist at all in my cassandra.yml) > i also checked the pending compactions. there are no pending compactions > at the moment. > > bg - roland otta > > On Fri, 2017-04-07 at 06:47 -0400, Jacob Shadix wrote: > > What version are you running? Do you see any errors in the system.log > (SocketTimeout, for instance)? > > And what values do you have for the following in cassandra.yaml: > - - stream_throughput_outbound_megabits_per_sec > - - compaction_throughput_mb_per_sec > - - streaming_socket_timeout_in_ms > > -- Jacob Shadix > > On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta <roland.o...@willhaben.at> > wrote: > > hi, > > we are trying to setup a new datacenter and are initalizing the data > with nodetool rebuild. > > after some hours it seems that the node stopped streaming (at least > there is no more streaming traffic on the network interface). > > nodetool netstats shows that the streaming is still in progress > > Mode: NORMAL > Bootstrap 6918dc90-1ad6-11e7-9f16-51230e2be4e9 > Rebuild 41606030-1ad9-11e7-9f16-51230e2be4e9 > /192.168.0.26 > Receiving 257 files, 145444246572 bytes total. Already received > 1 files, 1744027 bytes total > bds/adcounter_total 76456/47310255 bytes(0%) received from > idx:0/192.168.0.26 > bds/upselling_event 1667571/1667571 bytes(100%) received > from idx:0/192.168.0.26 > /192.168.0.188 > /192.168.0.27 > Receiving 169 files, 79355302464 bytes total. Already received > 1 files, 81585975 bytes total > bds/ad_event_history 81585975/81585975 bytes(100%) received > from idx:0/192.168.0.27 > /192.168.0.189 > Receiving 140 files, 19673034809 bytes total. Already received > 1 files, 5996604 bytes total > bds/adcounter_per_day 5956840/42259846 bytes(14%) received > from idx:0/192.168.0.189 > bds/user_event 39764/39764 bytes(100%) received from > idx:0/192.168.0.189 > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed Dropped > Large messages n/a 2 3 0 > Small messages n/a 0 68632465 0 > Gossip messages n/a 0 217661 0 > > > > it is in that state for approx 15 hours now > > does it make sense waiting for the streaming to finish or do i have to > restart the node, discard data and restart the rebuild? > > >