I don't know if this is going to help or not but we had a problem disk (was going bad) and would develop bad blocks, requiring raid repair. As nearly as we could guess, that would cause "some sort of timing issue" for nodetool - we were never able to duplicate the exact occurrence on demand. The clue came when the same exact process worked on a different setup that did not use the same disk subsystem. Once we isolated and replaced the degrading disk, the problem went away.
Beyond what Jeff suggested, you may want to look at any logs your raid produces for potential clues. Also, if it is always failing when sending the same file, you might want to review what is going on there. I know this is fairly "general" response but I hope it gives you some ideas. Good luck! Bob -----Original Message----- From: Jeff Jirsa [mailto:jji...@apache.org] Sent: Thursday, August 10, 2017 7:26 PM To: dev@cassandra.apache.org Subject: Re: rebuild constantly fails, 3.11 On 2017-08-08 01:00 (-0700), Micha <mich...@fantasymail.de> wrote: > Hi, > > it seems I'm not able to add add 3 node dc to a 3 node dc. After > starting the rebuild on a new node, nodetool netstats show it will > receive 1200 files from node-1 and 5000 from node-2. The stream from > node-1 completes but the stream from node-2 allways fails, after > sending ca 4000 files. > > After restarting the rebuild it again starts to send the 5000 files. > The whole cluster is connected via one switch only , no firewall > between, the networks shows no errors. > The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. > the logs show no errors. The size of the data is ca 1TB. Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? Is node3 running? --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
