Hi Roni, The repair results is following (we run it Friday): Cannot proceed on repair because a neighbor (/192.168.61.201) is dead: session failed
But to be honest the neighbor did not died. It seemed to trigger a series of full GC events on the initiating node. The results form logs are: [2015-02-20 16:47:54,884] Starting repair command #2, repairing 7 ranges for keyspace prem_maelstrom_2 (parallelism=PARALLEL, full=false) [2015-02-21 02:21:55,640] Lost notification. You should check server log for repair status of keyspace prem_maelstrom_2 [2015-02-21 02:22:55,642] Lost notification. You should check server log for repair status of keyspace prem_maelstrom_2 [2015-02-21 02:23:55,642] Lost notification. You should check server log for repair status of keyspace prem_maelstrom_2 [2015-02-21 02:24:55,644] Lost notification. You should check server log for repair status of keyspace prem_maelstrom_2 [2015-02-21 04:41:08,607] Repair session d5d01dd0-b917-11e4-bc97-e9a66e5b2124 for range (85070591730234615865843651857942052874,102084710076281535261119195933814292480] failed with error org.apache.cassandra.exceptions.RepairException: [repair #d5d01dd0-b917-11e4-bc97-e9a66e5b2124 on prem_maelstrom_2/customer_events, (85070591730234615865843651857942052874,102084710076281535261119195933814292480]] Sync failed between /192.168.71.196 and /192.168.61.199 [2015-02-21 04:41:08,608] Repair session eb8d8d10-b967-11e4-bc97-e9a66e5b2124 for range (68056473384187696470568107782069813248,85070591730234615865843651857942052874] failed with error java.io.IOException: Endpoint /192.168.61.199 died [2015-02-21 04:41:08,608] Repair session c48aef00-b971-11e4-bc97-e9a66e5b2124 for range (0,10] failed with error java.io.IOException: Cannot proceed on repair because a neighbor (/ 192.168.61.201) is dead: session failed [2015-02-21 04:41:08,609] Repair session c48d38f0-b971-11e4-bc97-e9a66e5b2124 for range (42535295865117307932921825928971026442,68056473384187696470568107782069813248] failed with error java.io.IOException: Cannot proceed on repair because a neighbor (/192.168.61.201) is dead: session failed [2015-02-21 04:41:08,609] Repair session c48d38f1-b971-11e4-bc97-e9a66e5b2124 for range (127605887595351923798765477786913079306,136112946768375392941136215564139626496] failed with error java.io.IOException: Cannot proceed on repair because a neighbor (/192.168.61.201) is dead: session failed [2015-02-21 04:41:08,619] Repair session c48d6000-b971-11e4-bc97-e9a66e5b2124 for range (136112946768375392941136215564139626496,0] failed with error java.io.IOException: Cannot proceed on repair because a neighbor (/ 192.168.61.201) is dead: session failed [2015-02-21 04:41:08,620] Repair session c48d6001-b971-11e4-bc97-e9a66e5b2124 for range (102084710076281535261119195933814292480,127605887595351923798765477786913079306] failed with error java.io.IOException: Cannot proceed on repair because a neighbor (/192.168.61.201) is dead: session failed [2015-02-21 04:41:08,620] Repair command #2 finished We tried to run repair one more time. After 24 hour have some streaming errors. Moreover, 2-3 hours later, we have to stop it because we start to have write timeouts on client and our system starts to dying. The iostats from "dying" time plus tpstats are available here: https://drive.google.com/file/d/0B4N_AbBPGGwLc25nU0lnY3Z5NDA/view On Wed, Feb 25, 2015 at 7:50 PM, Roni Balthazar <ronibaltha...@gmail.com> wrote: > Hi Piotr, > > Are your repairs finishing without errors? > > Regards, > > Roni Balthazar > > On 25 February 2015 at 15:43, Ja Sam <ptrstp...@gmail.com> wrote: > > Hi, Roni, > > They aren't exactly balanced but as I wrote before they are in range from > > 2500-6000. > > If you need exactly data I will check them tomorrow morning. But all > nodes > > in AGRAF have small increase of pending compactions during last week, > which > > is "wrong direction" > > > > I will check in the morning get compaction throuput, but my feeling about > > this parameter is that it doesn't change anything. > > > > Regards > > Piotr > > > > > > > > > > On Wed, Feb 25, 2015 at 7:34 PM, Roni Balthazar <ronibaltha...@gmail.com > > > > wrote: > >> > >> Hi Piotr, > >> > >> What about the nodes on AGRAF? Are the pending tasks balanced between > >> this DC nodes as well? > >> You can check the pending compactions on each node. > >> > >> Also try to run "nodetool getcompactionthroughput" on all nodes and > >> check if the compaction throughput is set to 999. > >> > >> Cheers, > >> > >> Roni Balthazar > >> > >> On 25 February 2015 at 14:47, Ja Sam <ptrstp...@gmail.com> wrote: > >> > Hi Roni, > >> > > >> > It is not balanced. As I wrote you last week I have problems only in > DC > >> > in > >> > which we writes (on screen it is named as AGRAF: > >> > https://drive.google.com/file/d/0B4N_AbBPGGwLR21CZk9OV1kxVDA/view). > The > >> > problem is on ALL nodes in this dc. > >> > In second DC (ZETO) only one node have more than 30 SSTables and > pending > >> > compactions are decreasing to zero. > >> > > >> > In AGRAF the minimum pending compaction is 2500 , maximum is 6000 (avg > >> > on > >> > screen from opscenter is less then 5000) > >> > > >> > > >> > Regards > >> > Piotrek. > >> > > >> > p.s. I don't know why my mail client display my name as Ja Sam instead > >> > of > >> > Piotr Stapp, but this doesn't change anything :) > >> > > >> > > >> > On Wed, Feb 25, 2015 at 5:45 PM, Roni Balthazar > >> > <ronibaltha...@gmail.com> > >> > wrote: > >> >> > >> >> Hi Ja, > >> >> > >> >> How are the pending compactions distributed between the nodes? > >> >> Run "nodetool compactionstats" on all of your nodes and check if the > >> >> pendings tasks are balanced or they are concentrated in only few > >> >> nodes. > >> >> You also can check the if the SSTable count is balanced running > >> >> "nodetool cfstats" on your nodes. > >> >> > >> >> Cheers, > >> >> > >> >> Roni Balthazar > >> >> > >> >> > >> >> > >> >> On 25 February 2015 at 13:29, Ja Sam <ptrstp...@gmail.com> wrote: > >> >> > I do NOT have SSD. I have normal HDD group by JBOD. > >> >> > My CF have SizeTieredCompactionStrategy > >> >> > I am using local quorum for reads and writes. To be precise I have > a > >> >> > lot > >> >> > of > >> >> > writes and almost 0 reads. > >> >> > I changed "cold_reads_to_omit" to 0.0 as someone suggest me. I used > >> >> > set > >> >> > compactionthrouput to 999. > >> >> > > >> >> > So if my disk are idle, my CPU is less then 40%, I have some free > RAM > >> >> > - > >> >> > why > >> >> > SSTables count is growing? How I can speed up compactions? > >> >> > > >> >> > On Wed, Feb 25, 2015 at 5:16 PM, Nate McCall < > n...@thelastpickle.com> > >> >> > wrote: > >> >> >> > >> >> >> > >> >> >>> > >> >> >>> If You could be so kind and validate above and give me an answer > is > >> >> >>> my > >> >> >>> disk are real problems or not? And give me a tip what should I do > >> >> >>> with > >> >> >>> above > >> >> >>> cluster? Maybe I have misconfiguration? > >> >> >>> > >> >> >>> > >> >> >> > >> >> >> You disks are effectively idle. What consistency level are you > using > >> >> >> for > >> >> >> reads and writes? > >> >> >> > >> >> >> Actually, 'await' is sort of weirdly high for idle SSDs. Check > your > >> >> >> interrupt mappings (cat /proc/interrupts) and make sure the > >> >> >> interrupts > >> >> >> are > >> >> >> not being stacked on a single CPU. > >> >> >> > >> >> >> > >> >> > > >> > > >> > > > > > >
