Re: Bootstrapping a node fails because of compactions not keeping up

Nice catch! I’ve totally overlooked it. Thanks a lot! Stefano On Sun, 15 Oct 2017 at 22:14, Jeff Jirsa wrote: > (Should still be able to complete, unless you’re running out of disk or > memory or similar, but that’s why it’s streaming more than you expect) > > > -- > Jeff

Re: Bootstrapping a node fails because of compactions not keeping up

(Should still be able to complete, unless you’re running out of disk or memory or similar, but that’s why it’s streaming more than you expect) -- Jeff Jirsa > On Oct 15, 2017, at 1:51 PM, Jeff Jirsa wrote: > > I > You’re adding the new node as rac3 > > The rack aware

Re: Bootstrapping a node fails because of compactions not keeping up

I You’re adding the new node as rac3 The rack aware policy is going to make sure you get the rack diversity you asked for by making sure one replica of each partition is in rac3, which is going to blow up that instance -- Jeff Jirsa > On Oct 15, 2017, at 1:42 PM, Stefano Ortolani

Re: Bootstrapping a node fails because of compactions not keeping up

Hi Jeff, this my third attempt bootstrapping the node so I tried several tricks that might partially explain the output I am posting. * To make the bootstrap incremental, I have been throttling the streams on all nodes to 1Mbits. I have selectively unthrottling one node at a time hoping that

Re: Bootstrapping a node fails because of compactions not keeping up

Can you post (anonymize as needed) nodetool status, nodetool netstats, nodetool tpstats, and nodetool compctionstats ? -- Jeff Jirsa > On Oct 15, 2017, at 1:14 PM, Stefano Ortolani wrote: > > Hi Jeff, > > that would be 3.0.15, single disk, vnodes enabled (num_tokens

Re: Bootstrapping a node fails because of compactions not keeping up

Hi Jeff, that would be 3.0.15, single disk, vnodes enabled (num_tokens 256). Stefano On Sun, Oct 15, 2017 at 9:11 PM, Jeff Jirsa wrote: > What version? > > Single disk or JBOD? > > Vnodes? > > -- > Jeff Jirsa > > > On Oct 15, 2017, at 12:49 PM, Stefano Ortolani

Re: Bootstrapping a node fails because of compactions not keeping up

What version? Single disk or JBOD? Vnodes? -- Jeff Jirsa > On Oct 15, 2017, at 12:49 PM, Stefano Ortolani wrote: > > Hi all, > > I have been trying "-Dcassandra.disable_stcs_in_l0=true", but no luck so far. > Based on the source code it seems that this option doesn't

Re: Bootstrapping a node fails because of compactions not keeping up

Hi all, I have been trying "-Dcassandra.disable_stcs_in_l0=true", but no luck so far. Based on the source code it seems that this option doesn't affect compactions while bootstrapping. I am getting quite confused as it seems I am not able to bootstrap a node if I don't have at least 6/7 times

Re: Bootstrapping a node fails because of compactions not keeping up

Other little update: at the same time I see the number of pending tasks stuck (in this case at 1847); restarting the node doesn't help, so I can't really force the node to "digest" all those compactions. In the meanwhile the disk occupied is already twice the average load I have on other nodes.

Re: Bootstrapping a node fails because of compactions not keeping up

I have been trying to add another node to the cluster (after upgrading to 3.0.15) and I just noticed through "nodetool netstats" that all nodes have been streaming to the joining node approx 1/3 of their SSTables, basically their whole primary range (using RF=3)? Is this expected/normal? I was

Re: Bootstrapping a node fails because of compactions not keeping up

> > But if it also streams, it means I'd still be under-pressure if I am not > mistaken. I am under the assumption that the compactions are the by-product > of streaming too many SStables at the same time, and not because of my > current write load. > Ah yeah I wasn't thinking about the capacity

Re: Bootstrapping a node fails because of compactions not keeping up

Hi Kurt, On Wed, Aug 23, 2017 at 11:32 AM, kurt greaves wrote: > > ​1) You mean restarting the node in the middle of the bootstrap with >> join_ring=false? Would this option require me to issue a nodetool boostrap >> resume, correct? I didn't know you could instruct the

Re: Bootstrapping a node fails because of compactions not keeping up

> ​1) You mean restarting the node in the middle of the bootstrap with > join_ring=false? Would this option require me to issue a nodetool boostrap > resume, correct? I didn't know you could instruct the join via JMX. Would > it be the same of the nodetool boostrap command? write_survey is

Re: Bootstrapping a node fails because of compactions not keeping up

Hi Kurt, 1) You mean restarting the node in the middle of the bootstrap with join_ring=false? Would this option require me to issue a nodetool boostrap resume, correct? I didn't know you could instruct the join via JMX. Would it be the same of the nodetool boostrap command? 2) Yes, they are

Re: Bootstrapping a node fails because of compactions not keeping up

Well, that sucks. Be interested if you could find out if any of the streamed SSTables are retaining their levels. To answer your questions: 1) No. However, you could set your nodes to join in write_survey mode, which will stop them from joining the ring and you can initiate the join over JMX when

Re: Bootstrapping a node fails because of compactions not keeping up

Hi Kurt, sorry, I forgot to specify. I am on 3.0.14. Cheers, Stefano On Wed, Aug 23, 2017 at 12:11 AM, kurt greaves wrote: > What version are you running? 2.2 has an improvement that will retain > levels when streaming and this shouldn't really happen. If you're on 2.1 >

Re: Bootstrapping a node fails because of compactions not keeping up

What version are you running? 2.2 has an improvement that will retain levels when streaming and this shouldn't really happen. If you're on 2.1 best bet is to upgrade