> > Yes num_tokens is set to 256. initial_token is blank on all nodes > including the new one.
Ok so you have num_tokens set to 256 for all nodes with initial_token commented out, this means you are using vnodes and the new node will automatically grab a list of tokens to take over responsibility for. Pool Name Active Pending Completed Blocked All > time blocked > FlushWriter 0 0 1136 0 > 512 > > Looks like about 50% of flushes are blocked. > This is a problem as it indicates that the IO system cannot keep up. Just ran this on the new node: > nodetool netstats | grep "Streaming from" | wc -l > 10 This is normal as the new node will most likely take tokens from all nodes in the cluster. Sorry for the multiple updates, but another thing I found was all the other > existing nodes have themselves in the seeds list, but the new node does not > have itself in the seeds list. Can that cause this issue? Seeds are only used when a new node is bootstrapping into the cluster and needs a set of ips to contact and discover the cluster, so this would have no impact on data sizes or streaming. In general it would be considered best practice to have a set of 2-3 seeds from each data center, with all nodes having the same seed list. What is the current output of 'nodetool compactionstats'? Could you also paste the output of nodetool status <keyspace>? Mark On Tue, Aug 5, 2014 at 3:59 PM, Ruchir Jha <ruchir....@gmail.com> wrote: > Sorry for the multiple updates, but another thing I found was all the > other existing nodes have themselves in the seeds list, but the new node > does not have itself in the seeds list. Can that cause this issue? > > > On Tue, Aug 5, 2014 at 10:30 AM, Ruchir Jha <ruchir....@gmail.com> wrote: > >> Just ran this on the new node: >> >> nodetool netstats | grep "Streaming from" | wc -l >> 10 >> >> Seems like the new node is receiving data from 10 other nodes. Is that >> expected in a vnodes enabled environment? >> >> Ruchir. >> >> >> >> On Tue, Aug 5, 2014 at 10:21 AM, Ruchir Jha <ruchir....@gmail.com> wrote: >> >>> Also not sure if this is relevant but just noticed the nodetool tpstats >>> output: >>> >>> Pool Name Active Pending Completed Blocked >>> All time blocked >>> FlushWriter 0 0 1136 0 >>> 512 >>> >>> Looks like about 50% of flushes are blocked. >>> >>> >>> On Tue, Aug 5, 2014 at 10:14 AM, Ruchir Jha <ruchir....@gmail.com> >>> wrote: >>> >>>> Yes num_tokens is set to 256. initial_token is blank on all nodes >>>> including the new one. >>>> >>>> >>>> On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy <mark.re...@boxever.com> >>>> wrote: >>>> >>>>> My understanding was that if initial_token is left empty on the new >>>>>> node, it just contacts the heaviest node and bisects its token range. >>>>> >>>>> >>>>> If you are using vnodes and you have num_tokens set to 256 the new >>>>> node will take token ranges dynamically. What is the configuration of your >>>>> other nodes, are you setting num_tokens or initial_token on those? >>>>> >>>>> >>>>> Mark >>>>> >>>>> >>>>> On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha <ruchir....@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks Patricia for your response! >>>>>> >>>>>> On the new node, I just see a lot of the following: >>>>>> >>>>>> INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line >>>>>> 400) Writing Memtable >>>>>> INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132 >>>>>> CompactionTask.java (line 262) Compacted 12 sstables to >>>>>> >>>>>> so basically it is just busy flushing, and compacting. Would you have >>>>>> any ideas on why the 2x disk space blow up. My understanding was that if >>>>>> initial_token is left empty on the new node, it just contacts the >>>>>> heaviest >>>>>> node and bisects its token range. And the heaviest node is around 2.1 TB, >>>>>> and the new node is already at 4 TB. Could this be because compaction is >>>>>> falling behind? >>>>>> >>>>>> Ruchir >>>>>> >>>>>> >>>>>> On Mon, Aug 4, 2014 at 7:23 PM, Patricia Gorla < >>>>>> patri...@thelastpickle.com> wrote: >>>>>> >>>>>>> Ruchir, >>>>>>> >>>>>>> What exactly are you seeing in the logs? Are you running major >>>>>>> compactions on the new bootstrapping node? >>>>>>> >>>>>>> With respect to the seed list, it is generally advisable to use 3 >>>>>>> seed nodes per AZ / DC. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha <ruchir....@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I am trying to bootstrap the thirteenth node in a 12 node cluster >>>>>>>> where the average data size per node is about 2.1 TB. The bootstrap >>>>>>>> streaming has been going on for 2 days now, and the disk size on the >>>>>>>> new >>>>>>>> node is already above 4 TB and still going. Is this because the new >>>>>>>> node is >>>>>>>> running major compactions while the streaming is going on? >>>>>>>> >>>>>>>> One thing that I noticed that seemed off was the seeds property in >>>>>>>> the yaml of the 13th node comprises of 1..12. Where as the seeds >>>>>>>> property >>>>>>>> on the existing 12 nodes consists of all the other nodes except the >>>>>>>> thirteenth node. Is this an issue? >>>>>>>> >>>>>>>> Any other insight is appreciated? >>>>>>>> >>>>>>>> Ruchir. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Patricia Gorla >>>>>>> @patriciagorla >>>>>>> >>>>>>> Consultant >>>>>>> Apache Cassandra Consulting >>>>>>> http://www.thelastpickle.com <http://thelastpickle.com> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
