Re: rt jump during new node bootstrap

Hello, I am not sure what 'RT' stands for (I feel like I miss something obvious :)) and there is no scales in this dashboard, it's hard for me to say anything here. Could you give us more details? --- Alain Rodriguez - @arodream - al...@thelastpickle.com France / Spain The

rt jump during new node bootstrap

Dear All, We are using C* 2.1.18,when we bootstrap a new node,the rt will jump when the new node start up,then it back to normal.Could anyone please advise? Thanks, Peng Xiao

Node bootstrap error

Hello, Am adding a new node to a cluster, bootstrap failed with the following errors: WARN [STREAM-IN-/111.11.11.193] 2017-11-30 04:44:05,825 CompressedStreamReader.java:119 - [Stream c9d90a60-d56f-11e7-a95c-39e84a08567f] Error while reading partition null from stream on ks='system_auth' and

Re: Disk full during new node bootstrap

Hi, Well, I guess knowing the disk behaviour would be useful to understand if it is really filling up and why. - What is the disk capacity? - Does it actually fill up? - If it is filling up, it might mean that all your nodes are not running with enough available space and that any node

Re: Disk full during new node bootstrap

Cluster Information: Name: Cluster Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: 397560b8-7245-3903-8828-60a97e5be4aa: [xxx.xxx.xxx.75, xxx.xxx.xxx.134, xxx.xxx.xxx.192, xxx.xxx.xxx.132, xxx.xxx.xxx.133,

Re: Disk full during new node bootstrap

Hi, could you share with us the following informations ? - "nodetool status" output - Keyspace definitions (we need to check the replication strategy you're using on all keyspaces) - Specifics about what you're calling "groups" in a DC. Are these racks ? Thanks On Sat, Feb 4, 2017 at 10:41 AM

Re: Disk full during new node bootstrap

Yes .. same number of tokens... 256 On Sat, Feb 4, 2017 at 11:56 AM, Jonathan Haddad wrote: > Are you using the same number of tokens on the new node as the old ones? > > On Fri, Feb 3, 2017 at 8:31 PM techpyaasa . wrote: > >> Hi, >> >> We are using c*

Re: Disk full during new node bootstrap

Are you using the same number of tokens on the new node as the old ones? On Fri, Feb 3, 2017 at 8:31 PM techpyaasa . wrote: > Hi, > > We are using c* 2.0.17 , 2 DCs , RF=3. > > When I try to add new node to one group in a DC , I got disk full. Can > someone please tell what

Disk full during new node bootstrap

Hi, We are using c* 2.0.17 , 2 DCs , RF=3. When I try to add new node to one group in a DC , I got disk full. Can someone please tell what is the best way to resolve this? Run compaction for nodes in that group(to which I'm going to add new node, as data streams to new nodes from nodes of group

Re: Node bootstrap

Still having issues with node bootstrapping. The new node just died, because it Full Gced, the nodes it had actual streams with noticed its down. After the full gc finished the new node printed this log : ERROR 02:52:36,259 Stream failed because /10.10.20.35 died or was restarted/removed (streams

Re: Node bootstrap

Thanks Patricia for your response! On the new node, I just see a lot of the following: INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line 400) Writing Memtable INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132 CompactionTask.java (line 262) Compacted 12 sstables to so basically

Re: Node bootstrap

Yes num_tokens is set to 256. initial_token is blank on all nodes including the new one. On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy mark.re...@boxever.com wrote: My understanding was that if initial_token is left empty on the new node, it just contacts the heaviest node and bisects its token

Re: Node bootstrap

Also not sure if this is relevant but just noticed the nodetool tpstats output: Pool NameActive Pending Completed Blocked All time blocked FlushWriter 0 0 1136 0 512 Looks like about 50% of flushes are

Re: Node bootstrap

Yes num_tokens is set to 256. initial_token is blank on all nodes including the new one. Ok so you have num_tokens set to 256 for all nodes with initial_token commented out, this means you are using vnodes and the new node will automatically grab a list of tokens to take over responsibility

Re: Node bootstrap

nodetool status: Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.10.20.27 1.89 TB256 25.4% 76023cdd-c42d-4068-8b53-ae94584b8b04 rack1 UN

Re: Node bootstrap

Also Mark to your comment on my tpstats output, below is my iostat output, and the iowait is at 4.59%, which means no IO pressure, but we are still seeing the bad flush performance. Should we try increasing the flush writers? Linux 2.6.32-358.el6.x86_64 (ny4lpcas13.fusionts.corp) 08/05/2014

Re: Node bootstrap

Hi Ruchir, With the large number of blocked flushes and the number of pending compactions would still indicate IO contention. Can you post the output of 'iostat -x 5 5' If you do in fact have spare IO, there are several configuration options you can tune such as increasing the number of flush

Re: Node bootstrap

Right now, we have 6 flush writers and compaction_throughput_mb_per_sec is set to 0, which I believe disables throttling. Also, Here is the iostat -x 5 5 output: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 10.00

Re: Node bootstrap

Also, right now the top command shows that we are at 500-700% CPU, and we have 23 total processors, which means we have a lot of idle CPU left over, so throwing more threads at compaction and flush should alleviate the problem? On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha ruchir@gmail.com

Node bootstrap

I am trying to bootstrap the thirteenth node in a 12 node cluster where the average data size per node is about 2.1 TB. The bootstrap streaming has been going on for 2 days now, and the disk size on the new node is already above 4 TB and still going. Is this because the new node is running major

Re: Node bootstrap

Ruchir, What exactly are you seeing in the logs? Are you running major compactions on the new bootstrapping node? With respect to the seed list, it is generally advisable to use 3 seed nodes per AZ / DC. Cheers, On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha ruchir@gmail.com wrote: I am

Re: Cassandra 1.1.6 - New node bootstrap not completing

On 11/01/2013 03:03 PM, Robert Coli wrote: On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma narendra.sha...@gmail.comwrote: I was successfully able to bootstrap the node. The issue was RF 2. Thanks again Robert. For the record, I'm not entirely clear why bootstrapping two nodes into the

Re: Cassandra 1.1.6 - New node bootstrap not completing

I was successfully able to bootstrap the node. The issue was RF 2. Thanks again Robert. On Wed, Oct 30, 2013 at 10:29 AM, Narendra Sharma narendra.sha...@gmail.com wrote: Thanks Robert. I didn't realize that some of the keyspaces (not all and esp. the biggest one I was focusing on) had RF

Re: Cassandra 1.1.6 - New node bootstrap not completing

On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma narendra.sha...@gmail.comwrote: I was successfully able to bootstrap the node. The issue was RF 2. Thanks again Robert. For the record, I'm not entirely clear why bootstrapping two nodes into the same range should have caused your specific

Cassandra 1.1.6 - New node bootstrap not completing

We had a cluster of 4 nodes in AWS. The average load on each node was approx 750GB. We added 4 new nodes. It is now more than 30 hours and the node is still in JOINING mode. Specifically I am analyzing the one with IP 10.3.1.29. There is no compaction or streaming or index building happening. $

Re: Cassandra 1.1.6 - New node bootstrap not completing

On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma narendra.sha...@gmail.com wrote: We had a cluster of 4 nodes in AWS. The average load on each node was approx 750GB. We added 4 new nodes. It is now more than 30 hours and the node is still in JOINING mode. Specifically I am analyzing the one

Re: Cassandra 1.1.6 - New node bootstrap not completing

Thanks Robert. I didn't realize that some of the keyspaces (not all and esp. the biggest one I was focusing on) had RF 2. I wasted 3 days on it. Thanks again for the pointers. I will try again and share the results. On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli rc...@eventbrite.com wrote: On