Re: High load on few nodes in a DC.
Yes we are using token aware but not shuffling replicas. On Wed, Sep 21, 2016 at 10:04 AM, Romain Hardouin <romainh...@yahoo.fr> wrote: > Hi, > > Do you shuffle the replicas with TokenAwarePolicy? > TokenAwarePolicy(LoadBalancingPolicy childPolicy, boolean > shuffleReplicas) > > Best, > > Romain > Le Mardi 20 septembre 2016 15h47, Pranay akula <pranay.akula2...@gmail.com> > a écrit : > > > I was a able to find the hotspots causing the load,but the size of these > partitions are in KB and no tombstones and no.of sstables is only 2 what > else i need to debug to find the reason for high load for some nodes. > we are also using unlogged batches is that can be the reason ?? how to > find which node is serving as a coordinator for un logged batches?? we are > using token-aware policy. > > thanks > > > > On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pranay.akula2...@gmail.com > > wrote: > > I was able to see most used partitions but the nodes with less load are > serving more read and write requests for that particular partitions when > compared to nodes with high load, how can i find if these nodes are serving > as co-coordinators for those read and write requests ?? how can i find the > token range for these particular partitions and which node is the primary > for these partition ?? > > > Thanks > > On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2...@gmail.com > > wrote: > > Hai Jeff, > > Thank, we are using RF 3 and cassandra version 2.1.8. > > Thanks > Pranay. > > On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> > wrote: > > Is your replication_factor 2? Or is it 3? What version are you using? > > The most likely answer is some individual partition that’s either being > written/read more than others, or is somehow impacting the cluster (wide > rows are a natural candidate). > > You don’t mention your version, but most modern versions of Cassandra ship > with ‘nodetool toppartitions’, which will help you identify frequently > written/read partitions – perhaps you can use that to identify a hotspot > due to some external behavior (some partition being read thousands of > times, over and over could certainly drive up load). > > - Jeff > > *From: *Pranay akula <pranay.akula2...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Monday, September 19, 2016 at 7:53 AM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *High load on few nodes in a DC. > > when our cluster was under load i am seeing 1 or 2 nodes are on more > load consistently when compared to others in dc i am not seeing any GC > pauses or wide partitions is this can be those nodes are continuously > serving as coordinators ?? how can i find what is the reason for high load > on those two nodes ?? We are using Vnode. > > > Thanks > Pranay. > > > > > > >
Re: High load on few nodes in a DC.
Hi, Do you shuffle the replicas with TokenAwarePolicy?TokenAwarePolicy(LoadBalancingPolicy childPolicy, boolean shuffleReplicas) Best, RomainLe Mardi 20 septembre 2016 15h47, Pranay akula <pranay.akula2...@gmail.com> a écrit : I was a able to find the hotspots causing the load,but the size of these partitions are in KB and no tombstones and no.of sstables is only 2 what else i need to debug to find the reason for high load for some nodes. we are also using unlogged batches is that can be the reason ?? how to find which node is serving as a coordinator for un logged batches?? we are using token-aware policy. thanks On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pranay.akula2...@gmail.com> wrote: I was able to see most used partitions but the nodes with less load are serving more read and write requests for that particular partitions when compared to nodes with high load, how can i find if these nodes are serving as co-coordinators for those read and write requests ?? how can i find the token range for these particular partitions and which node is the primary for these partition ?? Thanks On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2...@gmail.com> wrote: Hai Jeff, Thank, we are using RF 3 and cassandra version 2.1.8. ThanksPranay. On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: Is your replication_factor 2? Or is it 3? What version are you using? The most likely answer is some individual partition that’s either being written/read more than others, or is somehow impacting the cluster (wide rows are a natural candidate). You don’t mention your version, but most modern versions of Cassandra ship with ‘nodetool toppartitions’, which will help you identify frequently written/read partitions – perhaps you can use that to identify a hotspot due to some external behavior (some partition being read thousands of times, over and over could certainly drive up load). - Jeff From: Pranay akula <pranay.akula2...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Monday, September 19, 2016 at 7:53 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: High load on few nodes in a DC. when our cluster was under load i am seeing 1 or 2 nodes are on more load consistently when compared to others in dc i am not seeing any GC pauses or wide partitions is this can be those nodes are continuously serving as coordinators ?? how can i find what is the reason for high load on those two nodes ?? We are using Vnode. ThanksPranay.
Re: High load on few nodes in a DC.
I was a able to find the hotspots causing the load,but the size of these partitions are in KB and no tombstones and no.of sstables is only 2 what else i need to debug to find the reason for high load for some nodes. we are also using unlogged batches is that can be the reason ?? how to find which node is serving as a coordinator for un logged batches?? we are using token-aware policy. thanks On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pranay.akula2...@gmail.com> wrote: > I was able to see most used partitions but the nodes with less load are > serving more read and write requests for that particular partitions when > compared to nodes with high load, how can i find if these nodes are serving > as co-coordinators for those read and write requests ?? how can i find the > token range for these particular partitions and which node is the primary > for these partition ?? > > > Thanks > > On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2...@gmail.com > > wrote: > >> Hai Jeff, >> >> Thank, we are using RF 3 and cassandra version 2.1.8. >> >> Thanks >> Pranay. >> >> On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> >> wrote: >> >>> Is your replication_factor 2? Or is it 3? What version are you using? >>> >>> >>> >>> The most likely answer is some individual partition that’s either being >>> written/read more than others, or is somehow impacting the cluster (wide >>> rows are a natural candidate). >>> >>> >>> >>> You don’t mention your version, but most modern versions of Cassandra >>> ship with ‘nodetool toppartitions’, which will help you identify frequently >>> written/read partitions – perhaps you can use that to identify a hotspot >>> due to some external behavior (some partition being read thousands of >>> times, over and over could certainly drive up load). >>> >>> >>> >>> - Jeff >>> >>> >>> >>> *From: *Pranay akula <pranay.akula2...@gmail.com> >>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >>> *Date: *Monday, September 19, 2016 at 7:53 AM >>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >>> *Subject: *High load on few nodes in a DC. >>> >>> >>> >>> when our cluster was under load i am seeing 1 or 2 nodes are on more >>> load consistently when compared to others in dc i am not seeing any GC >>> pauses or wide partitions is this can be those nodes are continuously >>> serving as coordinators ?? how can i find what is the reason for high load >>> on those two nodes ?? We are using Vnode. >>> >>> >>> >>> >>> >>> Thanks >>> >>> Pranay. >>> >> >> >
Re: High load on few nodes in a DC.
I was able to see most used partitions but the nodes with less load are serving more read and write requests for that particular partitions when compared to nodes with high load, how can i find if these nodes are serving as co-coordinators for those read and write requests ?? how can i find the token range for these particular partitions and which node is the primary for these partition ?? Thanks On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2...@gmail.com> wrote: > Hai Jeff, > > Thank, we are using RF 3 and cassandra version 2.1.8. > > Thanks > Pranay. > > On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> > wrote: > >> Is your replication_factor 2? Or is it 3? What version are you using? >> >> >> >> The most likely answer is some individual partition that’s either being >> written/read more than others, or is somehow impacting the cluster (wide >> rows are a natural candidate). >> >> >> >> You don’t mention your version, but most modern versions of Cassandra >> ship with ‘nodetool toppartitions’, which will help you identify frequently >> written/read partitions – perhaps you can use that to identify a hotspot >> due to some external behavior (some partition being read thousands of >> times, over and over could certainly drive up load). >> >> >> >> - Jeff >> >> >> >> *From: *Pranay akula <pranay.akula2...@gmail.com> >> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >> *Date: *Monday, September 19, 2016 at 7:53 AM >> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> >> *Subject: *High load on few nodes in a DC. >> >> >> >> when our cluster was under load i am seeing 1 or 2 nodes are on more >> load consistently when compared to others in dc i am not seeing any GC >> pauses or wide partitions is this can be those nodes are continuously >> serving as coordinators ?? how can i find what is the reason for high load >> on those two nodes ?? We are using Vnode. >> >> >> >> >> >> Thanks >> >> Pranay. >> > >
Re: High load on few nodes in a DC.
Hai Jeff, Thank, we are using RF 3 and cassandra version 2.1.8. Thanks Pranay. On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > Is your replication_factor 2? Or is it 3? What version are you using? > > > > The most likely answer is some individual partition that’s either being > written/read more than others, or is somehow impacting the cluster (wide > rows are a natural candidate). > > > > You don’t mention your version, but most modern versions of Cassandra ship > with ‘nodetool toppartitions’, which will help you identify frequently > written/read partitions – perhaps you can use that to identify a hotspot > due to some external behavior (some partition being read thousands of > times, over and over could certainly drive up load). > > > > - Jeff > > > > *From: *Pranay akula <pranay.akula2...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Monday, September 19, 2016 at 7:53 AM > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *High load on few nodes in a DC. > > > > when our cluster was under load i am seeing 1 or 2 nodes are on more > load consistently when compared to others in dc i am not seeing any GC > pauses or wide partitions is this can be those nodes are continuously > serving as coordinators ?? how can i find what is the reason for high load > on those two nodes ?? We are using Vnode. > > > > > > Thanks > > Pranay. >
Re: High load on few nodes in a DC.
Is your replication_factor 2? Or is it 3? What version are you using? The most likely answer is some individual partition that’s either being written/read more than others, or is somehow impacting the cluster (wide rows are a natural candidate). You don’t mention your version, but most modern versions of Cassandra ship with ‘nodetool toppartitions’, which will help you identify frequently written/read partitions – perhaps you can use that to identify a hotspot due to some external behavior (some partition being read thousands of times, over and over could certainly drive up load). - Jeff From: Pranay akula <pranay.akula2...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Monday, September 19, 2016 at 7:53 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: High load on few nodes in a DC. when our cluster was under load i am seeing 1 or 2 nodes are on more load consistently when compared to others in dc i am not seeing any GC pauses or wide partitions is this can be those nodes are continuously serving as coordinators ?? how can i find what is the reason for high load on those two nodes ?? We are using Vnode. Thanks Pranay. smime.p7s Description: S/MIME cryptographic signature
High load on few nodes in a DC.
when our cluster was under load i am seeing 1 or 2 nodes are on more load consistently when compared to others in dc i am not seeing any GC pauses or wide partitions is this can be those nodes are continuously serving as coordinators ?? how can i find what is the reason for high load on those two nodes ?? We are using Vnode. Thanks Pranay.
