Hi Sean, Are you using any rack aware topology? --> we are using gossip file Are you using any rack aware topology? --> we are using gossip file What are your partition keys? --> Partition key is uniqIs it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)? --> No, we verified it. If it was partition key issue, we would see similar number of partition keys across nodes. If we look closely number of keys across nodes vary a lot.
Number of partitions (estimate): 3142552Number of partitions (estimate): 15625442Number of partitions (estimate): 15244021Number of partitions (estimate): 9592992Number of partitions (estimate): 15839280 On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R <sean_r_dur...@homedepot.com> wrote: #yiv2571733579 #yiv2571733579 -- _filtered #yiv2571733579 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv2571733579 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv2571733579 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv2571733579 #yiv2571733579 p.yiv2571733579MsoNormal, #yiv2571733579 li.yiv2571733579MsoNormal, #yiv2571733579 div.yiv2571733579MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;font-family:New serif;}#yiv2571733579 a:link, #yiv2571733579 span.yiv2571733579MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv2571733579 a:visited, #yiv2571733579 span.yiv2571733579MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv2571733579 p {margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579 p.yiv2571733579msonormal0, #yiv2571733579 li.yiv2571733579msonormal0, #yiv2571733579 div.yiv2571733579msonormal0 {margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579 p.yiv2571733579ydpe3237830p1, #yiv2571733579 li.yiv2571733579ydpe3237830p1, #yiv2571733579 div.yiv2571733579ydpe3237830p1 {margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579 span.yiv2571733579ydpe3237830s1 {}#yiv2571733579 span.yiv2571733579ydpe3237830apple-converted-space {}#yiv2571733579 span.yiv2571733579EmailStyle22 {font-family:sans-serif;color:#1F497D;}#yiv2571733579 .yiv2571733579MsoChpDefault {font-size:10.0pt;} _filtered #yiv2571733579 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv2571733579 div.yiv2571733579WordSection1 {}#yiv2571733579 Are you using any rack aware topology? What are your partition keys? Is it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)? Sean Durity lord of the (C*) rings (Staff Systems Engineer – Cassandra) MTC 2250 #cassandra - for the latest news and updates From: learner dba <cassandra...@yahoo.com.INVALID> Sent: Monday, June 18, 2018 2:06 PM To: User cassandra.apache.org <user@cassandra.apache.org> Subject: [EXTERNAL] Cluster is unbalanced Hi, Data volume varies a lot in our two DC cluster: Load Tokens Owns 20.01 GiB 256 ? 65.32 GiB 256 ? 60.09 GiB 256 ? 46.95 GiB 256 ? 50.73 GiB 256 ? kaiprodv2 ========= /Leaving/Joining/Moving Load Tokens Owns 25.19 GiB 256 ? 30.26 GiB 256 ? 9.82 GiB 256 ? 20.54 GiB 256 ? 9.7 GiB 256 ? I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on heavier nodes instead of decreasing. Based on nodetool cfstats, I can see partition keys on each node varies a lot: Number of partitions (estimate): 3142552 Number of partitions (estimate): 15625442 Number of partitions (estimate): 15244021 Number of partitions (estimate): 9592992 Number of partitions (estimate): 15839280 How can I diagnose this imbalance further?