What version are you running? On Sat, Mar 7, 2015 at 2:14 PM, Jason Wee <peich...@gmail.com> wrote:
> > Hi Jan, thanks for your time to prepare the question and answer below, > > > - How many nodes do you have on the ring ? > 12 > > - What is the activity when this occurs - reads / writes/ compactions > ? > This cluster has a lot of writes and read. off peak period, ops center > shown cluster write is about 5k/sec, read about 1k/sec and during peak > period, write could be 22k/sec and read about 10k/sec. So this particular > one node hang like every moment irrespect if it is peak or non peak, or > compaction. > > - Is there anything that is unique about this node that makes it > different from the other nodes ? > Our nodes are same in term of operating system (centos 6) and > cassandra configuration settings. Other than that, there are no other > resources intensive application running in cassandra nodes. > > > - Is this a periodic occurance OR a single occurence - I am trying > to determine a pattern about when this shows up. > it *always* happened and in fact, it is happening now. > > - What is the load distribution the ring (ie: is this node carrying > more load than the others). > As of this moment, > > - Address DC Rack Status State Load > Owns Token > - > 155962751505430129087380028406227096910 > - <node1> us-east 1e Up Normal 498.66 GB > 8.33% 0 > - <node2> us-east 1e Up Normal 503.36 GB > 8.33% 14178431955039102644307275309657008810 > - <node3> us-east 1e Up Normal 492.08 GB > 8.33% 28356863910078205288614550619314017619 > - <node4> us-east 1e Up Normal 499.54 GB > 8.33% 42535295865117307932921825928971026430 > - <node5> us-east 1e Up Normal 523.76 GB > 8.33% 56713727820156407428984779325531226109 > - <node6> us-east 1e Up Normal 515.36 GB > 8.33% 70892159775195513221536376548285044050 > - <node7> us-east 1e Up Normal 588.93 GB > 8.33% 85070591730234615865843651857942052860 > - <node8> us-east 1e Up Normal 498.51 GB > 8.33% 99249023685273718510150927167599061670 > - <node9> us-east 1e Up Normal 531.81 GB > 8.33% 113427455640312814857969558651062452221 > - <node10> us-east 1e Up Normal 501.85 GB > 8.33% 127605887595351923798765477786913079290 > - <node11> us-east 1e Up Normal 501.13 GB > 8.33% 141784319550391026443072753096570088100 > - <node12> us-east 1e Up Normal 508.45 GB > 8.33% 155962751505430129087380028406227096910 > > so that one node is node5. At this instance ring output, yea, it is > second highest in the ring but unlikely this is the cause. > > > Jason > > On Sat, Mar 7, 2015 at 3:35 PM, Jan <cne...@yahoo.com> wrote: > >> HI Jason; >> >> The single node showing the anomaly is a hint that the problem is >> probably local to a node (as you suspected). >> >> - How many nodes do you have on the ring ? >> - What is the activity when this occurs - reads / writes/ >> compactions ? >> - Is there anything that is unique about this node that makes it >> different from the other nodes ? >> - Is this a periodic occurance OR a single occurence - I am trying >> to determine a pattern about when this shows up. >> - What is the load distribution the ring (ie: is this node carrying >> more load than the others). >> >> >> The system.log should have more info., about it. >> >> hope this helps >> Jan/ >> >> >> >> >> >> On Friday, March 6, 2015 4:50 AM, Jason Wee <peich...@gmail.com> wrote: >> >> >> well, StatusLogger.java started shown in cassandra >> system.log, MessagingService.java also shown some stage (e.g. read, >> mutation) dropped. >> >> It's strange it only happen in this node but this type of message does >> not shown in other node log file at the same time... >> >> Jason >> >> On Thu, Mar 5, 2015 at 4:26 AM, Jan <cne...@yahoo.com> wrote: >> >> HI Jason; >> >> Whats in the log files at the moment jstat shows 100%. >> What is the activity on the cluster & the node at the specific point in >> time (reads/ writes/ joins etc) >> >> Jan/ >> >> >> On Wednesday, March 4, 2015 5:59 AM, Jason Wee <peich...@gmail.com> >> wrote: >> >> >> Hi, our cassandra node using java 7 update 72 and we ran jstat on one of >> the node, and notice some strange behaviour as indicated by output below. >> any idea why when eden space stay the same for few seconds like 100% and >> 18.02% for few seconds? we suspect such "stalling" cause timeout to our >> cluster. >> >> any idea what happened, what went wrong and what could cause this? >> >> >> $ jstat -gcutil 32276 1s >> >> 0.00 5.78 91.21 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 >> 0.00 4.65 29.66 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 70.88 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 71.58 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 72.15 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 72.33 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 72.73 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 73.20 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 73.71 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 73.84 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 73.91 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.18 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 >> 0.00 5.43 12.64 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 69.24 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 78.05 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 78.97 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 79.07 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 79.18 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 80.09 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 80.36 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 80.51 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 80.70 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 81.31 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 81.63 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 5.43 82.94 71.09 60.07 2661 73.534 4 0.056 73.590 >> 0.00 9.37 65.64 71.21 60.07 2663 73.583 4 0.056 73.639 >> 0.00 3.49 11.89 71.25 60.10 2665 73.623 4 0.056 73.679 >> 0.00 3.49 66.11 71.25 60.10 2665 73.623 4 0.056 73.679 >> 14.26 0.00 9.91 71.25 60.10 2666 73.649 4 0.056 73.705 >> 14.26 0.00 73.25 71.25 60.10 2666 73.649 4 0.056 73.705 >> 0.00 10.49 14.43 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 66.07 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> 0.00 10.49 100.00 71.29 60.10 2667 73.675 4 0.056 73.731 >> >> >> thanks. >> >> jason >> >> >> >> >> >> >> >
