Re: cassandra node jvm stall intermittently

2015-03-10 Thread Jason Wee
heh on the midst of upgrading , Rob ;-) Jason On Tue, Mar 10, 2015 at 2:04 AM, Robert Coli rc...@eventbrite.com wrote: On Sat, Mar 7, 2015 at 1:44 AM, Jason Wee peich...@gmail.com wrote: hey Ali, 1.0.8 On Sat, Mar 7, 2015 at 5:20 PM, Ali Akhtar ali.rac...@gmail.com wrote: What version are

Re: cassandra node jvm stall intermittently

2015-03-07 Thread Jason Wee
Hi Jan, thanks for your time to prepare the question and answer below, - How many nodes do you have on the ring ? 12 - What is the activity when this occurs - reads / writes/ compactions ? This cluster has a lot of writes and read. off peak period, ops center shown cluster

Re: cassandra node jvm stall intermittently

2015-03-07 Thread Jason Wee
hey Ali, 1.0.8 On Sat, Mar 7, 2015 at 5:20 PM, Ali Akhtar ali.rac...@gmail.com wrote: What version are you running? On Sat, Mar 7, 2015 at 2:14 PM, Jason Wee peich...@gmail.com wrote: Hi Jan, thanks for your time to prepare the question and answer below, - How many nodes do you have

Re: cassandra node jvm stall intermittently

2015-03-07 Thread Ali Akhtar
What version are you running? On Sat, Mar 7, 2015 at 2:14 PM, Jason Wee peich...@gmail.com wrote: Hi Jan, thanks for your time to prepare the question and answer below, - How many nodes do you have on the ring ? 12 - What is the activity when this occurs - reads / writes/

Re: cassandra node jvm stall intermittently

2015-03-06 Thread Jason Wee
well, StatusLogger.java started shown in cassandra system.log, MessagingService.java also shown some stage (e.g. read, mutation) dropped. It's strange it only happen in this node but this type of message does not shown in other node log file at the same time... Jason On Thu, Mar 5, 2015 at 4:26

Re: cassandra node jvm stall intermittently

2015-03-06 Thread Jan
HI Jason;  The single node showing the anomaly is a hint that the problem is probably local to a node (as you suspected).  - How many nodes do you have on the ring ?  - What is the activity when this occurs  - reads / writes/ compactions  ?    - Is there anything that is

Re: cassandra node jvm stall intermittently

2015-03-04 Thread Jan
HI Jason;  Whats in the log files at the moment jstat shows 100%. What is the activity on the cluster the node at the specific point in time (reads/ writes/ joins etc) Jan/ On Wednesday, March 4, 2015 5:59 AM, Jason Wee peich...@gmail.com wrote: Hi, our cassandra node using java 7

cassandra node jvm stall intermittently

2015-03-04 Thread Jason Wee
Hi, our cassandra node using java 7 update 72 and we ran jstat on one of the node, and notice some strange behaviour as indicated by output below. any idea why when eden space stay the same for few seconds like 100% and 18.02% for few seconds? we suspect such stalling cause timeout to our cluster.