Re: READ messages dropped
Hi! Thanks for the response. My cluster is in a bad state those recent days. I have 29 CFs, and my disk is 5% full... So I guess the VMs still have more space to go, and I am not sure this is considered many CFs. But maybe I have memory issues. I enlarge cassandra memory from about ~2G to ~4G (out of ~8G). This was done because at that stage I had lots of key caches. I then reduced them to almost 0 on all CF. I guess now I can reduce the memory back to ~2 or ~3 G. Will that help? Thanks *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Thu, Oct 11, 2012 at 10:46 PM, Tyler Hobbs ty...@datastax.com wrote: On Wed, Oct 10, 2012 at 3:10 PM, Tamar Fraenkel ta...@tok-media.comwrote: What I did noticed while looking at the logs (which are also running OpsCenter), is that there is some correlation between the dropped reads and flushes of OpsCenter column families to disk and or compactions. What are the rollups CFs? why is there so much traffic in them? The rollups CFs hold the performance metric data that OpsCenter stores about your cluster. Typically these aren't actually very high traffic column families, but that depends on how many column families you have (more CFs require more metrics to be stored). If you have a lot of column families, you have a couple of options for reducing the amount of metric data that's stored: http://www.datastax.com/docs/opscenter/trouble_shooting_opsc#limiting-the-metrics-collected-by-opscenter Assuming you don't have a large number of CFs, your nodes may legitimately be nearing capacity. -- Tyler Hobbs DataStax http://datastax.com/ tokLogo.png
Re: READ messages dropped
On Fri, Oct 12, 2012 at 2:24 AM, Tamar Fraenkel ta...@tok-media.com wrote: Thanks for the response. My cluster is in a bad state those recent days. I have 29 CFs, and my disk is 5% full... So I guess the VMs still have more space to go, and I am not sure this is considered many CFs. That's not too many CFs. I don't know how much 5% of your disk space is in absolute numbers, which is more important. The most important measure for whether you are approaching limits is really disk utilization (as in how busy the disk is, not how much data it's holding). OpsCenter exposes metrics for this that you should check. But maybe I have memory issues. I enlarge cassandra memory from about ~2G to ~4G (out of ~8G). This was done because at that stage I had lots of key caches. I then reduced them to almost 0 on all CF. I guess now I can reduce the memory back to ~2 or ~3 G. Will that help? I would leave your heap at 4G. You really do want key caching enabled in almost all circumstances; it can save you a lot of disk activity on reads. If you need to bump your heap up to 4.5G to accommodate key caches, it's worth it. -- Tyler Hobbs DataStax http://datastax.com/
Re: READ messages dropped
Hi! Thanks for the answer. I don't see much change in the load this Cassandra cluster is under, so why is the sudden surge of such messages? What I did noticed while looking at the logs (which are also running OpsCenter), is that there is some correlation between the dropped reads and flushes of OpsCenter column families to disk and or compactions. What are the rollups CFs? why is there so much traffic in them? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Wed, Oct 10, 2012 at 1:00 AM, aaron morton aa...@thelastpickle.comwrote: or how to solve it? Simple solution is move to m1.xlarge :) In the last 3 days I see many messages of READ messages dropped in last 5000ms on one of my 3 nodes cluster. The node is not able to keep up with the load. Possible causes include excessive GC, aggressive compaction, or simply too many requests. it also a good idea to take a look at iostat to see if the disk is keeping up. Hope that helps - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/10/2012, at 9:08 AM, Tamar Fraenkel ta...@tok-media.com wrote: Hi! In the last 3 days I see many messages of READ messages dropped in last 5000ms on one of my 3 nodes cluster. I see no errors in the log. There are also messages of Finished hinted handoff of 0 rows to endpoint but I had those for a while now, so I don't know if they are related. I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large instances. Rep factor 3 (Quorum read and write) Does anyone have a clue what I should be looking for, or how to solve it? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media tokLogo.png ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 tokLogo.png
Re: READ messages dropped
or how to solve it? Simple solution is move to m1.xlarge :) In the last 3 days I see many messages of READ messages dropped in last 5000ms on one of my 3 nodes cluster. The node is not able to keep up with the load. Possible causes include excessive GC, aggressive compaction, or simply too many requests. it also a good idea to take a look at iostat to see if the disk is keeping up. Hope that helps - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 9/10/2012, at 9:08 AM, Tamar Fraenkel ta...@tok-media.com wrote: Hi! In the last 3 days I see many messages of READ messages dropped in last 5000ms on one of my 3 nodes cluster. I see no errors in the log. There are also messages of Finished hinted handoff of 0 rows to endpoint but I had those for a while now, so I don't know if they are related. I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large instances. Rep factor 3 (Quorum read and write) Does anyone have a clue what I should be looking for, or how to solve it? Thanks, Tamar Fraenkel Senior Software Engineer, TOK Media tokLogo.png ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956
READ messages dropped
Hi! In the last 3 days I see many messages of READ messages dropped in last 5000ms on one of my 3 nodes cluster. I see no errors in the log. There are also messages of Finished hinted handoff of 0 rows to endpoint but I had those for a while now, so I don't know if they are related. I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large instances. Rep factor 3 (Quorum read and write) Does anyone have a clue what I should be looking for, or how to solve it? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 tokLogo.png
