Re: C* 2.1.2 invokes oom-killer
After couple of days it's still behaving fine. Case closed. On Thu, Feb 19, 2015 at 11:15 PM, Michał Łowicki mlowi...@gmail.com wrote: Upgrade to 2.1.3 seems to help so far. After ~12 hours total memory consumption grew from 10GB to 10.5GB. On Thu, Feb 19, 2015 at 2:02 PM, Carlos Rolo r...@pythian.com wrote: Then you are probably hitting a bug... Trying to find out in Jira. The bad news is the fix is only to be released on 2.1.4. Once I find it out I will post it here. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 12:16 PM, Michał Łowicki mlowi...@gmail.com wrote: |trickle_fsync| has been enabled for long time in our settings (just noticed): trickle_fsync: true trickle_fsync_interval_in_kb: 10240 On Thu, Feb 19, 2015 at 12:12 PM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo r...@pythian.com wrote: Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory. Another question, is always the same nodes that die? Or is 2 out of 4 that die? Always the same nodes. Upgraded to 2.1.3 two hours ago so we'll monitor if maybe issue has been fixed there. If not will try to enable |tricke_fsync| Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 10:49 AM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo r...@pythian.com wrote: So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). pending tasks: 0 How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. Write request latency is below 0.05 ms/op (avg). Checked with OpsCenter. -- -- BR, Michał Łowicki -- -- BR, Michał Łowicki -- BR, Michał Łowicki -- -- BR, Michał Łowicki -- BR, Michał Łowicki
Re: C* 2.1.2 invokes oom-killer
In all tables SSTable counts is below 30. On Thu, Feb 19, 2015 at 9:43 AM, Carlos Rolo r...@pythian.com wrote: Can you check how many SSTables you have? It is more or less a know fact that 2.1.2 has lots of problems with compaction so a upgrade can solve it. But a high number of SSTables can confirm that indeed compaction is your problem not something else. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 9:16 AM, Michał Łowicki mlowi...@gmail.com wrote: We don't have other things running on these boxes and C* is consuming all the memory. Will try to upgrade to 2.1.3 and if won't help downgrade to 2.1.2. — Michał On Thu, Feb 19, 2015 at 2:39 AM, Jacob Rhoden jacob.rho...@me.com wrote: Are you tweaking the nice priority on Cassandra? (Type: man nice) if you don't know much about it. Certainly improving cassandra's nice score becomes important when you have other things running on the server like scheduled jobs of people logging in to the server and doing things. __ Sent from iPhone On 19 Feb 2015, at 5:28 am, Michał Łowicki mlowi...@gmail.com wrote: Hi, Couple of times a day 2 out of 4 members cluster nodes are killed root@db4:~# dmesg | grep -i oom [4811135.792657] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Nodes are using 8GB heap (confirmed with *nodetool info*) and aren't using row cache. Noticed that couple of times a day used RSS is growing really fast within couple of minutes and I see CPU spikes at the same time - https://www.dropbox.com/s/khco2kdp4qdzjit/Screenshot%202015-02-18%2015.10.54.png?dl=0 . Could be related to compaction but after compaction is finished used RSS doesn't shrink. Output from pmap when C* process uses 50GB RAM (out of 64GB) is available on http://paste.ofcode.org/ZjLUA2dYVuKvJHAk9T3Hjb. At the time dump was made heap usage is far below 8GB (~3GB) but total RSS is ~50GB. Any help will be appreciated. -- BR, Michał Łowicki -- -- BR, Michał Łowicki
Re: C* 2.1.2 invokes oom-killer
On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo r...@pythian.com wrote: So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). pending tasks: 0 How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. Write request latency is below 0.05 ms/op (avg). Checked with OpsCenter. -- -- BR, Michał Łowicki
Re: C* 2.1.2 invokes oom-killer
So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. -- --
Re: C* 2.1.2 invokes oom-killer
Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory. Another question, is always the same nodes that die? Or is 2 out of 4 that die? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 10:49 AM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo r...@pythian.com wrote: So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). pending tasks: 0 How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. Write request latency is below 0.05 ms/op (avg). Checked with OpsCenter. -- -- BR, Michał Łowicki -- --
Re: C* 2.1.2 invokes oom-killer
Then you are probably hitting a bug... Trying to find out in Jira. The bad news is the fix is only to be released on 2.1.4. Once I find it out I will post it here. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 12:16 PM, Michał Łowicki mlowi...@gmail.com wrote: |trickle_fsync| has been enabled for long time in our settings (just noticed): trickle_fsync: true trickle_fsync_interval_in_kb: 10240 On Thu, Feb 19, 2015 at 12:12 PM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo r...@pythian.com wrote: Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory. Another question, is always the same nodes that die? Or is 2 out of 4 that die? Always the same nodes. Upgraded to 2.1.3 two hours ago so we'll monitor if maybe issue has been fixed there. If not will try to enable |tricke_fsync| Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 10:49 AM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo r...@pythian.com wrote: So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). pending tasks: 0 How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. Write request latency is below 0.05 ms/op (avg). Checked with OpsCenter. -- -- BR, Michał Łowicki -- -- BR, Michał Łowicki -- BR, Michał Łowicki -- --
C* 2.1.2 invokes oom-killer
Hi, Couple of times a day 2 out of 4 members cluster nodes are killed root@db4:~# dmesg | grep -i oom [4811135.792657] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Nodes are using 8GB heap (confirmed with *nodetool info*) and aren't using row cache. Noticed that couple of times a day used RSS is growing really fast within couple of minutes and I see CPU spikes at the same time - https://www.dropbox.com/s/khco2kdp4qdzjit/Screenshot%202015-02-18%2015.10.54.png?dl=0 . Could be related to compaction but after compaction is finished used RSS doesn't shrink. Output from pmap when C* process uses 50GB RAM (out of 64GB) is available on http://paste.ofcode.org/ZjLUA2dYVuKvJHAk9T3Hjb. At the time dump was made heap usage is far below 8GB (~3GB) but total RSS is ~50GB. Any help will be appreciated. -- BR, Michał Łowicki
Re: C* 2.1.2 invokes oom-killer
On Wed, Feb 18, 2015 at 10:28 AM, Michał Łowicki mlowi...@gmail.com wrote: Couple of times a day 2 out of 4 members cluster nodes are killed This sort of issue is usually best handled/debugged interactively on IRC. But briefly : - 2.1.2 is IMO broken for production. Downgrade (officially unsupported but fine between these versions) to 2.1.1 or upgrade to 2.1.3. - Beyond that, look at the steady state heap consumption. With 2.1.2, it would likely take at least 1TB of data to fill heap in steady state to near-failure. =Rob
