I'm switching the instances to machines with 61G of RAM, in this case would you still recommend using 8G of heap space?

Here is a gist of my heap settings from jvm.options

https://gist.github.com/dlsteuer/40e80280029897e6bb5fd12f2a86cbbe

On May 11 2017, at 3:08 pm, Alain RODRIGUEZ <arodr...@gmail.com> wrote:
For some context, I'm trying to get regular repairs going but am having issues with it.

You're not the only one, repairs are a real concern for many people.

For what it is worth, my team is actively working on this project initiated at Spotify: https://github.com/thelastpickle/cassandra-reaper.

C*heers,
-----------------------
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting

2017-05-11 23:04 GMT+01:00 Alain RODRIGUEZ <arodr...@gmail.com>:
Hi Daniel,

Could you paste the exact GC options in use?

Also 30 GB is not much. I would not use more than 8 GB for the JVM and probably CMS in those conditions for what it is worth. The thing is if memtables, bloom filter, caches, indexes, etc are off heap, then you probably ran out of Native memory. In any case it is good to have some space for page cache.

As a reminder you can try new GC option in a canary node, see how it goes.

C*heers,
-----------------------
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting

2017-05-11 22:29 GMT+01:00 Daniel Steuernol <dan...@sendwithus.com>:
Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G in the jvm.options for cassandra, and there is 30G on the machine.



On May 11 2017, at 2:22 pm, Cogumelos Maravilha <cogumelosmaravi...@sapo.pt> wrote:

Have a look at dmesg. It have already happened to me regarding type i instances at AWS.


On 11-05-2017 22:17, Daniel Steuernol wrote:
I had 2 nodes go down today, here is the ERRORs from the system log on both nodes
For some context, I'm trying to get regular repairs going but am having issues with it.


On May 11 2017, at 2:10 pm, Cogumelos Maravilha <cogumelosmaravi...@sapo.pt> wrote:

Can you grep ERROR system.log


On 11-05-2017 21:52, Daniel Steuernol wrote:
There is nothing in the system log about it being drained or shutdown, I'm not sure how else it would be pre-empted. No one else on the team is on the servers and I haven't been shutting them down. There also is no java memory dump on the server either. It appears that the process just died.


On May 11 2017, at 1:36 pm, Varun Gupta <var...@uber.com> wrote:

What do you mean by "no obvious error in the logs", do you see node was drained or shutdown. Are you sure, no other process is calling nodetool drain or shutdown, OR pre-empting cassandra process?

On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol <dan...@sendwithus.com> wrote:

I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any suggestions on how to track down what is causing the node to stop? --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org

--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org

--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org

--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org


--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org

Reply via email to