Hello Chris, I must admit I am a bit confused about what you need exactly, I'll try to do my best :).
> would like to place limits on it to avoid it becoming a “noisy neighbor” But we also don’t want it killed by the oom killer, so just placing limits > on the container won't help. This sound contradictory to me. When the available memory is fully used and the memory cannot be taken elsewhere the program cannot continue and OOM cannot be avoided. So it's probably one thing or the other I would say ¯\_(ツ)_/¯. Is there’s a way to limit Cassandra’s off-heap memory usage? If we consider this perspective: "cointainerMemory = NativeMemory + HeapMemory", Then controlling the JVM heap memory and the container memory, you also control the off-heap/native memory. So practically yes, you can set the off-heap memory size, not directly, but by reducing the JVM heap size. The option is in jvm.options (or cassandra-env.sh): MAX_HEAP_SIZE="X" (it comes with some tradeoff as well of course, if you're going this path I recommend this post from Jon about JVM/GarbageCollection which is a tricky piece of Cassandra operations http://thelastpickle.com/blog/2018/04/11/gc-tuning.html) You can also control each (most?) of the off-heap structures individually. It's a bit split here and there between the distinct configuration files and at the table level. For example, if you're running out of Native Memory, you can maybe: - Consider adding RAM or use a bigger instance type in the cloud. - Reduce bloom filters ? --> http://cassandra.apache.org/doc/latest/operating/bloom_filters.html?highlight=bloom_filter_fp_chance#bloom-filters - Disable Row caches ? If you have troubles with memory, I would start there probably (You did not give us your version of Cassandra though). - Reduce the max_index_interval? --> https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableMax_index_interval - ... *Long Story :)* It's the C* operator job to ensure that the hardware choice and usage is optimal or at least that the sum of the memory needed by off-heap structures stays below what's available and does not produce any OOM. Some structures are capped (like the key cache size) some other will grow with the data (like bloom filters). Thus it's good to have some breathing room for growth and to have a monitoring system in place (this is something I advocate for at any occasion :D). Finding the right balance is part of the job of many of us here around :). That being said, it's rare we are fighting this kind of OOM because, in a huge majority of the cluster, we rely strongly on page caching and we try to have as much possible 'free' native memory for that purpose. We run into problems way before running out of native memory in many cases. Generally, a Cassandra cluster with the recommended (64 GB of RAM maybe?) or at least decent (32 GB?) hardware and the default configuration should hopefully work nicely. The schema design might make things worse and of course, you can optimize and reduce the cost, sometimes in a substantial way. But globally Cassandra and the default configuration give a good starting point I think. One last thing is that the more details you share, the sharper and accurate our answers can be. I feel like I told you *everything* I know about memory because it wasn't clear to me what you needed :). Specifying the Cassandra version, and telling something about your specific case like the memory total size or the JVM configuration would probably induce (faster/better) responses from us :). I hope this still helps. C*heers, ----------------------- Alain Rodriguez - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com Le jeu. 3 janv. 2019 à 00:03, Chris Mildebrandt <ch...@octarinesec.com> a écrit : > Hi, > > Is there’s a way to limit Cassandra’s off-heap memory usage? I can’t find > a way to limit the memory used for row caches, bloom filters, etc. We’re > running Cassandra in a container and would like to place limits on it to > avoid it becoming a “noisy neighbor”. But we also don’t want it killed by > the oom killer, so just placing limits on the container won't help. > > Thanks, > -Chris >
