Re: Disabling Swap for Cassandra
I think there is some potential yak shaving to worrying excessively about swap. The reality is that you should know the memory demands of what you are running on your C* nodes and have things configured so that significant swap would be a highly abnormal situation. I'd expect to see excessive churn on buffer cache long before I'd see excessive swap kicking in, but sometimes a little swap usage doesn't mean much beyond the O/S detecting that some memory allocation is so stale that it may as well push it out of the way. This can happen for perfectly reasonable situations if, for example, you make heavy use of crond for automating system maintenance. Also, if you are running on Dell boxes, Dell software updates can get a bit cranky and you see resource locking that has zilch to do with your application stack. I'd worry less about how to crank down swap beyond the advice to make it a last resort, and more about how to monitor and alert on abnormal system behavior. When it's abnormal, you want a chance to see what is going on so you can fix it. OOM'ing problems out of visibility makes it hard to investigate root causes. I'd rather be paged while the cause is visible, than be paged anyways for the down node and have nothing to inspect. R On 4/17/20, 6:12 AM, "Alex Ott" wrote: Message from External Sender I usually recommend following document: https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Ddev_datastax-5Fenterprise_config_configRecommendedSettings.html=DwIFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=TQPtBiV2cZow-OW1xEFTxqIlaA6VWPwM9PbMdScHIGw=_YR4k-l76UU-LxTd7WCtHAV6_LdRP2qzNiBAD1dAzdU= - it's about DSE, but applicable to OSS Cassandra as well... Kunal at "Thu, 16 Apr 2020 15:49:35 -0700" wrote: K> Hello, K> K> I need some suggestion from you all. I am new to Cassandra and was reading Cassandra best practices. On one document, it was K> mentioned that Cassandra should not be using swap, it degrades the performance. K> My question is instead of disabling swap system wide, can we force Cassandra not to use swap? Some documentation suggests to use K> memory_locking_policy in cassandra.yaml. K> How do I check if our Cassandra already has this parameter and still uses swap ? Is there any way i can check this. I already K> checked cassandra.yaml and dont see this parameter. Is there any other place i can check and confirm? K> Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap (Xms = Xmx) can be locked at node startup ? Will that K> help? K> Or if you have any other suggestions, please let me know. K> K> K> Regards, K> Kunal K> -- With best wishes,Alex Ott Principal Architect, DataStax https://urldefense.proofpoint.com/v2/url?u=http-3A__datastax.com_=DwIFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=TQPtBiV2cZow-OW1xEFTxqIlaA6VWPwM9PbMdScHIGw=ddXQN2wa2-ikDaE4LFM7Z-g-V369ObwXmt6_BeWRXPU= - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Disabling Swap for Cassandra
I usually recommend following document: https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configRecommendedSettings.html - it's about DSE, but applicable to OSS Cassandra as well... Kunal at "Thu, 16 Apr 2020 15:49:35 -0700" wrote: K> Hello, K> K> I need some suggestion from you all. I am new to Cassandra and was reading Cassandra best practices. On one document, it was K> mentioned that Cassandra should not be using swap, it degrades the performance. K> My question is instead of disabling swap system wide, can we force Cassandra not to use swap? Some documentation suggests to use K> memory_locking_policy in cassandra.yaml. K> How do I check if our Cassandra already has this parameter and still uses swap ? Is there any way i can check this. I already K> checked cassandra.yaml and dont see this parameter. Is there any other place i can check and confirm? K> Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap (Xms = Xmx) can be locked at node startup ? Will that K> help? K> Or if you have any other suggestions, please let me know. K> K> K> Regards, K> Kunal K> -- With best wishes,Alex Ott Principal Architect, DataStax http://datastax.com/ - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Disabling Swap for Cassandra
> On Apr 16, 2020, at 5:50 PM, Dor Laor wrote: > > You should configure swap for safety, better be slow than crash, > For most production use cases, it’s almost always better to crash than be slow. - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Disabling Swap for Cassandra
On Thu, Apr 16, 2020 at 5:09 PM Kunal wrote: > > Thanks for the responses. Appreciae it. > > @Dor, so you are saying if we add "memlock unlimited" in limits.conf, the > entire heap (Xms=Xmx) can be locked at startup ? Will this be applied to all > Java processes ? We have couple of Java programs running with the same owner. Each process is responsible for calling mlock on its own (in the code itself). I only see mlock in C* under JNA, my knowledge is mostly in scylla, so not sure about this. The limits.conf just makes sure the limits are high enough You should configure swap for safety, better be slow than crash, the memory locking is another safety measure and isn't a must. You can also run your daemons in separate cgroup and cap their memory usage as explained in one of the answers here: https://stackoverflow.com/questions/12520499/linux-how-to-lock-the-pages-of-a-process-in-memory > > > Thanks > Kunal > > On Thu, Apr 16, 2020 at 4:31 PM Dor Laor wrote: >> >> It is good to configure swap for the OS but exempt Cassandra >> from swapping. Why is it good? Since you never know the >> memory utilization of additional agents and processes you or >> other admins will run on your server. >> >> So do configure a swap partition. >> You can control the eagerness of the kernel by the swappiness >> sysctl parameter. You can even control it per cgroup: >> https://askubuntu.com/questions/967588/how-can-i-prevent-certain-process-from-being-swapped >> >> You should make sure Cassandra locks its memory so the kernel >> won't choose its memory to be swapped out (since it will kill >> your latency). You do it by mlock. Read more on: >> https://stackoverflow.com/questions/578137/can-i-tell-linux-not-to-swap-out-a-particular-processes-memory >> >> The scylla /dist/common/limits.d/scylladb.com looks like this: >> scylla - core unlimited >> scylla - memlock unlimited >> scylla - nofile 20 >> scylla - as unlimited >> scylla - nproc8096 >> >> On Thu, Apr 16, 2020 at 3:57 PM Nitan Kainth wrote: >> > >> > Swap is controlled by OS and will use it when running short of memory. I >> > don’t think you can disable at Cassandra level >> > >> > >> > Regards, >> > >> > Nitan >> > >> > Cell: 510 449 9629 >> > >> > >> > On Apr 16, 2020, at 5:50 PM, Kunal wrote: >> > >> > >> > >> > Hello, >> > >> > >> > >> > I need some suggestion from you all. I am new to Cassandra and was reading >> > Cassandra best practices. On one document, it was mentioned that Cassandra >> > should not be using swap, it degrades the performance. >> > >> > My question is instead of disabling swap system wide, can we force >> > Cassandra not to use swap? Some documentation suggests to use >> > memory_locking_policy in cassandra.yaml. >> > >> > >> > How do I check if our Cassandra already has this parameter and still uses >> > swap ? Is there any way i can check this. I already checked cassandra.yaml >> > and dont see this parameter. Is there any other place i can check and >> > confirm? >> > >> > >> > Also, Can I set memlock parameter to unlimited (64kB default), so entire >> > Heap (Xms = Xmx) can be locked at node startup ? Will that help? >> > >> > >> > Or if you have any other suggestions, please let me know. >> > >> > >> > >> > >> > >> > Regards, >> > >> > Kunal >> > >> > >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> > > > -- > > > > Regards, > Kunal Vaid - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Disabling Swap for Cassandra
Thanks for the responses. Appreciae it. @Dor, so you are saying if we add "memlock unlimited" in limits.conf, the entire heap (Xms=Xmx) can be locked at startup ? Will this be applied to all Java processes ? We have couple of Java programs running with the same owner. Thanks Kunal On Thu, Apr 16, 2020 at 4:31 PM Dor Laor wrote: > It is good to configure swap for the OS but exempt Cassandra > from swapping. Why is it good? Since you never know the > memory utilization of additional agents and processes you or > other admins will run on your server. > > So do configure a swap partition. > You can control the eagerness of the kernel by the swappiness > sysctl parameter. You can even control it per cgroup: > > https://askubuntu.com/questions/967588/how-can-i-prevent-certain-process-from-being-swapped > > You should make sure Cassandra locks its memory so the kernel > won't choose its memory to be swapped out (since it will kill > your latency). You do it by mlock. Read more on: > > https://stackoverflow.com/questions/578137/can-i-tell-linux-not-to-swap-out-a-particular-processes-memory > > The scylla /dist/common/limits.d/scylladb.com looks like this: > scylla - core unlimited > scylla - memlock unlimited > scylla - nofile 20 > scylla - as unlimited > scylla - nproc8096 > > On Thu, Apr 16, 2020 at 3:57 PM Nitan Kainth > wrote: > > > > Swap is controlled by OS and will use it when running short of memory. I > don’t think you can disable at Cassandra level > > > > > > Regards, > > > > Nitan > > > > Cell: 510 449 9629 > > > > > > On Apr 16, 2020, at 5:50 PM, Kunal wrote: > > > > > > > > Hello, > > > > > > > > I need some suggestion from you all. I am new to Cassandra and was > reading Cassandra best practices. On one document, it was mentioned that > Cassandra should not be using swap, it degrades the performance. > > > > My question is instead of disabling swap system wide, can we force > Cassandra not to use swap? Some documentation suggests to use > memory_locking_policy in cassandra.yaml. > > > > > > How do I check if our Cassandra already has this parameter and still > uses swap ? Is there any way i can check this. I already checked > cassandra.yaml and dont see this parameter. Is there any other place i can > check and confirm? > > > > > > Also, Can I set memlock parameter to unlimited (64kB default), so entire > Heap (Xms = Xmx) can be locked at node startup ? Will that help? > > > > > > Or if you have any other suggestions, please let me know. > > > > > > > > > > > > Regards, > > > > Kunal > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- Regards, Kunal Vaid
Re: Disabling Swap for Cassandra
It is good to configure swap for the OS but exempt Cassandra from swapping. Why is it good? Since you never know the memory utilization of additional agents and processes you or other admins will run on your server. So do configure a swap partition. You can control the eagerness of the kernel by the swappiness sysctl parameter. You can even control it per cgroup: https://askubuntu.com/questions/967588/how-can-i-prevent-certain-process-from-being-swapped You should make sure Cassandra locks its memory so the kernel won't choose its memory to be swapped out (since it will kill your latency). You do it by mlock. Read more on: https://stackoverflow.com/questions/578137/can-i-tell-linux-not-to-swap-out-a-particular-processes-memory The scylla /dist/common/limits.d/scylladb.com looks like this: scylla - core unlimited scylla - memlock unlimited scylla - nofile 20 scylla - as unlimited scylla - nproc8096 On Thu, Apr 16, 2020 at 3:57 PM Nitan Kainth wrote: > > Swap is controlled by OS and will use it when running short of memory. I > don’t think you can disable at Cassandra level > > > Regards, > > Nitan > > Cell: 510 449 9629 > > > On Apr 16, 2020, at 5:50 PM, Kunal wrote: > > > > Hello, > > > > I need some suggestion from you all. I am new to Cassandra and was reading > Cassandra best practices. On one document, it was mentioned that Cassandra > should not be using swap, it degrades the performance. > > My question is instead of disabling swap system wide, can we force Cassandra > not to use swap? Some documentation suggests to use memory_locking_policy in > cassandra.yaml. > > > How do I check if our Cassandra already has this parameter and still uses > swap ? Is there any way i can check this. I already checked cassandra.yaml > and dont see this parameter. Is there any other place i can check and confirm? > > > Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap > (Xms = Xmx) can be locked at node startup ? Will that help? > > > Or if you have any other suggestions, please let me know. > > > > > > Regards, > > Kunal > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Disabling Swap for Cassandra
Cassandra attempts to lock the heap at startup, but all the memory allocated after startup is not locked. So you do want to make sure the allowed locked memory is large. Disabling or vastly dialing down swappiness is a best practice for all server software, not just Cassandra, so you should still at the very least set the swappiness to some small number of you don’t want to completely disable it. -Jeremiah > On Apr 16, 2020, at 5:57 PM, Nitan Kainth wrote: > > Swap is controlled by OS and will use it when running short of memory. I > don’t think you can disable at Cassandra level > > > Regards, > Nitan > Cell: 510 449 9629 > >>> On Apr 16, 2020, at 5:50 PM, Kunal wrote: >>> >> >> Hello, >> >> I need some suggestion from you all. I am new to Cassandra and was reading >> Cassandra best practices. On one document, it was mentioned that Cassandra >> should not be using swap, it degrades the performance. >> My question is instead of disabling swap system wide, can we force Cassandra >> not to use swap? Some documentation suggests to use memory_locking_policy in >> cassandra.yaml. >> >> How do I check if our Cassandra already has this parameter and still uses >> swap ? Is there any way i can check this. I already checked cassandra.yaml >> and dont see this parameter. Is there any other place i can check and >> confirm? >> >> Also, Can I set memlock parameter to unlimited (64kB default), so entire >> Heap (Xms = Xmx) can be locked at node startup ? Will that help? >> >> Or if you have any other suggestions, please let me know. >> >> >> Regards, >> Kunal >>
Re: Disabling Swap for Cassandra
Swap is controlled by OS and will use it when running short of memory. I don’t think you can disable at Cassandra level Regards, Nitan Cell: 510 449 9629 > On Apr 16, 2020, at 5:50 PM, Kunal wrote: > > > Hello, > > I need some suggestion from you all. I am new to Cassandra and was reading > Cassandra best practices. On one document, it was mentioned that Cassandra > should not be using swap, it degrades the performance. > My question is instead of disabling swap system wide, can we force Cassandra > not to use swap? Some documentation suggests to use memory_locking_policy in > cassandra.yaml. > > How do I check if our Cassandra already has this parameter and still uses > swap ? Is there any way i can check this. I already checked cassandra.yaml > and dont see this parameter. Is there any other place i can check and confirm? > > Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap > (Xms = Xmx) can be locked at node startup ? Will that help? > > Or if you have any other suggestions, please let me know. > > > Regards, > Kunal >
Disabling Swap for Cassandra
Hello, I need some suggestion from you all. I am new to Cassandra and was reading Cassandra best practices. On one document, it was mentioned that Cassandra should not be using swap, it degrades the performance. My question is instead of disabling swap system wide, can we force Cassandra not to use swap? Some documentation suggests to use memory_locking_policy in cassandra.yaml. How do I check if our Cassandra already has this parameter and still uses swap ? Is there any way i can check this. I already checked cassandra.yaml and dont see this parameter. Is there any other place i can check and confirm? Also, Can I set memlock parameter to unlimited (64kB default), so entire Heap (Xms = Xmx) can be locked at node startup ? Will that help? Or if you have any other suggestions, please let me know. Regards, Kunal
