Re: OOM only on one datacenter nodes
Centos 6.10 is a bit aged as a production server O/S platform, and I recall some odd-ball interactions with hardware variations, particularly around high-priority memory and network cards. How good is your O/S level metric monitoring? Not beyond the realm of possibility that your memory issues are outside of the JVM. It isn’t easy to tell you what to specifically look for, but I would begin with metrics around memory and swap. If you don’t see high consistent memory use outside of the JVM usage, saves wasting time chasing down details that are unlikely to matter. You need to be used to seeing what those metrics are normally like though, so you aren’t chasing phantoms. I second Jeff’s feedback. You need the information you need. It seems counterproductive to not configure these nodes to do what you need. A fundamental value of C* is the ability to bring nodes up and down without risking availability. When your existing technology approach is part of why you can’t gather the data you need, it helps to give yourself permission to improve what you have so you don’t remain in that situation. From: Surbhi Gupta Date: Monday, April 6, 2020 at 12:44 AM To: "user@cassandra.apache.org" Cc: Reid Pinchback Subject: Re: OOM only on one datacenter nodes Message from External Sender We are using JRE and not JDK , hence not able to take heap dump . On Sun, 5 Apr 2020 at 19:21, Jeff Jirsa mailto:jji...@gmail.com>> wrote: Set the jvm flags to heap dump on oom Open up the result in a heap inspector of your preference (like yourkit or similar) Find a view that counts objects by total retained size. Take a screenshot. Send that. On Apr 5, 2020, at 6:51 PM, Surbhi Gupta mailto:surbhi.gupt...@gmail.com>> wrote: I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. What specific parameter I should check on OS ? We are using CentOS release 6.10. Currently disk_access_modeis not set hence it is auto in our env. Should setting disk_access_mode to mmap_index_only will help ? Thanks Surbhi On Sun, 5 Apr 2020 at 01:31, Alex Ott mailto:alex...@gmail.com>> wrote: Have you set -Xmx32g ? In this case you may get significantly less available memory because of switch to 64-bit references. See http://java-performance.info/over-32g-heap-java/<https://urldefense.proofpoint.com/v2/url?u=http-3A__java-2Dperformance.info_over-2D32g-2Dheap-2Djava_&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=J9_U9AIV95JbVJ-0c1OyjqGOmdLCltCRwMPnOsS7BCE&s=rB9HFbb7t-FJQZUGJNtN0wOPIGZj7Fn8cE271bR63HE&e=> for details, and set slightly less than 32Gb Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: RP> Surbi: RP> If you aren’t seeing connection activity in DC2, I’d check to see if the operations hitting DC1 are quorum ops instead of local quorum. That RP> still wouldn’t explain DC2 nodes going down, but would at least explain them doing more work than might be on your radar right now. RP> The hint replay being slow to me sounds like you could be fighting GC. RP> You mentioned bumping the DC2 nodes to 32gb. You might have already been doing this, but if not, be sure to be under 32gb, like 31gb. RP> Otherwise you’re using larger object pointers and could actually have less effective ability to allocate memory. RP> As the problem is only happening in DC2, then there has to be a thing that is true in DC2 that isn’t true in DC1. A difference in hardware, a RP> difference in O/S version, a difference in networking config or physical infrastructure, a difference in client-triggered activity, or a RP> difference in how repairs are handled. Somewhere, there is a difference. I’d start with focusing on that. RP> From: Erick Ramirez mailto:erick.rami...@datastax.com>> RP> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> RP> Date: Saturday, April 4, 2020 at 8:28 PM RP> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> RP> Subject: Re: OOM only on one datacenter nodes RP> Message from External Sender RP> With a lack of heapdump for you to analyse, my hypothesis is that your DC2 nodes are taking on traffic (from some client somewhere) but you're RP> just not aware of it. The hints replay is just a side-effect of the nodes getting overloaded. RP> To rule out my hypothesis in the first instance, my recommendation is to monitor the incoming connections to the nodes in DC2. If you don't RP> have monitoring in place, you could simply run netstat at regular intervals and go from there. Cheers! RP> GOT QUESTIONS? Apache Cassandra experts fro
Re: OOM only on one datacenter nodes
Nobody is going to do any better than guessing without a heap histogram I’ve got pretty good intuition with cassandra in real prod environments and can think of like 8-9 different possible causes, but none of them really stand out as likely enough to describe in detail (mybe the memtable deadlock on flush, or maybe repair coordination in that dc), but really need a heap If it’s causing you enough pain to email a list, seems like it’s worth making the changes you need to make to get a heap and debug properly. > On Apr 5, 2020, at 9:44 PM, Surbhi Gupta wrote: > > > We are using JRE and not JDK , hence not able to take heap dump . > >> On Sun, 5 Apr 2020 at 19:21, Jeff Jirsa wrote: >> >> Set the jvm flags to heap dump on oom >> >> Open up the result in a heap inspector of your preference (like yourkit or >> similar) >> >> Find a view that counts objects by total retained size. Take a screenshot. >> Send that. >> >> >> >>>> On Apr 5, 2020, at 6:51 PM, Surbhi Gupta wrote: >>>> >>> >>> I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. >>> >>> I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. >>> What specific parameter I should check on OS ? >>> We are using CentOS release 6.10. >>> >>> Currently disk_access_modeis not set hence it is auto in our env. Should >>> setting disk_access_mode to mmap_index_only will help ? >>> >>> Thanks >>> Surbhi >>> >>>> On Sun, 5 Apr 2020 at 01:31, Alex Ott wrote: >>>> Have you set -Xmx32g ? In this case you may get significantly less >>>> available memory because of switch to 64-bit references. See >>>> http://java-performance.info/over-32g-heap-java/ for details, and set >>>> slightly less than 32Gb >>>> >>>> Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: >>>> RP> Surbi: >>>> >>>> RP> If you aren’t seeing connection activity in DC2, I’d check to see if >>>> the operations hitting DC1 are quorum ops instead of local quorum. That >>>> RP> still wouldn’t explain DC2 nodes going down, but would at least >>>> explain them doing more work than might be on your radar right now. >>>> >>>> RP> The hint replay being slow to me sounds like you could be fighting GC. >>>> >>>> RP> You mentioned bumping the DC2 nodes to 32gb. You might have already >>>> been doing this, but if not, be sure to be under 32gb, like 31gb. >>>> RP> Otherwise you’re using larger object pointers and could actually have >>>> less effective ability to allocate memory. >>>> >>>> RP> As the problem is only happening in DC2, then there has to be a thing >>>> that is true in DC2 that isn’t true in DC1. A difference in hardware, a >>>> RP> difference in O/S version, a difference in networking config or >>>> physical infrastructure, a difference in client-triggered activity, or a >>>> RP> difference in how repairs are handled. Somewhere, there is a >>>> difference. I’d start with focusing on that. >>>> >>>> RP> From: Erick Ramirez >>>> RP> Reply-To: "user@cassandra.apache.org" >>>> RP> Date: Saturday, April 4, 2020 at 8:28 PM >>>> RP> To: "user@cassandra.apache.org" >>>> RP> Subject: Re: OOM only on one datacenter nodes >>>> >>>> RP> Message from External Sender >>>> >>>> RP> With a lack of heapdump for you to analyse, my hypothesis is that >>>> your DC2 nodes are taking on traffic (from some client somewhere) but >>>> you're >>>> RP> just not aware of it. The hints replay is just a side-effect of the >>>> nodes getting overloaded. >>>> >>>> RP> To rule out my hypothesis in the first instance, my recommendation is >>>> to monitor the incoming connections to the nodes in DC2. If you don't >>>> RP> have monitoring in place, you could simply run netstat at regular >>>> intervals and go from there. Cheers! >>>> >>>> RP> GOT QUESTIONS? Apache Cassandra experts from the community and >>>> DataStax have answers! Share your expertise on >>>> https://community.datastax.com/. >>>> >>>> >>>> >>>> -- >>>> With best wishes,Alex Ott >>>> Principal Architect, DataStax >>>> http://datastax.com/ >>>> >>>> - >>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>>
Re: OOM only on one datacenter nodes
We are using JRE and not JDK , hence not able to take heap dump . On Sun, 5 Apr 2020 at 19:21, Jeff Jirsa wrote: > > Set the jvm flags to heap dump on oom > > Open up the result in a heap inspector of your preference (like yourkit or > similar) > > Find a view that counts objects by total retained size. Take a screenshot. > Send that. > > > > On Apr 5, 2020, at 6:51 PM, Surbhi Gupta wrote: > > > I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. > > I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. > What specific parameter I should check on OS ? > We are using CentOS release 6.10. > > Currently disk_access_modeis not set hence it is auto in our env. Should > setting disk_access_mode to mmap_index_only will help ? > > Thanks > Surbhi > > On Sun, 5 Apr 2020 at 01:31, Alex Ott wrote: > >> Have you set -Xmx32g ? In this case you may get significantly less >> available memory because of switch to 64-bit references. See >> http://java-performance.info/over-32g-heap-java/ for details, and set >> slightly less than 32Gb >> >> Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: >> RP> Surbi: >> >> RP> If you aren’t seeing connection activity in DC2, I’d check to see if >> the operations hitting DC1 are quorum ops instead of local quorum. That >> RP> still wouldn’t explain DC2 nodes going down, but would at least >> explain them doing more work than might be on your radar right now. >> >> RP> The hint replay being slow to me sounds like you could be fighting >> GC. >> >> RP> You mentioned bumping the DC2 nodes to 32gb. You might have already >> been doing this, but if not, be sure to be under 32gb, like 31gb. >> RP> Otherwise you’re using larger object pointers and could actually >> have less effective ability to allocate memory. >> >> RP> As the problem is only happening in DC2, then there has to be a >> thing that is true in DC2 that isn’t true in DC1. A difference in >> hardware, a >> RP> difference in O/S version, a difference in networking config or >> physical infrastructure, a difference in client-triggered activity, or a >> RP> difference in how repairs are handled. Somewhere, there is a >> difference. I’d start with focusing on that. >> >> RP> From: Erick Ramirez >> RP> Reply-To: "user@cassandra.apache.org" >> RP> Date: Saturday, April 4, 2020 at 8:28 PM >> RP> To: "user@cassandra.apache.org" >> RP> Subject: Re: OOM only on one datacenter nodes >> >> RP> Message from External Sender >> >> RP> With a lack of heapdump for you to analyse, my hypothesis is that >> your DC2 nodes are taking on traffic (from some client somewhere) but you're >> RP> just not aware of it. The hints replay is just a side-effect of the >> nodes getting overloaded. >> >> RP> To rule out my hypothesis in the first instance, my recommendation >> is to monitor the incoming connections to the nodes in DC2. If you don't >> RP> have monitoring in place, you could simply run netstat at regular >> intervals and go from there. Cheers! >> >> RP> GOT QUESTIONS? Apache Cassandra experts from the community and >> DataStax have answers! Share your expertise on >> https://community.datastax.com/. >> >> >> >> -- >> With best wishes,Alex Ott >> Principal Architect, DataStax >> http://datastax.com/ >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >>
Re: OOM only on one datacenter nodes
Set the jvm flags to heap dump on oom Open up the result in a heap inspector of your preference (like yourkit or similar) Find a view that counts objects by total retained size. Take a screenshot. Send that. > On Apr 5, 2020, at 6:51 PM, Surbhi Gupta wrote: > > > I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. > > I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. > What specific parameter I should check on OS ? > We are using CentOS release 6.10. > > Currently disk_access_modeis not set hence it is auto in our env. Should > setting disk_access_mode to mmap_index_only will help ? > > Thanks > Surbhi > >> On Sun, 5 Apr 2020 at 01:31, Alex Ott wrote: >> Have you set -Xmx32g ? In this case you may get significantly less >> available memory because of switch to 64-bit references. See >> http://java-performance.info/over-32g-heap-java/ for details, and set >> slightly less than 32Gb >> >> Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: >> RP> Surbi: >> >> RP> If you aren’t seeing connection activity in DC2, I’d check to see if >> the operations hitting DC1 are quorum ops instead of local quorum. That >> RP> still wouldn’t explain DC2 nodes going down, but would at least explain >> them doing more work than might be on your radar right now. >> >> RP> The hint replay being slow to me sounds like you could be fighting GC. >> >> RP> You mentioned bumping the DC2 nodes to 32gb. You might have already >> been doing this, but if not, be sure to be under 32gb, like 31gb. >> RP> Otherwise you’re using larger object pointers and could actually have >> less effective ability to allocate memory. >> >> RP> As the problem is only happening in DC2, then there has to be a thing >> that is true in DC2 that isn’t true in DC1. A difference in hardware, a >> RP> difference in O/S version, a difference in networking config or >> physical infrastructure, a difference in client-triggered activity, or a >> RP> difference in how repairs are handled. Somewhere, there is a >> difference. I’d start with focusing on that. >> >> RP> From: Erick Ramirez >> RP> Reply-To: "user@cassandra.apache.org" >> RP> Date: Saturday, April 4, 2020 at 8:28 PM >> RP> To: "user@cassandra.apache.org" >> RP> Subject: Re: OOM only on one datacenter nodes >> >> RP> Message from External Sender >> >> RP> With a lack of heapdump for you to analyse, my hypothesis is that your >> DC2 nodes are taking on traffic (from some client somewhere) but you're >> RP> just not aware of it. The hints replay is just a side-effect of the >> nodes getting overloaded. >> >> RP> To rule out my hypothesis in the first instance, my recommendation is >> to monitor the incoming connections to the nodes in DC2. If you don't >> RP> have monitoring in place, you could simply run netstat at regular >> intervals and go from there. Cheers! >> >> RP> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax >> have answers! Share your expertise on https://community.datastax.com/. >> >> >> >> -- >> With best wishes,Alex Ott >> Principal Architect, DataStax >> http://datastax.com/ >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >>
Re: OOM only on one datacenter nodes
I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. What specific parameter I should check on OS ? We are using CentOS release 6.10. Currently disk_access_modeis not set hence it is auto in our env. Should setting disk_access_mode to mmap_index_only will help ? Thanks Surbhi On Sun, 5 Apr 2020 at 01:31, Alex Ott wrote: > Have you set -Xmx32g ? In this case you may get significantly less > available memory because of switch to 64-bit references. See > http://java-performance.info/over-32g-heap-java/ for details, and set > slightly less than 32Gb > > Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: > RP> Surbi: > > RP> If you aren’t seeing connection activity in DC2, I’d check to see if > the operations hitting DC1 are quorum ops instead of local quorum. That > RP> still wouldn’t explain DC2 nodes going down, but would at least > explain them doing more work than might be on your radar right now. > > RP> The hint replay being slow to me sounds like you could be fighting GC. > > RP> You mentioned bumping the DC2 nodes to 32gb. You might have already > been doing this, but if not, be sure to be under 32gb, like 31gb. > RP> Otherwise you’re using larger object pointers and could actually have > less effective ability to allocate memory. > > RP> As the problem is only happening in DC2, then there has to be a thing > that is true in DC2 that isn’t true in DC1. A difference in hardware, a > RP> difference in O/S version, a difference in networking config or > physical infrastructure, a difference in client-triggered activity, or a > RP> difference in how repairs are handled. Somewhere, there is a > difference. I’d start with focusing on that. > > RP> From: Erick Ramirez > RP> Reply-To: "user@cassandra.apache.org" > RP> Date: Saturday, April 4, 2020 at 8:28 PM > RP> To: "user@cassandra.apache.org" > RP> Subject: Re: OOM only on one datacenter nodes > > RP> Message from External Sender > > RP> With a lack of heapdump for you to analyse, my hypothesis is that > your DC2 nodes are taking on traffic (from some client somewhere) but you're > RP> just not aware of it. The hints replay is just a side-effect of the > nodes getting overloaded. > > RP> To rule out my hypothesis in the first instance, my recommendation is > to monitor the incoming connections to the nodes in DC2. If you don't > RP> have monitoring in place, you could simply run netstat at regular > intervals and go from there. Cheers! > > RP> GOT QUESTIONS? Apache Cassandra experts from the community and > DataStax have answers! Share your expertise on > https://community.datastax.com/. > > > > -- > With best wishes,Alex Ott > Principal Architect, DataStax > http://datastax.com/ > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: OOM only on one datacenter nodes
Have you set -Xmx32g ? In this case you may get significantly less available memory because of switch to 64-bit references. See http://java-performance.info/over-32g-heap-java/ for details, and set slightly less than 32Gb Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: RP> Surbi: RP> If you aren’t seeing connection activity in DC2, I’d check to see if the operations hitting DC1 are quorum ops instead of local quorum. That RP> still wouldn’t explain DC2 nodes going down, but would at least explain them doing more work than might be on your radar right now. RP> The hint replay being slow to me sounds like you could be fighting GC. RP> You mentioned bumping the DC2 nodes to 32gb. You might have already been doing this, but if not, be sure to be under 32gb, like 31gb. RP> Otherwise you’re using larger object pointers and could actually have less effective ability to allocate memory. RP> As the problem is only happening in DC2, then there has to be a thing that is true in DC2 that isn’t true in DC1. A difference in hardware, a RP> difference in O/S version, a difference in networking config or physical infrastructure, a difference in client-triggered activity, or a RP> difference in how repairs are handled. Somewhere, there is a difference. I’d start with focusing on that. RP> From: Erick Ramirez RP> Reply-To: "user@cassandra.apache.org" RP> Date: Saturday, April 4, 2020 at 8:28 PM RP> To: "user@cassandra.apache.org" RP> Subject: Re: OOM only on one datacenter nodes RP> Message from External Sender RP> With a lack of heapdump for you to analyse, my hypothesis is that your DC2 nodes are taking on traffic (from some client somewhere) but you're RP> just not aware of it. The hints replay is just a side-effect of the nodes getting overloaded. RP> To rule out my hypothesis in the first instance, my recommendation is to monitor the incoming connections to the nodes in DC2. If you don't RP> have monitoring in place, you could simply run netstat at regular intervals and go from there. Cheers! RP> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/. -- With best wishes,Alex Ott Principal Architect, DataStax http://datastax.com/ - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: OOM only on one datacenter nodes
Surbi: If you aren’t seeing connection activity in DC2, I’d check to see if the operations hitting DC1 are quorum ops instead of local quorum. That still wouldn’t explain DC2 nodes going down, but would at least explain them doing more work than might be on your radar right now. The hint replay being slow to me sounds like you could be fighting GC. You mentioned bumping the DC2 nodes to 32gb. You might have already been doing this, but if not, be sure to be under 32gb, like 31gb. Otherwise you’re using larger object pointers and could actually have less effective ability to allocate memory. As the problem is only happening in DC2, then there has to be a thing that is true in DC2 that isn’t true in DC1. A difference in hardware, a difference in O/S version, a difference in networking config or physical infrastructure, a difference in client-triggered activity, or a difference in how repairs are handled. Somewhere, there is a difference. I’d start with focusing on that. From: Erick Ramirez Reply-To: "user@cassandra.apache.org" Date: Saturday, April 4, 2020 at 8:28 PM To: "user@cassandra.apache.org" Subject: Re: OOM only on one datacenter nodes Message from External Sender With a lack of heapdump for you to analyse, my hypothesis is that your DC2 nodes are taking on traffic (from some client somewhere) but you're just not aware of it. The hints replay is just a side-effect of the nodes getting overloaded. To rule out my hypothesis in the first instance, my recommendation is to monitor the incoming connections to the nodes in DC2. If you don't have monitoring in place, you could simply run netstat at regular intervals and go from there. Cheers! GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__community.datastax.com_&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=fGh0C8n6vv7LqulA6b4vVVTavsof7CZt6ESlDCr-uP8&s=oZjamTdUMswHVohvkHQQftZdYivh1qRAmRn1-dap_Uo&e=>.
Re: OOM only on one datacenter nodes
With a lack of heapdump for you to analyse, my hypothesis is that your DC2 nodes are taking on traffic (from some client somewhere) but you're just not aware of it. The hints replay is just a side-effect of the nodes getting overloaded. To rule out my hypothesis in the first instance, my recommendation is to monitor the incoming connections to the nodes in DC2. If you don't have monitoring in place, you could simply run netstat at regular intervals and go from there. Cheers! GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/.