Re: OOM only on one datacenter nodes

2020-04-05 Thread Surbhi Gupta
We are using JRE and not JDK , hence not able to take heap dump . On Sun, 5 Apr 2020 at 19:21, Jeff Jirsa wrote: > > Set the jvm flags to heap dump on oom > > Open up the result in a heap inspector of your preference (like yourkit or > similar) > > Find a view that counts objects by total

Re: OOM only on one datacenter nodes

2020-04-05 Thread Jeff Jirsa
Set the jvm flags to heap dump on oom Open up the result in a heap inspector of your preference (like yourkit or similar) Find a view that counts objects by total retained size. Take a screenshot. Send that. > On Apr 5, 2020, at 6:51 PM, Surbhi Gupta wrote: > >  > I just checked, we

Re: OOM only on one datacenter nodes

2020-04-05 Thread Surbhi Gupta
I just checked, we have setup the Heapsize to be 31GB not 32GB in DC2. I checked the CPU and RAM both are same on all the nodes in DC1 and DC2. What specific parameter I should check on OS ? We are using CentOS release 6.10. Currently disk_access_modeis not set hence it is auto in our env.

Re: unconfigured table logtabl

2020-04-05 Thread Jeff Jirsa
Be really cautious here, this can be deceptive There are races in some versions of cassandra that can leave you with different combinations of cfid The cfid is on disk for schema It’s in memory for schema It’s used for the table path on disk Those three have to match for things to work

Re: unconfigured table logtabl

2020-04-05 Thread Erick Ramirez
> > Another suggestion before resetlocalschema. Try rolling restart all the > nodes in the cluster and see if it fix the problem. After the restart all > the nodes will use the same schema for the table. > That's a little bit heavy-handed.  Resetting a node's schema is a simple, online operation

Re: unconfigured table logtabl

2020-04-05 Thread Jai Bheemsen Rao Dhanwada
Another suggestion before resetlocalschema. Try rolling restart all the nodes in the cluster and see if it fix the problem. After the restart all the nodes will use the same schema for the table. On Sunday, April 5, 2020, David Ni wrote: > Hi Erick > Thank you very much for your friendly note.

Re:Re:Re: Re: Re: unconfigured table logtabl

2020-04-05 Thread David Ni
Hi Erick Thank you very much for your friendly note. ERROR [AntiEntropyStage:1] 2020-04-04 13:57:09,614 RepairMessageVerbHandler.java:177 - Table with id 21a3fa90-74c7-11ea-978a-b556b0c3a5ea was dropped during prepare phase of repair cassandra@cqlsh:system_schema> select

Re: OOM only on one datacenter nodes

2020-04-05 Thread Alex Ott
Have you set -Xmx32g ? In this case you may get significantly less available memory because of switch to 64-bit references. See http://java-performance.info/over-32g-heap-java/ for details, and set slightly less than 32Gb Reid Pinchback at "Sun, 5 Apr 2020 00:50:43 +" wrote: RP> Surbi: