Re: Almost nodes in Solrcloud dead suddently

2020-07-04 Thread Tran Van Hoan
*total thread is 25.6k when solr hang. On Sunday, July 5, 2020, 2:55:26 AM GMT+7, Tran Van Hoan wrote: All server only run Solr, zookeeper, exporters (node-exporter, process-exporter, solr-exporter, zoo-exporter). - network: no package loss, TCP no issue before incident, TCP drop

Re: Almost nodes in Solrcloud dead suddently

2020-07-04 Thread Tran Van Hoan
ons, only one collection down causes node down (the remain collection is still green) On Sunday, July 5, 2020, 1:30:59 AM GMT+7, Rodrigo Oliveira wrote: Network it's ok? Between nodes? The use? Swap it's disabled? Swapiness rhe value it's 0? Em sáb, 4 de jul de 2020 15:19, Tran Van H

Re: Almost nodes in Solrcloud dead suddently

2020-07-04 Thread Tran Van Hoan
at host physical (until backup it's a problem over veeam). Regards Em sáb, 4 de jul de 2020 12:30, Tran Van Hoan escreveu: > The problem reoccurs repeatly in recent days. > To day i tried dump heap and thread. Only dumping thread, heap can not > because solr instance was hang. > A

Re: Almost nodes in Solrcloud dead suddently

2020-07-04 Thread Tran Van Hoan
The problem reoccurs repeatly in recent days. To day i tried dump heap and thread. Only dumping thread, heap can not because solr instance was hang.Almost thread was blocked. On Tuesday, June 23, 2020, 10:42:36 PM GMT+7, Tran Van Hoan wrote: I checked node exporter metrics and saw

Re: Almost nodes in Solrcloud dead suddently

2020-06-23 Thread Tran Van Hoan
I checked node exporter metrics and saw network no problem On Tuesday, June 23, 2020, 8:37:41 PM GMT+7, Tran Van Hoan wrote: I check node exporter, no problem with OS, hardware and network.I attached images about solr metrics 7 days and 12h. On Tuesday, June 23, 2020, 2:23:05

Almost nodes in Solrcloud dead suddently

2020-06-22 Thread Tran Van Hoan
dear all, I have a solr cloud 8.2.0 with 6 instance per 6 server (64G RAM), each instance has xmx = xms = 30G. Today almost nodes in the solrcloud were dead 2 times from 8:00AM (5/6 nodes were down) and 1:00PM (2/6 nodes  were down). yesterday,  One node were down. almost metrics didn't