Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

FYI. This issue went away after solrconfig.xml was tuned. "Hard commits blocked | non-solrcloud v6.6.2" thread has the details. http://lucene.472066.n3.nabble.com/Hard-commits-blocked-non-solrcloud-v6-6-2-td4374386.html -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

> Maybe this is the issue: https://github.com/eclipse/jetty.project/issues/2169 Looks like it is the issue. (I've readacted IP addresses below for security reasons) solr [ /opt/solr ]$ netstat -ptan | awk '{print $6 " " $7 }' | sort | uniq -c 8425 CLOSE_WAIT - 92 ESTABLISHED - 1

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

On 2/5/2018 12:29 PM, mmb1234 wrote: > Maybe this is the issue: https://github.com/eclipse/jetty.project/issues/2169 Could be.  That was one of the Jetty issues I found, and the one that said 9.4 doesn't have the problem.  Your Solr version would be using Jetty 9.3, not 9.2. > The custom client

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

Maybe this is the issue: https://github.com/eclipse/jetty.project/issues/2169 I have noticed when number of http requests / sec are increased, CLOSE_WAITS increase linearly until solr stops accepting socket connections. Netstat output is $ netstat -ptan | awk '{print $6 " " $7 }' | sort | uniq -c

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

On 2/2/2018 10:00 AM, mmb1234 wrote: Client / java8 app: An AsyncHTTPClient POST-ing gzip payloads. PoolingNHttpClientConnectionManager maxtotal=10,000 and maxperroute=1000) ConnectionRequestTimeout = ConnectTimeout = SocketTimeout = 4000 (4 secs) I have to concur with the person who

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

Hi Arcadius, Most of the clients use Solrj to interact with Solr. Does it not automatically handle the connection pools? SG On Fri, Feb 2, 2018 at 4:47 PM, Arcadius Ahouansou wrote: > I have seen a lot of CLOSE_WAIT in the past. > In many cases, it was that the client

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

I have seen a lot of CLOSE_WAIT in the past. In many cases, it was that the client application was not releasing/closing or pooling connections properly. I would suggest you double check the client code first. Arcadius. On 2 February 2018 at 23:52, mmb1234 wrote: > > You said

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

> You said that you're running Solr 6.2.2, but there is no 6.2.2 version. > but the JVM argument list includes "-Xmx512m" which is a 512MB heap My typos. They're 6.6.2 and -Xmx30g respectively. > many open connections causes is a large number of open file handles, solr [ /opt/solr/server/logs

Re: 9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

On 2/2/2018 10:00 AM, mmb1234 wrote: In our solr non-cloud env., we are seeing lots of CLOSE_WAIT, causing jvm to stop "working" with 3 mins of solr start. solr [ /opt/solr ]$ netstat -anp | grep 8983 | grep CLOSE_WAIT | grep 10.xxx.xxx.xxx | wc -l 9453 Solr isn't handling network services. 

9000+ CLOSE_WAIT connections in solr v6.2.2 causing it to "die"

Hello, In our solr non-cloud env., we are seeing lots of CLOSE_WAIT, causing jvm to stop "working" with 3 mins of solr start. solr [ /opt/solr ]$ netstat -anp | grep 8983 | grep CLOSE_WAIT | grep 10.xxx.xxx.xxx | wc -l 9453 Only option is then`kill -9` because even `jcmd Thread.print` is