Hi friends! *solr version: 4.9.0* I came across a problem when use solrcloud, it becomes dead lock, we got the java core log, it looks like the http connection pool is exhausted and most threads are waiting to get a free connection..
I posted the problem in JIRA, the link is https://issues.apache.org/jira/browse/SOLR-9253 I have increased http connection defaults for the SolrJ client, and also configed the connection defaults in solr.xml for all shard servers as below. <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory"> <int name="socketTimeout">60000</int> <int name="connTimeout">30000</int> <int name="maxConnections">10000</int> <int name="maxConnectionsPerHost">500</int> </shardHandlerFactory> *besides, we use the singleton pattern to connect solrlcoud.* public synchronized static CloudSolrServer getSolrCloudReadServer() { if (reviewSolrCloudServer == null) { ModifiableSolrParams params = new ModifiableSolrParams(); params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000); params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500); HttpClient client = HttpClientUtil.createClient(params); LBHttpSolrServer lbServer = new LBHttpSolrServer(client); lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut()); lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut()); reviewSolrCloudServer = new CloudSolrServer(ReviewProperties.getZkHost(),lbServer); reviewSolrCloudServer.setDefaultCollection("commodityReview"); reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout()); reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout()); reviewSolrCloudServer.connect(); } return reviewSolrCloudServer; } *the java stack as below* "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000 nid=0x20af waiting on condition [0x00007f79fd490000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000605710068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208) at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301) at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) please help me, look forward to your reply. Thanks a lot! Best Regards! Kent
