Dear Mr. Mangar! Thanks for your reply. we have 3 cores with 5 shards, and each shard with one leader and one replication. now the data account goes to be 100 million, and it is still growing up.I see no connection timeout. actually we have schedule job to commit the data every 2 hours. so the action to commit is not that frequent, and no auto-commit. the write operation seems to be OK.
I just see the reply by Mr. Heisey. He finds "jboss running info in java core". and he says that we need to raise the maxThread in JBOSS, which is 200 as default. so I decide to config it on next week, and to see if it will be OK. Thanks again! 2016-07-07 16:08 GMT+08:00 Shalin Shekhar Mangar <shalinman...@gmail.com>: > Hi Kent, > > There is no point sending multiple emails for the same subject. It > distracts people from the other messages, distributes the conversation and > discourages people from helping you. > > Please provide more details about your cluster. > 1. How many nodes? > 2. How many collections? > 3. How many shards? > 4. What is the replication factor? > 5. Are there connection and read timeouts specified on both client and > server? If yes, what are the values? > 6. How often do you commit? Is there auto-commit or explicit commits from > clients? > 7. What else is happening on your cluster? What is the write and query > load? > > The important thing is to figure out why these pools are getting exhausted. > 10000 max connections and 500 connections per host is already pretty high > so there is an underlying cause for such stuck requests. > > On Wed, Jul 6, 2016 at 4:56 PM, Kent Mu <solr.st...@gmail.com> wrote: > > > Hi friends! > > *solr version: 4.9.0* > > > > I came across a problem when use solrcloud, it becomes dead lock, we got > > the java core log, it looks like the http connection pool is exhausted > and > > most threads are waiting to get a free connection.. > > > > I posted the problem in JIRA, the link is > > https://issues.apache.org/jira/browse/SOLR-9253 > > I have increased http connection defaults for the SolrJ client, and also > > configed the connection defaults in solr.xml for all shard servers as > > below. > > > > <shardHandlerFactory name="shardHandlerFactory" > > class="HttpShardHandlerFactory"> > > <int name="socketTimeout">60000</int> > > <int name="connTimeout">30000</int> > > <int name="maxConnections">10000</int> > > <int name="maxConnectionsPerHost">500</int> > > </shardHandlerFactory> > > > > *besides, we use the singleton pattern to connect solrlcoud.* > > > > public synchronized static CloudSolrServer getSolrCloudReadServer() { > > if (reviewSolrCloudServer == null) { > > ModifiableSolrParams params = new ModifiableSolrParams(); > > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 2000); > > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 500); > > HttpClient client = HttpClientUtil.createClient(params); > > LBHttpSolrServer lbServer = new LBHttpSolrServer(client); > > > > > > > lbServer.setConnectionTimeout(ReviewProperties.getCloudConnectionTimeOut()); > > lbServer.setSoTimeout(ReviewProperties.getCloudSoTimeOut()); > > > > reviewSolrCloudServer = new > > CloudSolrServer(ReviewProperties.getZkHost(),lbServer); > > reviewSolrCloudServer.setDefaultCollection("commodityReview"); > > > > > > > reviewSolrCloudServer.setZkClientTimeout(ReviewProperties.getZkClientTimeout()); > > > > > > > reviewSolrCloudServer.setZkConnectTimeout(ReviewProperties.getZkConnectTimeout()); > > reviewSolrCloudServer.connect(); > > } > > return reviewSolrCloudServer; > > } > > > > *the java stack as below* > > > > "httpShardExecutor-3-thread-541" prio=10 tid=0x00007f7b1c02b000 > nid=0x20af > > waiting on condition [0x00007f79fd490000] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x0000000605710068> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > > at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) > > at > > > > > org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) > > at > > > org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) > > at > > > > > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) > > at > > > > > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) > > at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) > > at > > > > > org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208) > > at > > > > > org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195) > > at > > > > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422) > > at > > > > > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) > > at > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > > at > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) > > at > > > > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452) > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) > > at > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340) > > at > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301) > > at > > > > > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205) > > at > > > > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) > > at > > > > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:724) > > > > please help me, look forward to your reply. Thanks a lot! > > > > Best Regards! > > Kent > > > > > > -- > Regards, > Shalin Shekhar Mangar. >
