Hi, we have problems with our application: we use the solrj client that uses internally commons httpclient / MultiThreadedHttpConnectionManager, the httpclient is release 3.1.
From time to time (once/twice a week, with 150.000 requests per day) the
requests are blocking, then we have lots of threads (e.g. 300, 400)
waiting in MTHCM.doGetConnection (l. 518):
"TP-Processor149" daemon prio=10 tid=0x00002aab431b2c00 nid=0x4bc8 in
Object.wait() [0x0000000043250000..0x0000000043253bc0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaabf751468> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked <0x00002aaabf751468> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
Additionally I see the MultiThreadedHttpConnectionManager cleanup thread
waiting at ReferenceQueue.remove:
"MultiThreadedHttpConnectionManager cleanup" daemon prio=10
tid=0x00002aab439de800 nid=0x11d1 in Object.wait() [0x0000000
04241b000..0x000000004241bd40]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaabfca80e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaabfca80e8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
Now I wonder, why so many threads are waiting in doGetConnection. They
should be interrupted when MTHCM.notifyWaitingThread is invoked, so it
seems as if this is not called.
Though, this should be the case, asuming that there are/were connections
in use that should be released by solrj otherwise no thread should reach
the wait.
One thread dump containing 302 MTHCM.doGetConnection in state waiting
has 6 threads where the solrj client is processing the response, and
these are not blocking or anything.
AFAICS the solrj client uses httpclient correctly, especially in that it
releases the connection within a finally block (see also [1]).
Some additional facts:
- DefaultMaxConnectionsPerHost is set to 32
- MaxTotalConnections is set to 128
- ConnectionManagerTimeout is not set / 0
Has someone an idea what might be wrong here, or experienced anything
similar?
Thanx in advance,
cheers,
Martin
[1]
http://svn.apache.org/viewvc/lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/impl/CommonsHttpSolrServer.java?view=markup
signature.asc
Description: This is a digitally signed message part
