[
https://issues.apache.org/jira/browse/SOLR-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099200#comment-18099200
]
David Smiley commented on SOLR-18312:
-------------------------------------
Nice analysis!
The root cause, though, is misconfiguration of the Executor. I've seen this
before – see SOLR-17391. It's a JDK footgun that I wish various static
analyzers would catch; ah well. Our ExecutorUtil has factory methods that
avoids that problem. However if code directly constructs
MDCAwareThreadPoolExecutor then it can side-step the issue. I'm going to do
another PR/issue to help prevent this problem from happening again. I already
see HttpJettySolrClient is affected!
It does make sense to use a separate pool because there's a producer/consumer
relationship. On the other hand, if we guarantee an infinite pool then no
need. Anyway, I like the separate pool.
> Thread-Pool Starvation in HttpJdkSolrClient for large Queries.
> --------------------------------------------------------------
>
> Key: SOLR-18312
> URL: https://issues.apache.org/jira/browse/SOLR-18312
> Project: Solr
> Issue Type: Bug
> Reporter: Renato Haeberli
> Priority: Minor
> Labels: pull-request-available
> Attachments: HttpJdkSolrClient_thread-dump.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HttpJdkSolrClient uses the same thread-pool (default size of 4) for its
> HttpClient and also to write the request using RequestWriter.ContentWriter.
> If HttpJdkSolrClient is used with 4 or more threads and a large query,
> roughly around the 1KB mark, all 4 threads from the pool become body-writers
> and no thread is left for the HttpClient to read, see attached thread-dump.
> I belief under 1KB, the body is written in one go and the body-writer thread
> is freed up before anything is actually read. With larger bodies, the thread
> is held and the first 1 KB is flushed, but HttpClient has no thread to
> consume it.
> This happens only with Http 1.1. and not with Http 2, as far as I understand
> this is due to a different mechanism Http 2 uses under the hood.
> From the thread-dump:
> 4 Writer threads waiting for the first 1 KB of the body to be consumed, but
> it can't be consumed since no thread is available from the pool.
> {code:java}
> "HttpJdkSolrClient-1-thread-4" #41 [34051] prio=5 os_prio=31 cpu=5.07ms
> elapsed=41.77s tid=0x00000009260d3200 nid=34051 in Object.wait()
> [0x000000016f1f5000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait0([email protected]/Native Method)
> - waiting on <no object reference available>
> at java.lang.Object.wait([email protected]/Object.java:389)
> at
> java.io.PipedInputStream.awaitSpace([email protected]/PipedInputStream.java:280)
> at
> java.io.PipedInputStream.receive([email protected]/PipedInputStream.java:238)
> - locked <0x000000703d0003f8> (a java.io.PipedInputStream)
> at
> java.io.PipedOutputStream.write([email protected]/PipedOutputStream.java:155)
> at
> sun.nio.cs.StreamEncoder.writeBytes([email protected]/StreamEncoder.java:220)
> at
> sun.nio.cs.StreamEncoder.implFlushBuffer([email protected]/StreamEncoder.java:315)
> at
> sun.nio.cs.StreamEncoder.implFlush([email protected]/StreamEncoder.java:320)
> at sun.nio.cs.StreamEncoder.flush([email protected]/StreamEncoder.java:150)
> - locked <0x000000703d000420> (a java.io.OutputStreamWriter)
> at
> java.io.OutputStreamWriter.flush([email protected]/OutputStreamWriter.java:249)
> at org.apache.solr.common.util.Utils.writeJson(Utils.java:237)
> at
> org.apache.solr.client.solrj.request.json.JsonQueryRequest$1.write(JsonQueryRequest.java:430)
> at
> org.apache.solr.client.solrj.impl.HttpJdkSolrClient.lambda$preparePutOrPost$2(HttpJdkSolrClient.java:310)
> at
> org.apache.solr.client.solrj.impl.HttpJdkSolrClient$$Lambda/0x00003c00010cac08.run(Unknown
> Source)
> at
> java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:545)
> at
> java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:328)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda/0x00003c00010cb938.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1090)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:614)
> at java.lang.Thread.runWith([email protected]/Thread.java:1487)
> at java.lang.Thread.run([email protected]/Thread.java:1474) {code}
> A work-around is to set an own Executor when creating HttpJdkSolrClient, like
> this:
> {code:java}
> new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
> .withExecutor(Executors.newVirtualThreadPerTaskExecutor()) {code}
> or to set useHttp1_1 to false:
> {code:java}
> return new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
> .useHttp1_1(flase) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
