[
https://issues.apache.org/jira/browse/SOLR-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renato Haeberli updated SOLR-18312:
-----------------------------------
Description:
HttpJdkSolrClient uses the same thread-pool (default size of 4) for its
HttpClient and also to write the response using RequestWriter.ContentWriter.
If HttpJdkSolrClient is used with 4 or more threads and a large query, roughly
around the 1KB mark, all 4 threads from the pool become body-writers and no
thread is left for the HttpClient to read, see attached thread-dump.
I belief under 1KB, the body is written in one go and the body-writer thread is
freed up before anything is actually read. With larger bodies, the thread is
held and the first 1 KB is flushed, but HttpClient has no thread to consume it.
This happens only with Http 1.1. and not with Http 2, as far as I understand
this is due to a different mechanism Http 2 uses under the hood.
>From the thread-dump:
4 Reader threads waiting for the first 1 KB of the body to be consumed, but it
can't be consumed since no thread is available from the pool.
{code:java}
"HttpJdkSolrClient-1-thread-4" #41 [34051] prio=5 os_prio=31 cpu=5.07ms
elapsed=41.77s tid=0x00000009260d3200 nid=34051 in Object.wait()
[0x000000016f1f5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait0([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:389)
at
java.io.PipedInputStream.awaitSpace([email protected]/PipedInputStream.java:280)
at
java.io.PipedInputStream.receive([email protected]/PipedInputStream.java:238)
- locked <0x000000703d0003f8> (a java.io.PipedInputStream)
at
java.io.PipedOutputStream.write([email protected]/PipedOutputStream.java:155)
at
sun.nio.cs.StreamEncoder.writeBytes([email protected]/StreamEncoder.java:220)
at
sun.nio.cs.StreamEncoder.implFlushBuffer([email protected]/StreamEncoder.java:315)
at
sun.nio.cs.StreamEncoder.implFlush([email protected]/StreamEncoder.java:320)
at sun.nio.cs.StreamEncoder.flush([email protected]/StreamEncoder.java:150)
- locked <0x000000703d000420> (a java.io.OutputStreamWriter)
at
java.io.OutputStreamWriter.flush([email protected]/OutputStreamWriter.java:249)
at org.apache.solr.common.util.Utils.writeJson(Utils.java:237)
at
org.apache.solr.client.solrj.request.json.JsonQueryRequest$1.write(JsonQueryRequest.java:430)
at
org.apache.solr.client.solrj.impl.HttpJdkSolrClient.lambda$preparePutOrPost$2(HttpJdkSolrClient.java:310)
at
org.apache.solr.client.solrj.impl.HttpJdkSolrClient$$Lambda/0x00003c00010cac08.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:545)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:328)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda/0x00003c00010cb938.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1090)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:614)
at java.lang.Thread.runWith([email protected]/Thread.java:1487)
at java.lang.Thread.run([email protected]/Thread.java:1474) {code}
A work-around is to set an own Executor when creating HttpJdkSolrClient, like
this:
{code:java}
new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
.withExecutor(Executors.newVirtualThreadPerTaskExecutor()) {code}
or to set useHttp1_1 to false:
{code:java}
return new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
.useHttp1_1(flase) {code}
was:
HttpJdkSolrClient uses the same thread-pool (default size of 4) for its
HttpClient and also to write the response using RequestWriter.ContentWriter.
If HttpJdkSolrClient is used with 4 or more threads and a large query, roughly
around the 1KB mark, all 4 threads from the pool become body-writers and no
thread is left for the HttpClient to read, see attached thread-dump.
I belief under 1KB, the body is written in one go and the body-writer thread is
freed up before anything is actually read. With larger bodies, the thread is
held and the first 1 KB is flushed, but HttpClient has no thread to consume it.
This happens only with Http 1.1. and not with Http 2, as far as I understand
this is due to a different mechanism Http 2 uses under the hood.
>From the thread-dump:
4 Reader threads waiting for the first 1 KB of the body to be consumed, but it
can't be consumed since no thread is available from the pool.
{code:java}
"HttpJdkSolrClient-1-thread-4" #41 [34051] prio=5 os_prio=31 cpu=5.07ms
elapsed=41.77s tid=0x00000009260d3200 nid=34051 in Object.wait()
[0x000000016f1f5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait0([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:389)
at
java.io.PipedInputStream.awaitSpace([email protected]/PipedInputStream.java:280)
at
java.io.PipedInputStream.receive([email protected]/PipedInputStream.java:238)
- locked <0x000000703d0003f8> (a java.io.PipedInputStream)
at
java.io.PipedOutputStream.write([email protected]/PipedOutputStream.java:155)
at
sun.nio.cs.StreamEncoder.writeBytes([email protected]/StreamEncoder.java:220)
at
sun.nio.cs.StreamEncoder.implFlushBuffer([email protected]/StreamEncoder.java:315)
at
sun.nio.cs.StreamEncoder.implFlush([email protected]/StreamEncoder.java:320)
at sun.nio.cs.StreamEncoder.flush([email protected]/StreamEncoder.java:150)
- locked <0x000000703d000420> (a java.io.OutputStreamWriter)
at
java.io.OutputStreamWriter.flush([email protected]/OutputStreamWriter.java:249)
at org.apache.solr.common.util.Utils.writeJson(Utils.java:237)
at
org.apache.solr.client.solrj.request.json.JsonQueryRequest$1.write(JsonQueryRequest.java:430)
at
org.apache.solr.client.solrj.impl.HttpJdkSolrClient.lambda$preparePutOrPost$2(HttpJdkSolrClient.java:310)
at
org.apache.solr.client.solrj.impl.HttpJdkSolrClient$$Lambda/0x00003c00010cac08.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:545)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:328)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda/0x00003c00010cb938.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1090)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:614)
at java.lang.Thread.runWith([email protected]/Thread.java:1487)
at java.lang.Thread.run([email protected]/Thread.java:1474) {code}
> Thread-Pool Starvation in HttpJdkSolrClient for large Queries.
> --------------------------------------------------------------
>
> Key: SOLR-18312
> URL: https://issues.apache.org/jira/browse/SOLR-18312
> Project: Solr
> Issue Type: Bug
> Reporter: Renato Haeberli
> Priority: Minor
> Attachments: HttpJdkSolrClient_thread-dump.txt
>
>
> HttpJdkSolrClient uses the same thread-pool (default size of 4) for its
> HttpClient and also to write the response using RequestWriter.ContentWriter.
> If HttpJdkSolrClient is used with 4 or more threads and a large query,
> roughly around the 1KB mark, all 4 threads from the pool become body-writers
> and no thread is left for the HttpClient to read, see attached thread-dump.
> I belief under 1KB, the body is written in one go and the body-writer thread
> is freed up before anything is actually read. With larger bodies, the thread
> is held and the first 1 KB is flushed, but HttpClient has no thread to
> consume it.
> This happens only with Http 1.1. and not with Http 2, as far as I understand
> this is due to a different mechanism Http 2 uses under the hood.
> From the thread-dump:
> 4 Reader threads waiting for the first 1 KB of the body to be consumed, but
> it can't be consumed since no thread is available from the pool.
> {code:java}
> "HttpJdkSolrClient-1-thread-4" #41 [34051] prio=5 os_prio=31 cpu=5.07ms
> elapsed=41.77s tid=0x00000009260d3200 nid=34051 in Object.wait()
> [0x000000016f1f5000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait0([email protected]/Native Method)
> - waiting on <no object reference available>
> at java.lang.Object.wait([email protected]/Object.java:389)
> at
> java.io.PipedInputStream.awaitSpace([email protected]/PipedInputStream.java:280)
> at
> java.io.PipedInputStream.receive([email protected]/PipedInputStream.java:238)
> - locked <0x000000703d0003f8> (a java.io.PipedInputStream)
> at
> java.io.PipedOutputStream.write([email protected]/PipedOutputStream.java:155)
> at
> sun.nio.cs.StreamEncoder.writeBytes([email protected]/StreamEncoder.java:220)
> at
> sun.nio.cs.StreamEncoder.implFlushBuffer([email protected]/StreamEncoder.java:315)
> at
> sun.nio.cs.StreamEncoder.implFlush([email protected]/StreamEncoder.java:320)
> at sun.nio.cs.StreamEncoder.flush([email protected]/StreamEncoder.java:150)
> - locked <0x000000703d000420> (a java.io.OutputStreamWriter)
> at
> java.io.OutputStreamWriter.flush([email protected]/OutputStreamWriter.java:249)
> at org.apache.solr.common.util.Utils.writeJson(Utils.java:237)
> at
> org.apache.solr.client.solrj.request.json.JsonQueryRequest$1.write(JsonQueryRequest.java:430)
> at
> org.apache.solr.client.solrj.impl.HttpJdkSolrClient.lambda$preparePutOrPost$2(HttpJdkSolrClient.java:310)
> at
> org.apache.solr.client.solrj.impl.HttpJdkSolrClient$$Lambda/0x00003c00010cac08.run(Unknown
> Source)
> at
> java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:545)
> at
> java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:328)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda/0x00003c00010cb938.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1090)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:614)
> at java.lang.Thread.runWith([email protected]/Thread.java:1487)
> at java.lang.Thread.run([email protected]/Thread.java:1474) {code}
> A work-around is to set an own Executor when creating HttpJdkSolrClient, like
> this:
> {code:java}
> new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
> .withExecutor(Executors.newVirtualThreadPerTaskExecutor()) {code}
> or to set useHttp1_1 to false:
> {code:java}
> return new HttpJdkSolrClient.Builder(DEFAULT_SOLR_URL)
> .useHttp1_1(flase) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
