[
https://issues.apache.org/jira/browse/SOLR-16129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515233#comment-17515233
]
Ishan Chattopadhyaya edited comment on SOLR-16129 at 3/31/22, 1:01 PM:
-----------------------------------------------------------------------
-The test mentioned in SOLR-15840 fails with Solr 8.10 (after about 1 hour of
running), but passes on jira/solr-16129 branch (built from your patch). I've
started the run on main branch, and will report back the findings.-
Edit: The test fails on both main and jira/solr-16129 branch. After changing
both to solr.http1=true, both branches pass.
Steps to reproduce:
Run it on a 8 core (or 6 core machine)
1. Clone the solr-bench's master
2. Install the tools: `apt install wget unzip zip ant ivy lsof git netcat make
openjdk-11-jdk maven jq`
3. Edit the http2-bug.json. Build Solr locally, copy it to solr-bench folder,
then point the `solr-package` to that filename (tgz). Change the startup params
from `solr.http1=true` to false.
4. mvn clean compile assembly:single
5. ./stress.sh http2-bug.json
After about 40-45 minutes, I see:
{code}
[task3-threadpool] INFO StressMain - Running benchmarking task:
matchalldocs.json
[task3-threadpool] INFO StressMain - Global variables: {INDEX=${query-counter}}
[task3-threadpool] INFO StressMain - Query benchmarks for collection:
ecommerce-events-1
[task3-threadpool] INFO org.apache.solr.benchmarks.BenchmarksMain - Starting
querying benchmarks...
Total queries: 1
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
waiting response from server at: http://localhost:50000/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:692)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at
org.apache.solr.benchmarks.BenchmarksMain.lambda$getQuerySupplier$0(BenchmarksMain.java:237)
at
org.apache.solr.benchmarks.ControlledExecutor.lambda$run$0(ControlledExecutor.java:75)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at
java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
... 9 more
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
waiting response from server at: http://localhost:50000/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:692)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at
org.apache.solr.benchmarks.BenchmarksMain.lambda$getQuerySupplier$0(BenchmarksMain.java:237)
at
org.apache.solr.benchmarks.ControlledExecutor.lambda$run$0(ControlledExecutor.java:75)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
{code}
indicating failure. Solr nodes are in
SolrNightlyBenchmarksWorkDirectory/RunDirectory/ for checking logs.
was (Author: ichattopadhyaya):
The test mentioned in SOLR-15840 fails with Solr 8.10 (after about 1 hour of
running), but passes on jira/solr-16129 branch (built from your patch). I've
started the run on main branch, and will report back the findings.
> Solr specific InputStreamResponseListener to prevent client threads from
> hanging forever
> ----------------------------------------------------------------------------------------
>
> Key: SOLR-16129
> URL: https://issues.apache.org/jira/browse/SOLR-16129
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Chris M. Hostetter
> Assignee: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-16129.patch
>
>
> This issue tracks the implementation of workaround I suggested for SOLR-16099
> - it does not _fix_ the underlying bug (which as of this writting doesn't
> have an identified root cause) but it does ensure that client threads which
> encounter the bug won't hang forever...
> {quote}One thing we may want to consider (in Solr) is replacing our usage of
> {{InputStreamResponseListener}} with a variant implementation that uses a
> "timeout" instead of an unlimited {{wait()}} (along the lines of a [spin-off
> jetty enhancement issue|https://github.com/eclipse/jetty.project/issues/7259]
> one of the jetty devs filed). We could probably (with some effort) tweak the
> impacted Solr APIs to propogate the (remaining) {{timeAllowed}} (if that
> option was specified) down to this class – and/or have an "extreme" default
> (ie: 30min) just to prevent threads from sticking around forever.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
