Okay, that should solve the hung threads on the leader. When you stop the jetty service then it is a graceful shutdown where existing requests finish before the searcher thread pool is shutdown completely. A EC2 terminate probably just kills the processes and leader threads just wait due to a lack of read/connection timeouts.
The Solr leader should stop sending requests to the stopped replica once that replica's live node is removed from ZK (after session expiry). I think most of these issues are because of the lack of timeouts. Just add them and if there are more problems, we can discuss more. On Fri, Dec 12, 2014 at 8:14 PM, Peter Keegan <peterlkee...@gmail.com> wrote: > No, I wasn't aware of these. I will give that a try. If I stop the Solr > jetty service manually, things recover fine, but the hang occurs when I > 'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a > 15-sec timeout from the stopped node, and expires the session, but the Solr > leader never gets notified. This seems like a bug in ZK. > > Thanks, > Peter > > > On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > > > Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to > > reasonable values in your solr.xml? These are the timeouts used for > > inter-shard update requests. > > > > On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <peterlkee...@gmail.com> > > wrote: > > > > > We are running SolrCloud in AWS and using their auto scaling groups to > > spin > > > up new Solr replicas when CPU utilization exceeds a threshold for a > > period > > > of time. All is well until the replicas are terminated when CPU > > utilization > > > falls below another threshold. What happens is that index updates sent > to > > > the Solr leader hang forever in both the Solr leader and the SolrJ > client > > > app. Searches work fine. Here are 2 thread stack traces from the Solr > > > leader and 2 from the client app: > > > > > > 1) Solr-leader thread doing a distributed commit: > > > > > > Thread 23527: (state = IN_NATIVE) > > > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, > byte[], > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise) > > > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, > > line=150 > > > (Compiled frame) > > > - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 > > > (Compiled frame) > > > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() > > @bci=71, > > > line=166 (Compiled frame) > > > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, > line=90 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) > > > @bci=137, line=281 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=16, line=92 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=2, line=61 (Compiled frame) > > > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, > > line=254 > > > (Compiled frame) > > > - > > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() > > > @bci=8, line=289 (Compiled frame) > > > - > > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() > > > @bci=1, line=252 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() > > > @bci=6, line=191 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=62, line=300 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=60, line=127 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, > > > org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=574, line=520 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=344, line=906 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) > > > @bci=6, line=784 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, > > > org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395 > > > (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=17, line=199 (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=101, line=293 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req) > > > @bci=127, line=226 (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand, > > > java.util.List, org.apache.solr.common.params.ModifiableSolrParams) > > > @bci=112, line=195 (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand) > > > @bci=174, line=1250 (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand) > > > @bci=61, line=157 (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.update.processor.UpdateRequestProcessor, > > > org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69 > > > (Interpreted frame) > > > - > > > > > > > > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=60, line=68 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler, > > > org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 > (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest, > > > org.apache.solr.request.SolrRequestHandler, > > > org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean) > > > @bci=1380, line=417 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, > > line=201 > > > (Compiled frame) > > > - > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame) > > > - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled > > frame) > > > - > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled > > frame) > > > - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled > > > frame) > > > - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled > > frame) > > > - > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled > > frame) > > > - > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled > > > frame) > > > - > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled > > frame) > > > - > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled > > frame) > > > - > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled > > frame) > > > - > > > > > > > > > org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) > > > @bci=146, line=368 (Compiled frame) > > > - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() > > > @bci=323, line=489 (Compiled frame) > > > - org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() > > @bci=1, > > > line=53 (Interpreted frame) > > > - > > > > > > > > > org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer) > > > @bci=13, line=953 (Interpreted frame) > > > - > > > > > > > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer) > > > @bci=5, line=1014 (Interpreted frame) > > > - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921 > > > (Compiled frame) > > > - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240 > > > (Compiled frame) > > > - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51, > > > line=72 (Compiled frame) > > > - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run() > > > @bci=100, line=264 (Compiled frame) > > > - > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable) > > > @bci=1, line=608 (Interpreted frame) > > > - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, > > line=543 > > > (Compiled frame) > > > - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame) > > > > > > 2) Solr-leader thread doing a delete by query: > > > > > > Thread 19580: (state = BLOCKED) > > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > > information > > > may be imprecise) > > > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) > @bci=14, > > > line=186 (Compiled frame) > > > - > > > > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() > > > @bci=1, line=834 (Compiled frame) > > > - > > > > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, > > > int) @bci=67, line=867 (Compiled frame) > > > - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) > > > @bci=17, line=1197 (Compiled frame) > > > - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, > > > line=214 (Compiled frame) > > > - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished() > > > @bci=81, line=391 (Compiled frame) > > > - org.apache.solr.update.StreamingSolrServers.blockUntilFinished() > > > @bci=35, line=98 (Compiled frame) > > > - org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand) > > > @bci=1088, line=1088 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand) > > > @bci=14, line=825 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand) > > > @bci=61, line=121 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.client.solrj.request.UpdateRequest, > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208, > > line=150 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse, java.io.InputStream, > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68, > > line=111 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse, > > > org.apache.solr.common.util.ContentStream, > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18, > line=58 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse, > > > org.apache.solr.common.util.ContentStream, > > > org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165, > > line=92 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=129, line=74 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler, > > > org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 > (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest, > > > org.apache.solr.request.SolrRequestHandler, > > > org.apache.solr.request.SolrQueryRequest, > > > org.apache.solr.response.SolrQueryResponse) @bci=31, line=721 (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean) > > > @bci=1380, line=417 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, > > line=201 > > > (Compiled frame) > > > - > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, > > > javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame) > > > - org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled > > frame) > > > - > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled > > frame) > > > - org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled > > > frame) > > > - org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled > > frame) > > > - > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled > > frame) > > > - > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled > > > frame) > > > - > > org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled > > frame) > > > - > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled > > frame) > > > - > > > > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled > > frame) > > > - > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String, > > > org.eclipse.jetty.server.Request, > javax.servlet.http.HttpServletRequest, > > > javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled > > frame) > > > - > > > > > > > > > org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection) > > > @bci=146, line=368 (Compiled frame) > > > - org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() > > > @bci=323, line=489 (Compiled frame) > > > - org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866 > > > (Compiled frame) > > > - org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240 > > > (Compiled frame) > > > - org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51, > > > line=72 (Compiled frame) > > > - org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run() > > > @bci=100, line=264 (Compiled frame) > > > - > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable) > > > @bci=1, line=608 (Interpreted frame) > > > - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, > > line=543 > > > (Compiled frame) > > > - java.lang.Thread.run() @bci=11, line=724 (Interpreted frame) > > > > > > 3) SolrJ client thread doing commit: > > > > > > Thread 1872: (state = IN_NATIVE) > > > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, > byte[], > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise) > > > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, > > line=150 > > > (Compiled frame) > > > - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 > > > (Compiled frame) > > > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() > > @bci=71, > > > line=166 (Compiled frame) > > > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, > line=90 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) > > > @bci=137, line=281 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) > > > @bci=5, line=115 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=16, line=92 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=2, line=62 (Compiled frame) > > > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, > > line=254 > > > (Compiled frame) > > > - > > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() > > > @bci=8, line=289 (Compiled frame) > > > - > > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() > > > @bci=1, line=252 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() > > > @bci=6, line=191 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=73, line=301 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=60, line=127 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, > > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=597, line=522 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=344, line=906 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) > > > @bci=6, line=784 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, > > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 > > (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=17, line=199 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req) > > > @bci=132, line=285 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=838, line=640 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer) > > > @bci=17, line=117 (Interpreted frame) > > > - org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean) > > > @bci=16, line=168 (Interpreted frame) > > > - org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146 > > > (Interpreted frame) > > > > > > 4) SolrJ client thread doing delete by query: > > > > > > Thread 1873: (state = IN_NATIVE) > > > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, > byte[], > > > int, int, int) @bci=0 (Compiled frame; information may be imprecise) > > > - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, > > line=150 > > > (Compiled frame) > > > - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 > > > (Compiled frame) > > > - org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() > > @bci=71, > > > line=166 (Compiled frame) > > > - org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1, > line=90 > > > (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) > > > @bci=137, line=281 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer) > > > @bci=5, line=115 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=16, line=92 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) > > > @bci=2, line=62 (Compiled frame) > > > - org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, > > line=254 > > > (Compiled frame) > > > - > > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader() > > > @bci=8, line=289 (Compiled frame) > > > - > > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader() > > > @bci=1, line=252 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader() > > > @bci=6, line=191 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=73, line=301 (Compiled frame) > > > - > > > > > > > > > org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, > > > org.apache.http.HttpClientConnection, > > org.apache.http.protocol.HttpContext) > > > @bci=60, line=127 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest, > > > org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=597, line=522 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost, > > > org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) > > > @bci=344, line=906 (Compiled frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, > > > org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled > frame) > > > - > > > > > > > > > org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) > > > @bci=6, line=784 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest, > > > org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 > > (Compiled > > > frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=17, line=199 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req) > > > @bci=132, line=285 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest) > > > @bci=838, line=640 (Compiled frame) > > > - > > > > > > > > > org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer) > > > @bci=17, line=117 (Compiled frame) > > > - > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String, > > > int) @bci=21, line=285 (Compiled frame) > > > - > > org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String) > > > @bci=3, line=271 (Compiled frame) > > > > > > It looks as if the Solr-leader threads are waiting on a response from > the > > > terminated replicas, but I would have expected an IOException to occur. > > Any > > > ideas on what may be happening? > > > > > > Thanks, > > > Peter > > > > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > -- Regards, Shalin Shekhar Mangar.
