The solution is to move to cursors, but you may as a safety net try to apply the RequestSanitizerComponent to disallow large offsets, see https://github.com/cominvent/request-sanitizer-component <https://github.com/cominvent/request-sanitizer-component>
-- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 21. aug. 2018 kl. 11:08 skrev Ere Maijala <ere.maij...@helsinki.fi>: > > Hi, > > Just my short comment here. It's difficult to say for someone else, but we > identified deep paging as the definite reason for running out of memory or at > least grinding to semi-halt because of long stop-the-world garbage collection > pauses in an application running on a similar SolrCloud. You can often get > away without issues as long as you only have a single shard, but for the > reason Erick mentioned deep paging in a sharded index is a heavy operation. > > Regards, > Ere > > Ash Ramesh kirjoitti 21.8.2018 klo 8.09: >> Hi Erick, >> Sorry I phrased that the wrong way. I meant to ask whether there is a high >> probability that that could be the correlated cause for the issue. Do you >> know why Solr itself isn't able to recover or is that to be expected with >> allowing such deep pagination. We are going to be removing it going >> forwards, but want to make sure that we find the root cause. >> Appreciate your help as always :) >> Ash >> On Tue, Aug 21, 2018 at 2:59 PM Erick Erickson <erickerick...@gmail.com> >> wrote: >>> Did the large offsets _definitely_ cause the OOM? How do you expect >>> that to be answerable? It's likely though. To return rows 1,000,000 >>> through 1,000,010 the system has to keep a list of 1,000,010 top >>> documents. It has to be this way because you don't know (and can't >>> guess) the score or a doc prior to, well, scoring it. And these very >>> large structures are kept for every query being processed. Not only >>> will that chew up memory, it'll chew up CPU cycles as well as this an >>> ordered list. >>> >>> This is an anti-pattern, cursors were invented because this pattern is >>> very costly (as you're finding out). >>> >>> Further, 4G isn't very much memory by modern standards. >>> >>> So it's very likely (but not guaranteed) that using cursors will fix >>> this problem. >>> >>> Best, >>> Erick >>> >>> >>> >>> On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh <ash...@canva.com> wrote: >>>> Hi everyone, >>>> >>>> We ran into an issue yesterday where all our ec2 machines, running solr, >>>> ran out of memory and could not heal themselves. I'll try break down what >>>> happened here. >>>> >>>> *System Architecture:* >>>> >>>> - Solr Version: 7.3.1 >>>> - Replica Types: TLOG/PULL >>>> - Num Shards: 8 (default hashing mechanism) >>>> - Doc Count: > 20m >>>> - Index Size: 17G >>>> - EC2 Machine Spec: 16 Core | 32G ram | 100G SSD >>>> - Num EC2 Machines: 7+ (scales up and down) >>>> - Max Shards per node (one node per EC2 instance): 8 (some nodes had 4, >>>> some had 8) >>>> - Num TLOG shard replicas: 3 (3 copies of each shard as TLOG) >>>> - Num PULL shard replicas: 3+ >>>> - Heap: 4G >>>> >>>> *What was run prior to the issue:* >>>> >>>> We ran these queries around 2.55pm >>>> >>>> We ran a bunch of deep paginated queries (offset of 1,000,000) with a >>>> filter query. We set the timeout to 5 seconds and it did timeout. We >>> aren't >>>> sure if this is what caused the irrecoverable failure, but by reading >>> this >>>> - >>>> >>> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging >>>> , we feel that this was the cause. >>>> >>>> We did not use a cursor. >>>> >>>> This cluster was healthy for about 1 week, but we noticed the degradation >>>> soon after (within 30min) of running the offset queries mentioned above. >>> We >>>> currently use a single sharded collection in production, however are >>>> transitioning to an 8 shard cluster. We hit this issue in a controlled 8 >>>> sharded environment, but don't notice any issues on our production >>> (single >>>> sharded) cluster. On production the query still timed out (with same num >>>> docs etc.) but didn't go into a crazy state. >>>> >>>> *What Happened:* >>>> >>>> - All the EC2 instances started logging OOM error. None of the nodes were >>>> responsive to new requests. >>>> - We saw that the Heap usage jumped from an average of 2.7G to the max of >>>> 4G within a 5 minute window. >>>> - CPU across all 16 cores was at 100% >>>> - We saw that the distributed requests were timing out across all >>> machines. >>>> - We shutdown all the machines that only had PULL replicas on them and it >>>> still didn't 'fix' itself. >>>> - Eventually we shut down SOLR on the main node which had all the master >>>> TLOG replicas. Once restarted, the machine started working again. >>>> >>>> >>>> *Questions:* >>>> - Did this deep pagination query *DEFINITELY* cause this issue? >>>> - Is each node single threaded? I don't think so, but I'd like to confirm >>>> that. >>>> - Is there any configuration that we could use to avoid this in the >>> future? >>>> - Why could the nodes not recover by themselves? When we ran the same >>> query >>>> on the single shard cluster it failed and didn't spin out of control. >>>> >>>> Thanks for all your help, Logs are pasted below from different >>> timestamps. >>>> >>>> Regards, >>>> Ash >>>> >>>> *Logs:* >>>> >>>> Here are some logs we collected. Not sure if it tells a lot outside of >>> what >>>> we know. >>>> >>>> *Time: 2.55pm ~ Requests are failing to complete in time* >>>> >>>>> ERROR RequestHandlerBase org.apache.solr.common.SolrException: >>>>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle >>>>> this request exceeded:[ >>>>> http://10.0.9.204:8983/solr/media_shard1_replica_p57, >>>>> http://10.0.9.204:8983/solr/media_shard4_replica_p80, >>>>> http://10.0.9.204:8983/solr/media_shard3_replica_p73, >>>>> http://10.0.9.204:8983/solr/media_shard2_replica_p68] >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.s... >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: ERROR RequestHandlerBase org.apache.solr.common.SolrException: >>>>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle >>>>> this request exceeded:[ >>>>> http://10.0.9.204:8983/solr/media_shard1_replica_p57, >>>>> http://10.0.9.204:8983/solr/media_shard4_replica_p80, >>>>> http://10.0.9.204:8983/solr/media_shard3_replica_p73, >>>>> http://10.0.9.204:8983/solr/media_shard2_replica_p68] >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> ERROR RequestHandlerBase org.apache.solr.common.SolrException: >>>>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle >>>>> this request exceeded:[ >>>>> http://10.0.9.204:8983/solr/media_shard1_replica_p57, >>>>> http://10.0.9.204:8983/solr/media_shard4_replica_p80, >>>>> http://10.0.9.204:8983/solr/media_shard3_replica_p73, >>>>> http://10.0.9.204:8983/solr/media_shard2_replica_p68] >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.s... >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> ERROR HttpSolrCall null:org.apache.solr.common.SolrException: >>>>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle >>>>> this request exceeded:[ >>>>> http://10.0.9.204:8983/solr/media_shard1_replica_p57, >>>>> http://10.0.9.204:8983/solr/media_shard4_replica_p80, >>>>> http://10.0.9.204:8983/solr/media_shard3_replica_p73, >>>>> http://10.0.9.204:8983/solr/media_shard2_replica_p68] >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>> >>>> >>>> *Time: 3.00pm ~ Seeing out of memory* >>>> >>>>> The request took too long to iterate over terms. Timeout: timeoutAt: >>>>> 440824897486723 (System.nanoTime(): 440825204861492), >>>>> TermsEnum=org.apache.lucene.codecs.blocktree.SegmentTermsEnum@3a9d2efc >>>>> solr: ERROR RequestHandlerBase java.lang.ClassCastException: >>>>> org.apache.solr.response.BasicResultContext cannot be cast to >>>>> org.apache.solr.common.SolrDocumentList >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:315) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: ERROR HttpSolrCall null:java.lang.ClassCastException: >>>>> org.apache.solr.response.BasicResultContext cannot be cast to >>>>> org.apache.solr.common.SolrDocumentList >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:315) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>> >>>> >>>> *3.15~ Health checks failing, Nodes are unresponsive.* >>>> >>>>> WARN DistributedUpdateProcessor Error sending update to >>>>> http://10.0.9.47:8983/solr >>>>> Failed to connect to Solr. May not be online yet. Retrying in 0 >>> seconds... >>>>> ERROR HttpSolrCall null:org.apache.solr.common.SolrException: Error >>>>> trying to proxy request for url: >>>>> http://10.0.10.117:8983/solr/media/admin/ping >>>>> #011at >>>>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:647) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:501) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at o... >>>>> ERROR RequestHandlerBase org.apache.solr.common.SolrException: >>>>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle >>>>> this request exceeded >>>>> #011at >>>>> >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) >>>>> #011at org.apache.solr.... >>>>> RROR ErrorReportingConcurrentUpdateSolrClient error >>>>> solr: ERROR RequestHandlerBase java.io.IOException: >>>>> java.util.concurrent.TimeoutException: Idle timeout expired: >>> 120394/120000 >>>>> ms >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1069) >>>>> #011at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:309) >>>>> #011at >>>>> >>> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100) >>>>> #011at >>>>> >>> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79) >>>>> #011at >>>>> >>> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88) >>>>> #011at >>>>> >>> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60) >>>>> #011at >>>>> >>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107) >>>>> #011at >>>>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55) >>>>> #011at >>>>> >>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) >>>>> #011at >>>>> >>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) >>>>> #011at >>>>> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) >>>>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) >>>>> #011at >>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: >>>>> 120394/120000 ms >>>>> #011at >>>>> org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) >>>>> #011at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) >>>>> #011at >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> #011... 1 more >>>>> #011Suppressed: java.lang.Throwable: HttpInput failure >>>>> #011#011at >>> org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:816) >>>>> #011#011at >>>>> >>> org.eclipse.jetty.server.HttpConnection$BlockingReadCallback.failed(HttpConnection.java:639) >>>>> #011#011at org.eclipse.jetty.io >>> .FillInterest.onFail(FillInterest.java:134) >>>>> #011#011at >>>>> org.eclipse.jetty.io >>> .AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:398) >>>>> #011#011... 8 more >>>>> solr: ERROR HttpSolrCall null:org.apache.solr.common.SolrException: >>> Error >>>>> trying to proxy request for url: >>>>> http://10.0.10.117:8983/solr/media/admin/ping >>>>> #011at >>>>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:647) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:501) >>>>> ERROR HttpSolrCall null:org.apache.solr.common.SolrException: Error >>>>> trying to proxy request for url: >>>>> http://10.0.10.117:8983/solr/media/admin/ping >>>>> #011at >>>>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:647) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:501) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(Servl... >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at >>>>> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >>>>> #011at org.eclipse.jetty.server.Server.handle(Server.java:530) >>>>> #011at >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >>>>> #011at >>>>> org.eclipse.jetty.io >>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >>>>> #011at org.eclipse.jetty.io >>> .FillInterest.fillable(FillInterest.java:102) >>>>> #011at >>>>> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >>>>> #011at >>>>> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to >>>>> 10.0.10.117:8983 [/10.0.10.117] failed: connect timed out >>>>> #011at >>>>> >>> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) >>>>> #011at >>>>> >>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >>>>> #011at >>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) >>>>> #011at >>>>> >>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >>>>> #011at >>>>> >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >>>>> #011at >>>>> >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >>>>> #011at >>>>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:619) >>>> >>>> >>>> *3.40~ Cannot do updates. Everything is broke :(* >>>> >>>>> #011at >>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:664) >>>>> #011at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:532) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) >>>>> #011at >>>>> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) >>>>> #011at >>>>> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >>>>> #011at >>>>> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> #011at >>>>> >>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) >>>>> #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Ha... >>>>> solr: WARN DistributedUpdateProcessor Error sending update to >>>>> http://10.0.8.157:8983/solr >>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: >>>>> Error from server at >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35: >>>>> Server Error >>>>> ERROR DistributedUpdateProcessor Setting up to try to start recovery on >>>>> replica core_node36 with url >>>>> http://10.0.8.157:8983/solr/media_shard6_replica_t35/ by increasing >>>>> leader term >>>>> message repeated 2 times: [] >>>>> request: >>>>> >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F10.0.10.117%3A8983%2Fsolr%2Fmedia_shard6_replica_t10%2F&wt=javabin&version=2 >>>>> Remote error message: java.util.concurrent.TimeoutException: Idle >>> timeout >>>>> expired: 120485/120000 ms >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:383) >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:182) >>>>> #011at >>>>> >>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >>>>> #011at >>>>> >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: ERROR DistributedUpdateProcessor Setting up to try to start >>>>> recovery on replica core_node36 with url >>>>> http://10.0.8.157:8983/solr/media_shard6_replica_t35/ by increasing >>>>> leader term >>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: >>>>> Error from server at >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35: >>>>> Server Error >>>>> message repeated 2 times: [] >>>>> request: >>>>> >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F10.0.10.117%3A8983%2Fsolr%2Fmedia_shard6_replica_t10%2F&wt=javabin&version=2 >>>>> Remote error message: java.util.concurrent.TimeoutException: Idle >>> timeout >>>>> expired: 120485/120000 ms >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:383) >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:182) >>>>> #011at >>>>> >>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >>>>> #011at >>>>> >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: WARN DistributedUpdateProcessor Error sending update to >>>>> http://10.0.8.157:8983/solr >>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: >>>>> Error from server at >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35: >>>>> Server Error >>>>> message repeated 2 times: [] >>>>> request: >>>>> >>> http://10.0.8.157:8983/solr/media_shard6_replica_t35/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F10.0.10.117%3A8983%2Fsolr%2Fmedia_shard6_replica_t10%2F&wt=javabin&version=2 >>>>> Remote error message: java.util.concurrent.TimeoutException: Idle >>> timeout >>>>> expired: 120307/120000 ms >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:383) >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:182) >>>>> #011at >>>>> >>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >>>>> #011at >>>>> >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: WARN DistributedUpdateProcessor Error sending update to >>>>> http://10.0.9.47:8983/solr >>>>> org.apache.http.NoHttpResponseException: 10.0.9.47:8983 failed to >>> respond >>>>> #011at >>>>> >>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) >>>>> #011at >>>>> >>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) >>>>> #011at >>>>> org.apache.http.impl.io >>> .AbstractMessageParser.parse(AbstractMessageParser.java:259) >>>>> #011at >>>>> >>> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) >>>>> #011at >>>>> >>> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) >>>>> #011at >>>>> >>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) >>>>> #011at >>>>> >>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) >>>>> #011at >>>>> >>> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) >>>>> #011at >>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) >>>>> #011at >>>>> >>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) >>>>> #011at >>>>> >>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) >>>>> #011at >>>>> >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) >>>>> #011at >>>>> >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:347) >>>>> #011at >>>>> >>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:182) >>>>> #011at >>>>> >>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >>>>> #011at >>>>> >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) >>>>> #011at >>>>> >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> #011at java.base/java.lang.Thread.run(Thread.java:844) >>>>> solr: WARN DistributedUpdateProcessor Error sending update to >>>>> http://10.0.8.157:8983/solr >>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: >>>>> Error from server at >>> http://10.0.8.157:8983/solr/media_shard2_replica_t20: >>>>> Server Error >>>> >>>> -- >>>> *P.S. We've launched a new blog to share the latest ideas and case >>> studies >>>> from our team. Check it out here: product.canva.com >>>> <http://product.canva.com/>. *** >>>> ** <https://canva.com>Empowering the world >>>> to design >>>> Also, we're hiring. Apply here! >>>> <https://about.canva.com/careers/> >>>> <https://twitter.com/canva> >>>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva> >>>> <https://instagram.com/canva> >>>> >>>> >>>> >>>> >>>> >>> >>> > > -- > Ere Maijala > Kansalliskirjasto / The National Library of Finland >
