Node A receives batch of documents to index. It forwards documents to shards that are on the node B. Node B is having issues with GC so it takes a while to respond. Node A sees it as read timeout and reports it in logs. So the issue is on node B not node A.
Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> wrote: > > Yes, the nodes are well balanced. I am just using these boxes for indexing > the data and is not serving any traffic at this time. The error indicates > it is having issues errors on the shards that are hosted on the box and not > on the other box. > I will check GC logs to see if there were any issues. > thanks > > Thanks > Jay Potharaju > > > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > >> Hi Jay, >> My first guess would be that there was some major GC on other box so it >> did not respond on time. Are your nodes well balanced - do they serve equal >> amount of data? >> >> Thanks, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> wrote: >>> >>> Hi, >>> I am seeing the following lines in the error log. My setup has 2 nodes in >>> the solrcloud cluster, each node has 3 shards with no replication. From >> the >>> error log it seems like all the shards on this box are throwing async >>> exception errors. Other node in the cluster does not have any errors in >> the >>> logs. Any suggestions on how to tackle this error? >>> >>> Solr setup >>> Solr:6.6.3 >>> 2Nodes: 3 shards each >>> >>> >>> ERROR org.apache.solr.servlet.HttpSolrCall [test_shard3_replica1] ? >>> null:org.apache.solr.update.processor.DistributedUpdateProcessor$ >> DistributedUpdatesAsyncException: >>> Async exception during distributed update: Read timed out >>> at >>> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( >> DistributedUpdateProcessor.java:972) >>> at >>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish( >> DistributedUpdateProcessor.java:1911) >>> at >>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody( >> ContentStreamHandlerBase.java:78) >>> at >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( >> RequestHandlerBase.java:173) >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723) >>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) >>> at >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >> SolrDispatchFilter.java:361) >>> at >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >> SolrDispatchFilter.java:305) >>> at >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. >> doFilter(ServletHandler.java:1691) >>> at >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( >> ServletHandler.java:582) >>> at >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >> ScopedHandler.java:143) >>> at >>> org.eclipse.jetty.security.SecurityHandler.handle( >> SecurityHandler.java:548) >>> at >>> org.eclipse.jetty.server.session.SessionHandler. >> doHandle(SessionHandler.java:226) >>> at >>> org.eclipse.jetty.server.handler.ContextHandler. >> doHandle(ContextHandler.java:1180) >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( >> ServletHandler.java:512) >>> at >>> org.eclipse.jetty.server.session.SessionHandler. >> doScope(SessionHandler.java:185) >>> at >>> org.eclipse.jetty.server.handler.ContextHandler. >> doScope(ContextHandler.java:1112) >>> at >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >> ScopedHandler.java:141) >>> at >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( >> ContextHandlerCollection.java:213) >>> at >>> org.eclipse.jetty.server.handler.HandlerCollection. >> handle(HandlerCollection.java:119) >>> at >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >> HandlerWrapper.java:134) >>> at >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( >> RewriteHandler.java:335) >>> at >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >> HandlerWrapper.java:134) >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) >>> at >>> org.eclipse.jetty.server.HttpConnection.onFillable( >> HttpConnection.java:251) >>> at >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( >> AbstractConnection.java:273) >>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) >>> at >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( >> SelectChannelEndPoint.java:93) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( >> QueuedThreadPool.java:671) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( >> QueuedThreadPool.java:589) >>> at java.lang.Thread.run(Unknown Source) >>> >>> >>> Thanks >>> Jay >> >>
