How do you send documents? Large batches? Complex analysis? Do you send all batches to the same node? How do you commit? Do you delete by query while indexing?
Emir On Tue, May 8, 2018, 12:30 AM Jay Potharaju <jspothar...@gmail.com> wrote: > I didn't see any OOM errors in the logs on either of the nodes. I saw GC > pause of 1 second on the box that was throwing error ...but nothing on the > other node. Any other recommendations? > Thanks > > > Thanks > Jay Potharaju > > > On Mon, May 7, 2018 at 9:48 AM, Jay Potharaju <jspothar...@gmail.com> > wrote: > > > Ah thanks for explaining that! > > > > Thanks > > Jay Potharaju > > > > > > On Mon, May 7, 2018 at 9:45 AM, Emir Arnautović < > > emir.arnauto...@sematext.com> wrote: > > > >> Node A receives batch of documents to index. It forwards documents to > >> shards that are on the node B. Node B is having issues with GC so it > takes > >> a while to respond. Node A sees it as read timeout and reports it in > logs. > >> So the issue is on node B not node A. > >> > >> Emir > >> -- > >> Monitoring - Log Management - Alerting - Anomaly Detection > >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > >> > >> > >> > >> > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> wrote: > >> > > >> > Yes, the nodes are well balanced. I am just using these boxes for > >> indexing > >> > the data and is not serving any traffic at this time. The error > >> indicates > >> > it is having issues errors on the shards that are hosted on the box > and > >> not > >> > on the other box. > >> > I will check GC logs to see if there were any issues. > >> > thanks > >> > > >> > Thanks > >> > Jay Potharaju > >> > > >> > > >> > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < > >> > emir.arnauto...@sematext.com> wrote: > >> > > >> >> Hi Jay, > >> >> My first guess would be that there was some major GC on other box so > it > >> >> did not respond on time. Are your nodes well balanced - do they serve > >> equal > >> >> amount of data? > >> >> > >> >> Thanks, > >> >> Emir > >> >> -- > >> >> Monitoring - Log Management - Alerting - Anomaly Detection > >> >> Solr & Elasticsearch Consulting Support Training - > >> http://sematext.com/ > >> >> > >> >> > >> >> > >> >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> > wrote: > >> >>> > >> >>> Hi, > >> >>> I am seeing the following lines in the error log. My setup has 2 > >> nodes in > >> >>> the solrcloud cluster, each node has 3 shards with no replication. > >> From > >> >> the > >> >>> error log it seems like all the shards on this box are throwing > async > >> >>> exception errors. Other node in the cluster does not have any errors > >> in > >> >> the > >> >>> logs. Any suggestions on how to tackle this error? > >> >>> > >> >>> Solr setup > >> >>> Solr:6.6.3 > >> >>> 2Nodes: 3 shards each > >> >>> > >> >>> > >> >>> ERROR org.apache.solr.servlet.HttpSolrCall [test_shard3_replica1] ? > >> >>> null:org.apache.solr.update.processor.DistributedUpdateProcessor$ > >> >> DistributedUpdatesAsyncException: > >> >>> Async exception during distributed update: Read timed out > >> >>> at > >> >>> > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( > >> >> DistributedUpdateProcessor.java:972) > >> >>> at > >> >>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish( > >> >> DistributedUpdateProcessor.java:1911) > >> >>> at > >> >>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody( > >> >> ContentStreamHandlerBase.java:78) > >> >>> at > >> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( > >> >> RequestHandlerBase.java:173) > >> >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) > >> >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall. > >> java:723) > >> >>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) > >> >>> at > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> >> SolrDispatchFilter.java:361) > >> >>> at > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> >> SolrDispatchFilter.java:305) > >> >>> at > >> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. > >> >> doFilter(ServletHandler.java:1691) > >> >>> at > >> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( > >> >> ServletHandler.java:582) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> >> ScopedHandler.java:143) > >> >>> at > >> >>> org.eclipse.jetty.security.SecurityHandler.handle( > >> >> SecurityHandler.java:548) > >> >>> at > >> >>> org.eclipse.jetty.server.session.SessionHandler. > >> >> doHandle(SessionHandler.java:226) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > >> >> doHandle(ContextHandler.java:1180) > >> >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( > >> >> ServletHandler.java:512) > >> >>> at > >> >>> org.eclipse.jetty.server.session.SessionHandler. > >> >> doScope(SessionHandler.java:185) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > >> >> doScope(ContextHandler.java:1112) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> >> ScopedHandler.java:141) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > >> >> ContextHandlerCollection.java:213) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.HandlerCollection. > >> >> handle(HandlerCollection.java:119) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> >> HandlerWrapper.java:134) > >> >>> at > >> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( > >> >> RewriteHandler.java:335) > >> >>> at > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> >> HandlerWrapper.java:134) > >> >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) > >> >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > >> >>> at > >> >>> org.eclipse.jetty.server.HttpConnection.onFillable( > >> >> HttpConnection.java:251) > >> >>> at > >> >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > >> >> AbstractConnection.java:273) > >> >>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > >> >>> at > >> >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > >> >> SelectChannelEndPoint.java:93) > >> >>> at > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > >> >> QueuedThreadPool.java:671) > >> >>> at > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > >> >> QueuedThreadPool.java:589) > >> >>> at java.lang.Thread.run(Unknown Source) > >> >>> > >> >>> > >> >>> Thanks > >> >>> Jay > >> >> > >> >> > >> > >> > > >
