How many concurrent updates can be sent? Do you always send updates to the same node? Do you use solrj?
Emir On Tue, May 8, 2018, 1:02 AM Jay Potharaju <jspothar...@gmail.com> wrote: > The updates are pushed in real time not batched. No complex analysis and > everything is committed using autocommit settings in solr. > > Thanks > Jay Potharaju > > > On Mon, May 7, 2018 at 4:00 PM, Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > > > How do you send documents? Large batches? Complex analysis? Do you send > all > > batches to the same node? How do you commit? Do you delete by query while > > indexing? > > > > Emir > > > > On Tue, May 8, 2018, 12:30 AM Jay Potharaju <jspothar...@gmail.com> > wrote: > > > > > I didn't see any OOM errors in the logs on either of the nodes. I saw > GC > > > pause of 1 second on the box that was throwing error ...but nothing on > > the > > > other node. Any other recommendations? > > > Thanks > > > > > > > > > Thanks > > > Jay Potharaju > > > > > > > > > On Mon, May 7, 2018 at 9:48 AM, Jay Potharaju <jspothar...@gmail.com> > > > wrote: > > > > > > > Ah thanks for explaining that! > > > > > > > > Thanks > > > > Jay Potharaju > > > > > > > > > > > > On Mon, May 7, 2018 at 9:45 AM, Emir Arnautović < > > > > emir.arnauto...@sematext.com> wrote: > > > > > > > >> Node A receives batch of documents to index. It forwards documents > to > > > >> shards that are on the node B. Node B is having issues with GC so it > > > takes > > > >> a while to respond. Node A sees it as read timeout and reports it in > > > logs. > > > >> So the issue is on node B not node A. > > > >> > > > >> Emir > > > >> -- > > > >> Monitoring - Log Management - Alerting - Anomaly Detection > > > >> Solr & Elasticsearch Consulting Support Training - > > http://sematext.com/ > > > >> > > > >> > > > >> > > > >> > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> > > wrote: > > > >> > > > > >> > Yes, the nodes are well balanced. I am just using these boxes for > > > >> indexing > > > >> > the data and is not serving any traffic at this time. The error > > > >> indicates > > > >> > it is having issues errors on the shards that are hosted on the > box > > > and > > > >> not > > > >> > on the other box. > > > >> > I will check GC logs to see if there were any issues. > > > >> > thanks > > > >> > > > > >> > Thanks > > > >> > Jay Potharaju > > > >> > > > > >> > > > > >> > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < > > > >> > emir.arnauto...@sematext.com> wrote: > > > >> > > > > >> >> Hi Jay, > > > >> >> My first guess would be that there was some major GC on other box > > so > > > it > > > >> >> did not respond on time. Are your nodes well balanced - do they > > serve > > > >> equal > > > >> >> amount of data? > > > >> >> > > > >> >> Thanks, > > > >> >> Emir > > > >> >> -- > > > >> >> Monitoring - Log Management - Alerting - Anomaly Detection > > > >> >> Solr & Elasticsearch Consulting Support Training - > > > >> http://sematext.com/ > > > >> >> > > > >> >> > > > >> >> > > > >> >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> > > > wrote: > > > >> >>> > > > >> >>> Hi, > > > >> >>> I am seeing the following lines in the error log. My setup has 2 > > > >> nodes in > > > >> >>> the solrcloud cluster, each node has 3 shards with no > replication. > > > >> From > > > >> >> the > > > >> >>> error log it seems like all the shards on this box are throwing > > > async > > > >> >>> exception errors. Other node in the cluster does not have any > > errors > > > >> in > > > >> >> the > > > >> >>> logs. Any suggestions on how to tackle this error? > > > >> >>> > > > >> >>> Solr setup > > > >> >>> Solr:6.6.3 > > > >> >>> 2Nodes: 3 shards each > > > >> >>> > > > >> >>> > > > >> >>> ERROR org.apache.solr.servlet.HttpSolrCall > > [test_shard3_replica1] ? > > > >> >>> > null:org.apache.solr.update.processor.DistributedUpdateProcessor$ > > > >> >> DistributedUpdatesAsyncException: > > > >> >>> Async exception during distributed update: Read timed out > > > >> >>> at > > > >> >>> > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( > > > >> >> DistributedUpdateProcessor.java:972) > > > >> >>> at > > > >> >>> org.apache.solr.update.processor.DistributedUpdateProcessor. > > finish( > > > >> >> DistributedUpdateProcessor.java:1911) > > > >> >>> at > > > >> >>> org.apache.solr.handler.ContentStreamHandlerBase. > > handleRequestBody( > > > >> >> ContentStreamHandlerBase.java:78) > > > >> >>> at > > > >> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( > > > >> >> RequestHandlerBase.java:173) > > > >> >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) > > > >> >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall. > > > >> java:723) > > > >> >>> at org.apache.solr.servlet.HttpSolrCall.call( > > HttpSolrCall.java:529) > > > >> >>> at > > > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > > > >> >> SolrDispatchFilter.java:361) > > > >> >>> at > > > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > > > >> >> SolrDispatchFilter.java:305) > > > >> >>> at > > > >> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. > > > >> >> doFilter(ServletHandler.java:1691) > > > >> >>> at > > > >> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( > > > >> >> ServletHandler.java:582) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > > > >> >> ScopedHandler.java:143) > > > >> >>> at > > > >> >>> org.eclipse.jetty.security.SecurityHandler.handle( > > > >> >> SecurityHandler.java:548) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.session.SessionHandler. > > > >> >> doHandle(SessionHandler.java:226) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > > > >> >> doHandle(ContextHandler.java:1180) > > > >> >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( > > > >> >> ServletHandler.java:512) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.session.SessionHandler. > > > >> >> doScope(SessionHandler.java:185) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.ContextHandler. > > > >> >> doScope(ContextHandler.java:1112) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > > > >> >> ScopedHandler.java:141) > > > >> >>> at > > > >> >>> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > > > >> >> ContextHandlerCollection.java:213) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.HandlerCollection. > > > >> >> handle(HandlerCollection.java:119) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > > >> >> HandlerWrapper.java:134) > > > >> >>> at > > > >> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( > > > >> >> RewriteHandler.java:335) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > > >> >> HandlerWrapper.java:134) > > > >> >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) > > > >> >>> at org.eclipse.jetty.server.HttpChannel.handle( > > HttpChannel.java:320) > > > >> >>> at > > > >> >>> org.eclipse.jetty.server.HttpConnection.onFillable( > > > >> >> HttpConnection.java:251) > > > >> >>> at > > > >> >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > > > >> >> AbstractConnection.java:273) > > > >> >>> at org.eclipse.jetty.io.FillInterest.fillable( > > FillInterest.java:95) > > > >> >>> at > > > >> >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > > > >> >> SelectChannelEndPoint.java:93) > > > >> >>> at > > > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > > > >> >> QueuedThreadPool.java:671) > > > >> >>> at > > > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > > > >> >> QueuedThreadPool.java:589) > > > >> >>> at java.lang.Thread.run(Unknown Source) > > > >> >>> > > > >> >>> > > > >> >>> Thanks > > > >> >>> Jay > > > >> >> > > > >> >> > > > >> > > > >> > > > > > > > > > >
