There are some deletes by query. I have not had any issues with DBQ, currently have 5.3 running in production.
Thanks Jay Potharaju On Mon, May 7, 2018 at 4:02 PM, Jay Potharaju <jspothar...@gmail.com> wrote: > The updates are pushed in real time not batched. No complex analysis and > everything is committed using autocommit settings in solr. > > Thanks > Jay Potharaju > > > On Mon, May 7, 2018 at 4:00 PM, Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > >> How do you send documents? Large batches? Complex analysis? Do you send >> all >> batches to the same node? How do you commit? Do you delete by query while >> indexing? >> >> Emir >> >> On Tue, May 8, 2018, 12:30 AM Jay Potharaju <jspothar...@gmail.com> >> wrote: >> >> > I didn't see any OOM errors in the logs on either of the nodes. I saw GC >> > pause of 1 second on the box that was throwing error ...but nothing on >> the >> > other node. Any other recommendations? >> > Thanks >> > >> > >> > Thanks >> > Jay Potharaju >> > >> > >> > On Mon, May 7, 2018 at 9:48 AM, Jay Potharaju <jspothar...@gmail.com> >> > wrote: >> > >> > > Ah thanks for explaining that! >> > > >> > > Thanks >> > > Jay Potharaju >> > > >> > > >> > > On Mon, May 7, 2018 at 9:45 AM, Emir Arnautović < >> > > emir.arnauto...@sematext.com> wrote: >> > > >> > >> Node A receives batch of documents to index. It forwards documents to >> > >> shards that are on the node B. Node B is having issues with GC so it >> > takes >> > >> a while to respond. Node A sees it as read timeout and reports it in >> > logs. >> > >> So the issue is on node B not node A. >> > >> >> > >> Emir >> > >> -- >> > >> Monitoring - Log Management - Alerting - Anomaly Detection >> > >> Solr & Elasticsearch Consulting Support Training - >> http://sematext.com/ >> > >> >> > >> >> > >> >> > >> > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> >> wrote: >> > >> > >> > >> > Yes, the nodes are well balanced. I am just using these boxes for >> > >> indexing >> > >> > the data and is not serving any traffic at this time. The error >> > >> indicates >> > >> > it is having issues errors on the shards that are hosted on the box >> > and >> > >> not >> > >> > on the other box. >> > >> > I will check GC logs to see if there were any issues. >> > >> > thanks >> > >> > >> > >> > Thanks >> > >> > Jay Potharaju >> > >> > >> > >> > >> > >> > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < >> > >> > emir.arnauto...@sematext.com> wrote: >> > >> > >> > >> >> Hi Jay, >> > >> >> My first guess would be that there was some major GC on other box >> so >> > it >> > >> >> did not respond on time. Are your nodes well balanced - do they >> serve >> > >> equal >> > >> >> amount of data? >> > >> >> >> > >> >> Thanks, >> > >> >> Emir >> > >> >> -- >> > >> >> Monitoring - Log Management - Alerting - Anomaly Detection >> > >> >> Solr & Elasticsearch Consulting Support Training - >> > >> http://sematext.com/ >> > >> >> >> > >> >> >> > >> >> >> > >> >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> >> > wrote: >> > >> >>> >> > >> >>> Hi, >> > >> >>> I am seeing the following lines in the error log. My setup has 2 >> > >> nodes in >> > >> >>> the solrcloud cluster, each node has 3 shards with no >> replication. >> > >> From >> > >> >> the >> > >> >>> error log it seems like all the shards on this box are throwing >> > async >> > >> >>> exception errors. Other node in the cluster does not have any >> errors >> > >> in >> > >> >> the >> > >> >>> logs. Any suggestions on how to tackle this error? >> > >> >>> >> > >> >>> Solr setup >> > >> >>> Solr:6.6.3 >> > >> >>> 2Nodes: 3 shards each >> > >> >>> >> > >> >>> >> > >> >>> ERROR org.apache.solr.servlet.HttpSolrCall >> [test_shard3_replica1] ? >> > >> >>> null:org.apache.solr.update.processor.DistributedUpdateProce >> ssor$ >> > >> >> DistributedUpdatesAsyncException: >> > >> >>> Async exception during distributed update: Read timed out >> > >> >>> at >> > >> >>> >> > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( >> > >> >> DistributedUpdateProcessor.java:972) >> > >> >>> at >> > >> >>> org.apache.solr.update.processor.DistributedUpdateProcessor. >> finish( >> > >> >> DistributedUpdateProcessor.java:1911) >> > >> >>> at >> > >> >>> org.apache.solr.handler.ContentStreamHandlerBase.handleReque >> stBody( >> > >> >> ContentStreamHandlerBase.java:78) >> > >> >>> at >> > >> >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( >> > >> >> RequestHandlerBase.java:173) >> > >> >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) >> > >> >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall. >> > >> java:723) >> > >> >>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java: >> 529) >> > >> >>> at >> > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >> > >> >> SolrDispatchFilter.java:361) >> > >> >>> at >> > >> >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >> > >> >> SolrDispatchFilter.java:305) >> > >> >>> at >> > >> >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. >> > >> >> doFilter(ServletHandler.java:1691) >> > >> >>> at >> > >> >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( >> > >> >> ServletHandler.java:582) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >> > >> >> ScopedHandler.java:143) >> > >> >>> at >> > >> >>> org.eclipse.jetty.security.SecurityHandler.handle( >> > >> >> SecurityHandler.java:548) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.session.SessionHandler. >> > >> >> doHandle(SessionHandler.java:226) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.ContextHandler. >> > >> >> doHandle(ContextHandler.java:1180) >> > >> >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( >> > >> >> ServletHandler.java:512) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.session.SessionHandler. >> > >> >> doScope(SessionHandler.java:185) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.ContextHandler. >> > >> >> doScope(ContextHandler.java:1112) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >> > >> >> ScopedHandler.java:141) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.ha >> ndle( >> > >> >> ContextHandlerCollection.java:213) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.HandlerCollection. >> > >> >> handle(HandlerCollection.java:119) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >> > >> >> HandlerWrapper.java:134) >> > >> >>> at >> > >> >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( >> > >> >> RewriteHandler.java:335) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >> > >> >> HandlerWrapper.java:134) >> > >> >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) >> > >> >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel. >> java:320) >> > >> >>> at >> > >> >>> org.eclipse.jetty.server.HttpConnection.onFillable( >> > >> >> HttpConnection.java:251) >> > >> >>> at >> > >> >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( >> > >> >> AbstractConnection.java:273) >> > >> >>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest. >> java:95) >> > >> >>> at >> > >> >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( >> > >> >> SelectChannelEndPoint.java:93) >> > >> >>> at >> > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( >> > >> >> QueuedThreadPool.java:671) >> > >> >>> at >> > >> >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( >> > >> >> QueuedThreadPool.java:589) >> > >> >>> at java.lang.Thread.run(Unknown Source) >> > >> >>> >> > >> >>> >> > >> >>> Thanks >> > >> >>> Jay >> > >> >> >> > >> >> >> > >> >> > >> >> > > >> > >> > >
