I did that already. The situation was worse. The autocommit part makes solr unavailable. On Dec 8, 2015 7:13 PM, "Emir Arnautovic" <emir.arnauto...@sematext.com> wrote:
> Hi Ali, > Can you try without explicit commits and see if threads will still be > blocked. > > Thanks, > Emir > > On 08.12.2015 16:19, Ali Nazemian wrote: > >> The indexing load is as follows: >> - Around 1000 documents every 5 mins. >> - The indexing speed is slow because of the complicated analyzer which is >> applied to each document. It takes around 60 seconds to index 1000 >> documents with applying this analyzer (It is really slow. However, based >> on >> the analyzing part I think it would be acceptable). >> - The concurrentsolrclient is used in all the indexing/updating cases. >> >> Regards. >> >> On Tue, Dec 8, 2015 at 6:36 PM, Ali Nazemian <alinazem...@gmail.com> >> wrote: >> >> Dear Emir, >>> Hi, >>> There are some cases that I have soft commit in my application. However, >>> the bulk update part has only hard commit for a bulk of 2500 documents. >>> Here are some information about the whole indexing/updating scenarios: >>> - Indexing part uses soft commit. >>> - In a single update cases soft commit is used. >>> - For bulk update batch hard commit is used (on 2500 documents) >>> - Auto hard commit :120 sec >>> - Auto soft commit: disable >>> >>> Best regards. >>> >>> >>> On Tue, Dec 8, 2015 at 12:35 PM, Emir Arnautovic < >>> emir.arnauto...@sematext.com> wrote: >>> >>> Hi Ali, >>>> This thread is blocked because cannot obtain update lock - in this >>>> particular case when doing soft commit. I am guessing that there others >>>> are >>>> blocked for the same reason. Can you tell us bit more about your setup >>>> and >>>> indexing load and procedure? Do you do explicit commits? >>>> >>>> Regards, >>>> Emir >>>> >>>> -- >>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> >>>> >>>> On 08.12.2015 08:16, Ali Nazemian wrote: >>>> >>>> Hi, >>>>> There is a while since I have had problem with Solr 5.2.1 and I could >>>>> not >>>>> fix it yet. The only think that is clear to me is when I send bulk >>>>> update >>>>> to Solr the commit thread will be blocked! Here is the thread dump >>>>> output: >>>>> >>>>> "qtp595445781-8207" prio=10 tid=0x00007f0bf68f5800 nid=0x5785 waiting >>>>> for >>>>> monitor entry [0x00007f081cf04000] >>>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:608) >>>>> - waiting to lock <0x000000067ba2e660> (a java.lang.Object) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1635) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1612) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:161) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:270) >>>>> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) >>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) >>>>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) >>>>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) >>>>> at >>>>> >>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >>>>> at org.eclipse.jetty.server.Server.handle(Server.java:497) >>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) >>>>> at >>>>> org.eclipse.jetty.io >>>>> .AbstractConnection$2.run(AbstractConnection.java:540) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >>>>> at >>>>> >>>>> >>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> FYI there are lots of blocked thread in thread dump report and Solr >>>>> becomes >>>>> really slow in this case. The temporary solution would be restarting >>>>> Solr. >>>>> But, I am really sick of restarting! I really appreciate if somebody >>>>> can >>>>> help me to solve this problem? >>>>> >>>>> Best regards. >>>>> >>>>> >>>>> >>> -- >>> A.Nazemian >>> >>> >> >> > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > >
