Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
On Tue, Nov 11, 2008 at 3:35 PM, Burton-West, Tom <[EMAIL PROTECTED]> wrote: > Hi Yonik, > > Thanks for the NIO suggestion. We are using Linux, but our indexes are > NFS mounted. That may be your biggest performance problem right there - try on a local index and see what the difference is. > I thought I saw something about problems with NIO and NFS, > but am fuzzy on the details. > > These results are with Solr 1.2 and I'm wondering if even without the > NIO change, upgrading to Solr 1.3 might help. You would need to try the latest nightly build to get the NIO stuff. > What confuses me is why > multiple searchers are locking the prx index file. I would think that > searching is a read-only operation. Reading a chunk of a file requires a seek() and a read()... that needs to be synchronized so the read can execute before another seek() happens in another thread. NIO has a read(position, length) call and we can thus avoid synchronization. -Yonik
Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Tom, have you tried copying your index to the local disk to see if that resolves the locking problem? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch From: "Burton-West, Tom" <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Cc: "Farber, Phillip" <[EMAIL PROTECTED]> Sent: Tuesday, November 11, 2008 3:35:20 PM Subject: RE: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory Hi Yonik, Thanks for the NIO suggestion. We are using Linux, but our indexes are NFS mounted. I thought I saw something about problems with NIO and NFS, but am fuzzy on the details. These results are with Solr 1.2 and I'm wondering if even without the NIO change, upgrading to Solr 1.3 might help. What confuses me is why multiple searchers are locking the prx index file. I would think that searching is a read-only operation. Perhaps we need to change something to tell Solr we aren't updating the index? Tom -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Friday, November 07, 2008 8:25 PM To: solr-user@lucene.apache.org Cc: Farber, Phillip Subject: Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory Hi Tom, if you're on a non Windows box, could you perhaps try your test on the latest Solr nightly build? We've recently improved this through the use of NIO. -Yonik On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <[EMAIL PROTECTED]> wrote: > Hello, > > We are testing Solr with a simulation of 30 concurrent users. We are > getting socket timeouts and the thread dump from the admin tool shows > about 100+ threads with a similar message about a lock. (Message > appended below). > > We supsect this may have something to do with one or more phrase > queries containing common terms since our index is very large and we > suspect one or more very large segments of the position index need to > be read into memory. > > Can someone point us to either the possible cause of this problem or > what we might change to reduce/eliminate it? > > Tom > > Tom Burton-West > Information Retrieval Programmer > Digital Library Production Services > University of Michigan Library > [EMAIL PROTECTED] > > -- > > 'http-8080-Processor54' Id=71, BLOCKED on > [EMAIL PROTECTED] > 47 , total cpu time=2070.ms user time=1460.ms at > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect > or > y.java:532) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.j > av > a:93) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput > .j > ava:34) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57) > at > org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(Segment > Te > rmPositions.java:70) > at > org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermP > os > itions.java:66) > at > org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions. > ja > va:76) > at > org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer. > java:45) > at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94) > at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81) > at org.apache.lucene.search.Scorer.score(Scorer.java:48) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > at org.apache.lucene.search.Searcher.search(Searcher.java:118) > at org.apache.lucene.search.Searcher.search(Searcher.java:97) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher. > java:888) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher > .j > ava:805) > at > org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher. > ja > va:698) > at > org.apache.solr.request.StandardRequestHandler.handleRequestBody(Stand > ar > dRequestHandler.java:122) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle > rB > ase.java:77) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter. > ja > va:191) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > .j > ava:159) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > ca > tionFilterChain.java:215) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lt > erChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lv > e.java:2
RE: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Hi Yonik, Thanks for the NIO suggestion. We are using Linux, but our indexes are NFS mounted. I thought I saw something about problems with NIO and NFS, but am fuzzy on the details. These results are with Solr 1.2 and I'm wondering if even without the NIO change, upgrading to Solr 1.3 might help. What confuses me is why multiple searchers are locking the prx index file. I would think that searching is a read-only operation. Perhaps we need to change something to tell Solr we aren't updating the index? Tom -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Friday, November 07, 2008 8:25 PM To: solr-user@lucene.apache.org Cc: Farber, Phillip Subject: Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory Hi Tom, if you're on a non Windows box, could you perhaps try your test on the latest Solr nightly build? We've recently improved this through the use of NIO. -Yonik On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <[EMAIL PROTECTED]> wrote: > Hello, > > We are testing Solr with a simulation of 30 concurrent users. We are > getting socket timeouts and the thread dump from the admin tool shows > about 100+ threads with a similar message about a lock. (Message > appended below). > > We supsect this may have something to do with one or more phrase > queries containing common terms since our index is very large and we > suspect one or more very large segments of the position index need to > be read into memory. > > Can someone point us to either the possible cause of this problem or > what we might change to reduce/eliminate it? > > Tom > > Tom Burton-West > Information Retrieval Programmer > Digital Library Production Services > University of Michigan Library > [EMAIL PROTECTED] > > -- > > 'http-8080-Processor54' Id=71, BLOCKED on > [EMAIL PROTECTED] > 47 , total cpu time=2070.ms user time=1460.ms at > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect > or > y.java:532) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.j > av > a:93) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput > .j > ava:34) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57) > at > org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(Segment > Te > rmPositions.java:70) > at > org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermP > os > itions.java:66) > at > org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions. > ja > va:76) > at > org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer. > java:45) > at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94) > at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81) > at org.apache.lucene.search.Scorer.score(Scorer.java:48) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > at org.apache.lucene.search.Searcher.search(Searcher.java:118) > at org.apache.lucene.search.Searcher.search(Searcher.java:97) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher. > java:888) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher > .j > ava:805) > at > org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher. > ja > va:698) > at > org.apache.solr.request.StandardRequestHandler.handleRequestBody(Stand > ar > dRequestHandler.java:122) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle > rB > ase.java:77) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter. > ja > va:191) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > .j > ava:159) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > ca > tionFilterChain.java:215) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lt > erChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lv > e.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > lv > e.java:174) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:5 > 48 > ) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > va > :127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja > va > :117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. > java:108)
Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Hi Tom, if you're on a non Windows box, could you perhaps try your test on the latest Solr nightly build? We've recently improved this through the use of NIO. -Yonik On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <[EMAIL PROTECTED]> wrote: > Hello, > > We are testing Solr with a simulation of 30 concurrent users. We are > getting socket timeouts and the thread dump from the admin tool shows > about 100+ threads with a similar message about a lock. (Message > appended below). > > We supsect this may have something to do with one or more phrase queries > containing common terms since our index is very large and we suspect one > or more very large segments of the position index need to be read into > memory. > > Can someone point us to either the possible cause of this problem or > what we might change to reduce/eliminate it? > > Tom > > Tom Burton-West > Information Retrieval Programmer > Digital Library Production Services > University of Michigan Library > [EMAIL PROTECTED] > > -- > > 'http-8080-Processor54' Id=71, BLOCKED on > [EMAIL PROTECTED] > , total cpu time=2070.ms user time=1460.ms > at > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector > y.java:532) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav > a:93) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j > ava:34) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57) > at > org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTe > rmPositions.java:70) > at > org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPos > itions.java:66) > at > org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.ja > va:76) > at > org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer. > java:45) > at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94) > at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81) > at org.apache.lucene.search.Scorer.score(Scorer.java:48) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) > at org.apache.lucene.search.Searcher.search(Searcher.java:118) > at org.apache.lucene.search.Searcher.search(Searcher.java:97) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher. > java:888) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.j > ava:805) > at > org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.ja > va:698) > at > org.apache.solr.request.StandardRequestHandler.handleRequestBody(Standar > dRequestHandler.java:122) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB > ase.java:77) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja > va:191) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j > ava:159) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica > tionFilterChain.java:215) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt > erChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv > e.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv > e.java:174) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548 > ) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java > :127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java > :117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. > java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1 > 74) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:87 > 4) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc > essConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint > .java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollow > erWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool > .java:689) > at java.lang.Thread.run(Thread.java:619) > >
