Matthew, just because an index is read-only on some server it doesn't mean it contains no deletes (no docs marked as deleted, but not yet removed from the index). So you still want to check isDeleted(doc) *unless* you are certain the index has no docs marked as deleted (this happens after optimization).
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Matthew Runo <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Tuesday, August 19, 2008 4:26:59 PM > Subject: Re: Deadlock in lucene? > > I know this isn't really the place for this, so please forgive me - > but does this patch look reasonably safe to use to skip the isDeleted > check inside of FunctionQuery? > > My reasoning behind this is that many people (us included) will be > building the index on a separate server, and then using the > replication scripts to publish the files out to several read-only > servers. On those instances, deletedDocs would always be empty, since > it's a read only instance - and so we can conveniently skip the Lucene > code in question. This flag would also be good for other optimizations > that can only be made when you assume the index is read-only. > > Solr seems to work with the flag set - any reasons why this will crash > and/or kill my kitten? > > (please forgive my posting this here instead of in solr-dev!) > > Index: src/java/org/apache/solr/search/FunctionQParser.java > =================================================================== > --- src/java/org/apache/solr/search/FunctionQParser.java (revision > 687135) > +++ src/java/org/apache/solr/search/FunctionQParser.java Tue Aug 19 > 11:08:45 PDT 2008 > @@ -49,7 +49,7 @@ > } > ***/ > > - return new FunctionQuery(vs); > + return new FunctionQuery(vs, > req.getSchema().getSolrConfig().isReadOnly() ); > } > > /** > Index: src/java/org/apache/solr/search/function/FunctionQuery.java > =================================================================== > --- src/java/org/apache/solr/search/function/FunctionQuery.java > (revision 687135) > +++ src/java/org/apache/solr/search/function/FunctionQuery.java Tue > Aug 19 11:08:45 PDT 2008 > @@ -31,12 +31,14 @@ > */ > public class FunctionQuery extends Query { > ValueSource func; > + Boolean readOnly; > > /** > * @param func defines the function to be used for scoring > */ > - public FunctionQuery(ValueSource func) { > + public FunctionQuery(ValueSource func, Boolean readOnly) { > this.func=func; > + this.readOnly=readOnly; > } > > /** @return The associated ValueSource */ > @@ -113,7 +115,7 @@ > if (doc>=maxDoc) { > return false; > } > - if (reader.isDeleted(doc)) continue; > + if (!readOnly && reader.isDeleted(doc)) continue; > // todo: maybe allow score() to throw a specific exception > // and continue on to the next document if it is thrown... > // that may be useful, but exceptions aren't really good > Index: src/java/org/apache/solr/core/Config.java > =================================================================== > --- src/java/org/apache/solr/core/Config.java (revision 687135) > +++ src/java/org/apache/solr/core/Config.java Tue Aug 19 11:08:45 PDT > 2008 > @@ -45,6 +45,8 @@ > private final String name; > private final SolrResourceLoader loader; > > + private Boolean readOnly; > + > /** > * @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String, > InputStream, String)} instead. > */ > @@ -254,6 +256,19 @@ > return val!=null ? Double.parseDouble(val) : def; > } > > + /** > + * Is the index set up to be readOnly? If so, this will cause the > FunctionQuery stuff to not check > + * for deleted documents. > + * @return boolean readOnly > + */ > + public boolean isReadOnly() { > + if( this.readOnly == null ){ > + readOnly = getBool("/mainIndex/readOnly", false); > + } > + > + return readOnly; > + } > + > // The following functions were moved to ResourceLoader > > //----------------------------------------------------------------------------- > > Index: example/solr/conf/solrconfig.xml > =================================================================== > --- example/solr/conf/solrconfig.xml (revision 687135) > +++ example/solr/conf/solrconfig.xml Tue Aug 19 11:13:13 PDT 2008 > @@ -114,6 +114,12 @@ > This is not needed if lock type is 'none' or 'single' > --> > false > + > + > + false > > > > use > > > > --- end patch --- > > On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote: > > > It's not a deadlock (just a synchronization bottleneck) , but it is a > > known issue in Lucene and there has been some progress in improving > > the situation. > > -Yonik > > > > > > On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo > > wrote: > >> Hello folks! > >> > >> I was just wondering if anyone else has seen this issue under heavy > >> load. We > >> had some servers set to very high thread limits (12 core servers > >> with 32 > >> gigs of ram), and found several threads would end up in this > >> state.... > >> > >> Name: http-8080-891 > >> State: BLOCKED on [EMAIL PROTECTED] > >> owned by: > >> http-8080-191 > >> Total blocked: 97,926 Total waited: 16 > >> > >> Stack trace: > >> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java: > >> 674) > >> org.apache.solr.search.function.FunctionQuery > >> $AllScorer.next(FunctionQuery.java:116) > >> org > >> .apache > >> .lucene > >> .util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:116) > >> org > >> .apache > >> .lucene > >> .search > >> .DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSumScorer.java: > >> 175) > >> org > >> .apache > >> .lucene > >> .search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java:228) > >> org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java: > >> 76) > >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java: > >> 357) > >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java: > >> 320) > >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:137) > >> org.apache.lucene.search.Searcher.search(Searcher.java:126) > >> org.apache.lucene.search.Searcher.search(Searcher.java:105) > >> org > >> .apache > >> .solr > >> .search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java: > >> 1148) > >> org > >> .apache > >> .solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java: > >> 834) > >> org > >> .apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java: > >> 269) > >> org > >> .apache > >> .solr.handler.component.QueryComponent.process(QueryComponent.java: > >> 160) > >> org > >> .apache > >> .solr > >> .handler > >> .component.SearchHandler.handleRequestBody(SearchHandler.java:169) > >> org > >> .apache > >> .solr > >> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java: > >> 128) > >> org.apache.solr.core.SolrCore.execute(SolrCore.java:1143) > >> org > >> .apache > >> .solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > >> org > >> .apache > >> .solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java: > >> 272) > >> org > >> .apache > >> .catalina > >> .core > >> .ApplicationFilterChain > >> .internalDoFilter(ApplicationFilterChain.java:235) > >> org > >> .apache > >> .catalina > >> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java: > >> 206) > >> org > >> .apache > >> .catalina > >> .core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > >> org > >> .apache > >> .catalina > >> .core.StandardContextValve.invoke(StandardContextValve.java:175) > >> org > >> .apache > >> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > >> org > >> .apache > >> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > >> org > >> .apache > >> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java: > >> 109) > >> org > >> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java: > >> 286) > >> org > >> .apache.coyote.http11.Http11Processor.process(Http11Processor.java: > >> 844) > >> org.apache.coyote.http11.Http11Protocol > >> $Http11ConnectionHandler.process(Http11Protocol.java:583) > >> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java: > >> 447) > >> java.lang.Thread.run(Thread.java:619) > >> > >> Thanks for your time! > >> > >> Matthew Runo > >> Software Engineer, Zappos.com > >> [EMAIL PROTECTED] - 702-943-7833 > >> > >> > >
