FYI, I just slipped this optimization into trunk.
-Yonik
On Tue, Aug 19, 2008 at 4:37 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> It doesn't matter that it's executed on the read-only server... it
> matters if any of the docs are marked as deleted. That's the
> condition that you probably want to check for.
>
> -Yonik
>
> On Tue, Aug 19, 2008 at 4:26 PM, Matthew Runo <[EMAIL PROTECTED]> wrote:
>> I know this isn't really the place for this, so please forgive me - but does
>> this patch look reasonably safe to use to skip the isDeleted check inside of
>> FunctionQuery?
>>
>> My reasoning behind this is that many people (us included) will be building
>> the index on a separate server, and then using the replication scripts to
>> publish the files out to several read-only servers. On those instances,
>> deletedDocs would always be empty, since it's a read only instance - and so
>> we can conveniently skip the Lucene code in question. This flag would also
>> be good for other optimizations that can only be made when you assume the
>> index is read-only.
>>
>> Solr seems to work with the flag set - any reasons why this will crash
>> and/or kill my kitten?
>>
>> (please forgive my posting this here instead of in solr-dev!)
>>
>> Index: src/java/org/apache/solr/search/FunctionQParser.java
>> ===================================================================
>> --- src/java/org/apache/solr/search/FunctionQParser.java (revision
>> 687135)
>> +++ src/java/org/apache/solr/search/FunctionQParser.java Tue Aug 19
>> 11:08:45 PDT 2008
>> @@ -49,7 +49,7 @@
>> }
>> ***/
>>
>> - return new FunctionQuery(vs);
>> + return new FunctionQuery(vs,
>> req.getSchema().getSolrConfig().isReadOnly() );
>> }
>>
>> /**
>> Index: src/java/org/apache/solr/search/function/FunctionQuery.java
>> ===================================================================
>> --- src/java/org/apache/solr/search/function/FunctionQuery.java (revision
>> 687135)
>> +++ src/java/org/apache/solr/search/function/FunctionQuery.java Tue Aug 19
>> 11:08:45 PDT 2008
>> @@ -31,12 +31,14 @@
>> */
>> public class FunctionQuery extends Query {
>> ValueSource func;
>> + Boolean readOnly;
>>
>> /**
>> * @param func defines the function to be used for scoring
>> */
>> - public FunctionQuery(ValueSource func) {
>> + public FunctionQuery(ValueSource func, Boolean readOnly) {
>> this.func=func;
>> + this.readOnly=readOnly;
>> }
>>
>> /** @return The associated ValueSource */
>> @@ -113,7 +115,7 @@
>> if (doc>=maxDoc) {
>> return false;
>> }
>> - if (reader.isDeleted(doc)) continue;
>> + if (!readOnly && reader.isDeleted(doc)) continue;
>> // todo: maybe allow score() to throw a specific exception
>> // and continue on to the next document if it is thrown...
>> // that may be useful, but exceptions aren't really good
>> Index: src/java/org/apache/solr/core/Config.java
>> ===================================================================
>> --- src/java/org/apache/solr/core/Config.java (revision 687135)
>> +++ src/java/org/apache/solr/core/Config.java Tue Aug 19 11:08:45 PDT 2008
>> @@ -45,6 +45,8 @@
>> private final String name;
>> private final SolrResourceLoader loader;
>>
>> + private Boolean readOnly;
>> +
>> /**
>> * @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,
>> InputStream,
>> String)} instead.
>> */
>> @@ -254,6 +256,19 @@
>> return val!=null ? Double.parseDouble(val) : def;
>> }
>>
>> + /**
>> + * Is the index set up to be readOnly? If so, this will cause the
>> FunctionQuery stuff to not check
>> + * for deleted documents.
>> + * @return boolean readOnly
>> + */
>> + public boolean isReadOnly() {
>> + if( this.readOnly == null ){
>> + readOnly = getBool("/mainIndex/readOnly", false);
>> + }
>> +
>> + return readOnly;
>> + }
>> +
>> // The following functions were moved to ResourceLoader
>>
>> //-----------------------------------------------------------------------------
>>
>> Index: example/solr/conf/solrconfig.xml
>> ===================================================================
>> --- example/solr/conf/solrconfig.xml (revision 687135)
>> +++ example/solr/conf/solrconfig.xml Tue Aug 19 11:13:13 PDT 2008
>> @@ -114,6 +114,12 @@
>> This is not needed if lock type is 'none' or 'single'
>> -->
>> <unlockOnStartup>false</unlockOnStartup>
>> +
>> + <!-- In the event that you are only using this index for reads,
>> + you can enable this flag. This will skip some checks that
>> + can cause performance issues when under high load
>> + -->
>> + <readOnly>false</readOnly>
>> </mainIndex>
>>
>> <!-- Enables JMX if and only if an existing MBeanServer is found, use
>>
>>
>>
>> --- end patch ---
>>
>> On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:
>>
>>> It's not a deadlock (just a synchronization bottleneck) , but it is a
>>> known issue in Lucene and there has been some progress in improving
>>> the situation.
>>> -Yonik
>>>
>>>
>>> On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Hello folks!
>>>>
>>>> I was just wondering if anyone else has seen this issue under heavy load.
>>>> We
>>>> had some servers set to very high thread limits (12 core servers with 32
>>>> gigs of ram), and found several threads would end up in this state....
>>>>
>>>> Name: http-8080-891
>>>> State: BLOCKED on [EMAIL PROTECTED] owned
>>>> by:
>>>> http-8080-191
>>>> Total blocked: 97,926 Total waited: 16
>>>>
>>>> Stack trace:
>>>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:674)
>>>>
>>>> org.apache.solr.search.function.FunctionQuery$AllScorer.next(FunctionQuery.java:116)
>>>>
>>>> org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:116)
>>>>
>>>> org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSumScorer.java:175)
>>>>
>>>> org.apache.lucene.search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java:228)
>>>> org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
>>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:357)
>>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:320)
>>>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:137)
>>>> org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>>> org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>>>
>>>> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1148)
>>>>
>>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:834)
>>>>
>>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>>>>
>>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>>>>
>>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>>>>
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:128)
>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1143)
>>>>
>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>>>>
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
>>>>
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>>
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
>>>>
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>>>
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>>
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>>
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>
>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>>>>
>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>>>> java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Thanks for your time!
>>>>
>>>> Matthew Runo
>>>> Software Engineer, Zappos.com
>>>> [EMAIL PROTECTED] - 702-943-7833
>>>>
>>>>
>>>
>>
>>
>
