It doesn't matter that it's executed on the read-only server... it
matters if any of the docs are marked as deleted. That's the
condition that you probably want to check for.
-Yonik
On Tue, Aug 19, 2008 at 4:26 PM, Matthew Runo <[EMAIL PROTECTED]> wrote:
> I know this isn't really the place for this, so please forgive me - but does
> this patch look reasonably safe to use to skip the isDeleted check inside of
> FunctionQuery?
>
> My reasoning behind this is that many people (us included) will be building
> the index on a separate server, and then using the replication scripts to
> publish the files out to several read-only servers. On those instances,
> deletedDocs would always be empty, since it's a read only instance - and so
> we can conveniently skip the Lucene code in question. This flag would also
> be good for other optimizations that can only be made when you assume the
> index is read-only.
>
> Solr seems to work with the flag set - any reasons why this will crash
> and/or kill my kitten?
>
> (please forgive my posting this here instead of in solr-dev!)
>
> Index: src/java/org/apache/solr/search/FunctionQParser.java
> ===================================================================
> --- src/java/org/apache/solr/search/FunctionQParser.java (revision
> 687135)
> +++ src/java/org/apache/solr/search/FunctionQParser.java Tue Aug 19
> 11:08:45 PDT 2008
> @@ -49,7 +49,7 @@
> }
> ***/
>
> - return new FunctionQuery(vs);
> + return new FunctionQuery(vs,
> req.getSchema().getSolrConfig().isReadOnly() );
> }
>
> /**
> Index: src/java/org/apache/solr/search/function/FunctionQuery.java
> ===================================================================
> --- src/java/org/apache/solr/search/function/FunctionQuery.java (revision
> 687135)
> +++ src/java/org/apache/solr/search/function/FunctionQuery.java Tue Aug 19
> 11:08:45 PDT 2008
> @@ -31,12 +31,14 @@
> */
> public class FunctionQuery extends Query {
> ValueSource func;
> + Boolean readOnly;
>
> /**
> * @param func defines the function to be used for scoring
> */
> - public FunctionQuery(ValueSource func) {
> + public FunctionQuery(ValueSource func, Boolean readOnly) {
> this.func=func;
> + this.readOnly=readOnly;
> }
>
> /** @return The associated ValueSource */
> @@ -113,7 +115,7 @@
> if (doc>=maxDoc) {
> return false;
> }
> - if (reader.isDeleted(doc)) continue;
> + if (!readOnly && reader.isDeleted(doc)) continue;
> // todo: maybe allow score() to throw a specific exception
> // and continue on to the next document if it is thrown...
> // that may be useful, but exceptions aren't really good
> Index: src/java/org/apache/solr/core/Config.java
> ===================================================================
> --- src/java/org/apache/solr/core/Config.java (revision 687135)
> +++ src/java/org/apache/solr/core/Config.java Tue Aug 19 11:08:45 PDT 2008
> @@ -45,6 +45,8 @@
> private final String name;
> private final SolrResourceLoader loader;
>
> + private Boolean readOnly;
> +
> /**
> * @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,
> InputStream,
> String)} instead.
> */
> @@ -254,6 +256,19 @@
> return val!=null ? Double.parseDouble(val) : def;
> }
>
> + /**
> + * Is the index set up to be readOnly? If so, this will cause the
> FunctionQuery stuff to not check
> + * for deleted documents.
> + * @return boolean readOnly
> + */
> + public boolean isReadOnly() {
> + if( this.readOnly == null ){
> + readOnly = getBool("/mainIndex/readOnly", false);
> + }
> +
> + return readOnly;
> + }
> +
> // The following functions were moved to ResourceLoader
>
> //-----------------------------------------------------------------------------
>
> Index: example/solr/conf/solrconfig.xml
> ===================================================================
> --- example/solr/conf/solrconfig.xml (revision 687135)
> +++ example/solr/conf/solrconfig.xml Tue Aug 19 11:13:13 PDT 2008
> @@ -114,6 +114,12 @@
> This is not needed if lock type is 'none' or 'single'
> -->
> <unlockOnStartup>false</unlockOnStartup>
> +
> + <!-- In the event that you are only using this index for reads,
> + you can enable this flag. This will skip some checks that
> + can cause performance issues when under high load
> + -->
> + <readOnly>false</readOnly>
> </mainIndex>
>
> <!-- Enables JMX if and only if an existing MBeanServer is found, use
>
>
>
> --- end patch ---
>
> On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:
>
>> It's not a deadlock (just a synchronization bottleneck) , but it is a
>> known issue in Lucene and there has been some progress in improving
>> the situation.
>> -Yonik
>>
>>
>> On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo <[EMAIL PROTECTED]> wrote:
>>>
>>> Hello folks!
>>>
>>> I was just wondering if anyone else has seen this issue under heavy load.
>>> We
>>> had some servers set to very high thread limits (12 core servers with 32
>>> gigs of ram), and found several threads would end up in this state....
>>>
>>> Name: http-8080-891
>>> State: BLOCKED on [EMAIL PROTECTED] owned
>>> by:
>>> http-8080-191
>>> Total blocked: 97,926 Total waited: 16
>>>
>>> Stack trace:
>>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:674)
>>>
>>> org.apache.solr.search.function.FunctionQuery$AllScorer.next(FunctionQuery.java:116)
>>>
>>> org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:116)
>>>
>>> org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSumScorer.java:175)
>>>
>>> org.apache.lucene.search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java:228)
>>> org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:357)
>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:320)
>>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:137)
>>> org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>> org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>>
>>> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1148)
>>>
>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:834)
>>>
>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>>>
>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>>>
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:128)
>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1143)
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
>>>
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>>
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
>>>
>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>> Thanks for your time!
>>>
>>> Matthew Runo
>>> Software Engineer, Zappos.com
>>> [EMAIL PROTECTED] - 702-943-7833
>>>
>>>
>>
>
>
