[
https://issues.apache.org/jira/browse/SOLR-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097170#comment-18097170
]
Jason Gerlowski commented on SOLR-18309:
----------------------------------------
I noticed this issue while investigating a Solr 9.10.1 heap dump. The primary
cause was that join queries weren't obeying timeAllowed and were accumulating
in the JVM. The stacktrace for a query running > 180s was still stuck in
scorer creation:
{code}
|- at
org.apache.lucene.index.MultiPostingsEnum.<init>(Lorg/apache/lucene/index/MultiTermsEnum;I)V
(MultiPostingsEnum.java:47)
|- at
org.apache.lucene.index.MultiTermsEnum.postings(Lorg/apache/lucene/index/PostingsEnum;I)Lorg/apache/lucene/index/PostingsEnum;
(MultiTermsEnum.java:327)
|- at
org.apache.solr.search.JoinQuery$JoinQueryWeight.getDocSetEnumerate()Lorg/apache/solr/search/DocSet;
(JoinQuery.java:402)
|- at
org.apache.solr.search.JoinQuery$JoinQueryWeight.getDocSet()Lorg/apache/solr/search/DocSet;
(JoinQuery.java:237)
|- at
org.apache.solr.search.JoinQuery$JoinQueryWeight.scorer(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/Scorer;
(JoinQuery.java:165)
|- at
org.apache.lucene.search.Weight.scorerSupplier(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/ScorerSupplier;
(Weight.java:135)
|- at
org.apache.lucene.search.BooleanWeight.scorerSupplier(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/ScorerSupplier;
(BooleanWeight.java:596)
|- at
org.apache.lucene.search.BooleanWeight.requiredBulkScorer(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/BulkScorer;
(BooleanWeight.java:289)
|- at
org.apache.lucene.search.BooleanWeight.booleanScorer(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/BulkScorer;
(BooleanWeight.java:402)
|- at
org.apache.lucene.search.BooleanWeight.bulkScorer(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/BulkScorer;
(BooleanWeight.java:443)
|- at
org.apache.lucene.search.IndexSearcher.searchLeaf(Lorg/apache/lucene/index/LeafReaderContext;Lorg/apache/lucene/search/Weight;Lorg/apache/lucene/search/Collector;)V
(IndexSearcher.java:763)
|- at
org.apache.lucene.search.IndexSearcher.search(Ljava/util/List;Lorg/apache/lucene/search/Weight;Lorg/apache/lucene/search/Collector;)V
(IndexSearcher.java:736)
|- at org.apache.solr.search.SolrIndexSearcher$5.searchWithTimeout()V
(SolrIndexSearcher.java:807)
|- at
org.apache.solr.search.SolrIndexSearcher.search(Ljava/util/List;Lorg/apache/lucene/search/Weight;Lorg/apache/lucene/search/Collector;)V
(SolrIndexSearcher.java:813)
|- at
org.apache.lucene.search.IndexSearcher.search(Lorg/apache/lucene/search/Query;Lorg/apache/lucene/search/Collector;)V
(IndexSearcher.java:560)
|- at
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(Lorg/apache/solr/search/QueryResult;Lorg/apache/lucene/search/Query;Lorg/apache/lucene/search/Collector;Lorg/apache/solr/search/QueryCommand;Lorg/apache/solr/search/DelegatingCollector;)Lorg/apache/lucene/search/Collector;
(SolrIndexSearcher.java:334)
|- at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)V
(SolrIndexSearcher.java:1986)
|- at
org.apache.solr.search.SolrIndexSearcher.getDocListC(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)Lorg/apache/solr/search/QueryResult;
(SolrIndexSearcher.java:1791)
|- at
org.apache.solr.search.SolrIndexSearcher.search(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)Lorg/apache/solr/search/QueryResult;
(SolrIndexSearcher.java:787)
|- at
org.apache.solr.search.SolrIndexSearcher.search(Lorg/apache/solr/search/QueryCommand;)Lorg/apache/solr/search/QueryResult;
(SolrIndexSearcher.java:782)
|- at
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(Lorg/apache/solr/handler/component/ResponseBuilder;Lorg/apache/solr/search/QueryCommand;)V
(QueryComponent.java:1788)
|- at
org.apache.solr.handler.component.QueryComponent.process(Lorg/apache/solr/handler/component/ResponseBuilder;)V
(QueryComponent.java:447)
|- at
org.apache.solr.handler.component.SearchHandler.processComponents(Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;Lorg/apache/solr/handler/component/ResponseBuilder;Lorg/apache/solr/util/RTimerTree;Ljava/util/List;)V
(SearchHandler.java:506)
|- at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;)V
(SearchHandler.java:428)
'- at
org.apache.solr.handler.RequestHandlerBase.handleRequest(Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;)V
(RequestHandlerBase.java:231)
{code}
> Query limits should apply to expensive-to-create scorers
> --------------------------------------------------------
>
> Key: SOLR-18309
> URL: https://issues.apache.org/jira/browse/SOLR-18309
> Project: Solr
> Issue Type: Improvement
> Components: Query Limits
> Affects Versions: main(11.0)
> Reporter: Jason Gerlowski
> Priority: Major
>
> Solr's timeAllowed enforcement relies on wrapping the BulkScorer returned by
> Weight.bulkScorer()/scorer() in Lucene's TimeLimitingBulkScorer. This
> wrapping class checks the deadline periodically while documents are actually
> being scored and collected.
> This is sufficient in most cases. But it leaves a gap: short-circuiting can
> only be done after the scorer has been created. This is a problem for
> queries such as "join" or "graph", where scorer-creation itself involves a
> ton of work, issuing sub-requests, etc. None of this scorer-creation work is
> subject to timeAllowed and as a result (e.g.) join queries regularly run
> waaaay over their timeAllowed or other query limits.
> We should find a way to make scorer-creation itself subject to query limits,
> if possible. Or alternatively, rework these query implementations so that
> scorer-creation is a cheaper operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
