I just ran into the same bottleneck with ValueSourceScorer. Here's a stack trace:
Name: QueryThread group 1,#4 State: BLOCKED on org.apache.lucene.index.segmentrea...@49d7fb83 owned by: QueryThread group 1,#8 Total blocked: 1,535,881 Total waited: 1,080 Stack trace: org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663) org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221) org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138) org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159) org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399) org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59) org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51) org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) org.apache.lucene.search.Searcher.search(Searcher.java:118) Peter On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <j...@apache.org>wrote: > > [ > https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] > > Michael McCandless resolved LUCENE-1316. > ---------------------------------------- > > Resolution: Fixed > > Committed revision 737513. Thanks everyone! > > > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer > > ------------------------------------------------------------------------ > > > > Key: LUCENE-1316 > > URL: https://issues.apache.org/jira/browse/LUCENE-1316 > > Project: Lucene - Java > > Issue Type: Bug > > Components: Query/Scoring > > Affects Versions: 2.3 > > Environment: All > > Reporter: Todd Feak > > Assignee: Michael McCandless > > Priority: Minor > > Fix For: 2.9 > > > > Attachments: LUCENE-1316.patch, LUCENE-1316.patch, > LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, > MatchAllDocsQuery.java > > > > Original Estimate: 1h > > Remaining Estimate: 1h > > > > The isDeleted() method on IndexReader has been mentioned a number of > times as a potential synchronization bottleneck. However, the reason this > bottleneck occurs is actually at a higher level that wasn't focused on (at > least in the threads I read). > > In every case I saw where a stack trace was provided to show the > lock/block, higher in the stack you see the MatchAllScorer.next() method. In > Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly > poor performance (order of magnitude) on our load tests for NOT queries, due > to this bottleneck. The problem is that every single document is run through > this isDeleted() method, which is synchronized. Having an optimized index > exacerbates this issues, as there is only a single SegmentReader to > synchronize on, causing a major thread pileup waiting for the lock. > > By simply having the MatchAllScorer see if there have been any deletions > in the reader, much of this can be avoided. Especially in a read-only > environment for production where you have slaves doing all the high load > searching. > > I modified line 67 in the MatchAllDocsQuery > > FROM: > > if (!reader.isDeleted(id)) { > > TO: > > if (!reader.hasDeletions() || !reader.isDeleted(id)) { > > In our micro load test for NOT queries only, this was a major performance > improvement. We also got the same query results. I don't believe this will > improve the situation for indexes that have deletions. > > Please consider making this adjustment for a future bug fix release. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
