Thanks, Mike. I'll be happy to help test the patch. Peter
On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > > OK I created https://issues.apache.org/jira/browse/LUCENE-1538. > > Mike > > > Peter Keegan wrote: > > I just ran into the same bottleneck with ValueSourceScorer. >> Here's a stack trace: >> >> Name: QueryThread group 1,#4 >> State: BLOCKED on org.apache.lucene.index.segmentrea...@49d7fb83 owned >> by: QueryThread group 1,#8 >> Total blocked: 1,535,881 Total waited: 1,080 >> >> Stack trace: >> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:663) >> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221) >> >> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.next(ValueSourceQuery.java:138) >> >> org.apache.lucene.search.function.ValueSourceQuery$ValueSourceScorer.skipTo(ValueSourceQuery.java:159) >> >> org.apache.lucene.search.function.CustomScoreQuery$CustomScorer.skipTo(CustomScoreQuery.java:399) >> >> org.apache.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59) >> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:51) >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319) >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146) >> org.apache.lucene.search.Searcher.search(Searcher.java:118) >> >> Peter >> >> >> On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) < >> j...@apache.org> wrote: >> >> [ >> https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel >> ] >> >> Michael McCandless resolved LUCENE-1316. >> ---------------------------------------- >> >> Resolution: Fixed >> >> Committed revision 737513. Thanks everyone! >> >> > Avoidable synchronization bottleneck in MatchAlldocsQuery$MatchAllScorer >> > ------------------------------------------------------------------------ >> > >> > Key: LUCENE-1316 >> > URL: https://issues.apache.org/jira/browse/LUCENE-1316 >> > Project: Lucene - Java >> > Issue Type: Bug >> > Components: Query/Scoring >> > Affects Versions: 2.3 >> > Environment: All >> > Reporter: Todd Feak >> > Assignee: Michael McCandless >> > Priority: Minor >> > Fix For: 2.9 >> > >> > Attachments: LUCENE-1316.patch, LUCENE-1316.patch, >> LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, LUCENE_1316.patch, >> MatchAllDocsQuery.java >> > >> > Original Estimate: 1h >> > Remaining Estimate: 1h >> > >> > The isDeleted() method on IndexReader has been mentioned a number of >> times as a potential synchronization bottleneck. However, the reason this >> bottleneck occurs is actually at a higher level that wasn't focused on (at >> least in the threads I read). >> > In every case I saw where a stack trace was provided to show the >> lock/block, higher in the stack you see the MatchAllScorer.next() method. In >> Solr paricularly, this scorer is used for "NOT" queries. We saw incredibly >> poor performance (order of magnitude) on our load tests for NOT queries, due >> to this bottleneck. The problem is that every single document is run through >> this isDeleted() method, which is synchronized. Having an optimized index >> exacerbates this issues, as there is only a single SegmentReader to >> synchronize on, causing a major thread pileup waiting for the lock. >> > By simply having the MatchAllScorer see if there have been any deletions >> in the reader, much of this can be avoided. Especially in a read-only >> environment for production where you have slaves doing all the high load >> searching. >> > I modified line 67 in the MatchAllDocsQuery >> > FROM: >> > if (!reader.isDeleted(id)) { >> > TO: >> > if (!reader.hasDeletions() || !reader.isDeleted(id)) { >> > In our micro load test for NOT queries only, this was a major >> performance improvement. We also got the same query results. I don't >> believe this will improve the situation for indexes that have deletions. >> > Please consider making this adjustment for a future bug fix release. >> >> -- >> This message is automatically generated by JIRA. >> - >> You can reply to this email to add a comment to the issue online. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
