OK I just put the patch on there -- let me know. Thanks,
Mike
Peter Keegan wrote:
Thanks, Mike. I'll be happy to help test the patch.
Peter
On Fri, Feb 6, 2009 at 6:44 PM, Michael McCandless <luc...@mikemccandless.com
> wrote:
OK I created https://issues.apache.org/jira/browse/LUCENE-1538.
Mike
Peter Keegan wrote:
I just ran into the same bottleneck with ValueSourceScorer.
Here's a stack trace:
Name: QueryThread group 1,#4
State: BLOCKED on org.apache.lucene.index.segmentrea...@49d7fb83
owned by: QueryThread group 1,#8
Total blocked: 1,535,881 Total waited: 1,080
Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
663)
org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:221)
org.apache.lucene.search.function.ValueSourceQuery
$ValueSourceScorer.next(ValueSourceQuery.java:138)
org.apache.lucene.search.function.ValueSourceQuery
$ValueSourceScorer.skipTo(ValueSourceQuery.java:159)
org.apache.lucene.search.function.CustomScoreQuery
$CustomScorer.skipTo(CustomScoreQuery.java:399)
org
.apache
.lucene.search.ConjunctionScorer.doNext(ConjunctionScorer.java:59)
org
.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:
51)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
org.apache.lucene.search.Searcher.search(Searcher.java:118)
Peter
On Sun, Jan 25, 2009 at 9:41 AM, Michael McCandless (JIRA) <j...@apache.org
> wrote:
[ https://issues.apache.org/jira/browse/LUCENE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless resolved LUCENE-1316.
----------------------------------------
Resolution: Fixed
Committed revision 737513. Thanks everyone!
> Avoidable synchronization bottleneck in MatchAlldocsQuery
$MatchAllScorer
>
------------------------------------------------------------------------
>
> Key: LUCENE-1316
> URL: https://issues.apache.org/jira/browse/LUCENE-1316
> Project: Lucene - Java
> Issue Type: Bug
> Components: Query/Scoring
> Affects Versions: 2.3
> Environment: All
> Reporter: Todd Feak
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1316.patch, LUCENE-1316.patch,
LUCENE-1316.patch, LUCENE_1316.patch, LUCENE_1316.patch,
LUCENE_1316.patch, MatchAllDocsQuery.java
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The isDeleted() method on IndexReader has been mentioned a number
of times as a potential synchronization bottleneck. However, the
reason this bottleneck occurs is actually at a higher level that
wasn't focused on (at least in the threads I read).
> In every case I saw where a stack trace was provided to show the
lock/block, higher in the stack you see the MatchAllScorer.next()
method. In Solr paricularly, this scorer is used for "NOT" queries.
We saw incredibly poor performance (order of magnitude) on our load
tests for NOT queries, due to this bottleneck. The problem is that
every single document is run through this isDeleted() method, which
is synchronized. Having an optimized index exacerbates this issues,
as there is only a single SegmentReader to synchronize on, causing a
major thread pileup waiting for the lock.
> By simply having the MatchAllScorer see if there have been any
deletions in the reader, much of this can be avoided. Especially in
a read-only environment for production where you have slaves doing
all the high load searching.
> I modified line 67 in the MatchAllDocsQuery
> FROM:
> if (!reader.isDeleted(id)) {
> TO:
> if (!reader.hasDeletions() || !reader.isDeleted(id)) {
> In our micro load test for NOT queries only, this was a major
performance improvement. We also got the same query results. I
don't believe this will improve the situation for indexes that have
deletions.
> Please consider making this adjustment for a future bug fix release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
