[
https://issues.apache.org/jira/browse/LUCENE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101296#comment-14101296
]
Michael McCandless commented on LUCENE-4396:
--------------------------------------------
OK I tested on the larger And.tasks:
{noformat}
Report after iter 19:
Task QPS base StdDev QPS comp StdDev
Pct diff
HighAnd3LowNot 12.63 (5.9%) 11.26 (2.0%)
-10.8% ( -17% - -3%)
HighAnd4LowNot 11.41 (5.8%) 10.24 (2.0%)
-10.3% ( -17% - -2%)
HighAnd7LowNot 7.69 (5.4%) 6.95 (1.8%)
-9.7% ( -15% - -2%)
HighAnd9LowNot 7.75 (4.8%) 7.06 (1.6%)
-8.8% ( -14% - -2%)
MedAnd3LowNot 26.09 (4.9%) 23.79 (1.8%)
-8.8% ( -14% - -2%)
HighAnd5LowNot 17.15 (4.4%) 15.81 (1.6%)
-7.8% ( -13% - -1%)
HighAnd6LowNot 16.26 (4.1%) 15.13 (1.4%)
-6.9% ( -11% - -1%)
MedAnd4LowNot 26.69 (3.7%) 25.14 (1.4%)
-5.8% ( -10% - 0%)
HighAnd8LowNot 14.88 (3.2%) 14.13 (1.1%)
-5.0% ( -9% - 0%)
MedAnd7LowNot 18.29 (3.0%) 17.38 (1.1%)
-5.0% ( -8% - 0%)
MedAnd8LowNot 15.96 (3.1%) 15.21 (1.0%)
-4.7% ( -8% - 0%)
HighAnd4MedNot 6.63 (3.0%) 6.32 (1.2%)
-4.7% ( -8% - 0%)
MedAnd5LowNot 23.66 (2.7%) 22.69 (1.1%)
-4.1% ( -7% - 0%)
HighAnd6MedNot 5.16 (2.5%) 4.95 (1.0%)
-3.9% ( -7% - 0%)
HighAnd5MedNot 5.95 (2.3%) 5.73 (1.0%)
-3.7% ( -6% - 0%)
MedAnd9LowNot 16.50 (2.6%) 15.91 (0.9%)
-3.6% ( -6% - 0%)
HighAnd3MedNot 11.60 (2.3%) 11.20 (1.4%)
-3.4% ( -6% - 0%)
MedAnd6LowNot 25.79 (2.6%) 24.91 (1.1%)
-3.4% ( -6% - 0%)
HighAnd7MedNot 4.51 (2.1%) 4.37 (1.0%)
-3.2% ( -6% - 0%)
LowAnd3LowNot 72.70 (2.4%) 70.53 (1.1%)
-3.0% ( -6% - 0%)
MedAnd3MedNot 15.50 (1.9%) 15.04 (1.1%)
-3.0% ( -5% - 0%)
HighAnd3HighNot 3.71 (1.9%) 3.61 (1.4%)
-2.8% ( -5% - 0%)
MedAnd4MedNot 12.83 (1.9%) 12.47 (1.0%)
-2.8% ( -5% - 0%)
LowAnd3MedNot 38.64 (1.8%) 37.79 (1.6%)
-2.2% ( -5% - 1%)
HighAnd8MedNot 4.42 (1.6%) 4.32 (1.0%)
-2.2% ( -4% - 0%)
HighAnd5HighNot 3.44 (1.5%) 3.36 (1.0%)
-2.2% ( -4% - 0%)
MedAnd3HighNot 7.56 (1.4%) 7.42 (1.3%)
-1.9% ( -4% - 0%)
LowAnd5LowNot 47.22 (1.9%) 46.38 (1.7%)
-1.8% ( -5% - 1%)
MedAnd4HighNot 8.43 (1.4%) 8.29 (1.1%)
-1.7% ( -4% - 0%)
LowAnd4LowNot 58.26 (2.3%) 57.28 (1.4%)
-1.7% ( -5% - 2%)
MedAnd5MedNot 14.06 (1.4%) 13.83 (1.1%)
-1.6% ( -4% - 0%)
LowAnd3HighNot 28.74 (2.4%) 28.36 (2.4%)
-1.3% ( -5% - 3%)
LowAnd5MedNot 23.30 (1.5%) 23.01 (1.8%)
-1.3% ( -4% - 2%)
HighAnd4HighNot 4.14 (1.2%) 4.09 (1.1%)
-1.1% ( -3% - 1%)
MedAnd5HighNot 4.83 (1.3%) 4.78 (1.2%)
-1.0% ( -3% - 1%)
LowAnd4HighNot 81.42 (1.8%) 80.63 (2.3%)
-1.0% ( -4% - 3%)
LowAnd4MedNot 33.59 (1.2%) 33.27 (1.2%)
-0.9% ( -3% - 1%)
LowAnd5HighNot 10.79 (2.2%) 10.72 (2.2%)
-0.6% ( -4% - 3%)
LowAnd4HighOr 14.62 (2.6%) 14.61 (2.5%)
-0.1% ( -5% - 5%)
LowAnd3HighOr 17.87 (2.8%) 17.87 (2.4%)
0.0% ( -4% - 5%)
LowAnd5HighOr 15.44 (2.3%) 15.46 (2.5%)
0.1% ( -4% - 5%)
LowAnd4MedOr 27.91 (1.8%) 27.95 (1.8%)
0.1% ( -3% - 3%)
LowAnd5MedOr 19.76 (2.0%) 19.79 (1.6%)
0.1% ( -3% - 3%)
LowAnd3MedOr 35.41 (2.5%) 35.56 (1.9%)
0.4% ( -3% - 4%)
LowAnd3LowOr 72.38 (3.2%) 72.95 (1.2%)
0.8% ( -3% - 5%)
MedAnd3HighOr 5.77 (2.6%) 5.82 (1.5%)
0.8% ( -3% - 5%)
LowAnd5LowOr 44.94 (3.3%) 45.33 (1.4%)
0.9% ( -3% - 5%)
MedAnd5HighOr 6.88 (2.0%) 6.94 (0.9%)
0.9% ( -2% - 3%)
MedAnd4HighOr 7.48 (2.2%) 7.55 (1.1%)
0.9% ( -2% - 4%)
HighAnd7LowOr 5.63 (12.5%) 5.69 (1.2%)
1.0% ( -11% - 16%)
MedAnd6LowOr 27.56 (4.0%) 27.84 (1.2%)
1.0% ( -4% - 6%)
HighAnd3LowOr 9.96 (13.4%) 10.06 (1.3%)
1.0% ( -12% - 18%)
MedAnd8LowOr 19.01 (3.3%) 19.21 (0.9%)
1.1% ( -3% - 5%)
MedAnd9LowOr 17.51 (2.9%) 17.73 (0.8%)
1.3% ( -2% - 5%)
MedAnd5LowOr 22.88 (7.5%) 23.19 (1.1%)
1.3% ( -6% - 10%)
HighAnd8LowOr 12.29 (8.6%) 12.45 (1.1%)
1.3% ( -7% - 12%)
MedAnd7LowOr 20.30 (4.4%) 20.58 (1.0%)
1.4% ( -3% - 7%)
HighAnd9LowOr 10.82 (8.7%) 10.98 (1.1%)
1.4% ( -7% - 12%)
HighAnd6MedOr 6.06 (4.0%) 6.15 (0.9%)
1.4% ( -3% - 6%)
MedAnd3LowOr 26.52 (10.5%) 26.91 (1.2%)
1.5% ( -9% - 14%)
LowAnd4LowOr 53.92 (3.5%) 54.71 (1.6%)
1.5% ( -3% - 6%)
HighAnd6LowOr 15.37 (9.0%) 15.60 (1.1%)
1.5% ( -7% - 12%)
MedAnd4LowOr 27.39 (8.7%) 27.80 (1.3%)
1.5% ( -7% - 12%)
HighAnd5LowOr 17.05 (9.5%) 17.31 (1.3%)
1.5% ( -8% - 13%)
HighAnd4LowOr 25.42 (9.2%) 25.83 (1.1%)
1.6% ( -8% - 13%)
MedAnd4MedOr 10.42 (3.3%) 10.59 (0.8%)
1.6% ( -2% - 5%)
HighAnd9MedOr 3.29 (3.7%) 3.34 (0.8%)
1.6% ( -2% - 6%)
HighAnd4MedOr 7.91 (5.6%) 8.04 (1.0%)
1.7% ( -4% - 8%)
MedAnd5MedOr 8.01 (3.6%) 8.15 (0.8%)
1.7% ( -2% - 6%)
HighAnd5MedOr 5.92 (4.2%) 6.02 (0.9%)
1.7% ( -3% - 7%)
HighAnd8MedOr 3.09 (3.4%) 3.14 (0.8%)
1.7% ( -2% - 6%)
MedAnd3MedOr 13.67 (5.1%) 13.91 (1.1%)
1.8% ( -4% - 8%)
HighAnd4HighOr 2.71 (4.1%) 2.76 (0.9%)
1.9% ( -3% - 7%)
HighAnd5HighOr 2.91 (3.2%) 2.96 (0.8%)
1.9% ( -2% - 6%)
HighAnd3MedOr 8.92 (5.5%) 9.10 (0.9%)
2.0% ( -4% - 8%)
HighAnd3HighOr 4.80 (5.2%) 4.89 (0.9%)
2.0% ( -3% - 8%)
MedAnd6MedOr 4.38 (4.3%) 4.52 (0.7%)
3.2% ( -1% - 8%)
MedAnd6MedNot 7.22 (1.5%) 7.60 (2.4%)
5.3% ( 1% - 9%)
HighAnd6HighOr 1.54 (3.7%) 1.63 (0.9%)
6.2% ( 1% - 11%)
HighAnd6HighNot 3.26 (1.2%) 3.49 (2.2%)
6.8% ( 3% - 10%)
LowAnd6LowNot 44.49 (1.3%) 48.71 (2.4%)
9.5% ( 5% - 13%)
LowAnd7LowNot 33.91 (1.6%) 37.82 (2.6%)
11.5% ( 7% - 15%)
LowAnd6HighNot 10.75 (2.1%) 12.21 (2.5%)
13.5% ( 8% - 18%)
LowAnd6MedNot 22.24 (1.5%) 25.81 (2.1%)
16.0% ( 12% - 19%)
MedAnd6HighNot 4.99 (1.2%) 5.81 (1.6%)
16.4% ( 13% - 19%)
MedAnd7MedNot 7.99 (0.9%) 9.35 (2.3%)
16.9% ( 13% - 20%)
LowAnd7HighNot 12.14 (2.1%) 14.24 (2.3%)
17.3% ( 12% - 22%)
LowAnd8LowNot 30.14 (1.3%) 35.43 (2.0%)
17.5% ( 14% - 21%)
HighAnd8HighNot 2.16 (1.0%) 2.55 (2.2%)
18.0% ( 14% - 21%)
LowAnd9LowNot 31.23 (1.6%) 36.93 (2.6%)
18.3% ( 13% - 22%)
HighAnd9MedNot 4.28 (1.2%) 5.07 (2.4%)
18.5% ( 14% - 22%)
LowAnd6LowOr 44.85 (2.9%) 53.16 (6.0%)
18.5% ( 9% - 28%)
LowAnd8HighNot 8.27 (2.1%) 9.82 (2.8%)
18.7% ( 13% - 24%)
LowAnd7MedNot 18.14 (1.4%) 21.67 (1.6%)
19.4% ( 16% - 22%)
HighAnd9HighNot 1.84 (1.2%) 2.20 (2.3%)
20.0% ( 16% - 23%)
HighAnd7MedOr 4.77 (2.5%) 5.73 (2.2%)
20.1% ( 15% - 25%)
LowAnd7HighOr 8.47 (2.3%) 10.19 (3.0%)
20.3% ( 14% - 26%)
LowAnd6HighOr 10.98 (2.3%) 13.25 (3.0%)
20.7% ( 14% - 26%)
HighAnd7HighNot 3.11 (1.0%) 3.77 (1.8%)
21.1% ( 18% - 24%)
MedAnd8MedNot 6.94 (1.1%) 8.48 (2.3%)
22.2% ( 18% - 25%)
LowAnd9HighOr 7.52 (2.5%) 9.23 (3.1%)
22.7% ( 16% - 29%)
MedAnd7MedOr 4.89 (2.5%) 6.00 (2.9%)
22.8% ( 16% - 28%)
LowAnd9HighNot 7.70 (1.8%) 9.46 (2.5%)
22.9% ( 18% - 27%)
LowAnd8MedNot 16.93 (1.3%) 20.81 (2.3%)
23.0% ( 19% - 26%)
MedAnd7HighNot 4.89 (1.1%) 6.07 (1.8%)
24.0% ( 20% - 27%)
LowAnd8HighOr 8.70 (2.2%) 10.82 (3.4%)
24.4% ( 18% - 30%)
LowAnd7MedOr 16.85 (2.0%) 20.99 (3.2%)
24.5% ( 18% - 30%)
MedAnd9MedNot 7.06 (1.1%) 8.85 (1.9%)
25.3% ( 22% - 28%)
LowAnd7LowOr 36.86 (2.2%) 46.19 (5.2%)
25.3% ( 17% - 33%)
LowAnd6MedOr 21.10 (1.7%) 26.47 (3.2%)
25.4% ( 20% - 30%)
LowAnd8LowOr 29.79 (1.9%) 37.49 (4.6%)
25.8% ( 18% - 32%)
LowAnd9MedNot 13.51 (1.5%) 17.31 (2.6%)
28.1% ( 23% - 32%)
MedAnd9HighNot 3.75 (1.1%) 4.84 (1.7%)
29.1% ( 26% - 32%)
MedAnd8HighNot 3.39 (1.1%) 4.40 (1.7%)
29.8% ( 26% - 32%)
MedAnd6HighOr 4.54 (1.9%) 5.90 (2.4%)
29.9% ( 25% - 34%)
MedAnd8MedOr 10.56 (1.8%) 13.76 (2.7%)
30.3% ( 25% - 35%)
HighAnd7HighOr 2.06 (2.3%) 2.69 (3.2%)
30.3% ( 24% - 36%)
MedAnd7HighOr 4.86 (1.8%) 6.44 (2.3%)
32.5% ( 28% - 37%)
LowAnd8MedOr 15.35 (1.8%) 20.45 (3.1%)
33.2% ( 27% - 38%)
MedAnd9MedOr 3.77 (2.0%) 5.03 (3.0%)
33.4% ( 27% - 39%)
HighAnd8HighOr 1.67 (2.0%) 2.25 (3.2%)
34.3% ( 28% - 40%)
MedAnd8HighOr 4.07 (1.8%) 5.49 (2.7%)
34.8% ( 29% - 39%)
HighAnd9HighOr 1.40 (2.2%) 1.89 (3.4%)
35.1% ( 28% - 41%)
LowAnd9MedOr 11.78 (1.9%) 16.23 (3.3%)
37.8% ( 31% - 43%)
MedAnd9HighOr 2.82 (1.6%) 3.92 (2.3%)
39.1% ( 34% - 43%)
LowAnd9LowOr 25.34 (1.6%) 37.25 (5.9%)
47.0% ( 38% - 55%)
{noformat}
> BooleanScorer should sometimes be used for MUST clauses
> -------------------------------------------------------
>
> Key: LUCENE-4396
> URL: https://issues.apache.org/jira/browse/LUCENE-4396
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Attachments: And.tasks, And.tasks, And.tasks, AndOr.tasks,
> AndOr.tasks, LUCENE-4396-simple.patch, LUCENE-4396-simple.patch,
> LUCENE-4396-simple.patch, LUCENE-4396-simple.patch, LUCENE-4396.patch,
> LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch,
> LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch,
> LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch,
> LUCENE-4396.patch, LUCENE-4396.patch, LUCENE-4396.patch, SIZE.perf, all.perf,
> luceneutil-score-equal.patch, luceneutil-score-equal.patch,
> merge-simple.perf, merge-simple.png, merge.perf, merge.png, perf.png,
> stat.cpp, stat.cpp, tasks.cpp, tasks.cpp
>
>
> Today we only use BooleanScorer if the query consists of SHOULD and MUST_NOT.
> If there is one or more MUST clauses we always use BooleanScorer2.
> But I suspect that unless the MUST clauses have very low hit count compared
> to the other clauses, that BooleanScorer would perform better than
> BooleanScorer2. BooleanScorer still has some vestiges from when it used to
> handle MUST so it shouldn't be hard to bring back this capability ... I think
> the challenging part might be the heuristics on when to use which (likely we
> would have to use firstDocID as proxy for total hit count).
> Likely we should also have BooleanScorer sometimes use .advance() on the subs
> in this case, eg if suddenly the MUST clause skips 1000000 docs then you want
> to .advance() all the SHOULD clauses.
> I won't have near term time to work on this so feel free to take it if you
> are inspired!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
