[
https://issues.apache.org/jira/browse/LUCENE-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415613#comment-13415613
]
Michael McCandless commented on LUCENE-4227:
--------------------------------------------
I ran perf tests on a 2M Wikipedia index (requires 8 GB heap: need
more RAM to go higher!).
Results without the specialized scorers (baseline is trunk w/ MMapDir):
{noformat}
Task QPS base StdDev base QPS directStdDev direct Pct
diff
PKLookup 259.28 11.94 227.96 5.85 -18% -
-5%
Fuzzy1 160.21 5.11 183.91 1.48 10% -
19%
TermGroup1M 18.33 0.21 21.60 0.11 15% -
19%
SpanNear 5.79 0.16 6.86 0.31 10% -
27%
TermBGroup1M 18.46 0.24 22.16 0.11 17% -
22%
TermBGroup1M1P 22.47 0.65 28.04 0.67 18% -
31%
SloppyPhrase 3.51 0.13 4.60 0.05 24% -
37%
IntNRQ 53.75 4.68 71.22 4.21 14% -
53%
OrHighHigh 18.85 0.42 26.89 2.16 28% -
57%
OrHighMed 37.93 0.91 54.57 5.71 25% -
62%
Respell 167.73 5.37 242.93 1.78 39% -
50%
Wildcard 46.64 1.74 69.98 3.43 37% -
63%
Prefix3 109.51 3.45 165.77 6.42 41% -
62%
Fuzzy2 56.48 2.37 88.25 0.91 48% -
64%
AndHighHigh 24.59 0.74 41.82 0.72 62% -
78%
Phrase 12.57 0.20 21.89 0.71 65% -
82%
Term 39.05 1.74 69.00 3.68 60% -
94%
AndHighMed 126.87 2.48 261.73 4.19 99% -
113%
{noformat}
Nice speedups!
Same run, but using trunk w/ RAMDirectory as the baseline:
{noformat}
Task QPS base StdDev base QPS directStdDev direct Pct
diff
PKLookup 248.50 4.73 222.03 4.43 -14% -
-7%
Fuzzy1 159.41 3.65 185.32 3.15 11% -
21%
SpanNear 5.74 0.08 6.75 0.17 13% -
22%
TermGroup1M 17.78 0.42 21.03 0.68 11% -
25%
TermBGroup1M 19.32 0.58 23.08 1.02 10% -
28%
IntNRQ 46.82 0.49 56.12 1.28 15% -
23%
TermBGroup1M1P 23.27 0.46 30.14 0.91 23% -
36%
Respell 163.36 3.42 221.10 2.48 31% -
39%
OrHighMed 30.62 1.94 42.94 5.70 14% -
69%
OrHighHigh 17.98 0.99 25.69 3.35 17% -
70%
Prefix3 114.41 0.67 164.19 2.22 40% -
46%
Wildcard 47.58 0.36 70.47 1.20 44% -
51%
Fuzzy2 53.92 1.37 83.54 2.66 46% -
64%
SloppyPhrase 5.07 0.23 8.12 0.74 39% -
82%
AndHighHigh 24.73 0.75 40.51 0.42 57% -
70%
Phrase 14.02 0.07 23.42 0.30 64% -
69%
Term 39.96 2.13 67.39 4.09 50% -
88%
AndHighMed 132.66 3.24 274.07 1.64 100% -
113%
{noformat}
Still good speedups over the "obvious hold index in RAM" option.
Then, just testing the specialized scorers (baseline = DirectPF without
specialized scorers):
{noformat}
Task QPS base StdDev base QPS directStdDev direct Pct
diff
IntNRQ 74.86 3.42 71.72 0.27 -8% -
0%
Wildcard 62.88 2.34 60.52 0.49 -7% -
0%
Prefix3 102.46 3.98 98.92 0.85 -7% -
1%
AndHighHigh 51.41 1.96 50.26 1.10 -7% -
3%
AndHighMed 238.18 5.17 234.14 2.83 -4% -
1%
Fuzzy1 179.64 1.73 177.96 3.27 -3% -
1%
SloppyPhrase 8.97 0.37 8.93 0.48 -9% -
9%
Respell 223.76 1.16 222.79 2.68 -2% -
1%
Fuzzy2 79.62 1.38 79.31 0.90 -3% -
2%
SpanNear 6.83 0.25 6.89 0.31 -7% -
9%
PKLookup 220.25 1.46 225.17 2.56 0% -
4%
OrHighMed 50.70 4.27 53.20 3.95 -10% -
23%
TermGroup1M 22.17 0.33 23.42 0.37 2% -
8%
TermBGroup1M 24.45 0.44 25.86 0.21 3% -
8%
TermBGroup1M1P 30.61 0.82 32.76 0.12 3% -
10%
Term 68.69 3.99 74.88 0.28 2% -
16%
OrHighHigh 26.61 1.95 29.07 2.24 -6% -
26%
Phrase 13.81 0.17 15.96 0.13 13% -
17%
{noformat}
Reasonable but not immense speedups by specializing query
scorers.
> DirectPostingsFormat, storing postings as simple int[] in memory, if you have
> tons of RAM
> -----------------------------------------------------------------------------------------
>
> Key: LUCENE-4227
> URL: https://issues.apache.org/jira/browse/LUCENE-4227
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Attachments: LUCENE-4227.patch
>
>
> This postings format just wraps Lucene40 (on disk) but then at search
> time it loads (up front) all terms postings into RAM.
> You'd use this if you have insane amounts of RAM and want the fastest
> possible search performance. The postings are not compressed: docIds,
> positions are stored as straight int[]s.
> The terms are stored as a skip list (array of byte[]), but I packed
> all terms together into a single long byte[]: I had started as actual
> separate byte[] per term but the added pointer deref and loss of
> locality was a lot (~2X) slower for terms-dict intensive queries like
> FuzzyQuery.
> Low frequency postings (docFreq <= 32 by default) store all docs, pos
> and offsets into a single int[]. High frequency postings store docs
> as int[], freqs as int[], and positions as int[][] parallel arrays.
> For skipping I just do a growing binary search.
> I also made specialized DirectTermScorer and DirectExactPhraseScorer
> for the high freq case that just pull the int[] and iterate
> themselves.
> All tests pass.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
