On Mon, 2012-08-13 at 17:36 +0200, Stefan Trcek wrote:
> * There must be 2 "caches" in the system: The performance degrades
> significantly beyond 1000 and again beyond 200000 documents
> in the index.
With 1000 records, everything is in level 2 cache and a lot in level 1,
which makes it blazingly fast. The index at 200K is 13MB, which means
that half of it fits in your level 2 cache. The next step 500K is 32MB,
so your cache only holds 1/5 of it.
In a real setup, with all the other processes competing for level 2
cache, performance will likely be markedly lower.
> * JIT is very significant
>
> * Index start up is very significant allthough the index is in the
> IO cache and the query is the most simple.
As some internal Lucene structures are initialized upon first search, it
is generally advisable to discard the results from the first few
searches.
> * The performance of index size 20000000 is strange.
> Worse performance than 50000000
When I ran your test (results attached), the index at 20M had 76 files
and the index at 50M had 46 files and I got the same slowdown at 20M as
you did. More segments = more merge overhead.
Thank you for sharing your test & measurements,
Toke Eskildsen
Environment:
lucene-3.6.1
cat /proc/cpuinfo | tail ...
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping : 7
cpu MHz : 1200.000
cache size : 20480 KB
physical id : 1
siblings : 8
core id : 7
cpu cores : 8
apicid : 46
initial apicid : 46
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm dca sse4_1 sse4_2 x2apic popcnt aes xsave avx lahf_lm ida arat epb
xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 5199.25
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
[summatst@mars performance_delete]$ java -cp
lucene-3.6.1/lucene-core-3.6.1.jar:lucene-3.6.1/contrib/analyzers/common/lucene-analyzers-3.6.1.jar:.
misc/LuceneKeyValuePerformance write
WRITING
#entries seconds entries/s
1000 0.26 3889
2000 0.19 10636
5000 0.24 20605
10000 0.22 44523
20000 0.19 106527
50000 0.52 96405
100000 0.44 227960
200000 1.03 194073
500000 2.57 194720
1000000 4.16 240508
2000000 7.93 252067
5000000 18.94 264044
10000000 28.86 346530
20000000 59.41 336668
50000000 168.54 296672
[summatst@mars performance_delete]$ sync; echo 3 > /proc/sys/vm/drop_caches
-bash: /proc/sys/vm/drop_caches: Permission denied
[summatst@mars performance_delete]$ free -m
total used free shared buffers cached
Mem: 32060 27402 4658 0 644 14661
-/+ buffers/cache: 12096 19964
Swap: 8191 567 7624
[summatst@mars performance_delete]$ cat test-lucene-*/* > /dev/null
[summatst@mars performance_delete]$ free -m
total used free shared buffers cached
Mem: 32060 28043 4016 0 644 15305
-/+ buffers/cache: 12093 19966
Swap: 8191 567 7624
[summatst@mars performance_delete]$ java -cp
lucene-3.6.1/lucene-core-3.6.1.jar:lucene-3.6.1/contrib/analyzers/common/lucene-analyzers-3.6.1.jar:.
misc/LuceneKeyValuePerformance read
START JIT WARMUP
1000 1000 0.21 4704
1000 1000 0.11 9253
END JIT WARMUP
READING
#searches #entries seconds Searches/s
1000 1000 0.09 11175
1000 2000 0.08 12032
1000 5000 0.06 15674
1000 10000 0.06 16353
1000 20000 0.05 19953
1000 50000 0.05 19977
1000 100000 0.04 22901
1000 200000 0.04 25645
1000 500000 0.08 12808
1000 1000000 0.11 8798
1000 2000000 0.29 3427
1000 5000000 0.09 11207
1000 10000000 0.14 7051
1000 20000000 0.19 5163
1000 50000000 0.32 3078
10000 1000 0.03 332830
10000 2000 0.08 119444
10000 5000 0.09 113888
10000 10000 0.08 118714
10000 20000 0.10 102044
10000 50000 0.09 108773
10000 100000 0.09 105440
10000 200000 0.11 91084
10000 500000 0.18 55360
10000 1000000 0.26 38334
10000 2000000 0.39 25441
10000 5000000 0.34 29555
10000 10000000 0.47 21140
10000 20000000 0.77 13034
10000 50000000 0.69 14454
100000 1000 0.29 339910
100000 2000 0.83 121067
100000 5000 0.88 113527
100000 10000 0.85 117357
100000 20000 0.88 113188
100000 50000 1.01 98554
100000 100000 0.97 103356
100000 200000 0.97 102955
100000 500000 1.64 61049
100000 1000000 2.06 48594
100000 2000000 3.68 27166
100000 5000000 2.74 36448
100000 10000000 3.90 25640
100000 20000000 6.53 15325
100000 50000000 4.43 22553
1000000 1000 2.83 353871
1000000 2000 8.39 119153
1000000 5000 8.79 113712
1000000 10000 9.35 106993
1000000 20000 10.58 94494
1000000 50000 9.48 105441
1000000 100000 9.63 103799
1000000 200000 9.89 101118
1000000 500000 15.92 62799
1000000 1000000 20.35 49144
1000000 2000000 35.98 27793
1000000 5000000 26.66 37512
1000000 10000000 37.44 26712
1000000 20000000 62.85 15910
1000000 50000000 40.97 24405
REPEAT THE FIRST TEST SEQUENCE
1000 1000 0.01 115473
1000 2000 0.01 110742
1000 5000 0.01 102031
1000 10000 0.01 109778
1000 20000 0.01 103655
1000 50000 0.01 96765
1000 100000 0.01 91129
1000 200000 0.01 80868
1000 500000 0.02 46260
1000 1000000 0.03 32825
1000 2000000 0.05 19108
1000 5000000 0.06 18058
1000 10000000 0.09 11215
1000 20000000 0.18 5562
1000 50000000 0.32 3134
FINISHED
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
