Hello
I used lucene a key-value database and made some performance tests.
I changed the number of documents in the index and the number of queries.
First all indexes are built. The number in the filename denotes the
number of documents in this index. Each document has a key and a value.
$ du -sk test-lucene-* | sort -n
84 test-lucene-1000
136 test-lucene-2000
308 test-lucene-5000
588 test-lucene-10000
1196 test-lucene-20000
3012 test-lucene-50000
6048 test-lucene-100000
12500 test-lucene-200000
31916 test-lucene-500000
64264 test-lucene-1000000
132876 test-lucene-2000000
340676 test-lucene-5000000
685700 test-lucene-10000000
1415940 test-lucene-20000000
3622336 test-lucene-50000000
Now the test goes this way:
* clear the OS cache
* fill the OS cache with all indexes, so lucene should work from main memory
* run all tests in one VM
foreach (element-of-number-of-queries) {
foreach (index) {
open index
foreach (query: number-of-queries) {
execute the query
}
close index
}
}
* repeat the first test batch to see JIT effects
The test output has 4 columns:
1) Number of queries
2) Number of documents in this index
3) Number of seconds used
4) Number of searches per second
The data
-----------------------------------------------------------------------
$ # sync; echo 3 > /proc/sys/vm/drop_caches
$ free -m
total used free shared buffers cached
Mem: 9970 1498 8472 0 0 57
-/+ buffers/cache: 1440 8530
Swap: 11443 190 11253
$ cat test-lucene-*/* > /dev/null
$ free -m
total used free shared buffers cached
Mem: 9970 7671 2299 0 1 6228
-/+ buffers/cache: 1441 8529
Swap: 11443 190 11253
$ java misc.LuceneKeyValuePerformance read
START JIT WARMUP
1000 1000 0,22 4505
1000 1000 0,10 9871
END JIT WARMUP
READING
#searches #entries seconds Searches/s
1000 1000 0,08 13044
1000 2000 0,07 13348
1000 5000 0,06 17687
1000 10000 0,06 17284
1000 20000 0,04 24195
1000 50000 0,03 32368
1000 100000 0,03 28953
1000 200000 0,04 27875
1000 500000 0,07 14120
1000 1000000 0,11 9130
1000 2000000 0,32 3131
1000 5000000 0,18 5559
1000 10000000 0,21 4858
1000 20000000 0,21 4833
1000 50000000 0,33 3006
10000 1000 0,02 431093
10000 2000 0,07 133357
10000 5000 0,07 147052
10000 10000 0,07 149199
10000 20000 0,07 141534
10000 50000 0,08 125821
10000 100000 0,08 126125
10000 200000 0,09 117365
10000 500000 0,15 67195
10000 1000000 0,19 53957
10000 2000000 0,33 29880
10000 5000000 0,28 36286
10000 10000000 0,40 24789
10000 20000000 0,64 15683
10000 50000000 0,56 17711
100000 1000 0,16 644298
100000 2000 0,64 156404
100000 5000 0,67 149330
100000 10000 0,67 150207
100000 20000 0,70 142779
100000 50000 0,74 136046
100000 100000 0,76 132347
100000 200000 0,79 126456
100000 500000 1,28 78079
100000 1000000 1,69 59105
100000 2000000 3,12 32061
100000 5000000 2,32 43092
100000 10000000 3,27 30623
100000 20000000 5,47 18282
100000 50000000 3,56 28090
1000000 1000 1,53 652867
1000000 2000 6,26 159768
1000000 5000 6,64 150497
1000000 10000 6,79 147360
1000000 20000 6,97 143484
1000000 50000 7,26 137750
1000000 100000 7,34 136321
1000000 200000 7,56 132270
1000000 500000 12,76 78399
1000000 1000000 16,53 60494
1000000 2000000 30,49 32793
1000000 5000000 22,37 44700
1000000 10000000 31,76 31485
1000000 20000000 53,04 18852
1000000 50000000 32,30 30956
REPEAT THE FIRST TEST SEQUENCE
1000 1000 0,01 80293
1000 2000 0,01 142244
1000 5000 0,01 130735
1000 10000 0,01 132253
1000 20000 0,01 121036
1000 50000 0,01 107158
1000 100000 0,01 110660
1000 200000 0,01 74441
1000 500000 0,02 54170
1000 1000000 0,03 39513
1000 2000000 0,04 22353
1000 5000000 0,05 21471
1000 10000000 0,08 12752
1000 20000000 0,16 6387
1000 50000000 0,27 3695
FINISHED
$ free -m
total used free shared buffers cached
Mem: 9970 7813 2156 0 14 6380
-/+ buffers/cache: 1418 8552
Swap: 11443 190 11253
(to be continued)
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
