On Tue, 2008-01-22 at 03:08 -0800, Michael Busch wrote: > OK, then Yonik might be right about the multi-level skiplists code which > is new in 2.2. I'd love to see the performance numbers of the same index > built with 2.3, if possible? You could simply migrate it to 2.3 by using > IndexWriter.addIndexes().
Ah, yes. Just use addIndexes... Silly me. A quick migrate and new tests paints a whole other picture. I don't have time for a proper analysis now, but here's some numbers: (lx is lucene version, v23 is index-version 23, tx is x threads with shared searcher, txu is x threads with separate searchers) == Average over the first 50.000 queries == metis_15000_RAID1_8GB_i37_t2_l21.log - 55.9 q/sec metis_15000_RAID1_8GB_i37_t2_l23.log - 56.3 q/sec metis_15000_RAID1_8GB_i37_v23_t2_l23.log - 57.1 q/sec metis_15000_RAID1_8GB_i37_t3_l21.log - 54.7 q/sec metis_15000_RAID1_8GB_i37_t3_l23.log - 52.9 q/sec metis_15000_RAID1_8GB_i37_v23_t3_l23.log - 54.4 q/sec metis_15000_RAID1_8GB_i37_t3u_l21.log - 70.0 q/sec metis_15000_RAID1_8GB_i37_t3u_l23.log - 70.3 q/sec metis_15000_RAID1_8GB_i37_v23_t3u_l23.log - 70.4 q/sec --- metis_flash_RAID0_8GB_i37_t2_l21.log - 279.6 q/sec metis_flash_RAID0_8GB_i37_t2_l23.log - 202.3 q/sec metis_flash_RAID0_8GB_i37_v23_t2_l23.log - 195.9 q/sec metis_flash_RAID0_8GB_i37_t3_l21.log - 138.9 q/sec metis_flash_RAID0_8GB_i37_t3_l23.log - 131.9 q/sec metis_flash_RAID0_8GB_i37_v23_t3_l23.log - 159.1 q/sec metis_flash_RAID0_8GB_i37_t3u_l21.log - 273.0 q/sec metis_flash_RAID0_8GB_i37_t3u_l23.log - 251.8 q/sec metis_flash_RAID0_8GB_i37_v23_t3u_l23.log - 278.3 q/sec == Average over the first 340.000 queries == metis_15000_RAID1_8GB_i37_t2_l21.log - 109.2 q/sec metis_15000_RAID1_8GB_i37_t2_l23.log - 111.7 q/sec metis_15000_RAID1_8GB_i37_v23_t2_l23.log - 118.2 q/sec metis_15000_RAID1_8GB_i37_t3_l21.log - 103.7 q/sec metis_15000_RAID1_8GB_i37_t3_l23.log - 95.7 q/sec metis_15000_RAID1_8GB_i37_v23_t3_l23.log - 108.3 q/sec metis_15000_RAID1_8GB_i37_t3u_l21.log - 156.8 q/sec metis_15000_RAID1_8GB_i37_t3u_l23.log - 153.4 q/sec metis_15000_RAID1_8GB_i37_v23_t3u_l23.log - 188.3 q/sec --- metis_flash_RAID0_8GB_i37_t2_l21.log - 305.3 q/sec metis_flash_RAID0_8GB_i37_t2_l23.log - 260.5 q/sec metis_flash_RAID0_8GB_i37_v23_t2_l23.log - 294.1 q/sec metis_flash_RAID0_8GB_i37_t3_l21.log - 198.3 q/sec metis_flash_RAID0_8GB_i37_t3_l23.log - 185.3 q/sec metis_flash_RAID0_8GB_i37_v23_t3_l23.log - 238.0 q/sec metis_flash_RAID0_8GB_i37_t3u_l21.log - 378.5 q/sec metis_flash_RAID0_8GB_i37_t3u_l23.log - 336.1 q/sec metis_flash_RAID0_8GB_i37_v23_t3u_l23.log - 433.3 q/sec > In my performance tests (LUCENE-866) I measured an average performance > gain of 20% for AND queries. The 20% seems to fit the case with 3 threads and large test, both for harddisks and SSDs. I'm happy that there was an explanation for the apparent slowdown for the higher versions and that Lucene 2.3 is markedly faster than 2.1 on our system. > Of course, if you run 2.3 against an index built with 2.1, then the more > complex multi-level skiplist code gets executed on the simpler 2.1 data > structures. However, I don't think the slowdown should be that big (13%). If you need me to run tests or generate graphs, just let me know. If not, I'll stop experimenting with Lucene 2.1 vs. 2.3 and concentrate on Lucene 2.3: There 's still the matter of how the number of threads and the number of searchers affects performance. Next step is to tweak the amount of RAM to get an idea of RAM/index size/speed trade-offs. (and yes, I'll also try and find the time to put the results on the Lucene wiki) --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
