Hi,
We are in the process of upgrading from lucene 3.6.0 to lucene 4.7.2,
and our tests show a significant search degradation on Windows platform.
Trying to figure this out, here are a couple of points we noticed.
Any suggestions/thoughts will be greatly appreciated.
Thanks!
1) Running search on an optimized collection.
Our first run on Windows machine showed the following results:
Lucene 3.6: 115 queries / sec
Lucene 4.7.2: 74 queries / sec
Looking at the collections themselves, we got the following
characterization:
Lucene 3.6
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 116
Total size of files in FOLDER: 81558862032 bytes (75.96 GB)
Commit Point Information:
=========================
Version: 1399567203042
Timestamp: 1399593668185
Generation: 6018
Segments file name: segments_4n6
Number of segments: 32
Committed size: 81216915273 bytes (75.64 GB)
Number of files in COMMIT POINT: 89
Total size of files in COMMIT POINT: 81216923390 bytes (75.64 GB)
Lucene 4.7.2:
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 301
Total size of files in FOLDER: 71019073768 bytes (66.14 GB)
Commit Point Information:
=========================
Generation: 4518
Segments file name: segments_3hi
Number of segments: 38
Committed size: 70635339707 bytes (65.78 GB)
Number of files in COMMIT POINT: 115
Total size of files in COMMIT POINT: 70635341223 bytes (65.78 GB)
We saw that the collection created by lucene 4.7.2 was10GB smaller but it
had a more segments.
We thought that more segments might account to the search degradation, and
so we decided to run optimization on the 4.7.2 index before rerunning the
search test.
The index was more compact:
Lucene 4.7.2
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 38
Total size of files in FOLDER: 70488334388 bytes (65.65 GB)
Commit Point Information:
=========================
Generation: 4519
Segments file name: segments_3hj
Number of segments: 12
Committed size: 70488333864 bytes (65.65 GB)
Number of files in COMMIT POINT: 37
Total size of files in COMMIT POINT: 70488334368 bytes (65.65 GB)
And as expected, the search results were much better:
4.7.2. 118 queries / sec
We thought that this might be a good direction, so our next step was to
simulate a more compact index as part of our indexing session without
running a full optimize at the end.
To do that we changed maxMergeMB from 4 GB to 6 GB. The collection was
indeed more compact:
Win64 4.7.2 merge=6000 commitPoints:
General Index Information:
==========================
Num docs: 10485760
Num deleted docs: 0
Deletion rate: 0%
Number of files in FOLDER: 213
Total size of files in FOLDER: 83038952682 bytes (77.34 GB)
Commit Point Information:
=========================
Generation: 4406
Segments file name: segments_3ee
Number of segments: 14
Committed size: 70324985193 bytes (65.50 GB)
Number of files in COMMIT POINT: 91
Total size of files in COMMIT POINT: 70324985781 bytes (65.50 GB)
But search results were not good at all:
4.7.2: 72 queries / sec
Does this make sense?
We thought of "Optimize" as mainly decreasing the number of segments in
the collection, and removing deletions.
In this scenario, we had no deletions, and we saw that the number of
segments did in fact decrease substantially,
So why are we not seeing this reflect in search performance? Is there any
other "optimize" influence/hidden-operation that we are missing here?
(Note that we are using LogByteSizeMergePolicy. We know that
TieredMergePolicy is suppose to be better in this aspect, but it is
important to us
To keep the order of the documents the same between commit points... )
2) Search Directory
On Lucene 3.6, we did comprehensive testing and saw that the best search
performance is reached when using an Mmap directory.
(for Indexing we are using SimpleFSDirectory).
We tried different directories again with lucene 4.7.2, and while the
differences were not big, it still seems that Mmap is no longer the best
option:
Lucene 4.7.2 with MMap: 72 queries / sec
Lucene 4.7.2 with SimpleFS: 84 queries / sec
Was there any changes around the MMap directory that might account for
this difference?
If so, do you think that those changes might account for the overall
performance we are seeing?
3) Java 6 / Java 7
We are currently running on Java 6 (that is also the reason we stopped at
lucene 4.7.2 and not 4.8).
Is there a reason to believe that the degradation might be connected to
this?
Thanks again in advance!
