On 12/3/2012 2:39 PM, Shawn Heisey wrote:
What's happening is that whenever it hits maxDocs, I get 2 segment
files, one of them significantly smaller than the other. Rarely, it
creates 3 segments! I know it's not a ramBuffer problem, because
initially the exact same thing was happening with maxDocs at 100000
and a 32MB ramBuffer. I raised the ramBuffer and lowered the
maxDocs. It takes significantly less than 5 minutes maxDocs to get
indexed, so the maxTime value should not be a factor.
See the previous full message for details referenced below.
Looking at those listings again, I can see that the _fq segment wasn't
complete either - the fnm, _nrm.cfe, and _nrm.cfs files were missing.
It looks like both of the segments that get created by each autocommit
are incomplete. The full-import I wrote about earlier is still going,
here are the last two segments:
-rw-r--r-- 1 ncindex ncindex 3505648 Dec 3 19:41 _nd.fdt
-rw-r--r-- 1 ncindex ncindex 2017 Dec 3 19:41 _nd.fdx
-rw-r--r-- 1 ncindex ncindex 2346930 Dec 3 19:41 _nd_Lucene41_0.doc
-rw-r--r-- 1 ncindex ncindex 3066874 Dec 3 19:41 _nd_Lucene41_0.pos
-rw-r--r-- 1 ncindex ncindex 3690545 Dec 3 19:41 _nd_Lucene41_0.tim
-rw-r--r-- 1 ncindex ncindex 64304 Dec 3 19:41 _nd_Lucene41_0.tip
-rw-r--r-- 1 ncindex ncindex 411 Dec 3 19:41 _nd.si
-rw-r--r-- 1 ncindex ncindex 22002 Dec 3 19:41 _nd.tvd
-rw-r--r-- 1 ncindex ncindex 6853272 Dec 3 19:41 _nd.tvf
-rw-r--r-- 1 ncindex ncindex 175793 Dec 3 19:41 _nd.tvx
-rw-r--r-- 1 ncindex ncindex 8814592 Dec 3 19:43 _ne_Lucene41_0.doc
-rw-r--r-- 1 ncindex ncindex 11911168 Dec 3 19:43 _ne_Lucene41_0.pos
-rw-r--r-- 1 ncindex ncindex 8830976 Dec 3 19:43 _ne_Lucene41_0.tim
-rw-r--r-- 1 ncindex ncindex 204396 Dec 3 19:43 _ne_Lucene41_0.tip
As you can see, _nd is missing the fnm and nrm files and _ne is missing
lots of files. When the next autocommit happened, _nf and _ng were
created, and both of the segments listed above were completed.
I still need to do some additional testing, but I am pretty sure that
when autocommit is turned off, all segments are very uniform in size and
only get created one at a time. I will also try with and without updateLog.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
