It's odd that optimize is creating such tiny segments, and then that
these tiny segments wind up consuming so much disk space.
Can you turn on IndexWriter's infoStream, and post the output of the
attempts to optimize?
Are you sure there are no unhandled exceptions being logged to the
system error log? By default, merges are done in BG threads (using
ConcurrentMergeScheduler), so exceptions hit in those threads may be
logged elsewhere (though optimize() tries to bring such exceptions to
the foreground thread).
Mike
Qingdi wrote:
(I posted this question to "solr user" forum, but didn't get a clear
answer.
So re-post it here.)
Our index size is about 60G. Most of the time, the optimization
works fine.
But this morning, the optimization kept creating new segment files
until all
the free disk space (300G) was used up.
Here is how the files generated during optimization look like:
============================================
....................................
-rw-r--r-- 1 root root 213464 Feb 10 07:33 _5w9a.tis
-rw-r--r-- 1 root root 3039 Feb 10 07:33 _5w9a.tii
-rw-r--r-- 1 root root 9360 Feb 10 07:33 _5w9a.prx
-rw-r--r-- 1 root root 19824 Feb 10 07:33 _5w9a.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w9a.fnm
-rw-r--r-- 1 root root 74884 Feb 10 07:33 _5w9a.fdx
-rw-r--r-- 1 root root 3753944 Feb 10 07:33 _5w9a.fdt
-rw-r--r-- 1 root root 212513 Feb 10 07:33 _5w99.tis
-rw-r--r-- 1 root root 3005 Feb 10 07:33 _5w99.tii
-rw-r--r-- 1 root root 9436 Feb 10 07:33 _5w99.prx
-rw-r--r-- 1 root root 20052 Feb 10 07:33 _5w99.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w99.fnm
-rw-r--r-- 1 root root 75492 Feb 10 07:33 _5w99.fdx
-rw-r--r-- 1 root root 3795439 Feb 10 07:33 _5w99.fdt
-rw-r--r-- 1 root root 215726 Feb 10 07:33 _5w98.tis
-rw-r--r-- 1 root root 2994 Feb 10 07:33 _5w98.tii
-rw-r--r-- 1 root root 9438 Feb 10 07:33 _5w98.prx
-rw-r--r-- 1 root root 20058 Feb 10 07:33 _5w98.frq
-rw-r--r-- 1 root root 27 Feb 10 07:33 _5w98.fnm
-rw-r--r-- 1 root root 75508 Feb 10 07:33 _5w98.fdx
-rw-r--r-- 1 root root 3779402 Feb 10 07:33 _5w98.fdt
....................................
============================================
This is the third time we encountered this issue. Any idea why the
optimization keeps creating new files? (If we restart the solr
server and
run optimization again, the index files can be recovered. we are on
solr
1.3.) Where could we find more info. about how the optimization
process
work?
Appreciate your help.
Qingdi
--
View this message in context:
http://www.nabble.com/optimization-problem-tp21966759p21966759.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
