Note that moving to Java 6 alone will not save you. You do need ti give this poor JVM more memory to play with when working with such a big index.
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: F Knudson <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Friday, February 29, 2008 11:22:04 AM > Subject: Re: Optimization taking days/weeks > > > Yes indeed - it was spending all of its time in garbage collection. We will > be moving to Java6. > Thanks for your suggestion. > > Frances > > > Yonik Seeley wrote: > > > > Have you checked if this is due to running out of heap memory? > > When that happens, the garbage collector can start taking a lot of CPU. > > If you are using a Java6 JVM, it should have management enabled by > > default and you should be able to connect to it via jconsole and > > check. > > > > -Yonik > > > > On Thu, Feb 28, 2008 at 9:54 AM, F Knudson wrote: > >> > >> Optimization time on solr index has turned into days/weeks. > >> We are using solr 1.2. > >> We use one box to build/optimize indexes. This index is copied to > >> another > >> box for searching purposes. > >> We welcome suggestions/comments, etc. We are a bit stumped on this. > >> Details are below. > >> > >> Box details > >> Proc: 8 Dual Core 2.6GHz > >> Mem: 32 GB > >> OS: Red Hat Linux Enterprise 4 > >> Kernel: 2.6.9-55.0.12.ELlargesmp > >> > >> These are details from the index currently in use. Search response time > >> is > >> very acceptable (searchers are very happy) > >> Optimization time - 10433 (12/11/07) > >> index size - 229486464 > >> # of records - 84960570 > >> index directory > >> flasher# ls -l > >> total 229486464 > >> -rw-r--r-- 1 flknud staff 22197926593 Dec 12 08:07 _2bl6.fdt > >> -rw-r--r-- 1 flknud staff 679684560 Dec 12 08:20 _2bl6.fdx > >> -rw-r--r-- 1 flknud staff 208 Dec 12 08:23 _2bl6.fnm > >> -rw-r--r-- 1 flknud staff 40176405625 Dec 12 09:28 _2bl6.frq > >> -rw-r--r-- 1 flknud staff 594723994 Dec 12 09:41 _2bl6.nrm > >> -rw-r--r-- 1 flknud staff 47616340310 Dec 12 12:07 _2bl6.prx > >> -rw-r--r-- 1 flknud staff 76708079 Dec 12 12:25 _2bl6.tii > >> -rw-r--r-- 1 flknud staff 6154384415 Dec 12 12:42 _2bl6.tis > >> -rw-r--r-- 1 flknud staff 20 Dec 12 12:48 segments.gen > >> -rw-r--r-- 1 flknud staff 44 Dec 12 12:48 segments_2c64 > >> -------------------------- > >> > >> current directory listing > >> indexed new records - Jan 22 and Jan 27 > >> # of records - 85032470 > >> optimization time - 558188 > >> > >> There were no out of memory errors. There was 800961792KB left in the > >> directory. The files were not > >> collapsed as expected. There are still files dated Jan 22 and Jan 27. > >> > >> A new optimization was started Feb. 11 and continues. > >> This is a snapshot of the index directory. > >> > >> We have at least another million records to add. Plus weekly updates of > >> approximately 103K records. > >> We are using the direct indexing method. > >> java settings used - java -Xmx1024M -Xms1024M > >> > >> The files continue to grow so work is progressing. > >> snapshot 2/21/08 > >> -bash-3.00$ ls -ltr > >> total 205396680 > >> -rw-r--r-- 1 flknud users 208 Jan 10 07:15 _2bm7.fnm > >> -rw-r--r-- 1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt > >> -rw-r--r-- 1 flknud users 679819760 Jan 10 08:09 _2bm7.fdx > >> -rw-r--r-- 1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq > >> -rw-r--r-- 1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx > >> -rw-r--r-- 1 flknud users 6155230704 Jan 16 18:16 _2bm7.tis > >> -rw-r--r-- 1 flknud users 76704158 Jan 16 18:16 _2bm7.tii > >> -rw-r--r-- 1 flknud users 594842294 Jan 16 18:18 _2bm7.nrm > >> -rw-r--r-- 1 flknud users 208 Jan 22 08:57 _2bpa.fnm > >> -rw-r--r-- 1 flknud users 10806426 Jan 22 08:57 _2bpa.fdt > >> -rw-r--r-- 1 flknud users 371200 Jan 22 08:57 _2bpa.fdx > >> -rw-r--r-- 1 flknud users 21114330 Jan 22 08:57 _2bpa.frq > >> -rw-r--r-- 1 flknud users 25683573 Jan 22 08:57 _2bpa.prx > >> -rw-r--r-- 1 flknud users 9225592 Jan 22 08:57 _2bpa.tis > >> -rw-r--r-- 1 flknud users 118660 Jan 22 08:57 _2bpa.tii > >> -rw-r--r-- 1 flknud users 324804 Jan 22 08:57 _2bpa.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 22 09:00 _2bpl.fnm > >> -rw-r--r-- 1 flknud users 1335931 Jan 22 09:00 _2bpl.fdt > >> -rw-r--r-- 1 flknud users 36800 Jan 22 09:00 _2bpl.fdx > >> -rw-r--r-- 1 flknud users 2646708 Jan 22 09:00 _2bpl.frq > >> -rw-r--r-- 1 flknud users 3781824 Jan 22 09:00 _2bpl.prx > >> -rw-r--r-- 1 flknud users 1429176 Jan 22 09:00 _2bpl.tis > >> -rw-r--r-- 1 flknud users 18582 Jan 22 09:00 _2bpl.tii > >> -rw-r--r-- 1 flknud users 32204 Jan 22 09:00 _2bpl.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 22 09:01 _2bpm.fnm > >> -rw-r--r-- 1 flknud users 121716 Jan 22 09:01 _2bpm.fdt > >> -rw-r--r-- 1 flknud users 3200 Jan 22 09:01 _2bpm.fdx > >> -rw-r--r-- 1 flknud users 205961 Jan 22 09:01 _2bpm.frq > >> -rw-r--r-- 1 flknud users 302114 Jan 22 09:01 _2bpm.prx > >> -rw-r--r-- 1 flknud users 233641 Jan 22 09:01 _2bpm.tis > >> -rw-r--r-- 1 flknud users 3036 Jan 22 09:01 _2bpm.tii > >> -rw-r--r-- 1 flknud users 2804 Jan 22 09:01 _2bpm.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 27 14:00 _2bpn.fnm > >> -rw-r--r-- 1 flknud users 227962 Jan 27 14:00 _2bpn.fdt > >> -rw-r--r-- 1 flknud users 7200 Jan 27 14:00 _2bpn.fdx > >> -rw-r--r-- 1 flknud users 437798 Jan 27 14:00 _2bpn.frq > >> -rw-r--r-- 1 flknud users 593858 Jan 27 14:00 _2bpn.prx > >> -rw-r--r-- 1 flknud users 516031 Jan 27 14:00 _2bpn.tis > >> -rw-r--r-- 1 flknud users 6814 Jan 27 14:00 _2bpn.tii > >> -rw-r--r-- 1 flknud users 6304 Jan 27 14:00 _2bpn.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpo.fnm > >> -rw-r--r-- 1 flknud users 231456 Jan 27 14:01 _2bpo.fdt > >> -rw-r--r-- 1 flknud users 7200 Jan 27 14:01 _2bpo.fdx > >> -rw-r--r-- 1 flknud users 448401 Jan 27 14:01 _2bpo.frq > >> -rw-r--r-- 1 flknud users 616557 Jan 27 14:01 _2bpo.prx > >> -rw-r--r-- 1 flknud users 587697 Jan 27 14:01 _2bpo.tis > >> -rw-r--r-- 1 flknud users 7801 Jan 27 14:01 _2bpo.tii > >> -rw-r--r-- 1 flknud users 6304 Jan 27 14:01 _2bpo.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpp.fnm > >> -rw-r--r-- 1 flknud users 229944 Jan 27 14:01 _2bpp.fdt > >> -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpp.fdx > >> -rw-r--r-- 1 flknud users 462865 Jan 27 14:01 _2bpp.frq > >> -rw-r--r-- 1 flknud users 652003 Jan 27 14:01 _2bpp.prx > >> -rw-r--r-- 1 flknud users 574096 Jan 27 14:01 _2bpp.tis > >> -rw-r--r-- 1 flknud users 7452 Jan 27 14:01 _2bpp.tii > >> -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpp.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpq.fnm > >> -rw-r--r-- 1 flknud users 53438 Jan 27 14:01 _2bpq.fdt > >> -rw-r--r-- 1 flknud users 1600 Jan 27 14:01 _2bpq.fdx > >> -rw-r--r-- 1 flknud users 103059 Jan 27 14:01 _2bpq.frq > >> -rw-r--r-- 1 flknud users 157149 Jan 27 14:01 _2bpq.prx > >> -rw-r--r-- 1 flknud users 179711 Jan 27 14:01 _2bpq.tis > >> -rw-r--r-- 1 flknud users 2489 Jan 27 14:01 _2bpq.tii > >> -rw-r--r-- 1 flknud users 1404 Jan 27 14:01 _2bpq.nrm > >> -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpr.fnm > >> -rw-r--r-- 1 flknud users 200777 Jan 27 14:01 _2bpr.fdt > >> -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpr.fdx > >> -rw-r--r-- 1 flknud users 416658 Jan 27 14:01 _2bpr.frq > >> -rw-r--r-- 1 flknud users 570959 Jan 27 14:01 _2bpr.prx > >> -rw-r--r-- 1 flknud users 489593 Jan 27 14:01 _2bpr.tis > >> -rw-r--r-- 1 flknud users 6451 Jan 27 14:01 _2bpr.tii > >> -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpr.nrm > >> -rw-r--r-- 1 flknud users 236 Jan 27 14:01 segments_2cap > >> -rw-r--r-- 1 flknud users 20 Jan 27 14:01 segments.gen > >> -rw-r--r-- 1 flknud users 0 Feb 11 09:56 write.lock > >> -rw-r--r-- 1 flknud users 208 Feb 11 09:56 _2bps.fnm > >> -rw-r--r-- 1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt > >> -rw-r--r-- 1 flknud users 680259760 Feb 11 12:26 _2bps.fdx > >> -rw-r--r-- 1 flknud users 24416256 Feb 21 06:42 _2bps.tii > >> -rw-r--r-- 1 flknud users 1685057536 Feb 21 06:56 _2bps.tis > >> -rw-r--r-- 1 flknud users 29885433856 Feb 21 07:32 _2bps.frq > >> -rw-r--r-- 1 flknud users 38229336064 Feb 21 07:32 _2bps.prx > >> -- > >> View this message in context: > >> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762017.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
