Optimization time on solr index has turned into days/weeks. We are using solr 1.2. We use one box to build/optimize indexes. This index is copied to another box for searching purposes. We welcome suggestions/comments, etc. We are a bit stumped on this. Details are below.
Box details Proc: 8 Dual Core 2.6GHz Mem: 32 GB OS: Red Hat Linux Enterprise 4 Kernel: 2.6.9-55.0.12.ELlargesmp These are details from the index currently in use. Search response time is very acceptable (searchers are very happy) Optimization time - 10433 (12/11/07) index size - 229486464 # of records - 84960570 index directory flasher# ls -l total 229486464 -rw-r--r-- 1 flknud staff 22197926593 Dec 12 08:07 _2bl6.fdt -rw-r--r-- 1 flknud staff 679684560 Dec 12 08:20 _2bl6.fdx -rw-r--r-- 1 flknud staff 208 Dec 12 08:23 _2bl6.fnm -rw-r--r-- 1 flknud staff 40176405625 Dec 12 09:28 _2bl6.frq -rw-r--r-- 1 flknud staff 594723994 Dec 12 09:41 _2bl6.nrm -rw-r--r-- 1 flknud staff 47616340310 Dec 12 12:07 _2bl6.prx -rw-r--r-- 1 flknud staff 76708079 Dec 12 12:25 _2bl6.tii -rw-r--r-- 1 flknud staff 6154384415 Dec 12 12:42 _2bl6.tis -rw-r--r-- 1 flknud staff 20 Dec 12 12:48 segments.gen -rw-r--r-- 1 flknud staff 44 Dec 12 12:48 segments_2c64 -------------------------- current directory listing indexed new records - Jan 22 and Jan 27 # of records - 85032470 optimization time - 558188 There were no out of memory errors. There was 800961792KB left in the directory. The files were not collapsed as expected. There are still files dated Jan 22 and Jan 27. A new optimization was started Feb. 11 and continues. This is a snapshot of the index directory. We have at least another million records to add. Plus weekly updates of approximately 103K records. We are using the direct indexing method. java settings used - java -Xmx1024M -Xms1024M The files continue to grow so work is progressing. snapshot 2/21/08 -bash-3.00$ ls -ltr total 205396680 -rw-r--r-- 1 flknud users 208 Jan 10 07:15 _2bm7.fnm -rw-r--r-- 1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt -rw-r--r-- 1 flknud users 679819760 Jan 10 08:09 _2bm7.fdx -rw-r--r-- 1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq -rw-r--r-- 1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx -rw-r--r-- 1 flknud users 6155230704 Jan 16 18:16 _2bm7.tis -rw-r--r-- 1 flknud users 76704158 Jan 16 18:16 _2bm7.tii -rw-r--r-- 1 flknud users 594842294 Jan 16 18:18 _2bm7.nrm -rw-r--r-- 1 flknud users 208 Jan 22 08:57 _2bpa.fnm -rw-r--r-- 1 flknud users 10806426 Jan 22 08:57 _2bpa.fdt -rw-r--r-- 1 flknud users 371200 Jan 22 08:57 _2bpa.fdx -rw-r--r-- 1 flknud users 21114330 Jan 22 08:57 _2bpa.frq -rw-r--r-- 1 flknud users 25683573 Jan 22 08:57 _2bpa.prx -rw-r--r-- 1 flknud users 9225592 Jan 22 08:57 _2bpa.tis -rw-r--r-- 1 flknud users 118660 Jan 22 08:57 _2bpa.tii -rw-r--r-- 1 flknud users 324804 Jan 22 08:57 _2bpa.nrm -rw-r--r-- 1 flknud users 198 Jan 22 09:00 _2bpl.fnm -rw-r--r-- 1 flknud users 1335931 Jan 22 09:00 _2bpl.fdt -rw-r--r-- 1 flknud users 36800 Jan 22 09:00 _2bpl.fdx -rw-r--r-- 1 flknud users 2646708 Jan 22 09:00 _2bpl.frq -rw-r--r-- 1 flknud users 3781824 Jan 22 09:00 _2bpl.prx -rw-r--r-- 1 flknud users 1429176 Jan 22 09:00 _2bpl.tis -rw-r--r-- 1 flknud users 18582 Jan 22 09:00 _2bpl.tii -rw-r--r-- 1 flknud users 32204 Jan 22 09:00 _2bpl.nrm -rw-r--r-- 1 flknud users 198 Jan 22 09:01 _2bpm.fnm -rw-r--r-- 1 flknud users 121716 Jan 22 09:01 _2bpm.fdt -rw-r--r-- 1 flknud users 3200 Jan 22 09:01 _2bpm.fdx -rw-r--r-- 1 flknud users 205961 Jan 22 09:01 _2bpm.frq -rw-r--r-- 1 flknud users 302114 Jan 22 09:01 _2bpm.prx -rw-r--r-- 1 flknud users 233641 Jan 22 09:01 _2bpm.tis -rw-r--r-- 1 flknud users 3036 Jan 22 09:01 _2bpm.tii -rw-r--r-- 1 flknud users 2804 Jan 22 09:01 _2bpm.nrm -rw-r--r-- 1 flknud users 198 Jan 27 14:00 _2bpn.fnm -rw-r--r-- 1 flknud users 227962 Jan 27 14:00 _2bpn.fdt -rw-r--r-- 1 flknud users 7200 Jan 27 14:00 _2bpn.fdx -rw-r--r-- 1 flknud users 437798 Jan 27 14:00 _2bpn.frq -rw-r--r-- 1 flknud users 593858 Jan 27 14:00 _2bpn.prx -rw-r--r-- 1 flknud users 516031 Jan 27 14:00 _2bpn.tis -rw-r--r-- 1 flknud users 6814 Jan 27 14:00 _2bpn.tii -rw-r--r-- 1 flknud users 6304 Jan 27 14:00 _2bpn.nrm -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpo.fnm -rw-r--r-- 1 flknud users 231456 Jan 27 14:01 _2bpo.fdt -rw-r--r-- 1 flknud users 7200 Jan 27 14:01 _2bpo.fdx -rw-r--r-- 1 flknud users 448401 Jan 27 14:01 _2bpo.frq -rw-r--r-- 1 flknud users 616557 Jan 27 14:01 _2bpo.prx -rw-r--r-- 1 flknud users 587697 Jan 27 14:01 _2bpo.tis -rw-r--r-- 1 flknud users 7801 Jan 27 14:01 _2bpo.tii -rw-r--r-- 1 flknud users 6304 Jan 27 14:01 _2bpo.nrm -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpp.fnm -rw-r--r-- 1 flknud users 229944 Jan 27 14:01 _2bpp.fdt -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpp.fdx -rw-r--r-- 1 flknud users 462865 Jan 27 14:01 _2bpp.frq -rw-r--r-- 1 flknud users 652003 Jan 27 14:01 _2bpp.prx -rw-r--r-- 1 flknud users 574096 Jan 27 14:01 _2bpp.tis -rw-r--r-- 1 flknud users 7452 Jan 27 14:01 _2bpp.tii -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpp.nrm -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpq.fnm -rw-r--r-- 1 flknud users 53438 Jan 27 14:01 _2bpq.fdt -rw-r--r-- 1 flknud users 1600 Jan 27 14:01 _2bpq.fdx -rw-r--r-- 1 flknud users 103059 Jan 27 14:01 _2bpq.frq -rw-r--r-- 1 flknud users 157149 Jan 27 14:01 _2bpq.prx -rw-r--r-- 1 flknud users 179711 Jan 27 14:01 _2bpq.tis -rw-r--r-- 1 flknud users 2489 Jan 27 14:01 _2bpq.tii -rw-r--r-- 1 flknud users 1404 Jan 27 14:01 _2bpq.nrm -rw-r--r-- 1 flknud users 198 Jan 27 14:01 _2bpr.fnm -rw-r--r-- 1 flknud users 200777 Jan 27 14:01 _2bpr.fdt -rw-r--r-- 1 flknud users 6400 Jan 27 14:01 _2bpr.fdx -rw-r--r-- 1 flknud users 416658 Jan 27 14:01 _2bpr.frq -rw-r--r-- 1 flknud users 570959 Jan 27 14:01 _2bpr.prx -rw-r--r-- 1 flknud users 489593 Jan 27 14:01 _2bpr.tis -rw-r--r-- 1 flknud users 6451 Jan 27 14:01 _2bpr.tii -rw-r--r-- 1 flknud users 5604 Jan 27 14:01 _2bpr.nrm -rw-r--r-- 1 flknud users 236 Jan 27 14:01 segments_2cap -rw-r--r-- 1 flknud users 20 Jan 27 14:01 segments.gen -rw-r--r-- 1 flknud users 0 Feb 11 09:56 write.lock -rw-r--r-- 1 flknud users 208 Feb 11 09:56 _2bps.fnm -rw-r--r-- 1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt -rw-r--r-- 1 flknud users 680259760 Feb 11 12:26 _2bps.fdx -rw-r--r-- 1 flknud users 24416256 Feb 21 06:42 _2bps.tii -rw-r--r-- 1 flknud users 1685057536 Feb 21 06:56 _2bps.tis -rw-r--r-- 1 flknud users 29885433856 Feb 21 07:32 _2bps.frq -rw-r--r-- 1 flknud users 38229336064 Feb 21 07:32 _2bps.prx -- View this message in context: http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html Sent from the Solr - User mailing list archive at Nabble.com.
