On 7/10/2013 5:58 PM, Petersen, Robert wrote: > Using solr 3.6.1 and the following settings, I am trying to run without > optimizes. I used to optimize nightly, but sometimes the optimize took a > very long time to complete and slowed down our indexing. We are continuously > indexing our new or changed data all day and night. After a few days running > without an optimize, the index size has nearly doubled and maxdocs is nearly > twice the size of numdocs. I understand deletes should be expunged on > merges, but even after trying lots of different settings for our merge policy > it seems this growth is somewhat unbounded. I have tried sending an optimize > with numSegments = 2 which is a lot lighter weight then a regular optimize > and that does bring the number down but not by too much. Does anyone have > any ideas for better settings for my merge policy that would help? Here is > my current index snapshot too:
Your merge settings are the equivalent of the old mergeFactor set to 35, and based on the fact that you have the Explicit set to 105, I'm guessing your settings originally came from something I posted - these are the numbers that I use. These settings can result in a very large number of segments on your disk. Because you index a lot (and probably reindex existing documents often), I can understand why you have high merge settings, but if you want to eliminate optimizes, you'll need to go lower. The default merge setting of 10 (with an Explicit value of 30) is probably a good starting point, but you might need to go even smaller. On Solr 3.6, an optimize probably cannot take place at the same time as index updates -- the optimize would probably delay updates until after it's finished. I remember running into problems on Solr 3.x, so I set up my indexing program to stop updates while the index was optimizing. Solr 4.x should lift any restriction where optimizes and updates can't happen at the same time. With an index size of 25GB, a six-drive RAID10 should be able to optimize in 10-15 minutes, but if your I/O system is single disk, RAID1, RAID5, or RAID6, the write performance may cause this to take longer. If you went with SSD, optimizes would happen VERY fast. Thanks, Shawn
