The 'maxSegments' feature is new with 1.4. I'm not sure that it will cause any less disk I/O during optimize.
The 'mergeFactor=2' idea is not what you think: in this case the index is always "mostly optimized", so you never need to run optimize. Indexing is always slower, because you amortize the optimize time into little continuous chunks during indexing. You never stop indexing. You should not lose documents. On Fri, Nov 13, 2009 at 1:07 PM, Jerome L Quinn <jlqu...@us.ibm.com> wrote: > > Mark Miller <markrmil...@gmail.com> wrote on 11/12/2009 07:18:03 PM: >> Ah, the pains of optimization. Its kind of just how it is. One solution >> is to use two boxes and replication - optimize on the master, and then >> queries only hit the slave. Out of reach for some though, and adds many >> complications. > > Yes, in my use case 2 boxes isn't a great option. > > >> Another kind of option is to use the partial optimize feature: >> >> <optimize maxOptimizeSegments="5"/> >> >> Using this, you can optimize down to n segments and take a shorter hit >> each time. > > Is this a 1.4 feature? I'm planning to migrate to 1.4, but it'll take a > while since > I have to port custom code forward, including a query parser. > > >> Also, if optimizing is so painful, you might lower the merge factor >> amortize that pain better. Thats another way to slowly get there - if >> you lower the merge factor, as merging takes place, the new merge factor >> will be respected, and semgents will merge down. A merge factor of 2 >> (the lowest) will make it so you only ever have 2 segments. Sometimes >> that works reasonably well - you could try 3-6 or something as well. >> Then when you do your partial optimizes (and eventually a full optimize >> perhaps), you want have so far to go. > > So this will slow down indexing but speed up optimize somewhat? > Unfortunately > right now I lose docs I'm indexing, as well slowing searching to a crawl. > Ugh. > > I've got plenty of CPU horsepower. This is where having the ability to > optimize > on another filesystem would be useful. > > Would it perhaps make sense to set up a master/slave on the same machine? > Then > I suppose I can have an index being optimized that might not clobber the > search. > Would new indexed items still be dropped on the floor? > > Thanks, > Jerry -- Lance Norskog goks...@gmail.com
