Re: Optimize for large index size

2008-01-20 Thread Michael McCandless
lucene.apache.org Sent: Friday, January 18, 2008 4:31:26 AM Subject: Optimize for large index size Hi, We are using Lucene 2.2. We have an index of size 70G (within 3-4 days) and growing. We run optimize pretty frequently (once every hour - due to large number of index updates every min -

Re: Optimize for large index size

2008-01-20 Thread vivek sar
ED]> > To: java-user@lucene.apache.org > Sent: Friday, January 18, 2008 4:31:26 AM > Subject: Optimize for large index size > > Hi, > > We are using Lucene 2.2. We have an index of size 70G (within 3-4 > days) and growing. We run optimize pretty frequently (once every h

Re: Optimize for large index size

2008-01-19 Thread Otis Gospodnetic
AIL PROTECTED]> To: java-user@lucene.apache.org Sent: Friday, January 18, 2008 4:31:26 AM Subject: Optimize for large index size Hi, We are using Lucene 2.2. We have an index of size 70G (within 3-4 days) and growing. We run optimize pretty frequently (once every hour - due to large number of index u

Re: Optimize for large index size

2008-01-19 Thread Michael McCandless
vivek sar wrote: Thanks Michael for the feedback. Couple more questions, 1) Doesn't Lucene do some sort of optimization internally based on mergefactor, i.e, if the number of segments grow over the mergefactor number Lucene would automatically merge them into one segment - is this different th

Re: Optimize for large index size

2008-01-18 Thread vivek sar
Thanks Michael for the feedback. Couple more questions, 1) Doesn't Lucene do some sort of optimization internally based on mergefactor, i.e, if the number of segments grow over the mergefactor number Lucene would automatically merge them into one segment - is this different than optimization? Does

Re: Optimize for large index size

2008-01-18 Thread Michael McCandless
vivek sar wrote: Hi, We are using Lucene 2.2. We have an index of size 70G (within 3-4 days) and growing. We run optimize pretty frequently (once every hour - due to large number of index updates every min - can be up to 100K new documents every min). I have seen every now and then the optim

Optimize for large index size

2008-01-18 Thread vivek sar
Hi, We are using Lucene 2.2. We have an index of size 70G (within 3-4 days) and growing. We run optimize pretty frequently (once every hour - due to large number of index updates every min - can be up to 100K new documents every min). I have seen every now and then the optimize takes 3-4 hours t