Re: Using solr during optimization

2011-11-15 Thread Isan Fulia
Hi Mark, Thanks for the reply. You are right.We need to test first by decreasing the mergefactor and see the indexing as well as searching performance and have some numbers in hand. Also after partial optimize with the same mergefactor how long the performance lasts(both searching and indexing)

Re: Using solr during optimization

2011-11-14 Thread Isan Fulia
Hi Mark, In the above case , what if the index is optimized partly ie. by specifying the max no of segments we want. It has been observed that after optimizing(even partly optimization), the indexing as well as searching had been faster than in case of an unoptimized one. Decreasing the merge

Re: Using solr during optimization

2011-11-14 Thread Mark Miller
On Nov 14, 2011, at 8:27 AM, Isan Fulia wrote: Hi Mark, In the above case , what if the index is optimized partly ie. by specifying the max no of segments we want. It has been observed that after optimizing(even partly optimization), the indexing as well as searching had been faster than

Re: Using solr during optimization

2011-11-13 Thread Kalika Mishra
Hi Mark, Thanks for your reply. What you saying is interesting; so are you suggesting that optimizations should be done usually when there not many updates. Also can you please point out further under what conditions optimizations might be beneficial. Thanks. On 11 November 2011 20:30, Mark

Using solr during optimization

2011-11-11 Thread Kalika Mishra
Hi, I would like to optimize solr core which is in Reader Writer mode. Since the Solr cores are huge in size (above 100 GB) the optimization takes hours to complete. When the optimization is going on say. on the Writer core, the application wants to continue using the indexes for both query and

Re: Using solr during optimization

2011-11-11 Thread Mark Miller
Do a you have something forcing you to optimize, or are you just doing it for the heck of it? On Nov 11, 2011, at 7:50 AM, Kalika Mishra wrote: Hi, I would like to optimize solr core which is in Reader Writer mode. Since the Solr cores are huge in size (above 100 GB) the optimization takes

Re: Using solr during optimization

2011-11-11 Thread Kalika Mishra
Hi Mark, We are performing almost 11,000 updates a day, we have around 50 million docs in the index (i understand we will need to shard) the core seg will get fragmented over a period of time. We will need to do optimize every few days or once in a month; do you have any reason not to optimize

Re: Using solr during optimization

2011-11-11 Thread Mark Miller
I would not optimize - it's very expensive. With 11,000 updates a day, I think it makes sense to completely avoid optimizing. That should be your default move in any case. If you notice performance suffers more than is acceptable (good chance you won't), then I'd use a lower merge factor. It