Hi Mark, In the above case , what if the index is optimized partly ie. by specifying the max no of segments we want. It has been observed that after optimizing(even partly optimization), the indexing as well as searching had been faster than in case of an unoptimized one. Decreasing the merge factor will affect the performance as it will increase the indexing time due to the frequent merges. So is it good that we optimize partly(let say once in a month), rather than decreasing the merge factor and affect the indexing speed.Also since we will be sharding, that 100 GB index will be divided in different shards.
Thanks, Isan Fulia. On 14 November 2011 11:28, Kalika Mishra <kalika.mis...@germinait.com>wrote: > Hi Mark, > > Thanks for your reply. > > What you saying is interesting; so are you suggesting that optimizations > should be done usually when there not many updates. Also can you please > point out further under what conditions optimizations might be beneficial. > > Thanks. > > On 11 November 2011 20:30, Mark Miller <markrmil...@gmail.com> wrote: > > > I would not optimize - it's very expensive. With 11,000 updates a day, I > > think it makes sense to completely avoid optimizing. > > > > That should be your default move in any case. If you notice performance > > suffers more than is acceptable (good chance you won't), then I'd use a > > lower merge factor. It defaults to 10 - lower numbers will lower the > number > > of segments in your index, and essentially amortize the cost of an > optimize. > > > > Optimize is generally only useful when you will have a mostly static > index. > > > > - Mark Miller > > lucidimagination.com > > > > > > On Nov 11, 2011, at 9:12 AM, Kalika Mishra wrote: > > > > > Hi Mark, > > > > > > We are performing almost 11,000 updates a day, we have around 50 > million > > > docs in the index (i understand we will need to shard) the core seg > will > > > get fragmented over a period of time. We will need to do optimize every > > few > > > days or once in a month; do you have any reason not to optimize the > core. > > > Please let me know. > > > > > > Thanks. > > > > > > On 11 November 2011 18:51, Mark Miller <markrmil...@gmail.com> wrote: > > > > > >> Do a you have something forcing you to optimize, or are you just doing > > it > > >> for the heck of it? > > >> > > >> On Nov 11, 2011, at 7:50 AM, Kalika Mishra wrote: > > >> > > >>> Hi, > > >>> > > >>> I would like to optimize solr core which is in Reader Writer mode. > > Since > > >>> the Solr cores are huge in size (above 100 GB) the optimization takes > > >> hours > > >>> to complete. > > >>> > > >>> When the optimization is going on say. on the Writer core, the > > >> application > > >>> wants to continue using the indexes for both query and write > purposes. > > >> What > > >>> is the best approach to do this. > > >>> > > >>> I was thinking of using a temporary index (empty core) to write the > > >>> documents and use the same Reader to read the documents. (Please note > > >> that > > >>> temp index and the Reader cannot be made Reader Writer as Reader is > > >> already > > >>> setup for the Writer on which optimization is taking place) But there > > >> could > > >>> be some updates to the temp index which I would like to get reflected > > in > > >>> the Reader. Whats the best setup to support this. > > >>> > > >>> Thanks, > > >>> Kalika > > >> > > >> - Mark Miller > > >> lucidimagination.com > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > -- > > > Thanks & Regards, > > > Kalika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Thanks & Regards, > Kalika > -- Thanks & Regards, Isan Fulia.
