On 10/23/2014 2:44 AM, Giovanni Bricconi wrote: > My user interface shows some boxes to describe results categories. After > half a day of small updates and delete I noticed with various queries that > the boxes started swapping while browsing. > For sure I relied too much in getting the same results on each call, now > I'm keeping the categories order in request parameters to avoid the blink > effect while browsing. > > The optimize process is really slow, and I can't use it. Since I have many > other parameters that should be carried along the request to make sure that > the navigation is consistent, I would like to understand if is there a > setup that can limit the idf change and keep it low enough > > I tried with > > <indexConfig> > > <mergeFactor>5</mergeFactor> > > </indexConfig> > In solrconfig but this morning /solr/admin/cores?action=STATUS still > reports a number of segments above ten for all cores of the shard. (I'm > sure I have reloaded each core after changing the value) > > Now I'm trying with expungeDeletes called from solrj, but still I don't see > the segment count decrease
It's completely normal to have more segments than the mergeFactor. Think about this scenario with a mergeFactor of 5: You index five segments. They get merged to one segment. Let's say that this happens a total of four times, so you've indexed a total of 20 segments and merging has reduced that to four larger segments. Let's say that you now index four more segments. You'll be completely stable with eight segments. If you index another one, that will result in a fifth larger segment. This sets conditions up just right for another merge -- to one even larger segment. This represents three "levels" of merging, and there can be even more levels, each of which can have four segments and remain stable. Starting at the last state I described, if you then indexed 24 more segments, you'd have a stable index with a total of nine segments - four of them would be "normal" sized, four of them would be about five times normal size, and the first one would be about 25 times normal size. The Solr default for the merge policy in all recent versions is TieredMergePolicy, and this can make things slightly more complicated than I've described, because it can merge *any* segments, not just those indexed sequentially, and I believe that it can delay merging until the right number of segments with suitable characteristics appear. I've got merge settings equivalent to a mergeFactor of 35, but I regularly see the segment count approach 100, and there's absolutely nothing wrong with my merging. If I understand it correctly, expungeDeletes will not decrease the segment count. It will simply rewrite segments that have deleted documents so there are none. I'm not 100% sure that I know exactly what expungeDeletes does, though. Thanks, Shawn
