Lookback is a good idea: you could at least gather statistics and assess, later, whether good merges had been selected, and maybe play "what if" games to explore if different merge selections would have resulted in less copying.
A time-based MergeScheduler would make sense: e.g., it would allow small merges to run any time, but big ones must wait until "after hours". Also, RateLimitedDirWrapper can be used to limit IO impact of ongoing merges. It's like a naive ionice, for merging. Mike McCandless http://blog.mikemccandless.com On Mon, Jul 8, 2013 at 10:41 PM, Otis Gospodnetic <otis.gospodne...@gmail.com> wrote: > Hi, > > I was (re-re-re-re)-reading Mike's post about Lucene segment merges - > http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html > > Mike mentioned lookhead as something that could possibly yield more > optimal merges. > > But what about lookback? :) > > What if some sort of stats were kept about about which segments were > picked for merges? With some sort of stats in hand, could one look > back and, knowing what happened after those merges, evaluate if more > optimal merge choices could have been made and then use that "next > time"? > > Also, what about time of day and query rates? Very often search > traffic follows the wave pattern, which could mean that more > aggressive merging could be done during periods with lower query > rates... or maybe during that time more segments could be allowed to > live in the index, assuming that after allowing that for some time, > the subsequent merge could be bigger/more thorough, so to speak. > > Thoughts? > > Otis > -- > Solr & ElasticSearch Support -- http://sematext.com/ > Performance Monitoring -- http://sematext.com/spm > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
