Thanks for showing I wasn't completely crazy to think this made sense, Mike.
I added:
https://issues.apache.org/jira/browse/LUCENE-5134
https://issues.apache.org/jira/browse/LUCENE-5135
Otis
On Mon, Jul 15, 2013 at 1:28 PM, Michael McCandless
luc...@mikemccandless.com wrote:
Lookback is a good idea: you could at least gather statistics and
assess, later, whether good merges had been selected, and maybe play
what if games to explore if different merge selections would have
resulted in less copying.
A time-based MergeScheduler would make sense: e.g., it would allow
small merges to run any time, but big ones must wait until after
hours.
Also, RateLimitedDirWrapper can be used to limit IO impact of ongoing
merges. It's like a naive ionice, for merging.
Mike McCandless
http://blog.mikemccandless.com
On Mon, Jul 8, 2013 at 10:41 PM, Otis Gospodnetic
otis.gospodne...@gmail.com wrote:
Hi,
I was (re-re-re-re)-reading Mike's post about Lucene segment merges -
http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
Mike mentioned lookhead as something that could possibly yield more
optimal merges.
But what about lookback? :)
What if some sort of stats were kept about about which segments were
picked for merges? With some sort of stats in hand, could one look
back and, knowing what happened after those merges, evaluate if more
optimal merge choices could have been made and then use that next
time?
Also, what about time of day and query rates? Very often search
traffic follows the wave pattern, which could mean that more
aggressive merging could be done during periods with lower query
rates... or maybe during that time more segments could be allowed to
live in the index, assuming that after allowing that for some time,
the subsequent merge could be bigger/more thorough, so to speak.
Thoughts?
Otis
--
Solr ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
