> From: Daniel Einspanjer
>
> Mozilla recently had to turn off
> automatic scheduled major compactions because if they happen
> during our peak load time, they cause an IO delay that can
> destabilize our client applications.
>
> We need to put a replacement system in place as soon as
> possible that can initiate the major compaction and ensure
> that it successfully completed (See HBASE-2770).
> It would be nice if the system could actually determine the
> health (i.e. does it need to happen right now?) and
> dynamically schedule it based on load
A suggestion for whomever might want to pick up the work:
First, fix HBASE-2770.
Then, it's possible to trigger major compaction on a single region using the
shell, therefore trivially programmatically via hbase shell/jruby. If Mozilla
is publishing DFS and HBase metrics via Ganglia (I suspect so, and app metrics
might be useful also) then getting a sense of current system load wouldn't be
difficult: connect to gmond, parse the metrics XML tree. Write a script that
runs major compaction against each region -- a few in parallel, determined by
config and current load estimates -- and defers the next or reduces the
compaction concurrency if load estimates across metrics of interest as reported
by gmond are too high.
- Andy
