Hi Ken,
Compactions are serviced by a thread which sleeps for a configurable interval
and then wakes to do work. As compaction requests are raised, they are queued
and the thread is signaled and wakes early. When a region server first starts
up, a limit is imposed on how many compaction requests can be serviced from the
queue before the thread goes back to sleep. The limit is gradually relaxed. The
net effect is to draw out compactions over an initial "safe mode". The reason
this is done is to limit load on DFS when the cluster is starting. Compactions
during this period often process flushes from the last shutdown and can well
trigger splits. Around the 0.19 timeframe I had a 25 node test cluster pushing
1000 regions per node. Startup compaction/split activity would crush DFS and
prevent successful (re)starts. The mechanism described here solved that issue.
- Andy
________________________________
From: Ken Weiner <[email protected]>
To: [email protected]
Sent: Wednesday, September 2, 2009 5:50:53 PM
Subject: Compactions no longer limited
What does it mean when the following INFO message appears in the
RegionServer logs? I generally see that message show up on all
RegionServers at the same time.
org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
limited
Thanks, Ken
