Have you taken a look at the logs on the RegionServers during the period? One possibility is compactions happening organically. If you were sustaining a certain level of writes most of the time, I could maybe see that every 3 hours enough store files build up to require compactions.
There's nothing else automated in HDFS or HBase that I could see causing this. On Fri, Dec 13, 2013 at 3:07 PM, Patrick Schless <[email protected]>wrote: > CDH4.1.2 > HBase 0.92.1 > HDFS 2.0.0 > > > Every 3 hours, our production HBase cluster does something that causes all > the data nodes to have a sustained spike in CPU/network/disk. The spike > lasts about 30 mins, and during this time the cluster has greatly increased > latencies for our typical application usage. > > I can't find anything in our application that would have such a periodic > and significant behavior. Is there anything that HBase/HDFS might be doing > on it's own that would cause this? We're on the default schedule for major > compactions, but I thought that was daily. > > Any ideas what could be causing this? > > Thanks, > > Patrick >
