Ah, sorry about the attachment (didn't realize they weren't allowed). Here's the picture I was trying to attach: http://www.plainlystated.com/hbase_major_compactions.png
It sounds like you're right, Vladimir, about the compaction storm, though I don't understand why it's about every three hours instead of every day. In the book [1] I see the suggestion that they be managed manually. I don't see, however, any advice on what do do after turning auto-compaction off. Are there best practices around scheduling and monitoring the process? Thanks, Patrick [1] http://hbase.apache.org/book/important_configurations.html#managed.compactions On Fri, Dec 13, 2013 at 6:07 PM, Vladimir Rodionov <[email protected]>wrote: > You forgot to mention that it won't go through because Apache mail server > blocks attachments. > What Patrick is observing is called compaction storms. The best way (as > since its 0.92.x) is to disable automatic compactions > and manage them manually (see HBase book how to do this). > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: [email protected] > > ________________________________________ > From: Ted Yu [[email protected]] > Sent: Friday, December 13, 2013 3:33 PM > To: [email protected] > Cc: user > Subject: Re: 3-Hour Periodic Network/CPU/Disk/Latency Spikes > > Patrick: > Attachment didn't go through. > > Cheers > > On Dec 13, 2013, at 3:18 PM, Patrick Schless <[email protected]> > wrote: > > > Very interesting, I think we may be on to something. I grabbed all the > timestamps for major compactions completing and put them on a graph (see > attached). Each horizontal line is an individual server, and the dots are > when compactions complete. Each server clearly has a cluster of compactions > about every 3 hours, and several of the servers are aligned such that they > are compacting at the same time. > > > > Should we be managing these compactions ourselves? Would it make more > sense to have them less frequently (but presumably more expensive), or > closer together? > > > > Thanks, > > Patrick > > > > > > On Fri, Dec 13, 2013 at 2:19 PM, Bryan Beaudreault < > [email protected]> wrote: > >> Have you taken a look at the logs on the RegionServers during the > period? > >> > >> One possibility is compactions happening organically. If you were > >> sustaining a certain level of writes most of the time, I could maybe see > >> that every 3 hours enough store files build up to require compactions. > >> > >> There's nothing else automated in HDFS or HBase that I could see causing > >> this. > >> > >> On Fri, Dec 13, 2013 at 3:07 PM, Patrick Schless > >> <[email protected]>wrote: > >> > >> > CDH4.1.2 > >> > HBase 0.92.1 > >> > HDFS 2.0.0 > >> > > >> > > >> > Every 3 hours, our production HBase cluster does something that > causes all > >> > the data nodes to have a sustained spike in CPU/network/disk. The > spike > >> > lasts about 30 mins, and during this time the cluster has greatly > increased > >> > latencies for our typical application usage. > >> > > >> > I can't find anything in our application that would have such a > periodic > >> > and significant behavior. Is there anything that HBase/HDFS might be > doing > >> > on it's own that would cause this? We're on the default schedule for > major > >> > compactions, but I thought that was daily. > >> > > >> > Any ideas what could be causing this? > >> > > >> > Thanks, > >> > > >> > Patrick > >> > > > > > Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or [email protected] and > delete or destroy any copy of this message and its attachments. >
