You forgot to mention that it won't go through because Apache mail server blocks attachments. What Patrick is observing is called compaction storms. The best way (as since its 0.92.x) is to disable automatic compactions and manage them manually (see HBase book how to do this).
Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com ________________________________________ From: Ted Yu [yuzhih...@gmail.com] Sent: Friday, December 13, 2013 3:33 PM To: user@hbase.apache.org Cc: user Subject: Re: 3-Hour Periodic Network/CPU/Disk/Latency Spikes Patrick: Attachment didn't go through. Cheers On Dec 13, 2013, at 3:18 PM, Patrick Schless <patrick.schl...@gmail.com> wrote: > Very interesting, I think we may be on to something. I grabbed all the > timestamps for major compactions completing and put them on a graph (see > attached). Each horizontal line is an individual server, and the dots are > when compactions complete. Each server clearly has a cluster of compactions > about every 3 hours, and several of the servers are aligned such that they > are compacting at the same time. > > Should we be managing these compactions ourselves? Would it make more sense > to have them less frequently (but presumably more expensive), or closer > together? > > Thanks, > Patrick > > > On Fri, Dec 13, 2013 at 2:19 PM, Bryan Beaudreault <bbeaudrea...@hubspot.com> > wrote: >> Have you taken a look at the logs on the RegionServers during the period? >> >> One possibility is compactions happening organically. If you were >> sustaining a certain level of writes most of the time, I could maybe see >> that every 3 hours enough store files build up to require compactions. >> >> There's nothing else automated in HDFS or HBase that I could see causing >> this. >> >> On Fri, Dec 13, 2013 at 3:07 PM, Patrick Schless >> <patrick.schl...@gmail.com>wrote: >> >> > CDH4.1.2 >> > HBase 0.92.1 >> > HDFS 2.0.0 >> > >> > >> > Every 3 hours, our production HBase cluster does something that causes all >> > the data nodes to have a sustained spike in CPU/network/disk. The spike >> > lasts about 30 mins, and during this time the cluster has greatly increased >> > latencies for our typical application usage. >> > >> > I can't find anything in our application that would have such a periodic >> > and significant behavior. Is there anything that HBase/HDFS might be doing >> > on it's own that would cause this? We're on the default schedule for major >> > compactions, but I thought that was daily. >> > >> > Any ideas what could be causing this? >> > >> > Thanks, >> > >> > Patrick >> > > Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
