bq. Output is written to either Hbase Looks like Akhtar wants to boost write performance to HBase. MapReduce over snapshot files targets higher read throughput.
Cheers On Sat, Jan 4, 2014 at 12:55 PM, Vladimir Rodionov <vrodio...@carrieriq.com>wrote: > You cay try MapReduce over snapshot files > https://issues.apache.org/jira/browse/HBASE-8369 > > but you will need to patch 0.94. > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: vrodio...@carrieriq.com > > ________________________________________ > From: Akhtar Muhammad Din [akhtar.m...@gmail.com] > Sent: Saturday, January 04, 2014 12:44 PM > To: user@hbase.apache.org > Subject: Re: Hbase Performance Issue > > im using CDH 4.5: > Hadoop: 2.0.0-cdh4.5.0 > HBase: 0.94.6-cdh4.5.0 > > Regards > > > On Sun, Jan 5, 2014 at 1:24 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > What version of HBase / hdfs are you running with ? > > > > Cheers > > > > > > > > On Sat, Jan 4, 2014 at 12:17 PM, Akhtar Muhammad Din > > <akhtar.m...@gmail.com>wrote: > > > > > Hi, > > > I have been running a map reduce job that joins 2 datasets of 1.3 and 4 > > GB > > > in size. Joining is done at reduce side. Output is written to either > > Hbase > > > or HDFS depending upon configuration. The problem I am having is that > > Hbase > > > takes about 60-80 minutes to write the processed data, on the other > hand > > > HDFS takes only 3-5 mins to write the same data. I really want to > improve > > > the Hbase speed and bring it down to 1-2 min. > > > > > > I am using amazon EC2 instances, launched a cluster of size 3 and later > > 10, > > > have tried both c3.4xlarge and c3.8xlarge instances. > > > > > > I can see significant increase in performance while writing to HDFS as > i > > > use cluster with more nodes, having high specifications, but in the > case > > of > > > Hbase there was no significant change in performance. > > > > > > I have been going through different posts, articles and have read Hbase > > > book to solve the Hbase performance issue but have not been able to > > succeed > > > so far. > > > Here are the few things i have tried out so far: > > > > > > *Client Side* > > > - Turned off writing to WAL > > > - Experimented with write buffer size > > > - Turned off auto flush on table > > > - Used cache, experimented with different sizes > > > > > > > > > *Hbase Server Side* > > > - Increased region servers heap size to 8 GB > > > - Experimented with handlers count > > > - Increased Memstore flush size to 512 MB > > > - Experimented with hbase.hregion.max.filesize, tried different sizes > > > > > > There are many other parameters i have tried out following the > > suggestions > > > from different sources, but nothing worked so far. > > > > > > Your help will be really appreciated. > > > > > > -- > > > Regards > > > Akhtar Muhammad Din > > > > > > > > > -- > Regards > Akhtar Muhammad Din > > Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or notificati...@carrieriq.com and > delete or destroy any copy of this message and its attachments. >
