im using CDH 4.5: Hadoop: 2.0.0-cdh4.5.0 HBase: 0.94.6-cdh4.5.0 Regards
On Sun, Jan 5, 2014 at 1:24 AM, Ted Yu <yuzhih...@gmail.com> wrote: > What version of HBase / hdfs are you running with ? > > Cheers > > > > On Sat, Jan 4, 2014 at 12:17 PM, Akhtar Muhammad Din > <akhtar.m...@gmail.com>wrote: > > > Hi, > > I have been running a map reduce job that joins 2 datasets of 1.3 and 4 > GB > > in size. Joining is done at reduce side. Output is written to either > Hbase > > or HDFS depending upon configuration. The problem I am having is that > Hbase > > takes about 60-80 minutes to write the processed data, on the other hand > > HDFS takes only 3-5 mins to write the same data. I really want to improve > > the Hbase speed and bring it down to 1-2 min. > > > > I am using amazon EC2 instances, launched a cluster of size 3 and later > 10, > > have tried both c3.4xlarge and c3.8xlarge instances. > > > > I can see significant increase in performance while writing to HDFS as i > > use cluster with more nodes, having high specifications, but in the case > of > > Hbase there was no significant change in performance. > > > > I have been going through different posts, articles and have read Hbase > > book to solve the Hbase performance issue but have not been able to > succeed > > so far. > > Here are the few things i have tried out so far: > > > > *Client Side* > > - Turned off writing to WAL > > - Experimented with write buffer size > > - Turned off auto flush on table > > - Used cache, experimented with different sizes > > > > > > *Hbase Server Side* > > - Increased region servers heap size to 8 GB > > - Experimented with handlers count > > - Increased Memstore flush size to 512 MB > > - Experimented with hbase.hregion.max.filesize, tried different sizes > > > > There are many other parameters i have tried out following the > suggestions > > from different sources, but nothing worked so far. > > > > Your help will be really appreciated. > > > > -- > > Regards > > Akhtar Muhammad Din > > > -- Regards Akhtar Muhammad Din
