Your post is missing the most important configurations, mainly the region server heap size and GC configs.
Also, how much of those 300GB do you need to serve? Does the working dataset fit in cache? J-D On Sun, Mar 11, 2012 at 12:39 PM, Антон Лыска <[email protected]> wrote: > Hi guys! > > I have a little instance of hbase cluster with only 2 machines (8core cpu, > 12G mem, 3*1GB hdd on each machine). > I use cloudera's cdh3u1 distro. > Cluster serves two tables and total data size is about 300 GB with 300 > regions. > The average Get time is usually 20-50ms, but sometimes it rises up to > 500-800ms which is unacceptable. > > Gets per day: 13*10^6 > Puts per day: 11*10^6 > Deletes per day: 2*10^6 > > My conf is: > <configuration> > .... > <property> > <name>dfs.replication</name> > <value>2</value> > </property> > > <property> > <name>hbase.regionserver.handler.count</name> > <value>50</value> > </property> > > <property> > <name>hbase.hregion.majorcompaction</name> > <value>86400000000</value> > </property> > </configuration> > > My scheme is: > {NAME => 'table1', MAX_FILESIZE => '536870912', FAMILIES => [{NAME => 'c', > BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', > VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '16384', IN_MEMORY => > 'false', BLOCKCACHE => 'true'}, {NAME => 'p', BLOOMFILTER => 'NONE', > REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => > 'true'}, {NAME => 's', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', > COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} > > {NAME => 'table2', FAMILIES => [{NAME => 'n', BLOOMFILTER => 'NONE', > REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', TTL => > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => > 'true'}]} > > I disabled major compaction by setting a big value, and run it manually > each day at 3:00am (server is least loaded at that time). > Get time usually starts increasing at around 23:00-24:00. > Once hbase is restarted, Get time returns to 20ms. > What it can be? what options should I set to avoid this issue? > > Also I have installed ganglia, but I haven't seen anything strange there. > > Thank you in advance! > > Best regards, Anton.
