If you're using Cloudera, you want to be on CDH3u3 because it has several HDFS performance fixes for low-latency reads.
That still doesn't address your 23:00-hour perf issue, but that's something that will help. On 3/11/12 3:39 PM, "Антон Лыска" <[email protected]> wrote: >Hi guys! > >I have a little instance of hbase cluster with only 2 machines (8core cpu, >12G mem, 3*1GB hdd on each machine). >I use cloudera's cdh3u1 distro. >Cluster serves two tables and total data size is about 300 GB with 300 >regions. >The average Get time is usually 20-50ms, but sometimes it rises up to >500-800ms which is unacceptable. > >Gets per day: 13*10^6 >Puts per day: 11*10^6 >Deletes per day: 2*10^6 > >My conf is: ><configuration> > .... ><property> ><name>dfs.replication</name> ><value>2</value> ></property> > ><property> ><name>hbase.regionserver.handler.count</name> > <value>50</value> ></property> > ><property> ><name>hbase.hregion.majorcompaction</name> > <value>86400000000</value> ></property> > </configuration> > >My scheme is: >{NAME => 'table1', MAX_FILESIZE => '536870912', FAMILIES => [{NAME => 'c', >BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', >VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '16384', IN_MEMORY => >'false', BLOCKCACHE => 'true'}, {NAME => 'p', BLOOMFILTER => 'NONE', >REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => >'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >'true'}, {NAME => 's', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', >COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => >'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} > >{NAME => 'table2', FAMILIES => [{NAME => 'n', BLOOMFILTER => 'NONE', >REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', TTL => >'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => >'true'}]} > >I disabled major compaction by setting a big value, and run it manually >each day at 3:00am (server is least loaded at that time). >Get time usually starts increasing at around 23:00-24:00. >Once hbase is restarted, Get time returns to 20ms. >What it can be? what options should I set to avoid this issue? > >Also I have installed ganglia, but I haven't seen anything strange there. > >Thank you in advance! > >Best regards, Anton.
