Re: how to optimize for heavy writes scenario

2017-03-23 Thread Hef
Hi Allan, I didn't see any improvement either after decrease the compact threads number or increase the memstore flush size. :( How much write tps can your cluster handler per region server? Thanks Hef On Wed, Mar 22, 2017 at 10:07 AM, Allan Yang wrote: >

Re: how to optimize for heavy writes scenario

2017-03-21 Thread Allan Yang
hbase.regionserver.thread.compaction.small = 30 Am I seeing it right? You used 30 threads for small compaction. That's too much. For heavy writes scenario, you used too much resource to do compactions. We also have OpenTSDB running on HBase in our company. IMHO, the conf should like this:

Re: how to optimize for heavy writes scenario

2017-03-21 Thread Dejan Menges
Regarding du -sk, take a look here https://issues.apache.org/jira/browse/HADOOP-9884 Also hardly waiting for this one to be fixed. On Tue, Mar 21, 2017 at 4:09 PM Hef wrote: > There were several curious things we have observed: > One the region servers, there were

Re: how to optimize for heavy writes scenario

2017-03-21 Thread Hef
There were several curious things we have observed: One the region servers, there were abnormal much more reads than writes: Device:tpskB_read/skB_wrtn/skB_readkB_wrtn sda 608.00 6552.00 0.00 6552 0 sdb 345.00

Re: how to optimize for heavy writes scenario

2017-03-17 Thread Yu Li
First please try out stack's suggestion, all good ones. And some supplement: since all disks in use are HDD w/ normal IO capability, it's important to control big IO rate like flush and compaction. Try below features out: 1. HBASE-8329 : Limit

Re: how to optimize for heavy writes scenario

2017-03-17 Thread Vladimir Rodionov
>> In my opinion, 1M/s input data will result in only 70MByte/s write Times 3 (default HDFS replication factor) Plus ... Do not forget about compaction read/write amplification. If you flush 10 MB and your max region size is 10 GB, with default min file to compact (3) your amplification is 6-7

Re: how to optimize for heavy writes scenario

2017-03-17 Thread Kevin O'Dell
Hey Hef, What is the memstore size setting(how much heap is it allowed) that you have on that cluster? What is your region count per node? Are you writing evenly across all those regions or are only a few regions active per region server at a time? Can you paste your GC settings that you are

Re: how to optimize for heavy writes scenario

2017-03-17 Thread Stack
On Fri, Mar 17, 2017 at 9:31 AM, Hef wrote: > Hi group, > I'm using HBase to store large amount of time series data, the usage case > is heavy on writes then reads. My application stops at writing 600k > requests per second and I can't tune up for better tps. > > Hardware:

Re: how to optimize for heavy writes scenario

2017-03-17 Thread Esteban Gutierrez
Hi, Its a little bit hard to tell, assuming that you have tuned the number of regions and already looked into common perf issues like networking or any other issue with HDFS, you should probably consider to try HBase 1.2 (see HBASE-15146) and distros with other fixes like HBASE-17072 and