Re: how to optimize for heavy writes scenario

First please try out stack's suggestion, all good ones. And some supplement: since all disks in use are HDD w/ normal IO capability, it's important to control big IO rate like flush and compaction. Try below features out: 1. HBASE-8329 : Limit

Re: Parallel range scanners, not getting all rows.

Yes, i already did. Den 17 mars 2017 22:40 skrev "Ted Yu" : > Please also notify AsyncHBase mailing list, if you haven't done so. > > 2017-03-17 9:01 GMT-07:00 Kristoffer Sjögren : > > > Thanks Ted, that was indeed the problem. > > > > 2017-03-17 4:29

Re: how to optimize for heavy writes scenario

>> In my opinion, 1M/s input data will result in only 70MByte/s write Times 3 (default HDFS replication factor) Plus ... Do not forget about compaction read/write amplification. If you flush 10 MB and your max region size is 10 GB, with default min file to compact (3) your amplification is 6-7

Re: Parallel range scanners, not getting all rows.

Please also notify AsyncHBase mailing list, if you haven't done so. 2017-03-17 9:01 GMT-07:00 Kristoffer Sjögren : > Thanks Ted, that was indeed the problem. > > 2017-03-17 4:29 GMT+01:00 Ted Yu : > > Have you considered the empty start row ? > > > >

Re: how to optimize for heavy writes scenario

Hey Hef, What is the memstore size setting(how much heap is it allowed) that you have on that cluster? What is your region count per node? Are you writing evenly across all those regions or are only a few regions active per region server at a time? Can you paste your GC settings that you are

Re: Optimizations for a Read-only database

Thanks Anoop. We'll be sure to keep it below 80% From: Anoop John Sent: Friday, March 17, 2017 12:49:30 PM To: user@hbase.apache.org Subject: Re: Optimizations for a Read-only database >From HBase server perspective we need restrict

Re: Optimizations for a Read-only database

>From HBase server perspective we need restrict memstore size + block cache size to be max 80%. And memstore size alone can go down to 5% if am not wrong. We need to be careful when using G1 and giving this 80%. The cache will be mostly full as u said it will be read workload. Making ur working

Re: how to optimize for heavy writes scenario

On Fri, Mar 17, 2017 at 9:31 AM, Hef wrote: > Hi group, > I'm using HBase to store large amount of time series data, the usage case > is heavy on writes then reads. My application stops at writing 600k > requests per second and I can't tune up for better tps. > > Hardware:

Re: Optimizations for a Read-only database

I'll go through these recommendations, Kevin. Thanks a lot From: Kevin O'Dell Sent: Friday, March 17, 2017 10:55:49 AM To: user@hbase.apache.org Subject: Re: Optimizations for a Read-only database Hi Jeff, You can definitely lower the

Re: Optimizations for a Read-only database

Sorry for the Spam, As a follow up, if you are using large heaps with G1GC(~100GB) you can set memstore to .1 and block cache to .8, though there might be a barrier for how much you are allowed to set the systems to use. Leaving 10GB for regular HBase operations is MORE than enough though. On

Re: Optimizations for a Read-only database

Hi Jeff, You can definitely lower the memstore, the last time I looked there it had to be set to .1 at lowest it could go. I would not recommend disabling compactions ever, bad things will occur and it can end up impacting your read performance greatly. I would recommend looking at the Intel

Re: Optimizations for a Read-only database

Hi Default memstore is 0.42 of the heap size. You can just make it 0.1 and rest give to block cache. It will ensure that you have maximum data in the block cache. It is better for reads. In case of latest trunk you can even use bucket cache in offheap mode but as it is not available as a release

Optimizations for a Read-only database

We're creating a readonly database and would like to know the recommended optimizations we could do. We'd be loading data via direct write to HFiles. One thing i could immediately think of is to eliminate the memory for Memstore. What is the minimum that we could get away with? How about

Re: how to optimize for heavy writes scenario

Hi, Its a little bit hard to tell, assuming that you have tuned the number of regions and already looked into common perf issues like networking or any other issue with HDFS, you should probably consider to try HBase 1.2 (see HBASE-15146) and distros with other fixes like HBASE-17072 and

Re: Parallel range scanners, not getting all rows.

Thanks Ted, that was indeed the problem. 2017-03-17 4:29 GMT+01:00 Ted Yu : > Have you considered the empty start row ? > > 2017-03-16 10:51 GMT-07:00 Kristoffer Sjögren : > >> Thanks Ted, I have posted the question to AsyncHBase. >> >> But, if I may ask. Is

Re: Incompatible ProtocolBuffers version for HBase-client 1.0

I change my application to use ProtocolBuffers 2.5 then this issue resolved. On Fri, Mar 17, 2017 at 12:36 PM, Hef wrote: > Hi group, > I have a problem using ProtocolBuffers 3 in my application with CDH5.6 > HBase 1.0. > > When creating BufferedMutator and flush data