Thanks Federico. I will look into this patch. We are using .94.6.1 version.
On Aug 29, 2013, at 8:37 AM, Federico Gaule <[email protected]> wrote: > In 0.94.11 Release, has been included an optimization for MultiGets: > https://issues.apache.org/jira/browse/HBASE-9087 > > What version have you deployed? > > > On 08/29/2013 01:29 AM, lars hofhansl wrote: >> A 1s SLA is tough in HBase (or any large memory JVM application). >> >> >> Maybe, if you presplit your table, play with JDK7 and the G1 collector, but >> nobody here will vouch for such an SLA in the 99th percentile. >> I heard some folks have experimented with 30GB heaps and G1 and have >> reported max GC times of 200ms, but I have not verified that. >> >> -- Lars >> >> >> >> ----- Original Message ----- >> From: Saurabh Yahoo <[email protected]> >> To: "[email protected]" <[email protected]> >> Cc: "[email protected]" <[email protected]> >> Sent: Wednesday, August 28, 2013 3:17 PM >> Subject: Re: experiencing high latency for few reads in HBase >> >> Hi Vlad, >> >> Thanks for your response. >> >> 1. Our SLA is less than one sec. we cannot afford latency more than 1 sec. >> >> We can increase heap size if that help, we have enough memory on server. >> What would be the optimal heap size? >> >> 2. Cache hit ratio is 95%. One thing I don't understand that we have >> allocated only 4gb for block cache out of 12gb. That left 8gb for rest of >> JVM. There is no write. Memcache is empty. Is 8gb not enough for hbase to >> process the requests? What are the most memory consuming objects in region >> server? >> >> 3. We will change the cf to IN_memory and report back performance difference. >> >> Thanks, >> Saurabh. >> >> On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov <[email protected]> >> wrote: >> >>> 1. 4 sec max latency is not that bad taking into account 12GB heap. It can >>> be much larger. What is your SLA? >>> 2. Block evictions is the result of a poor cache hit rate and the root >>> cause of a periodic stop-the-world GC pauses (max latencies >>> latencies you have been observing in the test) >>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, >>> 25% - permanent). Permanent part is for CF with >>> IN_MEMORY = true (you can specify this when you create CF). Block first >>> stored in 'young gen' space, then gets promoted to 'tenured gen' space >>> (or gets evicted). May be your 'perm gen' space is underutilized? This is >>> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the >>> space allocated for block cache - >>> there is no guarantee (as usual). If you don have in_memory column families >>> you may decrease >>> >>> >>> >>> Best regards, >>> Vladimir Rodionov >>> Principal Platform Engineer >>> Carrier IQ, www.carrieriq.com >>> e-mail: [email protected] >>> >>> ________________________________________ >>> From: Saurabh Yahoo [[email protected]] >>> Sent: Wednesday, August 28, 2013 5:10 AM >>> To: [email protected] >>> Subject: experiencing high latency for few reads in HBase >>> >>> Hi, >>> >>> We are running a stress test in our 5 node cluster and we are getting the >>> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 >>> million reads having latency more than 4 seconds. Can anyone provide the >>> insight what we can do to meet below second SLA for each and every read? >>> >>> We observe the following things - >>> >>> 1. Reads are evenly distributed among 5 nodes. CPUs remain under 5% >>> utilized. >>> >>> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block >>> cache got filled up but around 1gb remained free. There are a large number >>> of cache eviction. >>> >>> Questions to experts - >>> >>> 1. If there are still 1gb of free block cache available, why is hbase >>> evicting the block from cache? >>> >>> 4. We are seeing memory went up to 10gb three times before dropping sharply >>> to 5gb. >>> >>> Any help is highly appreciable, >>> >>> Thanks, >>> Saurabh. >>> >>> Confidentiality Notice: The information contained in this message, >>> including any attachments hereto, may be confidential and is intended to be >>> read only by the individual or entity to whom this message is addressed. If >>> the reader of this message is not the intended recipient or an agent or >>> designee of the intended recipient, please note that any review, use, >>> disclosure or distribution of this message or its attachments, in any form, >>> is strictly prohibited. If you have received this message in error, please >>> immediately notify the sender and/or [email protected] and delete >>> or destroy any copy of this message and its attachments. >
