[
https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15080717#comment-15080717
]
deepankar commented on HBASE-15064:
-----------------------------------
Pinging [~ishanc] he can add anything I miss.
Our use case was more of gets (working for RocketFuel Inc), singular workload
with response times requirement to be < 40ms (~16kb response size, ~20k - 30k
reqs per node per sec). We are encountering problems with frequent(~ish) Mixed
G1 GCs due to the heap block cache churning (eviction size matching the Mixed
GC clean up size).
We decided to experiment with Off heap Bucket Cache by running hbase 1.1.2
version to reduce mixed GCs. We did see a decrease in mixed GC frequency but
the young gcs started to become more rampant, so we decided to use the patches
from HBASE-11425 to decrease them. I ported most of the required patches from
trunk onto our internal branch and was giving it a spin on a single region
server along with our stable 1.1 version regionservers. That is where I was
encountering some of these issues. Initial results did not decrease young GCs
that much (our client does not support the cells by payload yet and might also
needs some more investigation from our side for the reason). This is all from
our side until now. Can definitely help with anything from our iside related
testing or reporting results.
Thanks
> Possbile Bug in MultiByteBuffer
> -------------------------------
>
> Key: HBASE-15064
> URL: https://issues.apache.org/jira/browse/HBASE-15064
> Project: HBase
> Issue Type: Bug
> Components: io
> Affects Versions: 2.0.0
> Reporter: deepankar
> Priority: Critical
>
> While running the newer patches on our production system, I saw this error
> come couple of times
> {noformat}
> ipc.RpcServer: Unexpected throwable object
> 2016-01-01 16:42:56,090 ERROR
> [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402)
>
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517)
>
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code
> {code}
> if (this.curItem.remaining() == 0) {
> if (items.length - 1 == this.curItemIndex) {
> // means cur item is the last one and we wont be able to read a long.
> Throw exception
> throw new BufferUnderflowException();
> }
> this.curItemIndex++;
> this.curItem = this.items[this.curItemIndex];
> }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make
> sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining()
> == 0)}}. This logic is repeated may make sense abstract to a new function if
> we plan to change to {{if}} to {{while}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
