[
https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-12270:
--------------------------------
Attachment: HBASE-12270-0.98-v2.diff
[~apurtell]
Sorry for late rely.
In patch v2, set the write queue of bucket cache big enough for caching all the
block in the queue. Please kindly help to test in your env , for I can produce
the failure in my dev machine. Thanks.
> A bug in the bucket cache, with cache blocks on write enabled
> -------------------------------------------------------------
>
> Key: HBASE-12270
> URL: https://issues.apache.org/jira/browse/HBASE-12270
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.11, 0.98.6.1
> Environment: I can reproduce it on a simple 2 node cluster, one
> running the master and another running a RS. I was testing on ec2.
> I used the following configurations for the cluster.
> hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G
> -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog
> gc:/tmp/hbase-regionserver-gc.log
> hbase-site:
> hbase.bucketcache.ioengine=offheap
> hbase.bucketcache.size=4196
> hbase.rs.cacheblocksonwrite=true
> hfile.block.index.cacheonwrite=true
> hfile.block.bloom.cacheonwrite=true
> Reporter: Khaled Elmeleegy
> Assignee: Liu Shaohui
> Priority: Critical
> Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
>
> Attachments: HBASE-12270-0.98-v1.diff, HBASE-12270-0.98-v2.diff,
> HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch,
> TestHBase.java, TestKey.java
>
>
> In my experiments, I have writers streaming their output to HBase. The reader
> powers a web page and does this scatter/gather, where it reads 1000 keys
> written last and passes them the the front end. With this workload, I get the
> exception below at the region server. Again, I am using HBAse (0.98.6.1). Any
> help is appreciated.
> 2014-10-10 15:06:44,173 ERROR
> [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:236)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:176)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3758)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
