[
https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-12270:
--------------------------------
Attachment: HBASE-12270-0.98-v1.diff
[~apurtell]
Update the patch for 0.98.
The cache data write in bucket cache is asynchronous and the bucket cache will
abandon the data to cache if the write queue is full. For we use the disk as
bucket cache in the test, the write of data will be slow, which may make the
queue is full and data are not cached in the bucket cache.
Changes: enlarge the write thread number and write queue size.
Please help to trigger the ci test for 0.98 and 0.98-on-Hadoop-1.1. Thanks~
PS: Using MockedBucketCache with wait_when_cache=true is another option, but
the DEFAULT_CACHE_WAIT_TIME (50ms) is too small.
> A bug in the bucket cache, with cache blocks on write enabled
> -------------------------------------------------------------
>
> Key: HBASE-12270
> URL: https://issues.apache.org/jira/browse/HBASE-12270
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.11, 0.98.6.1
> Environment: I can reproduce it on a simple 2 node cluster, one
> running the master and another running a RS. I was testing on ec2.
> I used the following configurations for the cluster.
> hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G
> -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog
> gc:/tmp/hbase-regionserver-gc.log
> hbase-site:
> hbase.bucketcache.ioengine=offheap
> hbase.bucketcache.size=4196
> hbase.rs.cacheblocksonwrite=true
> hfile.block.index.cacheonwrite=true
> hfile.block.bloom.cacheonwrite=true
> Reporter: Khaled Elmeleegy
> Assignee: Liu Shaohui
> Priority: Critical
> Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
>
> Attachments: HBASE-12270-0.98-v1.diff, HBASE-12270-v1.diff,
> HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java
>
>
> In my experiments, I have writers streaming their output to HBase. The reader
> powers a web page and does this scatter/gather, where it reads 1000 keys
> written last and passes them the the front end. With this workload, I get the
> exception below at the region server. Again, I am using HBAse (0.98.6.1). Any
> help is appreciated.
> 2014-10-10 15:06:44,173 ERROR
> [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:236)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:176)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3758)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
