[
https://issues.apache.org/jira/browse/HBASE-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396482#comment-15396482
]
huaxiang sun commented on HBASE-16293:
--------------------------------------
Spent some time internally with Matteo, we found the issue.
The issue is that compaction is disabled for the table.
hbase.hstore.blockingStoreFiles is 12, and the test tries to create 20 store
files. When 12 is reached, it tries to compact but compaction is disabled. Then
the memstore size is exceeded and region server starts to return
RegionTooBusyException to the client. The client will retry and stuck forever.
Looking into the logic of the unitest, it does not assume how many store files
it needs to create.
It is related with HBASE-16235.
2016-07-27 13:06:18,272 INFO [hconnection-0x5b178b4b-shared-pool15-t22]
client.AsyncProcess$AsyncRequestFutureImpl(1266): #26, table=test,
attempt=13/36 failed=4468ops, last exception:
org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
regionName=test,,1469649897929.2e3e92fa2512073f502609defa3845f8.,
server=172.16.1.53,53409,1469649884198, memstoreSize=1363248,
blockingMemStoreSize=100000
at
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3719)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2945)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2896)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:878)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:840)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2359)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:38447)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
on 172.16.1.53,53409,1469649884198, tracking started null, retrying
after=20180ms, replay=4468ops
> TestSnapshotFromMaster#testSnapshotHFileArchiving flakey
> --------------------------------------------------------
>
> Key: HBASE-16293
> URL: https://issues.apache.org/jira/browse/HBASE-16293
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0
> Reporter: huaxiang sun
> Assignee: huaxiang sun
>
> Got the following stack trace for this failure, not sure if it is related
> with HBASE-9072
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 336.042 sec
> <<< FAILURE! - in
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster)
> Time elapsed: 303.771 sec <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 300000
> milliseconds
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1810)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1784)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1860)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:241)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:979)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:576)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:2002)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1979)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1967)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1945)
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297)
> Results :
> Tests in error:
> TestSnapshotFromMaster.testSnapshotHFileArchiving:297->Object.wait:-2 ยป
> TestTimedOut
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
