[
https://issues.apache.org/jira/browse/HBASE-19647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304819#comment-16304819
]
stack commented on HBASE-19647:
-------------------------------
Now the reason for the RegionTooBusyException and the region comes out in the
RetriesExhaustedException....
Error:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 144
actions: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit,
regionName=IntegrationTestBigLinkedList,\xC3\x0C0\xC3\x0C0\xC3\x00,1514409144553.57c92c6ed260dfa2bdd4666b19cddfc7.,
server=ve0534.halxg.cloudera.com,16020,1514408601293,
memStoreSize.heapSize=537620032, blockingMemStoreHeapSize=536870912 at
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4184)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3806)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3746)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) : 144
times, servers with issues: ve0534.halxg.cloudera.com,16020,1514408601293 at
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:491)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:268)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:181)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:143)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:538)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:464)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:399)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> Logging cleanups; emit regionname when RegionTooBusyException inside
> RetriesExhausted... make netty connect/disconnect TRACE-level
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-19647
> URL: https://issues.apache.org/jira/browse/HBASE-19647
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 19647.patch
>
>
> In MR failures, i see this:
> Error:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 365 actions: RegionTooBusyException: 365 times, servers with issues:
> ve0534.halxg.cloudera.com,16020,1514392912363 at
> org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:491)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:268)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:225)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:541)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:464)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:399)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) [2017-12-27
> 09:09:11.570]Container killed by the ApplicationMaster. [2017-12-27
> 09:09:11.586]Container killed on request. Exit code is 143 [2017-12-27
> 09:09:11.600]Container exited with a non-zero exit code 143.
> Its missing the region name which is in the root exception,
> RegionTooBusyException -- we just skip it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
