[jira] [Comment Edited] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406670#comment-16406670
]
stack edited comment on HBASE-19639 at 3/20/18 4:53 PM:
bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the
client. I see them in the MapReduce task logs, not in hbase regionserver logs.
Let me see if I can put up a metric or log message so we we can see when a
Region is struggling...
bq. and most flushes are when the memstore size is ~128MB, these are good news
I guess.
Yes. This is good.
Let me fix the log line.
bq. Anyway, I would strongly suggest to compare performance to a setting where
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has
been enabled always. You folks think it should be off ([~anastas] you were
looking at this?) by default? It might make sense when G1GC is enabled but
proving this generally would be a bit of a project.
was (Author: stack):
bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the
client. I see them in the MapReduce task logs, not in hbase regionserver logs.
Let me see if I can put up a metric or log message so we we can see when a
Region is struggling...
bq. and most flushes are when the memstore size is ~128MB, these are good news
I guess.
Yes. This is good.
Let me fix the log line.
bq. Anyway, I would strongly suggest to compare performance to a setting where
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has
bee
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
>
[jira] [Comment Edited] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406149#comment-16406149
]
Anoop Sam John edited comment on HBASE-19639 at 3/20/18 11:14 AM:
--
Ya seems pool is there and the log formatting having an issue
{code}
LOG.info("{} allocating {} MemStoreChunkPool with chunk size {}, max count {},
" +
"initial count {}", label, StringUtils.byteDesc(chunkSize),
maxCount, initialCount);
{code}
It should have been
{code}
LOG.info("Allocating {} MemStoreChunkPool with chunk size {}, max count {}, " +
"initial count {}", label, StringUtils.byteDesc(chunkSize),
maxCount, initialCount);
{code}
Log should be like
{code}
regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2
MB, max count 2830, initial count 0
regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size
204.80 KB, max count 3145, initial count 0
{code}
Good observation.
was (Author: anoop.hbase):
Ya seems pool is there and the log formatting having an issue
LOG.info("{} allocating {} MemStoreChunkPool with chunk size {}, max count {},
" +
"initial count {}", label, StringUtils.byteDesc(chunkSize),
maxCount, initialCount);
It should have been
LOG.info("Allocating {} MemStoreChunkPool with chunk size {}, max count {}, " +
"initial count {}", label, StringUtils.byteDesc(chunkSize),
maxCount, initialCount);
Log should be like
{code}
regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2
MB, max count 2830, initial count 0
regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size
204.80 KB, max count 3145, initial count 0
{code}
Good observation.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641,
