subject:"\[jira\] \[Commented\] \(HBASE\-19639\) ITBLL can't go big because RegionTooBusyException... Above memstore limit"

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407107#comment-16407107
 ] 

stack commented on HBASE-19639:
---

I fixed the logging formatting in a sub-issue.

G1GC as default is another project as is changing MSLAB default.

I had the memstore multiplier at 8. Default is 4. I set it back to the default 
for the last few runs.

I found this metric: 
{{hbase.regionserver.ipc.exceptions.RegionTooBusyException}}. Its a count of 
how many {{RegionTooBusyException}} are thrown on the server. It says none for 
the last few runs. It looks like this metric is hooked up properly since other 
exception counts come and go and I just noticed a non-zero on a new run. So, I 
think that I'm over this issue now. Return to defaults and fix ups in 
assignment done over last month or so allow ITBLL to make progress not stalled 
on RegionTooBusyException.

I'll resolve this issue as no longer valid. Lets carry on the tuning, etc., 
discussion over in the perf and scale issues, HBASE-20179 and HBASE-20188

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406740#comment-16406740
 ] 

Anoop Sam John commented on HBASE-19639:


For us the default memstore global size is 40% and BC is 40%.  When it  is R+W 
workload with this default configs, it is like the working size of the server 
is always 80%+ of Xmx.  The default InitialHeapOccupancyPercentage (IHOP) is 
45% for G1GC.  The larger value for this does not make G1GC meaningful IMHO. 
G1GC is basically for predictable GC pause.   With these setup, we will get 
more GC pauses..  So tuning Xmx and this IHOP is very key for G1GC based usage. 
 

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406670#comment-16406670
 ] 

stack commented on HBASE-19639:
---

bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the 
client. I see them in the MapReduce task logs, not in hbase regionserver logs. 
Let me see if I can put up a metric or log message so we we can see when a 
Region is struggling...

bq. and most flushes are when the memstore size is ~128MB, these are good news 
I guess.
Yes. This is good.

Let me fix the log line.

bq. Anyway, I would strongly suggest to compare performance to a setting where 
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has 
bee

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406149#comment-16406149
 ] 

Anoop Sam John commented on HBASE-19639:


Ya seems pool is there and the log formatting having an issue
LOG.info("{} allocating {} MemStoreChunkPool with chunk size {}, max count {}, 
" +
"initial count {}", label, StringUtils.byteDesc(chunkSize), 
maxCount, initialCount);
It should have been
LOG.info("Allocating {} MemStoreChunkPool with chunk size {}, max count {}, " +
"initial count {}", label, StringUtils.byteDesc(chunkSize), 
maxCount, initialCount);
Log should be like
{code}
regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 
MB, max count 2830, initial count 0
regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 
204.80 KB, max count 3145, initial count 0
{code}
Good observation.


> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406124#comment-16406124
 ] 

Eshcar Hillel commented on HBASE-19639:
---

I might have misinterpreted a line in the log. How would you interpret the 
following lines:
{code:java}
regionserver.ChunkCreator: data allocating 2 MB MemStoreChunkPool with chunk 
size 2830, max count 0, initial count {}
regionserver.ChunkCreator: index allocating 204.80 KB MemStoreChunkPool with 
chunk size 3145, max count 0, initial count {}
{code}
Maybe it means there is a pool but no chunk is pre-allocated?

Anyway, I would strongly suggest to compare performance to a setting where G1GC 
is used and no MSLAB. 

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406071#comment-16406071
 ] 

Anoop Sam John commented on HBASE-19639:


MSLAB with 100% pool is the default for 2.0.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-20 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406022#comment-16406022
 ] 

Eshcar Hillel commented on HBASE-19639:
---

Thanks [~stack]

I went over the log - didn't see any {{RegionTooBusyException}} and most 
flushes are when the memstore size is ~128MB, these are good news I guess...

However, here are 2 comments w.r.t settings:
1. you use mark GC - might consider G1GC
2. you are using MSLAB with no pool, better to use chunk pool with MSLAB, and 
best not to work with MSLAB at all ;)
 

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405829#comment-16405829
 ] 

stack commented on HBASE-19639:
---

[~eshcar] Here is a log from one of the cluster servers. Over the life of the 
job, regions went from zero to 15 regions.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405125#comment-16405125
 ] 

stack commented on HBASE-19639:
---

These are 31G heaps.

bq. That looks to be too high boss. IMO it should be less than 4x.. ...

Yeah. I was playing around trying to get rid of the RegionTooBusyException. 
Forgot to set it back down. Will do.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-19 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405105#comment-16405105
 ] 

Anoop Sam John commented on HBASE-19639:


bq.I don't follow. 4x is the default. You suggesting we do 2x?
No boss. Yes it is 4x by default.  I was just saying the fact that you are 
seeing the Exception because flush seems to be slower compared to the writes 
speed. 4x seems good enough
bq.we should be able to test in isolation? 
Yes we can I believe.
bq.This is default. I'm trying to run w/ all defaults. I will change default if 
we come up w/ something better ("...95% of 
hbase.regionserver.global.memstore.size..."). We don't block writes now. 
Instead we throw the RegionTooBusyException. We don't put anything in the logs. 
Let me fix this. Would be good to note when the Region is struggling... or at 
least let me see if a metric I could use.
I mean to ask what is the size in GBs for the global memstore size. This is 
default 40% of Xmx.  We throw RegionTooBusyException from region when we try 
write on a memstore with size >= 4 x flush size.But when the global 
memstore size (Sum of all memstores) above this barrier we will block writes. I 
can not see throwing Exception in such case.
bq.Hmm.. This last run of mine had it at 8 *. Let me set it do the default (4 * 
).
That looks to be too high boss.  IMO it should be less than 4x.. Because at 
each region level, we have 4x as max size and above which we throw Exception.  
The global memstore size barrier being  8x of (regions count * per region flush 
size) means theoretically we allow each region to grow upto 8x of flush size.  
To be clear I mean this. For my tests
regions count * per region flush size * 2 =  40% xmx
Ya when writes are becoming fast, we badly need faster flushes.


> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-19 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405040#comment-16405040
 ] 

stack commented on HBASE-19639:
---

[~eshcar] any chance you can attach here the entire log file of some RS? If 
possible from start-hbase point and then throughout the test. Thanks.

Will do. Let me get you a good one (Cluster doing another test at mo will 
be back).

[~anoop.hbase]
bq. In tests, we try flush to SDD or HDD boss?

HDD

bq. Now I can see the issue you face is the exception because memstore size is 
4x larger than flush size.

I don't follow. 4x is the default. You suggesting we do 2x?

bq. But with compacting memstore, it will become fresh CSLM again and so very 
fast writes.

Before in-memory compaction, we used to snapshot; move aside current CSLM 
taking no more writes to it and replace it with a new one so new writes 
would go fast again.

For your #1 and #2, we should be able to test in isolation? No? A compare of an 
old-school flush against new in-memory compaction flush?

Wonder if a difference between read form CSLM and heap read from a bunch of 
segments.

bq. How about your global memstore size limit.

This is default. I'm trying to run w/ all defaults. I will change default if we 
come up w/ something better ("...95% of 
hbase.regionserver.global.memstore.size..."). We don't block writes now. 
Instead we throw the RegionTooBusyException. We don't put anything in the logs. 
Let me fix this. Would be good to note when the Region is struggling... or at 
least let me see if a metric I could use.

bq. When I did tests normally will select this barrier as 2 * regions# * flush 
size. 

Hmm.. This last run of mine had it at 8 *. Let me set it do the default (4 * ).

bq.  If not on SSD, any chance for an SSD based tests?

Nope. No SSD in these chassis.









> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-19 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404679#comment-16404679
 ] 

Anoop Sam John commented on HBASE-19639:


I some how ended up adding below comment into HBASE-20188 .. But what I try to 
say make more sense here.

In tests, we try flush to SDD or HDD boss?
In writes we normally see this compares as hottest. It used to be this way. Now 
I can see the issue you face is the exception because memstore size is 4x 
larger than flush size. As you said yes the flush seems NOT speedy enough. I 
can think of following
1. Now with compacting memstore, the flush op as such more time taking. With 
default memstore, it just a matter of iterating over a map and write cells. But 
now we have to read from multiple segments in a heap way and so more compares 
there. Anyway at flush size the flush op was triggered. But till it reaches 4x 
flush was not complete. This can be one reason
2. The writes to CSLM as such became fast. With default memstore, when we are 
at flush size and started the flush op, still writes happening to CSLM. we 
allow it anyway. But then CSLM state is such that it already having so many 
cells and writes might be bit more delayed. So the pace of this might be low 
enough for flush to complete. But with compacting memstore, it will become 
fresh CSLM again and so very fast writes.
How about your global memstore size limit. I guess this might be a very large 
number. Normally in tests what we see is this barrier breach and so forced 
flushes by blocking writes. Because there are enough regions in RS and write to 
all regions. So any region crossing this 4x mark is less likely compared to 
this global barrier breach. When I did tests normally will select this barrier 
as 2 * regions# * flush size. 
We very much need flush to be faster. If not on SSD, any chance for an SSD 
based tests? This is one reason why am a fan of that JMS issue of flush to SSD 
policy.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-18 Thread Eshcar Hillel (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403890#comment-16403890
 ] 

Eshcar Hillel commented on HBASE-19639:
---

[~stack] any chance you can attach here the entire log file of some RS? If 
possible from start-hbase point and then throughout the test. Thanks.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2018-03-16 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403240#comment-16403240
 ] 

stack commented on HBASE-19639:
---

Back again. Trying to go big again w/ ITBLL but its crawling because MR tasks 
are full of this kinda thing:

{code}
2018-03-13 21:20:22,346 INFO [htable-pool3-t3] 
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: id=2, 
table=IntegrationTestBigLinkedList, attempt=9/16 failed=99ops, last 
exception=org.apache.hadoop.hbase.RegionTooBusyException: 
org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit; 
regionName=95a7c69c73b3580487acdb0cdcc02b61, 
server=ve0542.halxg.cloudera.com,16020,1520972147901, 
blockingMemStoreSize=512.0M
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4191)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3812)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3752)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1015)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:947)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:910)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2636)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
 on ve0542.halxg.cloudera.com,16020,1520972147901, tracking started null, 
retrying after=10022ms, replay=99ops
{code}

We are not flushing fast enough.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305801#comment-16305801
 ] 

stack commented on HBASE-19639:
---

Ok. I put back in-memory compaction and the throttling of compactions. I left 
in place the upped retries -- from 10 to 15 -- and upped blocking store files 
from 10 to 16. Doing this I was able to get a 1B 1hour run that verified with 
monkeys killing servers. Previous I could not get a 200M run to pass. This is 
not enough for hbase-2.0.0 but will do for beta-1. Let me make the changes to 
defaults for beta-1 and then push this issue out to beta-2.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-28 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305546#comment-16305546
 ] 

stack commented on HBASE-19639:
---

After upping blocking size to 64, I was able to get a 1B run of ITBLL in w/o 
failures. Let me come up w/ a patch. Maybe upping retries and blocking file 
defaults is enough. Retries went from 30 to 10 (good) but makes it so a client 
can't ride over a server crash which messes up ITBLL. The blocking files limit 
stalls flushing. The throttling makes flushing slower and compactions not clean 
up storefiles ... which brings on blocking, and so on. All but the blocking 
file limit are new to 2.0.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305069#comment-16305069
 ] 

stack commented on HBASE-19639:
---

Again failed because a server took too long to recover.

Looking at this change now:

tree 9fe0b4e8c5a6b2072230ece1be781900396aa1bb
parent 22b90c4a647d0ffeec7778042eedd0a49a664ed0
author Guanghao Zhang  Tue Nov 28 21:08:19 2017 +0800
committer Michael Stack  Wed Nov 29 10:33:20 2017 -0800

HBASE-19359 Revisit the default config of hbase client retries number

Our MTTR should be faster but we seem to be giving up at 11th retry... most of 
the time, 11 retries is enough but sometimes it is not enough.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305049#comment-16305049
 ] 

stack commented on HBASE-19639:
---

HBASE-14969 added flush throughput controller for 2.0.0 (NoLimit is default in 
branch-1). The below added compaction limiting in branch-2 (branch-1 is no 
limit):

commit eb351b9ff8276228e725bcf58675ab75b640fbbf
Author: zhangduo 
Date:   Tue Feb 3 12:10:54 2015 +0800

HBASE-8329 Limit compaction speed

Signed-off-by: stack 

Disabling the limiters makes for different failure type. In this last run, 
failure was different. A server was offline because 41 WAL files to split. 
Looks like recovery took one minute. Client seems to have given up though. 
Retrying.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305013#comment-16305013
 ] 

stack commented on HBASE-19639:
---

Upping throughput limits didn't help. Trying [~Apache9] suggestion.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=12/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305003#comment-16305003
 ] 

stack commented on HBASE-19639:
---

Thanks [~Apache9] I'll try that. I think we should up the defaults for the 
throughput controller anyways... Will be back. Thanks for suggestion.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=12/11

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread Duo Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304977#comment-16304977
 ] 

Duo Zhang commented on HBASE-19639:
---

Just use NoLimitThroughputController for both flush and compaction and try 
again?

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=12/11 failed=524ops, last 
> exception:

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304976#comment-16304976
 ] 

stack commented on HBASE-19639:
---

Here is where I'm at currently:

{code}
Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
Failed 11 actions: org.apache.hadoop.hbase.RegionTooBusyException: Over 
memstore limit; 
regionName=IntegrationTestBigLinkedList,\x9Ey\xE7\x9Ey\xE7\x9Ep,1514418177121.b03df3a35a6e8360ddf4e4d1c4b0ae0b.,
 server=ve0538.halxg.cloudera.com,16020,1514418422581, 
blockingMemStoreSize=536870912 at 
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178) 
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799) 
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739) 
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
 at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
 at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
 at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) : 8 
times, org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit; 
regionName=IntegrationTestBigLinkedList,0\xC3\x0C0\xC3\x0C0\xC0,1514418177121.e49363c19163fc9a09dff4536b3d0750.,
 server=ve0532.halxg.cloudera.com,16020,1514419525586, 
blockingMemStoreSize=536870912 at 
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178) 
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799) 
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739) 
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
 at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
 at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
 at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) : 3 
times, servers with issues: ve0538.halxg.cloudera.com,16020,1514418422581, 
ve0532.halxg.cloudera.com,16020,1514419525586 at 
org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54) 
at 
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:491)
 at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:268)
 at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:181)
 at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:143)
 at 
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:538)
 at 
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:464)
 at 
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:399)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at 
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) [2017-12-27 
16:16:16.193]Container killed by the ApplicationMaster. [2017-12-27 
16:16:16.213]Container killed on request. Exit code is 143 [2017-12-27 
16:16:16.236]Container exited with a non-zero exit code 143.
{code}

Looking, we seem to be sticking sometimes because too many store files... The 
compaction throttler is slowing compactions.  Default for blocking store files 
of 10.

Trying to up the throttler 

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL:

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

2017-12-27 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304866#comment-16304866
 ] 

stack commented on HBASE-19639:
---

I tried disabling in-memory compaction but that doesn't seem to be it; we're 
just up against memory limit and not flushing. Trying to fix up logs so can 
see. Currently logs are filled w/ all kinda random observations. Needs edit.

> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> Running ITBLLs, the basic link generator keeps failing because I run into 
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main] 
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: 
> Persisting current.length=100, count=100, id=Job: 
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2, 
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last 
> exception: org.apache.hadoop.hbase.RegionTooBusyException: 
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, 
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
>  server=ve0538.halxg.cloudera.com,16020,1514343549993, 
> memstoreSize=538084641, blockingMemStoreSize=536870912
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
>  on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, 
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities 
> for 6 ms, close active connection. Will reconnect next time when there 
> are new requests.
> 2017-12-26 19:24:39,122 WARN [htable-pool3-t6] 
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, 
>

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit

23 matches

Site Navigation

Mail list logo

Footer information