Hi Guanghao,
This throttling error shows up as soon as I start hbase. So, ideally there
shouldn't be too many prior operations in play here. Plus, the error shows up
even when my table has hardly any data (possibly in kbs) and I have set
throttling limit to be ~10GB.
Thanks,Sumit
From: Guanghao Zhang <[email protected]>
To: [email protected]; Sumit Nigam <[email protected]>
Sent: Thursday, September 22, 2016 7:37 AM
Subject: Re: Hbase throttling issues
All scan operations were throttled? Now it use the avg size of all previous
operations to check quota. Maybe the previous scan operation read too much
data.
2016-09-22 1:18 GMT+08:00 Sumit Nigam <[email protected]>:
> Hi Hbase experts,
>
> Our application is unable to scan or read from hbase tables, when
> throttling is set. We are getting ThrottlingException every time. The error
> is seen more frequently when the number of hbase pre splits is
> increased. The size tables for which this error is showing is empty (
> during some runs it was very low in the order of few kbs). Have tried both
> rate limiters already - average and fixed. Can't understand why read rate
> limit exceeds when there is hardly any data in hbase. Anyone has faced this
> issue before?
>
> Setup Details:
>
> Hbase version : 1.1.2
> Number of region servers :4
> Number of regions : 116
> HeapMemory for Region Server : 2GB
>
> Quotas set :
> TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT
> => 10G/sec, SCOPE => MACHINE
> TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT
> => 10G/sec, SCOPE => MACHINE
>
> Following is the error we faced. Pasting debug logs of regions server:
>
> 2016-09-17 22:35:40,674 DEBUG
> [B.defaultRpcServer.handler=55,queue=1,port=58526]
> quotas.RegionServerQuotaManager: Throttling exception for user=root
> table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit
> exceeded - wait 0.00sec
>
> 2016-09-17 22:35:40,676 DEBUG
> [B.defaultRpcServer.handler=55,queue=1,port=58526]
> ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId:
> 52 service: ClientService methodName: Scan size: 28 connection:
> 10.65.141.170:42806
>
> org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit
> exceeded - wait 0.00sec
>
> at org.apache.hadoop.hbase.quotas.ThrottlingException.
> throwThrottlingException(ThrottlingException.java:107)
>
> at org.apache.hadoop.hbase.quotas.ThrottlingException.
> throwReadSizeExceeded(ThrottlingException.java:101)
>
> at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.
> checkQuota(TimeBasedLimiter.java:139)
>
> at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(
> DefaultOperationQuota.java:59)
>
> at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> RegionServerQuotaManager.java:180)
>
> at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> RegionServerQuotaManager.java:125)
>
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> scan(RSRpcServices.java:2265)
>
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
>
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:130)
>
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks,
>
> Sumit
