Re: Hbase throttling issues
No, there are no other quotas set. The list_quota shows only REQUEST_TYPE as set, which is what I want. Not sure why it tries validating READ_SIZE when I have set only REQUEST_TYPE. Is anyone aware of known issues with Hbase 1.1? Setting separate read and write quotas was only introduced in 1.2, but not sure why this quota exceeded error with READ_SIZE then shows up. Thanks,Sumit From: Guanghao Zhang To: user@hbase.apache.org; Sumit Nigam Sent: Thursday, September 22, 2016 12:49 PM Subject: Re: Hbase throttling issues Did you set any other quotas? The exception ThrottlingException.throwReadSizeExceeded shows that the throttle type is READ_SIZE, but your quota's throttle type is REQUEST_SIZE. You can use list_quota shell cmd to check whether there are READ_SIZE quota. 2016-09-22 12:06 GMT+08:00 Sumit Nigam : > No, did not try request number. I want to use size as my throttling > factor. > > Thanks again! > > From: Guanghao Zhang > To: user@hbase.apache.org; Sumit Nigam > Sent: Thursday, September 22, 2016 9:28 AM > Subject: Re: Hbase throttling issues > > Did you try to use REQUEST_NUMBER as throttle type? > > 2016-09-22 11:33 GMT+08:00 Sumit Nigam : > > > Hi Guanghao, > > > > This throttling error shows up as soon as I start hbase. So, ideally > there > > shouldn't be too many prior operations in play here. Plus, the error > shows > > up even when my table has hardly any data (possibly in kbs) and I have > set > > throttling limit to be ~10GB. > > > > Thanks,Sumit > > > > From: Guanghao Zhang > > To: user@hbase.apache.org; Sumit Nigam > > Sent: Thursday, September 22, 2016 7:37 AM > > Subject: Re: Hbase throttling issues > > > > All scan operations were throttled? Now it use the avg size of all > previous > > operations to check quota. Maybe the previous scan operation read too > much > > data. > > > > 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > > > > > Hi Hbase experts, > > > > > > Our application is unable to scan or read from hbase tables, when > > > throttling is set. We are getting ThrottlingException every time. The > > error > > > is seen more frequently when the number of hbase pre splits is > > > increased. The size tables for which this error is showing is empty ( > > > during some runs it was very low in the order of few kbs). Have tried > > both > > > rate limiters already - average and fixed. Can't understand why read > rate > > > limit exceeds when there is hardly any data in hbase. Anyone has faced > > this > > > issue before? > > > > > > Setup Details: > > > > > > Hbase version : 1.1.2 > > > Number of region servers :4 > > > Number of regions : 116 > > > HeapMemory for Region Server : 2GB > > > > > > Quotas set : > > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > > LIMIT > > > => 10G/sec, SCOPE => MACHINE > > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > > LIMIT > > > => 10G/sec, SCOPE => MACHINE > > > > > > Following is the error we faced. Pasting debug logs of regions server: > > > > > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler= > > 55,queue=1,port=58526] > > > quotas.RegionServerQuotaManager: Throttling exception for user=root > > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > > > exceeded - wait 0.00sec > > > > > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler= > > 55,queue=1,port=58526] > > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: > callId: > > > 52 service: ClientService methodName: Scan size: 28 connection: > > > 10.65.141.170:42806 > > > > > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > > > exceeded - wait 0.00sec > > > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > > throwThrottlingException(ThrottlingException.java:107) > > > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > > throwReadSizeExceeded(ThrottlingException.java:101) > > > > > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > > > checkQuota(TimeBasedLimiter.java:139) > > > > > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > > > DefaultOperationQuota.java:59) > > > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManage
Re: Hbase throttling issues
Did you set any other quotas? The exception ThrottlingException.throwReadSizeExceeded shows that the throttle type is READ_SIZE, but your quota's throttle type is REQUEST_SIZE. You can use list_quota shell cmd to check whether there are READ_SIZE quota. 2016-09-22 12:06 GMT+08:00 Sumit Nigam : > No, did not try request number. I want to use size as my throttling > factor. > > Thanks again! > > From: Guanghao Zhang > To: user@hbase.apache.org; Sumit Nigam > Sent: Thursday, September 22, 2016 9:28 AM > Subject: Re: Hbase throttling issues > > Did you try to use REQUEST_NUMBER as throttle type? > > 2016-09-22 11:33 GMT+08:00 Sumit Nigam : > > > Hi Guanghao, > > > > This throttling error shows up as soon as I start hbase. So, ideally > there > > shouldn't be too many prior operations in play here. Plus, the error > shows > > up even when my table has hardly any data (possibly in kbs) and I have > set > > throttling limit to be ~10GB. > > > > Thanks,Sumit > > > > From: Guanghao Zhang > > To: user@hbase.apache.org; Sumit Nigam > > Sent: Thursday, September 22, 2016 7:37 AM > > Subject: Re: Hbase throttling issues > > > > All scan operations were throttled? Now it use the avg size of all > previous > > operations to check quota. Maybe the previous scan operation read too > much > > data. > > > > 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > > > > > Hi Hbase experts, > > > > > > Our application is unable to scan or read from hbase tables, when > > > throttling is set. We are getting ThrottlingException every time. The > > error > > > is seen more frequently when the number of hbase pre splits is > > > increased. The size tables for which this error is showing is empty ( > > > during some runs it was very low in the order of few kbs). Have tried > > both > > > rate limiters already - average and fixed. Can't understand why read > rate > > > limit exceeds when there is hardly any data in hbase. Anyone has faced > > this > > > issue before? > > > > > > Setup Details: > > > > > > Hbase version : 1.1.2 > > > Number of region servers :4 > > > Number of regions : 116 > > > HeapMemory for Region Server : 2GB > > > > > > Quotas set : > > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > > LIMIT > > > => 10G/sec, SCOPE => MACHINE > > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > > LIMIT > > > => 10G/sec, SCOPE => MACHINE > > > > > > Following is the error we faced. Pasting debug logs of regions server: > > > > > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler= > > 55,queue=1,port=58526] > > > quotas.RegionServerQuotaManager: Throttling exception for user=root > > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > > > exceeded - wait 0.00sec > > > > > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler= > > 55,queue=1,port=58526] > > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: > callId: > > > 52 service: ClientService methodName: Scan size: 28 connection: > > > 10.65.141.170:42806 > > > > > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > > > exceeded - wait 0.00sec > > > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > > throwThrottlingException(ThrottlingException.java:107) > > > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > > throwReadSizeExceeded(ThrottlingException.java:101) > > > > > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > > > checkQuota(TimeBasedLimiter.java:139) > > > > > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > > > DefaultOperationQuota.java:59) > > > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > > RegionServerQuotaManager.java:180) > > > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > > RegionServerQuotaManager.java:125) > > > > > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > > scan(RSRpcServices.java:2265) > > > > > > at org.apache.hadoop.hbase.protobuf.generated. > > > ClientProtos$ClientService$2.callBlockingMethod( > ClientProtos.java:32205) > > > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > > > RpcExecutor.java:130) > > > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > Thanks, > > > > > > Sumit > > > > > > > > > > > >
Re: Hbase throttling issues
No, did not try request number. I want to use size as my throttling factor. Thanks again! From: Guanghao Zhang To: user@hbase.apache.org; Sumit Nigam Sent: Thursday, September 22, 2016 9:28 AM Subject: Re: Hbase throttling issues Did you try to use REQUEST_NUMBER as throttle type? 2016-09-22 11:33 GMT+08:00 Sumit Nigam : > Hi Guanghao, > > This throttling error shows up as soon as I start hbase. So, ideally there > shouldn't be too many prior operations in play here. Plus, the error shows > up even when my table has hardly any data (possibly in kbs) and I have set > throttling limit to be ~10GB. > > Thanks,Sumit > > From: Guanghao Zhang > To: user@hbase.apache.org; Sumit Nigam > Sent: Thursday, September 22, 2016 7:37 AM > Subject: Re: Hbase throttling issues > > All scan operations were throttled? Now it use the avg size of all previous > operations to check quota. Maybe the previous scan operation read too much > data. > > 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > > > Hi Hbase experts, > > > > Our application is unable to scan or read from hbase tables, when > > throttling is set. We are getting ThrottlingException every time. The > error > > is seen more frequently when the number of hbase pre splits is > > increased. The size tables for which this error is showing is empty ( > > during some runs it was very low in the order of few kbs). Have tried > both > > rate limiters already - average and fixed. Can't understand why read rate > > limit exceeds when there is hardly any data in hbase. Anyone has faced > this > > issue before? > > > > Setup Details: > > > > Hbase version : 1.1.2 > > Number of region servers :4 > > Number of regions : 116 > > HeapMemory for Region Server : 2GB > > > > Quotas set : > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > LIMIT > > => 10G/sec, SCOPE => MACHINE > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > LIMIT > > => 10G/sec, SCOPE => MACHINE > > > > Following is the error we faced. Pasting debug logs of regions server: > > > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler= > 55,queue=1,port=58526] > > quotas.RegionServerQuotaManager: Throttling exception for user=root > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > > exceeded - wait 0.00sec > > > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler= > 55,queue=1,port=58526] > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: > > 52 service: ClientService methodName: Scan size: 28 connection: > > 10.65.141.170:42806 > > > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > > exceeded - wait 0.00sec > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > throwThrottlingException(ThrottlingException.java:107) > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > throwReadSizeExceeded(ThrottlingException.java:101) > > > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > > checkQuota(TimeBasedLimiter.java:139) > > > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > > DefaultOperationQuota.java:59) > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > RegionServerQuotaManager.java:180) > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > RegionServerQuotaManager.java:125) > > > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > scan(RSRpcServices.java:2265) > > > > at org.apache.hadoop.hbase.protobuf.generated. > > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > > RpcExecutor.java:130) > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Thanks, > > > > Sumit > > > >
Re: Hbase throttling issues
Did you try to use REQUEST_NUMBER as throttle type? 2016-09-22 11:33 GMT+08:00 Sumit Nigam : > Hi Guanghao, > > This throttling error shows up as soon as I start hbase. So, ideally there > shouldn't be too many prior operations in play here. Plus, the error shows > up even when my table has hardly any data (possibly in kbs) and I have set > throttling limit to be ~10GB. > > Thanks,Sumit > > From: Guanghao Zhang > To: user@hbase.apache.org; Sumit Nigam > Sent: Thursday, September 22, 2016 7:37 AM > Subject: Re: Hbase throttling issues > > All scan operations were throttled? Now it use the avg size of all previous > operations to check quota. Maybe the previous scan operation read too much > data. > > 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > > > Hi Hbase experts, > > > > Our application is unable to scan or read from hbase tables, when > > throttling is set. We are getting ThrottlingException every time. The > error > > is seen more frequently when the number of hbase pre splits is > > increased. The size tables for which this error is showing is empty ( > > during some runs it was very low in the order of few kbs). Have tried > both > > rate limiters already - average and fixed. Can't understand why read rate > > limit exceeds when there is hardly any data in hbase. Anyone has faced > this > > issue before? > > > > Setup Details: > > > > Hbase version : 1.1.2 > > Number of region servers :4 > > Number of regions : 116 > > HeapMemory for Region Server : 2GB > > > > Quotas set : > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > LIMIT > > => 10G/sec, SCOPE => MACHINE > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, > LIMIT > > => 10G/sec, SCOPE => MACHINE > > > > Following is the error we faced. Pasting debug logs of regions server: > > > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler= > 55,queue=1,port=58526] > > quotas.RegionServerQuotaManager: Throttling exception for user=root > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > > exceeded - wait 0.00sec > > > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler= > 55,queue=1,port=58526] > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: > > 52 service: ClientService methodName: Scan size: 28 connection: > > 10.65.141.170:42806 > > > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > > exceeded - wait 0.00sec > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > throwThrottlingException(ThrottlingException.java:107) > > > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > > throwReadSizeExceeded(ThrottlingException.java:101) > > > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > > checkQuota(TimeBasedLimiter.java:139) > > > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > > DefaultOperationQuota.java:59) > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > RegionServerQuotaManager.java:180) > > > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > > RegionServerQuotaManager.java:125) > > > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > scan(RSRpcServices.java:2265) > > > > at org.apache.hadoop.hbase.protobuf.generated. > > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > > RpcExecutor.java:130) > > > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > Thanks, > > > > Sumit > > > >
Re: Hbase throttling issues
Hi Guanghao, This throttling error shows up as soon as I start hbase. So, ideally there shouldn't be too many prior operations in play here. Plus, the error shows up even when my table has hardly any data (possibly in kbs) and I have set throttling limit to be ~10GB. Thanks,Sumit From: Guanghao Zhang To: user@hbase.apache.org; Sumit Nigam Sent: Thursday, September 22, 2016 7:37 AM Subject: Re: Hbase throttling issues All scan operations were throttled? Now it use the avg size of all previous operations to check quota. Maybe the previous scan operation read too much data. 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > Hi Hbase experts, > > Our application is unable to scan or read from hbase tables, when > throttling is set. We are getting ThrottlingException every time. The error > is seen more frequently when the number of hbase pre splits is > increased. The size tables for which this error is showing is empty ( > during some runs it was very low in the order of few kbs). Have tried both > rate limiters already - average and fixed. Can't understand why read rate > limit exceeds when there is hardly any data in hbase. Anyone has faced this > issue before? > > Setup Details: > > Hbase version : 1.1.2 > Number of region servers :4 > Number of regions : 116 > HeapMemory for Region Server : 2GB > > Quotas set : > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > > Following is the error we faced. Pasting debug logs of regions server: > > 2016-09-17 22:35:40,674 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > quotas.RegionServerQuotaManager: Throttling exception for user=root > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > exceeded - wait 0.00sec > > 2016-09-17 22:35:40,676 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: > 52 service: ClientService methodName: Scan size: 28 connection: > 10.65.141.170:42806 > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > exceeded - wait 0.00sec > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwThrottlingException(ThrottlingException.java:107) > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwReadSizeExceeded(ThrottlingException.java:101) > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > checkQuota(TimeBasedLimiter.java:139) > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > DefaultOperationQuota.java:59) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:180) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:125) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > scan(RSRpcServices.java:2265) > > at org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > at java.lang.Thread.run(Thread.java:745) > > > Thanks, > > Sumit
Re: Hbase throttling issues
All scan operations were throttled? Now it use the avg size of all previous operations to check quota. Maybe the previous scan operation read too much data. 2016-09-22 1:18 GMT+08:00 Sumit Nigam : > Hi Hbase experts, > > Our application is unable to scan or read from hbase tables, when > throttling is set. We are getting ThrottlingException every time. The error > is seen more frequently when the number of hbase pre splits is > increased. The size tables for which this error is showing is empty ( > during some runs it was very low in the order of few kbs). Have tried both > rate limiters already - average and fixed. Can't understand why read rate > limit exceeds when there is hardly any data in hbase. Anyone has faced this > issue before? > > Setup Details: > > Hbase version : 1.1.2 > Number of region servers :4 > Number of regions : 116 > HeapMemory for Region Server : 2GB > > Quotas set : > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > > Following is the error we faced. Pasting debug logs of regions server: > > 2016-09-17 22:35:40,674 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > quotas.RegionServerQuotaManager: Throttling exception for user=root > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > exceeded - wait 0.00sec > > 2016-09-17 22:35:40,676 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: > 52 service: ClientService methodName: Scan size: 28 connection: > 10.65.141.170:42806 > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > exceeded - wait 0.00sec > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwThrottlingException(ThrottlingException.java:107) > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwReadSizeExceeded(ThrottlingException.java:101) > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > checkQuota(TimeBasedLimiter.java:139) > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > DefaultOperationQuota.java:59) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:180) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:125) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > scan(RSRpcServices.java:2265) > > at org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > at java.lang.Thread.run(Thread.java:745) > > > Thanks, > > Sumit
Hbase throttling issues
Hi Hbase experts, Our application is unable to scan or read from hbase tables, when throttling is set. We are getting ThrottlingException every time. The error is seen more frequently when the number of hbase pre splits is increased. The size tables for which this error is showing is empty ( during some runs it was very low in the order of few kbs). Have tried both rate limiters already - average and fixed. Can't understand why read rate limit exceeds when there is hardly any data in hbase. Anyone has faced this issue before? Setup Details: Hbase version : 1.1.2 Number of region servers :4 Number of regions : 116 HeapMemory for Region Server : 2GB Quotas set : TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE Following is the error we faced. Pasting debug logs of regions server: 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager: Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit exceeded - wait 0.00sec 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName: Scan size: 28 connection: 10.65.141.170:42806 org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107) at org.apache.hadoop.hbase.quotas.ThrottlingException.throwReadSizeExceeded(ThrottlingException.java:101) at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:139) at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(DefaultOperationQuota.java:59) at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:180) at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:125) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2265) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Thanks, Sumit
