[
https://issues.apache.org/jira/browse/HBASE-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-25634:
---------------------------------
Description:
When the client scan operation, the server frequently returns
RpcThrottlingException, which will cause the meta table request to become high.
/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
{code:java}
// code placeholder
@Override
public T callWithRetries(RetryingCallable<T> callable, int callTimeout)
throws IOException, RuntimeException {
List<RetriesExhaustedException.ThrowableWithExtraContext> exceptions = new
ArrayList<>();
tracker.start();
context.clear();
for (int tries = 0;; tries++) {
long expectedSleep;
try {
// bad cache entries are cleared in the call to
RetryingCallable#throwable() in catch block
Throwable t = null;
if (exceptions != null && !exceptions.isEmpty()) {
t = exceptions.get(exceptions.size() - 1).throwable;
}
if (!(t instanceof RpcThrottlingException)) {
callable.prepare(tries != 0);
}
interceptor.intercept(context.prepare(callable, tries));
return callable.call(getTimeout(callTimeout));
} catch (PreemptiveFastFailException e) {
throw e;
} catch (Throwable t) {
ExceptionUtil.rethrowIfInterrupt(t);
Throwable cause = t.getCause();
if (cause instanceof DoNotRetryIOException) {
// Fail fast
throw (DoNotRetryIOException) cause;
}
// translateException throws exception when should not retry: i.e. when
request is bad.
interceptor.handleFailure(context, t);
t = translateException(t);
{code}
!image-2021-03-05-12-00-33-522.png!
!image-2021-03-05-12-01-08-769.png!
was:
When the client scan operation, the server frequently returns
RpcThrottlingException, which will cause the meta table request to become high.
2021-03-03 10:35:07,372 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t269] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28243ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,372 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t267] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28322ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,380 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181570,4
replyHeader:: 181570,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,382 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t271] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28241ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,389 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181571,4
replyHeader:: 181571,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,390 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t268] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28324ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,429 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181572,4
replyHeader::
181572,38683795827,0 request:: '/hbase_qinglong/table/ClientTest:CPUTEST2,F
response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s
{38683609136,38683795820,1613703475740,1614738735797,7 ,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,431 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t272] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=2825
3ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in LF-HBASE-QINGLONG1-172-20-16
1-121.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,436 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181573,4
replyHeader::
181573,38683795827,0 request:: '/hbase_qinglong/table/ClientTest:CPUTEST2,F
response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7
,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,437 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t270] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=2830
5ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in LF-HBASE-QINGLONG1-172-20-16
1-121.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,468 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181574,4
replyHeader::
181574,38683795827,0 request:: '/hbase_qinglong/table/ClientTest:CPUTEST2,F
response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s
\{38683609136,38683795820,1613703475740,1614738735797,7
,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,469 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t275] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=2820
1ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in LF-HBASE-QINGLONG1-172-20-16
1-121.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,470 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181575,4
replyHeader::
181575,38683795827,0 request:: '/hbase_qinglong/table/ClientTest:CPUTEST2,F
response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7
,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,471 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t276] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=2818
9ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in LF-HBASE-QINGLONG1-172-20-16
1-121.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,475 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181576,4
replyHeader::181576,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,477 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t274] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28222ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:07,526 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181577,4
replyHeader:: 181577,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:07,527 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t273] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=9, retries=63, started=28293ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,389 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181578,4
replyHeader:: 181578,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,391 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181579,4
replyHeader:: 181579,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,391 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t271] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38250ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,406 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t268] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38340ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,429 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181580,4
replyHeader:: 181580,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,430 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t267] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38380ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,451 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181581,4
replyHeader:: 181581,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,452 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t270] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38320ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,465 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181582,4
replyHeader:: 181582,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,466 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t269] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38337ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,495 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181583,4
replyHeader:: 181583,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,496 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t275] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38228ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,509 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181584,4
replyHeader:: 181584,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,510 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t274] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38255ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,513 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181585,4
replyHeader:: 181585,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,514 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t276] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38232ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,524 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181586,4
replyHeader:: 181586,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
2021-03-03 10:35:17,525 [hconnection-0x12a53d4 clusterId:
824ef0d2-aeca-4880-b8a7-507a4253d976-shared--pool1-t272] WARN
[org.apache.hadoop.hbase.client.RpcRetryingCaller] - Client Excess Quota is
Retrying, tries=10, retries=63, started=38347ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException:
org.apache.hadoop.hbase.quotas.RpcThrottlingException: ClientTest:CPUTEST2 -
request size limit exceeded - wait 999hrs, 0sec in
LF-HBASE-QINGLONG1.hadoop.jd.local,16020,1614698288842
2021-03-03 10:35:17,534 [Thread-1-SendThread(172.20.161.121:2181)] DEBUG
[org.apache.zookeeper.ClientCnxn] - Reading reply sessionid:0x1756ec5a1bf2157,
packet:: clientPath:null serverPath:null finished:false header:: 181587,4
replyHeader:: 181587,38683795827,0 request::
'/hbase_qinglong/table/ClientTest:CPUTEST2,F response::
#ffffffff000146d61737465723a3136303030fffffffd1c16ffffffa1fffffffcffffffcb64ffffff8e5042554680,s\{38683609136,38683795820,1613703475740,1614738735797,7,0,0,0,31,0,38683609136}
!image-2021-03-05-12-00-33-522.png!
!image-2021-03-05-12-01-08-769.png!
> The client frequently exceeds the quota, which causes the meta table scan to
> be too high
> ----------------------------------------------------------------------------------------
>
> Key: HBASE-25634
> URL: https://issues.apache.org/jira/browse/HBASE-25634
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.3.4
> Reporter: zhengsicheng
> Assignee: zhengsicheng
> Priority: Minor
> Attachments: image-2021-03-05-12-00-33-522.png,
> image-2021-03-05-12-01-08-769.png
>
>
> When the client scan operation, the server frequently returns
> RpcThrottlingException, which will cause the meta table request to become
> high.
>
> /hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
> {code:java}
> // code placeholder
> @Override
> public T callWithRetries(RetryingCallable<T> callable, int callTimeout)
> throws IOException, RuntimeException {
> List<RetriesExhaustedException.ThrowableWithExtraContext> exceptions = new
> ArrayList<>();
> tracker.start();
> context.clear();
> for (int tries = 0;; tries++) {
> long expectedSleep;
> try {
> // bad cache entries are cleared in the call to
> RetryingCallable#throwable() in catch block
> Throwable t = null;
> if (exceptions != null && !exceptions.isEmpty()) {
> t = exceptions.get(exceptions.size() - 1).throwable;
> }
> if (!(t instanceof RpcThrottlingException)) {
> callable.prepare(tries != 0);
> }
> interceptor.intercept(context.prepare(callable, tries));
> return callable.call(getTimeout(callTimeout));
> } catch (PreemptiveFastFailException e) {
> throw e;
> } catch (Throwable t) {
> ExceptionUtil.rethrowIfInterrupt(t);
> Throwable cause = t.getCause();
> if (cause instanceof DoNotRetryIOException) {
> // Fail fast
> throw (DoNotRetryIOException) cause;
> }
> // translateException throws exception when should not retry: i.e. when
> request is bad.
> interceptor.handleFailure(context, t);
> t = translateException(t);
> {code}
>
>
>
> !image-2021-03-05-12-00-33-522.png!
> !image-2021-03-05-12-01-08-769.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
