Jinglun created HDFS-14562:
------------------------------
Summary: The behaviour of getContentSummaryInt() in
getQuotaUsage() should be configurable.
Key: HDFS-14562
URL: https://issues.apache.org/jira/browse/HDFS-14562
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.1.0
Reporter: Jinglun
Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. There is a ploblem
about the getQuotaUsage rpc. In FSDirStatAndListingOp.getQuotaUsage(), if there
isn't any quota on the dir, it will automatically count the dir to get the info
of usage. But count on big dirs are quit dangerous, it can slow the NameNode
and even cause a failover. We've encountered the case that 10 concurrent count
rpcs on big dir causes the NameNode failover.
In our cluster we always need to check whether the dir has got quota or not,
and the automatically count will make things dangerous. Making the behavior
configurable seems a good idea. Administrator can decide to fall back to count
or fill the consume with -1 when there is no quota on the dir.
When I try to make it configurable, I find another problem. When we convert
QuotaUsageProto and QuotaUsage in PBHelperClient.class, there are checks for
qu.hasTypeQuotaInfos() and qu.isTypeQuotaSet() || qu.isTypeConsumedAvailable().
Supposing we want to return a QuotaUsage with \{fileAndDirectoryCount=-1,
spaceConsumed=-1, typeConsumed={-1,-1,-1,-1,-1}} from Namenode to Client,
because of the check, the value got by Client will be
\{fileAndDirectoryCount=-1, spaceConsumed=-1, typeConsumed={0,0,0,0,0}}. It's
inconsistent and I can't see any good reason that spaceConsumed could return -1
while typeConsumed must be 0. In fact we don't need the checks, checking all
the assignment statement then we'll find that QuotaUsage.typeConsumed and
typeQuota will never be null. And it's not right for the Convert layer to
tamper the returned value. Since -1 represents undefined in quota and usage, we
should remove the check and let Namenode returns -1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
