[
https://issues.apache.org/jira/browse/HDFS-14562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863661#comment-16863661
]
Jinglun commented on HDFS-14562:
--------------------------------
Hi [~John Smith], thanks for your comments. I'll remove the change in
convert(QuotaUsageProto qu) to keep it compatible. About the semantics I think
it won't be a problem because the new behavior is configurable. The default
behavior is the same as the original and I can switch to 'not counting' by
configuring DFSConfigKeys.DFS_FALL_BACK_TO_COUNT=false on the NameNode.
The reason I changed the PBHelperClient is that I want to return -1 as the
undefined value of the consume. It doesn't make sense to return 0 as consume
after the 'counting on dir without quota' behaviour is switched off.
I'll upload a new patch soon.
> The behaviour of getContentSummaryInt() in getQuotaUsage() should be
> configurable.
> ----------------------------------------------------------------------------------
>
> Key: HDFS-14562
> URL: https://issues.apache.org/jira/browse/HDFS-14562
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HDFS-14562.001.patch
>
>
> Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. There is a ploblem
> about the getQuotaUsage rpc. In FSDirStatAndListingOp.getQuotaUsage(), if
> there isn't any quota on the dir, it will automatically count the dir to get
> the info of usage. But count on big dirs are quit dangerous, it can slow the
> NameNode and even cause a failover. We've encountered the case that 10
> concurrent count rpcs on big dir causes the NameNode failover.
> In our cluster we always need to check whether the dir has got quota or not,
> and the automatically count will make things dangerous. Making the behavior
> configurable seems a good idea. Administrator can decide to fall back to
> count or fill the consume with -1 when there is no quota on the dir.
> When I try to make it configurable, I find another problem. When we convert
> QuotaUsageProto and QuotaUsage in PBHelperClient.class, there are checks for
> qu.hasTypeQuotaInfos() and qu.isTypeQuotaSet() ||
> qu.isTypeConsumedAvailable(). Supposing we want to return a QuotaUsage with
> \{fileAndDirectoryCount=-1, spaceConsumed=-1, typeConsumed={-1,-1,-1,-1,-1}}
> from Namenode to Client, because of the check, the value got by Client will
> be \{fileAndDirectoryCount=-1, spaceConsumed=-1, typeConsumed={0,0,0,0,0}}.
> It's inconsistent and I can't see any good reason that spaceConsumed could
> return -1 while typeConsumed must be 0. In fact we don't need the checks,
> checking all the assignment statement then we'll find that
> QuotaUsage.typeConsumed and typeQuota will never be null. And it's not right
> for the Convert layer to tamper the returned value. Since -1 represents
> undefined in quota and usage, we should remove the check and let Namenode
> returns -1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
