[
https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912784#comment-16912784
]
Xiaoyu Yao commented on HDFS-14633:
-----------------------------------
Thanks [~LiJinglun] for reporting the issue along with the test case for repro.
That helps us understand and fix the issue.
Spend some time this week and agree with your analysis. Attached is a draft fix
for the issue. The basic idea is to use the destination parent's storage policy
to calculate the quota usage delta when validate quota. The code change passes
the rename case illustrated in the earlier patch.
> The StorageType quota and consume in QuotaFeature is not handled when rename
> and setStoragePolicy etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14633
> URL: https://issues.apache.org/jira/browse/HDFS-14633
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HDFS-14633-testcases-explanation, HDFS-14633.002.patch
>
>
> The NameNode manages the global state of the cluster. We should always take
> NameNode's records as the sole criterion because no matter what inconsistent
> is the NameNode should finally make everything right based on it's records.
> Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do
> all quota related rpcs, we do the quota check according to NameNode's records
> regardless of any inconsistent situation, such as the replicas doesn't match
> the storage policy of the file, or the replicas count doesn't match the
> file's set replica.
> The work SPS deals with the wrongly palced replicas. There is a thought
> about putting off the consume update of the DirectoryQuota until all replicas
> are re-placed by SPS. I can't agree that because if we do so we will abandon
> letting the NameNode's records to be the sole criterion. The block
> replication is a good example of the rule NSC. When we count the consume of a
> file(CONTIGUOUS), we multiply the replication factor with the file's length,
> no matter the blocks are under replicated or excessed. We should do the same
> thing for the storage type quota.
> Another concern is the change will let setStoragePolicy throw
> QuotaByStorageTypeExceededException which it doesn't before. I don't think
> it's a big problem since the setStoragePolicy already throws IOException. Or
> we can wrap the QuotaByStorageTypeExceededException in an IOException, but I
> won't recommend that because it's ugly.
> To make storage type consume follow the rule NSC, we need change
> rename(moving a file with storage policy inherited from it's parent) and
> setStoragePolicy.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
