[
https://issues.apache.org/jira/browse/HDFS-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348042#comment-16348042
]
Xiaoyu Yao commented on HDFS-13093:
-----------------------------------
Thanks [~liaoyuxiangqin] for reporting this. This could relate to HDFS-8898
which returns a cached quota usage with the new getQuotaUsage() API from INode
directly without recursive traversal and recalculate like getContentSummary()
API.
Before HDFS-8898, the "hdfs dfs -count" CLI uses getContentSummary(), which is
expensive because it always walks the whole sub-tree to recalculate quota and
usage. This guarantees the correctness no matter the order of step 3 and 4.
After HDFS-8898, we switch to use the getQuotaUsage() API for "hdfs dfs count"
CLI, the cached INode quota usage for storage type will be 0 if you don't set
storage policy before storage type quota.
This is because the storage type usage is strongly tied to the storage policy.
If there is no storage policy set first, we will not be able to determine the
quota usage for different storage type. As a result, 0 will be the default in
this case.
I believe this is a transient incorrectness that can be fixed by one of the
following three options.
1. This is a corner case. Document the procedure to set storage policy first
before set storage type quota. Otherwise, getQuotaUsage() API and "hdfs dfs
count" CLI will return an inconsistent result. No fix needed.
2. The cached quota usage will be recalculated correctly any way upon next NN
restart, no fix needed.
3. If we really want to allow setting storage type quota before setting storage
policy, we can provide an option for "hdfs dfs count" CLI to use
getContentSummary() to report the accurate usage.
cc: [~mingma], [~kihwal] for additional comments.
> Quota set don't compute usage of unspecified storage policy content
> -------------------------------------------------------------------
>
> Key: HDFS-13093
> URL: https://issues.apache.org/jira/browse/HDFS-13093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.1.0
> Environment: hdfs: hadoop-3.1.0-SNAPSHOT
> node:1 namenode, 9 datanodes
> Reporter: liaoyuxiangqin
> Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> test as following steps:
> 1. hdfs dfs -mkdir /hot
> 2. hdfs dfs -put 1G.img /hot/file1
> 3. hdfs dfsadmin -setSpaceQuota 6442450944 -storageType DISK /hot
> 4. hdfs storagepolicies -setStoragePolicy -path /hot -policy HOT
> 5. hdfs dfs -count -q -h -v -t DISK /hot
> {code:java}
> SSD_QUOTA REM_SSD_QUOTA DISK_QUOTA REM_DISK_QUOTA ARCHIVE_QUOTA
> REM_ARCHIVE_QUOTA PROVIDED_QUOTA REM_PROVIDED_QUOTA PATHNAME
> none inf 6 G 6 G none inf none inf /hot{code}
> In step5 i speculation the remaining quota is 3G(quota - 1G*3 replicas )，but
> 6G actually.
> if i change the turn of step3 and step4, then the remaining quota equal to
> what I think 3G.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
