[
https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-14813:
---------------------------
Description:
Make Global quota and remote quota consistent.
(Global quota: the quota on mount table, Remote quota: the quota on namespace)
HDFS administrator can use global quota to simplify the management for
federation paths. But there is no consistent constraint for the global quota
and the remote quota. As an hdfs administrator the inconsistent has 3
disadvantages on management:
1. The quota part of getQuotaUsage() on a federation path is not helpful.
It's neither the global quota nor one of the remote quotas.
2. The global quota could be different with the remote quota. When a
QuotaExceedException happens it needs the administrator to find out whether
it's a violation of the global quota or the remote quota.
3. For management simplicity, it's always a good idea to keep the global
quota and the remote quota the same. Now we need the administrator to keep the
consistent manually.
My proposal is to add a constraint for global quota:
1. For federation paths, global quota could be inherited from parent
federation path.
2. For all remote paths in mount tables, the remote quotas must be
consistent with the global quotas.
To implement this, my idea is:
1. Global quota could be inherited. Add a method getGlobalQuota(String
path) to Quota.java returning the global quota.
2. Each time RouterQuotaUpdateService updates the quota usage for mount
table entries, it also checks and updates the remote quota.
3. When getQuotaUsage() on a federation path, return the global quota.
4. When setQuota() on a federation path, first update the global quota in
mount table, then recompute global quota for the current path and its children
paths, finally update all the federation paths.
Implement 1+2 in HDFS-14814
Implement 4 in HDFS-14815
Implement 3 in HDFS-14955
was:
Make Global quota and remote quota consistent.
(Global quota: the quota on mount table, Remote quota: the quota on namespace)
HDFS administrator can use global quota to simplify the management for
federation paths. But there is no consistent constraint for the global quota
and the remote quota. As an hdfs administrator the inconsistent has 3
disadvantages on management:
1. The quota part of getQuotaUsage() on a federation path is not helpful.
It's neither the global quota nor one of the remote quotas.
2. The global quota could be different with the remote quota. When a
QuotaExceedException happens it needs the administrator to find out whether
it's a violation of the global quota or the remote quota.
3. For management simplicity, it's always a good idea to keep the global
quota and the remote quota the same. Now we need the administrator to keep the
consistent manually.
My proposal is to add a constraint for global quota:
1. For federation paths, global quota could be inherited from parent
federation path.
2. For all remote paths in mount tables, the remote quotas must be
consistent with the global quotas.
To implement this, my idea is:
1. Global quota could be inherited. Add a method getGlobalQuota(String
path) to Quota.java returning the global quota.
2. Each time RouterQuotaUpdateService updates the quota usage for mount
table entries, it also checks and updates the remote quota.
3. When getQuotaUsage() on a federation path, return the global quota.
4. When setQuota() on a federation path, first update the global quota in
mount table, then recompute global quota for the current path and its children
paths, finally update all the federation paths.
Implement 1+2 in HDFS-14814
Implement 4 in HDFS-14815
> RBF: Make Global quota and Remote quota consistent.
> ---------------------------------------------------
>
> Key: HDFS-14813
> URL: https://issues.apache.org/jira/browse/HDFS-14813
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
>
> Make Global quota and remote quota consistent.
> (Global quota: the quota on mount table, Remote quota: the quota on
> namespace)
> HDFS administrator can use global quota to simplify the management for
> federation paths. But there is no consistent constraint for the global quota
> and the remote quota. As an hdfs administrator the inconsistent has 3
> disadvantages on management:
> 1. The quota part of getQuotaUsage() on a federation path is not
> helpful. It's neither the global quota nor one of the remote quotas.
> 2. The global quota could be different with the remote quota. When a
> QuotaExceedException happens it needs the administrator to find out whether
> it's a violation of the global quota or the remote quota.
> 3. For management simplicity, it's always a good idea to keep the global
> quota and the remote quota the same. Now we need the administrator to keep
> the consistent manually.
> My proposal is to add a constraint for global quota:
> 1. For federation paths, global quota could be inherited from parent
> federation path.
> 2. For all remote paths in mount tables, the remote quotas must be
> consistent with the global quotas.
> To implement this, my idea is:
> 1. Global quota could be inherited. Add a method getGlobalQuota(String
> path) to Quota.java returning the global quota.
> 2. Each time RouterQuotaUpdateService updates the quota usage for mount
> table entries, it also checks and updates the remote quota.
> 3. When getQuotaUsage() on a federation path, return the global quota.
> 4. When setQuota() on a federation path, first update the global quota
> in mount table, then recompute global quota for the current path and its
> children paths, finally update all the federation paths.
>
> Implement 1+2 in HDFS-14814
> Implement 4 in HDFS-14815
> Implement 3 in HDFS-14955
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
