[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931063#comment-16931063 ] Jinglun commented on HDFS-14813: My bad, I miss [~ayushtkn] comments in HDFS-14814. Let's discuss inherit quota in HDFS-14814, I'll move my comments there. > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[
https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931054#comment-16931054
]
Jinglun commented on HDFS-14813:
Hi [~ayushtkn], very sorry for my late response. I want to do 1+2 together
because I need 1 to resolve global quota conflicts. For example I have mount
tables below, the question is what quotas do I expect for ns1->/dir-b and
ns2->/dir-c.
{quote}M1. /dir-a ns0->/dir-a \{nquota=10,squota=20}
M2. /dir-a/dir-b ns1->/dir-b \{nquota=-1, squota=30}
M3. /dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1, squota=-1 }
{quote}
At present when we call Quota.setQuota(), all its children paths will be set
too. Quota in M1 will be set on M1,M2,M3 and quota in M2 will be set on M2,M3.
So there will be conflicts on M2's \{nquota} and M3's \{nquota,squota}. I want
to use inherited global quota to resolve this. If one mount table's global
quota is unset, it inherits its parent mount table's global quota.
> RBF: Make Global quota and Remote quota consistent.
> ---
>
> Key: HDFS-14813
> URL: https://issues.apache.org/jira/browse/HDFS-14813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
>
> Make Global quota and remote quota consistent.
> (Global quota: the quota on mount table, Remote quota: the quota on
> namespace)
> HDFS administrator can use global quota to simplify the management for
> federation paths. But there is no consistent constraint for the global quota
> and the remote quota. As an hdfs administrator the inconsistent has 3
> disadvantages on management:
> 1. The quota part of getQuotaUsage() on a federation path is not
> helpful. It's neither the global quota nor one of the remote quotas.
> 2. The global quota could be different with the remote quota. When a
> QuotaExceedException happens it needs the administrator to find out whether
> it's a violation of the global quota or the remote quota.
> 3. For management simplicity, it's always a good idea to keep the global
> quota and the remote quota the same. Now we need the administrator to keep
> the consistent manually.
> My proposal is to add a constraint for global quota:
> 1. For federation paths, global quota could be inherited from parent
> federation path.
> 2. For all remote paths in mount tables, the remote quotas must be
> consistent with the global quotas.
> To implement this, my idea is:
> 1. Global quota could be inherited. Add a method getGlobalQuota(String
> path) to Quota.java returning the global quota.
> 2. Each time RouterQuotaUpdateService updates the quota usage for mount
> table entries, it also checks and updates the remote quota.
> 3. When getQuotaUsage() on a federation path, return the global quota.
> 4. When setQuota() on a federation path, first update the global quota
> in mount table, then recompute global quota for the current path and its
> children paths, finally update all the federation paths.
>
> Implement 1+2 in HDFS-14814
> Implement 4 in HDFS-14815
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929972#comment-16929972 ] Ayush Saxena commented on HDFS-14813: - Thanx for confirming, makes sense to me, if there is no as such cost, but I guess the 1 and 2 are pretty independent, May be we shouldn't merge it together. 2 is quite independent alone and valid in the present setup too. > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929966#comment-16929966 ] Jinglun commented on HDFS-14813: Yes, that exactly what I mean ! > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[
https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929961#comment-16929961
]
Ayush Saxena commented on HDFS-14813:
-
Thanx [~LiJinglun] If I am understanding it correct, Do you mean to say in
{{RouterQuotaUpdateService}} we even check whether the namespace quota is
correct in every iteration, to prevent or say counter, if any changes done
directly by a user on the namespace? if so we we sync the quota again to make
the namespace quota aligned with the mount table quota?
> RBF: Make Global quota and Remote quota consistent.
> ---
>
> Key: HDFS-14813
> URL: https://issues.apache.org/jira/browse/HDFS-14813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
>
> Make Global quota and remote quota consistent.
> (Global quota: the quota on mount table, Remote quota: the quota on
> namespace)
> HDFS administrator can use global quota to simplify the management for
> federation paths. But there is no consistent constraint for the global quota
> and the remote quota. As an hdfs administrator the inconsistent has 3
> disadvantages on management:
> 1. The quota part of getQuotaUsage() on a federation path is not
> helpful. It's neither the global quota nor one of the remote quotas.
> 2. The global quota could be different with the remote quota. When a
> QuotaExceedException happens it needs the administrator to find out whether
> it's a violation of the global quota or the remote quota.
> 3. For management simplicity, it's always a good idea to keep the global
> quota and the remote quota the same. Now we need the administrator to keep
> the consistent manually.
> My proposal is to add a constraint for global quota:
> 1. For federation paths, global quota could be inherited from parent
> federation path.
> 2. For all remote paths in mount tables, the remote quotas must be
> consistent with the global quotas.
> To implement this, my idea is:
> 1. Global quota could be inherited. Add a method getGlobalQuota(String
> path) to Quota.java returning the global quota.
> 2. Each time RouterQuotaUpdateService updates the quota usage for mount
> table entries, it also checks and updates the remote quota.
> 3. When getQuotaUsage() on a federation path, return the global quota.
> 4. When setQuota() on a federation path, first update the global quota
> in mount table, then recompute global quota for the current path and its
> children paths, finally update all the federation paths.
>
> Implement 1+2 in HDFS-14814
> Implement 4 in HDFS-14815
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929953#comment-16929953 ] Jinglun commented on HDFS-14813: Hi [~ayushtkn], yes HDFS-13346 will help! My thought is to enable RouterQuotaUpdateService to fix the not sync paths. The NameNode can still be accessed directly so the quota might be modified wrong by users. (In Xiaomi we enable users to set quota for sub-directories of users' home path). I think adding check in RouterQuotaUpdateService can make it safer. Since the RouterQuotaUpdateService already gets all QuotaUsage, checking and updating quota won't be additional big cost. [be a burden|http://dict.youdao.com/search?q=be%20a%20burden=chrome.extension] [详细|http://www.youdao.com/search?q=be%20a%20burden=utf8=chrome.extension]X 没有英汉互译结果 [请尝试网页搜索|http://www.youdao.com/search?q=be%20a%20burden=utf8=chrome.extension] > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929930#comment-16929930 ] Ayush Saxena commented on HDFS-14813: - Hey [~LiJinglun], Just started reading the proposal. A minor doubt : bq. 2. For all remote paths in mount tables, the remote quotas must be consistent with the global quotas. In HDFS-13346 the mount quota was made to be in sync with the NS quota, does that not help? > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929919#comment-16929919 ] Jinglun commented on HDFS-14813: Hi [~ayushtkn], could you help taking a look at the proposals? Looking forward to your comments. > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
