[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295480#comment-16295480
]
Íñigo Goiri commented on HDFS-12934:
------------------------------------
Thanks [~linyiqun] for proposing this. We internally don't use quotas yet so
I'm not very familiar with the semantics that users/admins expect from this.
But let me throw a couple thoughts here.
If I understood correctly, your proposal would be to do quota management per
mount table.
In that case, we may want to create a new Quota structure in the State Store to
keep track of the usage (and maybe even the quota definition) separately.
Not sure who would keep this data up to date across Routers though, we will
need some synchronization to do it distributed.
In addition to the global quota, we may want to think if it makes sense to
implement functions like {{setQuota()}} from {{ClientProtocol}}.
It could even be the interface for setting the quota instead of adding it to
the {{dfsrouteradmin}}.
In addition, we could move all the bookkeeping to the Namenodes and just query
the quotas from those and just aggregate the values from the subclusters.
[~ajayydv], HDFS-12512 will add WebHDFS and could support the second approach
using {{setQuota()}}.
For the pure admin interface, we don't have any REST interface.
This might be a good reason to implement quotas through {{ClientProtocol}}.
> RBF: Federation supports global quota
> -------------------------------------
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: 3.0.0
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Labels: RBF
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one <Path, QuotaUsage> cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns <nsQuota> -ss <ssQuota> <mount table>
> hdfs dfsrouteradmin -clrQuota <mount table>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
