[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307398#comment-16307398
]
Yiqun Lin edited comment on HDFS-12934 at 1/1/18 12:49 PM:
-----------------------------------------------------------
Hi [~elgoiri], following are my comments:
bq. I've checked MountTableResolver and it might make sense to use a similar
TreeMap for the map in RouterQuotaLocalCache. It wo...
Using TreeMap will benefit for function {{getChildrenPath}} I think. I don't
think this can simplify the recursive search for function
{{RouterQuotaLocalCache#getQuotaUsage}}. Am I missing something? In
additionally, use the read/write lock to make this thread-safe to use.
bq. In Router, I would rename getChildrenPaths(String path) into
getQuotaUsage(String path) and getQuotaUsageCache() getQuotaUsage()..
I didn't fully understanding on this. Do you mean following changes:
{noformat}
getChildrenPaths -->getQuotaUsage
getQuotaUsageCache-->getQuotaUsage
{noformat}
The first change looks strange. I only renamed getQuotaUsageCache to
getQuotaUsage.
{quote}
Typo NamaService in RouterRpcServer and loc.getNameserviceId() should be
extracted.
Update TestMountTable to include the new field for the quota.
{quote}
Addressed.
{quote}
In TestRouterQuota, I think we shouldn't use the wait for breaking the quota
but actually create successfully the first 3 and then fail in the next one.
{quote}
Since quota cache service is a periodical service, we should ensure the mount
tables are loaded and their quota are updated. So I use
{{GenericTestUtils#waitFor}} for waiting this instead of just sleeping fixed
time. Another way, we don't need to wait for quota being updated, we can just
call {{periodicInvoke}} for trigger this. But this is not a simulated real
scene that we want to test.
I found test of setQuota/getQuota can be simplified. Waiting operation is not
needed. I have updated on this.
bq. Do you mind opening the JIRA for the doc and the UI?
Had filed JIRA HDFS-12972 and HDFS-12973 for tracking this.
Attach the updated patch.
was (Author: linyiqun):
Hi [~elgoiri], following are my comments:
bq. I've checked MountTableResolver and it might make sense to use a similar
TreeMap for the map in RouterQuotaLocalCache. It wo...
Using TreeMap will benefit for function {{getChildrenPath}} I think. I don't
think this can simplify the recursive search for function
{{RouterQuotaLocalCache#getQuotaUsage}}. Am I missing something? In
additionally, use the read/write lock to make this thread-safe to use.
bq. In Router, I would rename getChildrenPaths(String path) into
getQuotaUsage(String path) and getQuotaUsageCache() getQuotaUsage()..
I didn't fully understanding on this. Do you mean following changes:
{noformat}
getChildrenPaths -->getQuotaUsage
getQuotaUsageCache-->getQuotaUsage
{noformat}
The first change looks strange. I only renamed getQuotaUsageCache to
getQuotaUsage.
{quote}
Typo NamaService in RouterRpcServer and loc.getNameserviceId() should be
extracted.
Update TestMountTable to include the new field for the quota.
{quote}
Addressed.
{quote}
In TestRouterQuota, I think we shouldn't use the wait for breaking the quota
but actually create successfully the first 3 and then fail in the next one.
{quote}
Since quota cache service is a periodical service, we should ensure the mount
tables are loaded and their quota are updated. So I use
{{GenericTestUtils#waitFor}} for waiting this instead of just sleeping fixed
time. This is a simulated real scene. I found test of setQuota/getQuota can be
simplified. Waiting operation is not needed. I have updated on this.
bq. Do you mind opening the JIRA for the doc and the UI?
Had filed JIRA HDFS-12972 and HDFS-12973 for tracking this.
Attach the updated patch.
> RBF: Federation supports global quota
> -------------------------------------
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: 3.0.0
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch,
> HDFS-12934.003.patch, HDFS-12934.004.patch, RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one <Path, QuotaUsage> cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns <nsQuota> -ss <ssQuota> <mount table>
> hdfs dfsrouteradmin -clrQuota <mount table>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
