[
https://issues.apache.org/jira/browse/HADOOP-17997?focusedWorklogId=680356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-680356
]
ASF GitHub Bot logged work on HADOOP-17997:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Nov/21 17:09
Start Date: 11/Nov/21 17:09
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3634:
URL: https://github.com/apache/hadoop/pull/3634#discussion_r747676443
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Quota.java
##########
@@ -50,7 +50,7 @@ public static Counts newInstance() {
* The quota is violated if quota is set and usage > quota.
*/
public static boolean isViolated(final long quota, final long usage) {
- return quota >= 0 && usage > quota;
+ return quota >= 0 && usage >= quota;
Review comment:
I think the method is correct only, It is used for LOGS in core HDFS
parts, and Quota is said to be violated when the count is more than the value
set, if the fileAndDirCount is 5 and quota is also 5, then quota isn't said to
be violated.
The javadoc also tells that.
Regarding RBF, I think you need to change the call here to pass the delta,
while verifying :
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUsage.java#L99
Rather than the method which you tweaked use the below one with proper delta:
```
static boolean isViolated(final long quota, final long usage,
final long delta) {
return quota >= 0 && delta > 0 && usage > quota - delta;
}
```
That I think should solve the problem here. Not sure how things will work in
case of storage quota though....
Wrote a UT in **TestRouterRPCMultipleDestinationMountTableResolver** , to
check this, putting it up here as well in case anyone wants to try, might save
some efforts.
```
@Test
public void testQuotaMultiDest() throws Exception {
// NS-Quota as 4
long nsQuota = 4;
Path path = new Path("/test10");
nnFs0.mkdirs(path);
nnFs1.mkdirs(path);
Map<String, String> destMap = new HashMap<>();
destMap.put("ns0", "/test10");
destMap.put("ns1", "/test10");
MountTable addEntry = MountTable.newInstance("/test10", destMap);
assertTrue(addMountTable(addEntry));
RouterQuotaUpdateService updateService =
routerContext.getRouter().getQuotaCacheUpdateService();
updateService.periodicInvoke();
// Set NS-Quota as 4
RouterAdmin admin = getRouterAdmin();
String[] argv = new String[] {"-setQuota", path.toString(), "-nsQuota",
String.valueOf(nsQuota)};
assertEquals(0, ToolRunner.run(admin, argv));
updateService.periodicInvoke();
resolver.loadCache(true);
// Touch two files under ns-fed/tes10
routerFs.create(new Path("/test10/file1")).close();
routerFs.create(new Path("/test10/file2")).close();
updateService.periodicInvoke();
// This should have thrown exception since 2 target directories & 2
files: total 4 & Quota is 4
routerFs.create(new Path("/test10/file3")).close();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 680356)
Time Spent: 1h 20m (was: 1h 10m)
> RBF: Namespace usage of mount table with multi subclusters can exceed quota
> ---------------------------------------------------------------------------
>
> Key: HADOOP-17997
> URL: https://issues.apache.org/jira/browse/HADOOP-17997
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: xichaomin
> Priority: Major
> Labels: pull-request-available
> Attachments: 1636424307319.jpg
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> # mkdir on each subcluster /test10
> # mount table: hdfs dfsrouteradmin -add /test10 ns1,ns2 /test10
> # set quota: hdfs dfsrouteradmin -setQuota /test10 -nsQuota 4
> # touch two files under hdfs://\{fed}/test10, now the namespace usage is 4(2
> directories and 2 files).
> # refresh and put a new file, the put is successfull, and the namespace
> usage comes to 5.
> !1636424307319.jpg!
> The router checks quota without considering the increments, It is difficult
> to limit quota precisely without increments , but throw exception when usage
> >= quota will be better.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
