[
https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180637#comment-17180637
]
Ayush Saxena commented on HDFS-15510:
-------------------------------------
Was looking after the quota stuff in case of {{Multiple Destinations}} and on a
quick glance the approach #3 actually made sense in the first go. But, Since
there was even #2 suggested above by [~elgoiri], I tried digging a bit more.
I think the approach #3 does good when quota is set on the mount entry as
{{/router_test}}, but if this directory follows {{isPathAll}} order and you
create a directory {{/router_test/dir}} and then set quota on it, If there are
two namespaces, the {{setQuota}} will be a concurrent call and set on both
namespaces, The point is with #3 in count command the quota will shows correct,
but will it throw Quota exceeded exception at the value? I tried a quick UT and
it didn't. since for the namespace directories the quota is maintained at
router, For that case something near to approach #2, makes the functionality
work(Not a best solution either, due to restriction on randomness of
distribution), Can you check once, if for directories quota is protected, not
just {{mount entries}}
if that is protected, then coming to current patch, I tried this as well for
the mount entries, The present code, overlooks the overflow of {{LONG}} due to
addition of quota, set nsQuota to {{LONG.MAX-2}} in the current code and this
goes wrong. So, that also I guess needs to taken care of.
This the modified UT for the directories stuff :
{code:java}
@Test
public void testContentSummaryWithMultipleDest() throws Exception {
MountTable addEntry;
long nsQuota = 5;
long ssQuota = 100;
Path path = new Path("/router_test");
Map<String, String> destMap = new HashMap<>();
destMap.put("ns0", "/router_test");
destMap.put("ns1", "/router_test");
nnFs0.mkdirs(path);
nnFs1.mkdirs(path);
addEntry = MountTable.newInstance("/router_test", destMap);
addEntry.setDestOrder(DestinationOrder.HASH_ALL);
assertTrue(addMountTable(addEntry));
RouterQuotaUpdateService updateService =
routerContext.getRouter().getQuotaCacheUpdateService();
updateService.periodicInvoke();
routerFs.mkdirs(new Path("/router_test/dir"));
routerFs.setQuota(new Path("/router_test/dir"),nsQuota,ssQuota);
for(int i=0; i<nsQuota;i++)
routerFs.create(new Path("/router_test/dir/file"+i)).close();
updateService.periodicInvoke();
routerFs.create(new Path("/router_test/dir/file")).close();
resolver.loadCache(true);
ContentSummary cs = routerFs.getContentSummary(new
Path("/router_test/dir"));
assertEquals(nsQuota, cs.getQuota());
assertTrue(cs.getFileAndDirectoryCount() <= nsQuota);
routerFs.create(new Path("/router_test/dir/file")).close();
assertEquals(100, cs.getSpaceQuota());
}
{code}
I tried all random stuff, loadCache, periodic invoke and stuff like that, but
didn't got {{QuotaExceededException}}
Try once, I might have missed something in this, couldn't get time to check,
if so, please feel free to ignore and move ahead, if others are happy. :)
> RBF: Quota and Content Summary was not correct in Multiple Destinations
> -----------------------------------------------------------------------
>
> Key: HDFS-15510
> URL: https://issues.apache.org/jira/browse/HDFS-15510
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Hemanth Boyina
> Assignee: Hemanth Boyina
> Priority: Critical
> Attachments: 15510.png, HDFS-15510.001.patch, HDFS-15510.002.patch
>
>
> steps :
> *) create a mount entry with multiple destinations ( for suppose 2)
> *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content
> Summary on the Mount Entry shows NS quota as 20
> *) Create 10 files through router, on creating 11th file , NS Quota Exceeded
> Exception is coming
> though the Content Summary showing the NS quota as 20 , we are not able to
> create 20 files
>
> the problem here is router stores the mount entry's NS quota as 10 , but
> invokes NS quota on both the name services by set NS quota as 10 , so content
> summary on mount entry aggregates the content summary of both the name
> services by making NS quota as 20
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
