[jira] [Comment Edited] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[
https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184173#comment-17184173
]
Íñigo Goiri edited comment on HDFS-15510 at 8/25/20, 4:20 PM:
--
* Why are we adding {{path}} to {{aggregateContentSummary()}}?
* There is a checkstyle warning.
was (Author: elgoiri):
* Why are we adding {{aggregateContentSummary()}}?
* There is a checkstyle warning.
> RBF: Quota and Content Summary was not correct in Multiple Destinations
> ---
>
> Key: HDFS-15510
> URL: https://issues.apache.org/jira/browse/HDFS-15510
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Critical
> Attachments: 15510.png, HDFS-15510.001.patch, HDFS-15510.002.patch,
> HDFS-15510.003.patch
>
>
> steps :
> *) create a mount entry with multiple destinations ( for suppose 2)
> *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content
> Summary on the Mount Entry shows NS quota as 20
> *) Create 10 files through router, on creating 11th file , NS Quota Exceeded
> Exception is coming
> though the Content Summary showing the NS quota as 20 , we are not able to
> create 20 files
>
> the problem here is router stores the mount entry's NS quota as 10 , but
> invokes NS quota on both the name services by set NS quota as 10 , so content
> summary on mount entry aggregates the content summary of both the name
> services by making NS quota as 20
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[ https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179141#comment-17179141 ] Brahma Reddy Battula edited comment on HDFS-15510 at 8/17/20, 5:30 PM: --- thanks [~hemanthboyina] reporting this issue. IMO, #3 will be better choice as you also suggested because application will aware what they set and based on that application might validate. Setting Quota on multiple destinations with same value should fine I feel * As these commnads will come from the router to namenodes,application/user might not interact directly. * On unavailablity of other namespaces user can able to create files ( e.g Say two namespaces NS1,NS2 with 10 quota, NS1 is unavaiable due to some reason). Hoping when applications configured with router they shouldn't connect to nameserice directly . Even #2 also better, but these needs change current invocation of the commands as the quota value will be different for nameservices and unavailable of name serice might mislead user. E.G : * ns1-5, ns2-5, while writing 9th file NS2 is avialble, then it will quota exceed exception with "5" even user set "10". * ns1-7,ns2-7,ns3-6,if ns3 is unaviable we always fail write after 14 files, even we've available namespace ns1 and ns2 #1, this might not better than above two, as this can allow more than application/user set. was (Author: brahmareddy): thanks [~hemanthboyina] reporting this issue. IMO, #3 will be better choice as you also suggested because application will aware what they set and based on that application might validate. Setting Quota on multiple destinations with same value should fine I feel * As these commnads will come from the router to namenodes,application/user might not interact direc * On unavailablity of other namespaces user can able to create files ( e.g Say two namespaces NS1,NS2 with 10 quota, NS1 is unavaiable due to some reason). Hoping when applications configured with router they shouldn't connect to nameserice directly . Even #2 also better, but these needs change current invocation of the commands as the quota value will be different for nameservices and unavailable of name serice might mislead user. E.G : * ns1-5, ns2-5, while writing 9th file NS2 is avialble, then it will quota exceed exception with "5" even user set "10". * ns1-7,ns2-7,ns3-6,if ns3 is unaviable we always fail write after 14 files, even we've available namespace ns1 and ns2 #1, this might not better than above two, as this can allow more than application/user set. > RBF: Quota and Content Summary was not correct in Multiple Destinations > --- > > Key: HDFS-15510 > URL: https://issues.apache.org/jira/browse/HDFS-15510 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Critical > Attachments: 15510.png > > > steps : > *) create a mount entry with multiple destinations ( for suppose 2) > *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content > Summary on the Mount Entry shows NS quota as 20 > *) Create 10 files through router, on creating 11th file , NS Quota Exceeded > Exception is coming > though the Content Summary showing the NS quota as 20 , we are not able to > create 20 files > > the problem here is router stores the mount entry's NS quota as 10 , but > invokes NS quota on both the name services by set NS quota as 10 , so content > summary on mount entry aggregates the content summary of both the name > services by making NS quota as 20 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[
https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176129#comment-17176129
]
Hemanth Boyina edited comment on HDFS-15510 at 8/12/20, 8:04 AM:
-
{quote}The issue here seems to be that the general one is not accounted
correctly, we would need to fix that.
{quote}
for an example : if we set namespace quota as 10 on mount entry with 2
destinations , for the both destinations we set quota as 10 each and in mount
table store for the mount entry we set namespace quota as 10 which is not
correct
Periodic Invoke gets quota usage from the destinations name service and gets
Quota set value from mount table store
was (Author: hemanthboyina):
{quote}The issue here seems to be that the general one is not accounted
correctly, we would need to fix that.
{quote}
for an example : if we set namespace quota as 10 on mount entry with 2
destinations , for the both destinations we set quota as 10 each and in mount
table store for the mount entry we set namespace quota as 10 which is not
correct
> RBF: Quota and Content Summary was not correct in Multiple Destinations
> ---
>
> Key: HDFS-15510
> URL: https://issues.apache.org/jira/browse/HDFS-15510
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Critical
>
> steps :
> *) create a mount entry with multiple destinations ( for suppose 2)
> *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content
> Summary on the Mount Entry shows NS quota as 20
> *) Create 10 files through router, on creating 11th file , NS Quota Exceeded
> Exception is coming
> though the Content Summary showing the NS quota as 20 , we are not able to
> create 20 files
>
> the problem here is router stores the mount entry's NS quota as 10 , but
> invokes NS quota on both the name services by set NS quota as 10 , so content
> summary on mount entry aggregates the content summary of both the name
> services by making NS quota as 20
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
