[jira] [Comment Edited] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809659#comment-16809659 ] Shardul Singh edited comment on HBASE-22142 at 4/4/19 9:36 AM: --- I would like to work on this issue..I'll provide a patch:) was (Author: shardulsingh): I would like to work on this issue..Can someone assign it to me:) > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry from cache during the drop command as part of MasterCP > hook only, so that correct usage is show instantaneously after the drop > command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22142: -- Description: steps to follow: 1.create a quota at namespace level 2.create 2 tables t1 and t2 inside namespace and write data. 3.write 5 mb of data each in both t1 and t2. 4. drop t1. 5. data usage for t2 will be shown 10 mb for 10 minutes. If table is dropped inside namespace, data size usage is still shown for 10 minutes because of the configuration "hbase.master.quotas.region.report.retention.millis". This conf maintains the region report in cache(regionSizes) and old entry is used for 10 minutes. We can remove the entry from cache during the drop command as part of MasterCP hook only, so that correct usage is show instantaneously after the drop command. was: steps to follow: 1.create a quota at namespace level 2.create 2 tables t1 and t2 inside namespace and write data. 3.write 5 mb of data each in both t1 and t2. 4. drop t1. 5. data usage for t2 will be shown 10 mb for 10 minutes. If table is dropped inside namespace, data size usage is still shown for 10 minutes because of the configuration "hbase.master.quotas.region.report.retention.millis". This conf maintains the region report in cache(regionSizes) and old entry is used for 10 minutes. We can remove the entry during the drop command only, so that correct usage is show instantaneously after the drop command. > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry from cache during the drop command as part of MasterCP > hook only, so that correct usage is show instantaneously after the drop > command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809659#comment-16809659 ] Shardul Singh commented on HBASE-22142: --- I would like to work on this issue..Can someone assign it to me:) > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry during the drop command only, so that correct usage is > show instantaneously after the drop command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22142) Space quota: If table is dropped inside namespace, data size usage is still considered for the drop table.
Shardul Singh created HBASE-22142: - Summary: Space quota: If table is dropped inside namespace, data size usage is still considered for the drop table. Key: HBASE-22142 URL: https://issues.apache.org/jira/browse/HBASE-22142 Project: HBase Issue Type: Bug Affects Versions: 2.1.1 Reporter: Shardul Singh steps to follow: 1.create a quota at namespace level 2.create 2 tables t1 and t2 inside namespace and write data. 3.write 5 mb of data each in both t1 and t2. 4. drop t1. 5. data usage for t2 will be shown 10 mb for 10 minutes. If table is dropped inside namespace, data size usage is still shown for 10 minutes because of the configuration "hbase.master.quotas.region.report.retention.millis". This conf maintains the region report in cache(regionSizes) and old entry is used for 10 minutes. We can remove the entry during the drop command only, so that correct usage is show instantaneously after the drop command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22142: -- Summary: Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table. (was: Space quota: If table is dropped inside namespace, data size usage is still considered for the drop table.) > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry during the drop command only, so that correct usage is > show instantaneously after the drop command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22012) SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable table
[ https://issues.apache.org/jira/browse/HBASE-22012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814077#comment-16814077 ] Shardul Singh commented on HBASE-22012: --- [~elserj] Hi josh, I think just adding failsafe when we're checking for a quota that has the DisableTable violation policy don't filter in *tablesToRemove* won't work because after retention period for region report expires it removes the entry from the cache regionSizes, but since region report size are not present in cache, *targetSnapshot* in quotaObserverChore will give 0B as usage and table still get's enabled during *processTablesWithQuotas* step*.* I think it's better to not remove the entries from cache regionSizes if table is in *violation* and has *DISABLE* policy set on it. > SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable > table > > > Key: HBASE-22012 > URL: https://issues.apache.org/jira/browse/HBASE-22012 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.1 >Reporter: Ajeet Rai >Assignee: Nihal Jain >Priority: Major > > Space Quota: Policy state is getting changed from disable to Observance after > sometime automatically. > Steps: > 1: Create a table with space quota policy as Disable > 2: Put some data so that table state is in space quota violation > 3: So observe that table state is in violation > 4: Now wait for some time > 5: Observe that after some time table state is changing to to Observance > however table is still disabled > edit (elserj): The table is automatically moved back from the violation state > because of the code added that tried to ride over RITs. When a Region is not > online (whether normally or abnormally), the RegionSizeReports are not sent > from RS to Master. Eventually, enough Regions are not reported which dips > below the acceptable threshold and we automatically move the table back to > the "acceptable" space quota state (not in violation). We could skip this > failsafe when we're checking for a quota that has the DisableTable violation > policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-22012) SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable table
[ https://issues.apache.org/jira/browse/HBASE-22012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814077#comment-16814077 ] Shardul Singh edited comment on HBASE-22012 at 4/10/19 6:02 AM: [~elserj] Hi josh, I think just adding failsafe when we're checking for a quota that has the DisableTable violation policy don't filter in *tablesToRemove* won't work because after retention period for region report expires it removes the entry from the cache regionSizes, but since region report size are not present in cache, *targetSnapshot* in quotaObserverChore will give 0B as usage and table still get's enabled during *processTablesWithQuotas* step. I think it's better to not remove the entries from cache regionSizes if table is in *violation* and has *DISABLE* policy set on it. was (Author: shardulsingh): [~elserj] Hi josh, I think just adding failsafe when we're checking for a quota that has the DisableTable violation policy don't filter in *tablesToRemove* won't work because after retention period for region report expires it removes the entry from the cache regionSizes, but since region report size are not present in cache, *targetSnapshot* in quotaObserverChore will give 0B as usage and table still get's enabled during *processTablesWithQuotas* step*.* I think it's better to not remove the entries from cache regionSizes if table is in *violation* and has *DISABLE* policy set on it. > SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable > table > > > Key: HBASE-22012 > URL: https://issues.apache.org/jira/browse/HBASE-22012 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.1 >Reporter: Ajeet Rai >Assignee: Nihal Jain >Priority: Major > > Space Quota: Policy state is getting changed from disable to Observance after > sometime automatically. > Steps: > 1: Create a table with space quota policy as Disable > 2: Put some data so that table state is in space quota violation > 3: So observe that table state is in violation > 4: Now wait for some time > 5: Observe that after some time table state is changing to to Observance > however table is still disabled > edit (elserj): The table is automatically moved back from the violation state > because of the code added that tried to ride over RITs. When a Region is not > online (whether normally or abnormally), the RegionSizeReports are not sent > from RS to Master. Eventually, enough Regions are not reported which dips > below the acceptable threshold and we automatically move the table back to > the "acceptable" space quota state (not in violation). We could skip this > failsafe when we're checking for a quota that has the DisableTable violation > policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22142: -- Attachment: HBASE-22142.branch-2.1.v001.patch > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > Attachments: HBASE-22142.branch-2.1.v001.patch > > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry from cache during the drop command as part of MasterCP > hook only, so that correct usage is show instantaneously after the drop > command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22142: -- Status: Patch Available (was: Open) > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Priority: Minor > Attachments: HBASE-22142.branch-2.1.v001.patch > > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry from cache during the drop command as part of MasterCP > hook only, so that correct usage is show instantaneously after the drop > command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835859#comment-16835859
]
Shardul Singh commented on HBASE-20821:
---
hi [~elserj]..can you please review this patch...thanks in advance? :-)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835859#comment-16835859
]
Shardul Singh edited comment on HBASE-20821 at 5/8/19 7:52 PM:
---
hi [~elserj]..can you please review this patch?...thanks in advance. :-)
was (Author: shardulsingh):
hi [~elserj]..can you please review this patch...thanks in advance? :-)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-22342) space quota usage is calculated almost double then actual usage in case of Split /Merge of region
[ https://issues.apache.org/jira/browse/HBASE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-22342: - Assignee: Shardul Singh (was: Uma Maheswari) > space quota usage is calculated almost double then actual usage in case of > Split /Merge of region > - > > Key: HBASE-22342 > URL: https://issues.apache.org/jira/browse/HBASE-22342 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Minor > Labels: quota, space > > space quota usage is calculated double for some time because it is > considering the usage of the parent region report received earlier. This is > normal behaviour because *hbase.master.quotas.region.report.retention.millis* > is set as 10 minutes.It's not calculating the usage from the store files, > it's taking the usage from the earlier report send to the master by the > parent region server till 10 minutes. > So after 10 minutes, it stops considering the earlier region report and quota > is not violated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834443#comment-16834443
]
Shardul Singh commented on HBASE-20821:
---
Hi [~nihaljain.cs] , I have a patch for this jiraif you are not working on
this..should i go ahead and attach the patch?
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834567#comment-16834567
]
Shardul Singh edited comment on HBASE-20821 at 5/7/19 9:36 AM:
---
ping [~elserj] - Hi , ..I would like to work on this jira...Can you add me as
a contributor?
was (Author: shardulsingh):
ping [~elserj] - I would like to work on this jira...Can you add me as a
contributor?
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834567#comment-16834567
]
Shardul Singh commented on HBASE-20821:
---
ping [~elserj] - I would like to work on this jira...Can you add me as a
contributor?
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Fix Version/s: 2.1.0
3.0.0
Affects Version/s: 3.0.0
2.1.0
Status: Patch Available (was: Open)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 3.0.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: HBASE-20821.master.v001patch
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Attachments: HBASE-20821.master.v001patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: HBASE-20821.master.v001patch
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: (was: HBASE-20821.master.v001patch)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: HBASE-20821.master.v002.patch
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-22142) Space quota: If table inside namespace having space quota is dropped, data size usage is still considered for the drop table.
[ https://issues.apache.org/jira/browse/HBASE-22142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-22142: - Assignee: Shardul Singh > Space quota: If table inside namespace having space quota is dropped, data > size usage is still considered for the drop table. > -- > > Key: HBASE-22142 > URL: https://issues.apache.org/jira/browse/HBASE-22142 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Shardul Singh >Assignee: Shardul Singh >Priority: Minor > Labels: Quota, space > Attachments: HBASE-22142.branch-2.1.v001.patch > > > steps to follow: > 1.create a quota at namespace level > 2.create 2 tables t1 and t2 inside namespace and write data. > 3.write 5 mb of data each in both t1 and t2. > 4. drop t1. > 5. data usage for t2 will be shown 10 mb for 10 minutes. > If table is dropped inside namespace, data size usage is still shown for 10 > minutes because of the configuration > "hbase.master.quotas.region.report.retention.millis". This conf maintains the > region report in cache(regionSizes) and old entry is used for 10 minutes. We > can remove the entry from cache during the drop command as part of MasterCP > hook only, so that correct usage is show instantaneously after the drop > command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: HBASE-20821.master.v003.patch
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835526#comment-16835526
]
Shardul Singh commented on HBASE-20821:
---
TestCase Failures are passing locally...Submiting the patch again.
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22012) SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable table
[ https://issues.apache.org/jira/browse/HBASE-22012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814835#comment-16814835 ] Shardul Singh commented on HBASE-22012: --- [~elserj] Hey thanks for replying...no I just tried your suggestion and it was not working so came up with an alternative. [~nihaljain.cs] Anything is fine with me..i can also work on it..but if you want to continue no problem..thanks :) > SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable > table > > > Key: HBASE-22012 > URL: https://issues.apache.org/jira/browse/HBASE-22012 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.1 >Reporter: Ajeet Rai >Assignee: Nihal Jain >Priority: Major > > Space Quota: Policy state is getting changed from disable to Observance after > sometime automatically. > Steps: > 1: Create a table with space quota policy as Disable > 2: Put some data so that table state is in space quota violation > 3: So observe that table state is in violation > 4: Now wait for some time > 5: Observe that after some time table state is changing to to Observance > however table is still disabled > edit (elserj): The table is automatically moved back from the violation state > because of the code added that tried to ride over RITs. When a Region is not > online (whether normally or abnormally), the RegionSizeReports are not sent > from RS to Master. Eventually, enough Regions are not reported which dips > below the acceptable threshold and we automatically move the table back to > the "acceptable" space quota state (not in violation). We could skip this > failsafe when we're checking for a quota that has the DisableTable violation > policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Fix Version/s: 2.1.0 > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-22013: - Assignee: Shardul Singh > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906878#comment-16906878 ] Shardul Singh commented on HBASE-22013: --- Thanks :) [~Uma Murugesan] > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HBASE-22012) SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable table
[ https://issues.apache.org/jira/browse/HBASE-22012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-22012: - Assignee: Shardul Singh (was: Nihal Jain) > SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable > table > > > Key: HBASE-22012 > URL: https://issues.apache.org/jira/browse/HBASE-22012 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.1 >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > > Space Quota: Policy state is getting changed from disable to Observance after > sometime automatically. > Steps: > 1: Create a table with space quota policy as Disable > 2: Put some data so that table state is in space quota violation > 3: So observe that table state is in violation > 4: Now wait for some time > 5: Observe that after some time table state is changing to to Observance > however table is still disabled > edit (elserj): The table is automatically moved back from the violation state > because of the code added that tried to ride over RITs. When a Region is not > online (whether normally or abnormally), the RegionSizeReports are not sent > from RS to Master. Eventually, enough Regions are not reported which dips > below the acceptable threshold and we automatically move the table back to > the "acceptable" space quota state (not in violation). We could skip this > failsafe when we're checking for a quota that has the DisableTable violation > policy. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: (was: HBASE-22013.master.v001.patch) > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: HBASE-22013.master.002.patch HBASE-22013.master.001.patch > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.001.patch, > HBASE-22013.master.002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: (was: HBASE-20821.master.v001patch)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: (was: HBASE-20821.master.v002.patch)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: (was: HBASE-20821.master.v003.patch)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Attachment: HBASE-20821.master.001.patch
HBASE-20821.master.002.patch
HBASE-20821.master.003.patch
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.001.patch,
> HBASE-20821.master.002.patch, HBASE-20821.master.003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908720#comment-16908720
]
Shardul Singh commented on HBASE-20821:
---
hi [~jatsakthi], can you review this patch?
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.001.patch,
> HBASE-20821.master.002.patch, HBASE-20821.master.003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908718#comment-16908718 ] Shardul Singh commented on HBASE-22013: --- Thanks [~jatsakthi] for reviewing the patch and sorry for the naming convention mistake. As told, I have handled the review comments given by you and have uploaded a new patch. > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.001.patch, > HBASE-22013.master.002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: (was: HBASE-22013.master.002.patch) > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.001.patch, > HBASE-22013.master.002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: HBASE-22013.master.002.patch > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.001.patch, > HBASE-22013.master.002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: (was: HBASE-22013.master.v002.patch) > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Status: Patch Available (was: Open) > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: HBASE-22013.master.v002.patch > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.v001.patch, > HBASE-22013.master.v002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911124#comment-16911124 ] Shardul Singh commented on HBASE-22013: --- Hi [~jatsakthi], can we push this in if everything is fine?..thanks :) > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.master.001.patch, > HBASE-22013.master.002.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22342) space quota usage is calculated almost double then actual usage in case of Split /Merge of region
[ https://issues.apache.org/jira/browse/HBASE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908831#comment-16908831 ] Shardul Singh commented on HBASE-22342: --- Hi, [~jatsakthi] Will submit a patch this weekend for this isssue :) > space quota usage is calculated almost double then actual usage in case of > Split /Merge of region > - > > Key: HBASE-22342 > URL: https://issues.apache.org/jira/browse/HBASE-22342 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Minor > Labels: quota, space > > space quota usage is calculated double for some time because it is > considering the usage of the parent region report received earlier. This is > normal behaviour because *hbase.master.quotas.region.report.retention.millis* > is set as 10 minutes.It's not calculating the usage from the store files, > it's taking the usage from the earlier report send to the master by the > parent region server till 10 minutes. > So after 10 minutes, it stops considering the earlier region report and quota > is not violated -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913067#comment-16913067 ] Shardul Singh commented on HBASE-22013: --- Yes sure..no problem :)..Yes it is better to refactor because lot of space quota issue might have same problem > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > hbase-22013.branch-2.001.patch, hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22895) Fix the flakey TestSpaceQuotas
[ https://issues.apache.org/jira/browse/HBASE-22895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22895: -- Description: branch-2.2 shows this tests frequent failure in the precommit jobs (808,809). Also master branch has this test on it's flakey list. Looks like bour TestSpaceQuotas is pretty overloaded because of all the sleeps(of 5 seconds) in the write-until-policy-violated attempts. Also to verify the violation, our numer of retries is 10 and each time we sleep 5 secs before checking again. So we could often cross the 13 mins limit and get timeout eventually leading to appearance of TestSpaceQuotas in the flakey list. An e.g. here: [https://builds.apache.org/view/H-L/view/HBase/job/HBase-adhoc-run-tests/27/] Let's refactor & break TestSpaceQuotas into more tests so that none of the tests cross the 13 min limit as we cannot make this into an IntegrationTest. was: branch-2.2 shows this tests frequent failure in the precommit jobs (808,809). Also master branch has this test on it's flakey list. Looks like our TestSpaceQuotas is pretty overloaded because of all the sleeps(of 5 seconds) in the write-until-policy-violated attempts. Also to verify the violation, our number of retries is 10 and each time we sleep 5 secs before checking again. So we could often cross the 13 mins limit and get timeout eventually leading to appearance of TestSpaceQuotas in the flakey list. An e.g. here: [https://builds.apache.org/view/H-L/view/HBase/job/HBase-adhoc-run-tests/27/] Let's refactor & break TestSpaceQuotas into more tests so that none of the tests cross the 13 min limit as we cannot make this into an IntegrationTest. > Fix the flakey TestSpaceQuotas > -- > > Key: HBASE-22895 > URL: https://issues.apache.org/jira/browse/HBASE-22895 > Project: HBase > Issue Type: Task > Components: Quotas, test >Affects Versions: 3.0.0 >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Labels: Quota, Space > > branch-2.2 shows this tests frequent failure in the precommit jobs (808,809). > Also master branch has this test on it's flakey list. Looks like bour > TestSpaceQuotas is pretty overloaded because of all the sleeps(of 5 seconds) > in the write-until-policy-violated attempts. Also to verify the violation, > our numer of retries is 10 and each time we sleep 5 secs before checking > again. So we could often cross the 13 mins limit and get timeout eventually > leading to appearance of TestSpaceQuotas in the flakey list. > An e.g. here: > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-adhoc-run-tests/27/] > > Let's refactor & break TestSpaceQuotas into more tests so that none of the > tests cross the 13 min limit as we cannot make this into an IntegrationTest. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920332#comment-16920332 ] Shardul Singh commented on HBASE-22013: --- Hi [~jatsakthi], done the changes and raised a PR. > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > HBASE-22013.master.003.patch, hbase-22013.branch-2.001.patch, > hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
[
https://issues.apache.org/jira/browse/HBASE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918363#comment-16918363
]
Shardul Singh commented on HBASE-22944:
---
Yes, agreed. chore interval should be very less.
> TableNotFoundException: hbase:quota is thrown when region server is
> restarted.
> ---
>
> Key: HBASE-22944
> URL: https://issues.apache.org/jira/browse/HBASE-22944
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> During Master startup if quota feature is enabled and region server is
> running TableNotFoundException occurs in regionServer logs
> SpaceQuotaRefresherChore does not checks whether hbase:quota table is
> present,before starting its operation which sometimes may result to
> TableNotFoundExceptions like below.
> This is because master has not created the hbase:quota table and
> SpaceQuotaRefresherChore is running.
> {code:java}
> java.io.UncheckedIOException: org.apache.hadoop.hbase.
> TableNotFoundException: hbase:quota
> at
> org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Status: Patch Available (was: In Progress) > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > HBASE-22013.master.003.patch, hbase-22013.branch-2.001.patch, > hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: HBASE-22013.master.003.patch > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > HBASE-22013.master.003.patch, hbase-22013.branch-2.001.patch, > hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915636#comment-16915636 ] Shardul Singh commented on HBASE-22013: --- Hi [~jatsakthi], I have added a new patch after the merging of HBASE-22895. Let the CI run. :) > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > HBASE-22013.master.003.patch, hbase-22013.branch-2.001.patch, > hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22013) SpaceQuotas - getNumRegions() returning wrong number of regions due to region replicas
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912048#comment-16912048 ] Shardul Singh commented on HBASE-22013: --- Hi [~jatsakthi], thanks for pushing and uploading the patches. Yes you can direct me to some few sample patches. I'll try to avoid these mistakes from next time. Thanks again > SpaceQuotas - getNumRegions() returning wrong number of regions due to region > replicas > -- > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Major > Labels: Quota, Space > Fix For: 2.1.0 > > Attachments: HBASE-22013.branch-2.1.001.patch, > HBASE-22013.master.001.patch, HBASE-22013.master.002.patch, > hbase-22013.branch-2.001.patch, hbase-22013.branch-2.2.001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918285#comment-16918285
]
Shardul Singh commented on HBASE-20821:
---
Yeah sure. no problem. Will raise a PR for each one. Thanks!
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.001.patch,
> HBASE-20821.master.002.patch, HBASE-20821.master.003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
Shardul Singh created HBASE-22944:
-
Summary: TableNotFoundException: hbase:quota is thrown when
region server is restarted.
Key: HBASE-22944
URL: https://issues.apache.org/jira/browse/HBASE-22944
Project: HBase
Issue Type: Bug
Components: Quotas
Reporter: Shardul Singh
Assignee: Shardul Singh
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is becuase master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
```
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at
org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
```
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
[
https://issues.apache.org/jira/browse/HBASE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22944:
--
Description:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
```
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
```
was:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is becuase master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
```
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at
org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
```
> TableNotFoundException: hbase:quota is thrown when region server is
> restarted.
> ---
>
> Key: HBASE-22944
> URL: https://issues.apache.org/jira/browse/HBASE-22944
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> SpaceQuotaRefresherChore does not checks whether hbase:quota table is
> present,before starting its operation which sometimes may result to
> TableNotFoundExceptions like below.
> This is because master has not created the hbase:quota table and
> SpaceQuotaRefresherChore is running.
> ```
> java.io.UncheckedIOException: org.apache.hadoop.hbase.
> {quote}TableNotFoundException: hbase:quota
> {quote}
> at
> org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
[jira] [Updated] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
[
https://issues.apache.org/jira/browse/HBASE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22944:
--
Description:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
{code:java}
java.io.UncheckedIOException: org.apache.hadoop.hbase.
TableNotFoundException: hbase:quota
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
{code:java}
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
> TableNotFoundException: hbase:quota is thrown when region server is
> restarted.
> ---
>
> Key: HBASE-22944
> URL: https://issues.apache.org/jira/browse/HBASE-22944
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> SpaceQuotaRefresherChore does not checks whether hbase:quota table is
> present,before starting its operation which sometimes may result to
> TableNotFoundExceptions like below.
> This is because master has not created the hbase:quota table and
> SpaceQuotaRefresherChore is running.
> {code:java}
> java.io.UncheckedIOException: org.apache.hadoop.hbase.
> TableNotFoundException: hbase:quota
> at
> org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
[jira] [Updated] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
[
https://issues.apache.org/jira/browse/HBASE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22944:
--
Description:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
{code:java}
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
```
java.io.UncheckedIOException: org.apache.hadoop.hbase.
{quote}TableNotFoundException: hbase:quota
{quote}
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
```
> TableNotFoundException: hbase:quota is thrown when region server is
> restarted.
> ---
>
> Key: HBASE-22944
> URL: https://issues.apache.org/jira/browse/HBASE-22944
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> SpaceQuotaRefresherChore does not checks whether hbase:quota table is
> present,before starting its operation which sometimes may result to
> TableNotFoundExceptions like below.
> This is because master has not created the hbase:quota table and
> SpaceQuotaRefresherChore is running.
> {code:java}
> java.io.UncheckedIOException: org.apache.hadoop.hbase.
> {quote}TableNotFoundException: hbase:quota
> {quote}
> at
> org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
[jira] [Updated] (HBASE-22944) TableNotFoundException: hbase:quota is thrown when region server is restarted.
[
https://issues.apache.org/jira/browse/HBASE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22944:
--
Description:
During Master startup if quota feature is enabled and region server is running
TableNotFoundException occurs in regionServer logs
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
{code:java}
java.io.UncheckedIOException: org.apache.hadoop.hbase.
TableNotFoundException: hbase:quota
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
SpaceQuotaRefresherChore does not checks whether hbase:quota table is
present,before starting its operation which sometimes may result to
TableNotFoundExceptions like below.
This is because master has not created the hbase:quota table and
SpaceQuotaRefresherChore is running.
{code:java}
java.io.UncheckedIOException: org.apache.hadoop.hbase.
TableNotFoundException: hbase:quota
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
at
org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{code}
> TableNotFoundException: hbase:quota is thrown when region server is
> restarted.
> ---
>
> Key: HBASE-22944
> URL: https://issues.apache.org/jira/browse/HBASE-22944
> Project: HBase
> Issue Type: Bug
> Components: Quotas
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> During Master startup if quota feature is enabled and region server is
> running TableNotFoundException occurs in regionServer logs
> SpaceQuotaRefresherChore does not checks whether hbase:quota table is
> present,before starting its operation which sometimes may result to
> TableNotFoundExceptions like below.
> This is because master has not created the hbase:quota table and
> SpaceQuotaRefresherChore is running.
> {code:java}
> java.io.UncheckedIOException: org.apache.hadoop.hbase.
> TableNotFoundException: hbase:quota
> at
> org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:84)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Component/s: mapreduce
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Environment: Instead of
> {code:java}
> if(i >= numPartitions-1)
> {code}
> It should be
> {code:java}
> if(i >= numPartitions)
> {code}
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92
> Consider the following scenario:
> 1. if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true. But for the last region
> when i=4, getPartition should return 4 but it returns 2 because it falls in
> the case of when we have less reduces than region and returns true for the
> above condition.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Description:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true.
# But for the last region when i=4, getPartition should return 4 but it
returns 2 because it falls in the case of when we have less reduces than region
and returns true for the above condition.
{code:java}
{code}
was:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true. But for the last region
when i=4, getPartition should return 4 but it returns 2 because it falls in the
case of when we have less reduces than region and returns true for the above
condition.
{code:java}
{code}
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> # if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true.
> # But for the last region when i=4, getPartition should return 4 but it
> returns 2 because it falls in the case of when we have less reduces than
> region and returns true for the above condition.
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Description:
*Problem:*
Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
getPartition(). In getPartition(), there is a scenario where we have check for
less number of reducers than region. This scenario seems incorrect because for
a rowKey present in last region(let us say nth region) , getPartition() should
return value (n-1). But it is not returning n-1 for the last region as it is
falling in the case where number of reducers < number of regions and returning
some random value.
So if a client uses this class as a partitioner class in HBase MapReduce jobs,
this method incorrectly partitions the regions because rowKeys present in the
last regions does not fall to the last region.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
if there are 5 regions for the table, partitions = 5 and number of reducers is
also 5.
So in this case above check for reducers < regions should not return true.
But for the last region when i=4(last region, 5th region) , getPartition should
return 4 but it returns 2 because it falls in the case of when we have less
reduces than region and returns true for the above condition even though we
have reducers = regions. So the condition is incorrect.
*Solution:*
Instead of
{code} if (i >= numPartitions-1) {code}
It should be
{code} if (i >= numPartitions){ {code}
was:
Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
getPartition(). In getPartition(), there is a scenario where we have check for
less number of reducers than region. This scenario seems incorrect because for
a rowKey present in last region(let us say nth region) , getPartition() should
return value (n-1). But it is not returning n-1 for the last region as it is
falling in the case where number of reducers < number of regions and returning
some random value.
So if a client uses this class as a partitioner class in HBase MapReduce jobs,
this method incorrectly partitions the regions because rowKeys present in the
last regions does not fall to the last region.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
if there are 5 regions for the table, partitions = 5 and number of reducers is
also 5.
So in this case above check for reducers < regions should not return true.
But for the last region when i=4(last region, 5th region) , getPartition should
return 4 but it returns 2 because it falls in the case of when we have less
reduces than region and returns true for the above condition even though we
have reducers = regions. So the condition is incorrect.
Solution:
Instead of
{code} if (i >= numPartitions-1) {code}
It should be
{code} if (i >= numPartitions){ {code}
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Major
>
> *Problem:*
> Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
> getPartition(). In getPartition(), there is a scenario where we have check
> for less number of reducers than region. This scenario seems incorrect
> because for a rowKey present in last region(let us say nth region) ,
> getPartition() should return value (n-1). But it is not returning n-1 for the
> last region as it is falling in the case where number of reducers < number of
> regions and returning some random value.
> So if a client uses this class as a partitioner class in HBase MapReduce
> jobs, this method incorrectly partitions the regions because rowKeys present
> in the last regions does not fall to the last region.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> if there are 5 regions for the table, partitions = 5 and number of reducers
> is also 5.
> So in this case above check for reducers < regions should not return true.
> But for the last region when i=4(last region, 5th region) , getPartition
> should return 4 but it returns 2 because it falls in the case of when we have
> less
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Description:
Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
getPartition(). In getPartition(), there is a scenario where we have check for
less number of reducers than region. This scenario seems incorrect because for
a rowKey present in last region(let us say nth region) , getPartition() should
return value (n-1). But it is not returning n-1 for the last region as it is
falling in the case where number of reducers < number of regions and returning
some random value.
So if a client uses this class as a partitioner class in HBase MapReduce jobs,
this method incorrectly partitions the regions because rowKeys present in the
last regions does not fall to the last region.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
if there are 5 regions for the table, partitions = 5 and number of reducers is
also 5.
So in this case above check for reducers < regions should not return true.
But for the last region when i=4(last region, 5th region) , getPartition should
return 4 but it returns 2 because it falls in the case of when we have less
reduces than region and returns true for the above condition even though we
have reducers = regions. So the condition is incorrect.
Solution:
Instead of
{code} if (i >= numPartitions-1) {code}
It should be
{code} if (i >= numPartitions){ {code}
was:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true.
# But for the last region when i=4, getPartition should return 4 but it
returns 2 because it falls in the case of when we have less reduces than region
and returns true for the above condition.
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
> getPartition(). In getPartition(), there is a scenario where we have check
> for less number of reducers than region. This scenario seems incorrect
> because for a rowKey present in last region(let us say nth region) ,
> getPartition() should return value (n-1). But it is not returning n-1 for the
> last region as it is falling in the case where number of reducers < number of
> regions and returning some random value.
> So if a client uses this class as a partitioner class in HBase MapReduce
> jobs, this method incorrectly partitions the regions because rowKeys present
> in the last regions does not fall to the last region.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> if there are 5 regions for the table, partitions = 5 and number of reducers
> is also 5.
> So in this case above check for reducers < regions should not return true.
> But for the last region when i=4(last region, 5th region) , getPartition
> should return 4 but it returns 2 because it falls in the case of when we have
> less reduces than region and returns true for the above condition even though
> we have reducers = regions. So the condition is incorrect.
>
> Solution:
> Instead of
> {code} if (i >= numPartitions-1) {code}
> It should be
>{code} if (i >= numPartitions){ {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Priority: Major (was: Minor)
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Major
>
> Partitioner class HRegionPartitioner in a HBase MapReduce job has a method
> getPartition(). In getPartition(), there is a scenario where we have check
> for less number of reducers than region. This scenario seems incorrect
> because for a rowKey present in last region(let us say nth region) ,
> getPartition() should return value (n-1). But it is not returning n-1 for the
> last region as it is falling in the case where number of reducers < number of
> regions and returning some random value.
> So if a client uses this class as a partitioner class in HBase MapReduce
> jobs, this method incorrectly partitions the regions because rowKeys present
> in the last regions does not fall to the last region.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> if there are 5 regions for the table, partitions = 5 and number of reducers
> is also 5.
> So in this case above check for reducers < regions should not return true.
> But for the last region when i=4(last region, 5th region) , getPartition
> should return 4 but it returns 2 because it falls in the case of when we have
> less reduces than region and returns true for the above condition even though
> we have reducers = regions. So the condition is incorrect.
>
> Solution:
> Instead of
> {code} if (i >= numPartitions-1) {code}
> It should be
>{code} if (i >= numPartitions){ {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924094#comment-16924094
]
Shardul Singh commented on HBASE-22980:
---
Solution:
Instead of
{code}
if (i >= numPartitions-1){
{code}
It should be
{code}
if (i >= numPartitions){
{code}
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> # if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true. But for the last region
> when i=4, getPartition should return 4 but it returns 2 because it falls in
> the case of when we have less reduces than region and returns true for the
> above condition.
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Description:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true.
# But for the last region when i=4, getPartition should return 4 but it
returns 2 because it falls in the case of when we have less reduces than region
and returns true for the above condition.
was:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true.
# But for the last region when i=4, getPartition should return 4 but it
returns 2 because it falls in the case of when we have less reduces than region
and returns true for the above condition.
{code:java}
{code}
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> # if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true.
> # But for the last region when i=4, getPartition should return 4 but it
> returns 2 because it falls in the case of when we have less reduces than
> region and returns true for the above condition.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
Shardul Singh created HBASE-22980:
-
Summary: HRegionPartioner getPartition() method incorrectly
partitions the regions of the table.
Key: HBASE-22980
URL: https://issues.apache.org/jira/browse/HBASE-22980
Project: HBase
Issue Type: Bug
Environment: Instead of
{code:java}
if(i >= numPartitions-1)
{code}
It should be
{code:java}
if(i >= numPartitions)
{code}
Reporter: Shardul Singh
Assignee: Shardul Singh
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92
Consider the following scenario:
1. if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true. But for the last region when
i=4, getPartition should return 4 but it returns 2 because it falls in the case
of when we have less reduces than region and returns true for the above
condition.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Environment: (was: Instead of
{code:java}
if(i >= numPartitions-1)
{code}
It should be
{code:java}
if(i >= numPartitions)
{code})
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92
> Consider the following scenario:
> 1. if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true. But for the last region
> when i=4, getPartition should return 4 but it returns 2 because it falls in
> the case of when we have less reduces than region and returns true for the
> above condition.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22980) HRegionPartioner getPartition() method incorrectly partitions the regions of the table.
[
https://issues.apache.org/jira/browse/HBASE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-22980:
--
Description:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
[https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
Consider the following scenario:
# if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true. But for the last region
when i=4, getPartition should return 4 but it returns 2 because it falls in the
case of when we have less reduces than region and returns true for the above
condition.
{code:java}
{code}
was:
In HRegionPartitioner getPartition(), the case where we have less reduces than
region is incorrect.
https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92
Consider the following scenario:
1. if there are 5 regions for the table, partitions = 5 and number of reducers
is also 5.
So in this case above code should not return true. But for the last region when
i=4, getPartition should return 4 but it returns 2 because it falls in the case
of when we have less reduces than region and returns true for the above
condition.
> HRegionPartioner getPartition() method incorrectly partitions the regions of
> the table.
> ---
>
> Key: HBASE-22980
> URL: https://issues.apache.org/jira/browse/HBASE-22980
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Shardul Singh
>Assignee: Shardul Singh
>Priority: Minor
>
> In HRegionPartitioner getPartition(), the case where we have less reduces
> than region is incorrect.
> [https://github.com/apache/hbase/blob/fbd5b5e32753104f88600b0f4c803ab5659bce64/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java#L92]
> Consider the following scenario:
> # if there are 5 regions for the table, partitions = 5 and number of
> reducers is also 5.
> So in this case above code should not return true. But for the last region
> when i=4, getPartition should return 4 but it returns 2 because it falls in
> the case of when we have less reduces than region and returns true for the
> above condition.
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897803#comment-16897803
]
Shardul Singh commented on HBASE-20821:
---
hi [~elserj]..can you please check this patch..been a long time i
submitted..thanks :)
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903590#comment-16903590 ] Shardul Singh commented on HBASE-22013: --- Hi, Since this issue is opened for a long time. Can I work on this jira issue if you are not working on it? [~Uma Murugesan] > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Uma Maheswari >Priority: Major > Labels: Quota, Space > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22012) SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable table
[ https://issues.apache.org/jira/browse/HBASE-22012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903777#comment-16903777 ] Shardul Singh commented on HBASE-22012: --- Hi [~nihaljain.cs], I have a patch for this issue..if it's not a problem should I go ahead and assign to myself? :) > SpaceQuota DisableTableViolationPolicy will cause cycles of enable/disable > table > > > Key: HBASE-22012 > URL: https://issues.apache.org/jira/browse/HBASE-22012 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.1 >Reporter: Ajeet Rai >Assignee: Nihal Jain >Priority: Major > Labels: Quota, Space > > Space Quota: Policy state is getting changed from disable to Observance after > sometime automatically. > Steps: > 1: Create a table with space quota policy as Disable > 2: Put some data so that table state is in space quota violation > 3: So observe that table state is in violation > 4: Now wait for some time > 5: Observe that after some time table state is changing to to Observance > however table is still disabled > edit (elserj): The table is automatically moved back from the violation state > because of the code added that tried to ride over RITs. When a Region is not > online (whether normally or abnormally), the RegionSizeReports are not sent > from RS to Master. Eventually, enough Regions are not reported which dips > below the acceptable threshold and we automatically move the table back to > the "acceptable" space quota state (not in violation). We could skip this > failsafe when we're checking for a quota that has the DisableTable violation > policy. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903770#comment-16903770 ] Shardul Singh commented on HBASE-22013: --- Submitting the initial patch. Please review. > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Uma Maheswari >Priority: Major > Labels: Quota, Space > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22013) SpaceQuota utilization issue with region replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-22013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh updated HBASE-22013: -- Attachment: HBASE-22013.master.v001.patch > SpaceQuota utilization issue with region replicas enabled > - > > Key: HBASE-22013 > URL: https://issues.apache.org/jira/browse/HBASE-22013 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Uma Maheswari >Priority: Major > Labels: Quota, Space > Attachments: HBASE-22013.master.v001.patch > > > Space Quota: Space Quota Issue: If a table is created with region replica > then quota calculation is not happening > Steps: > 1: Create a table with 100 regions with region replica 3 > 2: Observe that 'hbase:quota' table doesn't have entry of usage for this > table So In UI only policy Limit and Policy is shown but not Usage and State. > Reason: > It looks like File system utilization core is sending data of 100 reasons > but not the size of region replicas. > But in quota observer chore, it is considering total region(actual regions+ > replica reasons) > So the ratio of reported regions is less then configured > percentRegionsReportedThreshold. > SO quota calculation is not happening -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HBASE-21891) New space quota policy doesn't take effect if quota policy is changed after violation
[ https://issues.apache.org/jira/browse/HBASE-21891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-21891: - Assignee: Shardul Singh > New space quota policy doesn't take effect if quota policy is changed after > violation > - > > Key: HBASE-21891 > URL: https://issues.apache.org/jira/browse/HBASE-21891 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Minor > Labels: Quota, space > > *Steps to reproduce* > 1: set_quota TYPE => SPACE, TABLE => 'test25', LIMIT => '2M', POLICY => > NO_WRITES > 2: ./hbase pe --table="test25" --nomapred --rows=300 sequentialWrite 10 > 3: Observe that after some time data usage is 3 mb and policy is in violation > 4: now try to insert some data again in the table and observe that operation > fails due to NoWritesViolationPolicyEnforcement > 5: Now change the quota policy > set_quota TYPE => SPACE, TABLE => 'test25', LIMIT => '2M', POLICY => > NO_WRITES_COMPACTIONS > 6: Now again try to insert data once new policy takes effect > 7: Observe that still operation fails but because of old policy and not new > policy. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HBASE-21891) New space quota policy doesn't take effect if quota policy is changed after violation
[ https://issues.apache.org/jira/browse/HBASE-21891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-21891: - Assignee: (was: Shardul Singh) > New space quota policy doesn't take effect if quota policy is changed after > violation > - > > Key: HBASE-21891 > URL: https://issues.apache.org/jira/browse/HBASE-21891 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Priority: Minor > Labels: Quota, space > > *Steps to reproduce* > 1: set_quota TYPE => SPACE, TABLE => 'test25', LIMIT => '2M', POLICY => > NO_WRITES > 2: ./hbase pe --table="test25" --nomapred --rows=300 sequentialWrite 10 > 3: Observe that after some time data usage is 3 mb and policy is in violation > 4: now try to insert some data again in the table and observe that operation > fails due to NoWritesViolationPolicyEnforcement > 5: Now change the quota policy > set_quota TYPE => SPACE, TABLE => 'test25', LIMIT => '2M', POLICY => > NO_WRITES_COMPACTIONS > 6: Now again try to insert data once new policy takes effect > 7: Observe that still operation fails but because of old policy and not new > policy. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-20821) Re-creating a dropped namespace and contained table inherits previously set space quota settings
[
https://issues.apache.org/jira/browse/HBASE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shardul Singh updated HBASE-20821:
--
Labels: Quota Space (was: )
> Re-creating a dropped namespace and contained table inherits previously set
> space quota settings
>
>
> Key: HBASE-20821
> URL: https://issues.apache.org/jira/browse/HBASE-20821
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Shardul Singh
>Priority: Major
> Labels: Quota, Space
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20821.master.v001patch,
> HBASE-20821.master.v002.patch, HBASE-20821.master.v003.patch
>
>
> As demonstarted in
> [HBASE-20662.master.002.patch|https://issues.apache.org/jira/secure/attachment/12927187/HBASE-20662.master.002.patch]
> re-creating a dropped namespace and contained table inherits previously set
> space quota settings.
> *Steps:*
> * Create a namespace and a table in it
> * Set space quota on namespace
> * Violate namespace quota
> * Drop table and then namespace
> * Re create same namespace and same table
> * Put data into the table (more than the previosuly set namespace quota
> limit)
> {code:java}
> private void setQuotaAndThenDropNamespace(final String namespace,
> SpaceViolationPolicy policy)
> throws Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> // Do puts until we violate space policy
> final TableName tn =
> writeUntilNSSpaceViolationAndVerifyViolation(namespace, policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now, drop the namespace
> TEST_UTIL.getAdmin().deleteNamespace(namespace);
> LOG.debug("Successfully deleted the namespace ", namespace);
> // Now re-create the namespace
> createNamespaceIfNotExist(TEST_UTIL.getAdmin(), namespace);
> LOG.debug("Successfully re-created the namespace ", namespace);
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as namespace/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*: SpaceQuota settings should not exist on the newly re-created
> table and we should be able to put limit less data into the table
> *Actual:* We fail to put data into newly created table as SpaceQuota settings
> (systematically created due to previously added namespace space quota) exist
> on table
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22342) space quota usage is calculated almost double then actual usage in case of Split /Merge of region
[ https://issues.apache.org/jira/browse/HBASE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938275#comment-16938275 ] Shardul Singh edited comment on HBASE-22342 at 9/26/19 5:13 AM: Hi [~elserj] and [~jatsakthi], I need your expertise in this one.Problem is quotaUsage is shows double for 10 minutes because of the conf "hbase.master.quotas.region.report.retention.millis" default value. So the solution which I came up is as follows: 1. I was thinking that we should not remove the region report if it is split/merge from regionSize cache as we did in HBASE-22012. 2. Catalog Janitor can take up the responisbility of removing the region report from the cache while deleting references for the merge/split region. 3. During master startup we can add a logic for Calculating the size of region if it's a splitParent or mergeParent and put it in the cache if the region state still exists in the cluster. Please give your opinions, Thanks :) was (Author: shardulsingh): Hi [~elserj] and [~jatsakthi], I need your expertise in this one.Problem is quotaUsage is shows double for 10 minutes because of the conf "hbase.master.quotas.region.report.retention.millis" default value. So the solution which I came up is as follows: 1. I was thinking that we should not remove the region report if it is split/merge from regionSize cache as we did in HBASE-22012. 2. Catalog Janitor can take up the responisbility of removing the region report from the cache while deleting references for the merge/split region. 3. During master startup we can add a logic for Calculating the size of region if it's a splitParent or mergeParent and put it in the cache if the region state still exists in the cluster. > space quota usage is calculated almost double then actual usage in case of > Split /Merge of region > - > > Key: HBASE-22342 > URL: https://issues.apache.org/jira/browse/HBASE-22342 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Minor > Labels: quota, space > > space quota usage is calculated double for some time because it is > considering the usage of the parent region report received earlier. This is > normal behaviour because *hbase.master.quotas.region.report.retention.millis* > is set as 10 minutes.It's not calculating the usage from the store files, > it's taking the usage from the earlier report send to the master by the > parent region server till 10 minutes. > So after 10 minutes, it stops considering the earlier region report and quota > is not violated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22342) space quota usage is calculated almost double then actual usage in case of Split /Merge of region
[ https://issues.apache.org/jira/browse/HBASE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938275#comment-16938275 ] Shardul Singh commented on HBASE-22342: --- Hi [~elserj] and [~jatsakthi], I need your expertise in this one.Problem is quotaUsage is shows double for 10 minutes because of the conf "hbase.master.quotas.region.report.retention.millis" default value. So the solution which I came up is as follows: 1. I was thinking that we should not remove the region report if it is split/merge from regionSize cache as we did in HBASE-22012. 2. Catalog Janitor can take up the responisbility of removing the region report from the cache while deleting references for the merge/split region. 3. During master startup we can add a logic for Calculating the size of region if it's a splitParent or mergeParent and put it in the cache if the region state still exists in the cluster. > space quota usage is calculated almost double then actual usage in case of > Split /Merge of region > - > > Key: HBASE-22342 > URL: https://issues.apache.org/jira/browse/HBASE-22342 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Ajeet Rai >Assignee: Shardul Singh >Priority: Minor > Labels: quota, space > > space quota usage is calculated double for some time because it is > considering the usage of the parent region report received earlier. This is > normal behaviour because *hbase.master.quotas.region.report.retention.millis* > is set as 10 minutes.It's not calculating the usage from the store files, > it's taking the usage from the earlier report send to the master by the > parent region server till 10 minutes. > So after 10 minutes, it stops considering the earlier region report and quota > is not violated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22146) SpaceQuotaViolationPolicy Disable is not working in Namespace level
[ https://issues.apache.org/jira/browse/HBASE-22146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935868#comment-16935868 ] Shardul Singh commented on HBASE-22146: --- Hi [~nihaljain.cs], If you are not working on this issue..can I assign this to myself? :) > SpaceQuotaViolationPolicy Disable is not working in Namespace level > --- > > Key: HBASE-22146 > URL: https://issues.apache.org/jira/browse/HBASE-22146 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.0 >Reporter: Uma Maheswari >Assignee: Nihal Jain >Priority: Major > Labels: Quota, space > Fix For: 3.0.0, 2.1.7, 2.2.2, 2.0.7 > > > SpaceQuotaViolationPolicy Disable is not working in Namespace level > PFB the steps: > * Create Namespace and set Quota violation policy as Disable > * Create tables under namespace and violate Quota > Expected result: Tables to get disabled > Actual Result: Tables are not getting disabled > Note: mutation operation is not allowed on the table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22166) Space Quota issue: Namespace Quota policy is not getting imposed on the table in NS,when table level quota is set on the table and is violated and removed
[ https://issues.apache.org/jira/browse/HBASE-22166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935860#comment-16935860 ] Shardul Singh commented on HBASE-22166: --- Assigning this to me. > Space Quota issue: Namespace Quota policy is not getting imposed on the table > in NS,when table level quota is set on the table and is violated and removed > -- > > Key: HBASE-22166 > URL: https://issues.apache.org/jira/browse/HBASE-22166 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.0 >Reporter: Uma Maheswari >Priority: Minor > Labels: Quota, space > > Space Quota issue: Namespace Quota policy is not getting imposed on the table > in NS,when table level quota is set on the table and is violated and removed > PFB the steps. > * create a NS and Set Quota > * create table under the NS and set Quota and write data to violate it > * Write data in NS and violate it > * Now remove the Quota in table level > Expected result: > * table should be moved to violation policy at NS level > Actual Result: > * NS level violation is shown in UI for the table > * in hbase:quota table no enry is present for that table > * violation policy is not imposed on the table > * insertion is allowed on the table which quota is removed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-22166) Space Quota issue: Namespace Quota policy is not getting imposed on the table in NS,when table level quota is set on the table and is violated and removed
[ https://issues.apache.org/jira/browse/HBASE-22166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Singh reassigned HBASE-22166: - Assignee: Shardul Singh > Space Quota issue: Namespace Quota policy is not getting imposed on the table > in NS,when table level quota is set on the table and is violated and removed > -- > > Key: HBASE-22166 > URL: https://issues.apache.org/jira/browse/HBASE-22166 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.0 >Reporter: Uma Maheswari >Assignee: Shardul Singh >Priority: Minor > Labels: Quota, space > > Space Quota issue: Namespace Quota policy is not getting imposed on the table > in NS,when table level quota is set on the table and is violated and removed > PFB the steps. > * create a NS and Set Quota > * create table under the NS and set Quota and write data to violate it > * Write data in NS and violate it > * Now remove the Quota in table level > Expected result: > * table should be moved to violation policy at NS level > Actual Result: > * NS level violation is shown in UI for the table > * in hbase:quota table no enry is present for that table > * violation policy is not imposed on the table > * insertion is allowed on the table which quota is removed -- This message was sent by Atlassian Jira (v8.3.4#803005)
