[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278456#comment-17278456
]
Guanghao Zhang commented on HBASE-23044:
Sorry to hear that..
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278417#comment-17278417
]
Karthik Palanisamy commented on HBASE-23044:
Yes [~rsanwal] . Sometimes silent data-loss. Two users reported this issue
recently but we somehow recovered data from archive.
We requested users to disable NORMALIZATION or any aggressive manual region
merge.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276692#comment-17276692
]
Ravi Sanwal commented on HBASE-23044:
-
This bugger caused us so much pain.
We lost a few regions because of this when we tried to manually merge a few
regions.
Catalog janitor deleted the merge qualifiers and then the references to parent
regions got cleared and then the store files got archived and deleted.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933531#comment-16933531
]
stack commented on HBASE-23044:
---
[~zghao] How did you figure this one? It looks painful to figure. Nice fix.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933520#comment-16933520
]
stack commented on HBASE-23044:
---
bq. Yes, may cause data loss.. Introduced by HBASE-22777, only the latest
release was effected.
I should roll new releases?
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933147#comment-16933147
]
Hudson commented on HBASE-23044:
Results for branch branch-2
[build #2276 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2276/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2276//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2276//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2276//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2276//console].
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933069#comment-16933069
]
Hudson commented on HBASE-23044:
Results for branch branch-2.1
[build #1614 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1614/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1614//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1614//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1614//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1614//console].
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933063#comment-16933063
]
Hudson commented on HBASE-23044:
Results for branch branch-2.2
[build #620 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/620/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/620//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/620//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/620//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/620//console].
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933061#comment-16933061
]
Hudson commented on HBASE-23044:
Results for branch master
[build #1456 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1456/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1456//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1456//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1456//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/master/1456//console].
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
>
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932947#comment-16932947
]
Guanghao Zhang commented on HBASE-23044:
Pushed to branch-2.1+. Thanks [~Apache9] for reviewing.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932520#comment-16932520
]
Duo Zhang commented on HBASE-23044:
---
So maybe we should roll new releases then... And also remove the download link
of 2.0.6 on our download page.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.6, 2.2.1, 2.1.6
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932480#comment-16932480
]
Guanghao Zhang commented on HBASE-23044:
Yes, may cause data loss.. Introduced by HBASE-22777, only the latest
release was effected.
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23044) CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
[
https://issues.apache.org/jira/browse/HBASE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932416#comment-16932416
]
Duo Zhang commented on HBASE-23044:
---
All 2.x releases are effect? This is a critical problem...
> CatalogJanitor#cleanMergeQualifier may clean wrong parent regions
> -
>
> Key: HBASE-23044
> URL: https://issues.apache.org/jira/browse/HBASE-23044
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
>
> 2019-09-17,19:42:40,539 INFO [PEWorker-1]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223589,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[81b6fc3c560a00692bc7c3cd266a626a],
> [472500358997b0dc8f0002ec86593dcf] in 2.6470sec
> 2019-09-17,19:59:54,179 INFO [PEWorker-6]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=1223651,
> state=SUCCESS; GCMultipleMergedRegionsProcedure
> child={color:red}647600d28633bb2fe06b40682bab0593{color},
> parents:[9c52f24e0a9cc9b4959c1ebdfea29d64],
> [a623f298870df5581bcfae7f83311b33] in 1.0340sec
> The child is same region {color:red}647600d28633bb2fe06b40682bab0593{color}
> but the parent regions are different.
> MergeTableRegionProcedure#prepareMergeRegion will try to cleanMergeQualifier
> for the regions to merge.
> {code:java}
> for (RegionInfo ri: this.regionsToMerge) {
> if (!catalogJanitor.cleanMergeQualifier(ri)) {
> String msg = "Skip merging " +
> RegionInfo.getShortNameToLog(regionsToMerge) +
> ", because parent " + RegionInfo.getShortNameToLog(ri) + " has a
> merge qualifier";
> LOG.warn(msg);
> throw new MergeRegionException(msg);
> }
> {code}
> If region A and B merge to C, region D and E merge to F. When merge C and F,
> it will try to cleanMergeQualifier for C and F.
> catalogJanitor.cleanMergeQualifier for region C succeed but
> catalogJanitor.cleanMergeQualifier for region F failed as there are
> references in region F.
> When merge C and F again, it will try to cleanMergeQualifier for C and F
> again. But MetaTableAccessor.getMergeRegions will get wrong parents now. It
> use scan with filter to scan result. But region C's MergeQualifier already
> was deleted before. Then the scan will return a wrong result, may be anther
> region..
> {code:java}
> public boolean cleanMergeQualifier(final RegionInfo region) throws
> IOException {
> // Get merge regions if it is a merged region and already has merge
> qualifier
> List parents =
> MetaTableAccessor.getMergeRegions(this.services.getConnection(),
> region.getRegionName());
> if (parents == null || parents.isEmpty()) {
> // It doesn't have merge qualifier, no need to clean
> return true;
> }
> return cleanMergeRegion(region, parents);
> }
> public static List getMergeRegions(Connection connection, byte[]
> regionName)
> throws IOException {
> return getMergeRegions(getMergeRegionsRaw(connection, regionName));
> }
> private static Cell [] getMergeRegionsRaw(Connection connection, byte []
> regionName)
> throws IOException {
> Scan scan = new Scan().withStartRow(regionName).
> setOneRowLimit().
> readVersions(1).
> addFamily(HConstants.CATALOG_FAMILY).
> setFilter(new QualifierFilter(CompareOperator.EQUAL,
> new RegexStringComparator(HConstants.MERGE_QUALIFIER_PREFIX_STR+
> ".*")));
> try (Table m = getMetaHTable(connection); ResultScanner scanner =
> m.getScanner(scan)) {
> // Should be only one result in this scanner if any.
> Result result = scanner.next();
> if (result == null) {
> return null;
> }
> // Should be safe to just return all Cells found since we had filter in
> place.
> // All values should be RegionInfos or something wrong.
> return result.rawCells();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
