[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-13938:
--
Release Note: Use the master's timestamp when sending hbase:meta edits on
region merge to ensure proper ordering of new region addition and old region
deletes.
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Fix Version/s: 2.0.0
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to 0.98+. Thanks everyone!
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Fix Version/s: 1.0.2
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Attachment: hbase-13938_v3-0.98.patch
Updated 0.98 backport, adds test and fixes the other issues originally missed.
Particularly worrisome are these lines, but I guess it's what [~devaraj] and
[~enis] had in mind:
{noformat}
private static Put addLocation(final Put p, final ServerName sn, long
openSeqNum) {
-// using regionserver's local time as the timestamp of Put.
-// See: HBASE-11536
-long now = EnvironmentEdgeManager.currentTimeMillis();
-p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER, now,
+p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.SERVER_QUALIFIER,
Bytes.toBytes(sn.getHostAndPort()));
-p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER,
now,
+p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.STARTCODE_QUALIFIER,
Bytes.toBytes(sn.getStartcode()));
-p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.SEQNUM_QUALIFIER, now,
-Bytes.toBytes(openSeqNum));
+p.addImmutable(HConstants.CATALOG_FAMILY, HConstants.SEQNUM_QUALIFIER,
+ Bytes.toBytes(openSeqNum));
return p;
}
{noformat}
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Attachment: hbase-13938_master.patch
Here is the master patch. It should not have any behavioral change, since
MTA.mergeRegions() is only called by master. The patch is needed for forward
compatibility for PB. The rest of the changes are good for patch portability.
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch,
> hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch,
> hbase-13938_v3-branch-1.0.patch, hbase-13938_v3-branch-1.1.patch,
> hbase-13938_v3-branch-1.2.patch, hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Fix Version/s: (was: 2.0.0)
0.98.14
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Attachment: hbase-13938_v3-branch-1.patch
hbase-13938_v3-branch-1.2.patch
hbase-13938_v3-branch-1.0.patch
hbase-13938_v3-0.98.patch
backport patches. FYI [~enis], [~apurtell], [~busbey]
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
> hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
> hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
> hbase-13938_v3-branch-1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Fix Version/s: 1.3.0
1.2.0
2.0.0
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
> hbase-13938_v3-branch-1.1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Attachment: hbase-13938_v3-branch-1.1.patch
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Enis Soztutar
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
> hbase-13938_v3-branch-1.1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Attachment: hbase-13938_v2-branch-1.1.patch
v2 patch uses {{max}} and adds a simple unit test. cc [~ndimiduk].
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Status: Patch Available (was: Open)
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: 13938-branch-1.1.txt
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: (was: 13938-branch-1.1.txt)
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: (was: hbase-13895_v1-branch-1.1.patch)
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: 13938-branch-1.1.txt
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: 13938-branch-1.1.txt
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: hbase-13895_v1-branch-1.1.patch
Quick patch for 1.1
> Deletes done during the region merge transaction may get eclipsed
> -
>
> Key: HBASE-13938
> URL: https://issues.apache.org/jira/browse/HBASE-13938
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Devaraj Das
> Fix For: 1.1.1
>
> Attachments: hbase-13895_v1-branch-1.1.patch
>
>
> Was looking at an issue from our internal testing. It seems the Deletes of
> the region rows from the meta done during the merge transaction could be
> eclipsed by the Put of a region row that might have happened moments before.
> The master logs this for the merge:
> {noformat}
> 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
>
> region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
>
> region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
> on ddas-2-5.openstacklocal,16020,1434632778438
> {noformat}
> One of the regions that got merged got Opened a few seconds back:
> {noformat}
> 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
> regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
> sequenceid=182988
> {noformat}
> The above would have done a Put in the meta.
> Looking at the raw scan of the meta, for the new merged region, the creation
> timestamp is 1434633226101:
> {noformat}
>
> IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
> column=info:regioninfo, timestamp=1434633226101, value={ENCODED =>
> 0927319db6bf5e128e3bec2a420819aa, NAME =>
> 'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
> STARTKEY => 'a65c', ENDKEY => 'b328'}
> {noformat}
> Looking at the raw scan of the meta, the timestamp for the region open of the
> already merged region is 1434633226600. This is a little after the merge
> transaction's timestamp.
> {noformat}
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:seqnumDuringOpen, timestamp=1434633226600,
> value=\x00\x00\x00\x00\x00\x02\xCA\xCC
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:server, timestamp=1434633226600,
> value=ddas-2-5.openstacklocal:16020
>
> IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
> column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
> {noformat}
> We need to fix it so that the merge region transaction also takes the
> master's timestamp. Similar to HBASE-13875.
> When this happens, clients start to see a row in the meta with an empty
> HRegionInfo (this is because the Put done during the region open only updates
> the location information but not the HRI, and the HRI deleted during the
> merge transaction "remains deleted").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
