[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, 5689-v4.txt, HBASE-5689.patch,
HBASE-5689.patch, HBASE-5689v2.patch, HBASE-5689v3.patch, HBASE-5689v3.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-5689:
Attachment: HBASE-5689v2.patch
Sorry for the later response.
In the patchV2, I add some doc for getCompletedRecoveredEditsFilePath() as the
comment by Stack. All the other is same with patchV1.
Please review again.
Thanks!
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, HBASE-5689.patch, HBASE-5689.patch,
HBASE-5689v2.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-5689:
Attachment: HBASE-5689v3.patch
In the patchV3, append the optimize suggested by Ming Ma
：replayRecoveredEditsIfAny simply walk through all the recoverededit files and
skip those files that have max sequence id = current sequence id of the
region.
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, HBASE-5689.patch, HBASE-5689.patch,
HBASE-5689v2.patch, HBASE-5689v3.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Attachment: 5689-v4.txt
Patch v4 removes Math.abs() call.
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, 5689-v4.txt, HBASE-5689.patch,
HBASE-5689.patch, HBASE-5689v2.patch, HBASE-5689v3.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5689:
-
Attachment: HBASE-5689.patch
Reattaching for test run.
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, HBASE-5689.patch, HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5689:
-
Status: Patch Available (was: Open)
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, HBASE-5689.patch, HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Attachment: (was: HBASE-5689.patch)
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-simplified.txt, 5689-testcase.patch,
HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Attachment: (was: 5689-simplified.txt)
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-testcase.patch, HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Summary: Skipping RecoveredEdits may cause data loss (was: Skip
RecoveredEdits may cause data loss)
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: 5689-testcase.patch, HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5689:
--
Priority: Critical (was: Major)
Making it critical as it is data loss related.
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Attachments: 5689-testcase.patch, HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5689:
--
Attachment: 5689-simplified.txt
Removing the check should be enough.
With the attached patch, TestHRegion#testDataCorrectnessReplayingRecoveredEdits
passes.
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Attachments: 5689-simplified.txt, 5689-testcase.patch,
HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5689) Skipping RecoveredEdits may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5689:
-
Fix Version/s: 0.94.0
Skipping RecoveredEdits may cause data loss
---
Key: HBASE-5689
URL: https://issues.apache.org/jira/browse/HBASE-5689
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.94.0
Attachments: 5689-simplified.txt, 5689-testcase.patch,
HBASE-5689.patch
Let's see the following scenario:
1.Region is on the server A
2.put KV(r1-v1) to the region
3.move region from server A to server B
4.put KV(r2-v2) to the region
5.move region from server B to server A
6.put KV(r3-v3) to the region
7.kill -9 server B and start it
8.kill -9 server A and start it
9.scan the region, we could only get two KV(r1-v1,r2-v2), the third
KV(r3-v3) is lost.
Let's analyse the upper scenario from the code:
1.the edit logs of KV(r1-v1) and KV(r3-v3) are both recorded in the same
hlog file on server A.
2.when we split server B's hlog file in the process of ServerShutdownHandler,
we create one RecoveredEdits file f1 for the region.
2.when we split server A's hlog file in the process of ServerShutdownHandler,
we create another RecoveredEdits file f2 for the region.
3.however, RecoveredEdits file f2 will be skiped when initializing region
HRegion#replayRecoveredEditsIfAny
{code}
for (Path edits: files) {
if (edits == null || !this.fs.exists(edits)) {
LOG.warn(Null or non-existent edits file: + edits);
continue;
}
if (isZeroLengthThenDelete(this.fs, edits)) continue;
if (checkSafeToSkip) {
Path higher = files.higher(edits);
long maxSeqId = Long.MAX_VALUE;
if (higher != null) {
// Edit file name pattern, HLog.EDITFILES_NAME_PATTERN: -?[0-9]+
String fileName = higher.getName();
maxSeqId = Math.abs(Long.parseLong(fileName));
}
if (maxSeqId = minSeqId) {
String msg = Maximum possible sequenceid for this log is +
maxSeqId
+ , skipped the whole file, path= + edits;
LOG.debug(msg);
continue;
} else {
checkSafeToSkip = false;
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
