[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v5-again.patch Hmm, it looks like the problem was using ClassSize.REFERENCE in stead of Long as it is only 4 bytes on a 32 bit jvm. Let's see if this fixes this. Really sorry for the attachment-email-span folks (but couldn't repro it on my local+jenkins, so have to try this here). > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v5.patch Renamed the HLogService to ServiceIds, as suggested in the earlier review. TestHeapSize failure is a bit bizzare in the earlier run as it didn't happen in my testing (both on jenkins and local). I will keep an eye on it here. > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch, HBASE-8741-v5.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8741: - Fix Version/s: 0.95.2 > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8741: - Attachment: HBASE-8741-v4-again.patch Retry > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, > HBASE-8741-v4.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v4-again.patch Attaching the v4 patch again (along with minor refactoring in some tests methods) to let qa run again. The prior testing still holds. > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v4.patch Patch that contains latest review comments. Attaching here for qabot. Ran jenkins + mvn test -Dtest=TestHLog* multiple times. Also ran a patched version locally. > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch, HBASE-8741-v4.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v3.patch In this patch, I refactored the lines which have more than 100 characters. The failed test passes on local (I see it failing on other qa runs, so probably it is not related to this patch). Let the qa bot run. > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, > HBASE-8741-v3.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v2.patch Attaching here to let qa run. It is the same patch which is on rb. > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Status: Patch Available (was: Open) > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-v0.patch > Mutations on Regions in recovery mode might have same sequenceIDs > - > > Key: HBASE-8741 > URL: https://issues.apache.org/jira/browse/HBASE-8741 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8741-v0.patch > > > Currently, when opening a region, we find the maximum sequence ID from all > its HFiles and then set the LogSequenceId of the log (in case the later is at > a small value). This works good in recovered.edits case as we are not writing > to the region until we have replayed all of its previous edits. > With distributed log replay, if we want to enable writes while a region is > under recovery, we need to make sure that the logSequenceId > maximum > logSequenceId of the old regionserver. Otherwise, we might have a situation > where new edits have same (or smaller) sequenceIds. > We can store region level information in the WALTrailer, than this scenario > could be avoided by: > a) reading the trailer of the "last completed" file, i.e., last wal file > which has a trailer and, > b) completely reading the last wal file (this file would not have the > trailer, so it needs to be read completely). > In future, if we switch to multi wal file, we could read the trailer for all > completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
