[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Attachment: 4862-v6-trunk.txt Patch v6 with javadoc updated according to reviews Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-trunk.txt, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157918#comment-13157918 ] Hadoop QA commented on HBASE-4862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505251/4862-v6-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/379//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/379//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/379//console This message is automatically generated. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.txt, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Attachment: 4862-v6-90.txt Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.txt, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157919#comment-13157919 ] Hadoop QA commented on HBASE-4862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505252/4862-v6-90.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/380//console This message is automatically generated. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.txt, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4868: -- Attachment: 4868-v3.txt Patch v3 applies same change to TestOfflineMetaRebuildHole and TestOfflineMetaRebuildOverlap. TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157921#comment-13157921 ] Hadoop QA commented on HBASE-4868: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505253/4868-v3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/381//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/381//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/381//console This message is automatically generated. TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4868: -- Attachment: 4868-0.92.txt TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157922#comment-13157922 ] Hadoop QA commented on HBASE-4868: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505254/4868-0.92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/382//console This message is automatically generated. TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4868: -- Issue Type: Test (was: Bug) Integrated to 0.92 and TRUNK. Thanks for the patch, Jinchao. Thanks for the review Jonathan. TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Test Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4868: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505254/4868-0.92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/382//console This message is automatically generated.) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Test Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4868: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Test Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4336) Convert source tree into maven modules
[
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112206#comment-13112206
]
Ted Yu edited comment on HBASE-4336 at 11/27/11 5:45 PM:
-
Unfortunately, the top level folder needs to just be a pom, so making it
package into 1 nice jar is out. Which makes it necessary to move the source
down a level in the hierarchy. With the move though, I was planning on making
the folders essentially the same, but 1 level down.
So hbase/src/main/java/org/apache/hbase/thing becomes
hbase/hbase-{core|security|server|it}/src/main/java/org/apache/hbase/thing
The move for the code in the patches is pretty minimal. Also, when do the
rebasing, I _think_ git will handle reapplying your patches on top of the moved
files (which it recognizes is moved). Of course, I'm assuming here that
everyone uses git ;)
TL;DR not really, as dictated by maven. Its gonna be painful, but once its
done, it will be much nicer.
was (Author: jesse_yates):
Unfortunately, the top level folder needs to just be a pom, so making it
package into 1 nice jar is out. Which makes it necessary to move the source
down a level in the hierarchy. With the move though, I was planning on making
the folders essentially the same, but 1 level down.
So hbase/src/main/java/org/apache/hbase/thing -
hbase/hbase-{core|security|server|it}/src/main/java/org/apache/hbase/thing
The move for the code in the patches is pretty minimal. Also, when do the
rebasing, I _think_ git will handle reapplying your patches on top of the moved
files (which it recognizes is moved). Of course, I'm assuming here that
everyone uses git ;)
TL;DR not really, as dictated by maven. Its gonna be painful, but once its
done, it will be much nicer.
Convert source tree into maven modules
--
Key: HBASE-4336
URL: https://issues.apache.org/jira/browse/HBASE-4336
Project: HBase
Issue Type: Task
Components: build
Reporter: Gary Helmling
When we originally converted the build to maven we had a single core module
defined, but later reverted this to a module-less build for the sake of
simplicity.
It now looks like it's time to re-address this, as we have an actual need for
modules to:
* provide a trimmed down client library that applications can make use of
* more cleanly support building against different versions of Hadoop, in
place of some of the reflection machinations currently required
* incorporate the secure RPC engine that depends on some secure Hadoop classes
I propose we start simply by refactoring into two initial modules:
* core - common classes and utilities, and client-side code and interfaces
* server - master and region server implementations and supporting code
This would also lay the groundwork for incorporating the HBase security
features that have been developed. Once the module structure is in place,
security-related features could then be incorporated into a third module --
security -- after normal review and approval. The security module could
then depend on secure Hadoop, without modifying the dependencies of the rest
of the HBase code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4336: -- Priority: Critical (was: Major) Fix Version/s: 0.94.0 Raising priority. I think this work is important so that security related classes can be packaged in their own jar. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.94.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157951#comment-13157951 ] Hudson commented on HBASE-4868: --- Integrated in HBase-TRUNK #2488 (See [https://builds.apache.org/job/HBase-TRUNK/2488/]) HBASE-4868 TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails (Gao Jinchao) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Test Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4823) long running scans lose benefit of bloomfilters and timerange hints
[
https://issues.apache.org/jira/browse/HBASE-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157975#comment-13157975
]
Phabricator commented on HBASE-4823:
khemani has commented on the revision HBASE-4823 [jira] long running scans
lose benefit of bloomfilters and timerange hints.
+1
looks good. This same change should also apply to trunk even though it has
filesonly and memstote-only scans.
REVISION DETAIL
https://reviews.facebook.net/D519
long running scans lose benefit of bloomfilters and timerange hints
---
Key: HBASE-4823
URL: https://issues.apache.org/jira/browse/HBASE-4823
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Assignee: Amitanand Aiyer
Attachments: HBASE-4823.D519.1.patch, TestScannerResets-89fb.txt
When you have a long running scan due to say an MR job, you can lose the
benefit of timerange hints bloom filters midway if your scanner gets reset.
[Note: The scanners can get reset say due to a flush or compaction].
In one of our workloads, we periodically want to do rollups on recent 15
minutes of data in a column family... but the timerange hint benefit is lost
midway when this resetScannerStack (shown below) happens. And end result-- we
end up reading all the old HFiles rather than just the recent HFiles.
{code}
private void resetScannerStack(KeyValue lastTopKey) throws IOException {
if (heap != null) {
throw new RuntimeException(StoreScanner.reseek run on an existing
heap!);
}
/* When we have the scan object, should we not pass it to getScanners()
* to get a limited set of scanners? We did so in the constructor and we
* could have done it now by storing the scan object from the constructor
*/
ListKeyValueScanner scanners = getScanners();
{code}
The comment in the code seems to be aware of this issue and even has the
suggested fix!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Attachment: 4847_pom.v3.patch Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Open (was: Patch Available) Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4847: --- Status: Patch Available (was: Open) will run small medium test Activate single jvm for small tests on jenkins -- Key: HBASE-4847 URL: https://issues.apache.org/jira/browse/HBASE-4847 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.94.0 Environment: build Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v2.patch, 4847_pom.v3.patch This will not revolutionate performances alone. We will win between 1 to 4 minutes. But we win as well: - it's a step for parallelizing the tests - new tests are less expensive as they do not create a new jvm: it's a continuous win - it will allow to push it on dev env while having the same env on dev on build, and 3 minutes are 10% of small + medium tests execution time. I will do a few submit patch to see if it works well before asking for the real commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4868) TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails
[ https://issues.apache.org/jira/browse/HBASE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157986#comment-13157986 ] Hudson commented on HBASE-4868: --- Integrated in HBase-0.92-security #18 (See [https://builds.apache.org/job/HBase-0.92-security/18/]) HBASE-4868 TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails (Gao Jinchao) tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java TestOfflineMetaRebuildBase#testMetaRebuild occasionally fails - Key: HBASE-4868 URL: https://issues.apache.org/jira/browse/HBASE-4868 Project: HBase Issue Type: Test Components: test Affects Versions: 0.92.0 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4868-0.92.txt, 4868-v3.txt, HBASE-4868_trial.patch, HBASE-4868_trunkv2.patch looks: https://builds.apache.org/job/HBase-TRUNK-security/7/testReport/org.apache.hadoop.hbase.util.hbck/TestOfflineMetaRebuildBase/testMetaRebuild/ Please review, see whether the method makes sense? If it makes sense, I will check other cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4874) TestHCM#testClosing fails if not enough entropy is available
[ https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13157996#comment-13157996 ] Ted Yu commented on HBASE-4874: --- I tried the patch on MacBook. TestHCM#testClosing passed. TestHCM#testClosing fails if not enough entropy is available Key: HBASE-4874 URL: https://issues.apache.org/jira/browse/HBASE-4874 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Ted Yu Attachments: 4874.txt TestHCM#testClosing fails if not enough entropy is available -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[
https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158001#comment-13158001
]
Ted Yu commented on HBASE-4847:
---
{code}
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 204.241 sec
FAILURE!
{code}
Activate single jvm for small tests on jenkins
--
Key: HBASE-4847
URL: https://issues.apache.org/jira/browse/HBASE-4847
Project: HBase
Issue Type: Improvement
Components: build, test
Affects Versions: 0.94.0
Environment: build
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch,
4847_pom.v2.patch, 4847_pom.v3.patch
This will not revolutionate performances alone. We will win between 1 to 4
minutes.
But we win as well:
- it's a step for parallelizing the tests
- new tests are less expensive as they do not create a new jvm: it's a
continuous win
- it will allow to push it on dev env while having the same env on dev on
build, and 3 minutes are 10% of small + medium tests execution time.
I will do a few submit patch to see if it works well before asking for the
real commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[ https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158013#comment-13158013 ] Hadoop QA commented on HBASE-4847: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505261/4847_pom.v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.util.TestRegionSplitter org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.regionserver.wal.TestHLogBench org.apache.hadoop.hbase.rest.TestGzipFilter org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.rest.TestScannersWithFilters org.apache.hadoop.hbase.TestInfoServers org.apache.hadoop.hbase.regionserver.TestParallelPut org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.filter.TestColumnRangeFilter org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.rest.TestStatusResource org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort org.apache.hadoop.hbase.rest.TestVersionResource org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.rest.TestRowResource org.apache.hadoop.hbase.rest.TestScannerResource org.apache.hadoop.hbase.rest.client.TestRemoteAdmin org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.rest.TestTableResource org.apache.hadoop.hbase.regionserver.wal.TestWALReplay org.apache.hadoop.hbase.master.TestHMasterRPCException org.apache.hadoop.hbase.util.TestIdLock org.apache.hadoop.hbase.rest.TestTransform org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.catalog.TestMetaReaderEditor org.apache.hadoop.hbase.client.TestMetaScanner org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.client.TestTimestampsFilter org.apache.hadoop.hbase.master.TestMaster org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass org.apache.hadoop.hbase.rest.TestSchemaResource org.apache.hadoop.hbase.TestAcidGuarantees org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase org.apache.hadoop.hbase.avro.TestAvroServer org.apache.hadoop.hbase.rest.client.TestRemoteTable org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.util.TestHBaseFsck org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove org.apache.hadoop.hbase.client.TestHTableUtil org.apache.hadoop.hbase.thrift.TestThriftServer org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics org.apache.hadoop.hbase.util.TestMergeTable org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.rest.TestMultiRowResource org.apache.hadoop.hbase.TestMultiVersions org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
[jira] [Commented] (HBASE-4847) Activate single jvm for small tests on jenkins
[
https://issues.apache.org/jira/browse/HBASE-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158015#comment-13158015
]
nkeywal commented on HBASE-4847:
Seems that the machine was not in its best state:
{noformat}
java.lang.OutOfMemoryError: unable to create new native thread
{noformat}
{noformat}
Error while running command to get file permissions : java.io.IOException:
Cannot run program /bin/ls: java.io.IOException: error=11, Resource
temporarily unavailable at
java.lang.ProcessBuilder.start(ProcessBuilder.java:460) at
org.apache.hadoop.util.Shell.runCommand(Shell.java:200) at
org.apache.hadoop.util.Shell.run(Shell.java:182) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) at
org.apache.hadoop.util.Shell.execCommand(Shell.java:461) at
org.apache.hadoop.util.Shell.execCommand(Shell.java:444) at
org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703) at
{noformat}
Will retry later.
Activate single jvm for small tests on jenkins
--
Key: HBASE-4847
URL: https://issues.apache.org/jira/browse/HBASE-4847
Project: HBase
Issue Type: Improvement
Components: build, test
Affects Versions: 0.94.0
Environment: build
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 4847_pom.patch, 4847_pom.v2.patch, 4847_pom.v2.patch,
4847_pom.v2.patch, 4847_pom.v3.patch
This will not revolutionate performances alone. We will win between 1 to 4
minutes.
But we win as well:
- it's a step for parallelizing the tests
- new tests are less expensive as they do not create a new jvm: it's a
continuous win
- it will allow to push it on dev env while having the same env on dev on
build, and 3 minutes are 10% of small + medium tests execution time.
I will do a few submit patch to see if it works well before asking for the
real commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4869) Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has.
[ https://issues.apache.org/jira/browse/HBASE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158025#comment-13158025 ] Jimmy Xiang commented on HBASE-4869: There are some issue. It could be with my environment. This test hangs in starting the mini cluster (eclipse 3.7.1, ubuntu 11.10, 64bit, jdk 1.6.0_24) Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has. --- Key: HBASE-4869 URL: https://issues.apache.org/jira/browse/HBASE-4869 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 0.90.2 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.92.0 Attachments: 0001-HBASE-4869-Backport-to-0.92-HBASE-4797-availability-.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4877) TestHCM failing sporadically on jenkins and always for me on an ubuntu machine
[ https://issues.apache.org/jira/browse/HBASE-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158050#comment-13158050 ] stack commented on HBASE-4877: -- I get the hangs w/ your pom patch or not Lars. I think we need this one too... least my build always fails this test w/o this patch (and reasoning about it, we should be seeing this fail more often that we do). TestHCM failing sporadically on jenkins and always for me on an ubuntu machine -- Key: HBASE-4877 URL: https://issues.apache.org/jira/browse/HBASE-4877 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.92.0 Attachments: 4877.txt TestHCM takes 13 minutes for me on ubuntu and fails in testClosing. It runs fine on a mac. The problem test is not testClosing as I thought originally, its the test just previous, testConnectionUniqueness. testConnectionUniqueness creates the maximum cached HConnections + 10 to verify each is unique if the passed in Configuration has a unique hash. Problem comes when zk enforces its default max from single host of 30 connections which is (max cached + 10). The max does not seem to be enforced on mac for me. The max connections runs up to max of 31 -- zk max + 1 -- and works fine until we do the +10. On ubuntu, when we hit the zk max of 30, we'll then go into a fail mode where we cannot set up a zk session... each attempt takes a while. Test passes, it just takes a while. Only, the uniqueness test does not clean up after itself and so all sessions to zk are outstanding so then when the subsequent testClosing runs, it can't set up connections successfully so fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158053#comment-13158053 ] stack commented on HBASE-4838: -- I'm good on it going in. Port 2856 (TestAcidGuarantee is failing) to 0.92 Key: HBASE-4838 URL: https://issues.apache.org/jira/browse/HBASE-4838 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0 Attachments: 4838-v1.txt, 4838-v3.txt Moving back port into a separate issue (as suggested by JonH), because this not trivial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4874) Run tests with non-secure random (some tests may hang otherwise)
[ https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4874: - Description: TestHCM#testClosing fails on Linux if not enough entropy is available in /dev/random (was: TestHCM#testClosing fails if not enough entropy is available) Affects Version/s: 0.94.0 Fix Version/s: 0.94.0 0.92.0 Hadoop Flags: Reviewed Summary: Run tests with non-secure random (some tests may hang otherwise) (was: TestHCM#testClosing fails if not enough entropy is available) Run tests with non-secure random (some tests may hang otherwise) Key: HBASE-4874 URL: https://issues.apache.org/jira/browse/HBASE-4874 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0 Reporter: Ted Yu Fix For: 0.92.0, 0.94.0 Attachments: 4874.txt TestHCM#testClosing fails on Linux if not enough entropy is available in /dev/random -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4874) Run tests with non-secure random, some tests hang otherwise
[ https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4874: - Summary: Run tests with non-secure random, some tests hang otherwise (was: Run tests with non-secure random (some tests may hang otherwise)) Run tests with non-secure random, some tests hang otherwise --- Key: HBASE-4874 URL: https://issues.apache.org/jira/browse/HBASE-4874 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0 Reporter: Ted Yu Fix For: 0.92.0, 0.94.0 Attachments: 4874.txt TestHCM#testClosing fails on Linux if not enough entropy is available in /dev/random -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4874) Run tests with non-secure random, some tests hang otherwise
[ https://issues.apache.org/jira/browse/HBASE-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-4874. -- Resolution: Fixed Assignee: Lars Hofhansl Committed to 0.92 and trunk Run tests with non-secure random, some tests hang otherwise --- Key: HBASE-4874 URL: https://issues.apache.org/jira/browse/HBASE-4874 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0 Reporter: Ted Yu Assignee: Lars Hofhansl Fix For: 0.92.0, 0.94.0 Attachments: 4874.txt TestHCM#testClosing fails on Linux if not enough entropy is available in /dev/random -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-4838. -- Resolution: Fixed Hadoop Flags: Reviewed Committed to 0.92 Port 2856 (TestAcidGuarantee is failing) to 0.92 Key: HBASE-4838 URL: https://issues.apache.org/jira/browse/HBASE-4838 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0 Attachments: 4838-v1.txt, 4838-v3.txt Moving back port into a separate issue (as suggested by JonH), because this not trivial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158086#comment-13158086 ] Lars Hofhansl commented on HBASE-4838: -- Whoops, missing adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java. Fixed in addendum patch. Port 2856 (TestAcidGuarantee is failing) to 0.92 Key: HBASE-4838 URL: https://issues.apache.org/jira/browse/HBASE-4838 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0 Attachments: 4838-v1.txt, 4838-v3.txt Moving back port into a separate issue (as suggested by JonH), because this not trivial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4869) Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with edits we know older than what region currently has.
[
https://issues.apache.org/jira/browse/HBASE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158102#comment-13158102
]
Lars Hofhansl commented on HBASE-4869:
--
I applied the patch and ran TestHRegion:
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.regionserver.TestHRegion
Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.447 sec
Results :
Tests run: 60, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESSFUL
[INFO]
[INFO] Total time: 1 minute 8 seconds
[INFO] Finished at: Sun Nov 27 15:55:19 PST 2011
[INFO] Final Memory: 69M/316M
[INFO]
{noformat}
So might be your env?
Maybe related to HBASE-4874? I ran with that applied too.
Backport to 0.92: HBASE-4797 [availability] Skip recovered.edits files with
edits we know older than what region currently has.
---
Key: HBASE-4869
URL: https://issues.apache.org/jira/browse/HBASE-4869
Project: HBase
Issue Type: Improvement
Components: wal
Affects Versions: 0.90.2
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Fix For: 0.92.0
Attachments:
0001-HBASE-4869-Backport-to-0.92-HBASE-4797-availability-.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4823) long running scans lose benefit of bloomfilters and timerange hints
[
https://issues.apache.org/jira/browse/HBASE-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158105#comment-13158105
]
Phabricator commented on HBASE-4823:
lhofhansl has commented on the revision HBASE-4823 [jira] long running scans
lose benefit of bloomfilters and timerange hints.
+1 lgtm
INLINE COMMENTS
src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerResets.java:1
Wanna add a Copyright notice?
REVISION DETAIL
https://reviews.facebook.net/D519
long running scans lose benefit of bloomfilters and timerange hints
---
Key: HBASE-4823
URL: https://issues.apache.org/jira/browse/HBASE-4823
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Assignee: Amitanand Aiyer
Attachments: HBASE-4823.D519.1.patch, TestScannerResets-89fb.txt
When you have a long running scan due to say an MR job, you can lose the
benefit of timerange hints bloom filters midway if your scanner gets reset.
[Note: The scanners can get reset say due to a flush or compaction].
In one of our workloads, we periodically want to do rollups on recent 15
minutes of data in a column family... but the timerange hint benefit is lost
midway when this resetScannerStack (shown below) happens. And end result-- we
end up reading all the old HFiles rather than just the recent HFiles.
{code}
private void resetScannerStack(KeyValue lastTopKey) throws IOException {
if (heap != null) {
throw new RuntimeException(StoreScanner.reseek run on an existing
heap!);
}
/* When we have the scan object, should we not pass it to getScanners()
* to get a limited set of scanners? We did so in the constructor and we
* could have done it now by storing the scan object from the constructor
*/
ListKeyValueScanner scanners = getScanners();
{code}
The comment in the code seems to be aware of this issue and even has the
suggested fix!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158121#comment-13158121
]
Hadoop QA commented on HBASE-4605:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505275/java_HBASE-4605_v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/384//console
This message is automatically generated.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4879) Add Constraint support to shell
Add Constraint support to shell --- Key: HBASE-4879 URL: https://issues.apache.org/jira/browse/HBASE-4879 Project: HBase Issue Type: Improvement Affects Versions: 0.94.0 Reporter: Jesse Yates Follow-on ticket to HBASE-4605. Extend the shell functionality to include altering a table to add a Constraint. Discussion around this can be found at: http://search-hadoop.com/m/EeYb3dezMM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158086#comment-13158086 ] Lars Hofhansl edited comment on HBASE-4838 at 11/28/11 12:52 AM: - Whoops, missed adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java. Fixed in addendum patch. was (Author: lhofhansl): Whoops, missing adding MultiVersionConsistencyControl.java and removing ReadWriteConsistencyControl.java. Fixed in addendum patch. Port 2856 (TestAcidGuarantee is failing) to 0.92 Key: HBASE-4838 URL: https://issues.apache.org/jira/browse/HBASE-4838 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0 Attachments: 4838-v1.txt, 4838-v3.txt Moving back port into a separate issue (as suggested by JonH), because this not trivial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v2.patch
oops, wrong file. This should be right.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158126#comment-13158126
]
Hadoop QA commented on HBASE-4605:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505277/java_HBASE-4605_v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/385//console
This message is automatically generated.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v3.patch
Forgot --no-prefix. Let's try this again.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Attachment: 4862-v6-trunk.patch Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Status: Patch Available (was: Open) Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Attachment: (was: 4862-v6-trunk.txt) Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[ https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158139#comment-13158139 ] Hadoop QA commented on HBASE-4605: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505280/java_HBASE-4605_v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 69 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestRollingRestart org.apache.hadoop.hbase.util.TestRegionSplitter org.apache.hadoop.hbase.io.hfile.TestLruBlockCache org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.master.TestRestartCluster org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.regionserver.TestStore org.apache.hadoop.hbase.regionserver.wal.TestHLogBench org.apache.hadoop.hbase.rest.TestGzipFilter org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.rest.TestScannersWithFilters org.apache.hadoop.hbase.TestInfoServers org.apache.hadoop.hbase.regionserver.TestParallelPut org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.filter.TestColumnRangeFilter org.apache.hadoop.hbase.mapred.TestTableInputFormat org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.rest.TestStatusResource org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort org.apache.hadoop.hbase.rest.TestVersionResource org.apache.hadoop.hbase.client.TestScannerTimeout org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.regionserver.TestSplitTransaction org.apache.hadoop.hbase.rest.TestRowResource org.apache.hadoop.hbase.rest.TestScannerResource org.apache.hadoop.hbase.ipc.TestDelayedRpc org.apache.hadoop.hbase.rest.client.TestRemoteAdmin org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.rest.TestTableResource org.apache.hadoop.hbase.regionserver.wal.TestWALReplay org.apache.hadoop.hbase.master.TestHMasterRPCException org.apache.hadoop.hbase.util.TestIdLock org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster org.apache.hadoop.hbase.rest.TestTransform org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl org.apache.hadoop.hbase.client.TestMultipleTimestamps org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.catalog.TestMetaReaderEditor org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.regionserver.TestKeepDeletes
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158141#comment-13158141 ] Hadoop QA commented on HBASE-4862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505283/4862-v6-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/387//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/387//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/387//console This message is automatically generated. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-4862: Attachment: hbase-4862v7fortrunk.patch hbase-4862v7for0.90.patch Based on patchV6,update javadoc of HLog#getSplitEditFilesSorted Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158146#comment-13158146 ] Hadoop QA commented on HBASE-4862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505285/hbase-4862v7fortrunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/388//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/388//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/388//console This message is automatically generated. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505283/4862-v6-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 67 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/387//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/387//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/387//console This message is automatically generated.) Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4862: -- Attachment: 4862-0.92.txt Patch for 0.92 branch. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158151#comment-13158151 ] Hadoop QA commented on HBASE-4862: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505287/4862-0.92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/389//console This message is automatically generated. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4838) Port 2856 (TestAcidGuarantee is failing) to 0.92
[ https://issues.apache.org/jira/browse/HBASE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158159#comment-13158159 ] Hudson commented on HBASE-4838: --- Integrated in HBase-0.92-security #19 (See [https://builds.apache.org/job/HBase-0.92-security/19/]) HBASE-4838 addendum, missing file larsh : Files : * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java Port 2856 (TestAcidGuarantee is failing) to 0.92 Key: HBASE-4838 URL: https://issues.apache.org/jira/browse/HBASE-4838 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0 Attachments: 4838-v1.txt, 4838-v3.txt Moving back port into a separate issue (as suggested by JonH), because this not trivial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158177#comment-13158177 ] Ted Yu commented on HBASE-4862: --- Integrated to 0.90, 0.92 branches and TRUNK. Thanks for the patch Chunhui. Thanks for the review Jonathan. Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158178#comment-13158178 ] Hudson commented on HBASE-4862: --- Integrated in HBase-TRUNK #2490 (See [https://builds.apache.org/job/HBase-TRUNK/2490/]) HBASE-4862 Splitting hlog and opening region concurrently may cause data loss (Chunhui Shen) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be put
new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-4880:
Attachment: hbase-4880.patch
Region isn't on service unitl completing openRegionHanlder successfully.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158199#comment-13158199
]
ramkrishna.s.vasudevan commented on HBASE-4878:
---
@Chunhui
{code}
status.setStatus(Archiving logs after completed split);
{code}
also needs to be moved before
{code}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
{code}
Otherwise the status will be interleaved with
Finishing writing output logs and closing down.
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4878.diff
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-4878:
Attachment: hbase-4878v2.patch
@ramkrishna
Have done in patchv2
Thanks.
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4878.diff, hbase-4878v2.patch
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4773) HBaseAdmin leaks ZooKeeper connections
[
https://issues.apache.org/jira/browse/HBASE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xufeng updated HBASE-4773:
--
Attachment: trunk_4773_patch.patch
branches_4773.patch
submit the branches and trunk patch.
unit test:ok.
test in cluster:ok
HBaseAdmin leaks ZooKeeper connections
--
Key: HBASE-4773
URL: https://issues.apache.org/jira/browse/HBASE-4773
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.4
Reporter: gaojinchao
Priority: Critical
Fix For: 0.90.5
Attachments: 4773.patch, branches_4773.patch, trunk_4773_patch.patch
When master crashs, HBaseAdmin will leaks ZooKeeper connections
I think we should close the zk connetion when throw MasterNotRunningException
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException {
this.conf = HBaseConfiguration.create(c);
this.connection = HConnectionManager.getConnection(this.conf);
this.pause = this.conf.getLong(hbase.client.pause, 1000);
this.numRetries = this.conf.getInt(hbase.client.retries.number, 10);
this.retryLongerMultiplier =
this.conf.getInt(hbase.client.retries.longer.multiplier, 10);
//we should add this code and close the zk connection
try{
this.connection.getMaster();
}catch(MasterNotRunningException e){
HConnectionManager.deleteConnection(conf, false);
throw e;
}
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158208#comment-13158208
]
ramkrishna.s.vasudevan commented on HBASE-4878:
---
+1 on patch.
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4878.diff, hbase-4878v2.patch
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4881) Unhealthy region is on service caused by rollback of region splitting
Unhealthy region is on service caused by rollback of region splitting
-
Key: HBASE-4881
URL: https://issues.apache.org/jira/browse/HBASE-4881
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
If region splitting is failed in the state of JournalEntry.CLOSED_PARENT_REGION
It will be rollback as the following steps:
{code}
1.case CLOSED_PARENT_REGION:
this.parent.initialize();
break;
2.case CREATE_SPLIT_DIR:
this.parent.writestate.writesEnabled = true;
cleanupSplitDir(fs, this.splitdir);
break;
3.case SET_SPLITTING_IN_ZK:
if (server != null server.getZooKeeper() != null) {
cleanZK(server, this.parent.getRegionInfo());
}
break;
{code}
If this.parent.initialize() throws IOException in step 1,
If check filesystem is ok. it will do nothing.
However, the parent region is on service now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4878:
--
Assignee: chunhui shen
Affects Version/s: 0.92.0
Status: Patch Available (was: Open)
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4878.diff, hbase-4878v2.patch
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-4880:
-
Assignee: chunhui shen
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158215#comment-13158215
]
Jesse Yates commented on HBASE-4605:
Hmmm, so it looks like most of them failed. So far, its passing on my local
machine (OSX). Could this be a build system overload issue? Or something along
those lines?
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4881) Unhealthy region is on service caused by rollback of region splitting
[
https://issues.apache.org/jira/browse/HBASE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158216#comment-13158216
]
chunhui shen commented on HBASE-4881:
-
This case happen in our test.
It causes the regionserver can't do flushing cache, because step2 is not
executed, and this.parent's state is
closing=false,closed=false,writestate.writesEnabled =fasle.
Unhealthy region is on service caused by rollback of region splitting
-
Key: HBASE-4881
URL: https://issues.apache.org/jira/browse/HBASE-4881
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
If region splitting is failed in the state of
JournalEntry.CLOSED_PARENT_REGION
It will be rollback as the following steps:
{code}
1.case CLOSED_PARENT_REGION:
this.parent.initialize();
break;
2.case CREATE_SPLIT_DIR:
this.parent.writestate.writesEnabled = true;
cleanupSplitDir(fs, this.splitdir);
break;
3.case SET_SPLITTING_IN_ZK:
if (server != null server.getZooKeeper() != null) {
cleanZK(server, this.parent.getRegionInfo());
}
break;
{code}
If this.parent.initialize() throws IOException in step 1,
If check filesystem is ok. it will do nothing.
However, the parent region is on service now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158224#comment-13158224
]
ramkrishna.s.vasudevan commented on HBASE-4880:
---
If we are going to make this change, why don't we avoid the step of adding the
region into online region before step 3.
And add it only at the end?
Let's see what others have to say on this.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158227#comment-13158227
]
Ted Yu commented on HBASE-4605:
---
From
https://builds.apache.org/job/PreCommit-HBASE-Build/386//testReport/org.apache.hadoop.hbase/TestRegionRebalancing/org_apache_hadoop_hbase_TestRegionRebalancing/:
{code}
java.lang.OutOfMemoryError: unable to create new native thread
{code}
Report from HadoopQA was not reliable these days.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158234#comment-13158234
]
chunhui shen commented on HBASE-4880:
-
@ramkrishna
If we avoid the step of adding the region into online region before step 3.
We need modify the HReionserver#postOpenDeployTasks, it seems changing too much.
In my patchv1, remove it from online regions immediately at memory level after
updating .META.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158240#comment-13158240
]
Ted Yu commented on HBASE-4880:
---
@Chunhui:
HadoopQA isn't functional at the moment.
Can you let us know if test suite passes ?
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4878:
--
Fix Version/s: 0.94.0
0.92.0
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.92.0, 0.94.0
Attachments: hbase-4878.diff, hbase-4878v2.patch
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158242#comment-13158242
]
chunhui shen commented on HBASE-4880:
-
@Ted
I'm running test for trunk now.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4881) Unhealthy region is on service caused by rollback of region splitting
[
https://issues.apache.org/jira/browse/HBASE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158244#comment-13158244
]
Ted Yu commented on HBASE-4881:
---
{code}
+ // TODO: Verify.
+ this.parent.initialize();
{code}
Can you tell us your plan to validate the above ?
Thanks
Unhealthy region is on service caused by rollback of region splitting
-
Key: HBASE-4881
URL: https://issues.apache.org/jira/browse/HBASE-4881
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4881.patch
If region splitting is failed in the state of
JournalEntry.CLOSED_PARENT_REGION
It will be rollback as the following steps:
{code}
1.case CLOSED_PARENT_REGION:
this.parent.initialize();
break;
2.case CREATE_SPLIT_DIR:
this.parent.writestate.writesEnabled = true;
cleanupSplitDir(fs, this.splitdir);
break;
3.case SET_SPLITTING_IN_ZK:
if (server != null server.getZooKeeper() != null) {
cleanZK(server, this.parent.getRegionInfo());
}
break;
{code}
If this.parent.initialize() throws IOException in step 1,
If check filesystem is ok. it will do nothing.
However, the parent region is on service now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4773) HBaseAdmin leaks ZooKeeper connections
[
https://issues.apache.org/jira/browse/HBASE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158249#comment-13158249
]
Ted Yu commented on HBASE-4773:
---
@Xufeng:
HadoopQA isn't functional at the moment.
Please clarify whether you have run patch for TRUNK through unit test suite.
HBaseAdmin leaks ZooKeeper connections
--
Key: HBASE-4773
URL: https://issues.apache.org/jira/browse/HBASE-4773
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.4
Reporter: gaojinchao
Priority: Critical
Fix For: 0.90.5
Attachments: 4773.patch, branches_4773.patch, trunk_4773_patch.patch
When master crashs, HBaseAdmin will leaks ZooKeeper connections
I think we should close the zk connetion when throw MasterNotRunningException
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException {
this.conf = HBaseConfiguration.create(c);
this.connection = HConnectionManager.getConnection(this.conf);
this.pause = this.conf.getLong(hbase.client.pause, 1000);
this.numRetries = this.conf.getInt(hbase.client.retries.number, 10);
this.retryLongerMultiplier =
this.conf.getInt(hbase.client.retries.longer.multiplier, 10);
//we should add this code and close the zk connection
try{
this.connection.getMaster();
}catch(MasterNotRunningException e){
HConnectionManager.deleteConnection(conf, false);
throw e;
}
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4881) Unhealthy region is on service caused by rollback of region splitting
[
https://issues.apache.org/jira/browse/HBASE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158254#comment-13158254
]
chunhui shen commented on HBASE-4881:
-
@Ted
sorry，I forgot to delete this line(// TODO: Verify.)
I just update the IOException to RunTimeException which is thrown by
this.parent.initialize()
Unhealthy region is on service caused by rollback of region splitting
-
Key: HBASE-4881
URL: https://issues.apache.org/jira/browse/HBASE-4881
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Attachments: hbase-4881.patch
If region splitting is failed in the state of
JournalEntry.CLOSED_PARENT_REGION
It will be rollback as the following steps:
{code}
1.case CLOSED_PARENT_REGION:
this.parent.initialize();
break;
2.case CREATE_SPLIT_DIR:
this.parent.writestate.writesEnabled = true;
cleanupSplitDir(fs, this.splitdir);
break;
3.case SET_SPLITTING_IN_ZK:
if (server != null server.getZooKeeper() != null) {
cleanZK(server, this.parent.getRegionInfo());
}
break;
{code}
If this.parent.initialize() throws IOException in step 1,
If check filesystem is ok. it will do nothing.
However, the parent region is on service now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13158266#comment-13158266 ] Hudson commented on HBASE-4862: --- Integrated in HBase-0.92-security #20 (See [https://builds.apache.org/job/HBase-0.92-security/20/]) HBASE-4862 Splitting hlog and opening region concurrently may cause data loss (Chunhui Shen) move JIRA to 0.90 section in CHANGES.txt HBASE-4862 Splitting hlog and opening region concurrently may cause data loss (Chunhui Shen) tedyu : Files : * /hbase/branches/0.92/CHANGES.txt tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
