[
https://issues.apache.org/jira/browse/HBASE-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614184#comment-14614184
]
Hadoop QA commented on HBASE-14012:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12743615/14012.txt
against master branch at commit e640f1e76af8f32015f475629610da127897f01e.
ATTACHMENT ID: 12743615
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14667//testReport/
Release Findbugs (version 2.0.3) warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14667//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14667//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14667//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14667//console
This message is automatically generated.
> Double Assignment and Dataloss when ServerCrashProcedure runs during Master
> failover
> ------------------------------------------------------------------------------------
>
> Key: HBASE-14012
> URL: https://issues.apache.org/jira/browse/HBASE-14012
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 2.0.0, 1.2.0
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14012.txt
>
>
> (Rewrite to be more explicit about what the problem is)
> ITBLL. Master comes up (It is being killed every 1-5 minutes or so). It is
> joining a running cluster (all servers up except Master with most regions
> assigned out on cluster). ProcedureStore has two ServerCrashProcedures
> unfinished (RUNNABLE state) for two separate servers. One SCP is in the
> middle of the assign step when master crashes (SERVER_CRASH_ASSIGN). This SCP
> step has this comment on it:
> {code}
> // Assign may not be idempotent. SSH used to requeue the SSH if we
> got an IOE assigning
> // which is what we are mimicing here but it looks prone to double
> assignment if assign
> // fails midway. TODO: Test.
> {code}
> This issue is 1.2+ only since it is ServerCrashProcedure (Added in
> HBASE-13616, post hbase-1.1.x).
> Looking at ServerShutdownHandler, how we used to do crash processing before
> we moved over to the Pv2 framework, SSH may have (accidentally) avoided this
> issue since it does its processing in a big blob starting over if killed
> mid-crash. In particular, post-crash, SSH scans hbase:meta to find servers
> that were on the downed server. SCP scanneds Meta in one step, saves off the
> regions it finds into the ProcedureStore, and then in the next step, does
> actual assign. In this case, we crashed post-meta scan and during assign.
> Assign is a bulk assign. It mostly succeeded but got this:
> {code}
> 809622 2015-06-09 20:05:28,576 INFO [ProcedureExecutorThread-9]
> master.GeneralBulkAssigner: Failed assigning 3 regions to server
> c2021.halxg.cloudera.com,16020,1433905510696, reassigning them
> {code}
> So, most regions actually made it to new locations except for a few
> stragglers. All of the successfully assigned regions then are reassigned on
> other side of master restart when we replay the SCP assign step.
> Let me put together the scan meta and assign steps in SCP; this should do
> until we redo all of assign to run on Pv2.
> A few other things I noticed:
> In SCP, we only check if failover in first step, not for every step, which
> means ServerCrashProcedure will run if on reload it is beyond the first step.
> {code}
> // Is master fully online? If not, yield. No processing of servers unless
> master is up
> if (!services.getAssignmentManager().isFailoverCleanupDone()) {
> throwProcedureYieldException("Waiting on master failover to complete");
> }
> {code}
> This means we are assigning while Master is still coming up, a no-no (though
> it does not seem to have caused problem here). Fix.
> Also, I see that over the 8 hours of this particular log, each time the
> master crashes and comes back up, we queue a ServerCrashProcedure for c2022
> because an empty dir never gets cleaned up:
> {code}
> 39 2015-06-09 22:15:33,074 WARN [ProcedureExecutorThread-0]
> master.SplitLogManager: returning success without actually splitting and
> deleting all the log files in path
> hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2022.halxg.cloudera.com,16020,1433902151857-splitting
> {code}
> Fix this too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
