[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131120#comment-16131120 ]
Hudson commented on HBASE-18551: -------------------------------- FAILURE: Integrated in Jenkins build HBASE-14070.HLC #233 (See [https://builds.apache.org/job/HBASE-14070.HLC/233/]) HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers (stack: rev 2dd75d10f8818ed31fcc36bd89024e9ad728ae41) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java Revert "HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers" (stack: rev e4ba404a5a65d5224255551e26045b8d37fbfda8) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers (stack: rev 6f44b24860192d81dbf88ffd834d4b998a6fe636) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java HBASE-18551 [AMv2] UnassignProcedure and crashed regionservers; (stack: rev 1070888fff3a89d435018f11bfb2fd5609be8bab) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java > [AMv2] UnassignProcedure and crashed regionservers > -------------------------------------------------- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 > Reporter: stack > Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18551.master.001.patch, > HBASE-18551.master.002.patch, HBASE-18551.master.003.patch > > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)