[ 
https://issues.apache.org/jira/browse/HBASE-23808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067372#comment-17067372
 ] 

Michael Stack commented on HBASE-23808:
---------------------------------------

I just saw this bit in TestMasterShutdown:

{code}
// Switching to master registry exacerbated a race in the master bootstrap that 
can result
        // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is 
essentially because
        // the server manager in HMaster is not initialized by the time 
shutdown() RPC (below) is
        // made to the master. The suspected reason as to why it was uncommon 
before HBASE-18095
        // is because the connection creation with ZK registry is so slow that 
by then the server
        // manager is usually init'ed in time for the RPC to be made. For now, 
adding an explicit
        // wait() in the test, waiting for the server manager to become 
available.
        final long timeout = TimeUnit.MINUTES.toMillis(10);
        assertNotEquals("timeout waiting for server manager to become 
available.",
          -1, Waiter.waitFor(htu.getConfiguration(), timeout,
            () -> masterThread.getMaster().getServerManager() != null...
{code}

... which probably explains the 'hang' I see.

In RSProcedureDispatcher#start, we were getting NPEs... which correlated to the 
test fails. Above, I added catch and returning failed start which seemed to be 
because Master had already been stopped. Made a subtask adding more debug for 
now while tests run over night.

> [Flakey Test] 
> TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-23808
>                 URL: https://issues.apache.org/jira/browse/HBASE-23808
>             Project: HBase
>          Issue Type: Test
>          Components: test
>    Affects Versions: 2.3.0
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>            Priority: Major
>             Fix For: 3.0.0, 2.3.0, 2.2.4
>
>         Attachments: 
> TEST-org.apache.hadoop.hbase.master.TestMasterShutdown.xml
>
>
> Reproduces locally from time to time. Not much to go on here. Looks like the 
> test is trying to do some fancy HBase cluster initialization order on top of 
> a mini-cluster. Failure seems related to trying to start the HBase master 
> before HDFS is fully initialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to