Jose Luis López created HDFS-17987:
--------------------------------------

             Summary: TestDataNodeLifeline: stub NameNode spies before 
installing them in BPServiceActor
                 Key: HDFS-17987
                 URL: https://issues.apache.org/jira/browse/HDFS-17987
             Project: Hadoop HDFS
          Issue Type: Test
          Components: datanode, test
            Reporter: Jose Luis López


{{TestDataNodeLifeline#setup()}} installs Mockito spies of the NameNode and 
lifeline proxies into the running {{BPServiceActor}}. The tests then stub those 
spies while the actor thread is still calling them every second. Mockito 
stubbing is not thread-safe, so the actor thread can fail with:

{noformat}
java.lang.AssertionError
        at 
org.mockito.internal.stubbing.InvocationContainerImpl.setMethodForStubbing(InvocationContainerImpl.java:123)
        ...
        at 
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:571)
{noformat}

This ends the block pool service. {{testSendLifelineIfHeartbeatBlocked}} then 
fails when it reconfigures the data dirs, because there is no block pool left:

{noformat}
Caused by: java.io.IOException: FAILED to ADD: [DISK]file:.../data-new: 
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: 
Index 0 out of bounds for length 0
        at 
org.apache.hadoop.hdfs.server.datanode.DataStorage.prepareVolume(DataStorage.java:334)
{noformat}

Surefire reruns of the failed test then hang for 600 s in {{setup()}}: 
{{testHeartbeatAndLifelineOnError}} sets the static 
{{BlockManagerFaultInjector.instance}} to an injector that fails every 
heartbeat and never restores it, so a rerun in the same JVM never sees its 
DataNode heartbeat and waits in {{MiniDFSCluster#waitActive}} until the timeout.

Seen in CI for https://github.com/apache/hadoop/pull/8704. Reproducible on 
trunk: the unmodified test failed this way in 1 of the first 5 local runs.

*Fix:* stub the spies before installing them in the actor, and restore the 
fault injector in {{testHeartbeatAndLifelineOnError}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to