Jose Luis López created HDFS-17987:
--------------------------------------
Summary: TestDataNodeLifeline: stub NameNode spies before
installing them in BPServiceActor
Key: HDFS-17987
URL: https://issues.apache.org/jira/browse/HDFS-17987
Project: Hadoop HDFS
Issue Type: Test
Components: datanode, test
Reporter: Jose Luis López
{{TestDataNodeLifeline#setup()}} installs Mockito spies of the NameNode and
lifeline proxies into the running {{BPServiceActor}}. The tests then stub those
spies while the actor thread is still calling them every second. Mockito
stubbing is not thread-safe, so the actor thread can fail with:
{noformat}
java.lang.AssertionError
at
org.mockito.internal.stubbing.InvocationContainerImpl.setMethodForStubbing(InvocationContainerImpl.java:123)
...
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:571)
{noformat}
This ends the block pool service. {{testSendLifelineIfHeartbeatBlocked}} then
fails when it reconfigures the data dirs, because there is no block pool left:
{noformat}
Caused by: java.io.IOException: FAILED to ADD: [DISK]file:.../data-new:
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index 0 out of bounds for length 0
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.prepareVolume(DataStorage.java:334)
{noformat}
Surefire reruns of the failed test then hang for 600 s in {{setup()}}:
{{testHeartbeatAndLifelineOnError}} sets the static
{{BlockManagerFaultInjector.instance}} to an injector that fails every
heartbeat and never restores it, so a rerun in the same JVM never sees its
DataNode heartbeat and waits in {{MiniDFSCluster#waitActive}} until the timeout.
Seen in CI for https://github.com/apache/hadoop/pull/8704. Reproducible on
trunk: the unmodified test failed this way in 1 of the first 5 local runs.
*Fix:* stub the spies before installing them in the actor, and restore the
fault injector in {{testHeartbeatAndLifelineOnError}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]