[
https://issues.apache.org/jira/browse/HDFS-15762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258997#comment-17258997
]
Toshihiko Uchida commented on HDFS-15762:
-----------------------------------------
It seems that this unit test fails when at least one of the first two DNs are
always at the end of data pipelines.
Is is because in that case, qop of SaslClient in the DN will not be updated.
I ran the test multiple times in my local docker environment, and could
reproduce the failure. (For debugging, I added an assertion message to print DN
and also increased the sleep time period from 100 to 1,000.)
The following stack trace and standard output in a failed test might support my
guess.
- Stack Trace
{code}
TestMultipleNNPortQOP.testMultipleNNPortOverwriteDownStream:267 qop should be
equal to auth on
DataNode{data=FSDataset{dirpath='[/home/toshi/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1,
/home/toshi/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2]'},
localName='127.0.0.1:35861',
datanodeUuid='0d08c609-f49a-4b7d-99fc-8cd9f3184bc5', xmitsInProgress=0}
expected:<auth> but was:<null>
{code}
- Standard Output
{code}
2021-01-05 15:24:05,689 [IPC Server handler 0 on default port 34837] INFO
hdfs.StateChange (FSDirWriteFileOp.java:logAllocatedBlock(802)) - BLOCK*
allocate blk_1073741825_1001, replicas=127.0.0.1:43177, 127.0.0.1:42217,
127.0.0.1:35861 for /file1
(omitted)
2021-01-05 15:24:06,987 [IPC Server handler 3 on default port 34837] INFO
hdfs.StateChange (FSDirWriteFileOp.java:logAllocatedBlock(802)) - BLOCK*
allocate blk_1073741826_1002, replicas=127.0.0.1:43177, 127.0.0.1:42217,
127.0.0.1:35861 for /file1
(omitted)
2021-01-05 15:24:07,276 [IPC Server handler 9 on default port 34837] INFO
hdfs.StateChange (FSDirWriteFileOp.java:logAllocatedBlock(802)) - BLOCK*
allocate blk_1073741827_1003, replicas=127.0.0.1:43177, 127.0.0.1:42217,
127.0.0.1:35861 for /file1
{code}
> TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails
> intermittently
> --------------------------------------------------------------------------------
>
> Key: HDFS-15762
> URL: https://issues.apache.org/jira/browse/HDFS-15762
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Toshihiko Uchida
> Priority: Minor
> Labels: flaky-test
> Attachments: PR2585#1-TestMultipleNNPortQOP-output.txt
>
>
> This unit test failed in https://github.com/apache/hadoop/pull/2585 due to an
> AssertionError.
> {code}
> java.lang.AssertionError: expected:<auth> but was:<null>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.TestMultipleNNPortQOP.testMultipleNNPortOverwriteDownStream(TestMultipleNNPortQOP.java:267)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> The failure occurred at the following assertion.
> {code}
> doTest(fsPrivacy, PATH1);
> for (int i = 0; i < 2; i++) {
> DataNode dn = dataNodes.get(i);
> SaslDataTransferClient saslClient = dn.getSaslClient();
> String qop = null;
> // It may take some time for the qop to populate
> // to all DNs, check in a loop.
> for (int trial = 0; trial < 10; trial++) {
> qop = saslClient.getTargetQOP();
> if (qop != null) {
> break;
> }
> Thread.sleep(100);
> }
> assertEquals("auth", qop);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
