[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112223#comment-17112223 ] Jim Brennan commented on HDFS-9376: --- Thanks [~iwasakims]! I figured that was the case. > TestSeveralNameNodes fails occasionally > --- > > Key: HDFS-9376 > URL: https://issues.apache.org/jira/browse/HDFS-9376 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Kihwal Lee >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.0.0-alpha1, 2.10.1 > > Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch > > > TestSeveralNameNodes has been failing in precommit builds. It usually times > out on waiting for the last thread to finish writing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111617#comment-17111617 ] Masatake Iwasaki commented on HDFS-9376: Thanks for pinging me [~Jim_Brennan]. I cherry-picked this to branch-2.10. This was not in branch-2/branch-2.10 because multiple standby NN had not been supported before HDFS-14205. > TestSeveralNameNodes fails occasionally > --- > > Key: HDFS-9376 > URL: https://issues.apache.org/jira/browse/HDFS-9376 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Kihwal Lee >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch > > > TestSeveralNameNodes has been failing in precommit builds. It usually times > out on waiting for the last thread to finish writing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111269#comment-17111269 ] Jim Brennan commented on HDFS-9376: --- [~cnauroth], [~iwasakims], [~kihwal] I know this is a pretty old Jira, but we have seen this failure come up in our internal branch-2.10 builds. I downloaded the patch and verified that it applies cleanly to branch-2.10, builds and runs. Any chance we could get this pulled back to branch-2.10? > TestSeveralNameNodes fails occasionally > --- > > Key: HDFS-9376 > URL: https://issues.apache.org/jira/browse/HDFS-9376 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Kihwal Lee >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch > > > TestSeveralNameNodes has been failing in precommit builds. It usually times > out on waiting for the last thread to finish writing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080780#comment-15080780 ] Masatake Iwasaki commented on HDFS-9376: Thanks, [~cnauroth]. > TestSeveralNameNodes fails occasionally > --- > > Key: HDFS-9376 > URL: https://issues.apache.org/jira/browse/HDFS-9376 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Kihwal Lee >Assignee: Masatake Iwasaki > Fix For: 3.0.0 > > Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch > > > TestSeveralNameNodes has been failing in precommit builds. It usually times > out on waiting for the last thread to finish writing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074397#comment-15074397 ] Hudson commented on HDFS-9376: -- FAILURE: Integrated in Hadoop-trunk-Commit #9035 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9035/]) HDFS-9376. TestSeveralNameNodes fails occasionally. Contributed by (cnauroth: rev 84a81477912644290173518d566b586305b85bf7) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestSeveralNameNodes.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestSeveralNameNodes fails occasionally > --- > > Key: HDFS-9376 > URL: https://issues.apache.org/jira/browse/HDFS-9376 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Kihwal Lee >Assignee: Masatake Iwasaki > Fix For: 3.0.0 > > Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch > > > TestSeveralNameNodes has been failing in precommit builds. It usually times > out on waiting for the last thread to finish writing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073364#comment-15073364
]
Masatake Iwasaki commented on HDFS-9376:
The failure of {{TestReplicationPolicyConsiderLoad}} was already fixed by
HDFS-9597. Other tests are flaky regardless of the patch and succeeded on my
local environment.
> TestSeveralNameNodes fails occasionally
> ---
>
> Key: HDFS-9376
> URL: https://issues.apache.org/jira/browse/HDFS-9376
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9376.001.patch, HDFS-9376.002.patch
>
>
> TestSeveralNameNodes has been failing in precommit builds. It usually times
> out on waiting for the last thread to finish writing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073470#comment-15073470
]
Hadoop QA commented on HDFS-9376:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 56s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 44s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12779734/HDFS-9376.002.patch |
| JIRA Issue | HDFS-9376 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f7babf54ccf1 3.13.0-36-lowlatency
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071486#comment-15071486
]
Hadoop QA commented on HDFS-9376:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 38s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12779494/HDFS-9376.001.patch |
| JIRA Issue | HDFS-9376 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071388#comment-15071388
]
Masatake Iwasaki commented on HDFS-9376:
The failover thread in {{HAStressTestHarness}} will invoke failover
periodically with fixed sleep time. The {{msBetweenFailovers}} is set to 1000
ms for {{TestSeveralNameNodes}}.
{code}
for (int i = 0; i < nns; i++) {
int next = (i + 1) % nns;
...
cluster.transitionToStandby(i);
cluster.transitionToActive(next);
...
Thread.sleep(msBetweenFailovers);
{code}
Retry proxy of client have sleep time exponential to number of retries on
failover. The client is possible to sleep up to around 15 seconds if it
repeatedly fails on the operation. The client may not get enough effective run
time due to this.
{noformat}
2015-12-24 12:22:00,784 [Thread-250] INFO retry.RetryInvocationHandler
(RetryInvocationHandler.java:invoke(147)) - Exception while invoking create of
class ClientNamenodeProtocolTranslatorPB over localhost/127.0.0.1:42201 after 4
fail over attempts. Trying to fail over after sleeping for 10161ms.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category WRITE is not supported in state standby. Visit
https://s.apache.org/sbnn-error
{noformat}
> TestSeveralNameNodes fails occasionally
> ---
>
> Key: HDFS-9376
> URL: https://issues.apache.org/jira/browse/HDFS-9376
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Masatake Iwasaki
>
> TestSeveralNameNodes has been failing in precommit builds. It usually times
> out on waiting for the last thread to finish writing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989886#comment-14989886
]
Kihwal Lee commented on HDFS-9376:
--
This is what we get.
{panel}
Running org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 128.809 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
testCircularLinkedListWrites(org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes)
Time elapsed: 128.646 sec <<< FAILURE!
java.lang.AssertionError: Some writers didn't complete in expected runtime!
Current writer state:\[Circular Writer:
directory: /test-1
target length: 50
current item: 49
done: false
\] expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
{panel}
When I tried following, it passed more reliably, but there might be a better
fix or a more fundamental issue in the core code.
{code}
- private static final long RUNTIME = 10;
+ private static final long RUNTIME = 15;
{code}
> TestSeveralNameNodes fails occasionally
> ---
>
> Key: HDFS-9376
> URL: https://issues.apache.org/jira/browse/HDFS-9376
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>
> TestSeveralNameNodes has been failing in precommit builds. It usually times
> out on waiting for the last thread to finish writing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
