[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514175#comment-16514175 ] Hudson commented on HDFS-13673: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14435 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14435/]) HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming (inigoiri: rev 43d994e4a6dfd1c24eafb909d6f8a0663b20769a) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514131#comment-16514131 ] Íñigo Goiri commented on HDFS-13673: Let's go with this one for now and we can figure the flaky one later. +1 on [^HDFS-13673.001.patch]. Committing. > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512768#comment-16512768 ] Íñigo Goiri commented on HDFS-13673: Yes, I have been seeing it in the past. However, I think it would be good to fix it before committing this JIRA to make sure. There has been a bunch of JIRAs in the past for this: HDFS-2434, HDFS-3812, HDFS-4338, HDFS-4355 Actually, HDFS-6126 is still open. Can you check if you cna find why it is flaky? If so, we may be able to use HDFS-6126. > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512744#comment-16512744 ] Zuoming Zhang commented on HDFS-13673: -- So you mean testCorruptBlock is another issue? Just double confirm to make sure it's not caused by my patch. [~elgoiri] > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512741#comment-16512741 ] Íñigo Goiri commented on HDFS-13673: Can you try to fix testCorruptBlock too? It has been flaky lately. > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512720#comment-16512720 ] Zuoming Zhang commented on HDFS-13673: -- Interesting, for the error in TestNameNodeMetrics.testCorruptBlock, it doesn't seem to be related to my change, since the test has first passed in my previous patch, and the new patch is no difference in logic. I think the TestNameNodeMetrics.testCorruptBlock itself has random failure since it sometimes fails on my local machine. > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511972#comment-16511972
]
genericqa commented on HDFS-13673:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
35s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
22s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13673 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927752/HDFS-13673.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2b1f6585cade 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7547740 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24436/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24436/testReport/ |
| Max. process+thread count | 3287 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511864#comment-16511864 ] Zuoming Zhang commented on HDFS-13673: -- [~elgoiri] Uploaded > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Assignee: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, HDFS-13673.001.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt, > TestNameNodeMetrics-testVolumeFailures-Report.001.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511858#comment-16511858 ] Íñigo Goiri commented on HDFS-13673: bq. This error doesn't seem to be related to mine, any idea? No, it's not, this is happening in all JIRAs today. Can you do the var extract I mentioned before? > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511843#comment-16511843 ] Zuoming Zhang commented on HDFS-13673: -- [~elgoiri] This error doesn't seem to be related to mine, any idea? > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511790#comment-16511790
]
genericqa commented on HDFS-13673:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
22s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
11s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestFileCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13673 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927714/HDFS-13673.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 32fa9ceab2ef 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7566e0e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24432/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24432/testReport/ |
| Max. process+thread count | 3089 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24432/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511689#comment-16511689
]
Íñigo Goiri commented on HDFS-13673:
bq. What do you mean to extract the variable? I think this is only used once in
this file?
Just a code style nit to avoid massive one liners:
{code}
File storageDir = new File(dataDir, Storage.STORAGE_DIR_CURRENT);
DataNodeTestUtils.injectDataDirFailure(storageDir);
{code}
> TestNameNodeMetrics fails on Windows
>
>
> Key: HDFS-13673
> URL: https://issues.apache.org/jira/browse/HDFS-13673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Zuoming Zhang
>Priority: Minor
> Labels: Windows
> Fix For: 3.1.0, 2.9.1
>
> Attachments: HDFS-13673.000.patch,
> TestNameNodeMetrics-testVolumeFailures-Report.000.txt
>
>
> _TestNameNodeMetrics_ fails on Windows
>
> Problem:
> This is because in _testVolumeFailures_, it tries to call
> _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What
> _injectDataDirFailure_does is actually modifying the folder name from
> _volume_name_ to _volume_name_._origin_ and create a new file named as
> _volume_name_. Inside the folder, it has two things: 1. a directory named as
> "_current_", 2. a file named as "_in_use.lock_". Windows behaves different
> from Linux when renaming the parent folder of a locked file. Windows prevent
> you from renaming while Linux allows.
> Fix:
> So in order to inject data failure on to the volume. Instead of renaming the
> volume folder itself. Rename the folder inside it which doesn't hold a lock.
> Since the folder inside the volume is "_current_". Then we only need to
> inject data failure to _volume_name/current_.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511682#comment-16511682 ] Zuoming Zhang commented on HDFS-13673: -- Thanks [~elgoiri] Answers to your questions: * Nope. Actually for other places that are calling _DataNodeTestUtils.injectDataDirFailure_, they are not calling on the volume folder itself. So not affected by _in_use.lock_. I've also checked with all other tests that call the _injectDataDirFailure_, and the tests are not failing. * What do you mean to extract the variable? I think this is only used once in this file? > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13673) TestNameNodeMetrics fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511673#comment-16511673 ] Íñigo Goiri commented on HDFS-13673: The fix looks good. A couple comments: * Is there any other test affected by the same with a similar fix? * Can we extract the variable? > TestNameNodeMetrics fails on Windows > > > Key: HDFS-13673 > URL: https://issues.apache.org/jira/browse/HDFS-13673 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Zuoming Zhang >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-13673.000.patch, > TestNameNodeMetrics-testVolumeFailures-Report.000.txt > > > _TestNameNodeMetrics_ fails on Windows > > Problem: > This is because in _testVolumeFailures_, it tries to call > _DataNodeTestUtils.injectDataDirFailure_ on a volume folder. What > _injectDataDirFailure_does is actually modifying the folder name from > _volume_name_ to _volume_name_._origin_ and create a new file named as > _volume_name_. Inside the folder, it has two things: 1. a directory named as > "_current_", 2. a file named as "_in_use.lock_". Windows behaves different > from Linux when renaming the parent folder of a locked file. Windows prevent > you from renaming while Linux allows. > Fix: > So in order to inject data failure on to the volume. Instead of renaming the > volume folder itself. Rename the folder inside it which doesn't hold a lock. > Since the folder inside the volume is "_current_". Then we only need to > inject data failure to _volume_name/current_. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
