[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577609#comment-16577609 ] Íñigo Goiri commented on HADOOP-15475: -- One that should be easy to fix is [TestDatanodeStartupFixesLegacyStorageIDs|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.hdfs/TestDatanodeStartupFixesLegacyStorageIDs/]. It looks like a path issue from the log. [~huanbang1993], [~surmountian], can any of you take a look? > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577606#comment-16577606 ] Íñigo Goiri commented on HADOOP-15475: -- Stopped for a while working on this but I checked today and we have potential to easily fix around 50 tests. The current progress is: ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| |[483|https://builds.apache.org/job/hadoop-trunk-win/483/testReport/]|2018/05/30|19390|1473|348|17569| |[484|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/05/31|19390|1473|288|17629| |[485|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/06/01|19173|1473|270|17430| |[489|https://builds.apache.org/job/hadoop-trunk-win/489/testReport/]|2018/06/05|19394|1482|242|17670| |[495|https://builds.apache.org/job/hadoop-trunk-win/495/testReport/]|2018/06/11|19332|1482|213|17637| |[508|https://builds.apache.org/job/hadoop-trunk-win/508/testReport/]|2018/06/24|19247|1506|189|17552| |[508|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/]|2018/08/04|19508|1500|186|17822| The ones we could target are: * [TestServiceManager|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.yarn.service/TestServiceManager/] * [TestYarnNativeServices|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.yarn.service/TestYarnNativeServices/] * [TestServiceCLI|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.yarn.service.client/TestServiceCLI/] * [TestContainerManager|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.yarn.server.nodemanager.containermanager/TestContainerManager/] * [TestContainerLaunch|https://builds.apache.org/job/hadoop-trunk-win/548/testReport/org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher/TestContainerLaunch/] > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522534#comment-16522534 ] Íñigo Goiri commented on HADOOP-15475: -- ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| |[483|https://builds.apache.org/job/hadoop-trunk-win/483/testReport/]|2018/05/30|19390|1473|348|17569| |[484|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/05/31|19390|1473|288|17629| |[485|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/06/01|19173|1473|270|17430| |[489|https://builds.apache.org/job/hadoop-trunk-win/489/testReport/]|2018/06/05|19394|1482|242|17670| |[495|https://builds.apache.org/job/hadoop-trunk-win/495/testReport/]|2018/06/11|19332|1482|213|17637| |[508|https://builds.apache.org/job/hadoop-trunk-win/508/testReport/]|2018/06/24|19247|1506|189|17552| > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509849#comment-16509849 ] Íñigo Goiri commented on HADOOP-15475: -- I committed YARN-8359 and we cut the failures a bunch. There are also a couple smaller JIRAs that went in. ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| |[483|https://builds.apache.org/job/hadoop-trunk-win/483/testReport/]|2018/05/30|19390|1473|348|17569| |[484|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/05/31|19390|1473|288|17629| |[485|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/06/01|19173|1473|270|17430| |[489|https://builds.apache.org/job/hadoop-trunk-win/489/testReport/]|2018/06/05|19394|1482|242|17670| |[495|https://builds.apache.org/job/hadoop-trunk-win/495/testReport/]|2018/06/11|19332|1482|213|17637| Right now we are at 213 failures but I've seen it as small as 202 in build [494|https://builds.apache.org/job/hadoop-trunk-win/494/testReport/]. Today we should be committing MAPREDUCE-7108 which would take us <200. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503607#comment-16503607 ] Íñigo Goiri commented on HADOOP-15475: -- bq. One thing which would be good would be for some official ASF windows libs to ship. I do the set on github, but that's not quite the same as the ASF blessing. Even without all the tests passing, we can at least include the native binaries as signed artifacts I guess that now that we switched the way winutils are built, we should be able to build them from Linux, right? In that case, it should be fairly easy for the release manager to add the windows native libs. For the progress, we cut a bunch and we are pretty much caught up (242 in trunk) with our internal branch-2.9 runs (~230 in our internal run). The status right now is: ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| |[483|https://builds.apache.org/job/hadoop-trunk-win/483/testReport/]|2018/05/30|19390|1473|348|17569| |[484|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/05/31|19390|1473|288|17629| |[485|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/06/01|19173|1473|270|17430| |[489|https://builds.apache.org/job/hadoop-trunk-win/489/testReport/]|2018/06/05|19394|1482|242|17670| In HDFS we have fixed a bunch by using the randomized path per test case but we haven't really fixed the root causes. There is still a bunch of timeouts which we think are because of poll0 being very slow at the beginning. For MapReduce there are a bunch that should be fixable and we can cut the number substantially. In YARN, there a lot of failures but the fixes are not trivial (we still have the Linux tests ignore pending from YARN-8359). > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501946#comment-16501946 ] Steve Loughran commented on HADOOP-15475: - the graph is going in the right direction. One thing which would be good would be for some official ASF windows libs to ship. I do the set on [github|https://github.com/steveloughran/winutils], but that's not quite the same as the ASF blessing. Even without all the tests passing, we can at least include the native binaries as signed artifacts > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Affects Versions: 3.1.0, 2.9.1 >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498189#comment-16498189 ] Íñigo Goiri commented on HADOOP-15475: -- A couple more runs: ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| |[483|https://builds.apache.org/job/hadoop-trunk-win/483/testReport/]|2018/05/30|19390|1473|348|17569| |[484|https://builds.apache.org/job/hadoop-trunk-win/484/testReport/]|2018/05/31|19390|1473|288|17629| The one with 288 failures had HDFS-13632 and HDFS-13629 but I also think it was specially good, we may go up in the following days with flaky test. Right now there are 86 failed HDFS tests and there's nothing specially big other than the ones for TestReconstructStripedFile (all of those could use randomized paths for the MiniDFSCluster). There is also TestMetadataStore HDFS-13516 with a few. I think at this point we can cleanup YARN and MapReduce failures a lot. We are still pending YARN-8359 which should cut a bunch. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493803#comment-16493803 ] Íñigo Goiri commented on HADOOP-15475: -- A couple more builds: ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| Last week we resolved a couple JIRAs with a lot of failures in them so we reduced by more than 50. >From build #479 to #481 there was an increase in failed unit tests but I think >it's because of flaky unit tests that we should eventually fix too. We have been mostly focusing on HDFS so far and now we should try to target commons, YARN and MapReduce more. For YARN there are a lot of tests that we are Linux specific and we are thinking to disable in YARN-8359. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489589#comment-16489589 ] Íñigo Goiri commented on HADOOP-15475: -- bq. So you are seeing 239 failures on branch-2, but Jenkins is at 446? 239 for branch-2.9 on VSTS using Azure builders and 443 for trunk on Jenkins. They are not fully comparable though; this was more like a reference. We started with +700 on branch-2.9 and +800 for trunk. Once we narrow this down a little more (<100 in VSTS) we can go over the differences. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489521#comment-16489521 ] Steve Loughran commented on HADOOP-15475: - So you are seeing 239 failures on branch-2, but Jenkins is at 446? > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[
https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489337#comment-16489337
]
Íñigo Goiri commented on HADOOP-15475:
--
{quote}My ASF volunteer time is almost exclusively for other projects. As a
result, Hadoop builds are pretty much off my radar entirely. Sorry.
{quote}
Sure, I'll work with [~chris.douglas] on checking the build.
Feel free to provide feedback in this thread too.
Funny, enough the daily Windows build worked
[yesterday|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/].
This includes a couple big fixes like HDFS-13586 which reduced the number of
failed unit tests from 516 to 446.
Internally our errors in branch-2.9 on Windows are 239.
The evolution is as follows:
||Run||Date||Total||Skipped||Failed||Passed||
|[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075|
|[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734|
|[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240|
|[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272|
|[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388|
|[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436|
The unit tests with the most failures at this point are:
* TestHDFSFileSystemContract 44 failures: issues and deleting paths, I'm
guessing a failed unit test locks it
* TestAuditLoggerWithCommands 40 failures: I think this must be line separator
on Windows
* TestRawLocalFileSystemContract 21 failures: related to the path
* TestMetadataStore 18 failures
> Fix broken unit tests on Windows
>
>
> Key: HADOOP-15475
> URL: https://issues.apache.org/jira/browse/HADOOP-15475
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Anbang Hu
>Assignee: Anbang Hu
>Priority: Minor
>
> There are hundreds of unit tests that fail on Windows. This JIRA tracks the
> effort to fix them.
> The main reasons for unit test failures on Windows are:
> * Windows/Linux path formats (e.g., HDFS-10256).
> * Line separator.
> * Locked files: Windows locks files when opening them.
> ** The typical trigger is not cleaning MiniDFSCluster leaves files locked
> when a test times out; they need to be cleaned using After.
> * Memory lock size.
> * Slow DNS resolution (e.g., HDFS-13569).
> * Locked ports (e.g., HDFS-11700)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488140#comment-16488140 ] Allen Wittenauer commented on HADOOP-15475: --- My ASF volunteer time is almost exclusively for other projects. As a result, Hadoop builds are pretty much off my radar entirely. Sorry. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487843#comment-16487843 ] Íñigo Goiri commented on HADOOP-15475: -- [~aw], any insights on what's wrong with the daily Windows build? We used to see the UnsatisfiedLinkError in around 30% of the builds but now is almost 100%. I specially want to make sure we didn't break it with one of the fixes. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[
https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486563#comment-16486563
]
Íñigo Goiri commented on HADOOP-15475:
--
The number of unit tests failed in the last 6 builds are >5k.
The error seems to be with the native code.
One example
[here|https://builds.apache.org/job/hadoop-trunk-win/475/testReport/].
{code}
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:640)
at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:1220)
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:667)
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:620)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:371)
{code}
This is an issue that has been coming and going but seems like it's been there
for a while.
Not sure the real reason.
> Fix broken unit tests on Windows
>
>
> Key: HADOOP-15475
> URL: https://issues.apache.org/jira/browse/HADOOP-15475
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Anbang Hu
>Assignee: Anbang Hu
>Priority: Minor
>
> There are hundreds of unit tests that fail on Windows. This JIRA tracks the
> effort to fix them.
> The main reasons for unit test failures on Windows are:
> * Windows/Linux path formats (e.g., HDFS-10256).
> * Line separator.
> * Locked files: Windows locks files when opening them.
> ** The typical trigger is not cleaning MiniDFSCluster leaves files locked
> when a test times out; they need to be cleaned using After.
> * Memory lock size.
> * Slow DNS resolution (e.g., HDFS-13569).
> * Locked ports (e.g., HDFS-11700)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479649#comment-16479649 ] Giovanni Matteo Fumarola commented on HADOOP-15475: --- Thanks [~huanbang1993] and [~elgoiri] for taking care of those. Let's sync offline to fix the one in YARN side as well. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479496#comment-16479496 ] Íñigo Goiri commented on HADOOP-15475: -- In the [current Windows daily build|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/], the top offenders are: * [TestAuditLoggerWithCommands|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.hdfs.server.namenode/TestAuditLoggerWithCommands/] 40 failures * [TestRawLocalFileSystemContract|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.fs/TestRawLocalFileSystemContract/] 21 failures * [TestMetadataStore|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.ozone/TestMetadataStore/] 18 failures * [TestOpenFilesWithSnapshot|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.hdfs.server.namenode.snapshot/TestOpenFilesWithSnapshot/] 14 failures * [TestErasureCodingExerciseAPIs|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.hdfs/TestErasureCodingExerciseAPIs/] 13 failures * [TestQuorumJournalManager|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.hdfs.qjournal.client/TestQuorumJournalManager/] 12 failures * [TestContainerLaunch|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher/TestContainerLaunch/] 11 failures * [TestFileOutputCommitter|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.mapreduce.lib.output/TestFileOutputCommitter/] 10 failures * [TestReconstructStripedFile|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/org.apache.hadoop.hdfs/TestReconstructStripedFile/] 10 failures Fixing these, we should potentially reduce 149 failures. There are a lot more, but we should prioritize these ones. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479418#comment-16479418 ] Íñigo Goiri commented on HADOOP-15475: -- CC [~ste...@apache.org] and [~giovanni.fumarola] for awareness. This is not fully related to the effort in HADOOP-15461 but somewhat. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
