[jira] [Commented] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286276#comment-17286276 ] Hadoop QA commented on MAPREDUCE-6726: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 35m 50s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 9s{color} | {color:red}{color} | {color:red} Unprocessed flag(s): --findbugs-strict-precheck {color} | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/57/artifact/out/Dockerfile | | JIRA Issue | MAPREDUCE-6726 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13020600/MAPREDUCE-6726.005.trunk.patch | | Console output | https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/57/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > MAPREDUCE-6726.005.trunk.patch, TestReport.pdf, WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated MAPREDUCE-6726: Attachment: MAPREDUCE-6726.005.trunk.patch > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > MAPREDUCE-6726.005.trunk.patch, TestReport.pdf, WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated MAPREDUCE-6726: Attachment: TestReport.pdf > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > TestReport.pdf, WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=553981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-553981 ] ASF GitHub Bot logged work on MAPREDUCE-7320: - Author: ASF GitHub Bot Created on: 18/Feb/21 00:05 Start Date: 18/Feb/21 00:05 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2705: URL: https://github.com/apache/hadoop/pull/2705#issuecomment-780936238 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 14m 49s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 8 new or modified test files. | _ trunk Compile Tests _ | | +0 :ok: | mvndep | 14m 0s | | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 54s | | trunk passed | | +1 :green_heart: | compile | 21m 44s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 17m 54s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | | +1 :green_heart: | checkstyle | 3m 46s | | trunk passed | | +1 :green_heart: | mvnsite | 2m 15s | | trunk passed | | +1 :green_heart: | shadedclient | 19m 47s | | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 52s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | | +0 :ok: | spotbugs | 26m 18s | | Both FindBugs and SpotBugs are enabled, using SpotBugs. | | +0 :ok: | spotbugs | 0m 45s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no spotbugs output file (spotbugsXml.xml) | _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 15s | | the patch passed | | +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 19m 51s | | root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1956 unchanged - 3 fixed = 1956 total (was 1959) | | +1 :green_heart: | compile | 17m 45s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | | +1 :green_heart: | javac | 17m 45s | | root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1850 unchanged - 3 fixed = 1850 total (was 1853) | | +1 :green_heart: | checkstyle | 3m 43s | | root: The patch generated 0 new + 135 unchanged - 3 fixed = 135 total (was 138) | | +1 :green_heart: | mvnsite | 2m 12s | | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 13s | | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 55s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | | +0 :ok: | spotbugs | 0m 40s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests has no data from spotbugs | | -1 :x: | spotbugs | 1m 0s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2705/1/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient cannot run computeBugHistory from spotbugs | | -1 :x: | spotbugs | 1m 0s | [/patch-spotbugs-hadoop-tools_hadoop-streaming.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2705/1/artifact/out/patch-spotbugs-hadoop-tools_hadoop-streaming.txt) | hadoop-tools/hadoop-streaming cannot run computeBugHistory from spotbugs | _ Other Tests _ | | +1 :green_heart: | unit | 3m 24s | | hadoop-yarn-server-tests in the patch passed. | | -1 :x: | unit |
[jira] [Commented] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286110#comment-17286110 ] Hadoop QA commented on MAPREDUCE-6726: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color} | {color:red}{color} | {color:red} MAPREDUCE-6726 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | MAPREDUCE-6726 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13020586/MAPREDUCE-6726.004.patch | | Console output | https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/55/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated MAPREDUCE-6726: Fix Version/s: 3.2.2 > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated MAPREDUCE-6726: Attachment: MAPREDUCE-6726.004.patch > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated MAPREDUCE-7320: -- Labels: pull-request-available (was: ) > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286079#comment-17286079 ] Ahmed Hussein commented on MAPREDUCE-7320: -- This fix is cleaning up the clusters directories after finishing the unit tests. * It touches {{MiniYARNCluster.java}} in order to change the base dir to {{target/test-dir/$TEST_CLASS_NAME}} * test classes affected ** TestMRJobClient, ** TestStreamingBadRecords, ** TestClusterMapReduceTestCase, ** TestBadRecords. ** TestMRCJCJobClient, ** TestJobName I tested the TestUnits that use {{MiniYARNCluster.java}} such as: {code:bash} Class MiniYARNCluster TestOSSMiniYarnCluster (3 usages found) TestMRTimelineEventHandling (4 usages found) TestJobHistoryEventHandler (3 usages found) TestHadoopArchiveLogs (3 usages found) TestHadoopArchiveLogsRunner (3 usages found) TestDynamometerInfra (3 usages found) TestDSTimelineV10 TestDSTimelineV20 TestDSTimelineV15 TestUnmanagedAMLauncher (3 usages found) TestApplicationMasterServiceProtocolForTimelineV2 TestFederationRMFailoverProxyProvider (3 usages found) TestHedgingRequestRMFailoverProxyProvider (4 usages found) TestNoHaRMFailoverProxyProvider (5 usages found) TestRMFailover (4 usages found) TestAMRMClient TestAMRMClientPlacementConstraints TestAMRMProxy (5 usages found) TestNMClient (3 usages found) TestOpportunisticContainerAllocationE2E (3 usages found) TestYarnClient (3 usages found) TestYarnClientWithReservation (12 usages found) TestYarnCLI (7 usages found) TestContainerManagerSecurity (2 usages found) TestDiskFailures (2 usages found) TestMiniYarnCluster (9 usages found) TestMiniYARNClusterForHA (2 usages found) TestMiniYarnClusterNodeUtilization (3 usages found) {code} > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7320: Assignee: Ahmed Hussein > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=553793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-553793 ] ASF GitHub Bot logged work on MAPREDUCE-7320: - Author: ASF GitHub Bot Created on: 17/Feb/21 18:45 Start Date: 17/Feb/21 18:45 Worklog Time Spent: 10m Work Description: amahussein opened a new pull request #2705: URL: https://github.com/apache/hadoop/pull/2705 [MAPREDUCE-7320: ClusterMapReduceTestCase does not clean directories](https://issues.apache.org/jira/browse/MAPREDUCE-7320) Running Junits that extend ClusterMapReduceTestCase generate lots of directories and folders without cleaning them up. This PR addresses cleaning up the clusters directories after finishing the unit tests. - It touches `MiniYARNCluster.java` in order to change the base dir to `target/test-dir/$TEST_CLASS_NAME` - test classes affected - TestMRJobClient, - TestStreamingBadRecords, - TestClusterMapReduceTestCase, - TestBadRecords. TestMRCJCJobClient, - TestJobName I tested the TestUnits that use `MiniYARNCluster.java` such as: ```bash Class MiniYARNCluster TestOSSMiniYarnCluster (3 usages found) TestMRTimelineEventHandling (4 usages found) TestJobHistoryEventHandler (3 usages found) TestHadoopArchiveLogs (3 usages found) TestHadoopArchiveLogsRunner (3 usages found) TestDynamometerInfra (3 usages found) TestDSTimelineV10 TestDSTimelineV20 TestDSTimelineV15 TestUnmanagedAMLauncher (3 usages found) TestApplicationMasterServiceProtocolForTimelineV2 TestFederationRMFailoverProxyProvider (3 usages found) TestHedgingRequestRMFailoverProxyProvider (4 usages found) TestNoHaRMFailoverProxyProvider (5 usages found) TestRMFailover (4 usages found) TestAMRMClient TestAMRMClientPlacementConstraints TestAMRMProxy (5 usages found) TestNMClient (3 usages found) TestOpportunisticContainerAllocationE2E (3 usages found) TestYarnClient (3 usages found) TestYarnClientWithReservation (12 usages found) TestYarnCLI (7 usages found) TestContainerManagerSecurity (2 usages found) TestDiskFailures (2 usages found) TestMiniYarnCluster (9 usages found) TestMiniYARNClusterForHA (2 usages found) TestMiniYarnClusterNodeUtilization (3 usages found) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 553793) Remaining Estimate: 0h Time Spent: 10m > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7320 started by Ahmed Hussein. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285715#comment-17285715 ] Bilwa S T commented on MAPREDUCE-6749: -- Attached test report for this feature. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated MAPREDUCE-6749: - Attachment: Container Reuse Performance Report.pdf > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org