[jira] [Commented] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290657#comment-17290657 ] Brahma Reddy Battula commented on MAPREDUCE-6726: - [~junping_du]/[~srikanth.sampath]/[~jianhe] could you please review this and move forward, this is asked for 3.3.1 release.. > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > MAPREDUCE-6726.005.trunk.patch, TestReport.pdf, WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7324) ClientHSSecurityInfo class is in wrong META-INF file
[
https://issues.apache.org/jira/browse/MAPREDUCE-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290626#comment-17290626
]
Hadoop QA commented on MAPREDUCE-7324:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
55s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
32s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
46m 43s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557588
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 23:17
Start Date: 24/Feb/21 23:17
Worklog Time Spent: 10m
Work Description: amahussein commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582374895
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##
@@ -229,6 +229,31 @@ public static int uniqueSequenceId() {
return sequence.incrementAndGet();
}
+ /**
+ * Creates a directory for the data/logs of the unit test.
+ * It first delete the directory if it exists.
+ *
+ * @param testClass the unit test class.
+ * @param defaultTargetRootDir the directory where the class directory is
+ * created.
+ * @return the Path of the root directory.
+ */
+ public static Path setupTestRootDir(Class testClass,
+ String defaultTargetRootDir) {
+// setup the test root directory
+String targetTestDir =
+System.getProperty(SYSPROP_TEST_DATA_DIR, defaultTargetRootDir);
+Path testRootDirPath =
+new Path(targetTestDir, testClass.getSimpleName());
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
+testRootDirPath.toString());
Review comment:
Also, if I do not set this property, the diff will be very large to get
around it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557588)
Time Spent: 2h (was: 1h 50m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557585
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 23:14
Start Date: 24/Feb/21 23:14
Worklog Time Spent: 10m
Work Description: amahussein commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582373743
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/ClusterMapReduceTestCase.java
##
@@ -43,8 +45,21 @@
* The DFS filesystem is formated before the testcase starts and after it ends.
*/
public abstract class ClusterMapReduceTestCase {
+ private static final String TEST_ROOT_DEFAULT_PATH =
+ System.getProperty("test.build.data", "target/test-dir");
+ private static Path testRootDir;
+
private MiniDFSCluster dfsCluster = null;
- private MiniMRCluster mrCluster = null;
+ private MiniMRClientCluster mrCluster = null;
+
+ protected static void setupClassBase(Class testClass) throws Exception {
+// setup the test root directory
+testRootDir = GenericTestUtils.setupTestRootDir(testClass,
+TEST_ROOT_DEFAULT_PATH);
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
Review comment:
It will be set for the JVM executing the unittest. Each unit test should
then have its own property. Then they should not overlap.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557585)
Time Spent: 1h 50m (was: 1h 40m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557583
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 23:13
Start Date: 24/Feb/21 23:13
Worklog Time Spent: 10m
Work Description: amahussein commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582373195
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##
@@ -229,6 +229,31 @@ public static int uniqueSequenceId() {
return sequence.incrementAndGet();
}
+ /**
+ * Creates a directory for the data/logs of the unit test.
+ * It first delete the directory if it exists.
+ *
+ * @param testClass the unit test class.
+ * @param defaultTargetRootDir the directory where the class directory is
+ * created.
+ * @return the Path of the root directory.
+ */
+ public static Path setupTestRootDir(Class testClass,
+ String defaultTargetRootDir) {
+// setup the test root directory
+String targetTestDir =
+System.getProperty(SYSPROP_TEST_DATA_DIR, defaultTargetRootDir);
+Path testRootDirPath =
+new Path(targetTestDir, testClass.getSimpleName());
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
+testRootDirPath.toString());
Review comment:
There has to be a global parameter so that all the modules will use the
desired folder. The property is being read everywhere by different modules and
this ends up picking different directories.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557583)
Time Spent: 1h 40m (was: 1.5h)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557582
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 23:11
Start Date: 24/Feb/21 23:11
Worklog Time Spent: 10m
Work Description: amahussein commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582372117
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/ClusterMapReduceTestCase.java
##
@@ -43,8 +45,21 @@
* The DFS filesystem is formated before the testcase starts and after it ends.
*/
public abstract class ClusterMapReduceTestCase {
+ private static final String TEST_ROOT_DEFAULT_PATH =
+ System.getProperty("test.build.data", "target/test-dir");
Review comment:
The point that if the property "test.build.data" is set globally to the
unit tests, then the `ClusterMapReduceTestCase.java` should also reads this
value. Otherwise, all unit tests would not consume the global properties set to
the JVM. It will be a mess in the folders
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557582)
Time Spent: 1.5h (was: 1h 20m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7324) ClientHSSecurityInfo class is in wrong META-INF file
[
https://issues.apache.org/jira/browse/MAPREDUCE-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290538#comment-17290538
]
Daryn Sharp commented on MAPREDUCE-7324:
+1. This was a real pain to debug...
> ClientHSSecurityInfo class is in wrong META-INF file
>
>
> Key: MAPREDUCE-7324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7324
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: MAPREDUCE-7324.001.patch
>
>
> {{ClientHSSecurityInfo}} is located in
> {noformat}
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
> {noformat}
> But the actual class exists in
> {noformat}
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
> {noformat}
> Because of this issue, there is an ordering dependency between the
> client-jobclient and client-common that can cause failures if the ordering is
> not correct. Namely, if client-common is in the classpath _after_
> client-jobclient, the JVM won't find {{ClientHSSecurityInfo}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7324) ClientHSSecurityInfo class is in wrong META-INF file
[
https://issues.apache.org/jira/browse/MAPREDUCE-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-7324:
---
Attachment: MAPREDUCE-7324.001.patch
> ClientHSSecurityInfo class is in wrong META-INF file
>
>
> Key: MAPREDUCE-7324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7324
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: MAPREDUCE-7324.001.patch
>
>
> {{ClientHSSecurityInfo}} is located in
> {noformat}
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
> {noformat}
> But the actual class exists in
> {noformat}
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
> {noformat}
> Because of this issue, there is an ordering dependency between the
> client-jobclient and client-common that can cause failures if the ordering is
> not correct. Namely, if client-common is in the classpath _after_
> client-jobclient, the JVM won't find {{ClientHSSecurityInfo}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7324) ClientHSSecurityInfo class is in wrong META-INF file
[
https://issues.apache.org/jira/browse/MAPREDUCE-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-7324:
---
Status: Patch Available (was: Open)
> ClientHSSecurityInfo class is in wrong META-INF file
>
>
> Key: MAPREDUCE-7324
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7324
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: MAPREDUCE-7324.001.patch
>
>
> {{ClientHSSecurityInfo}} is located in
> {noformat}
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
> {noformat}
> But the actual class exists in
> {noformat}
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
> {noformat}
> Because of this issue, there is an ordering dependency between the
> client-jobclient and client-common that can cause failures if the ordering is
> not correct. Namely, if client-common is in the classpath _after_
> client-jobclient, the JVM won't find {{ClientHSSecurityInfo}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7324) ClientHSSecurityInfo class is in wrong META-INF file
Eric Badger created MAPREDUCE-7324:
--
Summary: ClientHSSecurityInfo class is in wrong META-INF file
Key: MAPREDUCE-7324
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7324
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Eric Badger
Assignee: Eric Badger
{{ClientHSSecurityInfo}} is located in
{noformat}
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
{noformat}
But the actual class exists in
{noformat}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
{noformat}
Because of this issue, there is an ordering dependency between the
client-jobclient and client-common that can cause failures if the ordering is
not correct. Namely, if client-common is in the classpath _after_
client-jobclient, the JVM won't find {{ClientHSSecurityInfo}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557572=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557572
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 22:55
Start Date: 24/Feb/21 22:55
Worklog Time Spent: 10m
Work Description: jbrennan333 commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582364631
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/ClusterMapReduceTestCase.java
##
@@ -43,8 +45,21 @@
* The DFS filesystem is formated before the testcase starts and after it ends.
*/
public abstract class ClusterMapReduceTestCase {
+ private static final String TEST_ROOT_DEFAULT_PATH =
+ System.getProperty("test.build.data", "target/test-dir");
Review comment:
Ahh. I see it now. They are different. But I don't think you need to
call getProperty. Just pass "target/test-dir" as the default for
setupTestRootDir.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557572)
Time Spent: 1h 20m (was: 1h 10m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557562
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 22:33
Start Date: 24/Feb/21 22:33
Worklog Time Spent: 10m
Work Description: jbrennan333 commented on a change in pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722#discussion_r582272551
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##
@@ -229,6 +229,31 @@ public static int uniqueSequenceId() {
return sequence.incrementAndGet();
}
+ /**
+ * Creates a directory for the data/logs of the unit test.
+ * It first delete the directory if it exists.
+ *
+ * @param testClass the unit test class.
+ * @param defaultTargetRootDir the directory where the class directory is
+ * created.
+ * @return the Path of the root directory.
+ */
+ public static Path setupTestRootDir(Class testClass,
+ String defaultTargetRootDir) {
+// setup the test root directory
+String targetTestDir =
+System.getProperty(SYSPROP_TEST_DATA_DIR, defaultTargetRootDir);
+Path testRootDirPath =
+new Path(targetTestDir, testClass.getSimpleName());
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
+testRootDirPath.toString());
Review comment:
I'm a little uneasy about overwriting this property. I think I see why
you are doing it, so getTestDir will do the right thing, but I worry that it
might lead to unexpected behavior because getTestDir is used everywhere. At a
minimum, I think the comment on the function should clearly state that this
changes the property.
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##
@@ -245,6 +270,18 @@ public static File getTestDir() {
return dir;
}
+ /**
+ * Cleans-up the root directory from the property
+ * {@link #SYSPROP_TEST_DATA_DIR}.
+ *
+ * @return the absolute file of the test root directory.
+ */
+ public static File clearTestRootDir() {
Review comment:
This is only called from setupTestRootDir(), so I don't think it needs
to be a separate public function. I would just do the fullyDelete() call
in-line in setup TestRootDir.
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/ClusterMapReduceTestCase.java
##
@@ -43,8 +45,21 @@
* The DFS filesystem is formated before the testcase starts and after it ends.
*/
public abstract class ClusterMapReduceTestCase {
+ private static final String TEST_ROOT_DEFAULT_PATH =
+ System.getProperty("test.build.data", "target/test-dir");
+ private static Path testRootDir;
+
private MiniDFSCluster dfsCluster = null;
- private MiniMRCluster mrCluster = null;
+ private MiniMRClientCluster mrCluster = null;
+
+ protected static void setupClassBase(Class testClass) throws Exception {
+// setup the test root directory
+testRootDir = GenericTestUtils.setupTestRootDir(testClass,
+TEST_ROOT_DEFAULT_PATH);
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
Review comment:
I don't think we should set the property here if we are doing it in
setupTestRootDir().
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/ClusterMapReduceTestCase.java
##
@@ -43,8 +45,21 @@
* The DFS filesystem is formated before the testcase starts and after it ends.
*/
public abstract class ClusterMapReduceTestCase {
+ private static final String TEST_ROOT_DEFAULT_PATH =
+ System.getProperty("test.build.data", "target/test-dir");
+ private static Path testRootDir;
+
private MiniDFSCluster dfsCluster = null;
- private MiniMRCluster mrCluster = null;
+ private MiniMRClientCluster mrCluster = null;
+
+ protected static void setupClassBase(Class testClass) throws Exception {
+// setup the test root directory
+testRootDir = GenericTestUtils.setupTestRootDir(testClass,
+TEST_ROOT_DEFAULT_PATH);
+System.setProperty(GenericTestUtils.SYSPROP_TEST_DATA_DIR,
Review comment:
One question is whether we should get the original value and then reset
it in an afterClass method? Is every unit test run in a separate jvm instance?
Wouldn't want a following test to get the wrong setting.
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##
@@ -229,6 +229,31 @@ public static int uniqueSequenceId() {
return sequence.incrementAndGet();
}
+ /**
+ * Creates a
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557213
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 18:17
Start Date: 24/Feb/21 18:17
Worklog Time Spent: 10m
Work Description: amahussein opened a new pull request #2722:
URL: https://github.com/apache/hadoop/pull/2722
[MAPREDUCE-7320: ClusterMapReduceTestCase does not clean
directories](https://issues.apache.org/jira/browse/MAPREDUCE-7320)
Running Junits that extend ClusterMapReduceTestCase generate lots of
directories and folders all over the place.
This PR addresses organizing the directories generated by the unit test,
cleaning them up at the beginning of the execution if necessary.
- It touches `MiniYARNCluster.java` in order to change the base dir to
`target/test-dir/$TEST_CLASS_NAME`
- test classes affected
- TestMRJobClient,
- TestStreamingBadRecords,
- TestClusterMapReduceTestCase,
- TestBadRecords.
- TestMRCJCJobClient,
- TestJobName
I tested the TestUnits that use `MiniYARNCluster.java` such as:
```bash
Class
MiniYARNCluster
TestOSSMiniYarnCluster (3 usages found)
TestMRTimelineEventHandling (4 usages found)
TestJobHistoryEventHandler (3 usages found)
TestHadoopArchiveLogs (3 usages found)
TestHadoopArchiveLogsRunner (3 usages found)
TestDynamometerInfra (3 usages found)
TestDSTimelineV10
TestDSTimelineV20
TestDSTimelineV15
TestUnmanagedAMLauncher (3 usages found)
TestApplicationMasterServiceProtocolForTimelineV2
TestFederationRMFailoverProxyProvider (3 usages found)
TestHedgingRequestRMFailoverProxyProvider (4 usages found)
TestNoHaRMFailoverProxyProvider (5 usages found)
TestRMFailover (4 usages found)
TestAMRMClient
TestAMRMClientPlacementConstraints
TestAMRMProxy (5 usages found)
TestNMClient (3 usages found)
TestOpportunisticContainerAllocationE2E (3 usages found)
TestYarnClient (3 usages found)
TestYarnClientWithReservation (12 usages found)
TestYarnCLI (7 usages found)
TestContainerManagerSecurity (2 usages found)
TestDiskFailures (2 usages found)
TestMiniYarnCluster (9 usages found)
TestMiniYARNClusterForHA (2 usages found)
TestMiniYarnClusterNodeUtilization (3 usages found)
TestEncryptedShuffle
```
MAPREDUCE-7320. fix TestEncryptedShuffle paths
MAPREDUCE-7320. move cleaning up root directory to setup
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557213)
Time Spent: 1h (was: 50m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?focusedWorklogId=557191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-557191
]
ASF GitHub Bot logged work on MAPREDUCE-7320:
-
Author: ASF GitHub Bot
Created on: 24/Feb/21 18:00
Start Date: 24/Feb/21 18:00
Worklog Time Spent: 10m
Work Description: amahussein closed pull request #2705:
URL: https://github.com/apache/hadoop/pull/2705
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 557191)
Time Spent: 50m (was: 40m)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated MAPREDUCE-7320:
---
Flags: (was: Patch)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated MAPREDUCE-7320:
---
Flags: Patch
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
