[jira] [Commented] (MAPREDUCE-7259) testSpeculateSuccessfulWithUpdateEvents fails Intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024764#comment-17024764
]
Hadoop QA commented on MAPREDUCE-7259:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 54 unchanged - 5 fixed = 54 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
15s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 56s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
44s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}217m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | MAPREDUCE-7259 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991946/MAPREDUCE-7259.005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7830b364aabe 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f40e66 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262-branch-2.10.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7262-branch-2.10.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
>
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated MAPREDUCE-7262:
--
Fix Version/s: 3.2.2
3.1.4
3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
+1. Thanks, [~ahussein]. I committed this to trunk, branch-3.2, and branch-3.1.
This patch doesn't apply to branch-2.10 due to the Logger vs Log api. With a
small change, this patch could apply to the branch-2.10.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024717#comment-17024717
]
Hudson commented on MAPREDUCE-7262:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17902 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17902/])
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) (jeagles: rev
3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024686#comment-17024686
]
Hadoop QA commented on MAPREDUCE-7262:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
46s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | MAPREDUCE-7262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991950/MAPREDUCE-7262.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 617687e95c3a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f40e66 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7727/testReport/ |
| Max. process+thread count | 582 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7727/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> MRApp helpers block for long intervals (500ms)
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024677#comment-17024677
]
Hadoop QA commented on MAPREDUCE-7262:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 0 new + 87 unchanged - 6 fixed = 87 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
24s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | MAPREDUCE-7262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991949/MAPREDUCE-7261.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 93e49da8099e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f40e66 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7726/testReport/ |
| Max. process+thread count | 583 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024642#comment-17024642
]
Jonathan Turner Eagles commented on MAPREDUCE-7262:
---
002 patch looks good. +1. Pending on Hadoop QA to catch what I may have missed.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
>
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: (was: MAPREDUCE-7261.002.patch)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Open (was: Patch Available)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Patch Available (was: Open)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7261.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
>
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024627#comment-17024627
]
Ahmed Hussein commented on MAPREDUCE-7262:
--
{quote}waitForState(TaskAttempt attempt, TaskAttemptState...finalStates) was
not updated. I assume this is because we assume MAPREDUCE-7259 will remove this
function. Can you confirm? If it turns out MAPREDUCE-7259 will continue to use
this, it will need to be updated.{quote}
Yes, MAPREDUCE-7259 will remove the method.
{quote}public void waitForState(Task task, TaskState finalState)
Seems to have a few problems. One is that it continues to use
System.out.println. Can you comment on that?.t{quote}
My bad..I did not remove that print statement. I will update the patch.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
>
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024621#comment-17024621
]
Jonathan Turner Eagles commented on MAPREDUCE-7262:
---
[~ahussein], Looks like a good fix. Couple of small things to clarify.
- public void waitForState(Task task, TaskState finalState)
Seems to have a few problems. One is that it continues to use
System.out.println. Can you comment on that?.
Also, it breaks the while loop pattern where it gets the updated "while
condition state" before sleeping. Seems a bit wasteful.
- waitForState(TaskAttempt attempt, TaskAttemptState...finalStates) was not
updated. I assume this is because we assume MAPREDUCE-7259 will remove this
function. Can you confirm? If it turns out MAPREDUCE-7259 will continue to use
this, it will need to be updated.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
>
[jira] [Updated] (MAPREDUCE-7259) testSpeculateSuccessfulWithUpdateEvents fails Intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7259:
-
Attachment: MAPREDUCE-7259.005.patch
> testSpeculateSuccessfulWithUpdateEvents fails Intermittently
> --
>
> Key: MAPREDUCE-7259
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7259
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7259.001.patch, MAPREDUCE-7259.002.patch,
> MAPREDUCE-7259.003.patch, MAPREDUCE-7259.004.patch, MAPREDUCE-7259.005.patch
>
>
> {{TestSpeculativeExecutionWithMRApp.testSpeculateSuccessfulWithUpdateEvents}}
> fails Intermittently with the exponential estimator. The problem happens
> because assertion fails waiting for the MRApp to stop.
> There maybe a need to redesign the test case because it does not work very
> well because of the racing and the timing between the speculator and the
> tasks. It works fine for the legacy estimator because the estimate is based
> on start-end rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7259) testSpeculateSuccessfulWithUpdateEvents fails Intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024547#comment-17024547
]
Ahmed Hussein commented on MAPREDUCE-7259:
--
[~jeagles], I moved the optimizations of the MRApp methods into a different
Jira (MAPREDUCE-7262). Changing the intervals of polling the App/Task status
reduces the elapsed time of test cases significantly.
> testSpeculateSuccessfulWithUpdateEvents fails Intermittently
> --
>
> Key: MAPREDUCE-7259
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7259
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7259.001.patch, MAPREDUCE-7259.002.patch,
> MAPREDUCE-7259.003.patch, MAPREDUCE-7259.004.patch
>
>
> {{TestSpeculativeExecutionWithMRApp.testSpeculateSuccessfulWithUpdateEvents}}
> fails Intermittently with the exponential estimator. The problem happens
> because assertion fails waiting for the MRApp to stop.
> There maybe a need to redesign the test case because it does not work very
> well because of the racing and the timing between the speculator and the
> tasks. It works fine for the legacy estimator because the estimate is based
> on start-end rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024537#comment-17024537
]
Hadoop QA commented on MAPREDUCE-7262:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
59s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | MAPREDUCE-7262 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991932/MAPREDUCE-7262.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux dded2de9faed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f40e66 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7724/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7724/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> MRApp helpers block for long intervals (500ms)
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024502#comment-17024502
]
Ahmed Hussein commented on MAPREDUCE-7262:
--
The table in MAPREDUCE-7262-elapsedTimes.pdf shows the ratio between elapsed
times of the affected JUnit tests before and after applying the patch.
Some test cases like {{TestRecovery}} and {{TestMRApp}} execution times are
reduce by 50%.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262.001.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
>
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024500#comment-17024500
]
Hadoop QA commented on MAPREDUCE-7262:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} MAPREDUCE-7262 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-7262 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7723/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262-elapsedTimes.pdf
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
>
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Patch Available (was: Open)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> org.apache.hadoop.mapreduce.v2.app (67 usages found)
> TestAMInfos (1 usage
