[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated MAPREDUCE-7262: -- Fix Version/s: 2.10.1 > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: MAPREDUCE-7262-branch-2.10.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch, > MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEven
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: MAPREDUCE-7262-branch-2.10.002.patch > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7262-branch-2.10.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch, > MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEve
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated MAPREDUCE-7262: -- Fix Version/s: 3.2.2 3.1.4 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) +1. Thanks, [~ahussein]. I committed this to trunk, branch-3.2, and branch-3.1. This patch doesn't apply to branch-2.10 due to the Logger vs Log api. With a small change, this patch could apply to the branch-2.10. > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7262-elapsedTimes.pdf, > MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usag
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: (was: MAPREDUCE-7261.002.patch) > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7262-elapsedTimes.pdf, > MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(),
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Status: Open (was: Patch Available) > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch, > MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: MAPREDUCE-7262.002.patch > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch, > MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Status: Patch Available (was: Open) > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch, > MAPREDUCE-7262.002.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAtt
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: MAPREDUCE-7261.002.patch > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.002.patch, > MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(), > TaskAttem
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: MAPREDUCE-7262.001.patch > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > org
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Attachment: MAPREDUCE-7262-elapsedTimes.pdf > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7262-elapsedTimes.pdf > > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > org.apache.hadoop.mapr
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Status: Patch Available (was: Open) > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > org.apache.hadoop.mapreduce.v2.app (67 usages found) > TestAMInfos (1 usage
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7262: - Summary: MRApp helpers block for long intervals (500ms) (was: MRApp helpers blocks for long intervals) > MRApp helpers block for long intervals (500ms) > -- > > Key: MAPREDUCE-7262 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > MRApp has a set of methods used as helpers in test cases such as: > {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc > When the condition fails, the thread sleeps for a minimum of 500ms before > rechecking the new state of the Job/TA. > Example: > {code:java} > public void waitForState(Task task, TaskState finalState) throws Exception { > int timeoutSecs = 0; > TaskReport report = task.getReport(); > while (!finalState.equals(report.getTaskState()) && > timeoutSecs++ < 20) { > System.out.println("Task State for " + task.getID() + " is : " > + report.getTaskState() + " Waiting for state : " + finalState > + " progress : " + report.getProgress()); > report = task.getReport(); > Thread.sleep(500); > } > System.out.println("Task State is : " + report.getTaskState()); > Assert.assertEquals("Task state is not correct (timedout)", finalState, > report.getTaskState()); > } > {code} > I suggest to reduce the interval 500 to 50, while incrementing the number of > retries to 200. this will potentially make the test cases run faster. Also, > the {{System.out}} calls need to be removed because they are not adding > information dumping the current state on every iteration. > A tentative list of Junits affected by the change: > {code:bash} > Method > waitForInternalState(JobImpl, JobStateInternal) > Found usages (12 usages found) > org.apache.hadoop.mapreduce.v2.app (10 usages found) > TestJobEndNotifier (3 usages found) > testNotificationOnLastRetry(boolean) (1 usage found) > 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1 > usage found) > 256 app.waitForInternalState(job, JobStateInternal.REBOOT); > testNotificationOnLastRetryUnregistrationFailure() (1 usage > found) > 289 app.waitForInternalState(job, JobStateInternal.REBOOT); > TestKill (5 usages found) > testKillJob() (1 usage found) > 70 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTask() (1 usage found) > 108 app.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testKillTaskWait() (1 usage found) > 219 app.waitForInternalState((JobImpl) job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobAfterTA_DONE() (1 usage found) > 266 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found) > 316 app.waitForInternalState((JobImpl)job, > JobStateInternal.KILLED); > TestMRApp (2 usages found) > testJobSuccess() (1 usage found) > 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED); > testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found) > 542 app.waitForInternalState((JobImpl) job, > JobStateInternal.REBOOT); > org.apache.hadoop.mapreduce.v2.app.rm (2 usages found) > TestRMContainerAllocator (2 usages found) > testReportedAppProgress() (1 usage found) > 1050 mrApp.waitForInternalState((JobImpl) job, > JobStateInternal.RUNNING); > testReportedAppProgressWithOnlyMaps() (1 usage found) > 1202 mrApp.waitForInternalState((JobImpl)job, > JobStateInternal.RUNNING); > -- > Method > waitForState(TaskAttempt, TaskAttemptState) > Found usages (72 usages found) > org.apache.hadoop.mapreduce.v2 (2 usages found) > TestSpeculativeExecutionWithMRApp (2 usages found) > testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found) > 212 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > testSpeculateSuccessfulWithUpdateEvents() (1 usage found) > 275 app.waitForState(taskAttempt.getValue(), > TaskAttemptState.SUCCEEDED); > org.apache.hadoop.