[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419144#comment-15419144 ] Hadoop QA commented on MAPREDUCE-6740: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s {color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 723 unchanged - 3 fixed = 725 total (was 726) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 6s {color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 13s {color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 29s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.mapreduce.v2.app.TestFail | | Timed out junit tests | org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncher | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12823386/mapreduce6740.003.patch | | JIRA Issue | MAPREDUCE-6740 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 8e8e48fae62a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 23c6e3c | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6670/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt | | unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6670/arti
[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6740: -- Status: Patch Available (was: Open) Uploaded a new patch to address Karthik's comment. This patch 1) enforces task timeout to be as least twice as long as progress interval 2) sets the default value of progress report interval to 1% of the default value of task timeout, which is still 3000 milliseconds. Regardless of what value the user set the progress-report-interval to, the task timeout will always be Max(2*progress-report-interval, configured value of task timeout). > Enforce mapreduce.task.timeout to be at least > mapreduce.task.progress-report.interval > - > > Key: MAPREDUCE-6740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.8.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, > mapreduce6740.003.patch > > > Mapreduce-6242 makes task status update interval configurable to ease the > pressure on MR AM to process status updates, but it did not ensure that > mapreduce.task.timeout is no smaller than the configured value of task report > interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Sampath updated MAPREDUCE-6754: Summary: Container Ids for an yarn application should be monotonically increasing in the scope of the application (was: Container Ids for a yarn application should be monotonically increasing in the scope of the application) > Container Ids for an yarn application should be monotonically increasing in > the scope of the application > > > Key: MAPREDUCE-6754 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Srikanth Sampath >Assignee: Srikanth Sampath > > Currently across application attempts, container Ids are reused. The > container id is stored in AppSchedulingInfo and it is reinitialized with > every application attempt. So the containerId scope is limited to the > application attempt. > In the MR Framework, It is important to note that the containerId is being > used as part of the JvmId. JvmId has 3 components containerId>. The JvmId is used in datastructures in TaskAttemptListener and > is passed between the AppMaster and the individual tasks. For an application > attempt, no two tasks have the same JvmId. > This works well currently, since inflight tasks get killed if the AppMaster > goes down. However, if we want to enable WorkPreserving nature for the AM, > containers (and hence containerIds) live across application attempts. If we > recycle containerIds across attempts, then two independent tasks (one > inflight from a previous attempt and another new in a succeeding attempt) > can have the same JvmId and cause havoc. > This can be solved in two ways: > *Approach A*: Include attempt id as part of the JvmId. This is a viable > solution, however, there is a change in the format of the JVMid. Changing > something that has existed so long for an optional feature is not persuasive. > *Approach B*: Keep the container id to be a monotonically increasing id for > the life of an application. So, container ids are not reused across > application attempts containers should be able to outlive an application > attempt. This is the preferred approach as it is clean, logical and is > backwards compatible. Nothing changes for existing applications or the > internal workings. > *How this is achieved:* > Currently, we maintain latest containerId only for application attempts and > reinitialize for new attempts. With this approach, we retrieve the latest > containerId from the just-failed attempt and initialize the new attempt with > the latest containerId (instead of 0). I can provide the patch if it helps. > It currently exists in MAPREDUCE-6726 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6754) Container Ids for a yarn application should be monotonically increasing in the scope of the application
Srikanth Sampath created MAPREDUCE-6754: --- Summary: Container Ids for a yarn application should be monotonically increasing in the scope of the application Key: MAPREDUCE-6754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Srikanth Sampath Assignee: Srikanth Sampath Currently across application attempts, container Ids are reused. The container id is stored in AppSchedulingInfo and it is reinitialized with every application attempt. So the containerId scope is limited to the application attempt. In the MR Framework, It is important to note that the containerId is being used as part of the JvmId. JvmId has 3 components . The JvmId is used in datastructures in TaskAttemptListener and is passed between the AppMaster and the individual tasks. For an application attempt, no two tasks have the same JvmId. This works well currently, since inflight tasks get killed if the AppMaster goes down. However, if we want to enable WorkPreserving nature for the AM, containers (and hence containerIds) live across application attempts. If we recycle containerIds across attempts, then two independent tasks (one inflight from a previous attempt and another new in a succeeding attempt) can have the same JvmId and cause havoc. This can be solved in two ways: *Approach A*: Include attempt id as part of the JvmId. This is a viable solution, however, there is a change in the format of the JVMid. Changing something that has existed so long for an optional feature is not persuasive. *Approach B*: Keep the container id to be a monotonically increasing id for the life of an application. So, container ids are not reused across application attempts containers should be able to outlive an application attempt. This is the preferred approach as it is clean, logical and is backwards compatible. Nothing changes for existing applications or the internal workings. *How this is achieved:* Currently, we maintain latest containerId only for application attempts and reinitialize for new attempts. With this approach, we retrieve the latest containerId from the just-failed attempt and initialize the new attempt with the latest containerId (instead of 0). I can provide the patch if it helps. It currently exists in MAPREDUCE-6726 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org