[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval

2016-08-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419144#comment-15419144
 ] 

Hadoop QA commented on MAPREDUCE-6740:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 2 new + 723 unchanged - 3 fixed = 725 total (was 726) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 6s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 13s {color} 
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 29s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.app.TestFail |
| Timed out junit tests | 
org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12823386/mapreduce6740.003.patch
 |
| JIRA Issue | MAPREDUCE-6740 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 8e8e48fae62a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 23c6e3c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6670/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
 |
| unit |

[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval

2016-08-12 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6740:
--
Status: Patch Available  (was: Open)

Uploaded a new patch to address Karthik's comment. This patch 
1) enforces task timeout to be as least twice as long as progress interval
2) sets the default value of progress report interval to 1% of the default 
value of task timeout, which is still 3000 milliseconds.
Regardless of what value the user set the progress-report-interval to, the task 
timeout will always be Max(2*progress-report-interval,  configured value of 
task timeout).

> Enforce mapreduce.task.timeout to be at least 
> mapreduce.task.progress-report.interval
> -
>
> Key: MAPREDUCE-6740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.8.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, 
> mapreduce6740.003.patch
>
>
> Mapreduce-6242 makes task status update interval configurable to ease the 
> pressure on MR AM to process status updates, but it did not ensure that 
> mapreduce.task.timeout is no smaller than the configured value of task report 
> interval. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

2016-08-12 Thread Srikanth Sampath (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Srikanth Sampath updated MAPREDUCE-6754:

Summary: Container Ids for an yarn application should be monotonically 
increasing in the scope of the application  (was: Container Ids for a yarn 
application should be monotonically increasing in the scope of the application)

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6754) Container Ids for a yarn application should be monotonically increasing in the scope of the application

2016-08-12 Thread Srikanth Sampath (JIRA) 
Srikanth Sampath created MAPREDUCE-6754:
---

 Summary: Container Ids for a yarn application should be 
monotonically increasing in the scope of the application
 Key: MAPREDUCE-6754
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Srikanth Sampath
Assignee: Srikanth Sampath


Currently across application attempts, container Ids are reused.  The container 
id is stored in AppSchedulingInfo and it is reinitialized with every 
application attempt.  So the containerId scope is limited to the application 
attempt.

In the MR Framework, It is important to note that the containerId is being used 
as part of the JvmId.  JvmId has 3 components .  
The JvmId is used in datastructures in TaskAttemptListener and is passed 
between the AppMaster and the individual tasks.  For an application attempt, no 
two tasks have the same JvmId.

This works well currently, since inflight tasks get killed if the AppMaster 
goes down.  However, if we want to enable WorkPreserving nature for the AM, 
containers (and hence containerIds) live across application attempts.  If we 
recycle containerIds across attempts, then two independent tasks (one inflight 
from a previous attempt  and another new in a succeeding attempt) can have the 
same JvmId and cause havoc.

This can be solved in two ways:

*Approach A*: Include attempt id as part of the JvmId. This is a viable 
solution, however, there is a change in the format of the JVMid. Changing 
something that has existed so long for an optional feature is not persuasive.

*Approach B*: Keep the container id to be a monotonically increasing id for the 
life of an application. So, container ids are not reused across application 
attempts containers should be able to outlive an application attempt. This is 
the preferred approach as it is clean, logical and is backwards compatible. 
Nothing changes for existing applications or the internal workings.  
*How this is achieved:*
Currently, we maintain latest containerId only for application attempts and 
reinitialize for new attempts.  With this approach, we retrieve the latest 
containerId from the just-failed attempt and initialize the new attempt with 
the latest containerId (instead of 0).   I can provide the patch if it helps.  
It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org