[jira] [Commented] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436245#comment-15436245
 ] 

Hadoop QA commented on MAPREDUCE-6638:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 48s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 35s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825396/mapreduce6638.002.patch
 |
| JIRA Issue | MAPREDUCE-6638 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 242a65b7551a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5a6fc5f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6697/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6697/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: 

[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436220#comment-15436220
 ] 

Jian He commented on MAPREDUCE-6754:


Thanks for the feedback, Jason, Vinod.
I think we can add a attemptId into the JvmID, given that it's internal only. 
[~srikanth.sampath], your opinion ?

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6638:
--
Status: Patch Available  (was: Open)

> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6638:
--
Attachment: mapreduce6638.002.patch

Uploading a new patch to address the unit test failure and check style warning

> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6638:
--
Status: Open  (was: Patch Available)

> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: mapreduce6638.001.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436138#comment-15436138
 ] 

Hudson commented on MAPREDUCE-6767:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10340 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10340/])
MAPREDUCE-6767. TestSlive fails after a common change. Contributed by 
(aajisaka: rev 5a6fc5f4831d130d9bdaca03f02f0ac4e7cfd64f)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TruncateOp.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/AppendOp.java


> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436130#comment-15436130
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-6754:


bq. Approach A: Include attempt id as part of the JvmId. This is a viable 
solution, however, there is a change in the format of the JVMid. Changing 
something that has existed so long for an optional feature is not persuasive.
bq. I don't understand the concern about changing the JvmID. It's not really 
public and only used within the scope of a single job
Agreed.

[~srikanth.sampath], JvmID originally was added to implement JVM reuse in 
Hadoop 1 MapReduce. When we moved to YARN + MR, we lost JVM reuse and I doubt 
if we are going to implement that now. So, I'd argue that we can completely 
remove JvmID. But if that's too much, like [~jlowe] says we can simply change 
the format of JvmID - it is not a public API.

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated MAPREDUCE-6767:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-3.0.0-alpha1. Thanks [~kihwal] for the 
report and thanks [~templedf] for the quick fix!

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated MAPREDUCE-6767:
-
Target Version/s: 3.0.0-alpha1

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436103#comment-15436103
 ] 

Akira Ajisaka commented on MAPREDUCE-6767:
--

LGTM, +1

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436073#comment-15436073
 ] 

Hadoop QA commented on MAPREDUCE-6767:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 117m 58s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 22s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825347/MAPREDUCE-6767.001.patch
 |
| JIRA Issue | MAPREDUCE-6767 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 95f76f787a10 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a1f3293 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6696/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6696/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435883#comment-15435883
 ] 

Daniel Templeton commented on MAPREDUCE-6767:
-

Oops.  Thanks, [~chris.douglas].

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435876#comment-15435876
 ] 

Hadoop QA commented on MAPREDUCE-6638:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s 
{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: 
The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 42s {color} 
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 52s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.app.TestRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825352/mapreduce6638.001.patch
 |
| JIRA Issue | MAPREDUCE-6638 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 63f482cc2a93 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a1f3293 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6695/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6695/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6695/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6695/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: 

[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated MAPREDUCE-6767:
-
Status: Patch Available  (was: Open)

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435862#comment-15435862
 ] 

Haibo Chen edited comment on MAPREDUCE-6765 at 8/24/16 10:42 PM:
-

Tested it on a pseudo-distributed cluster, ran a pi job that requests a 
container bigger than what is supported at maximum, worked fine.


was (Author: haibochen):
Tested it on a pseudo-distributed cluster, ran a pi job, worked fine.

> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: mapreduce6765.001.patch
>
>
> When mapper or reducer containers request resource larger than the 
> maxResourceRequest in the cluster, job is to be killed. In such cases, it is 
> unnecessary to still schedule container requests.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435862#comment-15435862
 ] 

Haibo Chen commented on MAPREDUCE-6765:
---

Tested it on a pseudo-distributed cluster, ran a pi job, worked fine.

> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: mapreduce6765.001.patch
>
>
> When mapper or reducer containers request resource larger than the 
> maxResourceRequest in the cluster, job is to be killed. In such cases, it is 
> unnecessary to still schedule container requests.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6638:
--
Status: Patch Available  (was: Open)

> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: mapreduce6638.001.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6638) Jobs with encrypted spills don't recover if the AM goes down

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6638:
--
Attachment: mapreduce6638.001.patch

The patch skips recovery if spill encryption is enabled and rerun all mappers 
and reducers.  Have not seen any safe place to store the spill encryption key 
in MRAppMaster.

> Jobs with encrypted spills don't recover if the AM goes down
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: mapreduce6638.001.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change

[Daniel Templeton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated MAPREDUCE-6767:

Attachment: MAPREDUCE-6767.001.patch

This resolves the Slive failures.

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6767.001.patch
>
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Assigned] (MAPREDUCE-6767) TestSlive fails after a common change

[Daniel Templeton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton reassigned MAPREDUCE-6767:
---

Assignee: Daniel Templeton

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Daniel Templeton
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435706#comment-15435706
 ] 

Kihwal Lee commented on MAPREDUCE-6767:
---

E.g.
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/143/testReport/junit/org.apache.hadoop.fs.slive/TestSlive/testSelection/

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6501) Improve reducer preemption

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435690#comment-15435690
 ] 

Haibo Chen commented on MAPREDUCE-6501:
---

Task 2 is already implemented with taskAttemptProgress, that is, SHUFFLE phase 
is expressed in terms of progress, and preemption also tries to preempt 
reducers that are of least progress.
Task 3 is inapplicable since reducer preemption never kicks in if there is one 
mapper can run.

> Improve reducer preemption
> --
>
> Key: MAPREDUCE-6501
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6501
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
>
> As discussed on MAPREDUCE-6302, we could improve the reducer preemption as 
> follows:
> # preempt enough reducers so there are enough mappers to match slowstart 
> # prioritize preempting reducers that are still in SHUFFLE phase
> # add an option to not preempt reducers that are past SHUFFLE phase 
> irrespective of slowstart as long as one mapper can run
> We could do it all in one patch or create subtasks as necessary. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6767) TestSlive fails after a common change

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435619#comment-15435619
 ] 

Daniel Templeton commented on MAPREDUCE-6767:
-

Any details you can give?

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6767) TestSlive fails after a common change

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-6767:
--
Description: It looks like this was broken after HADOOP-12726.

> TestSlive fails after a common change
> -
>
> Key: MAPREDUCE-6767
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Kihwal Lee
>
> It looks like this was broken after HADOOP-12726.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6767) TestSlive fails after a common change

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-6767:
-

 Summary: TestSlive fails after a common change
 Key: MAPREDUCE-6767
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Kihwal Lee






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435547#comment-15435547
 ] 

Hadoop QA commented on MAPREDUCE-6765:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 55s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 38s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825319/mapreduce6765.001.patch
 |
| JIRA Issue | MAPREDUCE-6765 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 69fa0eb9733c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3476156 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: 

[jira] [Updated] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6765:
--
Status: Patch Available  (was: Open)

> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: mapreduce6765.001.patch
>
>
> When mapper or reducer containers request resource larger than the 
> maxResourceRequest in the cluster, job is to be killed. In such cases, it is 
> unnecessary to still schedule container requests.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6765:
--
Attachment: mapreduce6765.001.patch

The patch is trivial.  In cases where the resource requested by container  is 
not supported, do not scheduler container requests, since the job will soon be 
killed. 

There is no test since the fix is trivial, and ScheduledRequests is an inner 
class whose ScheduledRequests.addMap or ScheduledRequests.addReduce are hard to 
spy on.

> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: mapreduce6765.001.patch
>
>
> When mapper or reducer containers request resource larger than the 
> maxResourceRequest in the cluster, job is to be killed. In such cases, it is 
> unnecessary to still schedule container requests.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435334#comment-15435334
 ] 

Haibo Chen commented on MAPREDUCE-6765:
---

Not a 100% sure, it seems that you'll have to be added by a committer to become 
a contributor first.

> MR should not schedule container requests in cases where reducer or mapper 
> containers demand resource larger than the maximum supported
> ---
>
> Key: MAPREDUCE-6765
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.9.0
>
>
> When mapper or reducer containers request resource larger than the 
> maxResourceRequest in the cluster, job is to be killed. In such cases, it is 
> unnecessary to still schedule container requests.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435248#comment-15435248
 ] 

Jason Lowe commented on MAPREDUCE-6754:
---

I don't understand the concern about changing the JvmID.  It's not really 
public and only used within the scope of a single job (i.e.: not by the JHS or 
anything like that) so from my perspective we can completely gut this class to 
do whatever we want.  Am I missing a use-case for the JvmID that is going to be 
problematic?  Adding the AM attempt ID to the JvmID seems like a 
straightforward and proper fix.  If we're really concerned about 
backwards-compatibility then we can preserve the existing methods and add new 
constructor methods that can take a specified attempt ID.  The existing 
constructors can assume an attempt ID of 1.  There's still the string 
representation if someone is trying to parse the JvmID string, but again I 
don't know where that would be seen by end-users outside of AM logs (if it's 
even there).

I'm not a fan of changing YARN's container ID semantics to fix this, as I see 
this as a MapReduce-specific problem that has a straightforward solution in 
MapReduce.

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6766) Concurrent local job failures due to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis())

[Mark S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark S updated MAPREDUCE-6766:
--
Summary: Concurrent local job failures due to uniqueNumberGenerator = new 
AtomicLong(System.currentTimeMillis())  (was: Concurrent Local Job Failures due 
to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis()))

> Concurrent local job failures due to uniqueNumberGenerator = new 
> AtomicLong(System.currentTimeMillis())
> ---
>
> Key: MAPREDUCE-6766
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6766
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.3.0, 2.7.1
> Environment: Druid 0.8.3
>Reporter: Mark S
>
> I am seeing the following exception when attempting to execute multiple 
> Hadoop Local Jobs with Druid.
> {code}
> java.io.IOException: Rename cannot overwrite non empty destination directory 
> /tmp/hadoop-username/mapred/local/1472019105135
> {code}
> From a quick look at the Hadoop code base, it seems that the 
> uniqueNumberGenerator for the LocalDistributedCacheManager is based on the 
> System time, and this appears to cause problems for concurrent jobs.
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
> new AtomicLong(System.currentTimeMillis());
> {code}
> I am pretty sure the following line of code is responsible, and this seems to 
> exist in latter versions of such as 2.7.1:
> * [Hadoop 2.3.0 - 
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.3.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> * [Hadoop 2.7.1 - 
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> h3.  Full Stack Trace
> {code}
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>   at com.google.common.base.Throwables.propagate(Throwables.java:160) 
> ~[guava-16.0.1.jar:?]
>   at 
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206) 
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285)
>  [druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265)
>  [druid-indexing-service-0.8.3.jar:0.8.3]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_71]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_71]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_71]
>   at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
>   at 
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   ... 7 more
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
> overwrite non empty destination directory 
> /tmp/hadoop-username/mapred/local/1472019105135
>   at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_71]
>   at 
> 

[jira] [Created] (MAPREDUCE-6766) Concurrent Local Job Failures due to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis())

[Mark S (JIRA)]

Mark S created MAPREDUCE-6766:
-

 Summary: Concurrent Local Job Failures due to 
uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis())
 Key: MAPREDUCE-6766
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6766
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.3.0
 Environment: Druid 0.8.3
Reporter: Mark S


I am seeing the following exception when attempting to execute multiple Hadoop 
Local Jobs with Druid.

{code}
java.io.IOException: Rename cannot overwrite non empty destination directory 
/tmp/hadoop-username/mapred/local/1472019105135
{code}

>From a quick look at the Hadoop code base, it seems that the 
>uniqueNumberGenerator for the LocalDistributedCacheManager is based on the 
>System time, and this appears to cause problems for concurrent jobs.

{code}
// Generating unique numbers for FSDownload.
AtomicLong uniqueNumberGenerator =
new AtomicLong(System.currentTimeMillis());
{code}


I am pretty sure the following line of code is responsible, and this seems to 
exist in latter versions of such as 2.7.1:

* [Hadoop 2.3.0 - 
LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.3.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
* [Hadoop 2.7.1 - 
LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]


h3.  Full Stack Trace
{code}
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) 
~[guava-16.0.1.jar:?]
at 
io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138)
 ~[druid-indexing-service-0.8.3.jar:0.8.3]
at 
io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206) 
~[druid-indexing-service-0.8.3.jar:0.8.3]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285)
 [druid-indexing-service-0.8.3.jar:0.8.3]
at 
io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265)
 [druid-indexing-service-0.8.3.jar:0.8.3]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_71]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_71]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_71]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
at 
io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
 ~[druid-indexing-service-0.8.3.jar:0.8.3]
... 7 more
Caused by: java.lang.RuntimeException: java.io.IOException: 
java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
overwrite non empty destination directory 
/tmp/hadoop-username/mapred/local/1472019105135
at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211) 
~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) 
~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
at 
io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) 
~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
at 
io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
 ~[druid-indexing-service-0.8.3.jar:0.8.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_71]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_71]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
at 
io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
 ~[druid-indexing-service-0.8.3.jar:0.8.3]
... 7 more
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: 
java.io.IOException: Rename cannot overwrite non empty destination directory 
/tmp/hadoop-username/mapred/local/1472019105135
at 

[jira] [Updated] (MAPREDUCE-6766) Concurrent Local Job Failures due to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis())

[Mark S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark S updated MAPREDUCE-6766:
--
Affects Version/s: 2.7.1

> Concurrent Local Job Failures due to uniqueNumberGenerator = new 
> AtomicLong(System.currentTimeMillis())
> ---
>
> Key: MAPREDUCE-6766
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6766
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.3.0, 2.7.1
> Environment: Druid 0.8.3
>Reporter: Mark S
>
> I am seeing the following exception when attempting to execute multiple 
> Hadoop Local Jobs with Druid.
> {code}
> java.io.IOException: Rename cannot overwrite non empty destination directory 
> /tmp/hadoop-username/mapred/local/1472019105135
> {code}
> From a quick look at the Hadoop code base, it seems that the 
> uniqueNumberGenerator for the LocalDistributedCacheManager is based on the 
> System time, and this appears to cause problems for concurrent jobs.
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
> new AtomicLong(System.currentTimeMillis());
> {code}
> I am pretty sure the following line of code is responsible, and this seems to 
> exist in latter versions of such as 2.7.1:
> * [Hadoop 2.3.0 - 
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.3.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> * [Hadoop 2.7.1 - 
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> h3.  Full Stack Trace
> {code}
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>   at com.google.common.base.Throwables.propagate(Throwables.java:160) 
> ~[guava-16.0.1.jar:?]
>   at 
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206) 
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285)
>  [druid-indexing-service-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265)
>  [druid-indexing-service-0.8.3.jar:0.8.3]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_71]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_71]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_71]
>   at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
>   at 
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   ... 7 more
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
> overwrite non empty destination directory 
> /tmp/hadoop-username/mapred/local/1472019105135
>   at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) 
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
>   at 
> io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
>  ~[druid-indexing-service-0.8.3.jar:0.8.3]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_71]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_71]
>   at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
>   at 
> 

[jira] [Commented] (MAPREDUCE-6761) Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435071#comment-15435071
 ] 

Hudson commented on MAPREDUCE-6761:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10337 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10337/])
MAPREDUCE-6761. Regression when handling providers - invalid (jlowe: rev 
6fa9bf4407a0b09be8ce4e52d2fde89adba34f1a)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* (add) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestCluster.java


> Regression when handling providers - invalid configuration 
> ServiceConfiguration causes Cluster initialization failure
> -
>
> Key: MAPREDUCE-6761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6761
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Peter Vary
>Assignee: Peter Vary
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6761.2.patch, MAPREDUCE-6761.3.patch, 
> MAPREDUCE-6761.4.patch, MAPREDUCE-6761.5.patch, MAPREDUCE-6761.patch
>
>
> When a rogue org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider 
> defines a provider that is not on classpath, then the initialization is 
> failed with the following exception:
> java.util.ServiceConfigurationError: 
> org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider: Provider 
> org.apache.hadoop.mapred.YarnClientProtocolProvider not found
>   at java.util.ServiceLoader.fail(ServiceLoader.java:239)
>   at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
>   at 
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
>   at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>   at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>   at org.apache.hadoop.mapreduce.Cluster.initProviderList(Cluster.java:84)
>   at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:114)
>   at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:108)
>   at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:101)
>   at org.apache.hadoop.mapred.JobClient.init(JobClient.java:477)
>   at org.apache.hadoop.mapred.JobClient.(JobClient.java:455)
> This regression is caused by MAPREDUCE-6473



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6761) Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-6761:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Thanks to [~pvary] for the contribution and to [~rchiang], [~templedf], and 
[~kshukla] for additional review!  I committed this to trunk, branch-2, and 
branch-2.8.

> Regression when handling providers - invalid configuration 
> ServiceConfiguration causes Cluster initialization failure
> -
>
> Key: MAPREDUCE-6761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6761
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 3.0.0-alpha2
>Reporter: Peter Vary
>Assignee: Peter Vary
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6761.2.patch, MAPREDUCE-6761.3.patch, 
> MAPREDUCE-6761.4.patch, MAPREDUCE-6761.5.patch, MAPREDUCE-6761.patch
>
>
> When a rogue org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider 
> defines a provider that is not on classpath, then the initialization is 
> failed with the following exception:
> java.util.ServiceConfigurationError: 
> org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider: Provider 
> org.apache.hadoop.mapred.YarnClientProtocolProvider not found
>   at java.util.ServiceLoader.fail(ServiceLoader.java:239)
>   at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
>   at 
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
>   at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>   at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>   at org.apache.hadoop.mapreduce.Cluster.initProviderList(Cluster.java:84)
>   at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:114)
>   at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:108)
>   at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:101)
>   at org.apache.hadoop.mapred.JobClient.init(JobClient.java:477)
>   at org.apache.hadoop.mapred.JobClient.(JobClient.java:455)
> This regression is caused by MAPREDUCE-6473



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435030#comment-15435030
 ] 

Hudson commented on MAPREDUCE-6578:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10336 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10336/])
MAPREDUCE-6578. Add support for HDFS heterogeneous storage testing to 
(kai.zheng: rev 0ce1ab95cc1178f9ea763fd1f5a65a890b23b0de)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java


> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6628) Potential memory leak in CryptoOutputStream

[Mariappan Asokan (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435024#comment-15435024
 ] 

Mariappan Asokan commented on MAPREDUCE-6628:
-

The test failures seem to be unrelated to this patch.  Can a committer please 
commit this Jira? Thanks.

> Potential memory leak in CryptoOutputStream
> ---
>
> Key: MAPREDUCE-6628
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6628
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.6.4
>Reporter: Mariappan Asokan
>Assignee: Mariappan Asokan
> Attachments: MAPREDUCE-6628.001.patch, MAPREDUCE-6628.002.patch, 
> MAPREDUCE-6628.003.patch, MAPREDUCE-6628.004.patch, MAPREDUCE-6628.005.patch, 
> MAPREDUCE-6628.006.patch, MAPREDUCE-6628.007.patch
>
>
> There is a potential memory leak in {{CryptoOutputStream.java.}}  It 
> allocates two direct byte buffers ({{inBuffer}} and {{outBuffer}}) that get 
> freed when {{close()}} method is called.  Most of the time, {{close()}} 
> method is called.  However, when writing to intermediate Map output file or 
> the spill files in {{MapTask}}, {{close()}} is never called since calling so  
> would close the underlying stream which is not desirable.  There is a single 
> underlying physical stream that contains multiple logical streams one per 
> partition of Map output.  
> By default the amount of memory allocated per byte buffer is 128 KB and  so 
> the total memory allocated is 256 KB,  This may not sound much.  However, if 
> the number of partitions (or number of reducers) is large (in the hundreds) 
> and/or there are spill files created in {{MapTask}}, this can grow into a few 
> hundred MB. 
> I can think of two ways to address this issue:
> h2. Possible Fix - 1
> According to JDK documentation:
> {quote}
> The contents of direct buffers may reside outside of the normal 
> garbage-collected heap, and so their impact upon the memory footprint of an 
> application might not be obvious.  It is therefore recommended that direct 
> buffers be allocated primarily for large, long-lived buffers that are subject 
> to the underlying system's native I/O operations.  In general it is best to 
> allocate direct buffers only when they yield a measureable gain in program 
> performance.
> {quote}
> It is not clear to me whether there is any benefit of allocating direct byte 
> buffers in {{CryptoOutputStream.java}}.  In fact, there is a slight CPU 
> overhead in moving data from {{outBuffer}} to a temporary byte array as per 
> the following code in {{CryptoOutputStream.java}}.
> {code}
> /*
>  * If underlying stream supports {@link ByteBuffer} write in future, needs
>  * refine here. 
>  */
> final byte[] tmp = getTmpBuf();
> outBuffer.get(tmp, 0, len);
> out.write(tmp, 0, len);
> {code}
> Even if the underlying stream supports direct byte buffer IO (or direct IO in 
> OS parlance), it is not clear whether it will yield any measurable 
> performance gain.
> The fix would be to allocate a ByteBuffer on the heap for inBuffer and wrap a 
> byte array in a {{ByteBuffer}} for {{outBuffer}}.  By the way, the 
> {{inBuffer}} and {{outBuffer}} have to be {{ByteBuffer}} as demanded by the 
> {{encrypt()}} method in {{Encryptor}}.
> h2. Possible Fix - 2
> Assuming that we want to keep the buffers as direct byte buffers, we can 
> create a new constructor to {{CryptoOutputStream}} and pass a boolean flag 
> {{ownOutputStream}} to indicate whether the underlying stream will be owned 
> by {{CryptoOutputStream}}. If it is true, then calling the {{close()}} method 
> will close the underlying stream.  Otherwise, when {{close()}} is called only 
> the direct byte buffers will be freed and the underlying stream will not be 
> closed.
> The scope of changes for this fix will be somewhat wider.  We need to modify 
> {{MapTask.java}}, {{CryptoUtils.java}}, and {{CryptoFSDataOutputStream.java}} 
> as well to pass the ownership flag mentioned above.
> I can post a patch for either of the above.  I welcome any other ideas from 
> developers to fix this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Kai Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435019#comment-15435019
 ] 

Kai Zheng commented on MAPREDUCE-6578:
--

Please ignore the above messy building message. It happened after the patch 
committed already. I fixed the minor check style (line too long) just before 
committing it.

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Kai Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated MAPREDUCE-6578:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha1
   Status: Resolved  (was: Patch Available)

Committed to 3.0.0 and trunk branches. Thanks [~zhouwei] and [~Sammi] for the 
contribution.

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435005#comment-15435005
 ] 

Hadoop QA commented on MAPREDUCE-6578:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color} 
| {color:red} MAPREDUCE-6578 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825269/MAPREDUCE-6578.04.patch
 |
| JIRA Issue | MAPREDUCE-6578 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6693/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Kai Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated MAPREDUCE-6578:
-
Attachment: MAPREDUCE-6578.04.patch

This fixed the minor style.

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Kai Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434999#comment-15434999
 ] 

Kai Zheng commented on MAPREDUCE-6578:
--

+1 on the updated patch except the minor style. Will commit the latest one that 
solved the style.

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch, MAPREDUCE-6578.04.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434807#comment-15434807
 ] 

Hadoop QA commented on MAPREDUCE-6578:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
 The patch generated 2 new + 49 unchanged - 1 fixed = 51 total (was 50) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 117m 30s 
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 19s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.slive.TestSlive |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825239/MAPREDUCE-6578.03.patch
 |
| JIRA Issue | MAPREDUCE-6578 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux aecca326fa35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 092b4d5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6692/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6692/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6692/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6692/testReport/ |
| modules | C: 

[jira] [Commented] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434646#comment-15434646
 ] 

SammiChen commented on MAPREDUCE-6578:
--

Thanks Kai's suggestion. If the user input {{storagePolicy}} parameter is 
incorrect or invalid, it's better provide user the knowledge that what's are 
current supported storage policy list. 
Update the patch based on above thoughts. 

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO

[Kai Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated MAPREDUCE-6578:
-
Attachment: MAPREDUCE-6578.03.patch

Uploading the updated patch provided by [~Sammi]. Thanks!

> Add support for HDFS heterogeneous storage testing to TestDFSIO
> ---
>
> Key: MAPREDUCE-6578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Attachments: MAPREDUCE-6578.00.patch, MAPREDUCE-6578.01.patch, 
> MAPREDUCE-6578.02.patch, MAPREDUCE-6578.03.patch
>
>
> HDFS heterogeneous storage allows user to store data blocks to different 
> storage medias according to predefined storage policies. Only 'Default' 
> policy is supported in current TestDFSIO implementation. This is going to add 
> an new option to enable tests of other storage polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434421#comment-15434421
 ] 

Jian He commented on MAPREDUCE-6754:


Hi [~jlowe], mind help shedding some light on this ? any reason the JvmID did 
not include the attemptId ? or any problem if we add that.  If we cannot add 
the attempt Id in the JvmID, we'll go with approach B to make 
ContainerId#getContainerId uniq across attempts.

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org