[jira] [Commented] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE

2016-08-25 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438291#comment-15438291
 ] 

Hadoop QA commented on MAPREDUCE-6768:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s 
{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: 
The patch generated 1 new + 118 unchanged - 1 fixed = 119 total (was 119) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 4s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 13s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825581/mapreduce6768.001.patch
 |
| JIRA Issue | MAPREDUCE-6768 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 51ccb7d6be4d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 81485db |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> TestRecovery.testSpeculative failed with NPE
> 
>
> Key: MAPREDUCE-6768
> URL: 

[jira] [Updated] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE

2016-08-25 Thread Haibo Chen (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6768:
--
Status: Patch Available  (was: Open)

> TestRecovery.testSpeculative failed with NPE
> 
>
> Key: MAPREDUCE-6768
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6768.001.patch
>
>
> 1 tests failed.
> REGRESSION:  org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative
> Error Message:
> null
> Stack Trace:
> java.lang.NullPointerException: null
> at 
> org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE

2016-08-25 Thread Haibo Chen (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6768:
--
Attachment: mapreduce6768.001.patch

> TestRecovery.testSpeculative failed with NPE
> 
>
> Key: MAPREDUCE-6768
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6768.001.patch
>
>
> 1 tests failed.
> REGRESSION:  org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative
> Error Message:
> null
> Stack Trace:
> java.lang.NullPointerException: null
> at 
> org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE

2016-08-25 Thread Haibo Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438260#comment-15438260
 ] 

Haibo Chen commented on MAPREDUCE-6768:
---

 {code:java}   
ContainerId t1a2contId = task1Attempt2.getAssignedContainerID();
LOG.info(t1a2contId.toString());
{code}

TaskAttempt2 may not have been assigned a container, this is why there is a 
NPE. Uploading a trivial patch to fix this.


> TestRecovery.testSpeculative failed with NPE
> 
>
> Key: MAPREDUCE-6768
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> 1 tests failed.
> REGRESSION:  org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative
> Error Message:
> null
> Stack Trace:
> java.lang.NullPointerException: null
> at 
> org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Created] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE

2016-08-25 Thread Haibo Chen (JIRA)
Haibo Chen created MAPREDUCE-6768:
-

 Summary: TestRecovery.testSpeculative failed with NPE
 Key: MAPREDUCE-6768
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Haibo Chen
Assignee: Haibo Chen


1 tests failed.
REGRESSION:  org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative

Error Message:
null

Stack Trace:
java.lang.NullPointerException: null
at 
org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6762) ControlledJob#toString failed with NPE when job status is not successfully updated

2016-08-25 Thread Varun Saxena (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated MAPREDUCE-6762:

Assignee: Weiwei Yang

> ControlledJob#toString failed with NPE when job status is not successfully 
> updated
> --
>
> Key: MAPREDUCE-6762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6762
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.9.0
>
> Attachments: MAPREDUCE-6762.001.patch
>
>
> This issue was found from a cluster where Pig query occasionally failed on 
> NPE. Pig uses JobControl API to track MR job status, but sometimes Job 
> History Server failed to flush job meta files to HDFS which caused the status 
> update failed. Then we get NPE in 
> {{org.apache.hadoop.mapreduce.Job.getJobName}}. The result of this situation 
> is quite confusing: Pig query failed, job history is missing, but the job 
> status on Yarn is succeed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437263#comment-15437263
 ] 

Hudson commented on MAPREDUCE-6764:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10348 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10348/])
MAPREDUCE-6764. Teragen LOG initialization bug. Contributed by Yufei Gu. 
(weichiu: rev 1360bd2d545134b582e70f2add33a105710dc80b)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java


> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
>  Labels: reviewed
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Yufei Gu (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437235#comment-15437235
 ] 

Yufei Gu commented on MAPREDUCE-6764:
-

Thanks [~jojochuang] for the review and commit! Thanks 
[~garlanaganarasi...@huawei.com] for the review!

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
>  Labels: reviewed
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Yufei Gu (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437235#comment-15437235
 ] 

Yufei Gu edited comment on MAPREDUCE-6764 at 8/25/16 5:06 PM:
--

Thanks [~jojochuang] for the review and commit! Thanks [~Naganarasimha] for the 
review!


was (Author: yufeigu):
Thanks [~jojochuang] for the review and commit! Thanks 
[~garlanaganarasi...@huawei.com] for the review!

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
>  Labels: reviewed
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Yufei Gu (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated MAPREDUCE-6764:

Labels: reviewed  (was: )

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
>  Labels: reviewed
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Yufei Gu (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated MAPREDUCE-6764:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
>  Labels: reviewed
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Yufei Gu (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated MAPREDUCE-6764:

Affects Version/s: 3.0.0-alpha1

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug

2016-08-25 Thread Wei-Chiu Chuang (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437222#comment-15437222
 ] 

Wei-Chiu Chuang commented on MAPREDUCE-6764:


Committed this patch to trunk, branch-2 and branch-2.8. Thanks [~yufeigu] for 
reporting and submitting a patch, and thanks [~Naganarasimha] for the reviewing 
the patch!

> Teragen LOG initialization bug
> --
>
> Key: MAPREDUCE-6764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: examples
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Minor
> Attachments: MAPREDUCE-6764.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application

2016-08-25 Thread Srikanth Sampath (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436757#comment-15436757
 ] 

Srikanth Sampath commented on MAPREDUCE-6754:
-

Thanks much [[~jlowe][[~vinodkv][[~jianhe]  Given the discussion above, we can 
just add the attemptId to the JvmId for now.  I will consider nuking out JvmId 
separately.  

> Container Ids for an yarn application should be monotonically increasing in 
> the scope of the application
> 
>
> Key: MAPREDUCE-6754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Srikanth Sampath
>Assignee: Srikanth Sampath
>
> Currently across application attempts, container Ids are reused.  The 
> container id is stored in AppSchedulingInfo and it is reinitialized with 
> every application attempt.  So the containerId scope is limited to the 
> application attempt.
> In the MR Framework, It is important to note that the containerId is being 
> used as part of the JvmId.  JvmId has 3 components  containerId>.  The JvmId is used in datastructures in TaskAttemptListener and 
> is passed between the AppMaster and the individual tasks.  For an application 
> attempt, no two tasks have the same JvmId.
> This works well currently, since inflight tasks get killed if the AppMaster 
> goes down.  However, if we want to enable WorkPreserving nature for the AM, 
> containers (and hence containerIds) live across application attempts.  If we 
> recycle containerIds across attempts, then two independent tasks (one 
> inflight from a previous attempt  and another new in a succeeding attempt) 
> can have the same JvmId and cause havoc.
> This can be solved in two ways:
> *Approach A*: Include attempt id as part of the JvmId. This is a viable 
> solution, however, there is a change in the format of the JVMid. Changing 
> something that has existed so long for an optional feature is not persuasive.
> *Approach B*: Keep the container id to be a monotonically increasing id for 
> the life of an application. So, container ids are not reused across 
> application attempts containers should be able to outlive an application 
> attempt. This is the preferred approach as it is clean, logical and is 
> backwards compatible. Nothing changes for existing applications or the 
> internal workings.  
> *How this is achieved:*
> Currently, we maintain latest containerId only for application attempts and 
> reinitialize for new attempts.  With this approach, we retrieve the latest 
> containerId from the just-failed attempt and initialize the new attempt with 
> the latest containerId (instead of 0).   I can provide the patch if it helps. 
>  It currently exists in MAPREDUCE-6726



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org