[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701077#comment-15701077 ] Varun Vasudev commented on MAPREDUCE-6743: -- +1 from my end. Applied the patch on my machine and ran the tests. The results are published in an xml format in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire-reports/TEST-nativetask-nttest.xml. > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701058#comment-15701058
]
Hadoop QA commented on MAPREDUCE-6743:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hadoop-mapreduce-client-nativetask in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827838/MAPREDUCE-5743.03.patch
|
| JIRA Issue | MAPREDUCE-6743 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 1d9ece14fc15 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 209e805 |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6821/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6821/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> nativetask unit tests need to provide usable output; fix link errors during
> mvn test
>
>
> Key: MAPREDUCE-6743
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: nativetask
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch,
> MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch
>
>
> Currently,
[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701036#comment-15701036 ] Allen Wittenauer commented on MAPREDUCE-6743: - This was built off of 1.8.0, the very latest release at the time of the patch. > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6743) nativetask unit tests need to provide usable output; fix link errors during mvn test
[ https://issues.apache.org/jira/browse/MAPREDUCE-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701016#comment-15701016 ] Akira Ajisaka commented on MAPREDUCE-6743: -- Sorry for being very late. Mostly looks good to me. I have one question: What version (or hashtag?) of the gtest in this patch? I'm thinking adding this information to the release note or description makes it easier for tracking if there is something to do. > nativetask unit tests need to provide usable output; fix link errors during > mvn test > > > Key: MAPREDUCE-6743 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6743 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: nativetask >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: MAPREDUCE-5743.00.patch, MAPREDUCE-5743.01.patch, > MAPREDUCE-5743.02.patch, MAPREDUCE-5743.03.patch > > > Currently, hadoop-mapreduce-client-nativetask creates a nttest binary which > provides an binary exit code to determine failure. This means there is no > output generated by the Jenkins run to actually debug or provide hints as to > what failed. Given that nttest is written with gtest, it should be > configured to either spit out junit or TAP which can then be used to provide > further analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1578#comment-1578 ] Naganarasimha G R edited comment on MAPREDUCE-6786 at 11/27/16 5:13 PM: Hi [~devaraj.k], Actually wanted to work on this instead specified MAPREDUCE-6784 ! Hope i can work on this too... was (Author: naganarasimha): Hi [~devaraj.k], Actually wanted to work on this instead specified MAPREDUCE-6786 ! Hope i can work on this too... > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1578#comment-1578 ] Naganarasimha G R commented on MAPREDUCE-6786: -- Hi [~devaraj.k], Actually wanted to work on this instead specified MAPREDUCE-6786 ! Hope i can work on this too... > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6711) JobImpl fails to handle preemption events on state COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1571#comment-1571
]
Prabhu Joseph commented on MAPREDUCE-6711:
--
[~djp] I think since the job is already in Committing state as the tasks are
finished, both those events can be ignored.
> JobImpl fails to handle preemption events on state COMMITTING
> -
>
> Key: MAPREDUCE-6711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Prabhu Joseph
> Attachments: MAPREDUCE-6711.1.patch, MAPREDUCE-6711.patch
>
>
> When a MR app being preempted on COMMITTING state, we saw the following
> exceptions in its log:
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_ATTEMPT_COMPLETED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> and
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_MAP_TASK_RESCHEDULED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> Seems like we need to handle those preemption related events when the job is
> being committed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated MAPREDUCE-6773: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: MR-6749 Status: Resolved (was: Patch Available) Thanks for the contribution [~devaraj.k], have committed it to MR-6749 branch ! > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Fix For: MR-6749 > > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-MR-6749.005.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6711) JobImpl fails to handle preemption events on state COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699773#comment-15699773
]
Prabhu Joseph commented on MAPREDUCE-6711:
--
Hi [~djp], Sorry missed the mail. Job is a Map only and has a single Map task.
Once Map Attempt and Task is SUCCEEDED, the job transitioned from RUNNING to
COMMITTING state. At this point, if the Succeeded Attempt is Killed as part of
Container Preemption, a T_ATTEMPT_KILLED is raised and the task transitioned
from Succeeded to Scheduled, TaskImpl#RetroactiveKilledTransition tells the job
about the rescheduling by raising both JOB_TASK_ATTEMPT_COMPLETED and
JOB_MAP_TASK_RESCHEDULED. The job now will receive both these events at
COMMITTING state and fails as the transition is not handled.
Looks like the fix can ignore the JOB_TASK_ATTEMPT_COMPLETED but not
JOB_MAP_TASK_RESCHEDULED instead move the COMMITTING job to RUNNING state again
and reschedule the Map Task like below.
addTransition(JobStateInternal.COMMITTING, JobStateInternal.RUNNING,
JobEventType.JOB_MAP_TASK_RESCHEDULED, new MapTaskRescheduledTransition())
Please share your comments.
> JobImpl fails to handle preemption events on state COMMITTING
> -
>
> Key: MAPREDUCE-6711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Prabhu Joseph
> Attachments: MAPREDUCE-6711.1.patch, MAPREDUCE-6711.patch
>
>
> When a MR app being preempted on COMMITTING state, we saw the following
> exceptions in its log:
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_ATTEMPT_COMPLETED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> and
> {code}
> ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_MAP_TASK_RESCHEDULED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1289)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1285)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> Seems like we need to handle those preemption related events when the job is
> being committed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
