[jira] [Commented] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620200#comment-16620200
]
Oleksandr Shevchenko commented on MAPREDUCE-7137:
-
Thanks, [~jlowe] , for the review and committing!
MRAppBenchmark.benchmark1() passes successfully without changes in
MAPREDUCE-7138. MAPREDUCE-7138 needed only for MRAppBenchmark.benchmark2().
These tests failed for different unrelated reasons. That's why I created two
separated tickets for each test. But I agree with you that changes were small
and can be in one commit.
> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: MAPREDUCE-7137
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6
>
> Attachments: MAPREDUCE-7137.001.patch
>
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be
> finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7138) ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619822#comment-16619822
]
Hudson commented on MAPREDUCE-7138:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15000 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15000/])
MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should (jlowe:
rev 8382b860d4ef4f20d000537ded42a88e98bd2190)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
> ThrottledContainerAllocator in MRAppBenchmark should implement
> RMHeartbeatHandler
> -
>
> Key: MAPREDUCE-7138
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7138
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6
>
> Attachments: MAPREDUCE-7138.001.patch, MAPREDUCE-7138.002.patch,
> MAPREDUCE-7138.003.patch
>
>
> MRAppBenchmark#benchmark2 test fails with the following exception:
> {noformat}
> java.lang.ClassCastException:
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark$ThrottledMRApp$ThrottledContainerAllocator
> cannot be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMHeartbeatHandler
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getRMHeartbeatHandler(MRAppMaster.java:718)
> at
> org.apache.hadoop.mapreduce.v2.app.MRApp.createCommitterEventHandler(MRApp.java:665)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.serviceInit(MRApp.java:266)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:294)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:279)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:69)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark2(MRAppBenchmark.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> {noformat}
> since ThrottledContainerAllocator doesn't implement RMHeartbeatHandler
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619821#comment-16619821
]
Hudson commented on MAPREDUCE-7137:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15000 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15000/])
MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with (jlowe: rev
34b2237e420cfbe3a97ddd44968de8bbe1ed30ab)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: MAPREDUCE-7137
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6
>
> Attachments: MAPREDUCE-7137.001.patch
>
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be
> finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7138) ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-7138:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.6
3.1.2
2.9.2
3.0.4
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Thanks, [~oshevchenko]! I committed this to trunk, branch-3.1, branch-3.0,
branch-2, branch-2.9, and branch-2.8.
> ThrottledContainerAllocator in MRAppBenchmark should implement
> RMHeartbeatHandler
> -
>
> Key: MAPREDUCE-7138
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7138
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6
>
> Attachments: MAPREDUCE-7138.001.patch, MAPREDUCE-7138.002.patch,
> MAPREDUCE-7138.003.patch
>
>
> MRAppBenchmark#benchmark2 test fails with the following exception:
> {noformat}
> java.lang.ClassCastException:
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark$ThrottledMRApp$ThrottledContainerAllocator
> cannot be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMHeartbeatHandler
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getRMHeartbeatHandler(MRAppMaster.java:718)
> at
> org.apache.hadoop.mapreduce.v2.app.MRApp.createCommitterEventHandler(MRApp.java:665)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.serviceInit(MRApp.java:266)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:294)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:279)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:69)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark2(MRAppBenchmark.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> {noformat}
> since ThrottledContainerAllocator doesn't implement RMHeartbeatHandler
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-7137:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.6
3.1.2
2.9.2
3.0.4
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Thanks, [~oshevchenko]! I committed this to trunk, branch-3.1, branch-3.0,
branch-2, branch-2.9, and branch-2.8.
> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: MAPREDUCE-7137
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6
>
> Attachments: MAPREDUCE-7137.001.patch
>
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be
> finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7138) ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619790#comment-16619790
]
Jason Lowe commented on MAPREDUCE-7138:
---
Thanks for the report and patch! +1 lgtm. Committing this.
> ThrottledContainerAllocator in MRAppBenchmark should implement
> RMHeartbeatHandler
> -
>
> Key: MAPREDUCE-7138
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7138
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Attachments: MAPREDUCE-7138.001.patch, MAPREDUCE-7138.002.patch,
> MAPREDUCE-7138.003.patch
>
>
> MRAppBenchmark#benchmark2 test fails with the following exception:
> {noformat}
> java.lang.ClassCastException:
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark$ThrottledMRApp$ThrottledContainerAllocator
> cannot be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMHeartbeatHandler
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getRMHeartbeatHandler(MRAppMaster.java:718)
> at
> org.apache.hadoop.mapreduce.v2.app.MRApp.createCommitterEventHandler(MRApp.java:665)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.serviceInit(MRApp.java:266)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:294)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:279)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:69)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark2(MRAppBenchmark.java:268)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> {noformat}
> since ThrottledContainerAllocator doesn't implement RMHeartbeatHandler
> interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619783#comment-16619783
]
Jason Lowe commented on MAPREDUCE-7137:
---
Thanks for the report and patch!
+1 for the patch, although I noted that this alone doesn't allow the unit test
to pass. As soon as this is fixed it leads to the problem reported in
MAPREDUCE-7138. In the future it would be simpler and less overhead to fix all
the issues with the test in one patch, especially since the changes are so
small.
> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: MAPREDUCE-7137
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Attachments: MAPREDUCE-7137.001.patch
>
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be
> finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3801: -- Resolution: Fixed Fix Version/s: 2.8.6 3.1.2 2.9.2 3.0.4 3.2.0 2.10.0 Status: Resolved (was: Patch Available) > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6 > > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619728#comment-16619728 ] Hudson commented on MAPREDUCE-3801: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14997 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14997/]) MAPREDUCE-3801: (ericp: rev 2cf8927683d49f4cdc3cba038480128c473c5461) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619665#comment-16619665 ] Eric Payne commented on MAPREDUCE-3801: --- Thanks for the patch [~jlowe]. +1 Will commit shortly > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619457#comment-16619457 ] Jason Lowe commented on MAPREDUCE-3801: --- No unit test since this is fixing the code to get an existing flaky test to pass. > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619351#comment-16619351
]
Hadoop QA commented on MAPREDUCE-3801:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
31s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-3801 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940241/MAPREDUCE-3801.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3b9a6f83b572 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e4fca6a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7484/testReport/ |
| Max. process+thread count | 597 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7484/console |
| Powered b
[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3801: -- Attachment: MAPREDUCE-3801.001.patch > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3801: -- Target Version/s: 2.10.0, 3.2.0, 3.0.4, 2.9.2, 3.1.2, 2.8.6 Status: Patch Available (was: Open) > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-3801.001.patch, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-3801: - Assignee: Jason Lowe Ran into this again, as it started failing semi-reliably on our test machines. Not sure this is the complete issue, but certainly part of the problem is the speculator.eventQueueEmpty() call in the test to wait for any pending speculations. That method is checking a queue _that is never used_. The queue is therefore always empty, so the test never waits. The method should be waiting on the scanControl queue rather than the unused eventQueue. > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Robert Joseph Evans >Assignee: Jason Lowe >Priority: Major > Attachments: > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7134) Setup and cleanup progress information are not available after job completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-7134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618988#comment-16618988
]
Oleksandr Shevchenko commented on MAPREDUCE-7134:
-
Could someone review the patch?
Thanks.
> Setup and cleanup progress information are not available after job completion
> -
>
> Key: MAPREDUCE-7134
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7134
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Attachments: MAPREDUCE-7134.001.patch
>
>
> Information about "setupProgress" and "cleanupProgress" are not available
> after a job completion. These properties are ignored during creating
> JobReport since they were not added to jhist file.
> We can get this information during a job is running:
> {code}
> import org.apache.hadoop.mapred.JobClient
> import org.apache.hadoop.conf.Configuration
> val client = new JobClient(new Configuration())
> val job = client.getJob("job_1535984902382_0006")
> val setup = job.setupProgress()
> {code}
> When job runnin:
> {code}
> scala> val setup = job.setupProgress()
> setup: Float = 1.0
> {code}
> When job already finished:
> {code}
> scala> val setup = job.setupProgress()
> setup: Float = 0.0
> {code}
> JobReport interface includes getCleanupProgress and getSetupProgress methods.
> But these methods ignore during creating JobReport in
> CompletedJob#constructJobReport(). setupProgress and cleanupProgress
> properties set in JobImpl class but JobFinishedEvent and
> JobUnsuccessfulCompletionEvent do not include this information. As the
> result, these properties do not include in jhist file.
> We can add these properties to jhist file but of course, this is the
> incompatible change. These changes should be applied for a major release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
