Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)
Thanks for driving the release, Junping! +1 (binding) - Verified signatures and digests - Successfully performed a native build from source - Successfully deployed a single-node cluster with the timeline server - Ran some sample jobs and examined the web UI and job logs Jason On Mon, Sep 10, 2018 at 7:00 AM, 俊平堵 wrote: > Hi all, > > I've created the first release candidate (RC0) for Apache > Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It > includes 33 important fixes and improvements. > > > The RC artifacts are available at: > http://home.apache.org/~junping_du/hadoop-2.8.5-RC0 > > > The RC tag in git is: release-2.8.5-RC0 > > > > The maven artifacts are available via repository.apache.org< > http://repository.apache.org> at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1140 > > > Please try the release and vote; the vote will run for the usual 5 > working > days, ending on 9/15/2018 PST time. > > > Thanks, > > > Junping >
[jira] [Created] (MAPREDUCE-7138) ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler
Oleksandr Shevchenko created MAPREDUCE-7138:
---
Summary: ThrottledContainerAllocator in MRAppBenchmark should
implement RMHeartbeatHandler
Key: MAPREDUCE-7138
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7138
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Oleksandr Shevchenko
Assignee: Oleksandr Shevchenko
MRAppBenchmark#benchmark2 test fails with the following exception:
{noformat}
java.lang.ClassCastException:
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark$ThrottledMRApp$ThrottledContainerAllocator
cannot be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMHeartbeatHandler
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getRMHeartbeatHandler(MRAppMaster.java:718)
at
org.apache.hadoop.mapreduce.v2.app.MRApp.createCommitterEventHandler(MRApp.java:665)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
at org.apache.hadoop.mapreduce.v2.app.MRApp.serviceInit(MRApp.java:266)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:294)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:279)
at
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:69)
at
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark2(MRAppBenchmark.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
{noformat}
since ThrottledContainerAllocator doesn't implement RMHeartbeatHandler
interface.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/ [Sep 9, 2018 5:07:30 AM] (msingh) HDDS-410. ozone scmcli list is not working properly. Contributed by [Sep 9, 2018 5:50:26 PM] (brahma) HDFS-13862. RBF: Router logs are not capturing few of the dfsrouteradmin [Sep 10, 2018 12:45:31 AM] (wangda) YARN-8698. [Submarine] Failed to reset Hadoop home environment when [Sep 10, 2018 3:40:51 AM] (vinayakumarb) HDFS-13806. EC: No error message for unsetting EC policy of the [Sep 10, 2018 3:52:59 AM] (vinayakumarb) HDFS-13895. EC: Fix Intermittent Failure in -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine Unread field:FSBasedSubmarineStorageImpl.java:[line 39] Found reliance on default encoding in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component):in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component): new java.io.FileWriter(File) At YarnServiceJobSubmitter.java:[line 195] org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component) may fail to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:[line 195] is not discharged org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.web.TestWebHdfsTimeouts cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-compile-javac-root.txt [304K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-checkstyle-root.txt [17M] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-ozone_c
[jira] [Created] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
Oleksandr Shevchenko created MAPREDUCE-7137:
---
Summary: MRAppBenchmark.benchmark1() fails with
NullPointerException
Key: MAPREDUCE-7137
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Oleksandr Shevchenko
MRAppBenchmark.benchmark1() fails with NullPointerException:
1. We do not set any queue for this test. As the result we got the following
exception:
{noformat}
2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator
(RMCommunicator.java:register(177)) - Exception while registering
java.lang.NullPointerException
at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
at org.apache.avro.util.Utf8.(Utf8.java:39)
at
org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
at
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
{noformat}
2. We override createSchedulerProxy method and do not set application priority
that was added later by MAPREDUCE-6515. We got the following error:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
at java.lang.Thread.run(Thread.java:748)
{noformat}
In both cases, the job never will be run and the test stuck and will not be
finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 2.8.5 (RC0)
Hi all, I've created the first release candidate (RC0) for Apache Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It includes 33 important fixes and improvements. The RC artifacts are available at: http://home.apache.org/~junping_du/hadoop-2.8.5-RC0 The RC tag in git is: release-2.8.5-RC0 The maven artifacts are available via repository.apache.org< http://repository.apache.org> at: https://repository.apache.org/content/repositories/orgapachehadoop-1140 Please try the release and vote; the vote will run for the usual 5 working days, ending on 9/15/2018 PST time. Thanks, Junping
[jira] [Created] (MAPREDUCE-7136) TestMRAppMetrics should shutdown DefaultMetricsSystem after completion
Oleksandr Shevchenko created MAPREDUCE-7136:
---
Summary: TestMRAppMetrics should shutdown DefaultMetricsSystem
after completion
Key: MAPREDUCE-7136
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7136
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Oleksandr Shevchenko
Attachments: image-2018-09-10-14-39-57-992.png
TestMRAppMetrics should invoke shutdown method in DefaultMetricsSystem after
completion. Since it can lead to failing other tests. For example,
TestRMContainerAllocator#testReportedAppProgress fails when run after
TestMRAppMetrics#testNames with the following error:
{noformat}
org.apache.hadoop.metrics2.MetricsException: Metrics source MRAppMetrics
already exists!
{noformat}
!image-2018-09-10-14-39-57-992.png!
We do not catch this on the trunk since the test
TestRMContainerAllocator#testUnsupportedMapContainerRequirement run first and
"DefaultMetricsSystem.shutdown();" invokes after completion. But since JUnit
does not guarantee the order of the tests we should fix it.
Also, this is affected by previous versions which run testReportedAppProgress
first (I faced it on 2.7.0 version).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7135) TestTaskAttemptContainerRequest should reset UserGroupInformation
Oleksandr Shevchenko created MAPREDUCE-7135: --- Summary: TestTaskAttemptContainerRequest should reset UserGroupInformation Key: MAPREDUCE-7135 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7135 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Oleksandr Shevchenko TestTaskAttemptContainerRequest should reset UserGroupInformation after end since this test cache UserGroupInformation and can be cause of other test fail. For example, all test in TestRMContainerAllocator will be failed if we run these tests after TestTaskAttemptContainerRequest (or we can change "UserGroupInformation.setLoginUser(null)" with UserGroupInformation.reset() in this class). Also, I think the WARNING message in TestTaskAttemptContainerRequest can be removed since we can reset UserGroupInformation after each test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
