Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
+1 (binding) - Verfied signatures and digests- Performed a native build from the release tag- Deployed to a single node cluster- Ran some sample jobs Jason On Friday, March 17, 2017 4:18 AM, Junping Duwrote: Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Thanks Junping for doing this. +1 (non-binding) 0. Download the src tar.gz file; checked the MD5 checksum 1. Build Hadoop from source successfully 2. Deploy a single node cluster and start the cluster successfully 3. Operate the HDFS from command line: ls, put, distcp, dfsadmin etc 4. Run hadoop mapreduce examples: grep 5. Operate AWS S3 using S3A schema from commandline: ls, cat, distcp 6. Check the HDFS service logs L > On Mar 17, 2017, at 2:18 AM, Junping Duwrote: > > Hi all, > With fix of HDFS-11431 get in, I've created a new release candidate (RC3) > for Apache Hadoop 2.8.0. > > This is the next minor release to follow up 2.7.0 which has been released > for more than 1 year. It comprises 2,900+ fixes, improvements, and new > features. Most of these commits are released for the first time in branch-2. > > More information about the 2.8.0 release plan can be found here: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > New RC is available at: > http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 > > The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: > 91f2b7a13d1e97be65db92ddabc627cc29ac0009 > > The maven artifacts are available via repository.apache.org at: > https://repository.apache.org/content/repositories/orgapachehadoop-1057 > > Please try the release and vote; the vote will run for the usual 5 days, > ending on 03/22/2017 PDT time. > > Thanks, > > Junping
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Thanks for the new RC, Junping. I built from source and tried it out on a 2-node cluster with HA enabled. I ran a pi job and some streaming jobs. I tested that localization and failover work correctly, and I played a little with the YARN and HDFS web UIs. I did encounter an old friend of mine, which is that if you submit a streaming job with input that is only 1 block, you will nonetheless get 2 mappers that both process the same split. What's new this time is that the second mapper was consistently failing on certain input sizes. I (re)verified that the issue also exists is 2.7.3, so it's not a regression. I'm pretty sure it's been there since at least 2.6.0. I filed MAPREDUCE-6864 for it. Given that my issue was not a regression, I'm +1 on the RC. Daniel On 3/17/17 2:18 AM, Junping Du wrote: Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
+1 Thanks, Junping, for your efforts to get this release out. I downloaded and built the source, and i did the following manual testing on a 2-node pseudo cluster: - Streaming job - Inter-queue (cross-queue) preemption--verified that only the expected amount of preemptions occured. - Inter-queue (in-queue) preemption with higher priority apps preempting lower ones. - Limited node label testing. - Yarn distributed shell, both with and without keeping containers across AM restart. - Killing apps from Application UI From: Junping DuTo: "common-dev@hadoop.apache.org" ; "hdfs-...@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" Sent: Friday, March 17, 2017 4:18 AM Subject: [VOTE] Release Apache Hadoop 2.8.0 (RC3) Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14196) Azure Data Lake doc is missing required config entry
Atul Sikaria created HADOOP-14196: - Summary: Azure Data Lake doc is missing required config entry Key: HADOOP-14196 URL: https://issues.apache.org/jira/browse/HADOOP-14196 Project: Hadoop Common Issue Type: Bug Components: fs/adl Affects Versions: 3.0.0-alpha2 Reporter: Atul Sikaria Assignee: Atul Sikaria The index.md for adl file system is missing one of the config entries needed for setting up OAuth with client credentials. Users need to set the key dfs.adls.oauth2.access.token.provider.type = ClientCredential, but the instructions do not say that. This has led to people not being able to connect to the backend after setting up a cluster with ADL. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/ [Mar 16, 2017 2:08:30 PM] (stevel) HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 2:30:10 PM] (jlowe) YARN-4051. ContainerKillEvent lost when container is still recovering [Mar 16, 2017 3:54:59 PM] (kihwal) HDFS-10601. Improve log message to include hostname when the NameNode is [Mar 16, 2017 7:06:51 PM] (stevel) Revert "HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 7:20:46 PM] (jitendra) HDFS-11533. reuseAddress option should be used for child channels in [Mar 16, 2017 10:07:38 PM] (wang) HDFS-10530. BlockManager reconstruction work scheduling should correctly [Mar 16, 2017 11:08:32 PM] (liuml07) HADOOP-14191. Duplicate hadoop-minikdc dependency in hadoop-common [Mar 17, 2017 1:13:43 AM] (arp) HDFS-10394. move declaration of okhttp version from hdfs-client to -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.app.TestRuntimeEstimators hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapreduce.TestMRJobClient Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [136K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [236K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K]
[jira] [Created] (HADOOP-14195) CredentialProviderFactory is not thread-safe
Vihang Karajgaonkar created HADOOP-14195: Summary: CredentialProviderFactory is not thread-safe Key: HADOOP-14195 URL: https://issues.apache.org/jira/browse/HADOOP-14195 Project: Hadoop Common Issue Type: Bug Reporter: Vihang Karajgaonkar Multi-threaded access to CredentialProviderFactory is not thread-safe because {{java.util.ServiceLoader}} is not thread-safe (as noted in its Java doc). Thanks to [~jzhuge] I was able to reproduce this issue but creating a simple multi-threaded application which executes the following code in parallel. {code:java} for (int i = 0; i < ITEMS; i++) { futures.add(executor.submit(new Callable() { @Override public Void call() throws Exception { boolean found = false; for (CredentialProviderFactory factory : serviceLoader) { CredentialProvider kp = factory.createProvider(uri, conf); if (kp != null) { result.add(kp); found = true; break; } } if (!found) { throw new IOException(Thread.currentThread() + "No CredentialProviderFactory for " + uri); } else { System.out.println(Thread.currentThread().getName() + " found credentialProvider for " + path); } return null; } })); } {code} I see the following exception trace when I execute the above code. {code:java} java.util.concurrent.ExecutionException: java.util.NoSuchElementException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at TestCredentialProvider.main(TestCredentialProvider.java:58) Caused by: java.util.NoSuchElementException at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:615) at java.net.URLClassLoader$3.nextElement(URLClassLoader.java:590) at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:61) at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357) at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393) at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474) at TestCredentialProvider$1.call(TestCredentialProvider.java:38) at TestCredentialProvider$1.call(TestCredentialProvider.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} I also see a NPE sometimes {code:java} java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at TestCredentialProvider.main(TestCredentialProvider.java:58) Caused by: java.lang.NullPointerException at java.util.ServiceLoader.parse(ServiceLoader.java:304) at java.util.ServiceLoader.access$200(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357) at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393) at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474) at TestCredentialProvider$1.call(TestCredentialProvider.java:38) at TestCredentialProvider$1.call(TestCredentialProvider.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/ [Mar 16, 2017 2:08:30 PM] (stevel) HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 2:30:10 PM] (jlowe) YARN-4051. ContainerKillEvent lost when container is still recovering [Mar 16, 2017 3:54:59 PM] (kihwal) HDFS-10601. Improve log message to include hostname when the NameNode is [Mar 16, 2017 7:06:51 PM] (stevel) Revert "HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 7:20:46 PM] (jitendra) HDFS-11533. reuseAddress option should be used for child channels in [Mar 16, 2017 10:07:38 PM] (wang) HDFS-10530. BlockManager reconstruction work scheduling should correctly [Mar 16, 2017 11:08:32 PM] (liuml07) HADOOP-14191. Duplicate hadoop-minikdc dependency in hadoop-common [Mar 17, 2017 1:13:43 AM] (arp) HDFS-10394. move declaration of okhttp version from hdfs-client to -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.TestReadStripedFileWithMissingBlocks hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-compile-javac-root.txt [180K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [296K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 2.8.0 (RC3)
Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping