[VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0
Hi all, As always, thanks to the many, many contributors who helped with this release! I've prepared an RC0 for 3.0.0-alpha4: http://home.apache.org/~wang/3.0.0-alpha4-RC0/ The standard 5-day vote would run until midnight on Tuesday, July 4th. Given that July 4th is a holiday in the US, I expect this vote might have to be extended, but I'd like to close the vote relatively soon after. I've done my traditional testing of a pseudo-distributed cluster with a single task pi job, which was successful. Normally my testing would end there, but I'm slightly more confident this time. At Cloudera, we've successfully packaged and deployed a snapshot from a few days ago, and run basic smoke tests. Some bugs found from this include HDFS-11956, which fixes backwards compat with Hadoop 2 clients, and the revert of HDFS-11696, which broke NN QJM HA setup. Vijay is working on a test run with a fuller test suite (the results of which we can hopefully post soon). My +1 to start, Best, Andrew
Heads up: branching 3.0.0-alpha4, use -beta1 for new commits to trunk
Hi folks, I'm in the process of moving out all the JIRA versions from 3.0.0-alpha4 to 3.0.0-beta1 in preparation for a 3.0.0-alpha4 release. I'm hoping to get an RC up tomorrow for a vote, with possibly extending it given the holiday week in the US for July 4th. Please use the beta1 target/fix version from now on unless committing to branch-3.0.0-alpha4. We're not planning on an alpha5. Best, Andrew
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/ [Jun 28, 2017 5:54:25 PM] (weichiu) HDFS-11870. Add CLI cmd to enable/disable an erasure code policy. [Jun 28, 2017 8:17:58 PM] (templedf) YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description [Jun 28, 2017 8:47:23 PM] (lei) Add -E option in 'ls' to list erasure coding policy of each file and [Jun 28, 2017 8:53:54 PM] (weichiu) HADOOP-14297. Update the documentation about the new ec codecs config [Jun 28, 2017 9:18:59 PM] (liuml07) HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if [Jun 28, 2017 10:32:04 PM] (junping_du) YARN-5311. Document graceful decommission CLI and usage. Contributed by [Jun 28, 2017 10:40:58 PM] (sunilg) YARN-6280. Introduce deselect query param to skip ResourceRequest from [Jun 29, 2017 12:33:44 AM] (aw) MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl [Jun 29, 2017 3:26:36 AM] (kai.zheng) HADOOP-14479. Erasurecode testcase failures with native enabled. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-mvninstall-root.txt [500K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [468K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/360/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K]
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/ [Jun 28, 2017 6:49:09 AM] (xiao) HADOOP-14515. Addendum. Specifically configure zookeeper-related log [Jun 28, 2017 9:22:13 AM] (stevel) HADOOP-14190. Add more on S3 regions to the s3a documentation. [Jun 28, 2017 5:54:25 PM] (weichiu) HDFS-11870. Add CLI cmd to enable/disable an erasure code policy. [Jun 28, 2017 8:17:58 PM] (templedf) YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description [Jun 28, 2017 8:47:23 PM] (lei) Add -E option in 'ls' to list erasure coding policy of each file and [Jun 28, 2017 8:53:54 PM] (weichiu) HADOOP-14297. Update the documentation about the new ec codecs config [Jun 28, 2017 9:18:59 PM] (liuml07) HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if [Jun 28, 2017 10:32:04 PM] (junping_du) YARN-5311. Document graceful decommission CLI and usage. Contributed by [Jun 28, 2017 10:40:58 PM] (sunilg) YARN-6280. Introduce deselect query param to skip ResourceRequest from [Jun 29, 2017 12:33:44 AM] (aw) MAPREDUCE-6536. hadoop-pipes doesn't use maven properties for openssl [Jun 29, 2017 3:26:36 AM] (kai.zheng) HADOOP-14479. Erasurecode testcase failures with native enabled. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 719] Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:[line 455] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus() makes inefficient use of keySet iterator instead of entrySet iterator At ContainerLocalizer.java:keySet iterator instead of entrySet iterator At ContainerLocalizer.java:[line 334] org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics is a mutable collection which should be package protected At ContainerMetrics.java:which should be package protected At ContainerMetrics.java:[line 134] Failed junit tests : hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 hadoop.fs.contract.hdfs.TestHDFSContractCreate hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestAMRMClient cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-compile-javac-root.txt [192K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/449/artifact/out/diff-patch-shelldocs.txt [12K] whitespace:
