Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/ [Jun 26, 2017 5:54:01 PM] (wang) HDFS-12032. Inaccurate comment on [Jun 26, 2017 6:20:07 PM] (wang) HDFS-11956. Do not require a storage ID or target storage IDs when [Jun 26, 2017 8:24:27 PM] (raviprak) HDFS-11993. Add log info when connect to datanode socket address failed. [Jun 26, 2017 10:43:50 PM] (lei) HDFS-12033. DatanodeManager picking EC recovery tasks should also [Jun 27, 2017 12:35:55 AM] (rkanter) MAPREDUCE-6904. HADOOP_JOB_HISTORY_OPTS should be [Jun 27, 2017 7:39:47 AM] (aengineer) HDFS-12045. Add log when Diskbalancer volume is transient storage type. [Jun 27, 2017 11:49:26 AM] (aajisaka) HDFS-12040. TestFsDatasetImpl.testCleanShutdownOfVolume fails. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-mvninstall-root.txt [500K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [488K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [76K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/358/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K]
[jira] [Created] (MAPREDUCE-6907) Hadoop "Spill failed" with custom ArrayWritable and data type - "java.lang.Exception: java.lang.NegativeArraySizeException"
Neel Chauhan created MAPREDUCE-6907: --- Summary: Hadoop "Spill failed" with custom ArrayWritable and data type - "java.lang.Exception: java.lang.NegativeArraySizeException" Key: MAPREDUCE-6907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6907 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.3 Environment: Ubuntu 16.04 (GNOME Desktop) on Dell Inspiron 7352, i7 5500U, 8GB RAM, 512GB SSD Reporter: Neel Chauhan When I run a MapReduce program which uses a custom ArrayWritable based on a custom data type, I get this error: java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1562) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1471) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2019) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NegativeArraySizeException at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:93) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146) at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170) at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:876) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1532) 17/06/26 16:41:54 INFO mapred.LocalJobRunner: map task executor complete. 17/06/26 16:41:54 WARN mapred.LocalJobRunner: job_local680262639_0001 java.lang.Exception: java.io.IOException: Spill failed at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1562) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$300(MapTask.java:876) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1372) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1349) at java.io.DataOutputStream.writeShort(DataOutputStream.java:167) at edu.nyu.cusp.umg.mapreduce.houghplane.CellWritable.write(CellWritable.java:55) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1149) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at edu.nyu.cusp.umg.mapreduce.houghplane.HoughTransformRasterCombiner$HoughTransformMapper.writeFull(HoughTransformRasterCombiner.java:253) at edu.nyu.cusp.umg.mapreduce.houghplane.HoughTransformRasterCombiner$HoughTransformMapper.map(HoughTransformRasterCombiner.java:158) at edu.nyu.cusp.umg.mapreduce.houghplane.HoughTransformRasterCombiner$HoughTransformMapper.map(HoughTransformRasterCombiner.java:59) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/ [Jun 26, 2017 7:41:00 AM] (aajisaka) HADOOP-14549. Use GenericTestUtils.setLogLevel when available in [Jun 26, 2017 8:26:09 AM] (kai.zheng) HDFS-11943. [Erasure coding] Warn log frequently print to screen in [Jun 26, 2017 12:39:47 PM] (stevel) HADOOP-14461 Azure: handle failure gracefully in case of missing account [Jun 26, 2017 5:54:01 PM] (wang) HDFS-12032. Inaccurate comment on [Jun 26, 2017 6:20:07 PM] (wang) HDFS-11956. Do not require a storage ID or target storage IDs when [Jun 26, 2017 8:24:27 PM] (raviprak) HDFS-11993. Add log info when connect to datanode socket address failed. [Jun 26, 2017 10:43:50 PM] (lei) HDFS-12033. DatanodeManager picking EC recovery tasks should also [Jun 27, 2017 12:35:55 AM] (rkanter) MAPREDUCE-6904. HADOOP_JOB_HISTORY_OPTS should be -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 719] Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:[line 455] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus() makes inefficient use of keySet iterator instead of entrySet iterator At ContainerLocalizer.java:keySet iterator instead of entrySet iterator At ContainerLocalizer.java:[line 334] org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics is a mutable collection which should be package protected At ContainerMetrics.java:which should be package protected At ContainerMetrics.java:[line 134] Failed junit tests : hadoop.ipc.TestProtoBufRpcServerHandoff hadoop.ipc.TestRPC hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport hadoop.hdfs.TestFileAppend hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.client.api.impl.TestAMRMProxy cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-compile-javac-root.txt [192K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/447/artifact/out/whitespace-tabs.txt [1.2M] findbugs:
[jira] [Created] (MAPREDUCE-6906) Print out current tokens in LauncherAM
Peter Cseh created MAPREDUCE-6906: - Summary: Print out current tokens in LauncherAM Key: MAPREDUCE-6906 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6906 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Peter Cseh We should print out the list of available tokens in the LauncherAM, similar to what MapReduce does: https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1709 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
