[jira] [Resolved] (MAPREDUCE-6987) JHS Log Scanner and Cleaner blocked

[ https://issues.apache.org/jira/browse/MAPREDUCE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt resolved MAPREDUCE-6987. - Resolution: Fixed Fix Version/s: 3.1.0 3.0.0

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer > wrote: > > > > With no other information or access to go on, my current hunch is that one of > the HDFS unit tests is ballooning in memory size. The easiest way to kill a > Linux machine is to eat all of the

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/10/ [Oct 23, 2017 4:58:04 PM] (epayne) YARN-4163: Audit getQueueInfo and getApplications calls [Oct 23, 2017 5:47:35 PM] (arp) HDFS-12683. DFSZKFailOverController re-order logic for logging [Oct 23, 2017

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

+1 (non-binding) - Verified all hashes and checksums - Built from source on macOS 10.12.6, Java 1.8.0u65 - Deployed a pseudo cluster - Ran some example jobs Thanks, Eric On Tue, Oct 24, 2017 at 12:59 AM, Mukul Kumar Singh wrote: > Thanks Junping, > > +1 (non-binding)

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/567/ [Oct 23, 2017 4:43:41 PM] (epayne) YARN-4163: Audit getQueueInfo and getApplications calls [Oct 23, 2017 5:47:16 PM] (arp) HDFS-12683. DFSZKFailOverController re-order logic for logging [Oct 23, 2017

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Thanks for all your hard work Junping! * Checked signature. * Ran a sleep job. * Checked NN File browser UI works. +1 (binding) Cheers Ravi On Tue, Oct 24, 2017 at 12:26 PM, Rakesh Radhakrishnan wrote: > Thanks Junping for getting this out. > > +1 (non-binding) > > *

[jira] [Created] (MAPREDUCE-6990) FileInputStream.skip function can return 0 when the file is corrupted, causing an infinite loop

Ting Dai created MAPREDUCE-6990: --- Summary: FileInputStream.skip function can return 0 when the file is corrupted, causing an infinite loop Key: MAPREDUCE-6990 URL:

[jira] [Created] (MAPREDUCE-6991) getRogueTaskPID and testProcessTree hangs when the file creation failed

Ting Dai created MAPREDUCE-6991: --- Summary: getRogueTaskPID and testProcessTree hangs when the file creation failed Key: MAPREDUCE-6991 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6991 Project:

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

My plan is currently to: * switch some of Hadoop’s Yetus jobs over to my branch with the YETUS-561 patch to test it out. * if the tests work, work on getting YETUS-561 committed to yetus master * switch jobs back to ASF yetus master either post-YETUS-561 or without it if it doesn’t work * go

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Sean/Junping- Ignoring the epistemology, it's a problem. Let's figure out what's causing memory to balloon and then we can work out the appropriate remedy. Is this reproducible outside the CI environment? To Junping's point, would YETUS-561 provide more detailed information to aid debugging? -C

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

> On Oct 24, 2017, at 4:10 PM, Andrew Wang wrote: > > FWIW we've been running branch-3.0 unit tests successfully internally, though > we have separate jobs for Common, HDFS, YARN, and MR. The failures here are > probably a property of running everything in the same

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Allen, Do we have any solid evidence to show the HDFS unit tests going through the roof are due to serious memory leak by HDFS? Normally, I don't expect memory leak are identified in our UTs - mostly, it (test jvm gone) is just because of test or deployment issues. Unless there is

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

FWIW we've been running branch-3.0 unit tests successfully internally, though we have separate jobs for Common, HDFS, YARN, and MR. The failures here are probably a property of running everything in the same JVM, which I've found problematic in the past due to OOMs. On Tue, Oct 24, 2017 at 4:04

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Just curious, Junping what would "solid evidence" look like? Is the supposition here that the memory leak is within HDFS test code rather than library runtime code? How would such a distinction be shown? On Tue, Oct 24, 2017 at 4:06 PM, Junping Du wrote: > Allen, > Do

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

In general, the "solid evidence" of memory leak comes from analysis of heapdump, jastack, gc log, etc. In many cases, we can locate/conclude which piece of code are leaking memory from the analysis. Unfortunately, I cannot find any conclusion from previous comments and it even cannot tell

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Thanks Junping for getting this out. +1 (non-binding) * Built from source on CentOS 7.3.1611, jdk1.8.0_111 * Deployed 3 node cluster * Ran some sample jobs * Ran balancer * Operate HDFS from command line: ls, put, dfsadmin etc * HDFS Namenode UI looks good Thanks, Rakesh On Fri, Oct 20, 2017

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

Allen, can we bump up the maven surefire heap size to max (if it already is not) for the branch-2 nightly build and see if it helps? Thanks, Subru On Tue, Oct 24, 2017 at 4:22 PM, Allen Wittenauer wrote: > > > On Oct 24, 2017, at 4:10 PM, Andrew Wang