It also looks like TestMasterFailover isn't cleaning up after itself. Each test method just does TEST_UTIL.shutdownMiniCluster() at the end, without use of a finally block. So it wouldn't get called in the case of an exception. Testing a patch for that now.
On Wed, Nov 23, 2011 at 9:52 AM, Gary Helmling <[email protected]> wrote: > I think this might be caused by: > https://issues.apache.org/jira/browse/HBASE-4857 > > There is a recursive loop on the leader > AuthenticationTokenSecretManger when a KeeperException is encountered. > > > > On Wed, Nov 23, 2011 at 8:53 AM, Ted Yu <[email protected]> wrote: >> The heap space error happened again for build #9 >> >> TestMasterFailover hung in build 9 too. >> >> On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[email protected]> wrote: >> >>> The interesting thing is that their counterparts in HBase-0.92 build >>> didn't run out of heap :-) >>> >>> >>> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[email protected]> wrote: >>> >>>> Usually IME this means that some test is generating many GB of logs, >>>> etc... before raising heap let's look and see if some of the tests are >>>> being excessively noisy >>>> >>>> -Todd >>>> >>>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[email protected]> wrote: >>>> > Hi, >>>> > I found the following in >>>> > >>>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console >>>> > : >>>> > >>>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting >>>> > Exception in thread "ThreadedStreamConsumer" >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > at java.util.Arrays.copyOf(Arrays.java:2882) >>>> > at >>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) >>>> > at >>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) >>>> > at java.lang.StringBuffer.append(StringBuffer.java:224) >>>> > at >>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201) >>>> > at >>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139) >>>> > at >>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112) >>>> > Running org.apache.hadoop.hbase.master.TestMasterFailover >>>> > Exception in thread "ThreadedStreamConsumer" >>>> > java.lang.OutOfMemoryError: Java heap space >>>> > at java.util.Arrays.copyOf(Arrays.java:2882) >>>> > at >>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) >>>> > at >>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) >>>> > at java.lang.StringBuffer.append(StringBuffer.java:224) >>>> > at >>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201) >>>> > at >>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139) >>>> > >>>> > Currently each unit test is given 1.9GB of heap. >>>> > I think we should give more heap to the tests. >>>> > >>>> > Cheers >>>> > >>>> >>>> >>>> >>>> -- >>>> Todd Lipcon >>>> Software Engineer, Cloudera >>>> >>> >>> >> >
