Easy to repro Gary? I just add in your above patch and then loop? The first run completely goes down? The regionserver is waiting on all regions to close before it can go down (and then the master go down). St.Ack
On Wed, Oct 5, 2011 at 12:13 PM, Gary Helmling <[email protected]> wrote: > Something else seems to be going on. With the call to shutdownMiniCluster() > the first run of TestHLog passes. But when I try running in a loop, the > second run always seems to hang. > > Thread dump here: http://pastebin.com/f18Wfa3T > > > On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[email protected]> wrote: > >> +CC Roman who worked on the patch identified by the bisect. >> >> Roman, does Gary's analysis make sense to you? >> >> -Todd >> >> On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[email protected]> >> wrote: >> > Somehow TestHLog was never actually shutting down the mini-cluster? >> > >> > The following change lets the test exit successfully: >> > >> > diff --git >> > a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java >> > b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java >> > index 663b318..13f821c 100644 >> > --- >> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java >> > +++ >> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java >> > @@ -54,6 +54,7 @@ import >> > org.apache.hadoop.hdfs.server.namenode.LeaseManager; >> > import org.apache.hadoop.io.SequenceFile; >> > import org.apache.log4j.Level; >> > import org.junit.After; >> > +import org.junit.AfterClass; >> > import org.junit.Before; >> > import org.junit.BeforeClass; >> > import org.junit.Test; >> > @@ -120,6 +121,11 @@ public class TestHLog { >> > oldLogDir = new Path(hbaseDir, ".oldlogs"); >> > dir = new Path(hbaseDir, getName()); >> > } >> > + @AfterClass >> > + public static void tearDownAfterClass() throws Exception { >> > + TEST_UTIL.shutdownMiniCluster(); >> > + } >> > + >> > private static String getName() { >> > // TODO Auto-generated method stub >> > return "TestHLog"; >> > >> > >> > On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[email protected]> >> wrote: >> > >> >> I've noticed that TestHLog is currently hanging in trunk (haven't >> checked >> >> other branches). Oddly the tests actually complete, but then the test >> hangs >> >> in teardown. >> >> >> >> Seems to be something in the server shutdown hooks. git bisect tracks >> down >> >> the hang to this commit: >> >> >> >> commit 9c195c7ef350a932a9901a2069b96694d202c675 >> >> Author: Michael Stack <[email protected]> >> >> Date: Fri Sep 30 21:45:20 2011 +0000 >> >> >> >> HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping >> it >> >> >> >> git-svn-id: >> https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68 >> >> >> >> >> >> Anyone else noticed this on TestHLog or other tests? I think it may be >> >> behind some of our odd test cleanup issues up in Jenkins. >> >> >> >> --gh >> >> >> >> >> > >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> >
