That TestMultiSlaveReplication doesn't pass means something pretty fundamental was broken...
J-D On Tue, Jul 17, 2012 at 10:08 AM, Andrew Purtell <[email protected]> wrote: > I dispatched each unit test individually to 20 EC2 c1.mediums (64 bit > system, 2 VCPUs, kind of slow on purpose but still allowing some > thread concurrency). On the instance each test was run 100 times or > until failure. For each iteration after Maven exited the process table > was checked to see if any surefire processes lingered, and if so the > test would also be reported failed. > > OS: Amazon Linux AMI release 2012.03 > uname: Linux 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC > 2012 x86_64 x86_64 x86_64 GNU/Linux > JVM: java version "1.6.0_24" > OpenJDK Runtime Environment (IcedTea6 1.11.3) > (amazon-52.1.11.3.45.amzn1-x86_64) > OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > > Here are the tests that failed to complete successful runs in the above: > > TestCatalogTracker > hangs on a join in testServerNotRunningIOException waiting on a CT > that is stuck on CatalogTracker.waitForMeta and will linger in the > background > > TestColumnSeeking > > testDuplicateVersions(org.apache.hadoop.hbase.regionserver.TestColumnSeeking): > expected:<0> but was:<200> > > TestAtomicOperation > > testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation): > expected:<0> but was:<1> > > TestSplitLogManager > > testOrphanTaskAcquisition(org.apache.hadoop.hbase.master.TestSplitLogManager): > java.lang.AssertionError > > TestRegionRebalancing > > testRebalanceOnRegionServerNumberChange(org.apache.hadoop.hbase.TestRegionRebalancing): > After 5 attempts, region assignments were not balanced. > > TestDrainingServer > junit.framework.AssertionFailedError from > org.apache.hadoop.hbase.TestDrainingServer.setUpBeforeClass > > TestMasterObserver > > testTableOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): > org.apache.hadoop.hbase.InvalidFamilyOperationException: Column family > 'fam2' does not exist > > testRegionTransitionOperations(org.apache.hadoop.hbase.coprocessor.TestMasterObserver): > org.apache.hadoop.hbase.TableExistsException: observed_table > > TestServerCustomProtocol > > testSingleMethod(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol): > Results should contain region > test,bbb,1342509423473.2c0326188f899f3e91ec5eb623959c13. for row 'bbb' > > TestFromClientSide > testPoolBehavior(org.apache.hadoop.hbase.client.TestFromClientSide): > expected:<3> but was:<4> > > TestZooKeeper > testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper) > > TestReplication > > testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication): > Shutting down > > TestMasterReplication > > testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication): > Waited too much time for put replication > > TestMultiSlaveReplication > > testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication): > Unable to add peer > > TestReplicationPeer > > testResetZooKeeperSession(org.apache.hadoop.hbase.replication.TestReplicationPeer): > ReplicationPeer ZooKeeper session was not properly expired. > > I didn't get to them all before AWS yanked back my spot instances but > I ordered the list from most likely to least, the remaining tests were > in io.hfile.*, thrift.*, and util.* > > I'll circle back, confirm each individually, and open JIRAs with more detail. > > The cluster of replication test failures are a concern, but I've seen > in other environments such as this one that the tests are timing > dependent. On a slow or busy test system they can fail with "waited > too much time ...". So a solution for this is to not use the system > clock but instead EnvironmentEdge or whatever incremented only when > the test process has CPU time. I haven't looked into this in detail > yet. > > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein (via Tom White)
