Somehow there's a ZK session expiration that I've not seen before...
2010-12-14 22:20:25,284 ERROR [Thread-282-EventThread]
zookeeper.ClientCnxn$EventThread(532): Error while calling watcher
java.lang.RuntimeException: Fatal ZK error, why=unittest-0x12ce6f79fca0009
unittest-0x12ce6f79fca0009 received expired from ZooKeeper, aborting
at
org.apache.hadoop.hbase.master.TestMasterFailover$1.abort(TestMasterFailover.java:236)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:328)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:246)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
... 4 more
So there are no RS online and the master is just sitting there waiting for RS
to show up. Eventually the test times out.
Need to trace through log more but not yet able to reproduce locally.
> -----Original Message-----
> From: jdcry...@gmail.com [mailto:jdcry...@gmail.com] On Behalf Of Jean-
> Daniel Cryans
> Sent: Tuesday, December 14, 2010 2:40 PM
> To: dev@hbase.apache.org
> Subject: Re: Build failed in Hudson: hbase-0.90 #27
>
> We should probably fix that test at some point...
>
> On Tue, Dec 14, 2010 at 2:35 PM, Apache Hudson Server
> <hud...@hudson.apache.org> wrote:
> > See <https://hudson.apache.org/hudson/job/hbase-0.90/27/changes>
> >
> > Changes:
> >
> > [jdcryans] HBASE-3355 Stopping a stopped cluster leaks an HMaster
> > HBASE-3356 Add more checks in replication if RS is stopped
> >
> > [jgray] HBASE-3332 Regions stuck in transition after RS failure
> >
> > [jgray] HBASE-3337 Restore HBCK fix of unassignment and dupe
> > assignment for new master
> >
> > ------------------------------------------
> > [...truncated 2734 lines...]
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.193
> > sec Running org.apache.hadoop.hbase.executor.TestExecutorService
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.048
> > sec Running org.apache.hadoop.hbase.client.TestFromClientSide
> > Tests run: 41, Failures: 0, Errors: 0, Skipped: 3, Time elapsed:
> > 329.059 sec Running
> > org.apache.hadoop.hbase.replication.TestReplication
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 159.547 sec Running
> > org.apache.hadoop.hbase.regionserver.TestCompaction
> > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 126.447 sec Running
> > org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054
> > sec Running
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.258
> > sec Running
> > org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031
> > sec Running
> > org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.105
> > sec Running org.apache.hadoop.hbase.master.TestClockSkewDetection
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042
> > sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 168.791 sec Running
> > org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.728
> > sec Running
> > org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.417
> > sec Running
> org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker
> > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.176
> > sec Running org.apache.hadoop.hbase.client.TestMetaScanner
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.34
> > sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034
> > sec Running org.apache.hadoop.hbase.TestRegionRebalancing
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.299
> > sec Running
> > org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
> > sec Running org.apache.hadoop.hbase.filter.TestInclusiveStopFilter
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
> > sec Running org.apache.hadoop.hbase.rest.TestTableResource
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.687
> > sec Running org.apache.hadoop.hbase.TestHBaseTestingUtility
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 113.548 sec Running
> > org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.249
> > sec Running org.apache.hadoop.hbase.rest.TestSchemaResource
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.451
> > sec Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.665
> > sec Running
> > org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015
> > sec Running org.apache.hadoop.hbase.rest.model.TestTableListModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.141
> > sec Running
> org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.038
> > sec Running org.apache.hadoop.hbase.regionserver.TestStoreFile
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.885
> > sec Running
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.469
> > sec Running org.apache.hadoop.hbase.rest.model.TestScannerModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145
> > sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.213
> > sec Running org.apache.hadoop.hbase.rest.TestVersionResource
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.366
> > sec Running org.apache.hadoop.hbase.filter.TestFilter
> > Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.074
> > sec Running org.apache.hadoop.hbase.regionserver.TestWideScanner
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.768
> > sec Running org.apache.hadoop.hbase.regionserver.TestSplitTransaction
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.848
> > sec Running
> > org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.763
> > sec Running
> > org.apache.hadoop.hbase.regionserver.TestGetClosestAtOrBefore
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.675
> > sec Running
> > org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.021
> > sec Running org.apache.hadoop.hbase.util.TestMergeTable
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.811
> > sec Running org.apache.hadoop.hbase.rest.model.TestVersionModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147
> > sec Running org.apache.hadoop.hbase.rest.model.TestCellSetModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.148
> > sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.159
> > sec Running
> > org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.156
> > sec Running org.apache.hadoop.hbase.avro.TestAvroServer
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.185
> > sec Running org.apache.hadoop.hbase.TestFullLogReconstruction
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.408
> > sec Running org.apache.hadoop.hbase.TestSerialization
> > Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.077
> > sec Running org.apache.hadoop.hbase.regionserver.TestKeyValueHeap
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142
> > sec Running org.apache.hadoop.hbase.regionserver.TestStoreScanner
> > Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.035
> > sec Running org.apache.hadoop.hbase.TestHMsg Tests run: 2, Failures:
> > 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec Running
> > org.apache.hadoop.hbase.io.hfile.TestReseekTo
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.239
> > sec Running org.apache.hadoop.hbase.regionserver.TestHRegionInfo
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
> > sec Running
> > org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016
> > sec Running org.apache.hadoop.hbase.util.TestKeying
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016
> > sec Running org.apache.hadoop.hbase.regionserver.TestScanner
> > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.425
> > sec Running org.apache.hadoop.hbase.util.TestHBaseFsck
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.167
> > sec Running org.apache.hadoop.hbase.master.TestRestartCluster
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.983
> > sec Running org.apache.hadoop.hbase.client.TestShell
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.955
> > sec Running org.apache.hadoop.hbase.rest.model.TestTableInfoModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142
> > sec Running org.apache.hadoop.hbase.TestInfoServers
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.727
> > sec Running org.apache.hadoop.hbase.master.TestLogsCleaner
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.19
> > sec Running
> > org.apache.hadoop.hbase.regionserver.TestMasterAddressManager
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.37
> > sec Running org.apache.hadoop.hbase.regionserver.TestQueryMatcher
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.158
> > sec Running org.apache.hadoop.hbase.zookeeper.TestZKTable
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.318
> > sec Running org.apache.hadoop.hbase.client.TestAdmin
> > Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 527.658 sec Running
> > org.apache.hadoop.hbase.regionserver.TestPriorityCompactionQueue
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.075
> > sec Running org.apache.hadoop.hbase.util.TestMergeTool
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.237
> > sec Running org.apache.hadoop.hbase.TestAcidGuarantees
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.016
> > sec Running org.apache.hadoop.hbase.util.TestBase64
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018
> > sec Running org.apache.hadoop.hbase.regionserver.TestMemStore
> > Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.429
> > sec Running org.apache.hadoop.hbase.rest.TestScannerResource
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.868
> > sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.062
> > sec Running org.apache.hadoop.hbase.master.TestDeadServer
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019
> > sec Running org.apache.hadoop.hbase.rest.TestGzipFilter
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.945
> > sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.015
> > sec Running org.apache.hadoop.hbase.TestKeyValue
> > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026
> > sec Running org.apache.hadoop.hbase.client.TestMultiParallel
> > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.235
> > sec Running org.apache.hadoop.hbase.master.TestMaster
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86 sec
> > Running org.apache.hadoop.hbase.master.TestRollingRestart
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 103.324 sec Running org.apache.hadoop.hbase.client.TestResult
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.038
> > sec Running org.apache.hadoop.hbase.client.TestHTablePool
> > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.927
> > sec Running org.apache.hadoop.hbase.rest.model.TestTableSchemaModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147
> > sec Running org.apache.hadoop.hbase.util.TestByteBloomFilter
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.571
> > sec Running org.apache.hadoop.hbase.filter.TestFilterList
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043
> > sec Running
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 333.16 sec Running org.apache.hadoop.hbase.regionserver.TestHRegion
> > Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 32.966 sec Running org.apache.hadoop.hbase.master.TestMasterFailover
> > Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 243.61 sec
> > <<<
> FAILURE!
> > Running org.apache.hadoop.hbase.rest.model.TestTableRegionModel
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034
> > sec Running org.apache.hadoop.hbase.client.TestTimestampsFilter
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.614
> > sec Running org.apache.hadoop.hbase.master.TestLoadBalancer
> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.218
> > sec Running org.apache.hadoop.hbase.util.TestRootPath
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021
> > sec Running org.apache.hadoop.hbase.client.TestScannerTimeout
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.937
> > sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.08
> > sec Running org.apache.hadoop.hbase.regionserver.wal.TestWALObserver
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.474
> > sec Running org.apache.hadoop.hbase.master.TestActiveMasterManager
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512
> > sec Running
> > org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource
> > Manager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> > elapsed: 0.876 sec Running org.apache.hadoop.hbase.util.TestBytes
> > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023
> > sec Running org.apache.hadoop.hbase.util.TestCompressionTest
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034
> > sec Running org.apache.hadoop.hbase.client.TestGetRowVersions
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.343
> > sec Running org.apache.hadoop.hbase.filter.TestDependentColumnFilter
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.386
> > sec Running org.apache.hadoop.hbase.rest.model.TestCellModel
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147
> > sec Running org.apache.hadoop.hbase.master.TestCatalogJanitor
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.558
> > sec Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.313
> > sec Running org.apache.hadoop.hbase.replication.TestReplicationSource
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.583
> > sec Running org.apache.hadoop.hbase.rest.TestRowResource
> > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 69.507 sec Running org.apache.hadoop.hbase.io.hfile.TestHFile
> > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.382
> > sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.389
> > sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
> > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045
> > sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.324
> > sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> > 108.305 sec Running org.apache.hadoop.hbase.client.TestTimestamp
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.817
> > sec Running org.apache.hadoop.hbase.TestScanMultipleVersions
> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.306
> > sec Running
> > org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03
> > sec Running org.apache.hadoop.hbase.client.TestHCM
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.917
> > sec
> >
> > Results :
> >
> > Tests in error:
> >
> > Tests run: 607, Failures: 0, Errors: 1, Skipped: 9
> >
> > [INFO]
> > ----------------------------------------------------------------------
> > --
> > [ERROR] BUILD FAILURE
> > [INFO]
> > ----------------------------------------------------------------------
> > --
> > [INFO] There are test failures.
> >
> > Please refer to <https://hudson.apache.org/hudson/job/hbase-
> 0.90/ws/trunk/target/surefire-reports> for the individual test results.
> > [INFO]
> > ----------------------------------------------------------------------
> > -- [INFO] For more information, run Maven with the -e switch [INFO]
> > ----------------------------------------------------------------------
> > -- [INFO] Total time: 93 minutes 5 seconds [INFO] Finished at: Tue Dec
> > 14 22:34:40 UTC 2010 [INFO] Final Memory: 93M/967M [INFO]
> > ----------------------------------------------------------------------
> > -- [locks-and-latches] Releasing all the locks [locks-and-latches] All
> > the locks released Archiving artifacts Recording test results
> >
> >
