I agree. +1 We can keep a list of flaky tests so that we can fix them later on.
Thanks, Jimmy On Wed, Nov 14, 2012 at 11:55 AM, lars hofhansl <[email protected]> wrote: > Sigh. > > It seems we're back at having a successful build being the exception rather > than the rule. > In this case it was some(?) timeout (all tests ran and passed), but many > previous runs had at least one test failing. > > Flaky tests are useless. They do not add confidence to a run, and worse they > add noise, which requires us to manually filter the good from the bad runs > but looking at the results. > > There was talk about separating the flaky tests from the good ones. > Short term I propose to disable or remove every test that failed more than > once in the last 10 runs. > > > This is getting quite frustrating. > > -- Lars > > > ----- Original Message ----- > From: Apache Jenkins Server <[email protected]> > To: [email protected] > Cc: > Sent: Wednesday, November 14, 2012 11:13 AM > Subject: Build failed in Jenkins: HBase-0.94 #589 > > See <https://builds.apache.org/job/HBase-0.94/589/> > > ------------------------------------------ > [...truncated 581 lines...] > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.706 sec > Running org.apache.hadoop.hbase.regionserver.TestAtomicOperation > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.647 sec > Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.495 sec > Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.374 sec > Running org.apache.hadoop.hbase.regionserver.metrics.TestSchemaMetrics > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.448 sec > Running org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.172 sec > Running org.apache.hadoop.hbase.regionserver.TestBlocksRead > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.987 sec > Running org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.313 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.529 sec > Running org.apache.hadoop.hbase.regionserver.TestHRegion > Tests run: 69, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.763 sec > Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.326 sec > Running org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor > Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.335 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.393 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.048 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestHLog > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.468 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestHLogBench > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.248 sec > Running org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.552 sec > Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.28 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 169.356 sec > Running org.apache.hadoop.hbase.filter.TestColumnRangeFilter > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.342 sec > Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.361 sec > Running org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol > Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.273 sec > Running org.apache.hadoop.hbase.coprocessor.TestClassLoading > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.9 sec > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.775 sec > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.899 sec > Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.647 sec > Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint > Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 33.635 sec > Running org.apache.hadoop.hbase.coprocessor.TestWALObserver > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.742 sec > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.368 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.997 sec > Running > org.apache.hadoop.hbase.coprocessor.example.TestZooKeeperScanPolicyObserver > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.064 sec > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.845 sec > Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit > Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 282.678 sec > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.9 sec > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.997 sec > Running org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.24 sec > Running org.apache.hadoop.hbase.TestGlobalMemStoreSize > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.046 sec > Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.018 sec > Running org.apache.hadoop.hbase.mapred.TestTableInputFormat > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.126 sec > Running org.apache.hadoop.hbase.mapred.TestTableMapReduce > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.751 sec > Running > org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.734 sec > Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.194 sec > Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.796 sec > Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 138.841 sec > Running org.apache.hadoop.hbase.mapreduce.TestImportTsv > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 171.43 sec > Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.682 sec > Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.211 sec > Running org.apache.hadoop.hbase.mapreduce.TestImportExport > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.202 sec > Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 367.455 sec > Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.107 sec > Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.901 sec > Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.01 sec > Running org.apache.hadoop.hbase.TestAcidGuarantees > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 141.048 sec > Running org.apache.hadoop.hbase.catalog.TestCatalogTracker > Tests run: 10, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 9.888 sec > Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.517 sec > Running org.apache.hadoop.hbase.client.TestTimestampsFilter > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.614 sec > Running org.apache.hadoop.hbase.client.TestHCM > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.206 sec > Running org.apache.hadoop.hbase.client.TestScannerTimeout > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.95 sec > Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 627.53 sec > Running org.apache.hadoop.hbase.client.TestMetaScanner > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.681 sec > Running org.apache.hadoop.hbase.client.TestHTableUtil > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.839 sec > Running org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.92 sec > Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.738 sec > Running org.apache.hadoop.hbase.client.TestFromClientSide > Tests run: 58, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 432.083 sec > Running > org.apache.hadoop.hbase.client.TestHTablePool$TestHTableThreadLocalPool > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.619 sec > Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.151 sec > Running org.apache.hadoop.hbase.client.TestMultipleTimestamps > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.605 sec > Running org.apache.hadoop.hbase.client.TestMultiParallel > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.126 sec > Running org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor > Tests run: 58, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 406.927 sec > Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.285 sec > Running org.apache.hadoop.hbase.client.TestHConnection > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.592 sec > Running org.apache.hadoop.hbase.rest.TestGzipFilter > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.234 sec > Running org.apache.hadoop.hbase.constraint.TestConstraint > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.295 sec > Running org.apache.hadoop.hbase.rest.TestMultiRowResource > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.757 sec > Running org.apache.hadoop.hbase.rest.TestScannersWithFilters > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.048 sec > Running org.apache.hadoop.hbase.rest.TestRowResource > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.37 sec > Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.817 sec > Running org.apache.hadoop.hbase.rest.client.TestRemoteTable > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.123 sec > Running org.apache.hadoop.hbase.rest.TestVersionResource > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.899 sec > Running org.apache.hadoop.hbase.rest.TestScannerResource > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.719 sec > Running org.apache.hadoop.hbase.rest.TestTableResource > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.198 sec > Running org.apache.hadoop.hbase.rest.TestStatusResource > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.926 sec > Running org.apache.hadoop.hbase.rest.TestSchemaResource > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.201 sec > Running org.apache.hadoop.hbase.thrift.TestThriftServer > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.049 sec > Running org.apache.hadoop.hbase.client.TestAdmin > Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 306.705 sec > Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.009 sec > Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.778 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.143 sec > Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 133.477 sec > Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.862 sec > Running org.apache.hadoop.hbase.io.hfile.TestChecksum > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.837 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.13 sec > Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache > Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.031 sec > Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.376 sec > Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.529 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.679 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.233 sec > Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock > Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.353 sec > Running org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.563 sec > Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite > Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.716 sec > Running org.apache.hadoop.hbase.TestClusterBootOrder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.724 sec > Running org.apache.hadoop.hbase.avro.TestAvroServer > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.122 sec > Running > org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.003 sec > Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 363.006 sec > Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.078 sec > Running org.apache.hadoop.hbase.replication.TestReplicationSource > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.047 sec > Running org.apache.hadoop.hbase.replication.TestReplicationZookeeper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.342 sec > Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.673 sec > Running org.apache.hadoop.hbase.ipc.TestDelayedRpc > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.628 sec > Running org.apache.hadoop.hbase.TestZooKeeper > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.229 sec > Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine > Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 588.246 sec > Running org.apache.hadoop.hbase.replication.TestMasterReplication > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 127.17 sec > Running org.apache.hadoop.hbase.TestHBaseTestingUtility > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.16 sec > Running org.apache.hadoop.hbase.replication.TestReplication > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.65 sec > > Results : > > Tests run: 1074, Failures: 0, Errors: 0, Skipped: 12 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 1:00:56.435s > [INFO] Finished at: Wed Nov 14 19:12:56 UTC 2012 > [INFO] Final Memory: 68M/639M > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test > (secondPartTestsExecution) on project hbase: Failure or timeout -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > Build step 'Invoke top-level Maven targets' marked build as failure > Archiving artifacts > Recording test results >
