See <https://builds.apache.org/job/HBase-1.0/201/changes>
Changes: [stack] HBASE-10134 Add Chaos Monkey that doesn't touch the master (Matteo Bertozzi ADDENDUM) ------------------------------------------ [...truncated 1627 lines...] Running org.apache.hadoop.hbase.rest.TestScannersWithLabels Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.599 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.358 sec Running org.apache.hadoop.hbase.rest.TestDeleteRow Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.144 sec Running org.apache.hadoop.hbase.rest.TestResourceFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.229 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.256 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.734 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.997 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.035 sec Running org.apache.hadoop.hbase.filter.TestFilterWrapper Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.708 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.738 sec Running org.apache.hadoop.hbase.filter.TestFuzzyRowAndColumnRangeFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.002 sec Running org.apache.hadoop.hbase.filter.TestFilterWithScanLimits Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.314 sec Running org.apache.hadoop.hbase.filter.TestColumnRangeFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.383 sec Running org.apache.hadoop.hbase.replication.TestReplicationWithTags Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.679 sec Running org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.805 sec Running org.apache.hadoop.hbase.replication.TestReplicationStateZKImpl Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.365 sec Running org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.397 sec Running org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.925 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.348 sec Running org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.987 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.313 sec Running org.apache.hadoop.hbase.replication.TestReplicationSmallTests Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.969 sec Running org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.664 sec Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.973 sec Running org.apache.hadoop.hbase.replication.TestMasterReplication Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.038 sec Running org.apache.hadoop.hbase.replication.TestReplicationTrackerZKImpl Tests run: 4, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 1.841 sec Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.522 sec Running org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.769 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.774 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationHLogReaderManager Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 251.469 sec Running org.apache.hadoop.hbase.replication.TestPerTableCFReplication Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.346 sec Running org.apache.hadoop.hbase.master.TestAssignmentManager Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.964 sec Running org.apache.hadoop.hbase.fs.TestBlockReorder Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.597 sec Running org.apache.hadoop.hbase.master.TestOpenedRegionHandler Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.87 sec Running org.apache.hadoop.hbase.master.cleaner.TestHFileCleaner Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.999 sec Running org.apache.hadoop.hbase.master.cleaner.TestLogsCleaner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.85 sec Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.517 sec Running org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.359 sec Running org.apache.hadoop.hbase.master.TestRegionPlacement Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.796 sec Running org.apache.hadoop.hbase.master.TestClusterStatusPublisher Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.437 sec Running org.apache.hadoop.hbase.master.handler.TestTableDescriptorModification Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.831 sec Running org.apache.hadoop.hbase.master.handler.TestTableDeleteFamilyHandler Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.05 sec Running org.apache.hadoop.hbase.master.handler.TestCreateTableHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.787 sec Running org.apache.hadoop.hbase.master.TestAssignmentListener Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.434 sec Running org.apache.hadoop.hbase.master.TestRestartCluster Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.724 sec Running org.apache.hadoop.hbase.master.TestMasterNoCluster Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.64 sec Running org.apache.hadoop.hbase.master.TestDeadServer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.445 sec Running org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.142 sec Running org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 121.064 sec Running org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.662 sec Running org.apache.hadoop.hbase.master.TestSplitLogManager Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.366 sec Running org.apache.hadoop.hbase.master.TestRollingRestart Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.373 sec Running org.apache.hadoop.hbase.master.TestMasterFileSystem Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.09 sec Running org.apache.hadoop.hbase.master.TestMasterTransitions Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 11.362 sec Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 148.934 sec Running org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.13 sec Running org.apache.hadoop.hbase.master.TestZKLessAMOnCluster Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.437 sec Running org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 534.673 sec Running org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.885 sec Running org.apache.hadoop.hbase.master.TestMasterMetrics Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.367 sec Running org.apache.hadoop.hbase.master.TestMaster Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.829 sec Running org.apache.hadoop.hbase.master.TestMasterFailover Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.662 sec Running org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.399 sec Running org.apache.hadoop.hbase.master.TestMasterStatusServlet Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.951 sec Running org.apache.hadoop.hbase.master.TestMasterShutdown Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.704 sec Running org.apache.hadoop.hbase.master.TestMasterMetricsWrapper Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.421 sec Running org.apache.hadoop.hbase.master.TestHMasterRPCException Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.507 sec Running org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.803 sec Running org.apache.hadoop.hbase.master.TestActiveMasterManager Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.846 sec Running org.apache.hadoop.hbase.master.TestTableLockManager Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.823 sec Results : Failed tests: testRebalanceOnRegionServerNumberChange[1](org.apache.hadoop.hbase.TestRegionRebalancing): After 5 attempts, region assignments were not balanced. Tests run: 2103, Failures: 1, Errors: 0, Skipped: 21 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] HBase ............................................. SUCCESS [2.505s] [INFO] HBase - Common .................................... SUCCESS [37.376s] [INFO] HBase - Protocol .................................. SUCCESS [9.102s] [INFO] HBase - Client .................................... SUCCESS [54.742s] [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [5.305s] [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [1.531s] [INFO] HBase - Prefix Tree ............................... SUCCESS [3.457s] [INFO] HBase - Server .................................... FAILURE [2:00:16.023s] [INFO] HBase - Testing Util .............................. SKIPPED [INFO] HBase - Thrift .................................... SKIPPED [INFO] HBase - Shell ..................................... SKIPPED [INFO] HBase - Integration Tests ......................... SKIPPED [INFO] HBase - Examples .................................. SKIPPED [INFO] HBase - Assembly .................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:02:10.809s [INFO] Finished at: Fri Sep 19 01:41:50 UTC 2014 [INFO] Final Memory: 39M/729M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase-server: There are test failures. [ERROR] [ERROR] Please refer to <https://builds.apache.org/job/HBase-1.0/ws/hbase-server/target/surefire-reports> for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hbase-server Build step 'Invoke top-level Maven targets' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l` if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then #It seems sometimes the tests are not dying immediately. Let's give them 10s echo "Suspicious java process found - waiting 10s to see if there are just slow to stop" sleep 10 ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l` if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then echo "There are $ZOMBIE_TESTS_COUNT zombie tests, they should have been killed by surefire but survived" echo "************ BEGIN zombies jstack extract" ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack | grep ".test" | grep "\.java"` jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack echo "************ END zombies jstack extract" JIRA_COMMENT="$JIRA_COMMENT {color:red}-1 core zombie tests{color}. There are ${ZOMBIE_TESTS_COUNT} zombie test(s): ${ZB_STACK}" BAD=1 jps | grep surefirebooter | cut -d ' ' -f 1 | xargs kill -9 else echo "We're ok: there is no zombie test, but some tests took some time to stop" fi else echo "We're ok: there is no zombie test" fi [HBase-1.0] $ /bin/bash -xe /tmp/hudson3626329626372225882.sh ++ jps ++ grep surefirebooter ++ wc -l + ZOMBIE_TESTS_COUNT=1 + [[ 1 != 0 ]] + echo 'Suspicious java process found - waiting 10s to see if there are just slow to stop' Suspicious java process found - waiting 10s to see if there are just slow to stop + sleep 10 ++ jps ++ grep surefirebooter ++ wc -l + ZOMBIE_TESTS_COUNT=1 + [[ 1 != 0 ]] + echo 'There are 1 zombie tests, they should have been killed by surefire but survived' There are 1 zombie tests, they should have been killed by surefire but survived + echo '************ BEGIN zombies jstack extract' ************ BEGIN zombies jstack extract ++ jps ++ grep surefirebooter ++ cut -d ' ' -f 1 ++ xargs -n 1 jstack ++ grep '\.java' ++ grep .test + ZB_STACK= POST BUILD TASK : FAILURE END OF POST BUILD TASK : 0 Archiving artifacts Sending artifact delta relative to HBase-1.0 #174 Archived 1813 artifacts Archive block size is 32768 Received 69 blocks and 665421431 bytes Compression is 0.3% Took 3 min 9 sec Recording test results Updating HBASE-10134
