This fail is OOME cannot create native thread. Two MR jobs fail:
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels <https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/lastCompletedBuild/jdk=latest1.8,label=Hadoop/testReport/org.apache.hadoop.hbase.mapreduce/TestImportTSVWithVisibilityLabels/org_apache_hadoop_hbase_mapreduce_TestImportTSVWithVisibilityLabels/>46 ms1 org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1 <https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/lastCompletedBuild/jdk=latest1.8,label=Hadoop/testReport/org.apache.hadoop.hbase.mapreduce/TestTableInputFormatScan1/org_apache_hadoop_hbase_mapreduce_TestTableInputFormatScan1/>42 ms1 Was running 1.3 tests on H0. Could try and purge resources used by these tests. I made HBASE-14955 to keep an eye on this rebirth of the OOME. St.Ack On Tue, Dec 8, 2015 at 3:58 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See < > https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/423/changes > > > > Changes: > > [enis] HBASE-14804 HBase shell's create table command ignores > > ------------------------------------------ > [...truncated 44593 lines...] > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.421 sec > - in org.apache.hadoop.hbase.procedure.TestZKProcedure > Running org.apache.hadoop.hbase.procedure.TestZKProcedureControllers > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.824 sec > - in org.apache.hadoop.hbase.procedure.TestZKProcedureControllers > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.241 sec > - in org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions > Running org.apache.hadoop.hbase.TestNamespace > Running org.apache.hadoop.hbase.constraint.TestConstraint > Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 289.524 > sec - in org.apache.hadoop.hbase.client.TestReplicasClient > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.454 sec > - in org.apache.hadoop.hbase.constraint.TestConstraint > Running org.apache.hadoop.hbase.TestZooKeeper > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.964 sec > - in org.apache.hadoop.hbase.TestNamespace > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 224.375 > sec - in org.apache.hadoop.hbase.client.TestMetaWithReplicas > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 195.99 sec > - in org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient > Running org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient > Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot > Running org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient > Running org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.25 sec > - in org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient > Running org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.774 > sec - in org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient > Running org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.784 sec > - in org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils > Running org.apache.hadoop.hbase.TestIOFencing > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.109 sec > - in org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries > Running > org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 227.832 > sec - in org.apache.hadoop.hbase.TestZooKeeper > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.213 sec > - in org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook > Running org.apache.hadoop.hbase.coprocessor.TestOpenTableInCoprocessor > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.678 sec > - in > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.208 sec > - in org.apache.hadoop.hbase.coprocessor.TestOpenTableInCoprocessor > Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorTableEndpoint > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorEndpoint > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.382 sec > - in org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorEndpoint > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.396 sec > - in org.apache.hadoop.hbase.coprocessor.TestCoprocessorTableEndpoint > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 151.189 > sec - in org.apache.hadoop.hbase.TestIOFencing > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface > Running > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.054 sec > - in > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove > Running org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol > Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.507 > sec - in org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol > Running org.apache.hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.53 sec - > in org.apache.hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint > Running org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver > Running org.apache.hadoop.hbase.coprocessor.TestHTableWrapper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.678 sec > - in org.apache.hadoop.hbase.coprocessor.TestHTableWrapper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.708 sec > - in org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort > Running org.apache.hadoop.hbase.coprocessor.TestClassLoading > Tests run: 14, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 77.414 > sec - in org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.858 sec > - in > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort > Running org.apache.hadoop.hbase.coprocessor.TestDoubleColumnInterpreter > Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.748 sec > - in org.apache.hadoop.hbase.coprocessor.TestDoubleColumnInterpreter > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.413 sec > - in org.apache.hadoop.hbase.coprocessor.TestClassLoading > Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.86 sec - > in org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass > Running org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint > Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorStop > Running > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.153 sec > - in org.apache.hadoop.hbase.coprocessor.TestCoprocessorStop > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.065 sec > - in org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.896 sec > - in > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove > Running org.apache.hadoop.hbase.coprocessor.TestBigDecimalColumnInterpreter > Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver > Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.531 sec > - in org.apache.hadoop.hbase.coprocessor.TestBigDecimalColumnInterpreter > Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.687 sec > - in org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint > Running org.apache.hadoop.hbase.TestGlobalMemStoreSize > Running org.apache.hadoop.hbase.coprocessor.TestWALObserver > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.848 sec > - in org.apache.hadoop.hbase.coprocessor.TestWALObserver > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.898 sec > - in org.apache.hadoop.hbase.TestGlobalMemStoreSize > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.593 sec > - in org.apache.hadoop.hbase.coprocessor.TestMasterObserver > Running org.apache.hadoop.hbase.TestMetaTableLocator > Running org.apache.hadoop.hbase.TestJMXListener > Running org.apache.hadoop.hbase.TestStochasticBalancerJmxMetrics > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.236 sec > - in org.apache.hadoop.hbase.TestMetaTableLocator > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.322 sec > - in org.apache.hadoop.hbase.TestStochasticBalancerJmxMetrics > Running org.apache.hadoop.hbase.zookeeper.TestZKTableStateManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.721 sec > - in org.apache.hadoop.hbase.zookeeper.TestZKTableStateManager > Running org.apache.hadoop.hbase.zookeeper.TestZKMulti > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.395 sec > - in org.apache.hadoop.hbase.zookeeper.TestZKMulti > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.569 sec > - in org.apache.hadoop.hbase.TestJMXListener > Running > org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock > Running org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.208 sec > - in org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager > Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperACL > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.703 sec > - in org.apache.hadoop.hbase.zookeeper.TestZooKeeperACL > Running org.apache.hadoop.hbase.zookeeper.TestRecoverableZooKeeper > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.518 sec > - in org.apache.hadoop.hbase.zookeeper.TestRecoverableZooKeeper > Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker > Running org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.902 sec > - in org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.34 sec - > in org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer > Running org.apache.hadoop.hbase.TestClusterBootOrder > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.959 sec > - in org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock > Running org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.797 sec > - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 566.282 > sec - in org.apache.hadoop.hbase.snapshot.TestExportSnapshot > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.231 sec > - in org.apache.hadoop.hbase.TestClusterBootOrder > Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 559.107 > sec - in org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot > > Results : > > Flaked tests: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes.testMROnTableWithInvalidOperationAttr(org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes) > Run 1: > TestImportTSVWithOperationAttributes.testMROnTableWithInvalidOperationAttr:144->doMROnTableTest:182 > expected:<0> but was:<1> > Run 2: PASS > > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs.testMROnTable(org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs) > Run 1: TestImportTSVWithTTLs.testMROnTable:113->doMROnTableTest:143 » IO > Unable to cl... > Run 2: PASS > > > org.apache.hadoop.hbase.mapreduce.TestImportTsv.testMROnTable(org.apache.hadoop.hbase.mapreduce.TestImportTsv) > Run 1: TestImportTsv.testMROnTable » Remote File > /tmp/hadoop-yarn/staging/jenkins/.st... > Run 2: PASS > > > org.apache.hadoop.hbase.mapreduce.TestImportTsv.testMROnTableWithCustomMapper(org.apache.hadoop.hbase.mapreduce.TestImportTsv) > Run 1: TestImportTsv.testMROnTableWithCustomMapper » Remote > java.lang.OutOfMemoryErro... > Run 2: PASS > > > Tests run: 2376, Failures: 0, Errors: 0, Skipped: 38, Flakes: 4 > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Apache HBase ...................................... SUCCESS > [1:27.761s] > [INFO] Apache HBase - Checkstyle ......................... SUCCESS [8.985s] > [INFO] Apache HBase - Resource Bundle .................... SUCCESS [0.373s] > [INFO] Apache HBase - Annotations ........................ SUCCESS [1.749s] > [INFO] Apache HBase - Protocol ........................... SUCCESS > [27.457s] > [INFO] Apache HBase - Common ............................. SUCCESS > [3:02.480s] > [INFO] Apache HBase - Procedure .......................... SUCCESS > [3:52.573s] > [INFO] Apache HBase - Client ............................. SUCCESS > [1:45.021s] > [INFO] Apache HBase - Hadoop Compatibility ............... SUCCESS > [10.473s] > [INFO] Apache HBase - Hadoop Two Compatibility ........... SUCCESS > [13.955s] > [INFO] Apache HBase - Prefix Tree ........................ SUCCESS > [16.251s] > [INFO] Apache HBase - Server ............................. FAILURE > [2:28:14.138s] > [INFO] Apache HBase - Testing Util ....................... SKIPPED > [INFO] Apache HBase - Thrift ............................. SKIPPED > [INFO] Apache HBase - Rest ............................... SKIPPED > [INFO] Apache HBase - Shell .............................. SKIPPED > [INFO] Apache HBase - Integration Tests .................. SKIPPED > [INFO] Apache HBase - Examples ........................... SKIPPED > [INFO] Apache HBase - External Block Cache ............... SKIPPED > [INFO] Apache HBase - Assembly ........................... SKIPPED > [INFO] Apache HBase - Shaded ............................. SKIPPED > [INFO] Apache HBase - Shaded - Client .................... SKIPPED > [INFO] Apache HBase - Shaded - Server .................... SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 2:40:10.312s > [INFO] Finished at: Tue Dec 08 23:54:13 UTC 2015 > [INFO] Final Memory: 446M/643M > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test > (secondPartTestsExecution) on project hbase-server: ExecutionException: > java.lang.RuntimeException: The forked VM terminated without properly > saying goodbye. VM crash or System.exit called? > [ERROR] Command was /bin/sh -c cd < > https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/ws/hbase-server> > && > /home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre/bin/java > -enableassertions -Dhbase.test -Xmx2800m -XX:MaxPermSize=256m > -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true > -Djava.awt.headless=true -jar < > https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/ws/hbase-server/target/surefire/surefirebooter1620578665641039176.jar> > < > https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/ws/hbase-server/target/surefire/surefire5891212082367218401tmp> > < > https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/ws/hbase-server/target/surefire/surefire_11185138431451721833397tmp > > > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn <goals> -rf :hbase-server > Build step 'Invoke top-level Maven targets' marked build as failure > Performing Post build task... > Match found for :.* : True > Logical operation result is TRUE > Running script : # Post-build task script. TODO: Check this in and have > all builds reference check-in. > pwd && ls > # NOTE!!!! The below code has been copied and pasted from > ./dev-tools/run-test.sh > # Do not change here without syncing there and vice-versa. > ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e '-Dhbase.test' > | wc -l` > if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then > echo "Suspicious java process found - waiting 30s to see if there are > just slow to stop" > sleep 30 > ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e '-Dhbase.test' > | wc -l` > if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then > echo " {color:red}There appear to be $ZOMBIE_TESTS_COUNT zombie > tests{color}, they should have been killed by surefire but survived" > jps -v | grep surefirebooter | grep -e '-Dhbase.test' > jps -v | grep surefirebooter | grep -e '-Dhbase.test' | cut -d ' ' -f 1 > | xargs -n 1 jstack > # Exit with error > exit 1 > else > echo "We're ok: there is no zombie test, but some tests took some time > to stop" > fi > else > echo "We're ok: there is no zombie test" > fi > [Hadoop] $ /bin/bash -xe /tmp/hudson1238770964816774274.sh > + pwd > <https://builds.apache.org/job/HBase-1.3/jdk=latest1.7,label=Hadoop/ws/> > + ls > bin > CHANGES.txt > conf > dev-support > hbase-annotations > hbase-assembly > hbase-checkstyle > hbase-client > hbase-common > hbase-examples > hbase-external-blockcache > hbase-hadoop2-compat > hbase-hadoop-compat > hbase-it > hbase-native-client > hbase-prefix-tree > hbase-procedure > hbase-protocol > hbase-resource-bundle > hbase-rest > hbase-server > hbase-shaded > hbase-shell > hbase-spark > hbase-testing-util > hbase-thrift > LICENSE.txt > NOTICE.txt > pom.xml > README.txt > src > target > ++ wc -l > ++ grep -e -Dhbase.test > ++ grep surefirebooter > ++ jps -v > + ZOMBIE_TESTS_COUNT=1 > + [[ 1 != 0 ]] > + echo 'Suspicious java process found - waiting 30s to see if there are > just slow to stop' > Suspicious java process found - waiting 30s to see if there are just slow > to stop > + sleep 30 > ++ jps -v > ++ grep surefirebooter > ++ grep -e -Dhbase.test > ++ wc -l > + ZOMBIE_TESTS_COUNT=0 > + [[ 0 != 0 ]] > + echo 'We'\''re ok: there is no zombie test, but some tests took some > time to stop' > We're ok: there is no zombie test, but some tests took some time to stop > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 0 > Archiving artifacts > Recording test results >
