See <https://builds.apache.org/job/HBase-1.1-JDK8/1681/changes>
Changes:
[tedyu] HBASE-14498 Master stuck in infinite loop when all Zookeeper servers
[tedyu] HBASE-14498 Revert for on-going review
------------------------------------------
[...truncated 3667 lines...]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4206)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4164)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4098)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:560)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:490)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplication(TestRegionReplicaReplicationEndpoint.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationWith2Replicas(TestRegionReplicaReplicationEndpoint.java:249)
Caused by: org.apache.hadoop.ipc.RemoteException: Failed after attempts=5,
exceptions:
Thu Nov 12 23:31:52 UTC 2015, RpcRetryingCaller{globalStartTime=1447371112852,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:31:53 UTC 2015, RpcRetryingCaller{globalStartTime=1447371112852,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:31:53 UTC 2015, RpcRetryingCaller{globalStartTime=1447371112852,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:31:54 UTC 2015, RpcRetryingCaller{globalStartTime=1447371112852,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:31:56 UTC 2015, RpcRetryingCaller{globalStartTime=1447371112852,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testRegionReplicaWithoutMemstoreReplication(org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint)
Time elapsed: 8.316 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=5, exceptions:
Thu Nov 12 23:32:01 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:01 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:02 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:03 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:05 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4206)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4164)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4098)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:560)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:490)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaWithoutMemstoreReplication(TestRegionReplicaReplicationEndpoint.java:269)
Caused by: org.apache.hadoop.ipc.RemoteException: Failed after attempts=5,
exceptions:
Thu Nov 12 23:32:01 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:01 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:02 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:03 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:32:05 UTC 2015, RpcRetryingCaller{globalStartTime=1447371121233,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testRegionReplicaReplicationIgnoresDroppedTables(org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint)
Time elapsed: 240.011 sec <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 240000
milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4168)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4098)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:1201)
at
org.apache.hadoop.hbase.HBaseTestingUtility.deleteTable(HBaseTestingUtility.java:1877)
at
org.apache.hadoop.hbase.HBaseTestingUtility.deleteTableIfAny(HBaseTestingUtility.java:1890)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationIgnoresDisabledTables(TestRegionReplicaReplicationEndpoint.java:348)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationIgnoresDroppedTables(TestRegionReplicaReplicationEndpoint.java:335)
testRegionReplicaReplicationPeerIsCreatedForModifyTable(org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint)
Time elapsed: 8.358 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=5, exceptions:
Thu Nov 12 23:36:09 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:09 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:10 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:11 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:13 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4206)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4164)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4098)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:560)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:490)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationPeerIsCreatedForModifyTable(TestRegionReplicaReplicationEndpoint.java:151)
Caused by: org.apache.hadoop.ipc.RemoteException: Failed after attempts=5,
exceptions:
Thu Nov 12 23:36:09 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:09 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:10 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:11 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
Thu Nov 12 23:36:13 UTC 2015, RpcRetryingCaller{globalStartTime=1447371369609,
pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This
server is in the failed servers list: hemera.apache.org/140.211.11.27:45412
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Tests run: 51, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 570.086 sec -
in org.apache.hadoop.hbase.util.TestHBaseFsck
Results :
Tests in error:
org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS.testWalRollOnLowReplication(org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS)
Run 1: TestWALProcedureStoreOnHDFS.testWalRollOnLowReplication:189 ? Runtime
sync abo...
Run 2: TestWALProcedureStoreOnHDFS.testWalRollOnLowReplication:189 ? Runtime
sync abo...
Run 3: TestWALProcedureStoreOnHDFS.testWalRollOnLowReplication:189 ? Runtime
sync abo...
TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationIgnoresDroppedTables:335->testRegionReplicaReplicationIgnoresDisabledTables:348
? TestTimedOut
TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationPeerIsCreated:121
? RetriesExhausted
TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationPeerIsCreatedForModifyTable:151
? RetriesExhausted
TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationWith10Replicas:259->testRegionReplicaReplication:181
? TestTimedOut
TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationWith2Replicas:249->testRegionReplicaReplication:178
? RetriesExhausted
TestRegionReplicaReplicationEndpoint.testRegionReplicaWithoutMemstoreReplication:269
? RetriesExhausted
Flaked tests:
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testRegionCrossingHFileSplitRowColBloom(org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles)
Run 1:
TestLoadIncrementalHFiles.testRegionCrossingHFileSplitRowColBloom:191->testRegionCrossingHFileSplit:195->runTest:223->runTest:232->runTest:285
? TestTimedOut
Run 2:
TestLoadIncrementalHFiles.testRegionCrossingHFileSplitRowColBloom:191->testRegionCrossingHFileSplit:195->runTest:223->runTest:232->runTest:285
? TestTimedOut
Run 3: PASS
org.apache.hadoop.hbase.regionserver.TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup(org.apache.hadoop.hbase.regionserver.TestWALLockup)
Run 1: TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup ? DamagedWAL On
sync
Run 2: TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup ? DamagedWAL On
sync
Run 3: PASS
Tests run: 2438, Failures: 0, Errors: 7, Skipped: 21, Flakes: 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ...................................... SUCCESS [7.307s]
[INFO] Apache HBase - Checkstyle ......................... SUCCESS [3.648s]
[INFO] Apache HBase - Resource Bundle .................... SUCCESS [1.391s]
[INFO] Apache HBase - Annotations ........................ SUCCESS [3.831s]
[INFO] Apache HBase - Protocol ........................... SUCCESS [10.556s]
[INFO] Apache HBase - Common ............................. SUCCESS [1:20.690s]
[INFO] Apache HBase - Procedure .......................... SUCCESS [1:54.554s]
[INFO] Apache HBase - Client ............................. SUCCESS [1:20.119s]
[INFO] Apache HBase - Hadoop Compatibility ............... SUCCESS [7.702s]
[INFO] Apache HBase - Hadoop Two Compatibility ........... SUCCESS [6.461s]
[INFO] Apache HBase - Prefix Tree ........................ SUCCESS [8.053s]
[INFO] Apache HBase - Server ............................. FAILURE
[1:59:21.011s]
[INFO] Apache HBase - Testing Util ....................... SKIPPED
[INFO] Apache HBase - Thrift ............................. SKIPPED
[INFO] Apache HBase - Rest ............................... SKIPPED
[INFO] Apache HBase - Shell .............................. SKIPPED
[INFO] Apache HBase - Integration Tests .................. SKIPPED
[INFO] Apache HBase - Examples ........................... SKIPPED
[INFO] Apache HBase - Assembly ........................... SKIPPED
[INFO] Apache HBase - Shaded ............................. SKIPPED
[INFO] Apache HBase - Shaded - Client .................... SKIPPED
[INFO] Apache HBase - Shaded - Server .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:04:47.486s
[INFO] Finished at: Thu Nov 12 23:42:00 UTC 2015
[INFO] Final Memory: 412M/584M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test
(secondPartTestsExecution) on project hbase-server: ExecutionException:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.maven.surefire.report.ReporterException: When writing xml report
stdout/stderr: /tmp/stderr6284834758747782625deferred (No such file or
directory) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hbase-server
Build step 'Invoke top-level Maven targets' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : # Post-build task script. TODO: Check this in and have all
builds reference check-in.
pwd && ls
# NOTE!!!! The below code has been copied and pasted from
./dev-tools/run-test.sh
# Do not change here without syncing there and vice-versa.
ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e '-Dhbase.test' | wc
-l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
echo "Suspicious java process found - waiting 30s to see if there are just
slow to stop"
sleep 30
ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e '-Dhbase.test' | wc
-l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
echo " {color:red}There appear to be $ZOMBIE_TESTS_COUNT zombie
tests{color}, they should have been killed by surefire but survived"
jps -v | grep surefirebooter | grep -e '-Dhbase.test'
jps -v | grep surefirebooter | grep -e '-Dhbase.test' | cut -d ' ' -f 1 |
xargs -n 1 jstack
# Exit with error
exit 1
else
echo "We're ok: there is no zombie test, but some tests took some time to
stop"
fi
else
echo "We're ok: there is no zombie test"
fi
[HBase-1.1-JDK8] $ /bin/bash -xe /tmp/hudson2249369848945700754.sh
+ pwd
/x1/jenkins/jenkins-slave/workspace/HBase-1.1-JDK8
+ ls
CHANGES.txt
LICENSE.txt
NOTICE.txt
README.txt
bin
conf
dev-support
hbase-annotations
hbase-assembly
hbase-checkstyle
hbase-client
hbase-common
hbase-examples
hbase-hadoop-compat
hbase-hadoop2-compat
hbase-it
hbase-native-client
hbase-prefix-tree
hbase-procedure
hbase-protocol
hbase-resource-bundle
hbase-rest
hbase-server
hbase-shaded
hbase-shell
hbase-testing-util
hbase-thrift
pom.xml
src
target
++ jps -v
++ grep surefirebooter
++ grep -e -Dhbase.test
++ wc -l
+ ZOMBIE_TESTS_COUNT=0
+ [[ 0 != 0 ]]
+ echo 'We'\''re ok: there is no zombie test'
We're ok: there is no zombie test
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Archiving artifacts
Recording test results
Updating HBASE-14498
