Hao Hao created KUDU-2880:
-----------------------------
Summary: TestSecurity is flaky
Key: KUDU-2880
URL: https://issues.apache.org/jira/browse/KUDU-2880
Project: Kudu
Issue Type: Test
Reporter: Hao Hao
A recent run of TestSecurity failed with the following error:
{noformat}
There was 1 failure:
1)
testExternallyProvidedSubjectRefreshedExternally(org.apache.kudu.client.TestSecurity)
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout:
KuduRpc(method=ListTabletServers, tablet=null, attempt=26,
TimeoutTracker(timeout=30000, elapsed=29608), Traces: [0ms] refreshing cache
from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.126:46581, [63ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.124:43241, [69ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.126:46581: Network error: Failed to
connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581, [70ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.125:43873, [250ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.125:43873: Network error: [peer
master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from
downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873],
[282ms] Sub RPC ConnectToMaster: received response from server
master-127.0.202.124:43241: OK, [336ms] delaying RPC due to: Service
unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
[peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception
from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873],
[357ms] refreshing cache from master, [358ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.126:46581, [358ms] Sub RPC ConnectToMaster:
sending RPC to server master-127.0.202.124:43241, [360ms] Sub RPC
ConnectToMaster: received response from server master-127.0.202.126:46581:
Network error: java.net.ConnectException: Connection refused:
/127.0.202.126:46581, [360ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.125:43873, [361ms] Sub RPC ConnectToMaster: received response
from server master-127.0.202.125:43873: Network error: Failed to connect to
peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
/127.0.202.125:43873, [363ms] Sub RPC ConnectToMaster: received response from
server master-127.0.202.124:43241: OK, [364ms] delaying RPC due to: Service
unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
refused: /127.0.202.125:43873, [376ms] refreshing cache from master, [377ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
[377ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.124:43241, [378ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.125:43873, [379ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.126:46581: Network error: Failed to
connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581, [381ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.125:43873: Network error:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [382ms]
Sub RPC ConnectToMaster: received response from server
master-127.0.202.124:43241: OK, [383ms] delaying RPC due to: Service
unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [397ms]
refreshing cache from master, [397ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [398ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.0.202.124:43241, [399ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.126:46581: Network error: Failed to
connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581, [402ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.124:43241: OK, [408ms] Sub RPC
ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [410ms] Sub
RPC ConnectToMaster: received response from server master-127.0.202.125:43873:
Network error: Failed to connect to peer
master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
/127.0.202.125:43873, [411ms] delaying RPC due to: Service unavailable: Master
config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no
leader. Exceptions received: org.apache.kudu.client.RecoverableException:
Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581):
Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
refused: /127.0.202.125:43873, [416ms] refreshing cache from master, [416ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
[417ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.124:43241, [418ms] Sub RPC ConnectToMaster: received response
from server master-127.0.202.126:46581: Network error:
java.net.ConnectException: Connection refused: /127.0.202.126:46581, [418ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873,
[419ms] Sub RPC ConnectToMaster: received response from server
master-127.0.202.125:43873: Network error: java.net.ConnectException:
Connection refused: /127.0.202.125:43873, [421ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.124:43241: OK, [422ms] delaying
RPC due to: Service unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [457ms]
refreshing cache from master, [457ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [457ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.0.202.124:43241, [460ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.125:43873, [460ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.126:46581: Network error:
java.net.ConnectException: Connection refused: /127.0.202.126:46581, [462ms]
Sub RPC ConnectToMaster: received response from server
master-127.0.202.125:43873: Network error: Failed to connect to peer
master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
/127.0.202.125:43873, [463ms] Sub RPC ConnectToMaster: received response from
server master-127.0.202.124:43241: OK, [465ms] delaying RPC due to: Service
unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
refused: /127.0.202.125:43873, [517ms] refreshing cache from master, [517ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
[517ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.124:43241, [518ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.125:43873, [520ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.126:46581: Network error:
java.net.ConnectException: Connection refused: /127.0.202.126:46581, [523ms]
Sub RPC ConnectToMaster: received response from server
master-127.0.202.124:43241: OK, [526ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.125:43873: Network error:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [527ms]
delaying RPC due to: Service unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [597ms]
refreshing cache from master, [597ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [598ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.0.202.124:43241, [599ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.125:43873, [601ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.126:46581: Network error: Failed
to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
refused: /127.0.202.126:46581, [602ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.124:43241: OK, [604ms] Sub RPC
ConnectToMaster: received response from server master-127.0.202.125:43873:
Network error: java.net.ConnectException: Connection refused:
/127.0.202.125:43873, [605ms] delaying RPC due to: Service unavailable: Master
config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no
leader. Exceptions received: org.apache.kudu.client.RecoverableException:
Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581):
Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [736ms]
refreshing cache from master, [737ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [737ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.0.202.124:43241, [738ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.125:43873, [739ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.126:46581: Network error:
java.net.ConnectException: Connection refused: /127.0.202.126:46581, [741ms]
Sub RPC ConnectToMaster: received response from server
master-127.0.202.125:43873: Network error: java.net.ConnectException:
Connection refused: /127.0.202.125:43873, [743ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.124:43241: OK, [744ms] delaying
RPC due to: Service unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [797ms]
refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [798ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.0.202.124:43241, [798ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.125:43873, [801ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.126:46581: Network error:
java.net.ConnectException: Connection refused: /127.0.202.126:46581, [802ms]
Sub RPC ConnectToMaster: received response from server
master-127.0.202.124:43241: OK, [802ms] Sub RPC ConnectToMaster: received
response from server master-127.0.202.125:43873: Network error:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [804ms]
delaying RPC due to: Service unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1417ms]
refreshing cache from master, [1417ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [1417ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.124:43241, [1421ms] Sub RPC ConnectToMaster:
sending RPC to server master-127.0.202.125:43873, [1421ms] Sub RPC
ConnectToMaster: received response from server master-127.0.202.126:46581:
Network error: java.net.ConnectException: Connection refused:
/127.0.202.126:46581, [1423ms] Sub RPC ConnectToMaster: received response from
server master-127.0.202.125:43873: Network error: java.net.ConnectException:
Connection refused: /127.0.202.125:43873, [1423ms] Sub RPC ConnectToMaster:
received response from server master-127.0.202.124:43241: OK, [1424ms] delaying
RPC due to: Service unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873, [2537ms]
refreshing cache from master, [2537ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.126:46581, [2537ms] Sub RPC ConnectToMaster: sending
RPC to server master-127.0.202.124:43241, [2540ms] Sub RPC ConnectToMaster:
sending RPC to server master-127.0.202.125:43873, [2541ms] Sub RPC
ConnectToMaster: received response from server master-127.0.202.126:46581:
Network error: java.net.ConnectException: Connection refused:
/127.0.202.126:46581, [2542ms] Sub RPC ConnectToMaster: received response from
server master-127.0.202.125:43873: Network error: Failed to connect to peer
master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
/127.0.202.125:43873, [2543ms] Sub RPC ConnectToMaster: received response from
server master-127.0.202.124:43241: OK, [2543ms] delaying RPC due to: Service
unavailable: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
refused: /127.0.202.125:43873, [3857ms] refreshing cache from master, [3857ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
[3858ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.0.202.124:43241, [3861ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.0.202.125:43873, [3861ms] too many traces: truncated at 100
traces, deferred=Deferred@531938157(state=PENDING, result=null,
callback=(continuation of Deferred@691636556 after retry RPC after
error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after
error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after
error@855033383) -> (continuation of Deferred@315773880 after retry RPC after
error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after
error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after
error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after
error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after
error@868489153) -> (continuation of Deferred@642229474 after retry RPC after
error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after
error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after
error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after
error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after
error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after
error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after
error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after
error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after
error@417732466) -> (continuation of Deferred@344634864 after retry RPC after
error@763356366) -> (continuation of Deferred@406794160 after retry RPC after
error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after
error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after
error@630803566) -> (continuation of Deferred@590705347 after retry RPC after
error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after
error@850814761) -> (continuation of Deferred@983414998 after retry RPC after
error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after
error@2040173048), errback=(continuation of Deferred@691636556 after retry RPC
after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC
after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC
after error@855033383) -> (continuation of Deferred@315773880 after retry RPC
after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC
after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC
after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC
after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC
after error@868489153) -> (continuation of Deferred@642229474 after retry RPC
after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC
after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC
after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC
after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC
after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC
after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC
after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC
after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC
after error@417732466) -> (continuation of Deferred@344634864 after retry RPC
after error@763356366) -> (continuation of Deferred@406794160 after retry RPC
after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC
after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC
after error@630803566) -> (continuation of Deferred@590705347 after retry RPC
after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC
after error@850814761) -> (continuation of Deferred@983414998 after retry RPC
after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC
after error@2040173048)))
at
org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at
org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:413)
at
org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:197)
at
org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:135)
at
org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Suppressed: org.apache.kudu.client.KuduException$OriginalException:
Original asynchronous stack trace
at
org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1676)
at
org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2121)
at
org.apache.kudu.client.AsyncKuduClient.access$1200(AsyncKuduClient.java:261)
at
org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1328)
at
org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1318)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at
com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at
org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:289)
at
org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
at
org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323)
at
org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at
com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at
org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275)
at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322)
at
org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145)
at
org.apache.kudu.client.Connection.messageReceived(Connection.java:390)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config
(127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused:
/127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873
at
org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:287)
at
org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
at
org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323)
at
org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275)
at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145)
at
org.apache.kudu.client.Connection.messageReceived(Connection.java:390)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.RecoverableException:
java.net.ConnectException: Connection refused: /127.0.202.125:43873
at org.apache.kudu.client.Connection.channelClosed(Connection.java:266)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at org.jboss.netty.channel.Channels$6.run(Channels.java:457)
at
org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
... 7 more
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
