[
https://issues.apache.org/jira/browse/KUDU-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Hao updated KUDU-2880:
--------------------------
Attachment: test-output.txt
> TestSecurity is flaky
> ---------------------
>
> Key: KUDU-2880
> URL: https://issues.apache.org/jira/browse/KUDU-2880
> Project: Kudu
> Issue Type: Test
> Reporter: Hao Hao
> Priority: Major
> Attachments: test-output.txt
>
>
> A recent run of TestSecurity failed with the following error:
> {noformat}
> There was 1 failure:
> 1)
> testExternallyProvidedSubjectRefreshedExternally(org.apache.kudu.client.TestSecurity)
> org.apache.kudu.client.NonRecoverableException: cannot complete before
> timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=26,
> TimeoutTracker(timeout=30000, elapsed=29608), Traces: [0ms] refreshing cache
> from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server
> master-127.0.202.126:46581, [63ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.124:43241, [69ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.126:46581: Network error: Failed to
> connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
> refused: /127.0.202.126:46581, [70ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.125:43873, [250ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.125:43873: Network error: [peer
> master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from
> downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873],
> [282ms] Sub RPC ConnectToMaster: received response from server
> master-127.0.202.124:43241: OK, [336ms] delaying RPC due to: Service
> unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
> refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception
> from downstream on [id: 0x2fae7299, /127.0.0.1:57014 =>
> /127.0.202.125:43873], [357ms] refreshing cache from master, [358ms] Sub RPC
> ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [358ms]
> Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241,
> [360ms] Sub RPC ConnectToMaster: received response from server
> master-127.0.202.126:46581: Network error: java.net.ConnectException:
> Connection refused: /127.0.202.126:46581, [360ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.125:43873, [361ms] Sub RPC
> ConnectToMaster: received response from server master-127.0.202.125:43873:
> Network error: Failed to connect to peer
> master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
> /127.0.202.125:43873, [363ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.124:43241: OK, [364ms] delaying RPC due to: Service
> unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
> refused: /127.0.202.125:43873, [376ms] refreshing cache from master, [377ms]
> Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
> [377ms] Sub RPC ConnectToMaster: sending RPC to server
> master-127.0.202.124:43241, [378ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.125:43873, [379ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.126:46581: Network error: Failed to
> connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
> refused: /127.0.202.126:46581, [381ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.125:43873: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [382ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.124:43241: OK, [383ms] delaying RPC due to: Service
> unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
> refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [397ms]
> refreshing cache from master, [397ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.126:46581, [398ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.124:43241, [399ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.126:46581: Network error:
> Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581):
> Connection refused: /127.0.202.126:46581, [402ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.124:43241: OK, [408ms] Sub RPC
> ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [410ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.125:43873: Network error: Failed to connect to peer
> master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
> /127.0.202.125:43873, [411ms] delaying RPC due to: Service unavailable:
> Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873)
> has no leader. Exceptions received:
> org.apache.kudu.client.RecoverableException: Failed to connect to peer
> master-127.0.202.126:46581(127.0.202.126:46581): Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
> refused: /127.0.202.125:43873, [416ms] refreshing cache from master, [416ms]
> Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
> [417ms] Sub RPC ConnectToMaster: sending RPC to server
> master-127.0.202.124:43241, [418ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.126:46581: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.126:46581, [418ms]
> Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873,
> [419ms] Sub RPC ConnectToMaster: received response from server
> master-127.0.202.125:43873: Network error: java.net.ConnectException:
> Connection refused: /127.0.202.125:43873, [421ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.124:43241: OK, [422ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [457ms]
> refreshing cache from master, [457ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.126:46581, [457ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.124:43241, [460ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.125:43873, [460ms] Sub RPC
> ConnectToMaster: received response from server master-127.0.202.126:46581:
> Network error: java.net.ConnectException: Connection refused:
> /127.0.202.126:46581, [462ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.125:43873: Network error: Failed to connect to peer
> master-127.0.202.125:43873(127.0.202.125:43873): Connection refused:
> /127.0.202.125:43873, [463ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.124:43241: OK, [465ms] delaying RPC due to: Service
> unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
> refused: /127.0.202.125:43873, [517ms] refreshing cache from master, [517ms]
> Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581,
> [517ms] Sub RPC ConnectToMaster: sending RPC to server
> master-127.0.202.124:43241, [518ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.125:43873, [520ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.126:46581: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.126:46581, [523ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.124:43241: OK, [526ms] Sub RPC ConnectToMaster: received
> response from server master-127.0.202.125:43873: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [527ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [597ms]
> refreshing cache from master, [597ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.126:46581, [598ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.124:43241, [599ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.125:43873, [601ms] Sub RPC
> ConnectToMaster: received response from server master-127.0.202.126:46581:
> Network error: Failed to connect to peer
> master-127.0.202.126:46581(127.0.202.126:46581): Connection refused:
> /127.0.202.126:46581, [602ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.124:43241: OK, [604ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.125:43873: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [605ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection
> refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [736ms]
> refreshing cache from master, [737ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.126:46581, [737ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.124:43241, [738ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.125:43873, [739ms] Sub RPC
> ConnectToMaster: received response from server master-127.0.202.126:46581:
> Network error: java.net.ConnectException: Connection refused:
> /127.0.202.126:46581, [741ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.125:43873: Network error: java.net.ConnectException:
> Connection refused: /127.0.202.125:43873, [743ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.124:43241: OK, [744ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [797ms]
> refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.126:46581, [798ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.124:43241, [798ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.125:43873, [801ms] Sub RPC
> ConnectToMaster: received response from server master-127.0.202.126:46581:
> Network error: java.net.ConnectException: Connection refused:
> /127.0.202.126:46581, [802ms] Sub RPC ConnectToMaster: received response from
> server master-127.0.202.124:43241: OK, [802ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.125:43873: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [804ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1417ms]
> refreshing cache from master, [1417ms] Sub RPC ConnectToMaster: sending RPC
> to server master-127.0.202.126:46581, [1417ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.124:43241, [1421ms] Sub RPC
> ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [1421ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.126:46581: Network error: java.net.ConnectException:
> Connection refused: /127.0.202.126:46581, [1423ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.125:43873: Network error:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1423ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.124:43241: OK, [1424ms] delaying RPC due to: Service
> unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873, [2537ms]
> refreshing cache from master, [2537ms] Sub RPC ConnectToMaster: sending RPC
> to server master-127.0.202.126:46581, [2537ms] Sub RPC ConnectToMaster:
> sending RPC to server master-127.0.202.124:43241, [2540ms] Sub RPC
> ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [2541ms]
> Sub RPC ConnectToMaster: received response from server
> master-127.0.202.126:46581: Network error: java.net.ConnectException:
> Connection refused: /127.0.202.126:46581, [2542ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.125:43873: Network error:
> Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873):
> Connection refused: /127.0.202.125:43873, [2543ms] Sub RPC ConnectToMaster:
> received response from server master-127.0.202.124:43241: OK, [2543ms]
> delaying RPC due to: Service unavailable: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to
> connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection
> refused: /127.0.202.125:43873, [3857ms] refreshing cache from master,
> [3857ms] Sub RPC ConnectToMaster: sending RPC to server
> master-127.0.202.126:46581, [3858ms] Sub RPC ConnectToMaster: sending RPC to
> server master-127.0.202.124:43241, [3861ms] Sub RPC ConnectToMaster: sending
> RPC to server master-127.0.202.125:43873, [3861ms] too many traces: truncated
> at 100 traces, deferred=Deferred@531938157(state=PENDING, result=null,
> callback=(continuation of Deferred@691636556 after retry RPC after
> error@610503197) -> (continuation of Deferred@1438562118 after retry RPC
> after error@1877676367) -> (continuation of Deferred@176002583 after retry
> RPC after error@855033383) -> (continuation of Deferred@315773880 after retry
> RPC after error@621038020) -> (continuation of Deferred@1127198301 after
> retry RPC after error@1206685665) -> (continuation of Deferred@1201883959
> after retry RPC after error@1089368174) -> (continuation of
> Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation
> of Deferred@1735340 after retry RPC after error@868489153) -> (continuation
> of Deferred@642229474 after retry RPC after error@269147125) -> (continuation
> of Deferred@1125522498 after retry RPC after error@1358918142) ->
> (continuation of Deferred@1258602423 after retry RPC after error@219615313)
> -> (continuation of Deferred@2115041725 after retry RPC after
> error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC
> after error@243071403) -> (continuation of Deferred@2002269384 after retry
> RPC after error@2049334820) -> (continuation of Deferred@650001986 after
> retry RPC after error@1340484255) -> (continuation of Deferred@2033105375
> after retry RPC after error@825298848) -> (continuation of
> Deferred@1722444778 after retry RPC after error@417732466) -> (continuation
> of Deferred@344634864 after retry RPC after error@763356366) -> (continuation
> of Deferred@406794160 after retry RPC after error@810214978) -> (continuation
> of Deferred@1907286158 after retry RPC after error@589636577) ->
> (continuation of Deferred@1107745434 after retry RPC after error@630803566)
> -> (continuation of Deferred@590705347 after retry RPC after error@696608623)
> -> (continuation of Deferred@1571007183 after retry RPC after
> error@850814761) -> (continuation of Deferred@983414998 after retry RPC after
> error@950083317) -> (continuation of Deferred@2014588668 after retry RPC
> after error@2040173048), errback=(continuation of Deferred@691636556 after
> retry RPC after error@610503197) -> (continuation of Deferred@1438562118
> after retry RPC after error@1877676367) -> (continuation of
> Deferred@176002583 after retry RPC after error@855033383) -> (continuation of
> Deferred@315773880 after retry RPC after error@621038020) -> (continuation of
> Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation
> of Deferred@1201883959 after retry RPC after error@1089368174) ->
> (continuation of Deferred@2114810728 after retry RPC after error@1564305615)
> -> (continuation of Deferred@1735340 after retry RPC after error@868489153)
> -> (continuation of Deferred@642229474 after retry RPC after error@269147125)
> -> (continuation of Deferred@1125522498 after retry RPC after
> error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC
> after error@219615313) -> (continuation of Deferred@2115041725 after retry
> RPC after error@1430980352) -> (continuation of Deferred@1004335897 after
> retry RPC after error@243071403) -> (continuation of Deferred@2002269384
> after retry RPC after error@2049334820) -> (continuation of
> Deferred@650001986 after retry RPC after error@1340484255) -> (continuation
> of Deferred@2033105375 after retry RPC after error@825298848) ->
> (continuation of Deferred@1722444778 after retry RPC after error@417732466)
> -> (continuation of Deferred@344634864 after retry RPC after error@763356366)
> -> (continuation of Deferred@406794160 after retry RPC after error@810214978)
> -> (continuation of Deferred@1907286158 after retry RPC after
> error@589636577) -> (continuation of Deferred@1107745434 after retry RPC
> after error@630803566) -> (continuation of Deferred@590705347 after retry RPC
> after error@696608623) -> (continuation of Deferred@1571007183 after retry
> RPC after error@850814761) -> (continuation of Deferred@983414998 after retry
> RPC after error@950083317) -> (continuation of Deferred@2014588668 after
> retry RPC after error@2040173048)))
> at
> org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
> at
> org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:413)
> at
> org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:197)
> at
> org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:135)
> at
> org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:431)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: org.apache.kudu.client.KuduException$OriginalException:
> Original asynchronous stack trace
> at
> org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1676)
> at
> org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2121)
> at
> org.apache.kudu.client.AsyncKuduClient.access$1200(AsyncKuduClient.java:261)
> at
> org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1328)
> at
> org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1318)
> at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
> at
> com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
> at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
> at
> org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:289)
> at
> org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
> at
> org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323)
> at
> org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312)
> at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
> at
> com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
> at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
> at
> org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275)
> at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322)
> at
> org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296)
> at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59)
> at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149)
> at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145)
> at
> org.apache.kudu.client.Connection.messageReceived(Connection.java:390)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config
> (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader.
> Exceptions received: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused:
> /127.0.202.126:46581,org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873
> at
> org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:287)
> at
> org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
> at
> org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323)
> at
> org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312)
> at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
> at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
> at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
> at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275)
> at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322)
> at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296)
> at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59)
> at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149)
> at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145)
> at
> org.apache.kudu.client.Connection.messageReceived(Connection.java:390)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> Caused by: org.apache.kudu.client.RecoverableException:
> java.net.ConnectException: Connection refused: /127.0.202.125:43873
> at org.apache.kudu.client.Connection.channelClosed(Connection.java:266)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
> at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
> at org.jboss.netty.channel.Channels$6.run(Channels.java:457)
> at
> org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
> ... 7 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
