Ok so it seems there was some sort of network issue. Then leader election.
But it seems it had some old state and kept trying to connect to the same
task machine over and over...?

2019-09-19 22:26:14,841 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Unable to read additional data from server sessionid 0xXXXXXX, likely
server has closed socket, closing socket connection and attempting reconnect
2019-09-19 22:26:14,946 INFO
 
org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager
 - State change: SUSPENDED
2019-09-19 22:26:14,947 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender http://XXXXXX-2:8081 no
longer participates in the leader election.
2019-09-19 22:26:14,947 WARN
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper suspended. Can no longer retrieve the leader
from ZooKeeper.
2019-09-19 22:26:14,947 WARN
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper suspended. Can no longer retrieve the leader
from ZooKeeper.
2019-09-19 22:26:14,948 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@XXXXXX-2:37697/user/resourcemanager
no longer participates in the leader election.
2019-09-19 22:26:14,948 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@fXXXXXX-2:37697/user/dispatcher
no longer participates in the leader election.
2019-09-19 22:26:14,949 WARN
 org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
ZooKeeper connection SUSPENDING. Changes to the submitted job graphs are
not monitored (temporarily).
2019-09-19 22:26:25,185 WARN
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No JAAS
configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-2423287132287811787.conf'. Will continue
connection to Zookeeper server without SASL authentication, if Zookeeper
server allows it.
2019-09-19 22:26:25,186 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Opening socket connection to server XXXXXX.71/XXXXXX.71:2181
2019-09-19 22:26:25,186 ERROR
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-09-19 22:26:25,192 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Socket connection established to XXXXXX.71/XXXXXX.71:2181, initiating
session
2019-09-19 22:26:25,199 WARN
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Unable to reconnect to ZooKeeper service, session 0x3017fc1a6660000 has
expired
2019-09-19 22:26:25,199 INFO
 
org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager
 - State change: LOST
2019-09-19 22:26:25,199 WARN
 org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  -
Session expired event received
2019-09-19 22:26:25,199 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper lost. The contender http://XXXXXX-2:8081 no longer
participates in the leader election.
2019-09-19 22:26:25,199 WARN
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper lost. Can no longer retrieve the leader from
ZooKeeper.
2019-09-19 22:26:25,200 WARN
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper lost. Can no longer retrieve the leader from
ZooKeeper.
2019-09-19 22:26:25,199 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper  -
Initiating client connection,
connectString=XXXXXX-1.XXXXXX:2181,XXXXXX-2.XXXXXX:2181,XXXXXX-3.XXXXXX:2181
sessionTimeout=60000
watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@2bec854f
2019-09-19 22:26:25,200 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper lost. The contender
akka.tcp://flink@XXXXXX-2:37697/user/resourcemanager
no longer participates in the leader election.
2019-09-19 22:26:25,200 WARN
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper lost. The contender
akka.tcp://flink@XXXXXX-2:37697/user/dispatcher
no longer participates in the leader election.
2019-09-19 22:26:25,201 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Unable to reconnect to ZooKeeper service, session 0x3017fc1a6660000 has
expired, closing socket connection
2019-09-19 22:26:25,201 WARN
 org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
ZooKeeper connection LOST. Changes to the submitted job graphs are not
monitored (permanently).
2019-09-19 22:26:25,220 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
EventThread shut down for session: 0x3017fc1a6660000
2019-09-19 22:26:25,231 WARN
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No JAAS
configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-2423287132287811787.conf'. Will continue
connection to Zookeeper server without SASL authentication, if Zookeeper
server allows it.
2019-09-19 22:26:25,232 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Opening socket connection to server XXXXXX.33/XXXXXX.33:2181
2019-09-19 22:26:25,232 ERROR
org.apache.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-09-19 22:26:25,233 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Socket connection established to XXXXXX.33/XXXXXX.33:2181, initiating
session
2019-09-19 22:26:25,247 INFO
 org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  -
Session establishment complete on server XXXXXX.33/XXXXXX.33:2181,
sessionid = 0x301db1787060000, negotiated timeout = 40000
2019-09-19 22:26:25,247 INFO
 
org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager
 - State change: RECONNECTED
2019-09-19 22:26:25,248 INFO
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-09-19 22:26:25,253 INFO
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper was reconnected. Leader retrieval can be
restarted.
2019-09-19 22:26:25,253 INFO
 org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
 - Connection to ZooKeeper was reconnected. Leader retrieval can be
restarted.
2019-09-19 22:26:25,253 INFO
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-09-19 22:26:25,253 INFO
 org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-09-19 22:26:25,253 INFO
 org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
ZooKeeper connection RECONNECTED. Changes to the submitted job graphs are
monitored again.
2019-09-19 22:26:34,376 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Disassociated]
2019-09-19 22:26:34,376 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:38091] has failed, address is now gated
for [50] ms. Reason: [Disassociated]
2019-09-19 22:26:35,147 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:26:35,149 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:26:45,167 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:26:45,168 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:26:55,151 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:26:55,153 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:27:05,159 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:27:05,160 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:27:15,157 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:27:15,161 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:27:25,152 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:27:25,160 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]
2019-09-19 22:27:35,161 WARN  akka.remote.transport.netty.NettyTransport
                 - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: /XXXXXX.11:46167
2019-09-19 22:27:35,165 WARN  akka.remote.ReliableDeliverySupervisor
                 - Association with remote system
[akka.tcp://[email protected]:46167] has failed, address is now gated for
[50] ms. Reason: [Association failed with [akka.tcp://[email protected]:46167]]
Caused by: [Connection refused: /XXXXXX.11:46167]



On Wed, 9 Oct 2019 at 19:44, Timothy Victor <[email protected]> wrote:

> We see a very similar (if not the same) error running version 1.9 on
> Kubernetes.   So far what we have discovered is that a taskmanager gets
> killed and a new one is created, but JM still thinks it needs to connect to
> the old (now dead TM).  I was even able to see the a taskmanager on the
> same host and port but with different TM instance ids in the Flink UI.  The
> issue seems to be persistent (i.e. doesn't clear after a few minutes).
>
> FWIW...TM was dying due to livenessprobe in K8s.   We have increased that,
> but still the above issue is a concern.
>
> Any ideas?
>
> Tim
>
> On Wed, Oct 9, 2019, 3:15 PM John Smith <[email protected]> wrote:
>
>> Sorry been away on leave. I'll check ASAP.
>>
>> On Thu, 3 Oct 2019 at 20:52, Zili Chen <[email protected]> wrote:
>>
>>> Does the log you attached above come from a TaskManager Node? If so,
>>> what state is the Job node it tried to connect to? Did it crash?
>>>
>>> BTW, it would be helpful if you can attach more logs of TM and JM except
>>> two lines said akka connection refused.
>>>
>>>
>>> John Smith <[email protected]> 于2019年10月4日周五 上午2:08写道:
>>>
>>>> So I guess it had some older state?
>>>>
>>>> On Thu., Oct. 3, 2019, 11:29 a.m. John Smith, <[email protected]>
>>>> wrote:
>>>>
>>>>> I'm running standalone cluster with Zookeeper. It seems it was trying
>>>>> to connect to an older node. I rebooted the Job node tha was complaining.
>>>>> It seems to be ok now...
>>>>>
>>>>> I have 3 Zookeepers, 3 Job Nodes and 3 Tasks Nodes
>>>>>
>>>>> On Thu, 3 Oct 2019 at 11:15, Zili Chen <[email protected]> wrote:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> could you provide some details such as which mode you runs
>>>>>> on(standalone/YARN)
>>>>>> and related configuration(jobmanager.address jobmanager.port and so
>>>>>> on)?
>>>>>>
>>>>>> Best,
>>>>>> tison.
>>>>>>
>>>>>>
>>>>>> John Smith <[email protected]> 于2019年10月3日周四 下午11:02写道:
>>>>>>
>>>>>>> Hi running 1.8 the cluster seems to be OK but I see these warnings
>>>>>>> in the logs...
>>>>>>>
>>>>>>> 2019-10-03 14:57:25,152 WARN
>>>>>>>  akka.remote.transport.netty.NettyTransport                    - Remote
>>>>>>> connection to [null] failed with java.net.ConnectException: Connection
>>>>>>> refused: /xxx.xxx.xxx.65:46167
>>>>>>> 2019-10-03 14:57:25,156 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>>                        - Association with remote system
>>>>>>> [akka.tcp://[email protected]:46167] has failed, address is now
>>>>>>> gated for [50] ms. Reason: [Association failed with
>>>>>>> [akka.tcp://[email protected]:46167]] Caused by: [Connection
>>>>>>> refused: /xxx.xxx.xxx.65:46167]
>>>>>>>
>>>>>>>
>>>>>>>

Reply via email to