Hi gang, a question regarding shard failover Here's my 3-node setup (using akka-2.3.0 rc3):
- backend 1 - backend 2 - frontend All nodes initialize ClusterSharding at startup -- frontend supplies an empty entryProps so that it does not host any regions. Frontend starts pinging a range of sharded actors host on the backends -- I can see that the entries are evenly distributed across the backends. When I invoke a clean shutdown of one of the backends, the frontend node is left in a bad state. It continually tries to connect to the dead backend node. It does this forever. Restarting the frontend is required to get it to find the failed over shards on the remaining backend. Am I missing something with shutting down cluster sharding? Some logs from frontend. They're a bit noisy -- I've tried to call out the interesting bits. DEBUG 15:40:42,389 akka.contrib.pattern.ShardRegion - Forwarding request for shard [0] to [Actor[akka.tcp:// [email protected]:50351/user/sharding/user#-576014717]] DEBUG 15:40:43,405 akka.contrib.pattern.ShardRegion - Forwarding request for shard [1] to [Actor[akka.tcp:// [email protected]:50324/user/sharding/user#-498786510]] DEBUG 15:40:44,425 akka.contrib.pattern.ShardRegion - Forwarding request for shard [2] to [Actor[akka.tcp:// [email protected]:50351/user/sharding/user#-576014717]] INFO 15:40:44,825 akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://ghost/deadLetters] to Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 15:40:44,881 akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://ghost/deadLetters] to Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. *DEBUG 15:40:44,884 akka.remote.EndpointWriter - Disassociated [akka.tcp://[email protected]:50373 <http://[email protected]:50373>] -> [akka.tcp://[email protected]:50351 <http://[email protected]:50351>]* INFO 15:40:44,884 akka.actor.LocalActorRef - Message [akka.actor.FSM$Timer] from Actor[akka://ghost/deadLetters] to Actor[akka://ghost/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1/endpointWriter#1517188237] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 15:40:45,091 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://ghost/system/cluster/core/daemon#-1507678631] to Actor[akka://ghost/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 15:40:45,097 com.kixeye.common.log.AkkaLogger - Cluster Node [akka.tcp://[email protected]:50373] - Marking exiting node(s) as UNREACHABLE [Member(address = akka.tcp://[email protected]:50351, status = Exiting)]. This is expected and they will be removed. *INFO 15:40:45,106 com.kixeye.common.cluster.ClusterModule - member is unreachable: Member(address = akka.tcp://[email protected]:50351 <http://[email protected]:50351>, status = Exiting)* ^-- node becomes unreachable DEBUG 15:40:45,445 akka.contrib.pattern.ShardRegion - Forwarding request for shard [3] to [Actor[akka.tcp:// [email protected]:50324/user/sharding/user#-498786510]] INFO 15:40:45,535 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] to Actor[akka://ghost/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. *DEBUG 15:40:46,465 akka.contrib.pattern.ShardRegion - Forwarding request for shard [4] to [Actor[akka.tcp://[email protected]:50351/user/sharding/user#-576014717 <http://[email protected]:50351/user/sharding/user#-576014717>]]* *^-- *forwarding msg to a known-unreachable node INFO 15:40:46,465 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] to Actor[akka://ghost/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 15:40:46,535 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] to Actor[akka://ghost/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 15:40:47,103 com.kixeye.common.cluster.ClusterModule - event: MemberRemoved(Member(address = akka.tcp://[email protected]:50351, status = Removed),Exiting) INFO 15:40:47,104 akka.contrib.pattern.ClusterSingletonManager - Member removed [akka.tcp://[email protected]:50351] DEBUG 15:40:47,486 akka.contrib.pattern.ShardRegion - Forwarding request for shard [5] to [Actor[akka.tcp:// [email protected]:50324/user/sharding/user#-498786510]] DEBUG 15:40:48,506 akka.contrib.pattern.ShardRegion - Forwarding request for shard [6] to [Actor[akka.tcp:// [email protected]:50351/user/sharding/user#-576014717]] INFO 15:40:48,506 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] to Actor[akka://ghost/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. DEBUG 15:40:49,526 akka.contrib.pattern.ShardRegion - Forwarding request for shard [7] to [Actor[akka.tcp:// [email protected]:50324/user/sharding/user#-498786510]] DEBUG 15:40:50,546 akka.contrib.pattern.ShardRegion - Forwarding request for shard [8] to [Actor[akka.tcp:// [email protected]:50351/user/sharding/user#-576014717]] DEBUG 15:40:51,552 akka.remote.EndpointWriter - AssociationError [akka.tcp://[email protected]:50373] -> [akka.tcp://[email protected]:50351]: Error [Association failed with [akka.tcp://[email protected]:50351]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:50351] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:50351 ] *WARN 15:40:51,554 akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://[email protected]:50351 <http://[email protected]:50351>] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://[email protected]:50351]].* ^-- this repeats every 5s, forever DEBUG 15:40:51,566 akka.contrib.pattern.ShardRegion - Forwarding request for shard [9] to [Actor[akka.tcp:// [email protected]:50324/user/sharding/user#-498786510]] DEBUG 15:40:51,579 akka.remote.EndpointWriter - Disassociated [akka.tcp:// [email protected]:50373] -> [akka.tcp://[email protected]:50351] INFO 15:40:51,579 akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] to Actor[akka://ghost/deadLetters] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
