I precisely explained for what event you need to wait :-) Proper graceful shutdown means you need to wait for _your own_ address to come back as MemberRemoved event, and THEN you can shut down the actor system and JVM.
-- Cheers, Konrad 'ktoso’ Malawski Akka @ Lightbend On 18 March 2016 at 18:24:58, Guido Medina ([email protected]) wrote: How long is enough time? I tried sleep 1 sec between leave and terminate calls and it didn't work. On Friday, March 18, 2016 at 5:13:50 PM UTC, Konrad Malawski wrote: Also, please don't do that: cluster.leave(cluster.selfAddress()); system.terminate(); It is wrong. You need to give the cluster gossip enough time to complete the leaving "nicely". Proper graceful shutdown means you need to wait for _your own_ address to come back as MemberRemoved event, and THEN you can shut down the actor system and JVM. Related: this mistake is exactly something I warn against in my recent talk "Zen of Akka", you may enjoy it :-) https://youtu.be/tC-joPMPJLs On Fri, Mar 18, 2016 at 5:58 PM, Patrik Nordwall <[email protected]> wrote: Hi Ben There are some subtle exceptions to the convergence rule. Unreachable members with status Down or Exiting are not counted. /Patrik On Thu, Mar 17, 2016 at 11:25 PM, Guido Medina <[email protected]> wrote: In fact in the other nodes you get the following: INFO 22:10:20,383 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Leader is moving node [akka.tcp://[email protected]:38845] to [Exiting] INFO 22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:20,403 orRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:21,403 orRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:22,403 orRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:23,403 orRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:24,384 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Marking exiting node(s) as UNREACHABLE [Member(address = akka.tcp://[email protected]:38845, status = Exiting)]. This is expected and they will be removed. INFO 22:10:24,403 orRef - Message [akka.cluster.ClusterHeartbeatSender$Heartbeat] from Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] to Actor[akka://DevCluster/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO 22:10:25,387 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Leader is removing exiting node [akka.tcp://[email protected]:38845] Notice the 1st status is [Exiting], then you see from [Exiting] to [Unreachable] and then you get [MemberRemoved] which isn't logged by Akka but I can assure you it happens, and if you log the events the last two will be: INFO 22:23:00,968 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Leader is auto-downing unreachable node [akka.tcp://[email protected]:54621] INFO 22:23:00,969 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Marking unreachable node [akka.tcp://[email protected]:54621] as [Down] INFO 22:23:01,952 ster) - Cluster Node [akka.tcp://[email protected]:2552] - Leader is removing unreachable node [akka.tcp://[email protected]:54621] INFO 22:23:01,966 visor - MemberRemoved(Member(address = akka.tcp://[email protected]:54621, status = Removed),Down) The last message is the one I rely upon, same as: INFO 22:22:39,954 visor - MemberUp(Member(address = akka.tcp://[email protected]:54621, status = Up)) Regards, Guido. On Thursday, March 17, 2016 at 10:15:52 PM UTC, Guido Medina wrote: I just tried this: final Cluster cluster = Cluster.get(system); cluster.leave(cluster.selfAddress()); try { Thread.sleep(1000); } catch (InterruptedException ignored) { } system.terminate(); and it didn't work, still the dead node has to timeout in the others, in other words I don't see any cluster event message coming to the node that just left so by default I'm just shutting down letting gossip failures take care of it, and reacting only upon MemberUp and MemberRemoved instead, I have that working for like 10 months now no problem. Regards, Guido. On Thursday, March 17, 2016 at 10:04:27 PM UTC, Benjamin Black wrote: Hi Guido, I think in your case you are shutting down before the node has communicated to the leader that it wants to leave. I wait to get the MemberExited message before shutting down the node. Maybe I should wait for the MemberRemoved? Either way the ultimate aim is to not have the unreachable logic kick in and have to wait x seconds (I use 10 seconds) for the node to be auto downed by the leader. And the reason why I don't want to wait is according to the docs the leader wouldn't be able to add nodes whilst any node in the cluster is considered unreachable, which is a problem if I'm doing a rolling restart of all the nodes. Regards, Ben On Thursday, March 17, 2016 at 4:51:30 PM UTC-4, Guido Medina wrote: As for cluster.leave(cluster.selfAddress) my micro-services use the following to leave: Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { final Cluster cluster = Cluster.get(system); cluster.leave(cluster.selfAddress()); system.terminate(); Configurator.shutdown((LoggerContext) LogManager.getContext()); } }); But honestly I have never seen that work, the other nodes just report it as unreachable until it times out and it is completely removed, maybe the shutdown happens so fast that it is useless in my case. HTH, Guido. On Thursday, March 17, 2016 at 8:33:29 PM UTC, Guido Medina wrote: Hi Benjamin, I also rely on cluster events and AFAIK you can expect (and trust) MemberUp and MemberRemoved, these IMHO are the only two consistent states you can trust. In other words, I register some actors only when their nodes reach MemberUp and unregister only when their nodes reach MemberRemoved Any other state in between I would treat them as information only. So far I haven't got any issue with my mini-shard implementation relying on these only 2 statuses, the draw back is that it will only have to wait for a longer time to react. HTH, Guido. On Thursday, March 17, 2016 at 6:07:48 PM UTC, Benjamin Black wrote: Hello, I'm adding logic to our service so that when a node is being restarted it gracefully leaves the cluster using cluster.leave(cluster.selfAddress). In the cluster specification doc it states: If a node is unreachable then gossip convergence is not possible and therefore any leader actions are also not possible (for instance, allowing a node to become a part of the cluster). To be able to move forward the state of theunreachable nodes must be changed. It must become reachable again or marked as down Is this totally true? If a node is unreachable and is the leaving/exiting/removed state will this stop the leader from adding a new node? I ask because I have an actor that subscribes to cluster events and I can see a node is being added whilst another node is considered unreachable and in the exiting status: 14:02:46.843 INFO Exited member Member(address = akka.tcp://[email protected]:7000, status = Exiting) 14:02:51.842 INFO Unreachable member Member(address = akka.tcp://[email protected]:7000, status = Exiting) 14:02:53.843 INFO Removing member Member(address = akka.tcp://[email protected]:7000, status = Removed) 14:02:57.843 INFO Exited member Member(address = akka.tcp://[email protected]:7000, status = Exiting) 14:03:02.760 INFO Unreachable member Member(address = akka.tcp://[email protected]:7000, status = Exiting) 14:03:04.843 INFO Adding member Member(address = akka.tcp://[email protected]:7000, status = Up) Thanks, Ben -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Akka Tech Lead Lightbend - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search ... -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
