Hi James, the sequence you describe makes perfect sense to me, and the ClusterSingletonManager tries to be overly thorough here; so much so that I would call it a bug<https://www.assembla.com/spaces/ddEDvgVAKr3QrUeJe5aVNr/tickets/3869>. Thanks for reporting!
Regards, Roland On Mon, Feb 10, 2014 at 9:34 PM, James Bellenger <[email protected]> wrote: > Hi gang. > What is the process for a node to gracefully exit a cluster? > Nodes in our system are going through this sequence: > > - jvm gets the shutdown signal > - node calls cluster.leave(cluster.selfAddress) > - node waits until it sees MemberRemoved with its own address > - node gives singletons a grace period to migrate > - actor system is shutdown > - jvm exits > > This *feels *correct, but the > docs<http://doc.akka.io/docs/akka/2.3.0-RC2/scala/cluster-usage.html#Leaving> > are > fuzzy on when the node can drop out. > Moreover, ClusterSingletonManager has a hard time with this flow. > Especially for 1-node clusters, it tries to handover to a non-existing > peer, fails, and then fails harder when it is restarted and the cluster > service is no longer running. > > Is there a better way for nodes to leave the cluster? > Logs below. > > INFO 12:19:40,586 com.kixeye.common.log.AkkaLogger - Cluster Node > [akka.tcp://[email protected]:50570] - Marked address [akka.tcp:// > [email protected]:50570] as [Leaving] > INFO 12:19:41,355 com.kixeye.common.log.AkkaLogger - Cluster Node > [akka.tcp://[email protected]:50570] - Leader is moving node [akka.tcp:// > [email protected]:50570] to [Exiting] > INFO 12:19:41,356 com.kixeye.common.cluster.ClusterModule - member > removed: leave completed! > INFO 12:19:41,362 com.kixeye.common.log.AkkaLogger - Cluster Node > [akka.tcp://[email protected]:50570] - Shutting down... > INFO 12:19:41,371 com.kixeye.common.log.AkkaLogger - Cluster Node > [akka.tcp://[email protected]:50570] - Successfully shut down > INFO 12:19:41,374 akka.contrib.pattern.ClusterSingletonManager - Exited > [akka.tcp://[email protected]:50570] > INFO 12:19:41,376 akka.contrib.pattern.ClusterSingletonManager - Oldest > observed OldestChanged: [akka.tcp://[email protected]:50570 -> None] > INFO 12:19:41,381 akka.contrib.pattern.ClusterSingletonManager - > ClusterSingletonManager state change [Oldest -> WasOldest] > INFO 12:19:41,396 akka.actor.LocalActorRef - Message > [akka.cluster.ClusterEvent$LeaderChanged] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon/autoDown#2017004581] was not > delivered. [1] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:41,396 akka.actor.LocalActorRef - Message > [akka.dispatch.sysmsg.Terminate] from > Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#1919962524] > to > Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#1919962524] > was not delivered. [2] dead letters encountered. This logging can be turned > off or adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:41,397 akka.actor.LocalActorRef - Message > [akka.cluster.ClusterEvent$RoleLeaderChanged] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon/autoDown#2017004581] was not > delivered. [3] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:41,397 akka.actor.LocalActorRef - Message > [akka.cluster.ClusterEvent$SeenChanged] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon/autoDown#2017004581] was not > delivered. [4] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:41,398 akka.actor.LocalActorRef - Message > [akka.cluster.InternalClusterAction$Unsubscribe] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon#1571353727] was not > delivered. [5] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:41,398 akka.actor.LocalActorRef - Message > [akka.cluster.InternalClusterAction$Unsubscribe] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon#1571353727] was not > delivered. [6] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:42,395 akka.contrib.pattern.ClusterSingletonManager - Retry > [1], sending TakeOverFromMe to [None] > INFO 12:19:43,415 akka.contrib.pattern.ClusterSingletonManager - Retry > [2], sending TakeOverFromMe to [None] > INFO 12:19:44,435 akka.contrib.pattern.ClusterSingletonManager - Retry > [3], sending TakeOverFromMe to [None] > INFO 12:19:45,455 akka.contrib.pattern.ClusterSingletonManager - Retry > [4], sending TakeOverFromMe to [None] > INFO 12:19:46,475 akka.contrib.pattern.ClusterSingletonManager - Retry > [5], sending TakeOverFromMe to [None] > ERROR 12:19:47,517 akka.actor.OneForOneStrategy - Expected hand-over to > [None] never occured > akka.contrib.pattern.ClusterSingletonManagerIsStuck: Expected hand-over to > [None] never occured > at > akka.contrib.pattern.ClusterSingletonManager$$anonfun$10.applyOrElse(ClusterSingletonManager.scala:556) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.contrib.pattern.ClusterSingletonManager$$anonfun$10.applyOrElse(ClusterSingletonManager.scala:548) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > ~[scala-library.jar:?] > at akka.actor.FSM$class.processEvent(FSM.scala:603) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.contrib.pattern.ClusterSingletonManager.processEvent(ClusterSingletonManager.scala:336) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:597) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:569) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.contrib.pattern.ClusterSingletonManager.aroundReceive(ClusterSingletonManager.scala:336) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:491) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.invoke_aroundBody2(ActorCell.scala:462) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.invoke_aroundBody3$advice(ActorCell.scala:536) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.invoke(ActorCell.scala:1) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [scala-library.jar:?] > INFO 12:19:47,520 akka.actor.LocalActorRef - Message > [akka.cluster.InternalClusterAction$Unsubscribe] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon#1571353727] was not > delivered. [7] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:47,520 akka.actor.LocalActorRef - Message > [akka.cluster.InternalClusterAction$Unsubscribe] from > Actor[akka://ghost/deadLetters] to > Actor[akka://ghost/system/cluster/core/daemon#1571353727] was not > delivered. [8] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:47,581 akka.actor.LocalActorRef - Message > [akka.actor.PoisonPill$] from > Actor[akka://ghost/user/$a/masterregion-DI_USA_1/queuemgr-DI_USA_1/$b#-1227772002] > to > Actor[akka://ghost/user/$a/masterregion-DI_USA_1/queuemgr-DI_USA_1/$b/$a#1401591931] > was not delivered. [9] dead letters encountered. This logging can be turned > off or adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO 12:19:47,611 akka.actor.LocalActorRef - Message > [akka.actor.PoisonPill$] from > Actor[akka://ghost/user/$a/masterregion-DI_USA_1/masterstats-DI_USA_1/$a#1216246889] > to > Actor[akka://ghost/user/$a/masterregion-DI_USA_1/masterstats-DI_USA_1/$a/$a#-146187624] > was not delivered. [10] dead letters encountered. This logging can be > turned off or adjusted with configuration settings 'akka.log-dead-letters' > and 'akka.log-dead-letters-during-shutdown'. > ERROR 12:19:47,616 akka.actor.OneForOneStrategy - requirement failed: > Cluster node must not be terminated > akka.actor.PostRestartException: exception post restart (class > akka.contrib.pattern.ClusterSingletonManagerIsStuck) > at > akka.actor.dungeon.FaultHandling$$anonfun$6.apply(FaultHandling.scala:240) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.FaultHandling$$anonfun$6.apply(FaultHandling.scala:238) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.FaultHandling$$anonfun$handleNonFatalOrInterruptedException$1.applyOrElse(FaultHandling.scala:293) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.FaultHandling$$anonfun$handleNonFatalOrInterruptedException$1.applyOrElse(FaultHandling.scala:288) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) > ~[scala-library.jar:?] > at > scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) > ~[scala-library.jar:?] > at > scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) > ~[scala-library.jar:?] > at > akka.actor.dungeon.FaultHandling$class.finishRecreate(FaultHandling.scala:238) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.FaultHandling$class.handleChildTerminated(FaultHandling.scala:281) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.handleChildTerminated(ActorCell.scala:344) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.DeathWatch$class.watchedActorTerminated(DeathWatch.scala:53) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.watchedActorTerminated(ActorCell.scala:344) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:430) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.systemInvoke_aroundBody0(ActorCell.scala:453) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.ActorCell.systemInvoke_aroundBody1$advice(ActorCell.scala:477) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:1) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > [akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [scala-library.jar:?] > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [scala-library.jar:?] > Caused by: java.lang.IllegalArgumentException: requirement failed: Cluster > node must not be terminated > at scala.Predef$.require(Predef.scala:233) ~[scala-library.jar:?] > at > akka.contrib.pattern.ClusterSingletonManager.preStart(ClusterSingletonManager.scala:389) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.Actor$class.postRestart(Actor.scala:547) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.contrib.pattern.ClusterSingletonManager.postRestart(ClusterSingletonManager.scala:336) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at akka.actor.Actor$class.aroundPostRestart(Actor.scala:485) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.contrib.pattern.ClusterSingletonManager.aroundPostRestart(ClusterSingletonManager.scala:336) > ~[akka-contrib_2.10-2.3.0-RC1.jar:2.3.0-RC1] > at > akka.actor.dungeon.FaultHandling$class.finishRecreate(FaultHandling.scala:229) > ~[akka-actor_2.10-2.3.0-RC1.jar:2.3.0-RC1] > ... 15 more > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
