Hi Everyone, Apologies for a long post. I am using Akka Cluster (version 2.4.10) to distribute an incoming HTTP request to a bunch of workers deployed on remote machines. I am following below sample from Lightbend. I am using Akka HTTP for the front-end role (as a REST layer)
http://www.lightbend.com/activator/template/akka-sample-cluster-scala The issue is, when I start multiple instances of the worker processes on different machines (15 instances on 4 different machines), some of them do not join the cluster and hence do not get the work sent by the front-end. For debugging, I reduced the problem to just 3 machines and 3 worker instances. All of these 3 instances are configured as seed nodes.The 3rd node joins the cluster only if I start the nodes in this particular order, node 2, node 3 and then node 1. If I start in any other order, node 1 and node 2 are forming the cluster, but node 3 never joins them. But the below Akka documentation says that the first node configured in seed-nodes element should be started first (node 1, in this case). The doc also says, "Note that you can only join to an existing cluster member, which means that for bootstrapping some node must join itself". but, I am not sure what does this mean and if I am doing this in my code. http://doc.akka.io/docs/akka/current/scala/cluster-usage.html#Joining_to_Seed_Nodes Can you help/guide me if there is anything wrong I am doing in my configuration/code or if I am missing something obvious. Appreciate your help. Below are the other details. *FrontEnd* val config = ConfigFactory.parseString("akka.cluster.roles = [frontend]"). withFallback(ConfigFactory.load("rr")) implicit val system = ActorSystem("APICluster", config) val server = new HttpServer(host, port) Cluster(system) registerOnMemberUp { system.actorOf(Props(classOf[FrontEnd], server, config), name = "FrontEnd") } *BackEnd* val config = ConfigFactory.parseString(s"akka.remote.netty.tcp.port=$port"). withFallback(ConfigFactory.parseString("akka.cluster.roles = [backend]")). withFallback(ConfigFactory.load("rr")) val system = ActorSystem("APICluster", config) system.actorOf(Props(classOf[Worker]), name = "Worker") system.actorOf(Props[MetricsListener], name = "metricsListener") *application.conf* akka { actor { provider = "akka.cluster.ClusterActorRefProvider" } remote { log-remote-lifecycle-events = off netty.tcp { hostname = "node1" port = 0 } } cluster { seed-nodes = [ "akka.tcp://APICluster@node1:2551", "akka.tcp://APICluster@node2:2581", "akka.tcp://APICluster@node3:2591"] } } *rr.conf* akka.cluster.min-nr-of-members = 2 akka.cluster.role { frontend.min-nr-of-members = 1 backend.min-nr-of-members = 2 } akka.actor.deployment { /frontEnd/backendRouter = { router = round-robin-group nr-of-instances = 20 routees.paths = ["/user/worker"] cluster { enabled = on use-role = backend allow-local-routees = on } } } Below are the logs when I started the nodes in the order node1, node2 and then node3. Node1 runs both frontend and a worker (2551), as 2 different instances, node2 runs a worker (2581) and node3 runs a worker (2591). *Node1* 13:17:27.540UTC INFO [APICluster-akka.actor.default-dispatcher-18] akka://APICluster/deadLetters RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://APICluster/system/cluster/core/daemon/firstSeedNodeProcess-1#-2136455265] to Actor[akka://APICluster/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 13:17:28.551UTC INFO [APICluster-akka.actor.default-dispatcher-17] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node1:2551] - Node [akka.tcp://APICluster@node1:2551] is JOINING, roles [backend] 13:17:30.195UTC INFO [APICluster-akka.actor.default-dispatcher-3] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node1:2551] - Node [akka.tcp://APICluster@node1:41935] is JOINING, roles [frontend] 13:18:02.002UTC INFO [APICluster-akka.actor.default-dispatcher-18] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node1:2551] - Node [akka.tcp://APICluster@node2:2581] is JOINING, roles [backend] 13:18:33.390UTC WARN [APICluster-akka.actor.default-dispatcher-14] akka.tcp://APICluster@node1:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAPICluster%40node3%3A2591-6 ReliableDeliverySupervisor - Association with remote system [akka.tcp://APICluster@node3:2591] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://APICluster@node3:2591]] Caused by: [Connection refused: node3/1.1.1.1:2591] *Node2* 13:18:02.050UTC INFO [APICluster-akka.actor.default-dispatcher-21] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node2:2581] - Welcome from [akka.tcp://APICluster@node1:2551] 13:18:03.239UTC INFO [APICluster-akka.actor.default-dispatcher-5] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node2:2581] - Leader is moving node [akka.tcp://APICluster@node2:2581] to [Up] 13:18:03.239UTC INFO [APICluster-akka.actor.default-dispatcher-5] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node2:2581] - Leader is moving node [akka.tcp://APICluster@node1:2551] to [Up] 13:18:03.239UTC INFO [APICluster-akka.actor.default-dispatcher-5] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node2:2581] - Leader is moving node [akka.tcp://APICluster@node1:41935] to [Up] 13:18:33.399UTC WARN [APICluster-akka.actor.default-dispatcher-21] akka.tcp://APICluster@node2:2581/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAPICluster%40node3%3A2591-6 ReliableDeliverySupervisor - Association with remote system [akka.tcp://APICluster@node3:2591] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://APICluster@node3:2591]] Caused by: [Connection refused: node3/1.1.1.1:2591] *Node3* 13:18:32.730UTC WARN [APICluster-akka.actor.default-dispatcher-15] akka.tcp://APICluster@node3:2591/system/cluster/core/daemon/downingProvider AutoDown - Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation. 13:18:32.797UTC DEBUG[APICluster-akka.actor.default-dispatcher-4] akka.tcp://APICluster@node3:2591/user/heatMapWorker HeatMapWorker - Inside HeatMapWorker's prestart 13:18:32.985UTC INFO [APICluster-akka.actor.default-dispatcher-20] akka.cluster.Cluster(akka://APICluster) Cluster(akka://APICluster) - Cluster Node [akka.tcp://APICluster@node3:2591] - Metrics collection has started successfully 14:05:55.480UTC WARN [APICluster-akka.actor.default-dispatcher-19] akka.tcp://APICluster@node3:2591/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAPICluster%40node1%3A2551-0 ReliableDeliverySupervisor - Association with remote system [akka.tcp://APICluster@node1:2551] has failed, address is now gated for [5000] ms. Reason: [Disassociated] Node 1 & Node 2 keep repeating the same message that association with node3 failed. And node 3 repeats it on node 1. Thanks for reading this long post. Appreciate your help and time. Thanks, Jegan -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
