Hello, I've now been working for a few weeks in a project that uses Akka combined with Spring Boot 4.2.4. We have started setting up a (small) cluster and ran into an odd situation; according to the documentation, I had expected that the oldest cluster node will be used as the cluster leader, and also (by default) as the node for ClusterSingleton actors.
However, I have encountered behaviour that seems to indicate that Akka remembers the address of earlier nodes that were removed and then re-added to the cluster; such a node is then immediately set to be the cluster leader when it re-joins the cluster after it has been re-started. On the other hand, our ClusterSingleton actor still remains on the oldest node (just as we expect). So the cluster leader, as reported by either of: - Cluster.get(getContext().system()).state().leader() - Cluster.get(getContext().system()).readView().isLeader() and the actual cluster on which our ClusterSingleton actor resides are different after the first node in a cluster was removed and then re-added to the cluster (but not in the reverse situation, where the younger node is removed an then re-added). This was a bit of a surprise to me, and I can't find any explanation for this. The details: our application is a Spring Boot Java application. Framework versions in use: Java 8 (1.8.0_64) Akka 2.4.2 (com.typesafe.akk:akka-cluster_2.11:2.4.2, com.typesafe.akk:akka-cluster-tools_2.11:2.4.2) Spring Boot 4.2.4 In my test, I run two copy of the application on my local machine; combined, they represent a simple cluster with two nodes. The copies are started with different Akka ports: node 1: [email protected]:2551 node 2: [email protected]:2552 I first start node 1: 15:31:41.288 INFO Member is Up: Member(address = akka.tcp:// [email protected]:2551, status = Up) 15:31:41.293 INFO curr leader: true, address: [email protected]:2551 * [i.e. the current node is the current leader; the leader address is [email protected]:2551]* 15:31:41.293 INFO LeaderChanged(Some(akka.tcp://[email protected]:2551)), getLeader: akka.tcp://[email protected]:2551 Then cluster node 2 is started: 15:39:57.351 INFO curr leader: false, address: [email protected]:2551 *[so node 2 is not the current leader; instead it's still [email protected]:2551, as expected]* 15:39:57.351 INFO Member is Up: Member(address = akka.tcp:// [email protected]:2551, status = Up), address: [email protected]:2551 15:39:57.352 INFO Member is Up: Member(address = akka.tcp:// [email protected]:2552, status = Up), address: [email protected]:2552 Node 1 notices this and adds the second node to the cluster; now when asked who the current leader is, cluster 1 is returned, as expected: cluster 1: 15:39:37.296 INFO Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2552] is JOINING, roles [] 15:39:37.298 INFO curr leader: true, address: [email protected]:2551 *[node 1 is still seen as the leader]* 15:39:37.298 INFO Member event MemberJoined(Member(address = akka.tcp:// [email protected]:2552, status = Joining)): 15:39:37.755 INFO Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2552] to [Up] 15:39:37.757 INFO Member is Up: Member(address = akka.tcp:// [email protected]:2552, status = Up) 15:39:37.757 INFO curr leader: true, address: [email protected]:2551 I then stop node 1, and node 2 assumes the position of cluster leader: node 2: 15:43:18.949 WARN Marking node(s) as UNREACHABLE [Member(address = akka.tcp://[email protected]:2551, status = Up)] 15:43:18.951 INFO Member detected as unreachable: Member(address = akka.tcp://[email protected]:2551, status = Up) 15:43:18.951 INFO curr leader: true, address: [email protected]:2552 15:43:18.951 INFO LeaderChanged(Some(akka.tcp://[email protected]:2552)), getLeader: akka.tcp://[email protected]:2552 ... 15:43:29.966 INFO Younger observed OldestChanged: [None -> myself] 15:43:29.966 INFO Singleton manager starting singleton actor [akka://blue/user/ourSupervisorActor/singleton] 15:43:29.966 INFO ClusterSingletonManager state change *[Younger -> Oldest]* Starting node 1 again should still keep the older node 2 as the cluster leader. Here's what happens, as seen from node 1: 15:46:15.776 INFO Cluster Node [akka.tcp://[email protected]:2551] - Welcome from [akka.tcp://[email protected]:2552] 15:46:17.224 INFO Singleton identified at [akka.tcp://[email protected]:2552/user/ourSupervisorActor/singleton] 15:46:17.291 INFO ClusterSingletonManager state change* [Start -> Younger]* ... 15:46:40.318 INFO Member is Up: Member(address = akka.tcp://[email protected]:2552, status = Up), address: [email protected]:2552 15:46:40.318 INFO curr leader: true, address: [email protected]:2551* [and node 1 is already leader, although it's the Younger node]* 15:46:40.319 INFO LeaderChanged(Some(akka.tcp://[email protected]:2551)), getLeader: akka.tcp://[email protected]:2551 And from the point of view of node 2, this is seen as follows: 15:46:15.731 INFO Member event MemberJoined(Member(address = akka.tcp:// [email protected]:2551, status = Joining)): 15:46:16.001 INFO Cluster Node [akka.tcp://[email protected]:2552] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up] 15:46:16.002 INFO Member is Up: Member(address = akka.tcp://[email protected]:2551, status = Up) 15:46:16.003 INFO curr leader: false, address: [email protected]:2551 *[now also sees node 1 as the new leader]* 15:46:16.003 INFO LeaderChanged(Some(akka.tcp://[email protected]:2551)), getLeader: akka.tcp://[email protected]:2551 So now node 1 is the leader again although it was just started up (and node 2 is definitively older). I can then trigger a message to our cluster singleton actor on node 1, and can then see that the message is sent to node 2 for processing, so our ClusterSingleton actor is still on node 2 at this point. When I then stop and start node 2 again, node 1 will be kept as the cluster leader. In other words, when node 2 is re-added it will not take the leader position. Node 1 says: 16:05:11.862 INFO Member event MemberJoined(Member(address = akka.tcp://[email protected]:2552, status = Joining)): 16:05:12.104 INFO Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2552] to [Up] 16:05:12.106 INFO curr leader: true, address: [email protected]:2551 16:05:12.106 INFO Member is Up: Member(address = akka.tcp://[email protected]:2552, status = Up), address: [email protected]:2552 And node 2 reports: 16:05:32.804 INFO Member is Up: Member(address = akka.tcp:/ /[email protected]:2551, status = Up) 16:05:32.804 INFO Member is Up: Member(address = akka.tcp:// [email protected]:2552, status = Up) 16:05:32.805 INFO curr leader: false, address: [email protected]:2551 16:05:32.807 INFO LeaderChanged: LeaderChanged(Some(akka.tcp://[email protected]:2551)), getLeader: akka.tcp:// [email protected]:2551 So here the leader is still node 1, i.e. the node that was already alive when node 2 was added. But when I re-added node 1 to the cluster that consisted of just node 2, the position of leader was immediately switched over to node 1. Now is this expected behaviour? I would have thought that the older, existing node will always remain cluster leader when a new cluster node is added (independently of the earlier history of such nodes, i.e. independently of whether they have been the leader in the cluster before). [The reason I'm trying to trace the cluster leader is to prevent long messages being sent to a ClusterSingleton actor from other cluster nodes: in our application it is sufficient if only the actors inside the ClusterSingleton node communicate with the ClusterSingleton actor, so we want to prevent messages being sent by stopping them on the senders' side already, not at the receiver's side.] Thanks for any insights and comments! Markus -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
