[akka-user] DistributedPubSub keeps sending to exited nodes

['Christian Hoffmeister' via Akka User List]

Hi,

I have a sample application where two nodes join an Akka cluster. On both 
nodes there is an actor running registered to DistributedPubSubMediator and 
both are sending a Ping once per second (that hits either itself or the 
other node).

Then I gracefully shut down Node B (leave cluster, wait for MemberRemoved 
event, wait some seconds, terminate actor system, wait for termination, 
exit the jvm).

But even though both nodes see Node B leaving through gossip convergence at 
22:16:58, Node A keeps the left Node B in the PubSub (you see this with the 
RECEIVED PONG(nr, ping sender, pong responder)).

Why does PubSub not remove actors from exited nodes, but only actors from 
removed nodes (remove of Node B on Node A happens 4 seconds after exiting 
of Node B). In my case I only do not loose messages, because Node B is 
waiting another 10 seconds after it sees its one remove (which again, is 4 
seconds earlier than Node A sees Node B removed).

Thanks in advance.
Christian


Output Node A

[info] Thu Jun 23 22:16:23 CEST 2016 RECEIVE 
Pong(0,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:23 CEST 2016 CLUSTER MemberUp(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Up))
[info] Thu Jun 23 22:16:23 CEST 2016 CLUSTER 
LeaderChanged(Some(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:23 CEST 2016 CLUSTER 
RoleLeaderChanged(api,Some(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:23 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:24 CEST 2016 RECEIVE 
Pong(1,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:25 CEST 2016 RECEIVE 
Pong(2,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:26 CEST 2016 RECEIVE 
Pong(3,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:28 CEST 2016 RECEIVE 
Pong(4,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:29 CEST 2016 RECEIVE 
Pong(5,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:30 CEST 2016 RECEIVE 
Pong(6,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:31 CEST 2016 RECEIVE 
Pong(7,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:32 CEST 2016 RECEIVE 
Pong(8,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:33 CEST 2016 RECEIVE 
Pong(9,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:34 CEST 2016 RECEIVE 
Pong(10,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:35 CEST 2016 RECEIVE 
Pong(11,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:36 CEST 2016 RECEIVE 
Pong(12,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:37 CEST 2016 RECEIVE 
Pong(13,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:38 CEST 2016 RECEIVE 
Pong(14,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:39 CEST 2016 RECEIVE 
Pong(15,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:40 CEST 2016 RECEIVE 
Pong(16,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:41 CEST 2016 RECEIVE 
Pong(17,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:42 CEST 2016 RECEIVE 
Pong(18,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:43 CEST 2016 RECEIVE 
Pong(19,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:44 CEST 2016 RECEIVE 
Pong(20,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:45 CEST 2016 RECEIVE 
Pong(21,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:46 CEST 2016 RECEIVE 
Pong(22,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:47 CEST 2016 RECEIVE 
Pong(23,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:48 CEST 2016 RECEIVE 
Pong(24,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER MemberJoined(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Joining))
[info] Thu Jun 23 22:16:49 CEST 2016 RECEIVE 
Pong(25,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2551, 
akka.tcp://airfocus@127.0.0.1:2552))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER MemberUp(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Up))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
SeenChanged(false,Set(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2551, 
akka.tcp://airfocus@127.0.0.1:2552))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:50 CEST 2016 RECEIVE 
Pong(26,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:51 CEST 2016 RECEIVE 
Pong(27,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:52 CEST 2016 RECEIVE 
Pong(28,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:53 CEST 2016 RECEIVE 
Pong(29,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:54 CEST 2016 RECEIVE 
Pong(30,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:55 CEST 2016 RECEIVE 
Pong(31,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:56 CEST 2016 RECEIVE 
Pong(32,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:57 CEST 2016 RECEIVE 
Pong(33,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:58 CEST 2016 RECEIVE 
Pong(34,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberLeft(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Leaving))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberExited(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Exiting))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:59 CEST 2016 RECEIVE 
Pong(35,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:17:00 CEST 2016 RECEIVE 
Pong(36,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:01 CEST 2016 RECEIVE 
Pong(37,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:17:02 CEST 2016 RECEIVE 
Pong(38,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:17:02 CEST 2016 CLUSTER 
UnreachableMember(Member(address = akka.tcp://airfocus@127.0.0.1:2552, 
status = Exiting))
[info] Thu Jun 23 22:17:02 CEST 2016 CLUSTER 
ReachabilityChanged(akka.tcp://airfocus@127.0.0.1:2551 -> 
akka.tcp://airfocus@127.0.0.1:2552: Unreachable [Unreachable] (1))
[info] Thu Jun 23 22:17:02 CEST 2016 CLUSTER MemberRemoved(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Removed),Exiting)
[info] Thu Jun 23 22:17:02 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:17:03 CEST 2016 RECEIVE 
Pong(39,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:04 CEST 2016 RECEIVE 
Pong(40,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:05 CEST 2016 RECEIVE 
Pong(41,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:06 CEST 2016 RECEIVE 
Pong(42,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:07 CEST 2016 RECEIVE 
Pong(43,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:08 CEST 2016 RECEIVE 
Pong(44,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:09 CEST 2016 RECEIVE 
Pong(45,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:10 CEST 2016 RECEIVE 
Pong(46,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:10 CEST 2016 CLUSTER MemberLeft(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Leaving))
[info] Thu Jun 23 22:17:11 CEST 2016 RECEIVE 
Pong(47,akka.tcp://airfocus@127.0.0.1:2551,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER MemberExited(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Exiting))
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER ClusterShuttingDown
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER MemberRemoved(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Removed),Exiting)
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER LeaderChanged(None)
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER RoleLeaderChanged(api,None)
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER SeenChanged(true,Set())
[info] Thu Jun 23 22:17:11 CEST 2016 CLUSTER ReachabilityChanged()


Output Node B

[info] Thu Jun 23 22:16:48 CEST 2016 RECEIVE 
Pong(0,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER MemberUp(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Up))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER MemberJoined(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Joining))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
LeaderChanged(Some(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
RoleLeaderChanged(api,Some(akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2551, 
akka.tcp://airfocus@127.0.0.1:2552))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:49 CEST 2016 RECEIVE 
Pong(1,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER MemberUp(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Up))
[info] Thu Jun 23 22:16:49 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:50 CEST 2016 RECEIVE 
Pong(2,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:51 CEST 2016 RECEIVE 
Pong(3,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:52 CEST 2016 RECEIVE 
Pong(4,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:53 CEST 2016 RECEIVE 
Pong(5,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:54 CEST 2016 RECEIVE 
Pong(6,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:55 CEST 2016 RECEIVE 
Pong(7,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:56 CEST 2016 RECEIVE 
Pong(8,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2551)
[info] Thu Jun 23 22:16:57 CEST 2016 RECEIVE 
Pong(9,akka.tcp://airfocus@127.0.0.1:2552,akka.tcp://airfocus@127.0.0.1:2552)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberLeft(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Leaving))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER 
SeenChanged(false,Set(akka.tcp://airfocus@127.0.0.1:2552))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER 
SeenChanged(true,Set(akka.tcp://airfocus@127.0.0.1:2552, 
akka.tcp://airfocus@127.0.0.1:2551))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberExited(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Exiting))
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER ReachabilityChanged()
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER ClusterShuttingDown
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberRemoved(Member(address = 
akka.tcp://airfocus@127.0.0.1:2551, status = Removed),Up)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER MemberRemoved(Member(address = 
akka.tcp://airfocus@127.0.0.1:2552, status = Removed),Exiting)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER LeaderChanged(None)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER RoleLeaderChanged(api,None)
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER SeenChanged(true,Set())
[info] Thu Jun 23 22:16:58 CEST 2016 CLUSTER ReachabilityChanged()



-- 
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: 
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.