We have a 3-node cluster （157,158 and 159）, we send 4000messages/s to 158
in 157, sometimes 157 can not receive the heartbeat from 158, but 158 is
already sent the heartbeat. The heartbeat between 159 and 158 is fine, and
157-159 also ok. Our version is akka2.3.10, and hope to be able to solve
this problem in this version, although we will soon be upgraded to 2.4.18
Here is the log of heartbeat, we add a log when receive heartbeat.
This is 157 log, as we can see, 157 can not receive the heartbeat from 158
since 20:59:15,985 until 20:59:32,558, Heartbeat is not lost but delay
2017-05-23 20:59:07,516 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:08,601 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:11,824 | DEBUG | t-dispatcher-852 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:13,004 | DEBUG | lt-dispatcher-29 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:13,166 | DEBUG | t-dispatcher-853 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:13,273 | DEBUG | t-dispatcher-853 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:14,287 | DEBUG | t-dispatcher-853 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:15,853 | DEBUG | t-dispatcher-853 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:15,985 | DEBUG | t-dispatcher-852 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,558 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,612 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,632 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,831 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,910 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,982 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:32,987 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,059 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,101 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,170 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,175 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,190 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,224 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,226 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,269 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,302 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,348 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:33,369 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:34,221 | DEBUG | ult-dispatcher-3 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
2017-05-23 20:59:34,603 | DEBUG | ult-dispatcher-2 |
ClusterHeartbeatReceiver |apply$mcV$sp() 77 |
com.typesafe.akka.slf4j | Received Heartbeat from
akka.tcp://actorsystem@192.168.0.158:2551
Here is 158 log, We can confirm that the heartbeat has been sent
successfully
2017-05-23 20:59:15,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:16,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:17,266 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:18,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:19,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:20,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:21,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:22,287 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:23,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:24,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:25,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:26,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:27,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:28,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:29,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:30,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:31,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:32,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:33,266 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:34,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:35,266 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:36,270 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:37,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:38,266 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:39,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:40,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:41,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:42,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
2017-05-23 20:59:43,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender
|apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node
[akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to
[akka.tcp://actorsystem@192.168.0.157:2551]
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
