I have a configuration of Akka cluster with 4 nodes. Let's call them Engine
and Warehouse.
In Engine:
*Everything is normal*
[DEBUG] [08/20/2015 07:32:25.082]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-80]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:25.082]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-80]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:32:25.082]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-80]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:32:25.084]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-98]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:32:25.084]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-98]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:25.084]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-98]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
*We have GC pause here*
2015-08-20T07:32:31.959-0700: [GC2015-08-20T07:32:31.960-0700: [ParNew
(promotion failed): 172544K->170208K(172544K), 0.8968931
secs]2015-08-20T07:32:32.857-0700: [CMS2015-08-20T07:32:39.682-0700:
[CMS-concurrent-sweep: 7.003/7.909 secs] [Times: user=8.78 sys=0.40,
real=7.91 secs]
(concurrent mode failure): 5760566K->2755481K(6099776K), 30.9310318 secs]
5924201K->2755481K(6272320K), [CMS Perm : 142090K->141971K(237016K)],
31.8286597 secs] [Times: user=32.22 sys=0.38, real=31.83 secs]
*We have almost 32 seconds GC pause. The following heartbeat request are
buffered and played right after GC pause in one single second!. All the
connection is being disassociated with other cluster members.*
[DEBUG] [08/20/2015 07:33:03.794]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-23]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.794]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-23]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.794]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-23]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-81]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-81]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-81]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-97]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-97]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.808]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-97]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
*After GC pause we have these disassociations*
[DEBUG] [08/20/2015 07:33:03.823]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.823]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[WARN] [08/20/2015 07:33:03.823]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0]
Association with remote system
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
[WARN] [08/20/2015 07:33:03.827]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2]
Association with remote system
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
[DEBUG] [08/20/2015 07:33:03.838]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.839]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:03.839]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[WARN] [08/20/2015 07:33:03.839]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8]
Association with remote system
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
[DEBUG] [08/20/2015 07:33:03.851]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:03.851]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-13]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.852]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-13]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
*But, after gaining back the associating, we have buffer drainage.*
[DEBUG] [08/20/2015 07:33:03.855]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[Remoting] Associated
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.856]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[Remoting] Associated
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.889]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter]
Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.890]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter]
Drained
buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: 0,
noBackoffCount: 0 , adaptiveBackoff: 1000
[DEBUG] [08/20/2015 07:33:03.891]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter]
Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.892]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter]
Drained
buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: 0,
noBackoffCount: 0 , adaptiveBackoff: 1000
[DEBUG] [08/20/2015 07:33:03.912]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter]
Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:03.912]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter]
Drained
buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: 0,
noBackoffCount: 0 , adaptiveBackoff: 1000
[DEBUG] [08/20/2015 07:33:03.938]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-10]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:03.938]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-10]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
*After that Everything is normal again*
[DEBUG] [08/20/2015 07:33:09.054]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-87]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:09.054]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-87]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:09.054]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-87]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:09.056]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-89]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:09.056]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-89]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551]
[DEBUG] [08/20/2015 07:33:09.056]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-87]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
*Meanwhile in Warehouse:*
*Everything is normal so far until*
[DEBUG] [08/20/2015 07:32:29.023]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:32:29.024]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:29.024]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:32:29.025]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-15]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:32:29.025]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-15]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:29.025]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
*Engine is running 32 seconds GC pause now*
[DEBUG] [08/20/2015 07:32:40.035]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-21]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-First
heartbeat req w/o reply
[DEBUG] [08/20/2015 07:32:40.035]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-21]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:40.035]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-21]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:32:40.036]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-20]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:40.037]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-21]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:32:51.033]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-30]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-2’nd
heartbeat req w/o reply
[DEBUG] [08/20/2015 07:32:51.033]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-30]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:51.033]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-30]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:32:51.034]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-20]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:32:51.035]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-30]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
*Failure Transport detector kicks in and disassociation happens*
[INFO] [08/20/2015 07:32:52.715]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-211]
No response from remote. Handshake timed out or transport failure detector
triggered.
[DEBUG] [08/20/2015 07:32:52.716]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-47]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-13/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[WARN] [08/20/2015 07:32:52.716]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-47]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-13]
Association with remote system
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552] has failed, address
is now gated for [5000] ms. Reason: [Disassociated]
[DEBUG] [08/20/2015 07:32:52.717]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-13/endpointWriter]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[INFO] [08/20/2015 07:32:55.594]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FEngineWarehouseActorSystem%4010.199.67.55%3A54627-212]
No response from remote. Handshake timed out or transport failure detector
triggered.
[DEBUG] [08/20/2015 07:32:55.595]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-47]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-18]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] <-
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:32:55.596]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-5]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-18]
Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] <-
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
*Association comes back but with drainage*
[DEBUG] [08/20/2015 07:33:02.034]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-26]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:33:02.034]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-26]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:02.034]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-26]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:02.036]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-30]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:02.036]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-26]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:03.853]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-47]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-13/endpointWriter]
Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] ->
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:33:03.854]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-47]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2552-13/endpointWriter]
Drained buffer with maxWriteCount: 50, fullBackoffCount: 1,
smallBackoffCount: 0, noBackoffCount: 0 , adaptiveBackoff: 1000
[DEBUG] [08/20/2015 07:33:03.910]
[EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-5]
[Remoting] Associated
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551] <-
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:33:03.912]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-14]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]<-Response is back
*After that Everything is normal again*
[DEBUG] [08/20/2015 07:33:13.040]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-16]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:33:13.040]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-16]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:13.040]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-16]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
[DEBUG] [08/20/2015 07:33:13.041]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod2:2552]
[DEBUG] [08/20/2015 07:33:13.041]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-25]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2551]
[DEBUG] [08/20/2015 07:33:13.042]
[EngineWarehouseActorSystem-akka.actor.default-dispatcher-16]
[akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender]
Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] -
Heartbeat response from
[akka.tcp://EngineWarehouseActorSystem@compprod1:2552]
In our Akka configuration I used
akka.cluster.failure-detector" : {
# String: 15
"acceptable-heartbeat-pause" : "121 s",
# String: 16
"expected-response-after" : "11 s",
# String: 13
"heartbeat-interval" : "11 s",
# reference.conf: 109
But seems like failure detector kicks in after 12s and dissociates the
nodes, instead of 121s. Our GC pause is 32s though. After gain associating
back BUT, we get Buffer Drainage issue and missed messages.
1- How can we filter out GC pause heartbeat delay?
2- Is Buffer Drainage due really due to associate/disassociate activity?
3- There's another Akka cluster transport failure detection. Does that play
role here too?
akka.cluster.transport-failure-detector" : {
# application.conf: 19
"acceptable-heartbeat-pause" : "33 s",
# reference.conf: 164
# How often keep-alive heartbeat messages should be sent to
each connection.
"heartbeat-interval" : "4 s",
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
