Hi Patrik, I'm quite sure his problem has anything to do with Akka but lack of proper JVM parameters, that 32 seconds pause shown in his logs would be enough to cause nodes de-association and connection failures/timeouts of Akka remote (without mentioning Akka cluster and other extensions) hence I suggested to switch to newest garbage collector, of course the ideal thing to do is for OP to post JVM parameters and more information about the JVM in use to aid better.
Hope that helps, Guido. On Monday, August 24, 2015 at 2:41:53 PM UTC+1, Patrik Nordwall wrote: > > Note that there is no: akka.cluster.transport-failure-detector > > you probably mean: akka.remote.transport-failure-detector > > The Disassociated might be triggered by that failure detector and you can > try to increase > > akka.remote.transport-failure-detector.acceptable-heartbeat-pause > > /Patrik > > On Fri, Aug 21, 2015 at 11:29 AM, Guido Medina <[email protected] > <javascript:>> wrote: > >> Try the JVM option -XX:+UseG1GC, assuming you are using at least JVM >> 7uLatest where latest should be 79 or 80. The new GC algorithm doesn't stop >> world, it works more often and can make your application a bit slower (on a >> very hyper micro-level) but GC will now be consistent. >> >> >> On Friday, August 21, 2015 at 6:36:11 AM UTC+1, Ara Tatous wrote: >> >>> I have a configuration of Akka cluster with 4 nodes. Let's call them >>> Engine and Warehouse. >>> >>> In Engine: >>> >>> >>> >>> *Everything is normal* >>> >>> [DEBUG] [08/20/2015 07:32:25.082] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-80] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:25.082] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-80] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:32:25.082] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-80] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:32:25.084] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-98] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:32:25.084] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-98] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:25.084] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-98] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> >>> >>> *We have GC pause here* >>> >>> 2015-08-20T07:32:31.959-0700: [GC2015-08-20T07:32:31.960-0700: [ParNew >>> (promotion failed): 172544K->170208K(172544K), 0.8968931 >>> secs]2015-08-20T07:32:32.857-0700: [CMS2015-08-20T07:32:39.682-0700: >>> [CMS-concurrent-sweep: 7.003/7.909 secs] [Times: user=8.78 sys=0.40, >>> real=7.91 secs] >>> >>> (concurrent mode failure): 5760566K->2755481K(6099776K), 30.9310318 >>> secs] 5924201K->2755481K(6272320K), [CMS Perm : 142090K->141971K(237016K)], >>> 31.8286597 secs] [Times: user=32.22 sys=0.38, real=31.83 secs] >>> >>> >>> >>> *We have almost 32 seconds GC pause. The following heartbeat request are >>> buffered and played right after GC pause in one single second!. All the >>> connection is being disassociated with other cluster members.* >>> >>> >>> >>> [DEBUG] [08/20/2015 07:33:03.794] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-23] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.794] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-23] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.794] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-23] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-81] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-81] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-81] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-97] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-97] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.808] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-97] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> >>> >>> *After GC pause we have these disassociations* >>> >>> [DEBUG] [08/20/2015 07:33:03.823] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.823] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [WARN] [08/20/2015 07:33:03.823] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0] >>> >>> Association with remote system >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] has failed, address >>> is now gated for [5000] ms. Reason: [Disassociated] >>> >>> [WARN] [08/20/2015 07:33:03.827] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2] >>> >>> Association with remote system >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] has failed, address >>> is now gated for [5000] ms. Reason: [Disassociated] >>> >>> [DEBUG] [08/20/2015 07:33:03.838] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.839] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.839] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [WARN] [08/20/2015 07:33:03.839] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8] >>> >>> Association with remote system >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] has failed, address >>> is now gated for [5000] ms. Reason: [Disassociated] >>> >>> [DEBUG] [08/20/2015 07:33:03.851] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.851] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-13] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.852] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-13] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> >>> >>> *But, after gaining back the associating, we have buffer drainage.* >>> >>> [DEBUG] [08/20/2015 07:33:03.855] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [Remoting] Associated >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.856] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [Remoting] Associated >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.889] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter] >>> >>> Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.890] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-0/endpointWriter] >>> Drained >>> buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: >>> 0, noBackoffCount: 0 , adaptiveBackoff: 1000 >>> >>> [DEBUG] [08/20/2015 07:33:03.891] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter] >>> >>> Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:03.892] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2552-2/endpointWriter] >>> Drained >>> buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: >>> 0, noBackoffCount: 0 , adaptiveBackoff: 1000 >>> >>> [DEBUG] [08/20/2015 07:33:03.912] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter] >>> >>> Associated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] -> >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.912] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod2%3A2551-8/endpointWriter] >>> Drained >>> buffer with maxWriteCount: 50, fullBackoffCount: 1, smallBackoffCount: >>> 0, noBackoffCount: 0 , adaptiveBackoff: 1000 >>> >>> [DEBUG] [08/20/2015 07:33:03.938] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-147] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-10] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:03.938] >>> [EngineWarehouseActorSystem-akka.remote.default-remote-dispatcher-79] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FEngineWarehouseActorSystem%40compprod1%3A2551-10] >>> >>> Disassociated [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <- >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> >>> >>> >>> >>> *After that Everything is normal again* >>> >>> [DEBUG] [08/20/2015 07:33:09.054] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-87] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:09.054] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-87] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:33:09.054] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-87] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:09.056] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-89] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:33:09.056] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-89] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] >>> >>> [DEBUG] [08/20/2015 07:33:09.056] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-87] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *Meanwhile in Warehouse:* >>> >>> >>> >>> *Everything is normal so far until* >>> >>> [DEBUG] [08/20/2015 07:32:29.023] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-25] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] >>> >>> [DEBUG] [08/20/2015 07:32:29.024] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-25] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:29.024] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-25] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:32:29.025] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-15] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] >>> >>> [DEBUG] [08/20/2015 07:32:29.025] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-15] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:29.025] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-25] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> *Engine is running 32 seconds GC pause now* >>> >>> [DEBUG] [08/20/2015 07:32:40.035] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-21] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-First >>> heartbeat req w/o reply >>> >>> [DEBUG] [08/20/2015 07:32:40.035] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-21] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:40.035] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-21] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:32:40.036] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-20] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:40.037] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-21] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> [DEBUG] [08/20/2015 07:32:51.033] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-30] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod2:2552] <-2’nd >>> heartbeat req w/o reply >>> >>> [DEBUG] [08/20/2015 07:32:51.033] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-30] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:51.033] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-30] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat to [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> [DEBUG] [08/20/2015 07:32:51.034] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-20] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2551] >>> >>> [DEBUG] [08/20/2015 07:32:51.035] >>> [EngineWarehouseActorSystem-akka.actor.default-dispatcher-30] >>> [akka.tcp://EngineWarehouseActorSystem@compprod2:2551/system/cluster/core/daemon/heartbeatSender] >>> >>> Cluster Node [akka.tcp://EngineWarehouseActorSystem@compprod2:2551] - >>> Heartbeat response from >>> [akka.tcp://EngineWarehouseActorSystem@compprod1:2552] >>> >>> >>> >>> *Failure Transport detector kicks in and disassociation happens* >>> ... >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
