Thanks Endre, I'll try increasing the akka.remote.transport-failure- detector.acceptable-heartbeat-pause
However I'm a little concerned in general that the ActorSystems are not recovering their connectivity without restarting the ActorSystem1 JVM. Do I need to change the config or but in a different Supervisor strategy to make this work? ActorSystem1 seems to make some attempt at it but it doesn't seem to work.. [2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor. default-dispatcher-11] akka.remote.EndpointWriter - restarted [2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted [2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939] [2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939] [2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840) [2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840) [2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://[email protected]:60789] -> [akka.tcp://DataAugmentation@server002:2552] [2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://[email protected]:60789] -> [akka.tcp://DataAugmentation@server002:2552] [2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] [2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] [2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d) [2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d) [2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] [2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] On Friday, 24 January 2014 15:23:05 UTC, Akka Team wrote: > > Hi Brian, > > > On Fri, Jan 24, 2014 at 4:14 PM, Brian Fowler > <[email protected]<javascript:> > > wrote: > >> >> Hi Endre, >> >> No the two systems can comunicate initially, the problem just occurs >> after a quiet period of about 5 minutes. The problem also does not occur >> with Akka 2.1.4 so I don't think it is a network issue. >> > > Akka 2.1.4 had no real recovery mechanism (and other important features) > so it probably never triggers it. > > That the systems can communicate initially is not really a proof because > actor systems reuse the incoming connection when possible. So it can be > like this: > > 1. system1 connects to system2 on address > akka.tcp://DataAugmentation@server002:2552 (this direction is fine) > 2. communication is fine > 3. for some reason system1 shuts of the connection (most probably > heartbeat misses, you should increase the > akka.remote.transport-failure-detector.acceptable-heartbeat-pause setting > to something like 10s -- but I need more log entries). > 4. Now system2 tries to recover by connecting to system1 on address > 127.0.0.1:60789 > and this direction fails. > > What is suspicious to me is this error: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: /127.0.0.1:60789 > This is thrown by netty and means the connection is not accepted. > > You can try to force this scenario by setting > akka.remote.use-passive-connections = off on both nodes (this prevents > system2 to reuse the connection from system1). Then we might see what > happens. > > -Endre > > >> >> Thanks, >> >> Brian >> >> >> On Friday, 24 January 2014 15:05:12 UTC, Akka Team wrote: >> >>> Hi Brian, >>> >>> >>> On ActorSystem1 I get >>>> >>> >>> Actually the interesting part is more probably earlier in this log. Btw, >>> is the port 60789 firewalled? It seems like ActorSystem2 is trying to >>> connect back to system1 and it cannot. >>> >>> -Endre >>> >>> >>>> >>>> [2014-01-24 14:26:14,620] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.remote.transport.ProtocolStateActor - stopped >>>> [2014-01-24 14:26:14,620] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.remote.transport.ProtocolStateActor - stopped >>>> [2014-01-24 14:26:14,620] [INFO] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>>> AssociationHandle$Disassociated] from >>>> Actor[akka://AkkaXLServer/deadLetters] >>>> to Actor[akka://AkkaXLServer/system/transports/ >>>> akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation% >>>> 40server002%3A2552-1#1651503039] was not delivered. [1] dead letters >>>> encountered. This logging can be turned off or adjusted with configuration >>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>> shutdown'. >>>> [2014-01-24 14:26:14,620] [INFO] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>>> AssociationHandle$Disassociated] from >>>> Actor[akka://AkkaXLServer/deadLetters] >>>> to Actor[akka://AkkaXLServer/system/transports/ >>>> akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation% >>>> 40server002%3A2552-1#1651503039] was not delivered. [1] dead letters >>>> encountered. This logging can be turned off or adjusted with configuration >>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>> shutdown'. >>>> [2014-01-24 14:26:14,621] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.remote.EndpointReader - stopped >>>> [2014-01-24 14:26:14,621] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>> akka.remote.EndpointReader - stopped >>>> [2014-01-24 14:26:14,623] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@ >>>> server002:2552] >>>> [2014-01-24 14:26:14,623] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@ >>>> server002:2552] >>>> [2014-01-24 14:26:14,628] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>> akka.remote.EndpointWriter - restarting >>>> [2014-01-24 14:26:14,628] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>> akka.remote.EndpointWriter - restarting >>>> [2014-01-24 14:26:14,630] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@ >>>> server002:2552] >>>> [2014-01-24 14:26:14,630] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@ >>>> server002:2552] >>>> [2014-01-24 14:26:14,630] [INFO] >>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>>> ActorTransportAdapter$DisassociateUnderlying] from >>>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. >>>> [2] dead letters encountered. This logging can be turned off or adjusted >>>> with configuration settings 'akka.log-dead-letters' and >>>> 'akka.log-dead-letters-during-shutdown'. >>>> [2014-01-24 14:26:14,630] [INFO] >>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>>> ActorTransportAdapter$DisassociateUnderlying] from >>>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. >>>> [2] dead letters encountered. This logging can be turned off or adjusted >>>> with configuration settings 'akka.log-dead-letters' and >>>> 'akka.log-dead-letters-during-shutdown'. >>>> [2014-01-24 14:26:14,632] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>> akka.remote.EndpointWriter - restarted >>>> [2014-01-24 14:26:14,632] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>> akka.remote.EndpointWriter - restarted >>>> [2014-01-24 14:26:14,633] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>>> 1273774939] >>>> [2014-01-24 14:26:14,633] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>>> 1273774939] >>>> [2014-01-24 14:26:14,634] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>> akka.remote.transport.ProtocolStateActor - started >>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>> [2014-01-24 14:26:14,634] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>> akka.remote.transport.ProtocolStateActor - started >>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>> [2014-01-24 14:26:17,126] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>> akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. >>>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>>> [2014-01-24 14:26:17,126] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>> akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. >>>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>>> [2014-01-24 14:26:17,127] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>> system/endpointManager/reliableEndpointWriter-akka. >>>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>>> endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% >>>> 40server002%3A2552-0#936701101] >>>> [2014-01-24 14:26:17,127] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>> system/endpointManager/reliableEndpointWriter-akka. >>>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>>> endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% >>>> 40server002%3A2552-0#936701101] >>>> [2014-01-24 14:26:17,129] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@ >>>> 1bd97d0d) >>>> [2014-01-24 14:26:17,129] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@ >>>> 1bd97d0d) >>>> [2014-01-24 14:26:17,129] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>> system/endpointManager/reliableEndpointWriter-akka. >>>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>>> endpointWriter#226701486] >>>> [2014-01-24 14:26:17,129] [DEBUG] >>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>> system/endpointManager/reliableEndpointWriter-akka. >>>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>>> endpointWriter#226701486] >>>> >>>> On ActorSystem2 >>>> >>>> [2014-01-24 14:26:16,473] [ERROR] >>>> [DataAugmentation-akka.actor.default-dispatcher-234] >>>> akka.remote.EndpointWriter - AssociationError >>>> [akka.tcp://DataAugmentation@server002:2552] >>>> -> [akka.tcp://[email protected]:60789]: Error [Association >>>> failed with [akka.tcp://[email protected]:60789]] [ >>>> akka.remote.EndpointAssociationException: Association failed with >>>> [akka.tcp://[email protected]:60789] >>>> Caused by: >>>> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: >>>> Connection refused: /127.0.0.1:60789 >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>> group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> >>> >>> -- >>> Akka Team >>> Typesafe - The software stack for applications that scale >>> Blog: letitcrash.com >>> Twitter: @akkateam >>> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Akka Team > Typesafe - The software stack for applications that scale > Blog: letitcrash.com > Twitter: @akkateam > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
