Hi Brian,
On Fri, Jan 24, 2014 at 4:14 PM, Brian Fowler <[email protected]>wrote: > > Hi Endre, > > No the two systems can comunicate initially, the problem just occurs after > a quiet period of about 5 minutes. The problem also does not occur with > Akka 2.1.4 so I don't think it is a network issue. > Akka 2.1.4 had no real recovery mechanism (and other important features) so it probably never triggers it. That the systems can communicate initially is not really a proof because actor systems reuse the incoming connection when possible. So it can be like this: 1. system1 connects to system2 on address akka.tcp://DataAugmentation@server002:2552 (this direction is fine) 2. communication is fine 3. for some reason system1 shuts of the connection (most probably heartbeat misses, you should increase the akka.remote.transport-failure-detector.acceptable-heartbeat-pause setting to something like 10s -- but I need more log entries). 4. Now system2 tries to recover by connecting to system1 on address 127.0.0.1:60789 and this direction fails. What is suspicious to me is this error: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:60789 This is thrown by netty and means the connection is not accepted. You can try to force this scenario by setting akka.remote.use-passive-connections = off on both nodes (this prevents system2 to reuse the connection from system1). Then we might see what happens. -Endre > > Thanks, > > Brian > > > On Friday, 24 January 2014 15:05:12 UTC, Akka Team wrote: > >> Hi Brian, >> >> >> On ActorSystem1 I get >>> >> >> Actually the interesting part is more probably earlier in this log. Btw, >> is the port 60789 firewalled? It seems like ActorSystem2 is trying to >> connect back to system1 and it cannot. >> >> -Endre >> >> >>> >>> [2014-01-24 14:26:14,620] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.remote.transport.ProtocolStateActor - stopped >>> [2014-01-24 14:26:14,620] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.remote.transport.ProtocolStateActor - stopped >>> [2014-01-24 14:26:14,620] [INFO] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>> AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] >>> was not delivered. [1] dead letters encountered. This logging can be turned >>> off or adjusted with configuration settings 'akka.log-dead-letters' and >>> 'akka.log-dead-letters-during-shutdown'. >>> [2014-01-24 14:26:14,620] [INFO] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>> AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] >>> was not delivered. [1] dead letters encountered. This logging can be turned >>> off or adjusted with configuration settings 'akka.log-dead-letters' and >>> 'akka.log-dead-letters-during-shutdown'. >>> [2014-01-24 14:26:14,621] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.remote.EndpointReader - stopped >>> [2014-01-24 14:26:14,621] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>> akka.remote.EndpointReader - stopped >>> [2014-01-24 14:26:14,623] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>> akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:14,623] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>> akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:14,628] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>> akka.remote.EndpointWriter - restarting >>> [2014-01-24 14:26:14,628] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>> akka.remote.EndpointWriter - restarting >>> [2014-01-24 14:26:14,630] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>> akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:14,630] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>> akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:14,630] [INFO] >>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>> ActorTransportAdapter$DisassociateUnderlying] from >>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. >>> [2] dead letters encountered. This logging can be turned off or adjusted >>> with configuration settings 'akka.log-dead-letters' and >>> 'akka.log-dead-letters-during-shutdown'. >>> [2014-01-24 14:26:14,630] [INFO] >>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>> akka.actor.LocalActorRef - Message [akka.remote.transport. >>> ActorTransportAdapter$DisassociateUnderlying] from >>> Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/ >>> system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F% >>> 2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. >>> [2] dead letters encountered. This logging can be turned off or adjusted >>> with configuration settings 'akka.log-dead-letters' and >>> 'akka.log-dead-letters-during-shutdown'. >>> [2014-01-24 14:26:14,632] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>> akka.remote.EndpointWriter - restarted >>> [2014-01-24 14:26:14,632] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>> akka.remote.EndpointWriter - restarted >>> [2014-01-24 14:26:14,633] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.AkkaProtocolManager - now supervising >>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>> 1273774939] >>> [2014-01-24 14:26:14,633] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.AkkaProtocolManager - now supervising >>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>> 1273774939] >>> [2014-01-24 14:26:14,634] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.ProtocolStateActor - started >>> (akka.remote.transport.ProtocolStateActor@549ad840) >>> [2014-01-24 14:26:14,634] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>> akka.remote.transport.ProtocolStateActor - started >>> (akka.remote.transport.ProtocolStateActor@549ad840) >>> [2014-01-24 14:26:17,126] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:17,126] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. >>> 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>> [2014-01-24 14:26:17,127] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka. >>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>> endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% >>> 40server002%3A2552-0#936701101] >>> [2014-01-24 14:26:17,127] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka. >>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>> endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% >>> 40server002%3A2552-0#936701101] >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@ >>> 1bd97d0d) >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@ >>> 1bd97d0d) >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka. >>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>> endpointWriter#226701486] >>> [2014-01-24 14:26:17,129] [DEBUG] >>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>> system/endpointManager/reliableEndpointWriter-akka. >>> tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ >>> endpointWriter#226701486] >>> >>> On ActorSystem2 >>> >>> [2014-01-24 14:26:16,473] [ERROR] >>> [DataAugmentation-akka.actor.default-dispatcher-234] >>> akka.remote.EndpointWriter - AssociationError >>> [akka.tcp://DataAugmentation@server002:2552] >>> -> [akka.tcp://[email protected]:60789]: Error [Association failed >>> with [akka.tcp://[email protected]:60789]] [ >>> akka.remote.EndpointAssociationException: Association failed with >>> [akka.tcp://[email protected]:60789] >>> Caused by: >>> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: >>> Connection refused: /127.0.0.1:60789 >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Akka Team >> Typesafe - The software stack for applications that scale >> Blog: letitcrash.com >> Twitter: @akkateam >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
