Hi Brian,
On Fri, Jan 24, 2014 at 4:44 PM, Brian Fowler <[email protected]>wrote: > Thanks Endre, I'll try increasing the akka.remote.transport-failure- > detector.acceptable-heartbeat-pause > > However I'm a little concerned in general that the ActorSystems are not > recovering their connectivity without restarting the ActorSystem1 JVM. Do I > need to change the config or but in a different Supervisor strategy to make > this work? > Restart is not needed, unless one of the nodes quarantined the other (but that would be in the logs). I suspect something else here: from your log it seems that system one connects fine: akka.remote.EndpointWriter - Associated [akka.tcp:// [email protected]:60789] -> [akka.tcp://DataAugmentation@server002 :2552] As I said, I suspect the direction from system2 to system1 to be problematic. I think the two systems are trying to connect at the same time, and therefore the connection from system1 on system2 to is accepted for reading only, but the writing direction (system2 to system1) is failing for some reason (Connection Refused). You can debug what messages are actually getting through: akka.remote.log-received-messages = on akka.remote.log-sent-messages = on Then you will see the actual remote messages that are attempted to be sent and that are actually received in the log. Btw without a longer log it is hard to see the whole picture. -Endre > > ActorSystem1 seems to make some attempt at it but it doesn't seem to work.. > > > [2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor. > default-dispatcher-11] akka.remote.EndpointWriter - restarted > [2014-01-24 14:26:14,632] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-11] > akka.remote.EndpointWriter - restarted > [2014-01-24 14:26:14,633] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-24] > akka.remote.transport.AkkaProtocolManager - now supervising > Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ > akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939] > [2014-01-24 14:26:14,633] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-24] > akka.remote.transport.AkkaProtocolManager - now supervising > Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ > akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939] > [2014-01-24 14:26:14,634] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-24] > akka.remote.transport.ProtocolStateActor - started (akka.remote.transport. > ProtocolStateActor@549ad840) > [2014-01-24 14:26:14,634] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-24] > akka.remote.transport.ProtocolStateActor - started (akka.remote.transport. > ProtocolStateActor@549ad840) > [2014-01-24 14:26:17,126] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-20] > akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. > 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] > [2014-01-24 14:26:17,126] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-20] > akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127. > 0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] > [2014-01-24 14:26:17,127] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-20] > akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ > system/endpointManager/reliableEndpointWriter-akka. > tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ > endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% > 40server002%3A2552-0#936701101] > [2014-01-24 14:26:17,127] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-20] > akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ > system/endpointManager/reliableEndpointWriter-akka. > tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ > endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation% > 40server002%3A2552-0#936701101] > [2014-01-24 14:26:17,129] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-18] > akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d) > [2014-01-24 14:26:17,129] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-18] > akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d) > [2014-01-24 14:26:17,129] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-18] > akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ > system/endpointManager/reliableEndpointWriter-akka. > tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ > endpointWriter#226701486] > [2014-01-24 14:26:17,129] [DEBUG] > [AkkaXLServer-akka.actor.default-dispatcher-18] > akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ > system/endpointManager/reliableEndpointWriter-akka. > tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/ > endpointWriter#226701486] > > > On Friday, 24 January 2014 15:23:05 UTC, Akka Team wrote: > >> Hi Brian, >> >> >> On Fri, Jan 24, 2014 at 4:14 PM, Brian Fowler <[email protected]>wrote: >> >>> >>> Hi Endre, >>> >>> No the two systems can comunicate initially, the problem just occurs >>> after a quiet period of about 5 minutes. The problem also does not occur >>> with Akka 2.1.4 so I don't think it is a network issue. >>> >> >> Akka 2.1.4 had no real recovery mechanism (and other important features) >> so it probably never triggers it. >> >> That the systems can communicate initially is not really a proof because >> actor systems reuse the incoming connection when possible. So it can be >> like this: >> >> 1. system1 connects to system2 on address >> akka.tcp://DataAugmentation@server002:2552 >> (this direction is fine) >> 2. communication is fine >> 3. for some reason system1 shuts of the connection (most probably >> heartbeat misses, you should increase the akka.remote.transport-failure- >> detector.acceptable-heartbeat-pause setting to something like 10s -- but >> I need more log entries). >> 4. Now system2 tries to recover by connecting to system1 on address >> 127.0.0.1:60789 >> and this direction fails. >> >> What is suspicious to me is this error: akka.remote.transport.netty. >> NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: / >> 127.0.0.1:60789 >> This is thrown by netty and means the connection is not accepted. >> >> You can try to force this scenario by setting >> akka.remote.use-passive-connections >> = off on both nodes (this prevents system2 to reuse the connection from >> system1). Then we might see what happens. >> >> -Endre >> >> >>> >>> Thanks, >>> >>> Brian >>> >>> >>> On Friday, 24 January 2014 15:05:12 UTC, Akka Team wrote: >>> >>>> Hi Brian, >>>> >>>> >>>> On ActorSystem1 I get >>>>> >>>> >>>> Actually the interesting part is more probably earlier in this log. >>>> Btw, is the port 60789 firewalled? It seems like ActorSystem2 is trying to >>>> connect back to system1 and it cannot. >>>> >>>> -Endre >>>> >>>> >>>>> >>>>> [2014-01-24 14:26:14,620] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.remote.transport.ProtocolStateActor - stopped >>>>> [2014-01-24 14:26:14,620] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.remote.transport.ProtocolStateActor - stopped >>>>> [2014-01-24 14:26:14,620] [INFO] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.Associa >>>>> tionHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>>>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server >>>>> 002%3A2552-1#1651503039] was not delivered. [1] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>> shutdown'. >>>>> [2014-01-24 14:26:14,620] [INFO] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.Associa >>>>> tionHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] >>>>> to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmana >>>>> ger.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server >>>>> 002%3A2552-1#1651503039] was not delivered. [1] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>> shutdown'. >>>>> [2014-01-24 14:26:14,621] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.remote.EndpointReader - stopped >>>>> [2014-01-24 14:26:14,621] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-10] >>>>> akka.remote.EndpointReader - stopped >>>>> [2014-01-24 14:26:14,623] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>> erver002:2552] >>>>> [2014-01-24 14:26:14,623] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-13] >>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>> erver002:2552] >>>>> [2014-01-24 14:26:14,628] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>>> akka.remote.EndpointWriter - restarting >>>>> [2014-01-24 14:26:14,628] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-19] >>>>> akka.remote.EndpointWriter - restarting >>>>> [2014-01-24 14:26:14,630] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>> erver002:2552] >>>>> [2014-01-24 14:26:14,630] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>> akka.remote.EndpointWriter - Disassociated [akka.tcp:// >>>>> [email protected]:60789] -> [akka.tcp://DataAugmentation@s >>>>> erver002:2552] >>>>> [2014-01-24 14:26:14,630] [INFO] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTr >>>>> ansportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/ >>>>> deadLetters] to Actor[akka://AkkaXLServer/system/transports/ >>>>> akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation% >>>>> 40server002%3A2552-1#1651503039] was not delivered. [2] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>> shutdown'. >>>>> [2014-01-24 14:26:14,630] [INFO] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-2] >>>>> akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTr >>>>> ansportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/ >>>>> deadLetters] to Actor[akka://AkkaXLServer/system/transports/ >>>>> akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation% >>>>> 40server002%3A2552-1#1651503039] was not delivered. [2] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during- >>>>> shutdown'. >>>>> [2014-01-24 14:26:14,632] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>>> akka.remote.EndpointWriter - restarted >>>>> [2014-01-24 14:26:14,632] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-11] >>>>> akka.remote.EndpointWriter - restarted >>>>> [2014-01-24 14:26:14,633] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>>>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>>>> 1273774939] >>>>> [2014-01-24 14:26:14,633] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>> akka.remote.transport.AkkaProtocolManager - now supervising >>>>> Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/ >>>>> akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2# >>>>> 1273774939] >>>>> [2014-01-24 14:26:14,634] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>> akka.remote.transport.ProtocolStateActor - started >>>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>>> [2014-01-24 14:26:14,634] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-24] >>>>> akka.remote.transport.ProtocolStateActor - started >>>>> (akka.remote.transport.ProtocolStateActor@549ad840) >>>>> [2014-01-24 14:26:17,126] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>> akka.remote.EndpointWriter - Associated [akka.tcp://[email protected] >>>>> .0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>>>> [2014-01-24 14:26:17,126] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>> akka.remote.EndpointWriter - Associated [akka.tcp://[email protected] >>>>> .0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552] >>>>> [2014-01-24 14:26:17,127] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>>>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>>>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>>>> [2014-01-24 14:26:17,127] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-20] >>>>> akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/ >>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>>>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader- >>>>> akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101] >>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>>>> d97d0d) >>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>> akka.remote.EndpointReader - started (akka.remote.EndpointReader@1b >>>>> d97d0d) >>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>>>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>>>> [2014-01-24 14:26:17,129] [DEBUG] >>>>> [AkkaXLServer-akka.actor.default-dispatcher-18] >>>>> akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/ >>>>> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F% >>>>> 2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486] >>>>> >>>>> On ActorSystem2 >>>>> >>>>> [2014-01-24 14:26:16,473] [ERROR] >>>>> [DataAugmentation-akka.actor.default-dispatcher-234] >>>>> akka.remote.EndpointWriter - AssociationError >>>>> [akka.tcp://DataAugmentation@ >>>>> server002:2552] -> [akka.tcp://[email protected]:60789]: Error >>>>> [Association failed with [akka.tcp://[email protected]:60789]] [ >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://[email protected]:60789] >>>>> Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$ >>>>> associate$1$$anon$2: Connection refused: /127.0.0.1:60789 >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>>> >>>> -- >>>> Akka Team >>>> Typesafe - The software stack for applications that scale >>>> Blog: letitcrash.com >>>> Twitter: @akkateam >>>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Akka Team >> Typesafe - The software stack for applications that scale >> Blog: letitcrash.com >> Twitter: @akkateam >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > -- Akka Team Typesafe - The software stack for applications that scale Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
