This could be one known issue in 2.3.2, please upgrade to 2.3.3 (or better, to 2.3.4 right away) and we’ll talk :-)
— k On 9 July 2014 at 10:44:17, Pierre Falda ([email protected]) wrote: Hello everyone, I had a simple two layer akka (test)deployment with akka 2.2.1 and was working well. It was quite simple: a machine is responsible for router creation and forwarding requests it receives from play servers to the worker. Some days ago I tried to switch to 2.3.2 and I found some problems in the process. The two nodes seems cannot connect anymore (i found no network activity at all on the specified tcp ports) and I don't get any very explicit message. just for paranoia I tried to add a err.print message in constructor of the actor that the router (/ResolveService) should spawn (ResolveServiceActor.class) and track the execution flow with intellij debugger , and according to the network inspection it seems it's not called at all. >From the previous environment, I changed only the akka/scala version and the >conf files. The application is written in java. I attached the conf of router/worker nodes. Here the router actorsystem stdout log: [DEBUG] [07/09/2014 10:13:23.861] [main] [EventStream] StandardOutLogger started [DEBUG] [07/09/2014 10:13:24.038] [main] [EventStream(akka://BrainRoutingSystem)] logger log1-Logging$DefaultLogger started [DEBUG] [07/09/2014 10:13:24.038] [main] [EventStream(akka://BrainRoutingSystem)] logger log1-Logging$DefaultLogger started [DEBUG] [07/09/2014 10:13:24.039] [main] [EventStream(akka://BrainRoutingSystem)] Default Loggers started [DEBUG] [07/09/2014 10:13:24.039] [main] [EventStream(akka://BrainRoutingSystem)] Default Loggers started [INFO] [07/09/2014 10:13:24.056] [main] [Remoting] Starting remoting [INFO] [07/09/2014 10:13:24.204] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://BrainRoutingSystem@Vortex:2553] [INFO] [07/09/2014 10:13:24.206] [main] [Remoting] Remoting now listens on addresses: [akka.tcp://BrainRoutingSystem@Vortex:2553] [DEBUG] [07/09/2014 10:13:24.224] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [RemoteActorRefProvider] [akka://BrainRoutingSystem/] Instantiating Remote Actor [akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c1] [DEBUG] [07/09/2014 10:13:24.227] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [RemoteActorRefProvider] [akka://BrainRoutingSystem/] Instantiating Remote Actor [akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c2] [DEBUG] [07/09/2014 10:13:24.229] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watching: [akka://BrainRoutingSystem/system/remote-deployment-watcher -> akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c1] [DEBUG] [07/09/2014 10:13:24.229] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watching: [akka://BrainRoutingSystem/user/ResolveService -> akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c1] [DEBUG] [07/09/2014 10:13:24.230] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watching: [akka://BrainRoutingSystem/system/remote-deployment-watcher -> akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c2] [DEBUG] [07/09/2014 10:13:24.230] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watching: [akka://BrainRoutingSystem/user/ResolveService -> akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c2] [DEBUG] [07/09/2014 10:13:24.230] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Unwatching: [akka://BrainRoutingSystem/user/ResolveService -> akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c2] [WARN] [07/09/2014 10:13:24.242] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDisBrainBackend%40Vortex%3A2552-0] Association with remote system [akka.tcp://DisBrainBackend@Vortex:2552] has failed, address is now gated for [10000] ms. Reason is: [exception during creation]. [INFO] [07/09/2014 10:13:24.247] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.DaemonMsgCreate] from Actor[akka://BrainRoutingSystem/deadLetters] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [07/09/2014 10:13:24.248] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.DaemonMsgCreate] from Actor[akka://BrainRoutingSystem/deadLetters] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [07/09/2014 10:13:24.348] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka://BrainRoutingSystem/deadLetters] Message [akka.actor.PoisonPill$] from Actor[akka://BrainRoutingSystem/deadLetters] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:25.228] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending first Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:25.230] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:26.228] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending first Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:26.229] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:27.228] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending first Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:27.229] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:28.228] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending first Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:28.229] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:28.248] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Trigger extra expected heartbeat from [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:29.228] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:29.229] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:30.228] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:30.229] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:31.228] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [INFO] [07/09/2014 10:13:31.229] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka://BrainRoutingSystem/deadLetters] Message [akka.remote.RemoteWatcher$Heartbeat$] from Actor[akka://BrainRoutingSystem/system/remote-watcher#-1659998873] to Actor[akka://BrainRoutingSystem/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [DEBUG] [07/09/2014 10:13:32.228] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:33.228] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:34.228] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [WARN] [07/09/2014 10:13:34.270] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDisBrainBackend%40Vortex%3A2552-0] Association with remote system [akka.tcp://DisBrainBackend@Vortex:2552] has failed, address is now gated for [10000] ms. Reason is: [exception during creation]. [DEBUG] [07/09/2014 10:13:35.228] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:36.228] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:37.228] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:38.227] [BrainRoutingSystem-akka.actor.default-dispatcher-4] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:39.228] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:40.228] [BrainRoutingSystem-akka.actor.default-dispatcher-3] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [DEBUG] [07/09/2014 10:13:41.228] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Sending Heartbeat to [akka.tcp://DisBrainBackend@Vortex:2552] [WARN] [07/09/2014 10:13:41.229] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Detected unreachable: [akka.tcp://DisBrainBackend@Vortex:2552] [WARN] [07/09/2014 10:13:41.229] [BrainRoutingSystem-akka.actor.default-dispatcher-2] [Remoting] Association to [akka.tcp://DisBrainBackend@Vortex:2552] with unknown UID is reported as quarantined, but address cannot be quarantined without knowing the UID, gating instead for 10000 ms. [DEBUG] [07/09/2014 10:13:41.237] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watchee terminated: [akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c1] [DEBUG] [07/09/2014 10:13:41.238] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Watchee terminated: [akka.tcp://DisBrainBackend@Vortex:2552/remote/akka.tcp/BrainRoutingSystem@Vortex:2553/user/ResolveService/c2] [DEBUG] [07/09/2014 10:13:41.238] [BrainRoutingSystem-akka.actor.default-dispatcher-24] [akka.tcp://BrainRoutingSystem@Vortex:2553/system/remote-watcher] Unwatched last watchee of node: [akka.tcp://DisBrainBackend@Vortex:2552] Do you have any idea on how to resolve/identify this problem? Thanks in advance Pierre Falda -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Konrad 'ktoso' Malawski hAkker @ typesafe http://akka.io -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
