I have a client/server application and recently we have started seeing issues with clients "disconnecting" themselves from the server whilst remaining up & running and *without receiving a Terminated event* (so the client just stops receiving any messages from the server but I cannot take any action because I don;t know this has happened). We have finally tracked down this happening and captured the client logs. The client just suddenly logs the message *"guardian failed, shutting down system"*. There's no obvious reason for this prior in the logs (no mention of any exceptions or errors, such as OOME)
Here are the full client logs. Note the exception at the end seems to be the result of my logger receiving a LogEvent with a null message (not clear whether this is a cause of the shutdown, or a symptom). Can anyone shed any light on what "*guardian failed, shutting down system*" means and any actions I can take to isolate/fix/workaround the issue. I'm using *akka 2.2.3* Cheers, Chris 14-Feb-2014 08:03:58:820: [akka.akka://papagui/system/remote-watcher - FINE] [38]: Sending Heartbeat to [akka.tcp://[email protected]:48173] 14-Feb-2014 08:03:58:822: [akka.akka://papagui/system/remote-watcher - FINE] [46]: Received heartbeat rsp from [akka.tcp://[email protected]:48173] *14-Feb-2014 08:04:03:334: [akka.LocalActorRefProvider(akka://papagui) - SEVERE] [30]: guardian failed, shutting down system* 14-Feb-2014 08:04:03:353: [akka.akka.serialization.Serialization(akka://papagui) - FINE] [30]: Using serializer[akka.serialization.JavaSerializer] for message [akka.dispatch.sysmsg.Unwatch] 14-Feb-2014 08:04:03:354: [akka.akka://papagui/system/remote-watcher - FINE] [42]: Unwatching: [akka://papagui/user/canary/client -> akka.tcp://[email protected]:48173/user/repl] 14-Feb-2014 08:04:03:354: [akka.akka://papagui/system/remote-watcher - FINE] [42]: Cleanup self watch of [akka.tcp://[email protected]:48173/user/repl] 14-Feb-2014 08:04:03:373: [akka.akka://papagui/system/remote-watcher - FINE] [42]: Unwatched last watchee of node: [akka.tcp://[email protected]:48173] 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO] [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from Actor[akka.tcp://[email protected]:48173/user/gekko/reporter#-494768958] to Actor[akka://papagui/user/canary/client#465055827] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO] [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from Actor[akka.tcp://[email protected]:48173/user/gekko/reporter#-494768958] to Actor[akka://papagui/user/canary/client#465055827] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO] [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from Actor[akka.tcp://[email protected]:48173/user/gekko/reporter#-494768958] to Actor[akka://papagui/user/canary/client#465055827] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO] [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from Actor[akka.tcp://[email protected]:48173/user/gekko/reporter#-494768958] to Actor[akka://papagui/user/canary/client#465055827] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO] [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from Actor[akka.tcp://[email protected]:48173/user/gekko/reporter#-494768958] to Actor[akka://papagui/user/canary/client#465055827] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:373: [akka.akka.serialization.Serialization(akka://papagui) - FINE] [52]: Using serializer[akka.serialization.JavaSerializer] for message [akka.dispatch.sysmsg.DeathWatchNotification] 14-Feb-2014 08:04:03:384: [akka.akka://papagui/system/remoting-terminator - INFO] [45]: Shutting down remote daemon. 14-Feb-2014 08:04:03:386: [akka.akka://papagui/system/remoting-terminator - INFO] [45]: Remote daemon shut down; proceeding with flushing remote transports. 14-Feb-2014 08:04:03:439: [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter - FINE] [42]: Disassociated [akka.tcp://[email protected]:63135] -> [akka.tcp://[email protected]:48173] 14-Feb-2014 08:04:03:440: [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0 - INFO] [54]: Message [akka.dispatch.sysmsg.Terminate] from Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpoint Writer/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0#-306838080] to Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0#-306838080] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:444: [akka.akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-1 - INFO] [45]: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://papagui/deadLetters] to Actor[akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-1#248771270] was not delivered. [7] dead let ters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:445: [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter - INFO] [42]: Message [akka.dispatch.sysmsg.DeathWatchNotification] from Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter#-407814177] to Actor[akka://papagui/system/endpointManager/reliableEn dpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter#-407814177] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14-Feb-2014 08:04:03:445: [akka.akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-1 - INFO] [49]: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://papagui/deadLetters] to Actor[akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%40loypcore4.intra.gsacapital.com%3A48173-1#248771270] was not delivered. [9] dead let ters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [ERROR] [02/14/2014 08:04:03.467] [papagui-akka.actor.default-dispatcher-4] [akka://papagui/system/log1-Logger] null java.lang.NullPointerException at gsa.papa.client.util.Logger.gsa$papa$client$util$Logger$$log_(Logger.scala:27) at gsa.papa.client.util.Logger$$anonfun$receive$1.applyOrElse(Logger.scala:12) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
