Hi Chris,
The default supervision strategy for the user guardian handles the
following exceptions:
case _: ActorInitializationException => Stop
case _: ActorKilledException => Stop
case _: DeathPactException => Stop
case _: Exception => Restart
Other exceptions are escalated and the root guardian will stop the user
guardian, and shutdown the system.
I would guess that you throw some kind of Error, e.g. AssertionError
The root guardian logs this: log.error(ex, "guardian failed, shutting down
system")
I don't know why you don't see the ex in your logs, maybe your logger
doesn't print that?
It is allowed with null message in LogEvent (according to its doc) so you
should handle that in the logger. NullPointerException should not be the
cause of the shutdown though.
Cheers,
Patrik
On Thu, Feb 20, 2014 at 1:04 PM, oxbow_lakes <[email protected]> wrote:
> I have a client/server application and recently we have started seeing
> issues with clients "disconnecting" themselves from the server whilst
> remaining up & running and *without receiving a Terminated event* (so the
> client just stops receiving any messages from the server but I cannot take
> any action because I don;t know this has happened). We have finally tracked
> down this happening and captured the client logs. The client just suddenly
> logs the message *"guardian failed, shutting down system"*. There's no
> obvious reason for this prior in the logs (no mention of any exceptions or
> errors, such as OOME)
>
> Here are the full client logs. Note the exception at the end seems to be
> the result of my logger receiving a LogEvent with a null message (not clear
> whether this is a cause of the shutdown, or a symptom).
>
> Can anyone shed any light on what "*guardian failed, shutting down system*"
> means and any actions I can take to isolate/fix/workaround the issue. I'm
> using *akka 2.2.3*
>
> Cheers,
>
> Chris
>
> 14-Feb-2014 08:03:58:820: [akka.akka://papagui/system/remote-watcher -
> FINE] [38]: Sending Heartbeat to [akka.tcp://
> [email protected]:48173]
> 14-Feb-2014 08:03:58:822: [akka.akka://papagui/system/remote-watcher -
> FINE] [46]: Received heartbeat rsp from [akka.tcp://
> [email protected]:48173]
> *14-Feb-2014 08:04:03:334: [akka.LocalActorRefProvider(akka://papagui) -
> SEVERE] [30]: guardian failed, shutting down system*
> 14-Feb-2014 08:04:03:353:
> [akka.akka.serialization.Serialization(akka://papagui) - FINE] [30]: Using
> serializer[akka.serialization.JavaSerializer] for message
> [akka.dispatch.sysmsg.Unwatch]
> 14-Feb-2014 08:04:03:354: [akka.akka://papagui/system/remote-watcher -
> FINE] [42]: Unwatching: [akka://papagui/user/canary/client -> akka.tcp://
> [email protected]:48173/user/repl]
> 14-Feb-2014 08:04:03:354: [akka.akka://papagui/system/remote-watcher -
> FINE] [42]: Cleanup self watch of [akka.tcp://
> [email protected]:48173/user/repl]
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/system/remote-watcher -
> FINE] [42]: Unwatched last watchee of node: [akka.tcp://
> [email protected]:48173]
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO]
> [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from
> Actor[akka.tcp://
> [email protected]:48173/user/gekko/reporter#-494768958]
> to Actor[akka://papagui/user/canary/client#465055827] was not delivered.
> [1] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO]
> [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from
> Actor[akka.tcp://
> [email protected]:48173/user/gekko/reporter#-494768958]
> to Actor[akka://papagui/user/canary/client#465055827] was not delivered.
> [2] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO]
> [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from
> Actor[akka.tcp://
> [email protected]:48173/user/gekko/reporter#-494768958]
> to Actor[akka://papagui/user/canary/client#465055827] was not delivered.
> [3] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO]
> [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from
> Actor[akka.tcp://
> [email protected]:48173/user/gekko/reporter#-494768958]
> to Actor[akka://papagui/user/canary/client#465055827] was not delivered.
> [4] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:373: [akka.akka://papagui/user/canary/client - INFO]
> [46]: Message [gsa.gekko.common.broadcast.impl.LeafImpl] from
> Actor[akka.tcp://
> [email protected]:48173/user/gekko/reporter#-494768958]
> to Actor[akka://papagui/user/canary/client#465055827] was not delivered.
> [5] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:373:
> [akka.akka.serialization.Serialization(akka://papagui) - FINE] [52]: Using
> serializer[akka.serialization.JavaSerializer] for message
> [akka.dispatch.sysmsg.DeathWatchNotification]
> 14-Feb-2014 08:04:03:384: [akka.akka://papagui/system/remoting-terminator
> - INFO] [45]: Shutting down remote daemon.
> 14-Feb-2014 08:04:03:386: [akka.akka://papagui/system/remoting-terminator
> - INFO] [45]: Remote daemon shut down; proceeding with flushing remote
> transports.
> 14-Feb-2014 08:04:03:439:
> [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter - FINE] [42]:
> Disassociated [akka.tcp://[email protected]:63135] -> [akka.tcp://
> [email protected]:48173]
> 14-Feb-2014 08:04:03:440:
> [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com
> %3A48173-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0 - INFO] [54]: Message
> [akka.dispatch.sysmsg.Terminate] from
> Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0/endpoint
> Writer/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0#-306838080] to
> Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com
> %3A48173-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0#-306838080] was not delivered.
> [6] dead letters encountered. This logging can be turned off or adjusted
> with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:444:
> [akka.akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-1 - INFO] [45]: Message
> [akka.remote.transport.AssociationHandle$Disassociated] from
> Actor[akka://papagui/deadLetters] to
> Actor[akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-1#248771270] was not delivered.
> [7] dead let
> ters encountered. This logging can be turned off or adjusted with
> configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:445:
> [akka.akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter - INFO] [42]:
> Message [akka.dispatch.sysmsg.DeathWatchNotification] from
> Actor[akka://papagui/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter#-407814177] to
> Actor[akka://papagui/system/endpointManager/reliableEn
> dpointWriter-akka.tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-0/endpointWriter#-407814177] was
> not delivered. [8] dead letters encountered. This logging can be turned off
> or adjusted with configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> 14-Feb-2014 08:04:03:445:
> [akka.akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-1 - INFO] [49]: Message
> [akka.remote.transport.AssociationHandle$Disassociated] from
> Actor[akka://papagui/deadLetters] to
> Actor[akka://papagui/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FgekkoRemoting%
> 40loypcore4.intra.gsacapital.com%3A48173-1#248771270] was not delivered.
> [9] dead let
> ters encountered. This logging can be turned off or adjusted with
> configuration settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> [ERROR] [02/14/2014 08:04:03.467]
> [papagui-akka.actor.default-dispatcher-4]
> [akka://papagui/system/log1-Logger] null
> java.lang.NullPointerException
> at
> gsa.papa.client.util.Logger.gsa$papa$client$util$Logger$$log_(Logger.scala:27)
> at
> gsa.papa.client.util.Logger$$anonfun$receive$1.applyOrElse(Logger.scala:12)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ: http://akka.io/faq/
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Patrik Nordwall
Typesafe <http://typesafe.com/> - Reactive apps on the JVM
Twitter: @patriknw
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
