We observe problems with both cluster sharding and cluster singletons. With sharders - usually problem is corrupted journal that prevents sharding coordinator from starting. In our situation easiest thing to do is to delete all data from journal and restart it - problem is that I can't find a way to detect that situation in a different way than observe logs - I can't find any way to detect such failure from the code. It should be pretty easy - as usually `akka.cluster.sharding.ShardCoordinator.State` throws exception with requirementFailed, but there is no way I can find a way to react on that - no easy way to put `supervisorStrategy` for shard coordinator or no other way to detect its state. We can't use `ddata` mode - as current implementation does not work in our environment, when we need to scale nodes up and down - as it requires majority of nodes to respond, it fails to work even on simplest cases of scaling down in a small cluster.
Similar situation applies to cluster singleton - if cluster singleton is stuck, there is no way to detect that situation from the code - only by observing logs. Does anybody have experience in handling such situations? I'm trying to implement some external monitoring for both things, with basically sending `Identity` message to actors that are supposed to exist - singletons - but it looks like rewriting already existing code inside akka. Maybe adding such failure detection capabilities to akka (publish event bus, adding ability to set supervisor strategy ) is a better approach? -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
