We have been using the following approach with success: Each cluster-member writes/updates an alive-record to the database each X-seconds. It also checks how many other alive nodes are found in the database.. (I know this solution does not scale massively) If number of alive nodes in database is different than the one it knows is in the cluster, we have a cluster-fragmentation and the node restarts itself to heal.
We also use this alive-nodes-info to generate correct seed-node-lists when a cluster is starting up. If no other nodes are present in database, we know we are the first one starting up - and should therefor be the first seednode in the list. If other are alive, we use them as the first seed-nodes. We also have a timeout when starting up, to recover from bad starts/restarts: If we started up NOT being the first seed node, and was unable to connect to cluster (due to the other nodes not being there - maybe also restarting?), we restart to heal. The above has, for us, turned out to be a robust solution for stopping/starting/restarting some or all of our nodes in our cluster. It has also auto-recovered issues after various network-related problems in our datacenter. I hope this information was helpful. Best regards, Morten Kjetland On Fri, Jun 19, 2015 at 12:13 AM tigerfoot <[email protected]> wrote: > Hello, > > Are there any methods or tools that help detect cluster fragmentation? > > Once fragmentation is detected what's the best practice to correct it? > > Thanks, > Greg > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
