On Fri, Feb 6, 2015 at 4:50 AM, Bourne Toulouss <[email protected]> wrote:
> please disregard my first question. I have that sorted out. Right now i'm > only concerned about the second case i mentioned where Terminated are not > received on both ends. > Thank you. > Best Regards, > Bourne. > > > On Thursday, February 5, 2015 at 7:01:02 PM UTC+7, Bourne Toulouss wrote: >> >> Hi all, >> >> I'm having a cluster system with many Master and many Slaves. I let >> Masters watch over Slaves and Slaves watch over their Master. So that when >> a Master is "terminated", corresponding slaves will re-register to a new >> Master and deny all jobs until a master is found . Or when a slave is >> "terminated", Master will get the jobs that were sent to that Slave and >> send it to other available slaves. >> >> >> Currently i'm testing the high-availability of the cluster. When i turn >> off the JVM of Workers or Masters, Terminated messages are generated >> immediately, but if i try to turn off the network adapter and turn it on >> right about 10 or 20 seconds later to simulate network lagging, Terminated >> messages are not thrown immediately,usually a moment before MemberDown is >> thrown. And when i turn the adapter back on, things went back to normal, >> but the data that was sent when network was down is lost. >> > Yes, you must design for that messages are delivered with at-most-once semantics <http://doc.akka.io/docs/akka/2.3.9/general/message-delivery-reliability.html>, i.e. can be lost. > i tried to catch DeadLetter and send them back to Master, >> > That will not always work. Implement at-least-once delivery with acknoledgments and resends instead. Perhaps use AtLeastOnceDelivery <http://doc.akka.io/docs/akka/2.3.9/java/persistence.html#At-Least-Once_Delivery> utility that is provided by Akka Persistence. > but there is a case where Terminated was receive by Slave but Master never >> caught any. So my HA flow was ruined.( this is because as i said above, >> Slave will deny all messages until a Master is found, to prevent the slave >> from trying to send the message back to a dead actor forever, and Master >> never detected any Terminated messages, so the jobs that were deny by Slave >> were never recovered) >> >> So there are two things i'm concerned about: >> -is there a config so that when an unreachable message is thrown, the >> actor that is watch will be considered Terminated? >> > Two things that can be configured. 1) The failure detector <http://doc.akka.io/docs/akka/2.3.9/java/cluster-usage.html#Failure_Detector> settings, akka.cluster.acceptable-heartbeat-pause. That defines how quickly a broken connection or crashed jvm will be deemed as unreachable. This will not trigger Terminated. A node can become reachable again if the network partition heals. 2) Downing of unreachable. A node is removed from the cluster when it has been downed. Terminated is triggered when the node has been removed. Selection of downing strategy <http://doc.akka.io/docs/akka/2.3.9/java/cluster-usage.html#Automatic_vs__Manual_Downing> might require some thought, but if you use auto downing the config property is akka.cluster.auto-down-unreachable-after > -And the case where Master watch over slave and slave watches over Master >> but only 1 of them received Terminated message for the other? >> > I don't understand how that can happen if you use Akka Cluster. When a member is removed that address is quarantined and no communication is allowed to that address. That should trigger things on the other side. By the way, do you use Akka 2.3.9? Regards, Patrik > >> Thank you very much for your help, >> Best Regards >> Bourne. >> >> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
