Thanks Endre. My take from this is that the QuarantinedEvent will eventually be fired on both sides, so if I subscribe to it on one side only and recycle the actor system on that side I should be good. The only issue is that it might take a long time to get notified of the QuarantinedEvent, and for that I have to go study the timeout settings and adjust them accordingly.
Azad On Friday, August 29, 2014 3:04:59 AM UTC-7, Akka Team wrote: > > Hi Azad, > > The Quarantined will likely to happen on both sides (but it might take a > long time depending on timeout settings) but you will get Terminated > messages only in those actors that are watching a remote actor. > > So if you have node A and B, and actor A1 on A watches the actor B1 on B, > and the link between A and B goes away, then you will see eventually (might > take long) Quarantined on both A and B, and A1 will receive a Terminated > for B1. Since B1 did not watch anything on A1, it of course does not > receive any Terminated messages. > > -Endre > > > On Thu, Aug 28, 2014 at 11:36 PM, Azad Bolour <[email protected] > <javascript:>> wrote: > >> Thank you Endre and Steven for your responses. >> >> A follow-up question. Do we have to set up remote death watch or >> subscribe to the QuarantinedEvent in both peers? Or do we get the >> terminated message and the quarantined event on one side no matter which >> side has quarantined the other? It would be a little simpler if we could >> just do this on one side, and recycle the actor system only on that side to >> re-establish the link. >> >> Thanks again. >> >> Azad >> >> >> On Thursday, August 28, 2014 6:42:23 AM UTC-7, Steven Scott wrote: >>> >>> There's also the Remote Events section of http://doc.akka.io/docs/ >>> akka/2.3.4/scala/remoting.html; I subscribe to akka.remote.QuarantinedEvent >>> events on the remoting ActorSystem. >>> >>> On Thursday, August 28, 2014 5:50:36 AM UTC-4, Akka Team wrote: >>>> >>>> Hi Azad, >>>> >>>> >>>> On Wed, Aug 27, 2014 at 6:18 PM, Azad Bolour <[email protected]> >>>> wrote: >>>> >>>>> >>>>> I am wondering how one peer in Akka remoting can detect that the other >>>>> peer has been quaranteed, so that it can restart the actor system used >>>>> for >>>>> remoting to clear the quarantine? Is there an API call for finding out >>>>> that >>>>> Akka has quarantined a peer? Or does the application need to use say >>>>> hearbeats to detect that the peer is unreachable and deduce that it has >>>>> been quarantined? >>>>> >>>> >>>> The easiest way is just to use DeathWatch and watch one of the actors >>>> on the remote machine. If the other host goes away and gets quarantined >>>> you >>>> will get a Terminated event. If this actor never stops otherwise (i.e. it >>>> only stops when the actor system goes away) then it basically does what >>>> you >>>> need. If you use clustering though you can just listen to cluster >>>> membership events, since that handles all these things for you. >>>> >>>> -Endre >>>> >>>> >>>>> >>>>> Many thanks. >>>>> >>>>> Azad >>>>> >>>>> >>>>> On Monday, June 2, 2014 10:16:17 PM UTC-7, rkuhn wrote: >>>>> >>>>>> Hi Steven, >>>>>> >>>>>> thanks for this write-up, your analysis is thorough and correct on >>>>>> all counts. >>>>>> >>>>>> Remoting needs to use a simplistic approach to the coroner problem >>>>>> (i.e. when to declare another system “dead”—and zombies are not >>>>>> tolerated), >>>>>> which is mostly just a timeout that you should set high enough to avoid >>>>>> false positives given your expected outages (network and GC). Using a >>>>>> dedicated (minimal) ActorSystem for the remoting should be the optimal >>>>>> solution for your use-case, the overhead is a few hundred milliseconds >>>>>> for >>>>>> starting it up plus its default dispatcher (which then should run the >>>>>> remoting etc.), saving the remoting dispatcher on the heavy local >>>>>> ActorSystem behind it. The added benefit you note is that you can then >>>>>> reconfigure the remoting part of the application at runtime. >>>>>> >>>>>> One thing to watch out for is that you don’t accidentally share an >>>>>> ActorRef from the local system in or with a remote message—including >>>>>> sender()—because that can of course not work if the originating system >>>>>> does >>>>>> not have remoting enabled. The symptom would be dropped messages. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Roland >>>>>> >>>>>> 23 maj 2014 kl. 20:27 skrev Steven Scott <[email protected]>: >>>>>> >>>>>> I've been slowly migrating an application to Akka since scala 2.10 >>>>>> came out and pushed me away from scala actors; sorry for any stupid >>>>>> questions as I'm always learning. >>>>>> >>>>>> As a general picture, the application consists of multiple long-lived >>>>>> JVMs communicating over ActiveMQ. The standard deployment is to a single >>>>>> machine, but with multiple services communicating over AMQ for the >>>>>> ability >>>>>> to move specific pieces of functionality to other boxes. As the >>>>>> migration >>>>>> and component rewrites have progressed, I'm solely left with actors >>>>>> communicating with each other over AMQ using akka-camel. The natural >>>>>> next >>>>>> step was to explore akka-remote. >>>>>> >>>>>> My questions started out as "is this an abuse/unintended usage of >>>>>> akka-remote? Is akka-remote meant to be used outside of akka-cluster? Is >>>>>> it >>>>>> useful for communicating to local JVMs? What about network hiccups for >>>>>> remote JVMs?" >>>>>> >>>>>> I did as much reading as I could and found that Victor Klang has said >>>>>> it's useful for transient networks: http://stackoverflow >>>>>> .com/questions/6401500/is-akka-suitable-for-systems-with-transient- >>>>>> network-coverage, and the smoking gun for same-machine inter-JVM >>>>>> communication being an expected use-case was Dr. Kuhn's comment here: >>>>>> http://stackoverflow.com/questions/10268613/whats-the- >>>>>> equivalent-of-akka/11787971#comment13246146_10268748 >>>>>> >>>>>> I went ahead and implemented a decent amount of code for using >>>>>> akka-remote to talk to one of the services after bumping our akka >>>>>> version >>>>>> to 2.3.3, and have to say I'm pleased, especially when comparing to >>>>>> ActiveMQ. Local machine communication is flawless, but once I started >>>>>> testing with remote machines and doing "ifdown eth0; sleep 20; ifup >>>>>> eth0" >>>>>> network disruption tests, I'm left with questions about how to handle >>>>>> quarantines. I looked at reference.conf and heeded the admonition to NOT >>>>>> change the quarantine timeout from 5 days - restarting one of the actor >>>>>> systems is the only alternative. >>>>>> >>>>>> So - what're the best practices concerning restarting the >>>>>> ActorSystem? >>>>>> >>>>>> - I'm not clustering - these are a few long-lived "heavy" services, >>>>>> not just nodes spinning up to do small processing tasks >>>>>> - Our general deployment is not HA, we don't usually have standbys >>>>>> waiting >>>>>> - Restarting the JVM isn't optimal >>>>>> * since the services are fairly substantial and there's a >>>>>> non-trivial amount of initialization including database hits to pre-fill >>>>>> caches, restarting the JVM is a possibility (less time than the remoting >>>>>> gate time), but isn't the first route I'd choose >>>>>> * we (very rarely) run on non-linux platforms and so tend to try >>>>>> to keep stuff in the JVM instead of relying on upstart/launchd/windows >>>>>> services/etc >>>>>> >>>>>> My only other thought is to run an additional ActorSystem for >>>>>> remoting. >>>>>> >>>>>> - allows programmatic configuration (our runtime configuration >>>>>> system could change remoting settings and restart the remoting >>>>>> ActorSystem >>>>>> with the new settings) >>>>>> - a quarantine situation would just require the remoting ActorSystem >>>>>> to be recreated, not a restart of the whole JVM >>>>>> >>>>>> However, one of the very earliest entries in the Akka documentation >>>>>> states "An ActorSystem is a heavyweight structure that will allocate >>>>>> 1…N Threads, so create one per logical application." I know creating >>>>>> multiple dispatchers in the same ActorSystem is fine, and sometimes (at >>>>>> least historically) a dedicated dispatcher was recommended for some >>>>>> remoting cases; I also know starting a new ActorSystem takes some amount >>>>>> of >>>>>> time to create dispatchers, parse configs, etc; so I'm thinking that the >>>>>> big yellow warning in the documentation is a general guideline for >>>>>> getting >>>>>> started with Akka, not a hard and fast rule. >>>>>> >>>>>> Sorry for the long post, can anybody give me some guidance on the >>>>>> situation? >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>> urrent/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>> p/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> >>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Dr. Roland Kuhn* >>>>>> *Akka Tech Lead* >>>>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >>>>>> twitter: @rolandkuhn >>>>>> <http://twitter.com/#!/rolandkuhn> >>>>>> >>>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>>>> current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>>> group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Akka Team >>>> Typesafe - The software stack for applications that scale >>>> Blog: letitcrash.com >>>> Twitter: @akkateam >>>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Akka Team > Typesafe - The software stack for applications that scale > Blog: letitcrash.com > Twitter: @akkateam > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
