Hi Steven, thanks for this write-up, your analysis is thorough and correct on all counts.
Remoting needs to use a simplistic approach to the coroner problem (i.e. when to declare another system “dead”—and zombies are not tolerated), which is mostly just a timeout that you should set high enough to avoid false positives given your expected outages (network and GC). Using a dedicated (minimal) ActorSystem for the remoting should be the optimal solution for your use-case, the overhead is a few hundred milliseconds for starting it up plus its default dispatcher (which then should run the remoting etc.), saving the remoting dispatcher on the heavy local ActorSystem behind it. The added benefit you note is that you can then reconfigure the remoting part of the application at runtime. One thing to watch out for is that you don’t accidentally share an ActorRef from the local system in or with a remote message—including sender()—because that can of course not work if the originating system does not have remoting enabled. The symptom would be dropped messages. Regards, Roland 23 maj 2014 kl. 20:27 skrev Steven Scott <ste...@codemettle.com>: > I've been slowly migrating an application to Akka since scala 2.10 came out > and pushed me away from scala actors; sorry for any stupid questions as I'm > always learning. > > As a general picture, the application consists of multiple long-lived JVMs > communicating over ActiveMQ. The standard deployment is to a single machine, > but with multiple services communicating over AMQ for the ability to move > specific pieces of functionality to other boxes. As the migration and > component rewrites have progressed, I'm solely left with actors communicating > with each other over AMQ using akka-camel. The natural next step was to > explore akka-remote. > > My questions started out as "is this an abuse/unintended usage of > akka-remote? Is akka-remote meant to be used outside of akka-cluster? Is it > useful for communicating to local JVMs? What about network hiccups for remote > JVMs?" > > I did as much reading as I could and found that Victor Klang has said it's > useful for transient networks: > http://stackoverflow.com/questions/6401500/is-akka-suitable-for-systems-with-transient-network-coverage, > and the smoking gun for same-machine inter-JVM communication being an > expected use-case was Dr. Kuhn's comment here: > http://stackoverflow.com/questions/10268613/whats-the-equivalent-of-akka/11787971#comment13246146_10268748 > > I went ahead and implemented a decent amount of code for using akka-remote to > talk to one of the services after bumping our akka version to 2.3.3, and have > to say I'm pleased, especially when comparing to ActiveMQ. Local machine > communication is flawless, but once I started testing with remote machines > and doing "ifdown eth0; sleep 20; ifup eth0" network disruption tests, I'm > left with questions about how to handle quarantines. I looked at > reference.conf and heeded the admonition to NOT change the quarantine timeout > from 5 days - restarting one of the actor systems is the only alternative. > > So - what're the best practices concerning restarting the ActorSystem? > > - I'm not clustering - these are a few long-lived "heavy" services, not just > nodes spinning up to do small processing tasks > - Our general deployment is not HA, we don't usually have standbys waiting > - Restarting the JVM isn't optimal > * since the services are fairly substantial and there's a non-trivial > amount of initialization including database hits to pre-fill caches, > restarting the JVM is a possibility (less time than the remoting gate time), > but isn't the first route I'd choose > * we (very rarely) run on non-linux platforms and so tend to try to keep > stuff in the JVM instead of relying on upstart/launchd/windows services/etc > > My only other thought is to run an additional ActorSystem for remoting. > > - allows programmatic configuration (our runtime configuration system could > change remoting settings and restart the remoting ActorSystem with the new > settings) > - a quarantine situation would just require the remoting ActorSystem to be > recreated, not a restart of the whole JVM > > However, one of the very earliest entries in the Akka documentation states > "An ActorSystem is a heavyweight structure that will allocate 1…N Threads, so > create one per logical application." I know creating multiple dispatchers in > the same ActorSystem is fine, and sometimes (at least historically) a > dedicated dispatcher was recommended for some remoting cases; I also know > starting a new ActorSystem takes some amount of time to create dispatchers, > parse configs, etc; so I'm thinking that the big yellow warning in the > documentation is a general guideline for getting started with Akka, not a > hard and fast rule. > > Sorry for the long post, can anybody give me some guidance on the situation? > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. Dr. Roland Kuhn Akka Tech Lead Typesafe – Reactive apps on the JVM. twitter: @rolandkuhn -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
