At the moment, all nodes register with etcd. Skydns uses those entries in order to make them addressable by logical hostname. We're using CoreOS so the actual hostname of the box is just ip-10-0-1-5 or whatever, skydns lets us address them as analytics-1.dev.us-east-1.conspire.local. IIRC Akka requires nodes to bind to hostname their want to be addressed at, so we need DNS to map those hostnames correctly (unless I misunderstood Akka's networking).
Right now, the node just publishes its IP and port in etcd and then eventually the skydns instances on every CoreOS box pick up that change. We aren't doing anything smart to handle downed nodes. There are TTLs on the etcd entries of 60 seconds and the entries are refreshed every 45 while the node is up. Fleet ensures that we don't accidentally deploy two nodes of the same name. If another node picks up a downed node as a potential seed, there might be an issue with that but the node will kill itself if it can't find the cluster. I think I'll probably wind up listing seeds elsewhere and controlling that with Akka cluster events while letting nodes still set their own etcd entries for DNS purposes. There will always be a slight lag in those seed values becoming consistent across the cluster—and I fully recognize that running a cluster on top of a cluster might be slightly nuts—but for the moment I'm content to tackle these issues as they become problems. Our etcd integration is no way Akka specific right now, we just run a sidekick service using fleet that has a while loop in bash refreshing that etcd entry. Seed nodes are fetched in the start script which passes them into Akka using -D flags. On Monday, September 29, 2014 11:11:27 AM UTC-6, Evan Chan wrote: > > Ryan, > > That sounds really interesting. How are you doing leader election, if I > may ask? > > Do all nodes register with etcd, or just the seed nodes? Is there some > duplication of state and conflicts, ie Akka detects a node is down but the > downed node still registers itself with etcd? > > On Mon, Sep 29, 2014 at 8:34 AM, Ryan Tanner <[email protected] > <javascript:>> wrote: > >> We're in the process of moving our Akka cluster over to CoreOS using etcd >> + skydns for service discovery. Our dev environment has been running on >> that for about two weeks now. Akka nodes publish their network location >> into etcd and then get their seed nodes out of the same, with skydns >> mapping logical node names (analytics-1.dev.blah) to their private EC2 IP. >> The change comes as part of moving our deployment towards Docker and making >> it easier to deploy more services along side our cluster (Kafka is up >> next), all using etcd for service discovery. >> >> For now it's KISS, just using TTLs on the etcd entries to remove downed >> nodes. Eventually we might tie that into cluster member events, letting >> the leader manage the etcd entries (ClusterSingleton probably), we'll see >> how it goes. >> >> On Monday, September 29, 2014 2:15:02 AM UTC-6, Evan Chan wrote: >>> >>> The discussion was very long but didn't produce any conclusions. >>> >>> One possible piece to start with is to make the Akka discovery of other >>> nodes pluggable (whereas the great threshold detection and up/down logic >>> would remain); for example, maybe the other nodes can be discovered via >>> Consul or ZK instead of purely peer to peer? This is likely to play >>> better with non-Akka services. >>> >>> On Fri, Sep 26, 2014 at 6:48 AM, Matlik <[email protected]> wrote: >>> >>>> Has there been any movement forward on this discussion? I like the >>>> idea of having a single layer of cluster coordination (discovery and >>>> availability) for all Akka and non-Akka services. If there is a clear >>>> technical direction and code base, I'd be interested in possibly >>>> contributing. Could/should this be a contrib project, or are the concerns >>>> more appropriately addressed in core functionality? >>>> >>>> Thanks, >>>> James >>>> >>>> >>>> On Monday, June 16, 2014 6:06:26 AM UTC-4, [email protected] wrote: >>>>> >>>>> >>>>> >>>>> On Monday, June 2, 2014 8:51:06 AM UTC-7, Evan Chan wrote: >>>>>> >>>>>> Hey guys, >>>>>> >>>>>> I would like to run an idea by the fine Akka community - which is to >>>>>> discuss what it would take to turn Akka into a platform for building a >>>>>> network of "microservices": each one independently redeployable and easy >>>>>> to >>>>>> change, yet through a common platform, take advantage of the distributed >>>>>> supervision, messaging and other goodies from Akka. >>>>>> >>>>>> >>>>>> *Related - Polyglot support* >>>>>> How would a Ruby/Python/etc process talk to an Akka network? My >>>>>> thoughts: >>>>>> - Easiest way would be to have a way to automagically generate HTTP >>>>>> endpoints that includes case class serialization to/from JSON. Type >>>>>> classes to handle special data types. >>>>>> - Did you guys define the Akka binary protocol and keep it stable? >>>>>> Client libraries could then be written for different langauges, but >>>>>> this >>>>>> doesn't solve the problem of message format -- Java serialization and >>>>>> Chill/Kryo won't work. >>>>>> >>>>>> >>>>> Thought I'd share how I'm using Jruby with Akka. My opinion is that >>>>> the best solution is an idiomatic wrapper around Akka for whatever >>>>> language >>>>> you are using. >>>>> >>>>> I created a ruby class that inherits from UntypedActor, and all of my >>>>> ruby actors inherit from that. That let me do a few things that were >>>>> more >>>>> idiomatic ruby. >>>>> >>>>> Say I have a ruby actor class called MyActor. To get an actor ref to >>>>> it, I can do this: >>>>> >>>>> actor_ref = MyActor.find >>>>> >>>>> Or all in one go... >>>>> >>>>> MyActor.find.tell(message) or MyActor.find.ask(message,100) >>>>> >>>>> find takes one argument, the default is the class name. So if you >>>>> created the actor with a different name you do this: >>>>> >>>>> MyActor.find("my_other_actor") (or just Actor::Base.find("my_other_ >>>>> actor") >>>>> >>>>> I can also do MyActor.find_remote(server,name) >>>>> >>>>> The find method actually returns an instance of my own actor ref >>>>> class. This was primarily to provide a simple interface to ask/tell, and >>>>> not have to deal with futures or ActorSelection/AskableActorSelection >>>>> directly. >>>>> >>>>> I also created a simple builder and factory that lets me create actors >>>>> like so: >>>>> >>>>> Actor::Builder.new(MyActor,arg1,arg2,arg3).with_name("myname >>>>> ").with_router(Javalib::RoundRobinRouter,10).start >>>>> >>>>> It also supports with_parent and with_dispatcher. >>>>> >>>>> The args are called on the actor's post_init method by the factory. >>>>> >>>>> I use protocol buffers for all messages. I use the protostuf library >>>>> as I just like it's api better and it's much more idiomatic to use from >>>>> ruby. >>>>> >>>>> Chris >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>>> current/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>> group/akka-user >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Akka User List" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/akka-user/QCYVY-dJlM8/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> The fruit of silence is prayer; >>> the fruit of prayer is faith; >>> the fruit of faith is love; >>> the fruit of love is service; >>> the fruit of service is peace. -- Mother Teresa >>> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Akka User List" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/akka-user/QCYVY-dJlM8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > The fruit of silence is prayer; > the fruit of prayer is faith; > the fruit of faith is love; > the fruit of love is service; > the fruit of service is peace. -- Mother Teresa > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
