On Mon, Jun 9, 2014 at 2:58 PM, Brian <[email protected]> wrote:
> > It seems to me that service discovery could be well addressed by CRDTs, > maybe an OR-Set or MC-Set. In the spare time that I never seem to have, I > had thought about grabbing Patrik's CRDT lib and trying to implement > service discovery on it to learn more about CRDTs. Fingers crossed, one of > these days i'll get a chance to do it. > Sounds good. /Patrik > > I would agree it's a good fit for eventual consistency, if a false > positive (service available) is received and the connection fails, a client > would move on to the next. Quorum-based strong consistency ( ZK / Curator > ) feels like overkill, but I may be missing an important use case. > > Excellent thread, thanks Evan! Look forward to the discussion on this. > > > On Thursday, June 5, 2014 9:57:38 AM UTC-4, rkuhn wrote: > >> Hi Evan, >> >> thanks for bringing this up! We have been discussing similar things at >> Akka HQ, so far without the ability to dedicate resources to such an >> effort. I agree with Thomas that Actors already cover the basics, I would >> call the Actor Model the ideal substrate on which we can efficiently and >> conveniently grow uniservices(*). >> >> [specific replies inline] >> >> (*) I prefer this term because it focuses on the Single Responsibility >> Principle (see also the introduction of modules >> <https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf>) >> instead of the rather irrelevant notion of “size” . >> >> 2 jun 2014 kl. 17:51 skrev Evan Chan <[email protected]>: >> >> Hey guys, >> >> I would like to run an idea by the fine Akka community - which is to >> discuss what it would take to turn Akka into a platform for building a >> network of "microservices": each one independently redeployable and easy to >> change, yet through a common platform, take advantage of the distributed >> supervision, messaging and other goodies from Akka. >> >> Here are some characteristics of such a platform: >> >> >> - Service discovery >> - Supporting different kinds of data flow topologies - request >> response, as well as streaming data; pub-sub, etc. >> - Provide common abstractions for efficient data serialization >> - Support backpressure and flow control, to rate limit requests >> - Support easy scaling of each component, including routing of >> messages or requests to multiple instances >> - Enable easy testing of multiple services (for example, see Akka’s >> sbt-multi-jvm plugin) >> - A common platform for application metrics >> - Distributed message or request tracing, to help with visibility and >> debugging >> - Support polyglot development - it should be possible to develop >> services in different languages >> >> >> I think many of these are already provided by Akka, but I wanted to run >> through each one in more detail: >> >> *Service Discovery* >> Right now every actor talks to another actor location-transparently; >> however, when looking up an external ActorRef, one does have to know the >> mechanism, ie is it looking up in cluster, or remote, etc.... is it >> another actorsystem etc... (this could have changed in 2.2 and 2.3, but >> I'm not up to date :-p) What I'm looking for is a >> mechanism-independent way of looking up actors, remote or not. IE, I >> should just need to do this: >> >> val downstreamActorRef = System.lookupByName(service = >> "tradingSystem", actor = "masterTrader", ....) >> >> Under the hood this looks up the actorRef using one of configurable >> mechanisms: >> - Akka Cluster is certainly one way to go, with nodes >> - At work we use Zookeeper and Curator. It would be great to make this >> platform support multiple discovery types >> >> >> This is indeed the central missing piece, and it is largely orthogonal to >> the other concerns. My intuitive starting point for the discovery service >> is that it is basically a function from name to ActorRef that is accessible >> throughout the deployed application cluster; I would also start from >> eventually consistent semantics to see whether that fits the bill since >> everything else will require specially maintained central (SPOF/SPOB) >> facilities. >> >> >> *Data Flow Topology* >> - Akka is pretty good at this already, supporting many types of data >> flow. The only concern I see is that you have to define the flow via the >> like of routers and such, which are defined in the code on each node, >> rather than externally via say a message queue (see ZMQ, NSQ etc). This >> can be mitigated through DI and configuration and things like that, of >> course. >> >> >> I’d propose to build only direct peer-to-peer communication into the >> fabric and model all other concerns (queues, pub-sub, etc.) as uniservices >> on top. These implementations will then naturally be part of the whole >> package, but I would rather not conflate different notions of >> message-passing under a common umbrella. >> >> >> *Data Serialization* >> If we are using native Akka protocol to talk over the wire, this is >> already really good. One defines case classes, and Akka transparently >> serializes them over the network if the actor is remote. This is one thing >> about Akka that really appeals to me. >> >> So the question is can we make this work for Play / Akka HTTP >> transparently as well? >> >> *Related - Polyglot support* >> How would a Ruby/Python/etc process talk to an Akka network? My >> thoughts: >> - Easiest way would be to have a way to automagically generate HTTP >> endpoints that includes case class serialization to/from JSON. Type >> classes to handle special data types. >> - Did you guys define the Akka binary protocol and keep it stable? >> Client libraries could then be written for different langauges, but this >> doesn't solve the problem of message format -- Java serialization and >> Chill/Kryo won't work. >> >> >> My current thoughts on this front (also for changing the Akka remote >> protocol to allow interoperability between different versions) go into the >> direction of CBOR <http://tools.ietf.org/html/rfc7049> as a kind-of >> binary JSON-like format that is friendly to streaming, compact, >> self-describing and standardized. In the end it will require a structural >> data format (i.e. not tied to JVM nominal types) to accomplish polyglot >> deployments. The reason to prefer it over protobuf is that it can be parsed >> without knowing the schema. >> >> The raw serialization implementation can be used for any number of >> use-cases, e.g. remoting, streams, persistence, etc. >> >> >> *Backpressure and Flow Control* >> Reactive streams looks really promising here. How it ties into routing, >> topologies, etc. I'd like to find out more about. >> Also, reactive streams won't work for request/response protocols. >> >> >> I’d contest that: I think it a uniservice should not be restricted to A >> => B, it could also be Stream[A] => Stream[B] (well, these are not the real >> types). This would take care of the back pressure for data ingesting >> services in a better fashion than individual calls would, but we need a >> solution for those as well as you point out. One of the reasons for >> exclusively targeting direct peer-to-peer communications is that back >> pressure otherwise is basically unmaintainable. >> >> One related feature is that we should include common patterns like the >> CircuitBreaker from the get-go. >> >> >> *Application Metrics and Tracing* >> Microservices means it becomes more and more important to figure out >> what's going on across many services. Fortunately there's a lot of work >> in this area; multiple third party libs to provide automatic metrics; >> somebody wrote an Akka integration with Twitter's Dapper-like tracing >> system, and I have written a tracing/graphing system as well. >> >> >> Yes, monitoring components must be added as well, I see them as >> uniservices themselves just like the message routing facilities. >> >> >> *Hot Reloads* >> I didn't include this in the list above, because the assumption is that >> with lots of independent small services, they will be naturally easier to >> redeploy. Some will argue that the JVM is heavyweight (actually I think a >> lightly loaded JVM app is under 50MB, which is very reasonable), and will >> want Erlang-style hot reloads of individual actors. This is really tricky >> area though. >> >> >> I fully agree, redeployment should be done on a per-JVM granularity, >> where one JVM possibly hosts multiple (related) uniservices. >> >> One thing that is commonly left out—even when citing Erlang—is >> resilience: it is of the utmost important to include a sane model for >> handling service failures, restarting or redeploying them, switching to >> backup topologies, etc. We have that within Akka and porting this to the >> way we express uniservice architectures is one of the most appealing >> aspects of this whole endeavor. >> >> >> Anyways, I'd love to get the thoughts of the community about this idea of >> using Akka as a "microservice" platform. >> >> >> Much appreciated that you started this, and sorry for taking so long to >> respond, the last few days were a bit hectic on my end. >> >> Regards, >> >> >> *Dr. Roland Kuhn* >> *Akka Tech Lead* >> Typesafe <http://typesafe.com/> – Reactive apps on the JVM. >> twitter: @rolandkuhn >> <http://twitter.com/#!/rolandkuhn> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw <http://www.scaladays.org/> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
