Hi Yaron, (Also apologies for the long reply)
>> First, I think that we need to clearly distinguish that service discovery, and request routing / load balancing are related but separate concerns. In my mind, at least, discovery is about finding a way to communicate with a service (much like a DNS query), whereas routing and load balancing concerns effectively and efficiently directing (handling) network traffic. It's very refreshing to hear you talk about service discovery this way. I think this is a very important point that often gets lost in discussions, and implementations don't always truly take this to heart so the result doesn't end up as system agnostic as intended. We've spend the last ~year deploying our own service discovery system because we felt nothing in the community really fitted *truly* into the sentence you described above... The result for us was something very similar to what you've come up with, a DNS (DNS-SD rfc6763) system that runs multi-datacenter backed by an distributed etcd database for the names and records. We built our own DNS server to do this as consul/weave didn't exist back then -> https://github.com/duedil-ltd/discodns. We're firm believers that DNS itself can provide us with the framework to achieve 99% of all use cases, assuming we ultimately build in support for things like dynamic updates via DNS (rfc2136) and possibly even the "push" based long-polling features that Apple use in Bonjour/mDNSResponder. Not to mention that using say, Marathon, for service discovery is increadiblly restricting. We'd like to use the same system for every service we run, ranging from things deployed in one cloud environment on Mesos to others in another environment deployed using Chef. There's no reason why this can't be the case, imo. The key idea very much the same, we want different systems to be able to register services in the way they need to (either via http to etcd, or via dns update) and using their own semantics. A good example is that some systems might want to have TTLs on records (in etcd, so the record automatically disappears) to remove unhealthy instances of services, however other systems might not want to relate their existence in service discovery with their health (think long running distributed databases). Currently we have some command line tools and chef cookbooks for service registering and a WIP branch for *dnsupdate* ( https://github.com/duedil-ltd/discodns/pull/31). (I'd be very very interested to hear more about your experience with Weave for this purpose, perhaps a blog post? :-)) > Regarding DNS: again I don’t think it makes sense to have a ‘mesos-dns’ and ‘weave-dns’ and ‘kubernets-dns’ - it makes much more sense to have a single DNS that easily integrates with multiple data sources. There's actually a ticket on mesos-dns to support plugins https://github.com/mesosphere/mesos-dns/issues/62 and I had an idea to write a discondns plugin that'd write the records into our etcd database, which might be an interesting way to achieve integration with these tools. Though I wonder whether this approach results is scalability problems because the state becomes too large for a single system to re-process on a regular basis, maybe it's best for the things running on mesos to register themselves, or even a mesos module for the slaves. > https://registry.hub.docker.com/u/yaronr/discovery We'd played around with the LUA support in NGINX to create some sort of dns-sd based service discovery proxy, though don't have anything worth sharing yet as far as I know! Thanks for sharing! On 12 April 2015 at 10:03, Yaron Rosenbaum <yaron.rosenb...@gmail.com> wrote: > Ok, this is a bit long, I apologize in advance > > I’ve been researching and experimenting with various challenges around > managing microservices at-scale. I’ve been working extensively with Docker, > CoreOS and recently Mesos. > > First, I think that we need to clearly distinguish that service discovery, > and request routing / load balancing are related but separate concerns. In > my mind, at least, discovery is about finding a way to communicate with a > service (much like a DNS query), whereas routing and load balancing > concerns effectively and efficiently directing (handling) network traffic. > > There are multiple solutions and approaches out there, but I don’t know if > any single technology could address all ‘microservices at-scale’ needs on > its own effectively and efficiently. In other words - mixing multiple > approaches, tools and technologies is probably the right way to go. > I’m saying this because many of the existing tools come with a single > technology in mind. Tools that come form the Mesos camp obviously have > Mesos in mind, tools that come from Kube have Kube in mind, tools coming > from CoreOS have CoreOS in mind, etc. > > I think It’s time to start mixing things together to really benefit from > all the goodness in all the various camps. > > I’ll give an example: > First, with respect to network traffic routing / balancing, I’ve created > an HAProxy based microservice, which supports multiple data sources as > inputs for auto-configuration. (BTW, I’m also experimenting with Velocity > scripts for the configuration template, and i’m finding it’s a big > improvement over Go scripting. Appreciate your feedback!) > https://registry.hub.docker.com/u/yaronr/discovery > For now just etcd and Marathon data sources are implemented. Some services > will rely of CoreOS/etcd for publishing their availability, and others on > Marathon. In my case, I run my Kernel cluster - Mesos-master, Marathon and > a a clustered REST DHCP like service on CoreOS, and allow my customers to > schedule their own microservices on Mesos/Marathon. All require discovery / > routing / load balancing, the only difference is where the data comes from. > > Regarding DNS: again I don’t think it makes sense to have a ‘mesos-dns’ > and ‘weave-dns’ and ‘kubernets-dns’ - it makes much more sense to have a > single DNS that easily integrates with multiple data sources. Network is a > common, low level resource, that’s used by the upper level abstractions / > frameworks such as Kube, Mesos, and possibly Fleet. Therefore, if a single > technology ’takes ownership’ of (parts of) this resource, that could create > a technology lock down the line. > I personally use Weave to abstract Netowrk altogether, and Weave DNS to > provide service discovery (not network routing). > > (Y) > >
