I've been working on this the past few weeks. Here's what I'm currently
doing.
Background:
* CoreOS as the host OS
* Using etcd/fleet for service discovery and scheduling
* Using SkyDNS in order to address nodes by their logical name, also allows
nodes to bind to that name rather than the IP of the CoreOS host or Docker
container
Every CoreOS host has a Docker container running SkyDNS. SkyDNS acts as an
authoritative DNS server for "conspire.local" (company name is Conspire,
use whatever you like) and it then falls back to 8.8.8.8 for other DNS
queries. SkyDNS watches specific paths in etcd in order to resolve those
queries to our services. For instance, if I place "{'host':10.0.1.5,
'port':2552}" at the path
/skydns/local/conspire/us-east-1/dev/backend/supervisor in etcd, a DNS
query for supervisor.backend.dev.us-east-1.conspire.local will resolve to
10.0.1.5, where 10.0.1.5 is the EC2 private IP of the CoreOS node hosting
the Docker container running an Akka node named "supervisor".
The Docker containers running our Akka nodes link to the SkyDNS container
(--link
skydns:skydns) and use it as their DNS server (SKYDNS_IP=$(docker inspect
skydns | grep IPAddress | cut -f4 -d'"'); --dns $SKYDNS_IP). The hostname
of the Docker container is the hostname we publish to etcd (-h
supervisor.backend.dev.us-east-1.conspire.local).
Every container running an Akka node has a sidekick service (scheduled by
fleet) which publishes the appropriate network location into etcd for
skydns. It's just a bash script in a while loop doing etcdctl set
/path/to/hostname "{'host':$COREOS_PRIVATE_IPV4}". We set a TTL on the etcd
entry so that if the Akka node dies, the sidekick will die with it and the
entry will expire.
Seed nodes are listed in another location in etcd and then resolved against
their SkyDNS entry. That's handled by a second sidekick. The DNS sidekick
runs *before* the Akka node is started, so that Akka can bind to its
logical hostname rather than container or host IP, the seed sidekick
advertises it as an available seed *after* Akka has started. If the two
aren't separated there's a nasty race condition.
This lets us easily (modulo etcd difficulties) run Akka in a multi-host
Docker environment with service discovery handled by etcd. Node locations
are advertised in etcd so no configuration is needed in advance. CoreOS
lets us easily (modulo etcd difficulties) add and remove EC2 nodes and
fleet decides where things are deployed. We're hoping to expand this
deployment strategy beyond our Akka cluster. We have plans to split some
services out of our cluster and also deploy Kafka. Ideally this will be our
unified deployment strategy for all these services.
We also use etcd for all of our application config parameters (DB hosts,
etc). I wrote a small Ruby script that plucks those out of etcd and outputs
a config.json which we pass in with -Dconfig.file=...
The caveat is that I spent all of yesterday battling etcd failures, hoping
all this work wasn't for nothing. ;)
I'm working on a blog post to detail all of this with code samples, I'll
try to post it next week.
On Wednesday, October 1, 2014 12:14:13 PM UTC-6, Alessandro Vermeulen wrote:
>
> Dear all,
>
> When running our Akka cluster in docker we need to bind netty to 0.0.0.0
> to make it accessible to the outside of the container and binding to the
> external ip of the docker host is not possible. However, the seed node uri
> needs to point to the external ip in order to connect to the seed nodes.
> Consequently, the seed node drops the messages from joining nodes because
> it thinks they are not meant for him.
>
> Is there an existing solution to this issue? One of the solutions would be
> to use host names and change the ip addresses through the hosts file. This
> would be very cumbersome for us to maintain.
>
> Would it be sensible to include a different variable that is used to
> configure the identifier of the node such that it is decoupled from the
> hostname used to bind the network interface?
>
> Cheers,
> Alessandro
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
