Nelz,
The drawback of using dynamic DNS is that you may end up using a
cached name, so when you remove nodes or add nodes you end up in a
period of time when clients are attempting/contacting servers in the
prior configuration when there is a change. This can be particularly
problematic in the case when a node has crashed, and clients are still
trying to connect to it. You can mitigate this by using a zero value
TTL on the given record, provided you control all points in the name
resolution path so that you don't end up with unexpected caching.
NorthScale has a configuration solution based on libconflate (http://labs.northscale.com/libconflate/
) that you could use to update configuration files on numerous hosts
at the same time. You could use a local file based configuration that
your spymemcached clients consult when connecting. I'd prefer that
sort of an approach over a DNS based solution.
The IP address of an EC2 instance does not change after the system
boots, so simply have it trigger a libconflate action upon boot-up,
and then use a monitoring approach to pull out unresponsive nodes when
they fail to respond to test probes.
Another option would be to use moxi (http://labs.northscale.com/moxi/)
to front-end your cluster so that your application always uses the
same string, but you add and remove nodes from the moxi instance. The
same file based configuration approach would be needed to manage a
cluster of moxi instances. Depending on your server topology you may
or may not need more than one.
Adrian
On Feb 19, 2010, at 2:55 PM, Nelz wrote:
Hey all,
We are using the spymemcached client with consistent hashing. In my
understanding, because we are using consistent hashing we need to
utilize the same exact strings on all clients for identifying the
physical nodes in a logical memcached cluster.
Does anyone have a good solution for handling the dynamic internal IPs
on EC2 instances?
We're going to try dynamic DNS and use hostnames, but wondered if
anyone has tried other solutions in large-scale implementations.
Let me know if my question is as clear as mud and could use some more
background.
- Nelz
