On 16/04/2016 1:52 am, Ben Tisdall <ben.tisd...@photobox.com> wrote: > > Hi, > > are there are plans to support DNS resolution based on TTL a la NGINX? This > would be helpful for use cases where the upstream is an ELB or similar > system. I've pasted a reply from AWS support based on some observations from > a couple of our services that use HAProxy 1.6 in front of ELBs. Note that I > am not contending that the issue of uneven distribution of upstream IPs is > HAProxy's fault (that is a consequence of our design), but the cycling of ELB > nodes when retirement occurs is something that NGINX would seem to handle in > a more satisfactory way. > > " > I think an explanation of what happens when ELB scales will be helpful as > background at this point. ELB employs what we term "Graceful Scaling". When a > scaling trigger is breached, let's say for sake of argument this is a > scale-up event, then ELBs controller immediately begins the process of > provisioning new more performant ELB nodes. This usually takes a few minutes, > and once these new nodes pass the controller health-checks, we remove the old > node IPs from the DNS record set, and add in the new ELB node IPs to the DNS > set. Since the TTL with this DNS record is published in 60 seconds, after > about 2 - 3 minutes, most traffic will migrate over to the new node. We do > however, do not de-provision the old ELB nodes, but instead we begin to > monitor them to determine when traffic received by these nodes drops to below > a certain threshold level, or a maximum age has expired (this is several > days). This happens to cater for the case where some clients are caching DNS > longer than the TTL value. > > Given the way that HA proxy works, when it starts up, it resolves the ELB > name, and obtains the current IPs. HealthChecks are a requirement for the > resolver clause, so HA also begins to perform the configured health-check on > the nodes it learned about at startup. > > If the ELB were to scale now, the new nodes would come online but HA proxy > would never learn of them, as the old nodes will continue to pass > health-checks. If traffic continues to increase, at some point the older ELB > nodes will become overwhelmed and will fail a health-check on HA proxy, at > which point that HA proxy node on which the health-check failed, will learn > of the new ELB nodes from DNS, and start to send traffic to the new one. > > Should traffic not increase sufficiently to cause the old nodes to fail a > health-check, then only new HA proxy instances in your fleet will learn of > the new nodes. Eventually the maximum graceful node lifetime will be reached, > and we will terminate the old nodes, at this point all your HA proxy > instances will fail health-checks on their upstream and learn of the new > nodes at the same time. > > This process happens in such a fashion that over a time, its conceivable that > each of your HA proxy nodes may know of different back-end IPs. As a result, > traffic on the inside ELB nodes will not be symmetrically distributed by the > HA proxy nodes over time. This is somewhat mitigated on the back-end by the > use of cross-zone load-balancing, so the asymmetry is not propagated to the > back-ends. We do monitor each ELB node individually, thus the ELB will scale > on the monitoring of a single node, rather than the entire ELB, which further > mitigates the effects of any asymmetry on your ELB nodes. > > There is no easy way to make HA proxy work perfectly in front of an ELB, due > to the nature of how HA proxy have implemented DNS resolution. > > We often recommend to customers using a reverse proxy in front of ELB, to > rather use Nginx, as this does have the ability to follow DNS TTLs of its > upstreams perfectly. In this case, given the way you have implemented it > means that HA will learn of failed ELB nodes, and eventually learn of > scaling, and the ELB mitigates the imbalance to your back-ends through cross > zone. So, in summary, as of now the only possible way to overcome this > behavior would be to consider using a different reverse proxy solution > between the two ELB tiers instead of HA proxy. I apologize for any > inconvenience. I hope the above information was helpful. Please let us know > if you have any other questions or concerns and we will be happy to assist > you. > " > > Regards, > > -- > Ben > > This message and any attachment are confidential and may be privileged or > otherwise protected from discl
Afaik haproxy already has runtime DNS resolution in 1.6 see http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/ although it is not going to dynamically update its configuration for you.
