On Fri, Apr 15, 2016 at 5:52 PM, Ben Tisdall <[email protected]> wrote: > Hi, > > are there are plans to support DNS resolution based on TTL a la NGINX? This > would be helpful for use cases where the upstream is an ELB or similar > system. I've pasted a reply from AWS support based on some observations from > a couple of our services that use HAProxy 1.6 in front of ELBs. Note that I > am not contending that the issue of uneven distribution of upstream IPs is > HAProxy's fault (that is a consequence of our design), but the cycling of > ELB nodes when retirement occurs is something that NGINX would seem to > handle in a more satisfactory way. > > " > I think an explanation of what happens when ELB scales will be helpful as > background at this point. ELB employs what we term "Graceful Scaling". When > a scaling trigger is breached, let's say for sake of argument this is a > scale-up event, then ELBs controller immediately begins the process of > provisioning new more performant ELB nodes. This usually takes a few > minutes, and once these new nodes pass the controller health-checks, we > remove the old node IPs from the DNS record set, and add in the new ELB node > IPs to the DNS set. Since the TTL with this DNS record is published in 60 > seconds, after about 2 - 3 minutes, most traffic will migrate over to the > new node. We do however, do not de-provision the old ELB nodes, but instead > we begin to monitor them to determine when traffic received by these nodes > drops to below a certain threshold level, or a maximum age has expired (this > is several days). This happens to cater for the case where some clients are > caching DNS longer than the TTL value. > > Given the way that HA proxy works, when it starts up, it resolves the ELB > name, and obtains the current IPs. HealthChecks are a requirement for the > resolver clause, so HA also begins to perform the configured health-check on > the nodes it learned about at startup. > > If the ELB were to scale now, the new nodes would come online but HA proxy > would never learn of them, as the old nodes will continue to pass > health-checks. If traffic continues to increase, at some point the older ELB > nodes will become overwhelmed and will fail a health-check on HA proxy, at > which point that HA proxy node on which the health-check failed, will learn > of the new ELB nodes from DNS, and start to send traffic to the new one. > > Should traffic not increase sufficiently to cause the old nodes to fail a > health-check, then only new HA proxy instances in your fleet will learn of > the new nodes. Eventually the maximum graceful node lifetime will be > reached, and we will terminate the old nodes, at this point all your HA > proxy instances will fail health-checks on their upstream and learn of the > new nodes at the same time. > > This process happens in such a fashion that over a time, its conceivable > that each of your HA proxy nodes may know of different back-end IPs. As a > result, traffic on the inside ELB nodes will not be symmetrically > distributed by the HA proxy nodes over time. This is somewhat mitigated on > the back-end by the use of cross-zone load-balancing, so the asymmetry is > not propagated to the back-ends. We do monitor each ELB node individually, > thus the ELB will scale on the monitoring of a single node, rather than the > entire ELB, which further mitigates the effects of any asymmetry on your ELB > nodes. > > There is no easy way to make HA proxy work perfectly in front of an ELB, due > to the nature of how HA proxy have implemented DNS resolution. > > We often recommend to customers using a reverse proxy in front of ELB, to > rather use Nginx, as this does have the ability to follow DNS TTLs of its > upstreams perfectly. In this case, given the way you have implemented it > means that HA will learn of failed ELB nodes, and eventually learn of > scaling, and the ELB mitigates the imbalance to your back-ends through cross > zone. So, in summary, as of now the only possible way to overcome this > behavior would be to consider using a different reverse proxy solution > between the two ELB tiers instead of HA proxy. I apologize for any > inconvenience. I hope the above information was helpful. Please let us know > if you have any other questions or concerns and we will be happy to assist > you. > " > > Regards, > > -- > Ben
Hi Ben, Could you share your configuration? I have the feeling here you're mixing too issues: DNS resolution at run time and the ability to expand backend with more servers. Baptiste
