Re: Server IP resolution using DNS in HAProxy
Hi Cyril, Baptiste, Just a quick response on this part to clear up some confusion. On Mon, Jul 20, 2015 at 01:11:22AM +0200, Cyril Bonté wrote: The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. IE, a new server directive could be init-addr (for inital IP address) and would take a list of directive from 'libc', 'dns', 'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to come...) Currently, HAProxy works like this: init-addr libc,dns A new value could be init-addr dns Or init-addr 1.2.3.4,dns In fact not, right now it's only init-addr libc since we cannot do DNS during init time. To be clear, does it mean that we could meet such configuration ? server server1 host1.somewhere:80 init-addr 127.0.0.1,libc,dns Which would mean : - Use the init IP first - At start up, try to resolve whith the libc resolver - At runtime, try to resolve with HAProxy resolvers. The purpose of init-addr was not this in fact but to declare how to resolve the initial address. Ultimately if Baptiste's DNS code can work out of the poll loop, we could consider using it for the very first resolution. In your case, the idea would be to have then : init-addr dns,libc,127.0.0.1 or : init-addr dns,libc,none This means : - try to resolve the name using DNS ; - if it fails, try again using libc ; - if it fails, use 127.0.0.1 or don't set any address This last option (hard-coded value or none) allows the server to start eventhough the resolution failed. We wanted to support an extra keyword such as last when haproxy is restarted and a dump file is present to provide the last IP address. It will solve the different needs of different people who want to set some preferences on config or on last dump, or on dns, etc... Then, maybe we'll have to prevent such (un)funny things : server server1 192.0.2.1:80 init-addr 127.0.0.1,libc,dns It's not absurd because this address is also the one used by the checks by default and some people might want it to continue to work this way. Regards, Willy
Re: Server IP resolution using DNS in HAProxy
Hi Baptiste, late feedback but it's better than none ;-) Le 12/07/2015 23:38, Baptiste a écrit : First, we want to fix the error when HAProxy fails starting up because the resolvers pointed by the system can't resolve a server's IP address (but HAProxy resolvers could). It is pretty awaited from some teams at work. I'll add that even if HAProxy resolvers can't resolve the entries or are not configured, we should allow to make it non fatal and leave such server entries definitely DOWN with healthchecks disabled (except if it is updated from the CLI). I see that Marco Corte has provided the same feedback ;-) The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. IE, a new server directive could be init-addr (for inital IP address) and would take a list of directive from 'libc', 'dns', 'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to come...) Currently, HAProxy works like this: init-addr libc,dns A new value could be init-addr dns Or init-addr 1.2.3.4,dns To be clear, does it mean that we could meet such configuration ? server server1 host1.somewhere:80 init-addr 127.0.0.1,libc,dns Which would mean : - Use the init IP first - At start up, try to resolve whith the libc resolver - At runtime, try to resolve with HAProxy resolvers. Then, maybe we'll have to prevent such (un)funny things : server server1 192.0.2.1:80 init-addr 127.0.0.1,libc,dns Second, we want to log server IP changes. For now, there are 2 ways to change a server IP address: DNS resolution or using the stats socket command: set server addr 2 options: - we setup a parameter to enable logging server IP changes, whatever has updated the server IP - we allow HAProxy to log server IP changes from a specific source only. IE, log only when DNS change a server's IP Why not something similar to init-addr ? like log-resolve-update libc,dns or log-resolve-update dns to prevent logging dns resolution during the init step (which will keep the current behaviour, to prevent flooding the logs when haproxy starts with a lot of servers declared). About the stats socket, I think it's important to always log server changes even it nothing is set in the configuration. Third, we have to handle DNS response errors. We thought about the 4 following cases: - NX domain : all DNS servers can't resolve this host name - response timeout : no response was received - query refused : the DNS servers refused our query - other : all other cases = For each error, we can maintain the latest good IP for a period decided by the user. IE, if you want to keep a server up for 5 minutes while your servers return NX, then setup hold nx 5m in your resolvers section Fourth, we need a new server state when a DNS resolution is in error. Currently, we have 2 types of state: operational or administrative - administrative states: ready, maint, drain - operational states: down, failed, stopped We have to create a new state (should be operational) which reports that HAProxy is not able to perform a proper DNS resolution for this server. Once in that state, the server won't be able to get new traffic, health checks will be stopped too. HAProxy will turn the server in this state after the hold period described in step #3. When servers fall to this state, maybe we'll have to define how haproxy will react with some configurations. For example, what's the effect of force-persist on a server in this state ? I'd say it's not like a DOWN server and it shouldn't persist, but we'll have to be clear on such use cases. -- Cyril Bonté
Re: Server IP resolution using DNS in HAProxy
On 07/17/2015 10:37 PM, Baptiste wrote: First would be resolution of SRV records and actually using the port supplied by the SRV record as the port for the server. I looked at the code and it doesn't seem like too much work, most of it would probably be changing the config stuff accordingly. You're right, this could be an interesting option. Actually, we though about the SRV records to populate a full backend server list with IP, name, port, weight using a single DNS query. Awesome! That's exactly what I want :) It's what I tried to express in the second part of my request, probably in a slightly weird way. The other one is... well you asked for it ;) so here it goes: it would be great to express in the config something like resolve this name and use up to X servers from the reply. The backend then allocates X servers. Assuming that the initial lookup returns Y results, the (sorted) records get assinged to the first Y servers, the other X-Y servers get marked as down. Upon a new lookup, same procedure for a potentially changing value of Y. I realize this a pretty bold feature request for several reasons, but I have actually spent some thought on it think it might be doable without really violating any of HAProxy's design paradigms. I would also be willing to invest some time (code) into this myself. If you think this might be at least worth a discussion, I'd be happy to share some more detailed thoughts and it would be great to hear your thoughts on that, too. First, there are some limitations about creating servers on the fly in a backend. So instead, we propose you to pre-allocate servers by configuration and then wait for the DNS to populate it. It's kind of what I meant, although I had something in mind where you say once preallocate 20 servers for this backend instead of listing 20 servers with the same hostname. But that's syntactic sugar and I would happily use the solution listing a server for each desired allocation. I don't speak about a request per server, I speak here about one request for the whole farm :) Again, awesome! That's what I'm looking for! You go one step further than the design we have about SRV records to populate the backend. We thought using priority to decide whether a server is active or backup. The advantage is that you don't need to reload HAProxy to change your X value ;) Yes, perfect. And, what I tried to express above, HAProxy should mark some servers down if the number records received is less than the number of servers configured. I would welcome a contribution about SRV record type. That said, before this, I have to rewrite part of the response parser to store the response in a real DNS packet structure instead of keeping data in a buffer. Sounds good. I am not sure what you have in mind there or what parts of the code would be touched by that, so if you let me know when it's done I'll happily send some patches as a base for further discussion! Thanks, Baptiste, this get's me even more excited about HAProxy than I am already :) Cheers, Conrad -- Conrad Hoffmann Traffic Engineer SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany Managing Director: Alexander Ljung | Incorporated in England Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg | HRB 110657B
Re: Server IP resolution using DNS in HAProxy
Hello, On 07/14/2015 10:11 PM, Baptiste wrote: snip I know the message above is very long, but we really need your feedback! /snip First, many thanks for tackling this! It surely makes many peoples live much easier. Reading this makes me want two things, one of them being a little not-haproxy-like maybe. First would be resolution of SRV records and actually using the port supplied by the SRV record as the port for the server. I looked at the code and it doesn't seem like too much work, most of it would probably be changing the config stuff accordingly. The other one is... well you asked for it ;) so here it goes: it would be great to express in the config something like resolve this name and use up to X servers from the reply. The backend then allocates X servers. Assuming that the initial lookup returns Y results, the (sorted) records get assinged to the first Y servers, the other X-Y servers get marked as down. Upon a new lookup, same procedure for a potentially changing value of Y. I realize this a pretty bold feature request for several reasons, but I have actually spent some thought on it think it might be doable without really violating any of HAProxy's design paradigms. I would also be willing to invest some time (code) into this myself. If you think this might be at least worth a discussion, I'd be happy to share some more detailed thoughts and it would be great to hear your thoughts on that, too. Cheers, Conrad -- Conrad Hoffmann Traffic Engineer SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany Managing Director: Alexander Ljung | Incorporated in England Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg | HRB 110657B
Re: Server IP resolution using DNS in HAProxy
On Wed, Jul 15, 2015 at 8:28 AM, Marco Corte ma...@marcocorte.it wrote: Il 14/07/2015 22:11, Baptiste ha scritto: - when parsing the configuration, HAProxy uses libc functions and resolvers provided by the operating system = if the server can't be resolved at this step, then HAProxy can't start [...] First, we want to fix the error when HAProxy fails starting up because the resolvers pointed by the system can't resolve a server's IP address (but HAProxy resolvers could). The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. Hi Marco, Why not providing an option to start haproxy even if not all servers can be resolved? That's the purpose of my mail. I need this feature, but I want to make it in a way which would satisfy the community. Your proposal of the init-addr could be useful for a trick: I can set a surely unreacheable address to let haproxy start and then force/wait for the name resolution to have a working server. That's what we want. The hidden feature, is that you can start large farms and simply turn on DNS when spawning new application servers. Scale up without reloading HAProxy ;) A NX server state would be very nice. noted, thx Baptiste
Re: Server IP resolution using DNS in HAProxy
Hi Robin, I don't understand the necessity of the hold valid config option. DNS has something that takes care of this for you called the TTL. Besides if hold valid is shorter then the TTL it would be kind of pointless since the resolvers you are querying won't re-resolve until the TTL expires. Your server won't wait until end of TTL to fail ;) So you don't want to followup TTLs and prefer force HAProxy to resolve more often. In some cases, you don't choose the TTL (amazon), so 'hold valid' allows you to choose your own TTL. Tbh I don't really see the point of configuring the resolvers in haproxy when the OS has perfectly fine working facilities for this? Imagine a big company. Imagine the ops team managing HAProxy and the IT team managing the DNS servers. (It's a real case) When the ops team start up a new server, DNS propagation can be long (several minutes) before the DNS servers managed by the IT team are aware of the update (we speak about worldwide deployment). In order to start up the new service asap, then the ops team want to use the regular DNS servers and their own DNS server... There are many cases like this one, where the ops team doesn't have the hand over the DNS server. Same if you use a service discovery, then HAProxy can point its DNS request to it instead of regular DNS servers. What is the benefit besides possibly causing lookups to happen twice, once from the OS resolving stack and once from haproxies? If you really want exactly the same behavior as described you could always configure a local resolver that queries multiple other resolvers instead of recursing itself. you say this because you have the hand over your OS. We have many customers and community users where it's not the case. Once again, HAProxy, is a load-balancer, it needs the most accurate information and as fast as possible. You don't want to tune your local bind or powerdns just for HAProxy and prevent any other service to operate as usual. Baptiste
Re: Server IP resolution using DNS in HAProxy
Actually a local resolver can take care of that for you as well since every resolver I know allows configuring a different destination on domain basis. Also as described in the first email, the server has to be resolvable via the OS resolving stack as well otherwise haproxy won't start. That's the purpose of this thread. We want / need to get rid of this limitation and that's why we ask our community if the way we wanted to fix it makes sense. This means you cannot use custom domains without configuring some sort of custom resolver anyway. HAProxy's internal resolver can be made flexible enough for this purpose without being intrusive in the underlying operating system. Baptiste -Robin- Nenad Merdanovic wrote on 7/15/2015 08:56: Hello Robin, On 07/15/2015 08:49 AM, Robin Geuze wrote: Tbh I don't really see the point of configuring the resolvers in haproxy when the OS has perfectly fine working facilities for this? What is the benefit besides possibly causing lookups to happen twice, once from the OS resolving stack and once from haproxies? If you really want exactly the same behavior as described you could always configure a local resolver that queries multiple other resolvers instead of recursing itself. Because this would perfectly integrate with things like Consul (https://www.consul.io/docs/agent/dns.html), which are currently very widely used to provide service discovery. -Robin- Regards,
Re: Server IP resolution using DNS in HAProxy
First would be resolution of SRV records and actually using the port supplied by the SRV record as the port for the server. I looked at the code and it doesn't seem like too much work, most of it would probably be changing the config stuff accordingly. You're right, this could be an interesting option. Actually, we though about the SRV records to populate a full backend server list with IP, name, port, weight using a single DNS query. The other one is... well you asked for it ;) so here it goes: it would be great to express in the config something like resolve this name and use up to X servers from the reply. The backend then allocates X servers. Assuming that the initial lookup returns Y results, the (sorted) records get assinged to the first Y servers, the other X-Y servers get marked as down. Upon a new lookup, same procedure for a potentially changing value of Y. I realize this a pretty bold feature request for several reasons, but I have actually spent some thought on it think it might be doable without really violating any of HAProxy's design paradigms. I would also be willing to invest some time (code) into this myself. If you think this might be at least worth a discussion, I'd be happy to share some more detailed thoughts and it would be great to hear your thoughts on that, too. First, there are some limitations about creating servers on the fly in a backend. So instead, we propose you to pre-allocate servers by configuration and then wait for the DNS to populate it. I don't speak about a request per server, I speak here about one request for the whole farm :) You go one step further than the design we have about SRV records to populate the backend. We thought using priority to decide whether a server is active or backup. The advantage is that you don't need to reload HAProxy to change your X value ;) I would welcome a contribution about SRV record type. That said, before this, I have to rewrite part of the response parser to store the response in a real DNS packet structure instead of keeping data in a buffer. Baptiste
Re: Server IP resolution using DNS in HAProxy
Hey, I don't understand the necessity of the hold valid config option. DNS has something that takes care of this for you called the TTL. Besides if hold valid is shorter then the TTL it would be kind of pointless since the resolvers you are querying won't re-resolve until the TTL expires. Tbh I don't really see the point of configuring the resolvers in haproxy when the OS has perfectly fine working facilities for this? What is the benefit besides possibly causing lookups to happen twice, once from the OS resolving stack and once from haproxies? If you really want exactly the same behavior as described you could always configure a local resolver that queries multiple other resolvers instead of recursing itself. -Robin- Marco Corte wrote on 7/15/2015 08:28: Il 14/07/2015 22:11, Baptiste ha scritto: - when parsing the configuration, HAProxy uses libc functions and resolvers provided by the operating system = if the server can't be resolved at this step, then HAProxy can't start [...] First, we want to fix the error when HAProxy fails starting up because the resolvers pointed by the system can't resolve a server's IP address (but HAProxy resolvers could). The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. Hi, Baptiste. Since I am used to IP address I cannot figure out all possible implication of the server name DNS resolution :-) IMHO HAproxy should start in any case if the configuration is valid; only the unresolvable items should be marked as disabled or failing or down or whatever. A wrong DNS entry could stop a otherwise perfectly working configuration. Why not providing an option to start haproxy even if not all servers can be resolved? Your proposal of the init-addr could be useful for a trick: I can set a surely unreacheable address to let haproxy start and then force/wait for the name resolution to have a working server. A NX server state would be very nice. .marcoc
Re: Server IP resolution using DNS in HAProxy
Hello Robin, On 07/15/2015 08:49 AM, Robin Geuze wrote: Tbh I don't really see the point of configuring the resolvers in haproxy when the OS has perfectly fine working facilities for this? What is the benefit besides possibly causing lookups to happen twice, once from the OS resolving stack and once from haproxies? If you really want exactly the same behavior as described you could always configure a local resolver that queries multiple other resolvers instead of recursing itself. Because this would perfectly integrate with things like Consul (https://www.consul.io/docs/agent/dns.html), which are currently very widely used to provide service discovery. -Robin- Regards, -- Nenad Merdanovic | PGP: 0x423edcb2 Linkedin: http://www.linkedin.com/in/nenadmerdanovic
Re: Server IP resolution using DNS in HAProxy
Il 14/07/2015 22:11, Baptiste ha scritto: - when parsing the configuration, HAProxy uses libc functions and resolvers provided by the operating system = if the server can't be resolved at this step, then HAProxy can't start [...] First, we want to fix the error when HAProxy fails starting up because the resolvers pointed by the system can't resolve a server's IP address (but HAProxy resolvers could). The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. Hi, Baptiste. Since I am used to IP address I cannot figure out all possible implication of the server name DNS resolution :-) IMHO HAproxy should start in any case if the configuration is valid; only the unresolvable items should be marked as disabled or failing or down or whatever. A wrong DNS entry could stop a otherwise perfectly working configuration. Why not providing an option to start haproxy even if not all servers can be resolved? Your proposal of the init-addr could be useful for a trick: I can set a surely unreacheable address to let haproxy start and then force/wait for the name resolution to have a working server. A NX server state would be very nice. .marcoc
Re: Server IP resolution using DNS in HAProxy
Hey Nenad, Actually a local resolver can take care of that for you as well since every resolver I know allows configuring a different destination on domain basis. Also as described in the first email, the server has to be resolvable via the OS resolving stack as well otherwise haproxy won't start. This means you cannot use custom domains without configuring some sort of custom resolver anyway. -Robin- Nenad Merdanovic wrote on 7/15/2015 08:56: Hello Robin, On 07/15/2015 08:49 AM, Robin Geuze wrote: Tbh I don't really see the point of configuring the resolvers in haproxy when the OS has perfectly fine working facilities for this? What is the benefit besides possibly causing lookups to happen twice, once from the OS resolving stack and once from haproxies? If you really want exactly the same behavior as described you could always configure a local resolver that queries multiple other resolvers instead of recursing itself. Because this would perfectly integrate with things like Consul (https://www.consul.io/docs/agent/dns.html), which are currently very widely used to provide service discovery. -Robin- Regards,
Re: Server IP resolution using DNS in HAProxy
On Sun, Jul 12, 2015 at 11:38 PM, Baptiste bed...@gmail.com wrote: hi all, As you may have noticed already, HAProxy 1.6-dev2 version has integrated a new feature: server IP address resolution using DNS. Main purpose of this dev is to make HAProxy aware of a server IP change when using environment such as AWS or docker. Here is the current status of HAProxy and server name resolution: - when parsing the configuration, HAProxy uses libc functions and resolvers provided by the operating system = if the server can't be resolved at this step, then HAProxy can't start - in order to make DNS resolution operational at run time, health checks must be enabled on the server. Actually, the health check triggers name resolution - HAProxy uses its own resolvers using the new section called resolvers. - HAProxy queries ALL resolvers and take the first non-error response - a resolution is considered in error when ALL resolvers failed (whatever the failure was) - When a resolution is successful, HAProxy keep it for hold valid period. Once hold valid has expired, next health check will trigger a new DNS resolution Documentation about it: - http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#resolvers%20%28Server%20and%20default-server%20options%29 - http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#5.3 Now current status is briefly explained, we have a few WIP tasks we want to discuss with the community. We want to here feedback about additional features we have in mind. First, we want to fix the error when HAProxy fails starting up because the resolvers pointed by the system can't resolve a server's IP address (but HAProxy resolvers could). The idea here would to create a new flag on the server to tell HAProxy which IP to use. The server would be enabled when the IP has been provided by the expected tool. IE, a new server directive could be init-addr (for inital IP address) and would take a list of directive from 'libc', 'dns', 'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to come...) Currently, HAProxy works like this: init-addr libc,dns A new value could be init-addr dns Or init-addr 1.2.3.4,dns Second, we want to log server IP changes. For now, there are 2 ways to change a server IP address: DNS resolution or using the stats socket command: set server addr 2 options: - we setup a parameter to enable logging server IP changes, whatever has updated the server IP - we allow HAProxy to log server IP changes from a specific source only. IE, log only when DNS change a server's IP Third, we have to handle DNS response errors. We thought about the 4 following cases: - NX domain : all DNS servers can't resolve this host name - response timeout : no response was received - query refused : the DNS servers refused our query - other : all other cases = For each error, we can maintain the latest good IP for a period decided by the user. IE, if you want to keep a server up for 5 minutes while your servers return NX, then setup hold nx 5m in your resolvers section Fourth, we need a new server state when a DNS resolution is in error. Currently, we have 2 types of state: operational or administrative - administrative states: ready, maint, drain - operational states: down, failed, stopped We have to create a new state (should be operational) which reports that HAProxy is not able to perform a proper DNS resolution for this server. Once in that state, the server won't be able to get new traffic, health checks will be stopped too. HAProxy will turn the server in this state after the hold period described in step #3. That's all for now. Looking forward to read your feedback! Baptiste Hey everyone! I know the message above is very long, but we really need your feedback! An other point I want to add: do you think it would make sense to allow updating the server hostname? It could be useful in environment where people want to pre-configure a farm for scalability, but server host names are not predictable (amazon ??). Baptiste
