Hi Baptiste, Sorry for the delayed response, had some urgent things come up that required more immediate attention... thanks again for your continued support.
> Why not using proxy-protocol between HAProxy and nginx? Sounds interesting; I'd definitely heard of it before, but hadn't looked into it since what we've been doing has been working. My initial impression is that it's a pretty big change from what we're currently doing (looks like it would at least require a brief maintenance to roll out since it requires coordinated change between client and load-balancer), but I'm not fundamentally opposed if there's significant advantages. I'll definitely take a look to see if it satisfies our requirements. > I disagree, it would be only 2: the 'real' IP addresses of the load-balancers only. OK, fair point. Maybe it's just being paranoid to think that unless we're explicitly setting the source, we should account for *all* possible sources. The VIP wouldn't be the default route, so we could probably get away with ignoring it. Come to think of it... maybe having keepalived change the default route on the primary and skipping hardcoding the source in haproxy would address what we're aiming for? seems worth further investigation, as I'm not sure whether it supports this out of the box. > there is no 0.0.0.0 magic values neither subnet values accepted in nginx XFF module? I wouldn't use 0.0.0.0 whether there is or not, as i wouldn't want it to be that open. It might be a different case for a subnet value, if we were able to put the load-balancer cluster in a separate subnet, but our current situation (managed private openstack deployment) doesn't give us quite that much network control. maybe someday soon with VXLAN or another overlay (of course, that comes with performance penalties, so maybe not). > Then instead of using a VIP, you can book 2 IPs in your subnet that could be used, whatever the LB is using. Pre-allocating network IPs from the subnet that aren't permitted to be assigned to anything other than whatever instance is currently filling the load-balancer role would certainly work (I like this idea!); that's actually pretty similar to what we're doing for the internal VIP currently (the external VIP is just an openstack floating IP, aka a DNAT in the underlying infrastructure), and then adding it as an allowed address for the instance-associated network "port" instance in Neutron's allowed-address-pairs... It'd be an extra step when creating an LB node, but a pretty reasonable one I think, and we're already treating them differently from "generic" instances anyways... definitely food for thought. > HAProxy rocks ! +1 * 100. :) > Can you start it up with strace ?? Yep! https://gist.github.com/nathwill/ea52324867072183b695 So far, I still like the "source 0.0.0.0 usesrc 10.240.36.13" solution the best, as it seems the most direct and easily understood. Fingers crossed the permissions issue is easily overcome. Cheers, Nathan W On Tue, Jul 14, 2015 at 2:58 PM Baptiste <bed...@gmail.com> wrote: > > As for details, it's advantageous for us for a couple of reasons... the > > realip module in nginx requires that you list "trusted" hosts which are > > permitted to set the X-Forwarded-For header before it will set the > "source" > > address in the logs to the x-forwarded-for address. as a result, using > > anything other than the VIP means: > > Why not using proxy-protocol between HAProxy and nginx? > http://blog.haproxy.com/haproxy/proxy-protocol/ > > So you can get rid of X-FF header limitation in nginx. (don't know if > proxy-protocol implementation in nginx suffers from the same > limitations). > > > - not using the vip means we have to trust 3 addresses instead of 1 to > set > > x-forwarded-for > > I disagree, it would be only 2: the 'real' IP addresses of the > load-balancers only. > > > - we have to update the list of allowed hosts on all of our backends any > > time we replace a load-balancer node. We're using config management, so > it's > > automated, but that's still more changes than should ideally be > necessary to > > replace a no-data node that we ideally can trash and replace at will. > > there is no 0.0.0.0 magic values neither subnet values accepted in > nginx XFF module? > If not, it deserves a patch ! > > > - there's a lag between the time of a change(e.g. node replacement) and > the > > next converge cycle of the config mgmt on the backends, so for some > period > > the backend config will be out of sync, incorrectly trusting IP(s) that > may > > now be associated with another host, or wrongly refusing to set the > "source" > > ip to the x-forwarded-for address. this is problematic for us, since we > have > > a highly-restricted internal environment, due to our business model > (online > > learn-to-code school) being essentially "running untrusted code as a > > service". > > Then instead of using a VIP, you can book 2 IPs in your subnet that > could be used, whatever the LB is using. > So you don't rely on the VIP, whatever the HAProxy box real IP, you > configure one of the IP above as an alias and you use it from HAProxy. > > > Happily, your suggested solution seems to achieve what we're aiming for > > (thanks!). The health-checks are coming from the local IP, and proxied > > requests from clients are coming from the VIP. The standby is seeing > > backends as "UP" since they're able to pass the health-checks. Progress! > > Finally we made it :) > HAProxy rocks ! > > > Unfortunately, this seems to cause another problem with our config... > though > > haproxy passes the config validation (haproxy -c -f /etc/haproxy.cfg), it > > fails to start up, logging an error like "Jul 14 20:22:48 > > lb01.stage.iad01.treehouse haproxy-systemd-wrapper[25225]: [ALERT] > > 194/202248 (25226) : [/usr/sbin/haproxy.main()] Some configuration > options > > require full privileges, so global.uid cannot be changed.". We can get > it to > > work by removing the user and group directives from the global section > and > > letting haproxy run as root, but having to escalate privileges is also > less > > than ideal... I almost hate to ask for further assistance, but do you > have > > any suggestions related to the above? FWIW, we're using haproxy 1.5.4 and > > kernel 4.0.4 on CentOS 7. > > Some features require root privileges, that said, from a documentation > point of view, It doesn't seem the 'source' keyword like I asked you > to set it up is one of them. > > Can you start it up with strace ?? > > Baptiste > > > > Regards, > > > > Nathan W > > > > On Tue, Jul 14, 2015 at 12:31 PM Baptiste <bed...@gmail.com> wrote: > >> > >> Nathan, > >> > >> The question is: why do you want to use the VIP to get connected on > >> your backend server? > >> > >> Please give a try to the following source line, instead of your current > >> one: > >> source 0.0.0.0 usesrc 10.240.36.13 > >> > >> Baptiste > >> > >> > >> On Tue, Jul 14, 2015 at 9:06 PM, Nathan Williams <nath.e.w...@gmail.com > > > >> wrote: > >> > OK, that did not seem to work, so I think the correct interpretation > of > >> > that > >> > "addr" option must be as an override for what address/port to perform > >> > the > >> > health-check *against* instead of from (which makes more sense in > >> > context of > >> > it being a server option). > >> > > >> > i was hoping for an option like "health-check-source" or similar, if > >> > that > >> > makes sense; I also tried removing the "source" directive and binding > >> > the > >> > frontend to the VIP explicitly, hoping that would cause the proxied > >> > requests > >> > to originate from the bound IP, but that didn't seem to do it either. > >> > While > >> > the standby was then able to see the backends as "up", the proxied > >> > requests > >> > to the backends came from the local IP instead of the VIP. > >> > > >> > Regards, > >> > > >> > Nathan W > >> > > >> > On Tue, Jul 14, 2015 at 8:58 AM Nathan Williams < > nath.e.w...@gmail.com> > >> > wrote: > >> >> > >> >> Hi Baptiste/Jarno, > >> >> > >> >> Thanks so much for responding. > >> >> > >> >> "addr" does indeed look like a promising option (though a strangely > >> >> lacking explanation in the docs, which explains what it makes > possible > >> >> while > >> >> leaving the reader to deduce what it actually does), thanks for > >> >> pointing > >> >> that out. > >> >> > >> >> Here's our config: > >> >> https://gist.github.com/nathwill/d30f2e9cc0c97bc5fc6f > >> >> (believe it or not this is the trimmed down version from what we used > >> >> to > >> >> have :), but backends, how they propagate in this > microservice-oriented > >> >> world of ours... ). > >> >> > >> >> As for addresses, the VIP is 10.240.36.13, and the active/standby > local > >> >> addresses are .11 and .12. > >> >> > >> >> fthe problem is basically that the way it's currently configured, > when > >> >> the > >> >> .11 is active and has the .13 address, health-checks from haproxy on > >> >> the .12 > >> >> host also originate from the .13 address (guessing due to the > "source" > >> >> line), and so never return and are (rightfully) marked by haproxy as > >> >> L4CON > >> >> network timeouts. > >> >> > >> >> i'm going to try the "addr" config and report back; fingers crossed! > >> >> > >> >> cheers, > >> >> > >> >> Nathan W > >> >> > >> >> On Tue, Jul 14, 2015 at 5:21 AM Baptiste <bed...@gmail.com> wrote: > >> >>> > >> >>> On Mon, Jul 13, 2015 at 6:03 PM, Nathan Williams > >> >>> <nath.e.w...@gmail.com> > >> >>> wrote: > >> >>> > Hi all, > >> >>> > > >> >>> > I'm hoping I can get some advice on how we can improve our > failover > >> >>> > setup. > >> >>> > > >> >>> > At present, we have an active-standby setup. Failover works really > >> >>> > well, but > >> >>> > on the standby, none of the backend servers are marked as "up" > since > >> >>> > haproxy > >> >>> > is bound to the VIP that is currently on the active member > (managed > >> >>> > with > >> >>> > keepalived). as a result, there's an initial period of a second or > >> >>> > two > >> >>> > after > >> >>> > the failover triggers and the standby claims the VIP where the > >> >>> > backend > >> >>> > servers have not yet passed a health-check on the new active > member. > >> >>> > > >> >>> > It seems like the easiest way to sort it out would be if the > >> >>> > health-checks > >> >>> > weren't also bound to the VIP so that the standby could complete > >> >>> > them > >> >>> > successfully. i do still want the proxied requests bound to the > VIP > >> >>> > though, > >> >>> > forthe benefit of our backends' real-ip configuration. > >> >>> > > >> >>> > is that doable? if not, is there some way to have the standby > >> >>> > "follow" > >> >>> > the > >> >>> > active-member's view on the backends, or another way i haven't > seen > >> >>> > yet? > >> >>> > > >> >>> > Thanks! > >> >>> > > >> >>> > Nathan W > >> >>> > >> >>> Hi Nathan, > >> >>> > >> >>> Maybe you could share your configuration. > >> >>> Please also let us know the real and virtual IPs configured on your > >> >>> master and slave HAProxy servers. > >> >>> > >> >>> Baptiste >
