oh, i think this comment thread explains it: http://comments.gmane.org/gmane.comp.web.haproxy/20366. I'll see about assigning
CAP_NET_ADMIN On Wed, Jul 15, 2015 at 4:56 PM Nathan Williams <nath.e.w...@gmail.com> wrote: > Hi Baptiste, > > Sorry for the delayed response, had some urgent things come up that > required more immediate attention... thanks again for your continued > support. > > > > Why not using proxy-protocol between HAProxy and nginx? > > Sounds interesting; I'd definitely heard of it before, but hadn't looked > into it since what we've been doing has been working. My initial impression > is that it's a pretty big change from what we're currently doing (looks > like it would at least require a brief maintenance to roll out since it > requires coordinated change between client and load-balancer), but I'm not > fundamentally opposed if there's significant advantages. I'll definitely > take a look to see if it satisfies our requirements. > > > > I disagree, it would be only 2: the 'real' IP addresses of the > load-balancers only. > > OK, fair point. Maybe it's just being paranoid to think that unless we're > explicitly setting the source, we should account for *all* possible > sources. The VIP wouldn't be the default route, so we could probably get > away with ignoring it. Come to think of it... maybe having keepalived > change the default route on the primary and skipping hardcoding the source > in haproxy would address what we're aiming for? seems worth further > investigation, as I'm not sure whether it supports this out of the box. > > > > there is no 0.0.0.0 magic values neither subnet values accepted in nginx > XFF module? > > I wouldn't use 0.0.0.0 whether there is or not, as i wouldn't want it to > be that open. It might be a different case for a subnet value, if we were > able to put the load-balancer cluster in a separate subnet, but our current > situation (managed private openstack deployment) doesn't give us quite that > much network control. maybe someday soon with VXLAN or another overlay (of > course, that comes with performance penalties, so maybe not). > > > > Then instead of using a VIP, you can book 2 IPs in your subnet that > could be used, whatever the LB is using. > > Pre-allocating network IPs from the subnet that aren't permitted to be > assigned to anything other than whatever instance is currently filling the > load-balancer role would certainly work (I like this idea!); that's > actually pretty similar to what we're doing for the internal VIP currently > (the external VIP is just an openstack floating IP, aka a DNAT in the > underlying infrastructure), and then adding it as an allowed address for > the instance-associated network "port" instance in Neutron's > allowed-address-pairs... It'd be an extra step when creating an LB node, > but a pretty reasonable one I think, and we're already treating them > differently from "generic" instances anyways... definitely food for thought. > > > HAProxy rocks ! > > +1 * 100. :) > > > > Can you start it up with strace ?? > > Yep! https://gist.github.com/nathwill/ea52324867072183b695 > > So far, I still like the "source 0.0.0.0 usesrc 10.240.36.13" solution the > best, as it seems the most direct and easily understood. Fingers crossed > the permissions issue is easily overcome. > > Cheers, > > Nathan W > > On Tue, Jul 14, 2015 at 2:58 PM Baptiste <bed...@gmail.com> wrote: > >> > As for details, it's advantageous for us for a couple of reasons... the >> > realip module in nginx requires that you list "trusted" hosts which are >> > permitted to set the X-Forwarded-For header before it will set the >> "source" >> > address in the logs to the x-forwarded-for address. as a result, using >> > anything other than the VIP means: >> >> Why not using proxy-protocol between HAProxy and nginx? >> http://blog.haproxy.com/haproxy/proxy-protocol/ >> >> So you can get rid of X-FF header limitation in nginx. (don't know if >> proxy-protocol implementation in nginx suffers from the same >> limitations). >> >> > - not using the vip means we have to trust 3 addresses instead of 1 to >> set >> > x-forwarded-for >> >> I disagree, it would be only 2: the 'real' IP addresses of the >> load-balancers only. >> >> > - we have to update the list of allowed hosts on all of our backends any >> > time we replace a load-balancer node. We're using config management, so >> it's >> > automated, but that's still more changes than should ideally be >> necessary to >> > replace a no-data node that we ideally can trash and replace at will. >> >> there is no 0.0.0.0 magic values neither subnet values accepted in >> nginx XFF module? >> If not, it deserves a patch ! >> >> > - there's a lag between the time of a change(e.g. node replacement) >> and the >> > next converge cycle of the config mgmt on the backends, so for some >> period >> > the backend config will be out of sync, incorrectly trusting IP(s) that >> may >> > now be associated with another host, or wrongly refusing to set the >> "source" >> > ip to the x-forwarded-for address. this is problematic for us, since we >> have >> > a highly-restricted internal environment, due to our business model >> (online >> > learn-to-code school) being essentially "running untrusted code as a >> > service". >> >> Then instead of using a VIP, you can book 2 IPs in your subnet that >> could be used, whatever the LB is using. >> So you don't rely on the VIP, whatever the HAProxy box real IP, you >> configure one of the IP above as an alias and you use it from HAProxy. >> >> > Happily, your suggested solution seems to achieve what we're aiming for >> > (thanks!). The health-checks are coming from the local IP, and proxied >> > requests from clients are coming from the VIP. The standby is seeing >> > backends as "UP" since they're able to pass the health-checks. Progress! >> >> Finally we made it :) >> HAProxy rocks ! >> >> > Unfortunately, this seems to cause another problem with our config... >> though >> > haproxy passes the config validation (haproxy -c -f /etc/haproxy.cfg), >> it >> > fails to start up, logging an error like "Jul 14 20:22:48 >> > lb01.stage.iad01.treehouse haproxy-systemd-wrapper[25225]: [ALERT] >> > 194/202248 (25226) : [/usr/sbin/haproxy.main()] Some configuration >> options >> > require full privileges, so global.uid cannot be changed.". We can get >> it to >> > work by removing the user and group directives from the global section >> and >> > letting haproxy run as root, but having to escalate privileges is also >> less >> > than ideal... I almost hate to ask for further assistance, but do you >> have >> > any suggestions related to the above? FWIW, we're using haproxy 1.5.4 >> and >> > kernel 4.0.4 on CentOS 7. >> >> Some features require root privileges, that said, from a documentation >> point of view, It doesn't seem the 'source' keyword like I asked you >> to set it up is one of them. >> >> Can you start it up with strace ?? >> >> Baptiste >> >> >> > Regards, >> > >> > Nathan W >> > >> > On Tue, Jul 14, 2015 at 12:31 PM Baptiste <bed...@gmail.com> wrote: >> >> >> >> Nathan, >> >> >> >> The question is: why do you want to use the VIP to get connected on >> >> your backend server? >> >> >> >> Please give a try to the following source line, instead of your current >> >> one: >> >> source 0.0.0.0 usesrc 10.240.36.13 >> >> >> >> Baptiste >> >> >> >> >> >> On Tue, Jul 14, 2015 at 9:06 PM, Nathan Williams < >> nath.e.w...@gmail.com> >> >> wrote: >> >> > OK, that did not seem to work, so I think the correct interpretation >> of >> >> > that >> >> > "addr" option must be as an override for what address/port to perform >> >> > the >> >> > health-check *against* instead of from (which makes more sense in >> >> > context of >> >> > it being a server option). >> >> > >> >> > i was hoping for an option like "health-check-source" or similar, if >> >> > that >> >> > makes sense; I also tried removing the "source" directive and binding >> >> > the >> >> > frontend to the VIP explicitly, hoping that would cause the proxied >> >> > requests >> >> > to originate from the bound IP, but that didn't seem to do it either. >> >> > While >> >> > the standby was then able to see the backends as "up", the proxied >> >> > requests >> >> > to the backends came from the local IP instead of the VIP. >> >> > >> >> > Regards, >> >> > >> >> > Nathan W >> >> > >> >> > On Tue, Jul 14, 2015 at 8:58 AM Nathan Williams < >> nath.e.w...@gmail.com> >> >> > wrote: >> >> >> >> >> >> Hi Baptiste/Jarno, >> >> >> >> >> >> Thanks so much for responding. >> >> >> >> >> >> "addr" does indeed look like a promising option (though a strangely >> >> >> lacking explanation in the docs, which explains what it makes >> possible >> >> >> while >> >> >> leaving the reader to deduce what it actually does), thanks for >> >> >> pointing >> >> >> that out. >> >> >> >> >> >> Here's our config: >> >> >> https://gist.github.com/nathwill/d30f2e9cc0c97bc5fc6f >> >> >> (believe it or not this is the trimmed down version from what we >> used >> >> >> to >> >> >> have :), but backends, how they propagate in this >> microservice-oriented >> >> >> world of ours... ). >> >> >> >> >> >> As for addresses, the VIP is 10.240.36.13, and the active/standby >> local >> >> >> addresses are .11 and .12. >> >> >> >> >> >> fthe problem is basically that the way it's currently configured, >> when >> >> >> the >> >> >> .11 is active and has the .13 address, health-checks from haproxy on >> >> >> the .12 >> >> >> host also originate from the .13 address (guessing due to the >> "source" >> >> >> line), and so never return and are (rightfully) marked by haproxy as >> >> >> L4CON >> >> >> network timeouts. >> >> >> >> >> >> i'm going to try the "addr" config and report back; fingers crossed! >> >> >> >> >> >> cheers, >> >> >> >> >> >> Nathan W >> >> >> >> >> >> On Tue, Jul 14, 2015 at 5:21 AM Baptiste <bed...@gmail.com> wrote: >> >> >>> >> >> >>> On Mon, Jul 13, 2015 at 6:03 PM, Nathan Williams >> >> >>> <nath.e.w...@gmail.com> >> >> >>> wrote: >> >> >>> > Hi all, >> >> >>> > >> >> >>> > I'm hoping I can get some advice on how we can improve our >> failover >> >> >>> > setup. >> >> >>> > >> >> >>> > At present, we have an active-standby setup. Failover works >> really >> >> >>> > well, but >> >> >>> > on the standby, none of the backend servers are marked as "up" >> since >> >> >>> > haproxy >> >> >>> > is bound to the VIP that is currently on the active member >> (managed >> >> >>> > with >> >> >>> > keepalived). as a result, there's an initial period of a second >> or >> >> >>> > two >> >> >>> > after >> >> >>> > the failover triggers and the standby claims the VIP where the >> >> >>> > backend >> >> >>> > servers have not yet passed a health-check on the new active >> member. >> >> >>> > >> >> >>> > It seems like the easiest way to sort it out would be if the >> >> >>> > health-checks >> >> >>> > weren't also bound to the VIP so that the standby could complete >> >> >>> > them >> >> >>> > successfully. i do still want the proxied requests bound to the >> VIP >> >> >>> > though, >> >> >>> > forthe benefit of our backends' real-ip configuration. >> >> >>> > >> >> >>> > is that doable? if not, is there some way to have the standby >> >> >>> > "follow" >> >> >>> > the >> >> >>> > active-member's view on the backends, or another way i haven't >> seen >> >> >>> > yet? >> >> >>> > >> >> >>> > Thanks! >> >> >>> > >> >> >>> > Nathan W >> >> >>> >> >> >>> Hi Nathan, >> >> >>> >> >> >>> Maybe you could share your configuration. >> >> >>> Please also let us know the real and virtual IPs configured on your >> >> >>> master and slave HAProxy servers. >> >> >>> >> >> >>> Baptiste >> >
