X-Forwarded-For header chaining
I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? -- Miguel Pilar
Re: X-Forwarded-For header chaining
On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. A.
Re: X-Forwarded-For header chaining
On 9/2/09 11:17 AM, Alexander Staubo a...@bengler.no wrote: On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. A. The typical way this is managed is an option to have the chaining software wipe the header to begin with or not. Essentially the default behavior should be optional (maybe have 'option chained-forwardfor', or something to the effect). For us haproxy isn't the first hop but we'd still like it to be in the chain, mostly because if we ever have to switch haproxy to their own server we can have the functionality. -- Miguel Pilar
Re: X-Forwarded-For header chaining
On Wed, Sep 02, 2009 at 05:17:28PM +0200, Alexander Staubo wrote: On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? Miguel, there might be something special in your config, because option forwardfor only appends a header, it does not remove it at all. If you want haproxy to remove the existing header, you have to explicitly tell it to do so using reqidel. The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. The problem with this header (and a few others such as Via) is that it can appear multiple times, but it must always be chained in the correct sequence. Haproxy respects this. However I've already seen some applications using any random occurrence of the header, instead of the one which corresponds to the proxy position in their architecture. For instance, if there are 2 proxies before the application, the application must use occurrence N-2 where N is the number of occurrences of the header, to find the original client's IP address. Hoping this helps, Willy
Re: stats socket, show errors always blank
On Tue, Sep 01, 2009 at 05:26:54PM -0500, Jeremy wrote: I've set up the stats socket and get results ok running show info or show stats etc, but for some reason show errors is always blank. Any tips to getting show errors to return the sexy output like is shown at the very bottom of http://haproxy.1wt.eu/download/1.3/doc/configuration.txt ? It simply means you have had no request nor response error. If you want to try to get one, send it an invalid request using telnet, such as : GET / / HTTP/1.0 You will find it in your show errors report. Basically we're still having an intermittent problem with a client that is getting 502 Bad Gateway errors, most likely due to some bad characters in their HTTP headers (specifically cookies we think), but I'm unable to tell the client exactly what is wrong since show errors is not showing anything. Do you see it in your logs too ? It is possible that another proxy between the client and haproxy returns the 502. Also, it is possible that haproxy sends a 502 because the application randomly kills sessions without responding. They want to track down and fix whatever's wrong with their website, so they have not wanted us to enable the options to accept invalid http requests/responses just yet. You can congratulate them, it's extremely rare these days to see people who want to comply with standards ! This particular load balancer just has 1 site running on it with a couple backends and one front end. I've also tried doing show errors IID using the various IID's in use but it always just comes back with nothing. The site in question has a ton of css includes, images, javascript files etc... The only thing I can think of is maybe once a valid request passes through HAProxy show errors is cleared, Not at all. It was designed exactly for the case you're trying to troubleshoot, which means a hosted customer having some bad characters once in a while without the ability to find which ones. So harpoxy will capture a bad request/response and keep it until you restart it or until a new invalid request/response replaces it, even one year after if needed. so we somehow need them to stop making any more requests once they run into 502 bad gateway until we can capture the show errors output? No, not at all. I have spammed show errors as fast as I possibly can while they're doing some testing but I've yet to actually see it return anything. Not needed either :-) However, I suggest to send it one last time before restarting haproxy, just in case you trap one. But quite honnestly, if you wait a long time without any, the application is fine IMHO. However, it's very likely that it breaks sessions. Check your logs for SH flags or even for 502 status codes. I'm sure your customer's server is randomly sending TCP RST packets in response to requests. One user reported such an issue with round-robin link load- balancing (which reorders packets) and made a windows server nervously send RST packets, thus aborting connections. Regards, Willy
Re: X-Forwarded-For header chaining
On Wed, Sep 02, 2009 at 05:06:01PM -0400, Miguel Pilar Vilagran wrote: The problem with this header (and a few others such as Via) is that it can appear multiple times, but it must always be chained in the correct sequence. Haproxy respects this. However I've already seen some applications using any random occurrence of the header, instead of the one which corresponds to the proxy position in their architecture. For instance, if there are 2 proxies before the application, the application must use occurrence N-2 where N is the number of occurrences of the header, to find the original client's IP address. Hoping this helps, Willy Willy what I meant by chaining is what Squid does, which is not append another header but if a header exists it appends the proxy IP to a comma+space separated list that may be on the header. There is no difference, it is 100% equivalent. See here: http://en.wikipedia.org/wiki/X-forwarded-for Better read the original doc, isn't it ? RFC2616 - Hypertext Transfer Protocol -- HTTP/1.1 Page 32, section 4.2 - Message headers Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one field-name: field-value pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded. It is exactly what is done here. Willy
