hash mapping on x-forwarded-for header?
Hi We have a backend cluster of 18 api servers which normally get hit from an haproxy instance on the public subnet. We like to use hash-type consistent to load balance and pin clients to specific servers in order to take advantage of local cache on the api servers. We recently deployed a few frontend nginx servers on a new project which are load balanced in this manner as well. However, when these servers hit the api cluster internally via haproxy they get pinned to only 3 backend api servers and cause them to melt. Is it possible to use hash-type consistent on the x-forwarded-for information from the request hitting the frontend nginx servers? Thank you Paul
Re: hash mapping on x-forwarded-for header?
On Tue, Sep 30, 2014 at 11:44 AM, Paul McIntire p...@skout.com wrote: Hi api servers and cause them to melt. Is it possible to use hash-type consistent on the x-forwarded-for information from the request hitting the frontend nginx servers? If you're using 1.5 the balance hdr(x-forwarded-for) option is probably what you want. http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#balance -Bryan
Re: [PATCH] Adding the client's IP address to the existing X-Forwarded-For header
Hi Matthieu, On Thu, Sep 15, 2011 at 01:55:22PM +0200, matthieu.lochegn...@altissemiconductor.com wrote: Hi, I'm facing a problem with the 'X-Forwarded-For' header. Although the HTTP RFC (2616) allows multiple occurences of the same header field, and describes how the values must be merged together, this is not correctly implemented by some servers. Especially JBoss or gSOAP only give one of the occurences to the target service, which cannot get nor build the whole 'X-Forwarded-For' comma separated list. Then why not raise bug reports to those products ? In some circumstances, it can even cause important security issues. Try to send them a POST with two content-length headers, one large then one small. If it is accepted it's possible that you can perform HTTP smugling attacks on them. So I have added a 'concat' keyword to the 'forwardfor' option, to have HAProxy append the client's IP address to the existing 'X-Forwarded-For' header, like traditional Squid proxies do. I am not sure of the correct behaviour this keyword should have together with the 'header' keyword. With the patches below, the 'X-Forwarded-For' header values are moved to the specified header, and the client's IP address is appended to the list.The old header is removed, the new one is added at the end of the header list. I strongly disagree about such a change for two reasons : - first, we can't have options to workaround all known products bugs, or configs will become quite awful and unmtainainable. - second, the way it is done makes the issue even worse as it now allows a client to hide its address : just send two xff headers, the first one will be modified by haproxy and the last one will remain. A sane product will consult the last one and see the address forged by the client. Oh and you can't simply change the logics to adjust the last header instead as it will not work with your buggy servers : haproxy would then change the last occurrence which will be ignored by the buggy servers, so they will still use only the client's information. So this is the wrong way of working around a dangerous server bug in my opinion and I really refuse to introduce nasty security issues that are hard to debug just because some products don't minimally follow protocols. The solution to your issues with those products is to remove the XFF header and let haproxy put its own version. As long as you let your clients send more than one header with those servers, you're screwed anyway. So basically you'd have this : reqidel ^x-forwarded-for option forwardfor Note: there are people who prefer to rename the old header instead of removing it. It's just a matter of taste. Regards, Willy
Redirect Loop when using X-Forwarded-Proto header.
I am using haproxy in combination with stunnel to perform SSL. My backend servers expect an X-Forwarded-Proto: https header to indicate that the request was sent over SSL. If this header is missing, the request is redirected to the https:// flavor of the URL. However, with haproxy-1.5-dev5, I am seeing that the header is only added to the first request of the connection. Subsequent requests are missing this header. Below is an example from a tcpdump. -- GET /private/ HTTP/1.1 Host: beta.mysite.com Connection: keep-alive Cache-Control: max-age=0 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.16 Safari/534.24 Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: sessionid=03412c52b518e63558dc3d2418b52dc9 X-Forwarded-Proto: http X-Forwarded-For: 10.10.10.10 HTTP/1.1 302 FOUND Date: Tue, 29 Mar 2011 16:28:45 GMT Server: Apache/2.2.3 (CentOS) Set-Cookie: sessionid=03412c52b518e63558dc3d2418b52dc9; expires=Tue, 29-Mar-2011 16:48:45 GMT; Max-Age=1200; Path=/ Location: https://beta.mysite.com/private/ Content-Length: 0 Keep-Alive: timeout=3, max=100 Connection: Keep-Alive Content-Type: text/html; charset=utf-8 GET /private/ HTTP/1.1 Host: beta.mysite.com Connection: keep-alive Cache-Control: max-age=0 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.16 Safari/534.24 Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: sessionid=03412c52b518e63558dc3d2418b52dc9 HTTP/1.1 302 FOUND Date: Tue, 29 Mar 2011 16:28:45 GMT Server: Apache/2.2.3 (CentOS) Set-Cookie: sessionid=03412c52b518e63558dc3d2418b52dc9; expires=Tue, 29-Mar-2011 16:48:45 GMT; Max-Age=1200; Path=/ Location: https://beta.mysite.com/private/ Content-Length: 0 Keep-Alive: timeout=3, max=99 Connection: Keep-Alive Content-Type: text/html; charset=utf-8 ... -- This redirect loop runs until the browser interrupts it and displays an error. My configuration follows: -- listen http-vip00 bind192.168.1.1:80 bind127.0.0.1:81 accept-proxy modehttp option httplog balance roundrobin reqidel ^X-Forwarded-For:.* acl is-ssl dst_port 81 reqadd X-Forwarded-Proto:\ https if is-ssl reqadd X-Forwarded-Proto:\ http unless is-ssl option forwardfor server www1 10.19.78.18:80 -- I have not yet had a chance to see if the same thing happens with previous versions of haproxy. Is this expected behavior or do I have something misconfigured?
Re: Redirect Loop when using X-Forwarded-Proto header.
I found the issue. From the haproxy manual: By default HAProxy operates in a tunnel-like mode with regards to persistent connections: for each connection it processes the first request and forwards everything else (including additional requests) to selected server. Once established, the connection is persisted both on the client and server sides. Use option http-server-close to preserve client persistent connections while handling every incoming request individually, dispatching them one after another to servers, in HTTP close mode. Use option httpclose to switch both sides to HTTP close mode. option forceclose and option http-pretend-keepalive help working around servers misbehaving in HTTP close mode. So: option http-server-close disables persistent connections to the backends, while keeping them for the frontend. This allows haproxy to modify each request to the backend and inject the needed headers. Sorry for the waste of bandwidth :-).
Re: X-Forwarded-For header
On Thu, Mar 24, 2011 at 09:12:46PM +0100, Willy Tarreau wrote: Hello Dmitry, On Thu, Mar 24, 2011 at 05:28:13PM +0300, Dmitry Sivachenko wrote: Hello! With option forwardfor, haproxy adds X-Forwarded-For header at the end of header list. But according to wikipedia: http://en.wikipedia.org/wiki/X-Forwarded-For and other HTTP proxies (say, nginx) there is standard format to specify several intermediate IP addresses: X-Forwarded-For: client1, proxy1, proxy2 Why don't you use these standard procedure to add client IP? Because these are not the standards. Standards are defined by RFCs, not by Wikipedia :-) I meant more like de-facto standard, sorry for the confusion. The format with single comma-delimited X-Forwarded-For is just more common. We already got this question anyway. The short answer is that both forms are strictly equivalent, and any intermediary is free to fold multiple header lines into a single one with values delimited by commas. Your application will not notice the difference (otherwise it's utterly broken and might possibly be sensible to many vulnerabilities such as request smugling attacks). Okay, thanks for the explanation.
X-Forwarded-For header
Hello! With option forwardfor, haproxy adds X-Forwarded-For header at the end of header list. But according to wikipedia: http://en.wikipedia.org/wiki/X-Forwarded-For and other HTTP proxies (say, nginx) there is standard format to specify several intermediate IP addresses: X-Forwarded-For: client1, proxy1, proxy2 Why don't you use these standard procedure to add client IP? (I mean if X-Forwarded-For already exists in request headers, modify its value with client IP and do not create another header with the same name). Thanks!
Re: X-Forwarded-For header
Hello Dmitry, On Thu, Mar 24, 2011 at 05:28:13PM +0300, Dmitry Sivachenko wrote: Hello! With option forwardfor, haproxy adds X-Forwarded-For header at the end of header list. But according to wikipedia: http://en.wikipedia.org/wiki/X-Forwarded-For and other HTTP proxies (say, nginx) there is standard format to specify several intermediate IP addresses: X-Forwarded-For: client1, proxy1, proxy2 Why don't you use these standard procedure to add client IP? Because these are not the standards. Standards are defined by RFCs, not by Wikipedia :-) We already got this question anyway. The short answer is that both forms are strictly equivalent, and any intermediary is free to fold multiple header lines into a single one with values delimited by commas. Your application will not notice the difference (otherwise it's utterly broken and might possibly be sensible to many vulnerabilities such as request smugling attacks). Hoping this helps, Willy
Re: X-Forwarded-For header
I know there have been several emails about this, but what is the most secure way of logging the client's IP address in the application code? Do you just log the full X-Forwarded-For comma delimited value? Also, can't they manipulate the X-Forwarded-For header in the HTTP request? Regards, Bradford On Thu, Mar 24, 2011 at 4:12 PM, Willy Tarreau w...@1wt.eu wrote: Hello Dmitry, On Thu, Mar 24, 2011 at 05:28:13PM +0300, Dmitry Sivachenko wrote: Hello! With option forwardfor, haproxy adds X-Forwarded-For header at the end of header list. But according to wikipedia: http://en.wikipedia.org/wiki/X-Forwarded-For and other HTTP proxies (say, nginx) there is standard format to specify several intermediate IP addresses: X-Forwarded-For: client1, proxy1, proxy2 Why don't you use these standard procedure to add client IP? Because these are not the standards. Standards are defined by RFCs, not by Wikipedia :-) We already got this question anyway. The short answer is that both forms are strictly equivalent, and any intermediary is free to fold multiple header lines into a single one with values delimited by commas. Your application will not notice the difference (otherwise it's utterly broken and might possibly be sensible to many vulnerabilities such as request smugling attacks). Hoping this helps, Willy
Re: X-Forwarded-For header
On Thu, Mar 24, 2011 at 4:35 PM, bradford fingerm...@gmail.com wrote: I know there have been several emails about this, but what is the most secure way of logging the client's IP address in the application code? Do you just log the full X-Forwarded-For comma delimited value? Also, can't they manipulate the X-Forwarded-For header in the HTTP request? Delete any existing headers using reqdel/reqidel. reqidel X-Forwarded-For option forwardfor This will ensure the only one the backed sees is the one you added.
Re: X-Forwarded-For header
On Thu, Mar 24, 2011 at 5:01 PM, Ben Timby bti...@gmail.com wrote: Delete any existing headers using reqdel/reqidel. reqidel X-Forwarded-For option forwardfor This will ensure the only one the backed sees is the one you added. Sorry, more like: reqidel ^X-Forwarded-For:.* Found that in the docs after sending my first reply :-).
Re: X-Forwarded-For header
On Thu, Mar 24, 2011 at 04:35:57PM -0400, bradford wrote: I know there have been several emails about this, but what is the most secure way of logging the client's IP address in the application code? Do you just log the full X-Forwarded-For comma delimited value? Also, can't they manipulate the X-Forwarded-For header in the HTTP request? Anyone can manipulate it. However, standards are clear : when headers are folded into one, the order must be respected. If a product such as haproxy, squid, nginx, apache or whatever says that when it sets the IP address in x-forwarded-for, it APPENDS it, it means that the IP will always be the last one in the list when going out of that product. Thus here's what the application should do : - get the list of values in the same order they appear. - take the last value, or last-1, or last-2, etc... depending on the number of proxies in the infrastructure. Here's an example : client --- rproxy1 (cache) --- rproxy2 (LB) --- app server Let's say the client goes out through a proxy farm and already presents : X-Forwarded-For: 192.168.0.1, 172.16.0.1 The client reaches rproxy1 with IP 11.11.11.11. Rproxy1 then appends this value to the list. Let's say it adds it to the end of the list, out of rproxy1 you'd have : X-Forwarded-For: 192.168.0.1, 172.16.0.1, 11.11.11.11 Now consider rproxy1 has 192.168.2.1, and rproxy2 adds its IP to the list, this time using a separate line. After it, you'll get : X-Forwarded-For: 192.168.0.1, 172.16.0.1, 11.11.11.11 X-Forwarded-For: 192.168.2.1 This request reaches the app server. Depending on the framework used, maybe the list will automatically be folded by the application server, maybe you'll get raw headers and will have to concat them yourself. Anyway, you now have this list : 192.168.0.1, 172.16.0.1, 11.11.11.11, 192.168.2.1 Your app server knows that it is behind two reverse proxies, so it will take not the last entry, but the (N-1)th, which is 11.11.11.11. The former ones are useless, because they were presented by the client. However sometimes it might be worth logging them because in case of litigation it might help an admin on the other site to discover that some action was not performed by the person you think, for instance. Hoping this helps, Willy
X-Forwarded-For header addition or header extension
Hi, I'm using HAproxy 1.4.8 on RHEL5 (fully up-to-date). I'm using the 'option forwardfor' In certain circumstances, the client has already got an X-Forwarded-For header in the request. HAproxy in this instance adds a second X-Forwarded-For header rather than extending the existing header. This causes a problem on our Tomcat backend because the RemoteIP valve (part of Tomcat) that is used appears to examine only the first header. 1. Is there some way to get HAproxy to extend the existing X-Forwarded-For header rather than adding a new one 2. Does the stunnel patch do the same thing or does it extend the existing X-Forwarded-For header? Thanks, -- Best Regards, Brett Delle Grazie
Re: X-Forwarded-For header addition or header extension
On Dec 10, 2010, at 6:42 AM, Brett Delle Grazie wrote: Hi, I'm using HAproxy 1.4.8 on RHEL5 (fully up-to-date). I'm using the 'option forwardfor' In certain circumstances, the client has already got an X-Forwarded-For header in the request. HAproxy in this instance adds a second X-Forwarded-For header rather than extending the existing header. This causes a problem on our Tomcat backend because the RemoteIP valve (part of Tomcat) that is used appears to examine only the first header. 1. Is there some way to get HAproxy to extend the existing X-Forwarded-For header rather than adding a new one 2. Does the stunnel patch do the same thing or does it extend the existing X-Forwarded-For header? It sounds like this is an issue with the RemoteIP valve. (I haven't looked at its code yet.) According to the HTTP spec, headers that allow multiple values can be specified as either a comma-separated list or multiple separate headers: Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one field-name: field-value pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded. That is, the following are semantically equivalent according to the HTTP spec: X-Forwarded-For: 192.168.100.123 X-Forwarded-For: 127.0.0.1 and X-Forwarded-For: 192.168.100.123, 127.0.0.1 In fact, proxies and caches may make these changes in-flight. If you don't need the client's existing X-Forwarded-For information and can't wait for a Tomcat fix, you can always do a `reqidel ^X-Forwarded-Proto:.*' in your haproxy config. Then the only XFF header you will get in Tomcat will be from haproxy. (This may cause issues if you're using stunnel, though.)
Re: X-Forwarded-For header addition or header extension
Jim, Thanks, I've raised the issue with tomcat users and they recommended to create a bugzilla entry, which I will do. I will also point them to this mailing list for the definition. I'm using stunnel so the second suggestion won't work. The most simple and obvious solution is to use a different header that won't be used elsewhere - but that's a work-around. Thanks for your help. On 10 December 2010 13:30, Jim Riggs freebsd-li...@christianserving.orgwrote: On Dec 10, 2010, at 6:42 AM, Brett Delle Grazie wrote: Hi, I'm using HAproxy 1.4.8 on RHEL5 (fully up-to-date). I'm using the 'option forwardfor' In certain circumstances, the client has already got an X-Forwarded-For header in the request. HAproxy in this instance adds a second X-Forwarded-For header rather than extending the existing header. This causes a problem on our Tomcat backend because the RemoteIP valve (part of Tomcat) that is used appears to examine only the first header. 1. Is there some way to get HAproxy to extend the existing X-Forwarded-For header rather than adding a new one 2. Does the stunnel patch do the same thing or does it extend the existing X-Forwarded-For header? It sounds like this is an issue with the RemoteIP valve. (I haven't looked at its code yet.) According to the HTTP spec, headers that allow multiple values can be specified as either a comma-separated list or multiple separate headers: Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one field-name: field-value pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded. That is, the following are semantically equivalent according to the HTTP spec: X-Forwarded-For: 192.168.100.123 X-Forwarded-For: 127.0.0.1 and X-Forwarded-For: 192.168.100.123, 127.0.0.1 In fact, proxies and caches may make these changes in-flight. If you don't need the client's existing X-Forwarded-For information and can't wait for a Tomcat fix, you can always do a `reqidel ^X-Forwarded-Proto:.*' in your haproxy config. Then the only XFF header you will get in Tomcat will be from haproxy. (This may cause issues if you're using stunnel, though.) __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ -- Best Regards, Brett Delle Grazie
X-Forwarded-For header chaining
I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? -- Miguel Pilar
Re: X-Forwarded-For header chaining
On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. A.
Re: X-Forwarded-For header chaining
On 9/2/09 11:17 AM, Alexander Staubo a...@bengler.no wrote: On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. A. The typical way this is managed is an option to have the chaining software wipe the header to begin with or not. Essentially the default behavior should be optional (maybe have 'option chained-forwardfor', or something to the effect). For us haproxy isn't the first hop but we'd still like it to be in the chain, mostly because if we ever have to switch haproxy to their own server we can have the functionality. -- Miguel Pilar
Re: X-Forwarded-For header chaining
On Wed, Sep 02, 2009 at 05:17:28PM +0200, Alexander Staubo wrote: On Wed, Sep 2, 2009 at 3:31 PM, Miguel Pilar Vilagranmiguel.pi...@apex.pr wrote: I am seeing (with option forwardfor) that HAProxy is replacing X-Fowarded-For instead of chaining the proxy chain. I know it's not an RFC but the defacto standard is to chain the proxies by appending to the header. For my usage it is not necessary but thought I'd point it out (Varnish also doesn't handle the header properly but there's a workaround in VCL for it). Is there a setting for this that I am missing? Miguel, there might be something special in your config, because option forwardfor only appends a header, it does not remove it at all. If you want haproxy to remove the existing header, you have to explicitly tell it to do so using reqidel. The issue is that X-Forwarded-For can be spoofed by clients, and to prevent this, the proxy would need a list of upstream IPs for which it will trust the X-Forwarded-For header and chain it. We would very much like this functionality as well. We are in a situation where we're using HAProxy simply to bounce requests onwards to another HAProxy (for legacy issues related to IP address ownership), and we've had to modify our app since the client IPs are sometimes no longer available. The problem with this header (and a few others such as Via) is that it can appear multiple times, but it must always be chained in the correct sequence. Haproxy respects this. However I've already seen some applications using any random occurrence of the header, instead of the one which corresponds to the proxy position in their architecture. For instance, if there are 2 proxies before the application, the application must use occurrence N-2 where N is the number of occurrences of the header, to find the original client's IP address. Hoping this helps, Willy
Re: X-Forwarded-For header chaining
On Wed, Sep 02, 2009 at 05:06:01PM -0400, Miguel Pilar Vilagran wrote: The problem with this header (and a few others such as Via) is that it can appear multiple times, but it must always be chained in the correct sequence. Haproxy respects this. However I've already seen some applications using any random occurrence of the header, instead of the one which corresponds to the proxy position in their architecture. For instance, if there are 2 proxies before the application, the application must use occurrence N-2 where N is the number of occurrences of the header, to find the original client's IP address. Hoping this helps, Willy Willy what I meant by chaining is what Squid does, which is not append another header but if a header exists it appends the proxy IP to a comma+space separated list that may be on the header. There is no difference, it is 100% equivalent. See here: http://en.wikipedia.org/wiki/X-forwarded-for Better read the original doc, isn't it ? RFC2616 - Hypertext Transfer Protocol -- HTTP/1.1 Page 32, section 4.2 - Message headers Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one field-name: field-value pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded. It is exactly what is done here. Willy
