Hi Craig, On Mon, Feb 07, 2011 at 09:24:24PM +0100, Craig wrote: > Hi, > > >> The X-Forwarded-For header is only added once at the end of all > processing. > >> Otherwise, having it in the defaults section would result in both your > >> frontend and your backend adding it. > Then the possibility to add it only to a frontend or a backend in the > defaults section would be nice?
It is already the case. The fact is that we're telling haproxy that we want an outgoing request to have the header. If you set the option in the frontend, it will have it. If you set it in the backend, it will have it. If you set it in both, it will only be added once. It's really a flag : when the request passes through a frontend or backend which has the option, then it will have the header appended. > >> So in your case, what happens is that you delete it in the frontend > (using > >> reqidel) then you tag the session for adding a new one after all > processing > >> is done. > >> > >> When at the last point we have to establish a connection to the > server, we > >> check the header and balance based on it. I agree we should always > have it > >> filled with the same value, so there's a bug. > So if I got it right, I cannot balance based on the new header because > it was not added yet. That behaviour comes really unexpected because one > usually would believe it was already added in the frontend. It can come unexpected when you reason with header addition, but it's sort of an implicit header addition. The opposite would be much more unexpected, you'd really not want the header to be added twice because it was enable in both sections. It's possible that the doc is not clear enough : "This option may be specified either in the frontend or in the backend. If at least one of them uses it, the header will be added. Note that the backend's setting of the header subargument takes precedence over the frontend's if both are defined." Maybe we should insist on the fact that it's done only at the end. We could try to add it in the frontend and tag the session to know it was already performed. But this would slightly change the semantics to a new one which might not necessarily be desirable. For instance, it's possible in a backend to delete the header and set the option. That way you know that your servers will receive exactly one occurrence of it. Many people are doing that because their servers are having issues with this header passed as a list. Changing the behaviour would result in the backend's delete rule to suppress the header that was just added, and the new one won't be added anymore since it already was. > >> My guess is that you're running a version prior to 1.4.10 which has the > >> header deletion bug : the header list can become corrupted when exactly > >> two consecutive headers are removed from the request (eg: connection and > >> x-forwarded-for). Then the newly added X-Forwarded-For could not be seen > >> by the code responsible for hashing it. > >> > >> If so, please try to upgrade to the last bug fix (1.4.10) and see if the > >> problem persists. > I am already using 1.4.10 - sorry, it seems I somehow forgot to mention > it! :/ OK so I'm interested in any reliabe reproducer for this bug (eg: config and/or request exhibiting the issue). You can send me your config privately if you don't want to post it to the list. > That is a good hint, but I also have a frontend for SSL (with stunnel > which adds the X-Forward-For header) that I'd want to use the same > backend. I did not like defining backends twice as it introduces > redundancy and might lead to inconsistency, it is a good workaround > though. Note: my testing and the bug happened with the normal frontend. OK I see. Be aware that this setup is not compatible with keep-alive though, as stunnel will only add the header in the first request. An alternative is to apply the patch for the proxy protocol to stunnel and use it with either haproxy 1.5-dev, or use the 1.4 backports that were recently posted to the list. > Also, I could leave out the reqidel of the header, but then a malicious > party could theoretically choose the server it accesses (by forging > x-forwarded-for) and overload one after another; I prefer to take away > this possibility (yea I am overdoing it, maybe). ;) Targetting a server is a false problem. It can also be done by forcing cookies when persistence is used. And even when persistence cookies are encrypted and not predictible, it's enough to make one valid request and replay very fast with the assigned cookie to always hit the same server ;-) Your infrastructure and configuration should ensure that your servers don't die from overloading. That will protect you from this specific issue. Regards, Willy
