Hi, I decided to narrow the bug a bit and deleted all other backends/frontends we have; I've defined three servers in the backend which all query www.google.de. Thus you do not have to set up your own server if you want to test this config, google can take the load. ;)
The problem is reproducable with this config, I used need netcat here.
Running with #1 config ("reqidel ^X-Forwarded-For:.*" in frontend_btg not set).
case a) jumps between backends:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
Connection: keep-alive
EOF
case b) stays on same backend:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
Connection: close
EOF
case c) stays on same backend:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
X-Forwarded-For: 127.0.0.1
Connection: close
EOF
case d) stays on same backend:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
X-Forwarded-For: 127.0.0.1
Connection: keep-alive
EOF
Expected behaviour:
Case a) should not jump between servers. An empty x-forwarded-for header means
that always the "same" header (an empty one) should be hashed, you should
always end up on the same server.
Case a) and b) should behave the same. What does it matter if the connection is
set to keep-alive or close, I've set option httpclose anyways.
Running with #2 config ("reqidel ^X-Forwarded-For:.*" in frontend_btg is set).
case e) jumps between backends:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
Connection: keep-alive
EOF
case f) stays on same backend:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
Connection: close
EOF
case g) stays on same backend:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
X-Forwarded-For: 127.0.0.1
Connection: close
EOF
case h) jumps between backends:
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
X-Forwarded-For: 127.0.0.1
Connection: keep-alive
EOF
Expected behaviour:
Case e) same expections as with case a) and config-1
Case h) should really stay on one backend. I haproxy to delete X-Forwarded-For
on the frontend, add new "X-Forwarded-For: SRC-IP", and balance based on that
header in the backend.
With this behaviour you will get some problems with http/https and sessions,
stunnel will add an X-Forwarded-For header which contains the actual IP, but
the user might have sent a different one (or none) resulting in the client to
access different backends with http than with https.
Best regards,
Craig
----- original Nachricht --------
Betreff: Re: balance (hdr) problem (maybe bug?)
Gesendet: Do, 10. Feb 2011
Von: Willy Tarreau<[email protected]>
> Hi Craig,
>
> On Mon, Feb 07, 2011 at 09:24:24PM +0100, Craig wrote:
> > Hi,
> >
> > >> The X-Forwarded-For header is only added once at the end of all
> > processing.
> > >> Otherwise, having it in the defaults section would result in both your
> > >> frontend and your backend adding it.
> > Then the possibility to add it only to a frontend or a backend in the
> > defaults section would be nice?
>
> It is already the case. The fact is that we're telling haproxy that we
> want an outgoing request to have the header. If you set the option in
> the frontend, it will have it. If you set it in the backend, it will
> have it. If you set it in both, it will only be added once. It's really
> a flag : when the request passes through a frontend or backend which
> has the option, then it will have the header appended.
>
> > >> So in your case, what happens is that you delete it in the frontend
> > (using
> > >> reqidel) then you tag the session for adding a new one after all
> > processing
> > >> is done.
> > >>
> > >> When at the last point we have to establish a connection to the
> > server, we
> > >> check the header and balance based on it. I agree we should always
> > have it
> > >> filled with the same value, so there's a bug.
> > So if I got it right, I cannot balance based on the new header because
> > it was not added yet. That behaviour comes really unexpected because one
> > usually would believe it was already added in the frontend.
>
> It can come unexpected when you reason with header addition, but it's sort
> of an implicit header addition. The opposite would be much more unexpected,
> you'd really not want the header to be added twice because it was enable in
> both sections. It's possible that the doc is not clear enough :
>
> "This option may be specified either in the frontend or in the backend. If
> at
> least one of them uses it, the header will be added. Note that the
> backend's
> setting of the header subargument takes precedence over the frontend's if
> both are defined."
>
> Maybe we should insist on the fact that it's done only at the end.
>
> We could try to add it in the frontend and tag the session to know it was
> already performed. But this would slightly change the semantics to a new
> one which might not necessarily be desirable. For instance, it's possible
> in a backend to delete the header and set the option. That way you know
> that your servers will receive exactly one occurrence of it. Many people
> are doing that because their servers are having issues with this header
> passed as a list. Changing the behaviour would result in the backend's
> delete rule to suppress the header that was just added, and the new one
> won't be added anymore since it already was.
>
> > >> My guess is that you're running a version prior to 1.4.10 which has
> the
> > >> header deletion bug : the header list can become corrupted when
> exactly
> > >> two consecutive headers are removed from the request (eg: connection
> and
> > >> x-forwarded-for). Then the newly added X-Forwarded-For could not be
> seen
> > >> by the code responsible for hashing it.
> > >>
> > >> If so, please try to upgrade to the last bug fix (1.4.10) and see if
> the
> > >> problem persists.
> > I am already using 1.4.10 - sorry, it seems I somehow forgot to mention
> > it! :/
>
> OK so I'm interested in any reliabe reproducer for this bug (eg: config
> and/or
> request exhibiting the issue). You can send me your config privately if you
> don't want to post it to the list.
>
> > That is a good hint, but I also have a frontend for SSL (with stunnel
> > which adds the X-Forward-For header) that I'd want to use the same
> > backend. I did not like defining backends twice as it introduces
> > redundancy and might lead to inconsistency, it is a good workaround
> > though. Note: my testing and the bug happened with the normal frontend.
>
> OK I see. Be aware that this setup is not compatible with keep-alive
> though,
> as stunnel will only add the header in the first request. An alternative is
> to apply the patch for the proxy protocol to stunnel and use it with either
> haproxy 1.5-dev, or use the 1.4 backports that were recently posted to the
> list.
>
> > Also, I could leave out the reqidel of the header, but then a malicious
> > party could theoretically choose the server it accesses (by forging
> > x-forwarded-for) and overload one after another; I prefer to take away
> > this possibility (yea I am overdoing it, maybe). ;)
>
> Targetting a server is a false problem. It can also be done by forcing
> cookies when persistence is used. And even when persistence cookies are
> encrypted and not predictible, it's enough to make one valid request and
> replay very fast with the assigned cookie to always hit the same server ;-)
>
> Your infrastructure and configuration should ensure that your servers don't
> die from overloading. That will protect you from this specific issue.
>
> Regards,
> Willy
>
>
>
--- original Nachricht Ende ----
haproxy-config-2.cfg
Description:
haproxy-config-1.cfg
Description:
