Hi Bartosz,
On Wed, Aug 24, 2016 at 09:54:36PM +0200, Bartosz Koni??ski wrote:
> Hey. I posted this bug report on
> http://discourse.haproxy.org/t/serious-security-issue-in-haproxy-1-6-8/600
> but then I read that bugs should be reported on this mailing list. So, here
> is the copy of my original bug report:
yes that's the right place, thanks for the report. I have some questions
below.
> Hello. It looks like there is serious security issue with HAProxy version
> 1.6.8. We are using same configuration for long time, and till yesterday
> (before we updated version from 1.6.7 to 1.6.8), everything worked just
> fine.
>
> Small overview on simplified version of our environment. Let's say that we
> have one frontend, and multiple backends. Routing to correct backend is
> based on host header of request ('acl host1 hdr_beg(host) -i
> host1.example.com', 'use_backend backend1 if host1', ...). Each backend has
> only one server. We have set 'errorfile 503 /some/maintenance/screen.http'.
>
> When backend is up - everything works just as expected.
> When backend is down - everything works just as expected.
>
> The magic happens, when backend is going down, in this small time window,
> when HAProxy checks are performed.
> Till yesterday, trying to connect to host1.example.com immediately after
> turning off service on one and only server of backend1 resulted in
> displaying our custom maintenance screen.
> Now trying to connect to host1.example.com results in serving content from
> random backend. ;-) So, in this short time period, when HAProxy is
> performing checks to ensure that backend1 is really DOWN, we can enter '
> host1.example.com' in browser, and hold F5 to get quick overview of all
> services available in our config (not sure if only backends accessible from
> frontend1, or all of them). After checks, when backend1 in 'haproxy?stats'
> is red, everything works fine - entering host1.example.com results in
> displaying our custom maintenance screen.
>
> Regarding the security.. In frontend1, we also have some ACLs, that allow
> access to some backends only from one IP address. When backend is going
> down, anyone trying to browse host1.example.com can bypass these ACLs, and
> randomly get access to restricted services. ;-)
>
> Downgrading to 1.6.7 resolves the issue, upgrading to 1.6.8 brings the
> issue back. For now, we are forced to use HAProxy 1.6.7.
I suspect it is a side effect of this fix :
1c12f4c ("BUG/MEDIUM: stream-int: completely detach connection on connect
error")
Could you try to revert it ? It properly detaches a connection upon a
failed connect() to a server before performing a retry. You're exactly
in this case. Now regarding how/why it could go to a random backend is
another question :-(
Do you have any http-reuse option ? I'm trying to figure how a connection
could be picked and used in this case. If you can post a simplified
anonymized config which reproduces it, it would help (ie: some apparently
harmless options may be needed to reproduce it).
Thanks!
Willy
