Hello,
On Fri, Jul 31, 2009 at 09:19:23AM +0200, Bo??tjan Mer??un wrote:
> Dear haproxy list,
>
> This message will be a bit longer, I hope that somebody will read it
> though and give me some opinion.
> I am testing HAProxy in our load-balanced environment, which now works
> with keepalived and the main reason for trying HAProxy is connection
> limiting and ACL, which keepalived doesn't have.
> We have more than 40 sites load balanced on 6 web servers and we have a
> problem that from time to time some of the sites gets huge amount of
> traffic. Much more then servers handle.
>
> I hoped I would be able to automate this with HAProxy, but I have some
> problems.
> I would like to have limits as centralized as possible, that is why I
> created one backend for all sites. In this case I can limit the taffic
> on backend and have frontends use whatever they can.
> The problem is that if I create the same rules for all frontends, they
> behave in the same way; when ACLs are true all stop working and then all
> start again. I would like to have all sites working even if one uses a
> lot of resources (for example: one using 80% of all resources and all
> others 20% or something like that).
Then you really need to use separate backends and assign them a certain
amount of connections. The advantage is that if a lot of traffic comes
to a backend, queueing will occur of that backend without impacting
other ones.
> My configuration is like that:
>
> frontend my_site1
> bind XXX.XXX.XXX.1:8880
> default_backend main
> acl my_site_toofast be_sess_rate(main) gt 6
> acl my_site_toomany connslots(main) lt 10
> acl my_site_slow fe_sess_rate lt 1
> use_backend sorry if my_site_toomany or my_site_toofast ! my_site_slow
I really don't know why you're limiting on the number of requests
per second. It is not the proper way to do this at all. In fact,
you should *only* need to play with the server's maxconn parameter,
as there should be no reason your servers would be sensible to a
number of requests per second.
(...)
> backend main
> option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
> server web1 YYY.YYY.YYY.1 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
> server web2 YYY.YYY.YYY.2 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
> server web3 YYY.YYY.YYY.3 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
> server web4 YYY.YYY.YYY.4 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
> server web5 YYY.YYY.YYY.5 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
> server web6 YYY.YYY.YYY.6 check port 8880 inter 10s fall 2 rise
> 3 weight 10 maxconn 50 maxqueue 1
>
> backend sorry
> option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
> server sorry YYY.YYY.YYY.7:1234 check inter 10s fall 5 rise 5
>
> When "be_sess_rate(main) gt 6" gets true, all sites stop working - I get
> the sorry page.
That's expected. Since all your traffic converges there, and it's the only
parameter you're monitoring from all frontends, it is normal that above a
given rate every frontend is affected.
> I don't want to split backends for each site because then I lose the
> control over all traffic; setting them to 1 request per second would
> limit each site too much and when all would work, they would flood the
> backend.
Once again, you don't have to limit in requests/s. You should just limit
per concurrent requests. The limit on requests/s was introduced for a few
reasons :
- it was easy since the value was already measured for stats
- it could be used by people experiencing heavy DDoS as a
temporary workaround
- it could be used by people who know their application is broken
to prevent it from reaching a known breaking point
But I don't know anyone using it right now. The limit on the frontend
session rate is useful for hosting companies though, as it allows them
to enforce an SLA per hosted application.
> I can not use fe_sess_rate to limit the traffic to frontend, because the
> way it works, when we get a lot of traffic to one site, no user will
> actually come through it. Or is there a way to redirect most users to
> sorry servers but still let some users to the site even when the
> connection rate to frontend is higher than configured fe_sess_rate?
Well you could use cookies to know if a user has a sesion on the site or
not, but it's still not the right solution.
> Can someout with more experience than me advice what would be the best
> way to handle this?
>
> I would like to limit number of current users on real servers and amount
> of new connections that can be created per some time unit.
This is the point I cannot agree with. I think you need to limit the
amount of concurrent connections on your servers. Otherwise, your
application is deadly broken, but that's not how you introduced it
first :-)
> And also to
> have sites as available as possible, meaning that when one gets a lot of
> traffic, others would still work and use some limited amount of
> resources, while the site with a lot of traffic gets what is left.
This is precisely where the splitting of the backend helps. Assuming
that your servers can handle, say, 200 concurrent connections, you
split your backends either per application or per application group
(you might want to guarantee a quality of service for a group of
applications). Then you adjust each server's maxconn so that the
total does not exceed the real server's maxconn (=MaxClients on
apache). That way, even if an application takes a lot of resources,
it will not saturate the server. Also, just enable health-checks
in one backend and configure the other ones to track the first one,
it will greatly reduce the number of health-checks sent to your
servers.
Regards,
Willy
