Hi Amol,
For example, this one:
# Shut the new connection as long as the client has already 40 opened
tcp-request connection reject if { src_conn_cur ge 40 }
Should be written
# Shut the new connection as long as the client has already 40 opened
tcp-request connection reject if { sc0_conn_cur ge 40 }
Baptiste
On Mon, Aug 17, 2015 at 4:53 AM, Amol <[email protected]> wrote:
> Hi Baptiste,
> I tried to read about SC0 and SRC, but i am not quite sure what i would gain
> by changing SRC to SCO for the acl paramters? did u have some example to
> explain?
>
> Thanks
>
> ________________________________
> From: Amol <[email protected]>
> To: Baptiste <[email protected]>
> Cc: HAproxy Mailing Lists <[email protected]>
> Sent: Friday, August 14, 2015 2:06 PM
>
> Subject: Re: Regarding using HAproxy for rate limiting
>
> Hi Baptiste,
> Yes sorry i might have confused you with some questions but to answer your
> questions
>
> "here, the question is: what kiils your server exactly?
> A high number of queries from a single users or whatever the number of
> users?
> I'm trying to understand what you need..."
> Yes i am trying to protect against high number of requests from a single
> user who can use API's or even mis-configure API's to generate high load.
>
> reposting the configuration
>
> frontend www-https
> bind xx.xx.xx.xx:443 ssl crt xxxx.pem ciphers AES128+EECDH:AES128+EDH
> no-sslv3 no-tls-tickets
>
> # Table definition
> stick-table type ip size 100k expire 30s store
> gpc0,conn_cur,conn_rate(3s),http_req_rate(10s),http_err_rate(10s)
>
> # Allow clean known IPs to bypass the filter
> tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
>
> # this is sending data defined in the stick-table and storing it the
> stick-table since by default nothing is restored in it
> tcp-request connection track-sc0 src
>
> # Shut the new connection as long as the client has already 40 opened
> tcp-request connection reject if { src_conn_cur ge 40 }
>
> # if someone has more than 40 connections in over a period of 3 seconds,
> REJECT
> tcp-request connection reject if { src_conn_rate ge 40 }
>
> # tracking connections that are not rejected from clients that don't have
> 10 connections/don't have 10 connections/3 seconds
> #tcp-request connection reject if { src_get_gpc0 gt 0 }
>
> acl abuse_err src_http_err_rate ge 10
> acl flag_abuser_err src_inc_gpc0 ge 0
> acl abuse src_http_req_rate ge 250
> #acl flag_abuser src_inc_gpc0 ge 0
> #tcp-request content reject if abuse_err flag_abuser_err
> #tcp-request content reject if abuse flag_abuser
>
> use_backend backend_slow_down if abuse flag_abuser
> use_backend backend_slow_down if abuse_err flag_abuser_err
> default_backend www-backend
>
> backend www-backend
> balance leastconn
> cookie BALANCEID insert indirect nocache secure httponly
> option httpchk HEAD /xxx.php HTTP/1.0
> redirect scheme https if !{ ssl_fc }
> server A1 xx.xx.xx.xx:80 cookie A check
> server A2 yy.yy.yy.yy:80 cookie B check
>
> backend backend_slow_down
> timeout tarpit 2s
> errorfile 500 /etc/haproxy/errors/429.http
> http-request tarpit
>
>
> ------
>
> Yes i will check out the difference between SC0 and SRC paramters in config
>
> regarding this .....
>> What i am doing here is that if the http_req_rate > 250 then i want to
>> send
>> them to a another backend which gives them a rate limiting message or if
>> the
>> number of concurrent connections are > 4, then i want to rate limit their
>> usage and allow on 40 connections to come in.
>
> i was trying to make 2 points i guess i should have been more clear...
> So i was saying that based on my config i am trying to achieve 2 things
>
> 1) to rate limit a client with high number of http requests in a certain
> time span (http_req_rate)
> 2) to rate limit a client with high number of concurrent connections in the
> certain time span. (src_conn_cur and src_conn_rate )
>
> Thanks once again for looking into this.
>
>
>
>
>
> ________________________________
> From: Baptiste <[email protected]>
> To: Amol <[email protected]>
> Cc: HAproxy Mailing Lists <[email protected]>
> Sent: Friday, August 14, 2015 1:40 PM
> Subject: Re: Regarding using HAproxy for rate limiting
>
> Hi Amol,
>
> On Fri, Aug 14, 2015 at 4:16 PM, Amol <[email protected]> wrote:
>> Hello,
>> I am been trying to configure my Haproxy for rate limiting our customer
>> usage, and wanted to know/understand some of my options
>> what i am trying to achieve is to throttle any clients requests/api calls
>> that can take lead to high load and can kill my servers.
>
> here, the question is: what kiils your server exactly?
> A high number of queries from a single users or whatever the number of
> users?
> I'm trying to understand what you need...
>
>
>> First of all here is my configuration i have so far from reading a few
>> articles
>>
>> frontend www-https
>> bind xx.xx.xx.xx:443 ssl crt xxxx.pem ciphers AES128+EECDH:AES128+EDH
>> no-sslv3 no-tls-tickets
>>
>> # Table definition
>> stick-table type ip size 100k expire 30s store
>> gpc0,conn_cur,conn_rate(3s),http_req_rate(10s),http_err_rate(10s)
>> # Allow clean known IPs to bypass the filter
>> tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
>> # this is sending data defined in the stick-table and storing it the
>> stick-table since by default nothing is restored in it
>> tcp-request connection track-sc0 src
>> # Shut the new connection as long as the client has already 10 opened
>> tcp-request connection reject if { src_conn_cur ge 40 }
>> # if someone has more than 100 connections in over a period of 3
>> seconds,
>> REJECT
>> tcp-request connection reject if { src_conn_rate ge 40 }
>> # tracking connections that are not rejected from clients that don't
>> have
>> 10 connections/don't have 10 connections/3 seconds
>> #tcp-request connection reject if { src_get_gpc0 gt 0 }
>>
>> acl abuse_err src_http_err_rate ge 10
>> acl flag_abuser_err src_inc_gpc0 ge 0
>> acl abuse src_http_req_rate ge 250
>> #acl flag_abuser src_inc_gpc0 ge 0
>> #tcp-request content reject if abuse_err flag_abuser_err
>> #tcp-request content reject if abuse flag_abuser
>>
>> use_backend backend_slow_down if abuse
>> #use_backend backend_slow_down if flag_abuser
>> use_backend backend_slow_down if abuse_err flag_abuser_err
>> default_backend www-backend
>>
>> backend www-backend
>> balance leastconn
>> cookie BALANCEID insert indirect nocache secure httponly
>> option httpchk HEAD /xxx.php HTTP/1.0
>> redirect scheme https if !{ ssl_fc }
>> server A1 xx.xx.xx.xx:80 cookie A check
>> server A2 yy.yy.yy.yy:80 cookie B check
>>
>> backend backend_slow_down
>> timeout tarpit 2s
>> errorfile 500 /etc/haproxy/errors/429.http
>> http-request tarpit
>
> you should use the sc0_conn_* functions instead of src_conn_* since
> you're tracking over sc0.
> Also, please repost your configuration with comments updated. For now,
> some comments doesn't match the statement you configured, which makes
> it hard to follow up.
>
>> What i am doing here is that if the http_req_rate > 250 then i want to
>> send
>> them to a another backend which gives them a rate limiting message or if
>> the
>> number of concurrent connections are > 4, then i want to rate limit their
>> usage and allow on 40 connections to come in.
>
> Please be more accurate on the context.
> Furthermore, you mix rate-limiting and concurrent connections for the
> same purpose in your sentence and I'm really confused about the real
> goal you want to achieve.
>
>
>> Please feel free to critique my config. Now on to questions,
>>
>> 1) is rate limiting based on IP a good way to do this or has anyone tried
>> of
>> other ways?
>
> The closest to the application layer the best.
> If you have a cookie or whatever header we can use to perform rate
> limiting, then it would be much better than source IP.
>
>> 2) Am i missing anything critical in the configuration?
>
> no idea as long as I still don't know what your primary goal was.
>
>> 3) when does the src_inc_gpc0 counter really increment? does it increment
>> for every subsequent request from the client in the given timeframe, i
>> have
>> seen it goes from 0 to 6 during my test but wasn't sure about it
>
> Each event may update a counter, such as a new connection or a new
> HTTP request coming in.
>
>
>
>
>> 4) can i not rate limit by just adding the maxconn to the server in the
>> backend or will that throttle everyone instead of the rogue IP...
>
>
> This will prevent your server from running too many request in
> parallel and then being overloaded.
> You can mix both technics. server's maxconn to protect servers against
> a huge load generated by many clients running 1 request + the
> configuration you setup above to prevent a single user to generate too
> many request and taking too much connections allowed by the maxconn.
>
>
> Baptiste
>
>
>>
>>
>>
>
>
>
>
