Re: tcp-request content track-sc2 with if statement doesn't work?
On Sat, Sep 6, 2014 at 9:16 PM, PiBa-NL piba.nl@gmail.com wrote: Hi list, Inspired by a blog about wordpress bruteforce protection [0] , i'm trying to use this same kind of method in a frontend/backend configuration. I did change the method from POST to GET, for easier testing, but that doesn't matter for retrieving the gpc counter, does it? So i was trying to use this: tcp-request content track-sc1 base32+src if METH_GET login It however doesn't seem to work using HAProxy 1.5.3, the acl containing sc1_get_gpc0 gt 0 never seems to get the correct gpc0 value, even though i have examined the stick-table and the gpc0 value there is increasing. If i change it to the following it starts working: tcp-request content track-sc1 base32+src Even though the use_backend in both cases checks those first criteria: acl flagged_as_abusersc1_get_gpc0 gt 0 use_backendpb3_453_http if METH_GET wp_login flagged_as_abuser Am i doing something wrong, is the blog outdated, or was a bug introduced somewhere? If more information perhaps -vv or full config is needed let me know, thanks for any reply. p.s. did anyone get my other emails a while back? [1] Kind regards, PiBa-NL [0] http://blog.haproxy.com/2013/04/26/wordpress-cms-brute-force-protection-with-haproxy/ [1] http://marc.info/?l=haproxym=140821298806125w=2 Hi, Plese let us know if you have the following configuration lines (or equivalent), before your tracking rule: tcp-request inspect-delay 10s tcp-request accept if HTTP Baptiste
Re: tcp-request content track-sc2 with if statement doesn't work?
On Sun, Sep 7, 2014 at 2:55 PM, PiBa-NL piba.nl@gmail.com wrote: Hi Baptiste, Thanks that fixes my issue indeed with the following: tcp-request inspect-delay 10s tcp-request content track-sc1 base32+src if METH_GET wp_login tcp-request content accept if HTTP I didn't think about inspect-delay because both frontend and backend are using 'mode http', and i only used to use inspect-delay with frontends using tcp mode. Though maybe the 'tcp-request' should have given my that hint. The 'accept' must be below the 'track-sc1' to make it work. Could you perhaps also add this to the blog article, or should i post a comment under it for other people to not fall into the same mistake? Thanks, PiBa-NL Baptiste schreef op 7-9-2014 11:38: On Sat, Sep 6, 2014 at 9:16 PM, PiBa-NL piba.nl@gmail.com wrote: Hi list, Inspired by a blog about wordpress bruteforce protection [0] , i'm trying to use this same kind of method in a frontend/backend configuration. I did change the method from POST to GET, for easier testing, but that doesn't matter for retrieving the gpc counter, does it? So i was trying to use this: tcp-request content track-sc1 base32+src if METH_GET login It however doesn't seem to work using HAProxy 1.5.3, the acl containing sc1_get_gpc0 gt 0 never seems to get the correct gpc0 value, even though i have examined the stick-table and the gpc0 value there is increasing. If i change it to the following it starts working: tcp-request content track-sc1 base32+src Even though the use_backend in both cases checks those first criteria: acl flagged_as_abusersc1_get_gpc0 gt 0 use_backendpb3_453_http if METH_GET wp_login flagged_as_abuser Am i doing something wrong, is the blog outdated, or was a bug introduced somewhere? If more information perhaps -vv or full config is needed let me know, thanks for any reply. p.s. did anyone get my other emails a while back? [1] Kind regards, PiBa-NL [0] http://blog.haproxy.com/2013/04/26/wordpress-cms-brute-force-protection-with-haproxy/ [1] http://marc.info/?l=haproxym=140821298806125w=2 Hi, Plese let us know if you have the following configuration lines (or equivalent), before your tracking rule: tcp-request inspect-delay 10s tcp-request accept if HTTP Baptiste Hi, Article updated. Baptiste
Re: tcp-request content track-sc2 with if statement doesn't work?
Baptiste schreef op 7-9-2014 17:13: On Sun, Sep 7, 2014 at 2:55 PM, PiBa-NL piba.nl@gmail.com wrote: Hi Baptiste, Thanks that fixes my issue indeed with the following: tcp-request inspect-delay 10s tcp-request content track-sc1 base32+src if METH_GET wp_login tcp-request content accept if HTTP I didn't think about inspect-delay because both frontend and backend are using 'mode http', and i only used to use inspect-delay with frontends using tcp mode. Though maybe the 'tcp-request' should have given my that hint. The 'accept' must be below the 'track-sc1' to make it work. Could you perhaps also add this to the blog article, or should i post a comment under it for other people to not fall into the same mistake? Thanks, PiBa-NL Baptiste schreef op 7-9-2014 11:38: On Sat, Sep 6, 2014 at 9:16 PM, PiBa-NL piba.nl@gmail.com wrote: Hi list, Inspired by a blog about wordpress bruteforce protection [0] , i'm trying to use this same kind of method in a frontend/backend configuration. I did change the method from POST to GET, for easier testing, but that doesn't matter for retrieving the gpc counter, does it? So i was trying to use this: tcp-request content track-sc1 base32+src if METH_GET login It however doesn't seem to work using HAProxy 1.5.3, the acl containing sc1_get_gpc0 gt 0 never seems to get the correct gpc0 value, even though i have examined the stick-table and the gpc0 value there is increasing. If i change it to the following it starts working: tcp-request content track-sc1 base32+src Even though the use_backend in both cases checks those first criteria: acl flagged_as_abusersc1_get_gpc0 gt 0 use_backendpb3_453_http if METH_GET wp_login flagged_as_abuser Am i doing something wrong, is the blog outdated, or was a bug introduced somewhere? If more information perhaps -vv or full config is needed let me know, thanks for any reply. p.s. did anyone get my other emails a while back? [1] Kind regards, PiBa-NL [0] http://blog.haproxy.com/2013/04/26/wordpress-cms-brute-force-protection-with-haproxy/ [1] http://marc.info/?l=haproxym=140821298806125w=2 Hi, Plese let us know if you have the following configuration lines (or equivalent), before your tracking rule: tcp-request inspect-delay 10s tcp-request accept if HTTP Baptiste Hi, Article updated. Baptiste Hi Baptiste, Thanks, however there are now 2 issues with that. - The 'accept' must be below the 'track-sc1' to make it work. (at least in my tests..) - Syntax error missing 'content' keyword in: tcp-request content accept if HTTP In the backend i didn't seem to need the inspect-delay, probably because the frontend has already filled buffers because it is in 'http' mode. Thanks, PiBa-NL
tcp-request content track-sc2 with if statement doesn't work?
Hi list, Inspired by a blog about wordpress bruteforce protection [0] , i'm trying to use this same kind of method in a frontend/backend configuration. I did change the method from POST to GET, for easier testing, but that doesn't matter for retrieving the gpc counter, does it? So i was trying to use this: tcp-request content track-sc1 base32+src if METH_GET login It however doesn't seem to work using HAProxy 1.5.3, the acl containing sc1_get_gpc0 gt 0 never seems to get the correct gpc0 value, even though i have examined the stick-table and the gpc0 value there is increasing. If i change it to the following it starts working: tcp-request content track-sc1 base32+src Even though the use_backend in both cases checks those first criteria: acl flagged_as_abusersc1_get_gpc0 gt 0 use_backendpb3_453_http if METH_GET wp_login flagged_as_abuser Am i doing something wrong, is the blog outdated, or was a bug introduced somewhere? If more information perhaps -vv or full config is needed let me know, thanks for any reply. p.s. did anyone get my other emails a while back? [1] Kind regards, PiBa-NL [0] http://blog.haproxy.com/2013/04/26/wordpress-cms-brute-force-protection-with-haproxy/ [1] http://marc.info/?l=haproxym=140821298806125w=2
Re: tcp-request content track
It would be easier to help you if you share your configuration! Baptiste On Wed, Mar 12, 2014 at 1:36 AM, Patrick Hemmer hapr...@stormcloud9.net wrote: 2 related questions: I'm trying to find a way to concat multiple samples to use in a stick table. Basically in my frontend I pattern match on the request path to determine which backend to send a request to. The client requests also have a client ID header. I want to rate limit based on a combination of this pattern that matched, and the client ID. Currently the way I do this is an http-request set-header rule that adds a new header combining a unique ID for the pattern that matched along with the client-ID header. Then in the backend I have a tcp-requst content track-sc2 on that header. This works, but I'm wondering if there's a better way. Secondly, the above works, but when I do a show table mybackend on the stats socket, both the conn_cur and use counters never decrease. They seem to be acting as the total number of requests, not the number of active connections. Is this a bug, or am I misunderstanding something? -Patrick
Re: tcp-request content track
Created a new config as an example. My existing config is huge, and hard
to read (generated programtically).
In regards to the bug, it appears it was a bug. I was using 1.5-dev19.
After upgrading to 1.5-dev22 it started behaving as expected.
Below is the config I'm using to accomplish what I want. As mentioned,
I'm basically rate limiting on a combination of the X-Client-Id header
and the matching URL. And as you can see, it's quite ugly and complex to
accomplish it :-(
For example, the same X-Client-Id should be able to hit /foo/bar 3 times
every 15 seconds, with only 1 open connection (the rules). It
should be able to hit /asdf at 5 times every 15 seconds with 3 open
connections (the rules).
global
log 127.0.0.1:514 local1 debug
maxconn 4096
daemon
stats socket /tmp/haproxy.sock level admin
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 200
timeout client 6
timeout server 17
option clitcpka
option srvtcpka
option abortonclose
stats enable
stats uri /haproxy/stats
frontend f1
bind *:1500
option httpclose
acl internal dst 127.0.0.2
acl have_request_id req.fhdr(X-Request-Id) -m found
http-request set-header X-API-URL %[path] if !internal
http-request add-header X-Request-Timestamp %Ts.%ms
http-request set-header X-Request-Id %[req.fhdr(X-Request-Id)] if
internal have_request_id
http-request set-header X-Request-Id %{+X}o%pid-%rt if !internal ||
!have_request_id
http-request set-header X-API-Host i-12345678
http-response set-header X-API-Host i-12345678
unique-id-format %{+X}o%pid-%rt
log-format %ci:%cp\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\
%ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %U/%B\ %ST\ %tsc\ %ID\ +\ %r
acl rewrite-found req.hdr(X-Rewrite-ID,1) -m found
acl _path path_reg ^/foo/([^\ ?]*)$
acl _method method GET
http-request set-header X-Rewrite-Id if !rewrite-found
_path _method
acl -rewrite req.hdr(X-Rewrite-Id) -m str
http-request set-header X-Limit-Id %[req.hdr(X-Client-Id)] if
-rewrite
use_backend b1 if -rewrite
reqrep ^(GET)\ /foo/([^\ ?]*)([\ ?].*|$) \1\ /echo/bar/\2\3 if
-rewrite
acl _path path_reg ^/([^\ ?]*)$
acl _method method GET
http-request set-header X-Rewrite-Id if !rewrite-found
_path _method
acl -rewrite req.hdr(X-Rewrite-Id) -m str
http-request set-header X-Limit-Id %[req.hdr(X-Client-Id)] if
-rewrite
use_backend b1 if -rewrite
reqrep ^(GET)\ /([^\ ?]*)([\ ?].*|$) \1\ /echo/\2\3 if -rewrite
backend b1
stick-table type string len 12 size 1000 expire 1h store
http_req_rate(15000),conn_cur
tcp-request content track-sc2 req.hdr(X-Limit-ID)
acl -rewrite req.hdr(X-Rewrite-Id) -m str
acl _req_rate sc2_http_req_rate gt 3
acl _conn_cur sc2_conn_cur gt 1
tcp-request content reject if -rewrite _req_rate
tcp-request content reject if -rewrite _conn_cur
acl -rewrite req.hdr(X-Rewrite-Id) -m str
acl _req_rate sc2_http_req_rate gt 5
acl _conn_cur sc2_conn_cur gt 3
tcp-request content reject if -rewrite _req_rate
tcp-request content reject if -rewrite _conn_cur
server s1 127.0.0.1:2700
server s2 127.0.0.1:2701
server s3 127.0.0.1:2702
-Patrick
*From: *Baptiste bed...@gmail.com
*Sent: * 2014-03-12 06:26:32 E
*To: *Patrick Hemmer hapr...@stormcloud9.net
*CC: *haproxy@formilux.org haproxy@formilux.org
*Subject: *Re: tcp-request content track
It would be easier to help you if you share your configuration!
Baptiste
On Wed, Mar 12, 2014 at 1:36 AM, Patrick Hemmer hapr...@stormcloud9.net
wrote:
2 related questions:
I'm trying to find a way to concat multiple samples to use in a stick table.
Basically in my frontend I pattern match on the request path to determine
which backend to send a request to. The client requests also have a client
ID header. I want to rate limit based on a combination of this pattern that
matched, and the client ID. Currently the way I do this is an http-request
set-header rule that adds a new header combining a unique ID for the
pattern that matched along with the client-ID header. Then in the backend I
have a tcp-requst content track-sc2 on that header. This works, but I'm
wondering if there's a better way.
Secondly, the above works, but when I do a show table mybackend on the
stats socket, both the conn_cur and use counters never decrease. They
seem to be acting as the total number of requests, not the number of active
connections. Is this a bug, or am I misunderstanding something?
-Patrick
tcp-request content track
2 related questions: I'm trying to find a way to concat multiple samples to use in a stick table. Basically in my frontend I pattern match on the request path to determine which backend to send a request to. The client requests also have a client ID header. I want to rate limit based on a combination of this pattern that matched, and the client ID. Currently the way I do this is an http-request set-header rule that adds a new header combining a unique ID for the pattern that matched along with the client-ID header. Then in the backend I have a tcp-requst content track-sc2 on that header. This works, but I'm wondering if there's a better way. Secondly, the above works, but when I do a show table mybackend on the stats socket, both the conn_cur and use counters never decrease. They seem to be acting as the total number of requests, not the number of active connections. Is this a bug, or am I misunderstanding something? -Patrick
Re: help with tcp-request content track-sc1
On Sat, Aug 27, 2011 at 5:26 AM, Willy Tarreau w...@1wt.eu wrote: Hi David, On Thu, Aug 25, 2011 at 12:28:43PM -0700, David Birdsong wrote: I've poured over 1.5 docs, and I'm pretty sure this should be possible. Is there a way to extract a header string from an http header and track that in a stick-table of type 'string'? If so, what is the syntax, where does the extraction take place? Right now it's not implemented, as the track-sc1 statement is only available at the TCP stage. I'm clearly thinking about having it before 1.5 is released, because at many places it's much more important than the source IP itself. Ok, thanks for the clarification. Is there a way to cast a header as an ip and track-sc1? In our setup we're terminating SSL in front of haproxy and so only the XFF header has the client ip address. Also, is there any way to concatenate two headers into one string value to track and store? If not, I can concatenate them upstream (close to client), but it'd be nice to keep the logic local to haproxy's config. No this is not possible. We need the pattern extraction feature which has not even started yet for this :-( Regards, Willy
Re: help with tcp-request content track-sc1
On Mon, Aug 29, 2011 at 01:40:53PM -0700, David Birdsong wrote: On Mon, Aug 29, 2011 at 1:36 PM, Willy Tarreau w...@1wt.eu wrote: On Mon, Aug 29, 2011 at 12:22:18PM -0700, David Birdsong wrote: On Sat, Aug 27, 2011 at 5:26 AM, Willy Tarreau w...@1wt.eu wrote: Hi David, On Thu, Aug 25, 2011 at 12:28:43PM -0700, David Birdsong wrote: I've poured over 1.5 docs, and I'm pretty sure this should be possible. Is there a way to extract a header string from an http header and track that in a stick-table of type 'string'? If so, what is the syntax, where does the extraction take place? Right now it's not implemented, as the track-sc1 statement is only available at the TCP stage. I'm clearly thinking about having it before 1.5 is released, because at many places it's much more important than the source IP itself. Ok, thanks for the clarification. Is there a way to cast a header as an ip and track-sc1? In our setup we're terminating SSL in front of haproxy and so only the XFF header has the client ip address. I understand the issue, it's the same everyone is facing when trying to do the same thing unfortunately :-( If you use a patched stunnel version which supports the PROXY protocol, then you can have the client's IP available as soon as tcp-request content rules are processed. Those rules support track-sc1 so you can do what you want at this level. It requires a patch on stunnel however, but it should not be an issue since you appear to be using the XFF We're actually terminating half of our ssl traffic with nginx and the other half with Amazon's elb offering with plans of moving all ssl termination to Amazon in the next week or so. The PROXY protocol should be ported to Amazon's ELB then ;-) Cheers, Willy
Re: help with tcp-request content track-sc1
On Mon, Aug 29, 2011 at 1:46 PM, Willy Tarreau w...@1wt.eu wrote: On Mon, Aug 29, 2011 at 01:40:53PM -0700, David Birdsong wrote: On Mon, Aug 29, 2011 at 1:36 PM, Willy Tarreau w...@1wt.eu wrote: On Mon, Aug 29, 2011 at 12:22:18PM -0700, David Birdsong wrote: On Sat, Aug 27, 2011 at 5:26 AM, Willy Tarreau w...@1wt.eu wrote: Hi David, On Thu, Aug 25, 2011 at 12:28:43PM -0700, David Birdsong wrote: I've poured over 1.5 docs, and I'm pretty sure this should be possible. Is there a way to extract a header string from an http header and track that in a stick-table of type 'string'? If so, what is the syntax, where does the extraction take place? Right now it's not implemented, as the track-sc1 statement is only available at the TCP stage. I'm clearly thinking about having it before 1.5 is released, because at many places it's much more important than the source IP itself. Ok, thanks for the clarification. Is there a way to cast a header as an ip and track-sc1? In our setup we're terminating SSL in front of haproxy and so only the XFF header has the client ip address. I understand the issue, it's the same everyone is facing when trying to do the same thing unfortunately :-( If you use a patched stunnel version which supports the PROXY protocol, then you can have the client's IP available as soon as tcp-request content rules are processed. Those rules support track-sc1 so you can do what you want at this level. It requires a patch on stunnel however, but it should not be an issue since you appear to be using the XFF We're actually terminating half of our ssl traffic with nginx and the other half with Amazon's elb offering with plans of moving all ssl termination to Amazon in the next week or so. The PROXY protocol should be ported to Amazon's ELB then ;-) Agreed, that would be a big help. Anybody know what the ELB's are? Some have speculated their just using Netscalers, but they toss around the word 'instance' in a way that makes me wonder if they're just using ec2 instances. Cheers, Willy
Re: help with tcp-request content track-sc1
Hi David, On Thu, Aug 25, 2011 at 12:28:43PM -0700, David Birdsong wrote: I've poured over 1.5 docs, and I'm pretty sure this should be possible. Is there a way to extract a header string from an http header and track that in a stick-table of type 'string'? If so, what is the syntax, where does the extraction take place? Right now it's not implemented, as the track-sc1 statement is only available at the TCP stage. I'm clearly thinking about having it before 1.5 is released, because at many places it's much more important than the source IP itself. Also, is there any way to concatenate two headers into one string value to track and store? If not, I can concatenate them upstream (close to client), but it'd be nice to keep the logic local to haproxy's config. No this is not possible. We need the pattern extraction feature which has not even started yet for this :-( Regards, Willy
help with tcp-request content track-sc1
I've poured over 1.5 docs, and I'm pretty sure this should be possible. Is there a way to extract a header string from an http header and track that in a stick-table of type 'string'? If so, what is the syntax, where does the extraction take place? Also, is there any way to concatenate two headers into one string value to track and store? If not, I can concatenate them upstream (close to client), but it'd be nice to keep the logic local to haproxy's config. The only examples I see for tcp-request track-sc1 are layer4 and below and yet the documentation alludes to contents being inspected as an option for any action (and not just accept/reject.)
