Re: Can you block this?
On Fri, Feb 24, 2023 at 10:18:14AM -0700, Bryan Arenal wrote:
> And would this work to reject any request that has the
> 'X-Forwarded-For' header?
>
> acl is-forwarded hdr_sub(x-forwarded-for)
> http-request reject if is-forwarded
No, not like this, as you're searching for sub-strings in this header
name but provide no substring. Instead just use the "found" match
method, which only searches for the element:
acl is-forwarded hdr(x-forwarded-for) -m found
http-request reject if is-forwarded
or:
http-request reject if { hdr(x-forwarded-for) -m found }
Note that I tend to find it more convenient to use anonymous ACLs like
above for simple definitions, especially when you're dealing with attacks
that tend to quickly require a lot of patterns, and when ACL names can
quickly become misleading. However declaring ACLs by names remains much
more convenient when you're starting to combine them. I think that
"is-forwarded" is sufficiently self-explanatory and definitely satisfies
this use case, but it was just to give an example.
> How resource intensive do you think this would this be?
It's very light. Just to give you an example, on my laptop, with one
single core assigned to haproxy, a config matching this rule achieves
133000 connections per second on a single core. And when the rule does
not match, just searching for it lowers the perf from 198k to 192k RPS
per core, or 158 nanoseconds of CPU for the whole rule evaluation, a
part of which is amortized if several rules are evaluated. 10 of them
only consume 832 ns here so the cost of starting evaluation is 75ns then
83ns per rule. It's reasonable to use a handful of such rules to fight
a DDoS if you need. The most important is to provide the least possible
information to the attacker about what you're detecting and how you
proceed (e.g. silent-drop and tarpit are great for this).
Willy
Re: Can you block this?
On Fri, Feb 24, 2023 at 12:33 AM Willy Tarreau wrote: > > On Fri, Feb 24, 2023 at 05:39:13AM +, Robin H. Johnson wrote: > > On Thu, Feb 23, 2023 at 06:48:14PM -0700, Bryan Arenal wrote: > > > Hi there, > > > > > > I'm seeing some traffic from what appears to be bad actors and am > > > wanting to block them. I see this in the existing config but being > > > new to haproxy, it doesn't seem like it's configured correctly but I'm > > > not sure: > > > > > > frontend main > > > bind :80 > > > acl bad_ip src > > > acl bad_ip_proxy hdr_ip(X-Forwarded-For) > > Off the top of my head, this should probably be: > > acl bad_ip src -f /etc/haproxy/blocklist.lst > > acl bad_ip_proxy hdr_ip(X-Forwarded-For) -f /etc/haproxy/blocklist.lst > > > > > tcp-request connection reject if bad_ip || bad_ip_proxy > > I'm not sure offhand about the processing order for the header case. > > > > You might need BOTH: > > tcp-request connection reject if bad_ip || bad_ip_proxy > > http-request connection reject if bad_ip || bad_ip_proxy > > In fact the tcp-request rule will not consider the bad_ip_proxy ACL since > it relies on header extraction that is not possible at this step. Indeed, > You'd need to deal with this using the http-request rule. I would also > suggest that if the IP is blocked at the TCP level then it's not needed > to evaluate it again at the HTTP layer. This would give: > > tcp-request connection reject if bad_ip > http-request reject if bad_ip_proxy > > > Depending on the scale of the traffic, the one problem you'll have here is > > that HAProxy still has to process the problematic requests. > > In fact it depends. For example if it's a POST with a heavy body, a "deny" > rule will return a 403 and will drain the rest of the body in order to try > to recover the connection, but a "reject" rule will instantly break the > connection, which instantly stops any processing. > > > In that case I suggest writing a feedback loop that adds the bad IP to an > > ipset > > set, to block the traffic before it gets to HAProxy, for some period of > > time. > > The trigger for the loop can either be a tail on the logfile, or using some > > variation of the set* functionality (set-acl, set-map, set-mark) and > > exporting > > the data to ipset. > > I used to proceed like this a very long time ago, and found that it was > not always convenient due to the lack of observability (i.e. "is that rule > still needed or has the attack stopped?"). Given the performance of most > machines nowadays, I *tend* to prefer only blocking using the ACLs in TCP > rules (that's still very fast). However you're right, sometimes it can be > required to block IPs at the packet level, particularly for multi-protocol > attacks that span over many ports. > > By the way, if attacks come from a small to medium sized set of clients, > the "tarpit" and "silent-drop" actions can be more effective because they > consume resources on the client by keeping the connection open. But *do not* > do that if there is a firewall or NAT box between the net and your LB or > it might not find this funny. For silent-drop there's a new "rst-ttl" > option that allows to send a reset with a limited TTL that only the > firewall/NAT box will see. This one might be considered as an option. Thank you both VERY much for your help -- I really appreciate it. That makes total sense that you'd have to use http-request rather than tcp-request since you've got to evaluate the headers. And would this work to reject any request that has the 'X-Forwarded-For' header? acl is-forwarded hdr_sub(x-forwarded-for) http-request reject if is-forwarded How resource intensive do you think this would this be? Thanks again!
Re: Can you block this?
On Fri, Feb 24, 2023 at 05:39:13AM +, Robin H. Johnson wrote: > On Thu, Feb 23, 2023 at 06:48:14PM -0700, Bryan Arenal wrote: > > Hi there, > > > > I'm seeing some traffic from what appears to be bad actors and am > > wanting to block them. I see this in the existing config but being > > new to haproxy, it doesn't seem like it's configured correctly but I'm > > not sure: > > > > frontend main > > bind :80 > > acl bad_ip src > > acl bad_ip_proxy hdr_ip(X-Forwarded-For) > Off the top of my head, this should probably be: > acl bad_ip src -f /etc/haproxy/blocklist.lst > acl bad_ip_proxy hdr_ip(X-Forwarded-For) -f /etc/haproxy/blocklist.lst > > > tcp-request connection reject if bad_ip || bad_ip_proxy > I'm not sure offhand about the processing order for the header case. > > You might need BOTH: > tcp-request connection reject if bad_ip || bad_ip_proxy > http-request connection reject if bad_ip || bad_ip_proxy In fact the tcp-request rule will not consider the bad_ip_proxy ACL since it relies on header extraction that is not possible at this step. Indeed, You'd need to deal with this using the http-request rule. I would also suggest that if the IP is blocked at the TCP level then it's not needed to evaluate it again at the HTTP layer. This would give: tcp-request connection reject if bad_ip http-request reject if bad_ip_proxy > Depending on the scale of the traffic, the one problem you'll have here is > that HAProxy still has to process the problematic requests. In fact it depends. For example if it's a POST with a heavy body, a "deny" rule will return a 403 and will drain the rest of the body in order to try to recover the connection, but a "reject" rule will instantly break the connection, which instantly stops any processing. > In that case I suggest writing a feedback loop that adds the bad IP to an > ipset > set, to block the traffic before it gets to HAProxy, for some period of time. > The trigger for the loop can either be a tail on the logfile, or using some > variation of the set* functionality (set-acl, set-map, set-mark) and exporting > the data to ipset. I used to proceed like this a very long time ago, and found that it was not always convenient due to the lack of observability (i.e. "is that rule still needed or has the attack stopped?"). Given the performance of most machines nowadays, I *tend* to prefer only blocking using the ACLs in TCP rules (that's still very fast). However you're right, sometimes it can be required to block IPs at the packet level, particularly for multi-protocol attacks that span over many ports. By the way, if attacks come from a small to medium sized set of clients, the "tarpit" and "silent-drop" actions can be more effective because they consume resources on the client by keeping the connection open. But *do not* do that if there is a firewall or NAT box between the net and your LB or it might not find this funny. For silent-drop there's a new "rst-ttl" option that allows to send a reset with a limited TTL that only the firewall/NAT box will see. This one might be considered as an option. Willy
Re: Can you block this?
On Thu, Feb 23, 2023 at 06:48:14PM -0700, Bryan Arenal wrote: > Hi there, > > I’m seeing some traffic from what appears to be bad actors and am > wanting to block them. I see this in the existing config but being > new to haproxy, it doesn’t seem like it’s configured correctly but I’m > not sure: > > frontend main > bind :80 > acl bad_ip src > acl bad_ip_proxy hdr_ip(X-Forwarded-For) Off the top of my head, this should probably be: acl bad_ip src -f /etc/haproxy/blocklist.lst acl bad_ip_proxy hdr_ip(X-Forwarded-For) -f /etc/haproxy/blocklist.lst > tcp-request connection reject if bad_ip || bad_ip_proxy I'm not sure offhand about the processing order for the header case. You might need BOTH: tcp-request connection reject if bad_ip || bad_ip_proxy http-request connection reject if bad_ip || bad_ip_proxy Depending on the scale of the traffic, the one problem you'll have here is that HAProxy still has to process the problematic requests. In that case I suggest writing a feedback loop that adds the bad IP to an ipset set, to block the traffic before it gets to HAProxy, for some period of time. The trigger for the loop can either be a tail on the logfile, or using some variation of the set* functionality (set-acl, set-map, set-mark) and exporting the data to ipset. -- Robin Hugh Johnson Pronouns : They/he E-Mail : robb...@orbis-terrarum.net signature.asc Description: PGP signature
