Re: PF rate limiting options valid for UDP?
On Thu, Jul 20, 2023 at 05:52:07PM +, mabi wrote:
> --- Original Message ---
> On Wednesday, July 19th, 2023 at 10:58 PM, Stuart Henderson
> wrote:
>
> > For rules that pass traffic to your authoritative DNS servers,
> > I don't think you need much longer than the time taken to answer a
> > query. So could be quite a bit less.
>
> Right good point, I will add custom state timeouts for this specific UDP pass
> rule on port 53.
>
> > Usually carp/ospf will enter the state table before the machines start
> > seeing large amounts of packets and stay there, which is what you would
> > normally want. If the state table is full, you have more problem
> > opening new connections that require state to be added than you do
> > maintaining existing ones.
> >
> > fwiw I typically use this on ospf+carp machines, "pass quick proto
> > {carp, ospf} keep state (no-sync) set prio 7"
>
> That's very interesting, I never realized there was a simple priority system
> ready to use in PF without the need of setting up any queues. Probably the
> "set prio 7" option on OSPF+CARP pass rules will juts do the trick and I will
> definitely also implement this.
>
> > DNS server software is written with this type of traffic in mind, and
> > has more information available (from inside the DNS request packet)
> > to make a decision about what to do with it, than is available in a
> > general-purpose packet filter like PF.
> >
> > Also it stores the tracking information in data structures that have
> > been chosen to make sense for this use (and common DNS servers default
> > to masking on common subnet sizes, reducing the amount they have to
> > store compared to tracking the full IP address).
> >
> > http://man.openbsd.org/nsd.conf#rrl
> > https://bind9.readthedocs.io/en/latest/reference.html#response-rate-limiting
> > https://www.knot-dns.cz/docs/2.4/html/reference.html#module-rrl
>
> Too bad I use PowereDNS, it does not seem to offer much parameters related to
> rate-limiting for UDP but for TCP I found at least max-tcp-connections. Maybe
> it's time for a change as Gabor mentions his tests in his reply (thanks
> btw!)...
>
In a typical PowerDNS setup the task of rate limiting is done by dnsdist.
-Otto
Re: PF rate limiting options valid for UDP?
--- Original Message ---
On Wednesday, July 19th, 2023 at 10:58 PM, Stuart Henderson
wrote:
> For rules that pass traffic to your authoritative DNS servers,
> I don't think you need much longer than the time taken to answer a
> query. So could be quite a bit less.
Right good point, I will add custom state timeouts for this specific UDP pass
rule on port 53.
> Usually carp/ospf will enter the state table before the machines start
> seeing large amounts of packets and stay there, which is what you would
> normally want. If the state table is full, you have more problem
> opening new connections that require state to be added than you do
> maintaining existing ones.
>
> fwiw I typically use this on ospf+carp machines, "pass quick proto
> {carp, ospf} keep state (no-sync) set prio 7"
That's very interesting, I never realized there was a simple priority system
ready to use in PF without the need of setting up any queues. Probably the "set
prio 7" option on OSPF+CARP pass rules will juts do the trick and I will
definitely also implement this.
> DNS server software is written with this type of traffic in mind, and
> has more information available (from inside the DNS request packet)
> to make a decision about what to do with it, than is available in a
> general-purpose packet filter like PF.
>
> Also it stores the tracking information in data structures that have
> been chosen to make sense for this use (and common DNS servers default
> to masking on common subnet sizes, reducing the amount they have to
> store compared to tracking the full IP address).
>
> http://man.openbsd.org/nsd.conf#rrl
> https://bind9.readthedocs.io/en/latest/reference.html#response-rate-limiting
> https://www.knot-dns.cz/docs/2.4/html/reference.html#module-rrl
Too bad I use PowerDNS, it does not seem to offer much parameters related to
rate-limiting for UDP but for TCP I found at least max-tcp-connections. Maybe
it's time for a change as Gabor mentions his tests in his reply (thanks btw!)...
Re: PF rate limiting options valid for UDP?
Hi, Are you already using your DNS server's response rate limiting features? Not yet, as I still believe I should stop as much as possible such traffic at the firewall before it even reaches the network behind my firewall. So at the software/daemon/service level it would be my last line of defense. If your hardware is powerful enough (e.g. at least 10Gbps Ethernet and the authoritative DNS server has let us say 32 CPU cores) you could also try fending off the DoS attack simply by using NSD or Knot DNS instead of BIND. According to my measurements, they both outperformed BIND by a factor or 10. If you are interested, you can find all the details in my open access paper: G. Lencse, "Benchmarking Authoritative DNS Servers", /IEEE Access/, vol. 8. pp. 130224-130238, July 2020. https://doi.org/10.1109/ACCESS.2020.3009141 Best regards, Gábor
Re: PF rate limiting options valid for UDP?
On 2023/07/19 19:54, mabi wrote:
> --- Original Message ---
> On Wednesday, July 19th, 2023 at 9:32 PM, Stuart Henderson
> wrote:
>
> > If PF is struggling as it is, there's a good chance it will buckle
> > completely if it has to do source tracking too
>
> That is also something I thought might be the case :|
>
> > Did you already tweak timeouts for the rule passing UDP DNS traffic?
> > Defaults are 60s/30s/60s for udp.first, udp.single and udp.multiple
> > respectively, that is much too high for a very busy DNS server -
> > you can set them on the specific rule itself rather than changing
> > defaults for all rules. For an auth server which is expected to
> > respond quickly they can be cranked way down.
>
> Yes, this at least I did since quite some time now and use the following
> timeout settings:
>
> set timeout udp.first 20
> set timeout udp.multiple 20
> set timeout udp.single 10
>
> Do you think I could go even lower? When I check the PF state entries during
> such a DDoS I see mostly states with the "SINGLE" state.
For rules that pass traffic to your authoritative DNS servers,
I don't think you need much longer than the time taken to answer a
query. So could be quite a bit less.
> > (If that is still too many states, I wonder if your network might
> > actually be happier if you "pass quick proto udp to $server port 53 no
> > state" and "pass quick proto udp from $server port 53 no state" right at
> > the top of the ruleset).
>
> That's actually an excellent idea to bypass PF states and hence consume less
> resources... Next thing to try out. I was also thinking I should use "no
> state" with CARP and OSPF rules in pf.conf so that in case the PF state table
> entries is full it does not prevent such important protocols to function.
> What do you think, would that also work?
Usually carp/ospf will enter the state table before the machines start
seeing large amounts of packets and stay there, which is what you would
normally want. If the state table is full, you have more problem
opening new connections that require state to be added than you do
maintaining existing ones.
fwiw I typically use this on ospf+carp machines, "pass quick proto
{carp, ospf} keep state (no-sync) set prio 7"
> > Are you already using your DNS server's response rate limiting features?
>
> Not yet, as I still believe I should stop as much as possible such traffic at
> the firewall before it even reaches the network behind my firewall. So at the
> software/daemon/service level it would be my last line of defense.
DNS server software is written with this type of traffic in mind, and
has more information available (from inside the DNS request packet)
to make a decision about what to do with it, than is available in a
general-purpose packet filter like PF.
Also it stores the tracking information in data structures that have
been chosen to make sense for this use (and common DNS servers default
to masking on common subnet sizes, reducing the amount they have to
store compared to tracking the full IP address).
http://man.openbsd.org/nsd.conf#rrl
https://bind9.readthedocs.io/en/latest/reference.html#response-rate-limiting
https://www.knot-dns.cz/docs/2.4/html/reference.html#module-rrl
Re: PF rate limiting options valid for UDP?
--- Original Message --- On Wednesday, July 19th, 2023 at 9:32 PM, Stuart Henderson wrote: > If PF is struggling as it is, there's a good chance it will buckle > completely if it has to do source tracking too That is also something I thought might be the case :| > Did you already tweak timeouts for the rule passing UDP DNS traffic? > Defaults are 60s/30s/60s for udp.first, udp.single and udp.multiple > respectively, that is much too high for a very busy DNS server - > you can set them on the specific rule itself rather than changing > defaults for all rules. For an auth server which is expected to > respond quickly they can be cranked way down. Yes, this at least I did since quite some time now and use the following timeout settings: set timeout udp.first 20 set timeout udp.multiple 20 set timeout udp.single 10 Do you think I could go even lower? When I check the PF state entries during such a DDoS I see mostly states with the "SINGLE" state. > (If that is still too many states, I wonder if your network might > actually be happier if you "pass quick proto udp to $server port 53 no > state" and "pass quick proto udp from $server port 53 no state" right at > the top of the ruleset). That's actually an excellent idea to bypass PF states and hence consume less resources... Next thing to try out. I was also thinking I should use "no state" with CARP and OSPF rules in pf.conf so that in case the PF state table entries is full it does not prevent such important protocols to function. What do you think, would that also work? > Are you already using your DNS server's response rate limiting features? Not yet, as I still believe I should stop as much as possible such traffic at the firewall before it even reaches the network behind my firewall. So at the software/daemon/service level it would be my last line of defense.
Re: PF rate limiting options valid for UDP?
On 2023/07/19 19:13, mabi wrote: > --- Original Message --- > On Wednesday, July 19th, 2023 at 12:40 PM, Stuart Henderson > wrote: > > > I don't think you understood what I wrote then - they are the > > opposite of helpful here. > > No, I do understand what you wrote but I should have explained my case > in more details. Behind my OpenBSD firewall I have two authoritative DNS > servers and because of recent DDoS originating from >12k IPs against UDP > port 53 on these two servers the whole network behind the firewall gets > unresponsive or has a high packet loss because there is over 2 million > states in the PF states table during the attack. So in my specific case > I don't care that cloudflare or other external DNS servers can not query > my DNS authoritative servers for a few seconds or minutes but I do care > a lot that my whole rest of my network and servers behind the OpenBSD > firewall stays responsive. It's a trade-off I can totally accept and > welcome. Furthermore when I have so many state entries due to a DDoS on > UDP port 53, CARP breaks as well as the OSPF sessions with my border > routers because it can not communicate properly within the defined > timeouts. If PF is struggling as it is, there's a good chance it will buckle completely if it has to do source tracking too Did you already tweak timeouts for the rule passing UDP DNS traffic? Defaults are 60s/30s/60s for udp.first, udp.single and udp.multiple respectively, that is much too high for a very busy DNS server - you can set them on the specific rule itself rather than changing defaults for all rules. For an auth server which is expected to respond quickly they can be cranked way down. (If that is still too many states, I wonder if your network might actually be happier if you "pass quick proto udp to $server port 53 no state" and "pass quick proto udp from $server port 53 no state" right at the top of the ruleset). Are you already using your DNS server's response rate limiting features?
Re: PF rate limiting options valid for UDP?
--- Original Message --- On Wednesday, July 19th, 2023 at 12:40 PM, Stuart Henderson wrote: > I don't think you understood what I wrote then - they are the > opposite of helpful here. No, I do understand what you wrote but I should have explained my case in more details. Behind my OpenBSD firewall I have two authoritative DNS servers and because of recent DDoS originating from >12k IPs against UDP port 53 on these two servers the whole network behind the firewall gets unresponsive or has a high packet loss because there is over 2 million states in the PF states table during the attack. So in my specific case I don't care that cloudflare or other external DNS servers can not query my DNS authoritative servers for a few seconds or minutes but I do care a lot that my whole rest of my network and servers behind the OpenBSD firewall stays responsive. It's a trade-off I can totally accept and welcome. Furthermore when I have so many state entries due to a DDoS on UDP port 53, CARP breaks as well as the OSPF sessions with my border routers because it can not communicate properly within the defined timeouts.
Re: PF rate limiting options valid for UDP?
On 19/07/2023 13:31, Stuart Henderson wrote: > On 2023-07-19, Kapetanakis Giannis wrote: >> Maybe even better, can it run under relayd (redirect) on top of carp? > That's just rdr-to behind the scenes, no problem with that, though if > you want to do per IP rate limiting alongside load-balancing you might > want "mode source-hash" rather than the default round-robin or one of > the random options. > > (I wouldn't recommend sticky-address, because then you get into more > complex paths inside PF because it has to maintain source-tracking > information). I don't think source tracking is that important in this case scenario. relayd will only have one host, which will be the dnsdist listening on localhost (on each load balancer). dnsdist will have whatever it can support with stickiness/source-tracking. pf rdr-to could also be an option, but then you loose the carp demotion which relayd provides. thanks G
Re: PF rate limiting options valid for UDP?
On 2023-07-19, mabi wrote: > --- Original Message --- > On Tuesday, July 18th, 2023 at 10:59 PM, Stuart Henderson > wrote: > > >> PF's state-tracking options are only for TCP. (Blocking an IP >> based on number of connections from easily spoofed UDP is a good >> way to let third parties prevent your machine from communicating >> with IPs that may well get in the way i.e. trigger a "self DoS"). > > What a pitty, these kind of rate limiting options for UDP would have been > quite useful. I don't think you understood what I wrote then - they are the opposite of helpful here. Say you are running a DNS recursive resolver with such protection; if someone were to send you spoofed high rate packets from the IPs of the root servers, some big gTLD/ccTLD servers, or big DNS hosters (cloudflare or someone), your lookups will be quite broken. Likewise for an authoritative server: send packets with source IPs of some large DNS recursive resolvers and you then won't be sending replies to legitimate requests from those resolvers. The difference with TCP is that someone sending packets needs to be able to see the response to those packets in order to carry out the handshake. That's not needed for UDP where a single packet in one direction is all that's needed.
Re: PF rate limiting options valid for UDP?
On 2023-07-19, Kapetanakis Giannis wrote: > On 18/07/2023 23:59, Stuart Henderson wrote: >> PF's state-tracking options are only for TCP. (Blocking an IP >> based on number of connections from easily spoofed UDP is a good >> way to let third parties prevent your machine from communicating >> with IPs that may well get in the way i.e. trigger a "self DoS"). >> >> You may be interested in looking into L7 methods of mitigating >> problems from high rates of DNS queries - for example dnsdist >> allows a lot of flexibility in this area. > > > dnsdist looks interesting. > > Can it run on top of carp interfaces? Don't think I tried it, but I don't see why not. > Maybe even better, can it run under relayd (redirect) on top of carp? That's just rdr-to behind the scenes, no problem with that, though if you want to do per IP rate limiting alongside load-balancing you might want "mode source-hash" rather than the default round-robin or one of the random options. (I wouldn't recommend sticky-address, because then you get into more complex paths inside PF because it has to maintain source-tracking information).
Re: PF rate limiting options valid for UDP?
--- Original Message --- On Tuesday, July 18th, 2023 at 10:59 PM, Stuart Henderson wrote: > PF's state-tracking options are only for TCP. (Blocking an IP > based on number of connections from easily spoofed UDP is a good > way to let third parties prevent your machine from communicating > with IPs that may well get in the way i.e. trigger a "self DoS"). What a pitty, these kind of rate limiting options for UDP would have been quite useful. > You may be interested in looking into L7 methods of mitigating > problems from high rates of DNS queries - for example dnsdist > allows a lot of flexibility in this area. Thanks for the hint about dnsdist, it looks powerful. Still whenever possible I would rather avoid having an extra piece of software and instead have that traffic controlled more upstream so ideally on the firewall directly.
Re: PF rate limiting options valid for UDP?
On 18/07/2023 23:59, Stuart Henderson wrote: > PF's state-tracking options are only for TCP. (Blocking an IP > based on number of connections from easily spoofed UDP is a good > way to let third parties prevent your machine from communicating > with IPs that may well get in the way i.e. trigger a "self DoS"). > > You may be interested in looking into L7 methods of mitigating > problems from high rates of DNS queries - for example dnsdist > allows a lot of flexibility in this area. dnsdist looks interesting. Can it run on top of carp interfaces? Maybe even better, can it run under relayd (redirect) on top of carp? G
Re: PF rate limiting options valid for UDP?
On 2023-07-18, mabi wrote: > Hello, > > From the following documentation, I am trying to figure out which PF tracking > options are also valid for UDP but unfortunately it is not quite clear to me: > > https://man.openbsd.org/pf.conf.5#Stateful_Tracking_Options > > My goal would be to do add rate limiting options to a PF UDP pass rule in > order to limit DDoS/DoS attacks on port 53. > > Interesting would be especially the "max-src-states" option. Is this option > also valid for UDP? > > Is it also possible to use the "overload" option with UDP in order to add > source IPs into a table of attackers which I will then block? PF's state-tracking options are only for TCP. (Blocking an IP based on number of connections from easily spoofed UDP is a good way to let third parties prevent your machine from communicating with IPs that may well get in the way i.e. trigger a "self DoS"). You may be interested in looking into L7 methods of mitigating problems from high rates of DNS queries - for example dnsdist allows a lot of flexibility in this area.
