Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Friday, 3 April 2020 11:13:17 UTC Steven Miller wrote:
> Essentially, yes.  Some increase in capacity on your side plus RRL will
> certainly keep you safer, but it's no guarantee.
> 
> ...

i saw the question differently:

> On 4/3/2020 5:03 AM, Tessa Plum wrote:
> > So no way to stop reflector attack unless migrating servers to
> > professional IDC?

you can subscribe to a ddos "scrubbing service" which reroutes your inbound 
traffic through a ddos filtering vendor (such as akamai) during attacks. they 
will deliver the non-ddos subset of the traffic to your servers in your own 
data center using a tunnel. so, technically, there is a way to mitigate a 
reflector attack without migrading your servers to a professional IDC.

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Steven Miller]

Adding more servers and going to 10G NICs seems relatively inexpensive 
and that should be helpful for "casual" attacks where you're being used 
as a reflector.  In those attacks, no one's out to attack you: they just 
want you to attack someone else, and don't mind eating all your 
bandwidth/CPU/whatever in order to do that.


Adding more bandwidth without enabling RRL or putting some sort of 
filtering in place will make your facilities more attractive to 
attackers, though.  I'd expect that attackers are passing around lists 
of particularly good sites for reflector attacks, and the fewer controls 
you have, and the more bandwidth you have, the more attractive you are 
for use in an attack -- and therefore the more likely you are to have 
your resources saturated.


I think RRL should be safe to run all the time.  You wouldn't want to 
scramble to enable it during an attack.


I don't know if there are commercial devices I would trust to be helpful 
in these situations, but when I was doing DNS DDoS work, nothing 
commercial was going to scale enough, so I didn't consider them much. :-)


The hard thing about these attacks is that there's always some time when 
local resources aren't enough: when you upgrade to 50Gbit/sec of 
capacity and the next attack is 60Gbit/sec of traffic.  I'd expect some 
correlation between "really high bandwidth attacks" and "attacks that 
are meant to hurt you instead of just use you as a reflector" but that 
correlation won't be perfect.  It's unfortunate that in the DNS attack 
world, for a lot of attacks, all you can do is have massively more 
capacity than you need on a daily basis.


The advantage to moving DNS into a cloud provider is that they have the 
resources to massively, crazily overprovision, to the point where it 
would be hard even for a nation-state to mount a big enough attack to 
take them down.  I'm most familiar with Cloudflare (I have never worked 
there, for the record) but certainly there are other companies worth 
looking at.  However, if you still have your nameservers in the public 
set of NS records for your domains, you'll still be vulnerable.  Some of 
these providers can probably load your zones using you as a shadow 
master: they just do a zone transfer from your DNS infrastructure, then 
serve all the queries from their own systems.


That's my perspective.  Hopefully it's not too out of date.

    -Steve

On 4/3/2020 4:18 AM, Tessa Plum wrote:

Hi Steve

I am so appreciate to get your kind private message, though I would 
like to reply my content to the list.


We are running authoritative name servers only, zone data are for the 
university only.


When the attack happened, the bandwidth watched in our gateway was 
about 20Gbps. That made name servers totally no response. Each name 
server has only 1Gbps interface to internet, so it dies.


We were considering the actions:
1. increase bandwidth to both inbound gateway and vlan for nameservers.
2. upgrade the network interface of nameserver to 10Gbps.
3. run multiple servers as cluster.
4. try to get a commercial device to analyst and stop such kind of 
attack.

5. enable RRL when attack happens.
6. I will try to suggest administrator to run secondary nameservers on 
professional hosting, such as cloudflare, Akamai, AWS route 53 etc.

  (also easyDNS, DNSimple, DNSMadeEasy, NS1 can be considered?)

How do you think of them?

Thank you.

regards
Tessa



___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Lanlan Pan]

TLD can make class identification to serve the most important client, such
as the backend server of isp recursive dns and important public recursive
dns. But there are many long-tail clients which can not reject directly,
pre-build greylist and whitelist.
Compare to TLD, SLD is simpler,  it can reject the history-rarely visited
clients with less critial infrastructure influnence concern.

ISP recursive dns  should make the access control, only serve their
customers.
Public recursive dns  and Root  are global anycast deployed.

To defend amplify attack, there should be more learning  on the open
resolvers.

Steven Miller  于2020年4月3日周五 下午7:23写道：

> Essentially, yes.  Some increase in capacity on your side plus RRL will
> certainly keep you safer, but it's no guarantee.
>
+1

>
> Though to be clear, every few years, someone's going to hit a public DNS
> provider with enough load to cause a problem.  IMHO that'll happen less
> on average, and the mitigation time will be lower on average, and the
> pain level for you will be lower on average (no scrambling for
> resources, the ability to say "yeah, a big chunk of the Internet's down,
> I'll let you know when it's over" :-)) than it will happen if you run
> your own infrastructure.
>
> It's a really unfortunate state of affairs.
>
>  -Steve
>
> On 4/3/2020 5:03 AM, Tessa Plum wrote:
> > So no way to stop reflector attack unless migrating servers to
> > professional IDC?
> >
> > Thanks.
> >
> > Steven Miller wrote:
> >> Adding more servers and going to 10G NICs seems relatively
> >> inexpensive and that should be helpful for "casual" attacks where
> >> you're being used as a reflector.  In those attacks, no one's out to
> >> attack you: they just want you to attack someone else, and don't mind
> >> eating all your bandwidth/CPU/whatever in order to do that.
> >>
> >> Adding more bandwidth without enabling RRL or putting some sort of
> >> filtering in place will make your facilities more attractive to
> >> attackers, though.  I'd expect that attackers are passing around
> >> lists of particularly good sites for reflector attacks, and the fewer
> >> controls you have, and the more bandwidth you have, the more
> >> attractive you are for use in an attack -- and therefore the more
> >> likely you are to have your resources saturated.
> >>
> >> I think RRL should be safe to run all the time.  You wouldn't want to
> >> scramble to enable it during an attack.
> >>
> >> I don't know if there are commercial devices I would trust to be
> >> helpful in these situations, but when I was doing DNS DDoS work,
> >> nothing commercial was going to scale enough, so I didn't consider
> >> them much. :-)
> >>
> >> The hard thing about these attacks is that there's always some time
> >> when local resources aren't enough: when you upgrade to 50Gbit/sec of
> >> capacity and the next attack is 60Gbit/sec of traffic.  I'd expect
> >> some correlation between "really high bandwidth attacks" and "attacks
> >> that are meant to hurt you instead of just use you as a reflector"
> >> but that correlation won't be perfect.  It's unfortunate that in the
> >> DNS attack world, for a lot of attacks, all you can do is have
> >> massively more capacity than you need on a daily basis.
> >>
> >> The advantage to moving DNS into a cloud provider is that they have
> >> the resources to massively, crazily overprovision, to the point where
> >> it would be hard even for a nation-state to mount a big enough attack
> >> to take them down.  I'm most familiar with Cloudflare (I have never
> >> worked there, for the record) but certainly there are other companies
> >> worth looking at.  However, if you still have your nameservers in the
> >> public set of NS records for your domains, you'll still be
> >> vulnerable.  Some of these providers can probably load your zones
> >> using you as a shadow master: they just do a zone transfer from your
> >> DNS infrastructure, then serve all the queries from their own systems.
> >>
> >> That's my perspective.  Hopefully it's not too out of date.
> >>
> >>  -Steve
> >>
> >> On 4/3/2020 4:18 AM, Tessa Plum wrote:
> >>> Hi Steve
> >>>
> >>> I am so appreciate to get your kind private message, though I would
> >>> like to reply my content to the list.
> >>>
> >>> We are running authoritative name servers only, zone data are for
> >>> the university only.
> >>>
> >>> When the attack happened, the bandwidth watched in our gateway was
> >>> about 20Gbps. That made name servers totally no response. Each name
> >>> server has only 1Gbps interface to internet, so it dies.
> >>>
> >>> We were considering the actions:
> >>> 1. increase bandwidth to both inbound gateway and vlan for nameservers.
> >>> 2. upgrade the network interface of nameserver to 10Gbps.
> >>> 3. run multiple servers as cluster.
> >>> 4. try to get a commercial device to analyst and stop such kind of
> >>> attack.
> >>> 5. enable RRL when attack happens.
> >>> 6. I will try to suggest administrator to 

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Steven Miller]

Essentially, yes.  Some increase in capacity on your side plus RRL will 
certainly keep you safer, but it's no guarantee.


Though to be clear, every few years, someone's going to hit a public DNS 
provider with enough load to cause a problem.  IMHO that'll happen less 
on average, and the mitigation time will be lower on average, and the 
pain level for you will be lower on average (no scrambling for 
resources, the ability to say "yeah, a big chunk of the Internet's down, 
I'll let you know when it's over" :-)) than it will happen if you run 
your own infrastructure.


It's a really unfortunate state of affairs.

    -Steve

On 4/3/2020 5:03 AM, Tessa Plum wrote:
So no way to stop reflector attack unless migrating servers to 
professional IDC?


Thanks.

Steven Miller wrote:
Adding more servers and going to 10G NICs seems relatively 
inexpensive and that should be helpful for "casual" attacks where 
you're being used as a reflector.  In those attacks, no one's out to 
attack you: they just want you to attack someone else, and don't mind 
eating all your bandwidth/CPU/whatever in order to do that.


Adding more bandwidth without enabling RRL or putting some sort of 
filtering in place will make your facilities more attractive to 
attackers, though.  I'd expect that attackers are passing around 
lists of particularly good sites for reflector attacks, and the fewer 
controls you have, and the more bandwidth you have, the more 
attractive you are for use in an attack -- and therefore the more 
likely you are to have your resources saturated.


I think RRL should be safe to run all the time.  You wouldn't want to 
scramble to enable it during an attack.


I don't know if there are commercial devices I would trust to be 
helpful in these situations, but when I was doing DNS DDoS work, 
nothing commercial was going to scale enough, so I didn't consider 
them much. :-)


The hard thing about these attacks is that there's always some time 
when local resources aren't enough: when you upgrade to 50Gbit/sec of 
capacity and the next attack is 60Gbit/sec of traffic.  I'd expect 
some correlation between "really high bandwidth attacks" and "attacks 
that are meant to hurt you instead of just use you as a reflector" 
but that correlation won't be perfect.  It's unfortunate that in the 
DNS attack world, for a lot of attacks, all you can do is have 
massively more capacity than you need on a daily basis.


The advantage to moving DNS into a cloud provider is that they have 
the resources to massively, crazily overprovision, to the point where 
it would be hard even for a nation-state to mount a big enough attack 
to take them down.  I'm most familiar with Cloudflare (I have never 
worked there, for the record) but certainly there are other companies 
worth looking at.  However, if you still have your nameservers in the 
public set of NS records for your domains, you'll still be 
vulnerable.  Some of these providers can probably load your zones 
using you as a shadow master: they just do a zone transfer from your 
DNS infrastructure, then serve all the queries from their own systems.


That's my perspective.  Hopefully it's not too out of date.

 -Steve

On 4/3/2020 4:18 AM, Tessa Plum wrote:

Hi Steve

I am so appreciate to get your kind private message, though I would 
like to reply my content to the list.


We are running authoritative name servers only, zone data are for 
the university only.


When the attack happened, the bandwidth watched in our gateway was 
about 20Gbps. That made name servers totally no response. Each name 
server has only 1Gbps interface to internet, so it dies.


We were considering the actions:
1. increase bandwidth to both inbound gateway and vlan for nameservers.
2. upgrade the network interface of nameserver to 10Gbps.
3. run multiple servers as cluster.
4. try to get a commercial device to analyst and stop such kind of 
attack.

5. enable RRL when attack happens.
6. I will try to suggest administrator to run secondary nameservers 
on professional hosting, such as cloudflare, Akamai, AWS route 53 etc.

  (also easyDNS, DNSimple, DNSMadeEasy, NS1 can be considered?)

How do you think of them?

Thank you.

regards
Tessa





___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

So no way to stop reflector attack unless migrating servers to 
professional IDC?


Thanks.

Steven Miller wrote:
Adding more servers and going to 10G NICs seems relatively inexpensive 
and that should be helpful for "casual" attacks where you're being used 
as a reflector.  In those attacks, no one's out to attack you: they just 
want you to attack someone else, and don't mind eating all your 
bandwidth/CPU/whatever in order to do that.


Adding more bandwidth without enabling RRL or putting some sort of 
filtering in place will make your facilities more attractive to 
attackers, though.  I'd expect that attackers are passing around lists 
of particularly good sites for reflector attacks, and the fewer controls 
you have, and the more bandwidth you have, the more attractive you are 
for use in an attack -- and therefore the more likely you are to have 
your resources saturated.


I think RRL should be safe to run all the time.  You wouldn't want to 
scramble to enable it during an attack.


I don't know if there are commercial devices I would trust to be helpful 
in these situations, but when I was doing DNS DDoS work, nothing 
commercial was going to scale enough, so I didn't consider them much. :-)


The hard thing about these attacks is that there's always some time when 
local resources aren't enough: when you upgrade to 50Gbit/sec of 
capacity and the next attack is 60Gbit/sec of traffic.  I'd expect some 
correlation between "really high bandwidth attacks" and "attacks that 
are meant to hurt you instead of just use you as a reflector" but that 
correlation won't be perfect.  It's unfortunate that in the DNS attack 
world, for a lot of attacks, all you can do is have massively more 
capacity than you need on a daily basis.


The advantage to moving DNS into a cloud provider is that they have the 
resources to massively, crazily overprovision, to the point where it 
would be hard even for a nation-state to mount a big enough attack to 
take them down.  I'm most familiar with Cloudflare (I have never worked 
there, for the record) but certainly there are other companies worth 
looking at.  However, if you still have your nameservers in the public 
set of NS records for your domains, you'll still be vulnerable.  Some of 
these providers can probably load your zones using you as a shadow 
master: they just do a zone transfer from your DNS infrastructure, then 
serve all the queries from their own systems.


That's my perspective.  Hopefully it's not too out of date.

     -Steve

On 4/3/2020 4:18 AM, Tessa Plum wrote:

Hi Steve

I am so appreciate to get your kind private message, though I would 
like to reply my content to the list.


We are running authoritative name servers only, zone data are for the 
university only.


When the attack happened, the bandwidth watched in our gateway was 
about 20Gbps. That made name servers totally no response. Each name 
server has only 1Gbps interface to internet, so it dies.


We were considering the actions:
1. increase bandwidth to both inbound gateway and vlan for nameservers.
2. upgrade the network interface of nameserver to 10Gbps.
3. run multiple servers as cluster.
4. try to get a commercial device to analyst and stop such kind of 
attack.

5. enable RRL when attack happens.
6. I will try to suggest administrator to run secondary nameservers on 
professional hosting, such as cloudflare, Akamai, AWS route 53 etc.

  (also easyDNS, DNSimple, DNSMadeEasy, NS1 can be considered?)

How do you think of them?

Thank you.

regards
Tessa




___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

Hi Steve

I am so appreciate to get your kind private message, though I would like 
to reply my content to the list.


We are running authoritative name servers only, zone data are for the 
university only.


When the attack happened, the bandwidth watched in our gateway was about 
20Gbps. That made name servers totally no response. Each name server has 
only 1Gbps interface to internet, so it dies.


We were considering the actions:
1. increase bandwidth to both inbound gateway and vlan for nameservers.
2. upgrade the network interface of nameserver to 10Gbps.
3. run multiple servers as cluster.
4. try to get a commercial device to analyst and stop such kind of attack.
5. enable RRL when attack happens.
6. I will try to suggest administrator to run secondary nameservers on 
professional hosting, such as cloudflare, Akamai, AWS route 53 etc.

  (also easyDNS, DNSimple, DNSMadeEasy, NS1 can be considered?)

How do you think of them?

Thank you.

regards
Tessa
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Friday, 3 April 2020 02:55:21 UTC Tessa Plum wrote:
> Where is "dnsdbq" coming from? I didn't see my ubuntu system has that a
> command.

it's an example of passive dns lookups. the source code is here:

https://github.com/dnsdb/dnsdbq

there are dozens of passive dns database systems; only two are currently 
supported by that tool. (more would be welcome, it's apache 2.0 licensed!)

> Paul Vixie wrote:
> > $ dnsdbq -r '\*.berkeley.edu/ns' -A 2020-01-01 -j | jq .rrname | uniq

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

Where is "dnsdbq" coming from? I didn't see my ubuntu system has that a 
command.


Thank you.


Paul Vixie wrote:

$ dnsdbq -r '\*.berkeley.edu/ns' -A 2020-01-01 -j | jq .rrname | uniq

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[John Levine]

In article <26a9dd93-91a3-dbc4-c34b-145f33e74...@plum.ovh> you write:
>Hi Stephane,
>
>I saw you were from FRNIC. May I ask a question that, since I got a 
>domain from .ovh, It seems anyone can have a domain extension? So how 
>can I have my own extension, such as .plum? Shall I contact the root 
>server operators to put .plum glues there?

Short answer: no.

ICANN manages the addition of top level domains through very, very, complicated
processes.

They are not currently accepting applications for new top-level domains, but
the last time they did, the fee to apply was $180,000.

R's,
John
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Friday, 3 April 2020 01:18:46 UTC Tessa Plum wrote:
> ...
> 
> Not only for those private domain names, but zone data also includes the
> administrative structure of corp/group.

nothing in the dns is private. if you don't want something viewed, cataloged, 
indexed, searched, and used, then do not put that something into DNS at all.

> For example, Colleges and Departmnts, administration offices and their
> sub-domains and MX records were defined in this zone file.

attached please find a list of all NS-record owning domain names at or below 
berkeley.edu which have been witnessed at least once since january 1 2020. i 
produced it using the following command:

$ dnsdbq -r '\*.berkeley.edu/ns' -A 2020-01-01 -j | jq .rrname | uniq

you should be able to extrapolate from this that DNS keeps _no_ secrets, and 
if you wonder why not, the answer is that keeping secrets is contrary to the 
goal of DNS. DNS is a publication system, not a secrecy system.

-- 
Paul"berkeley.edu."
"ce.berkeley.edu."
"cp.berkeley.edu."
"cs.berkeley.edu."
"dh.berkeley.edu."
"fs.berkeley.edu."
"ga.berkeley.edu."
"ih.berkeley.edu."
"is.berkeley.edu."
"ls.berkeley.edu."
"me.berkeley.edu."
"pi.berkeley.edu."
"rs.berkeley.edu."
"uc.berkeley.edu."
"v6.berkeley.edu."
"w3.berkeley.edu."
"2ol.berkeley.edu."
"_dmarc.2ol.berkeley.edu."
"_domainkey.2ol.berkeley.edu."
"aap.berkeley.edu."
"abc.berkeley.edu."
"api.berkeley.edu."
"are.berkeley.edu."
"bic.berkeley.edu."
"bjc.berkeley.edu."
"brc.berkeley.edu."
"bsp.berkeley.edu."
"cal.berkeley.edu."
"cbe.berkeley.edu."
"ced.berkeley.edu."
"chu.berkeley.edu."
"cnr.berkeley.edu."
"coe.berkeley.edu."
"dsp.berkeley.edu."
"dyn.berkeley.edu."
"ebi.berkeley.edu."
"ehs.berkeley.edu."
"erg.berkeley.edu."
"ets.berkeley.edu."
"fao.berkeley.edu."
"fxd.berkeley.edu."
"geo.berkeley.edu."
"ias.berkeley.edu."
"ica.berkeley.edu."
"igs.berkeley.edu."
"ihd.berkeley.edu."
"iir.berkeley.edu."
"ist.berkeley.edu."
"its.berkeley.edu."
"jlg.berkeley.edu."
"law.berkeley.edu."
"lhs.berkeley.edu."
"lib.berkeley.edu."
"mba.berkeley.edu."
"mcb.berkeley.edu."
"mfe.berkeley.edu."
"mri.berkeley.edu."
"mse.berkeley.edu."
"msp.berkeley.edu."
"mvz.berkeley.edu."
"nes.berkeley.edu."
"net.berkeley.edu."
"nuc.berkeley.edu."
"ocf.berkeley.edu."
"ohr.berkeley.edu."
"our.berkeley.edu."
"pmb.berkeley.edu."
"qb3.berkeley.edu."
"rac.berkeley.edu."
"sma.berkeley.edu."
"soc.berkeley.edu."
"soe.berkeley.edu."
"sph.berkeley.edu."
"spo.berkeley.edu."
"ssl.berkeley.edu."
"cse.ssl.berkeley.edu."
"local.ssl.berkeley.edu."
"tsc.berkeley.edu."
"ual.berkeley.edu."
"ucb.berkeley.edu."
"ucm.berkeley.edu."
"uga.berkeley.edu."
"uhs.berkeley.edu."
"vpn.berkeley.edu."
"xcf.berkeley.edu."
"1918.berkeley.edu."
"ist.1918.berkeley.edu."
"net.1918.berkeley.edu."
"haas.1918.berkeley.edu."
"calnet.1918.berkeley.edu."
"airbears2.1918.berkeley.edu."
"calvisitor.1918.berkeley.edu."
"2048.berkeley.edu."
"acuc.berkeley.edu."
"asuc.berkeley.edu."
"azul.berkeley.edu."
"bair.berkeley.edu."
"bear.berkeley.edu."
"bids.berkeley.edu."
"bioe.berkeley.edu."
"bnhm.berkeley.edu."
"brie.berkeley.edu."
"ceda.berkeley.edu."
"clas.berkeley.edu."
"clpr.berkeley.edu."
"cshe.berkeley.edu."
"csua.berkeley.edu."
"decf.berkeley.edu."
"dlab.berkeley.edu."
"econ.berkeley.edu."
"eecs.berkeley.edu."
"erso.berkeley.edu."
"espm.berkeley.edu."
"fhrp.berkeley.edu."
"geog.berkeley.edu."
"ggia.berkeley.edu."
"grad.berkeley.edu."
"gspp.berkeley.edu."
"haas.berkeley.edu."
"herb.berkeley.edu."
"icsi.berkeley.edu."
"notary.icsi.berkeley.edu."
"netalyzr.icsi.berkeley.edu."
"netalyser.icsi.berkeley.edu."
"netalyzer.icsi.berkeley.edu."
"ieor.berkeley.edu."
"ipsr.berkeley.edu."
"irle.berkeley.edu."
"iurd.berkeley.edu."
"kalx.berkeley.edu."
"lips.berkeley.edu."
"math.berkeley.edu."
"olac.berkeley.edu."
"path.berkeley.edu."
"peer.berkeley.edu."
"ppcs.berkeley.edu."
"rojo.berkeley.edu."
"rssp.berkeley.edu."
"sdsc.berkeley.edu."
"eis-github-prod-03.sdsc.berkeley.edu."
"sims.berkeley.edu."
"snap.berkeley.edu."
"sscl.berkeley.edu."
"stat.berkeley.edu."
"ucei.berkeley.edu."
"ucmp.berkeley.edu."
"udar.berkeley.edu."
"ugis.berkeley.edu."
"unex.berkeley.edu."
"urel.berkeley.edu."
"vcbf.berkeley.edu."
"vspa.berkeley.edu."
"astro.berkeley.edu."
"atmos.berkeley.edu."
"cchem.berkeley.edu."
"cnmat.berkeley.edu."
"cpsma.berkeley.edu."
"demog.berkeley.edu."
"gsppi.berkeley.edu."
"ipira.berkeley.edu."
"iucrp.berkeley.edu."
"media.berkeley.edu."
"neuro.berkeley.edu."
"psych.berkeley.edu."
"ucbso.berkeley.edu."
"voice.berkeley.edu."
"alumni.berkeley.edu."
"bampfa.berkeley.edu."
"bluext.berkeley.edu."
"calnet.berkeley.edu."
"calsol.berkeley.edu."
"campus.berkeley.edu."
"chance.berkeley.edu."
"citris.berkeley.edu."
"cspace.berkeley.edu."
"data8x.berkeley.edu."
"devlib.berkeley.edu."
"garden.berkeley.edu."
"github.berkeley.edu."
"humbio.berkeley.edu."
"physed.berkeley.edu."
"sccqb3.berkeley.edu."
"seismo.berkeley.edu."
"simons.berkeley.edu."
"socwel.berkeley.edu."
"summer.berkeley.edu."
"vision.berkeley.edu."
"y-plan.berkeley.edu."

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

Hi Stephane,

I saw you were from FRNIC. May I ask a question that, since I got a 
domain from .ovh, It seems anyone can have a domain extension? So how 
can I have my own extension, such as .plum? Shall I contact the root 
server operators to put .plum glues there?


Thank you.
Tessa

Stephane Bortzmeyer wrote:

Commercial boxes are typically optimised for HTTP, DNS is very

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

Fred Morris wrote:
There is this thing called a "search list". Love 'em or hate 'em (kind 
of like DNAMEs!).


Suppose your (ab)user is in a coffee shop (wearing appropriate hazmat 
gear of course). They load their web browser. It's visited 
secret-project.university-example.edu previously. Being extremely 
helpful, the browser tries to prefetch the address for 
secret-project.university-example.edu. When that doesn't work, it then 
tries secret-project.university-example.edu.coffeeshop-example.com. And 
so on, and so forth. (*cough* .cisco *cough* .belkin... no it's not 
COVID, I seem to have some DNS caught in my throat...)


Hello,

Not only for those private domain names, but zone data also includes the 
administrative structure of corp/group.


For example, Colleges and Departmnts, administration offices and their 
sub-domains and MX records were defined in this zone file.


Regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Fred Morris]

Yes, although if you don't believe us maybe you're looking in the wrong 
place


On Thu, 3 Apr 2020, John Levine wrote:

In article ,
Tessa Plum  wrote:

University has generally some private research projects who have their
domain names, but university won't let others see these domain names
unless the projects have got public.


If those names are ever retrieved by users on networks outside your
university, it's very likely that they're in public passive DNS
databases that are widely visible.  It is not realistic to believe
that you can put names in your public DNS and not have the world
know about them.


There is this thing called a "search list". Love 'em or hate 'em (kind of 
like DNAMEs!).


Suppose your (ab)user is in a coffee shop (wearing appropriate hazmat gear 
of course). They load their web browser. It's visited 
secret-project.university-example.edu previously. Being extremely helpful, 
the browser tries to prefetch the address for 
secret-project.university-example.edu. When that doesn't work, it then 
tries secret-project.university-example.edu.coffeeshop-example.com. And so 
on, and so forth. (*cough* .cisco *cough* .belkin... no it's not COVID, I 
seem to have some DNS caught in my throat...)


--

Fred Morris

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

John Levine wrote:

If those names are ever retrieved by users on networks outside your
university, it's very likely that they're in public passive DNS
databases that are widely visible.  It is not realistic to believe
that you can put names in your public DNS and not have the world
know about them.


I meant the domain names were setup in an education network environment.

regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[John Levine]

In article ,
Tessa Plum  wrote:
>University has generally some private research projects who have their 
>domain names, but university won't let others see these domain names 
>unless the projects have got public.

If those names are ever retrieved by users on networks outside your
university, it's very likely that they're in public passive DNS
databases that are widely visible.  It is not realistic to believe
that you can put names in your public DNS and not have the world
know about them.

-- 
Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Mark Andrews]

> On 3 Apr 2020, at 00:09, Tessa Plum  wrote:
> 
> On 2020/4/2 7:28 下午, Stephane Bortzmeyer wrote:
>> BCP38 is Good,*but*  it protects others against you. So, to be
>> protected, you need the*others*  to implement it.
> 
> Ah OK.
> So BCP38 is useless for my case. Others don't care if I am meeting the attack 
> or not.
> 
> regards.

No, it is not useless.  It requires you to talk to your upstream providers and 
have them traceback the attacks to their source.  Repeat with their upstreams.  
The sources can be cut off which can just be turn on BCP38 filtering on a link 
that is emitting spoofed traffic.  They can do that.  Every network that turns 
on BCP38 filtering is one more you don’t have to worry about in the future 
sending you spoofed traffic.

None of this saying don’t do the other measures.

Spoofed traffic has been a long term problem.  It does require getting people 
to spend time reconfiguring boxes.  That has a cost but it is a lot smaller 
cost globally than carrying the spoofed traffic past the earliest point where 
it can be blocked and defending against the spoofed traffic.  Unfortunately 
many ISPs don’t see that it is in their enlightened self interest to deploy 
BCP38 filters.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

strong +1 here. recommended reading or re-reading.

On Thursday, 2 April 2020 17:23:22 UTC Fred Morris wrote:
> On Thu, 2 Apr 2020, Davey Song wrote:
> > I'm very confused that why people on the list are suggesting RRL (even
> > BCP38) to the victim of DoS attack?
> 
> The reason rate limiting, of any kind (not just DNS, not just UDP; TCP SYN
> for example), helps in a spoofed source attack is because it makes you a
> less nourishing host for the parasites and hopefully they eventually move
> on.
> 
> It also means that a persistent legitimate party is more likely to get an
> answer.
> 
> It also means that the true victim (behind the spoofed source address) is
> less likely to mitigate by blocking traffic from you (your legitimate
> source address when you reply).

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Thursday, 2 April 2020 11:28:51 UTC Stephane Bortzmeyer wrote:
> On Thu, Apr 02, 2020 at 03:06:17PM +0800,
>  Tessa Plum  wrote
> 
>  a message of 18 lines which said:
> > I never knew BCP38 before. I will try to study it.
> 
> BCP38 is Good, *but* it protects others against you. So, to be
> protected, you need the *others* to implement it.

BCP38 is excellent for technical readers. a retelling of the BCP38 story meant 
for less-technical readers such as C-level or board risk committees is here:

https://dl.acm.org/doi/pdf/10.1145/2578902?download=true

which refers to this, which was also intended for less-technical readers:

https://www.icann.org/en/system/files/files/sac-004-en.pdf

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Fred Morris]

On Thu, 2 Apr 2020, Davey Song wrote:

I'm very confused that why people on the list are suggesting RRL (even
BCP38) to the victim of DoS attack?


The reason rate limiting, of any kind (not just DNS, not just UDP; TCP SYN 
for example), helps in a spoofed source attack is because it makes you a 
less nourishing host for the parasites and hopefully they eventually move 
on.


It also means that a persistent legitimate party is more likely to get an 
answer.


It also means that the true victim (behind the spoofed source address) is 
less likely to mitigate by blocking traffic from you (your legitimate 
source address when you reply).


--

Fred Morris

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 09:31:18PM +0800,
 Tessa Plum  wrote 
 a message of 7 lines which said:

> I think we can put the devices in our own network to protect such attacks.

Commercial boxes are typically optimised for HTTP, DNS is very
different. I remember a box which was creating an entry in memory for
every source IP address. Even with IPv4, an attack with randomised
addresses was sufficient to kill it. Not even mentioning IPv6 :-)

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tony Finch]

Tessa Plum  wrote:
>
> Does RRL work based on IP addr? but the requesting IP seems spoofed.

RRL is based on the contents of the DNS response as well as the IP
address. Usually for a DDoS attack the IP address is spoofed as the
address of the victim, so rate limiting reduces the amount of response
traffic sent to the victim.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Cromarty, Forth: Northwest 6 to gale 8, occasionally severe gale 9 at first in
Cromarty, backing west 4 to 6. Very rough at first in northeast Cromarty,
otherwise moderate or rough. Squally wintry showers. Good, occasionally poor.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

> I think we can put the devices in our own network to protect such attacks.
>

Sorry. I think no such kind of devices effective for your university, IMHO.
Usually anti-DoS solution providers are able to undertake huge amont
traffic and clean them bacause they can utilize huge amout of network
resouce like bandwidth, strong load-balancer and clusters, muliple IDC in
different regions etc.

In your case, the bottleneck is on the server and its interface.  You can
build your nameserver clusters with free software as I suggested.

Davey
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 9:29 下午, Davey Song wrote:
Usually the commercial DoS mitigation solution require you to put your 
service in their network


I think we can put the devices in our own network to protect such attacks.

regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

Usually the commercial DoS mitigation solution require you to put your
service in their network, like the secondary DNS in which you have privacy
concern.

Davey

On Thu, 2 Apr 2020 at 21:15, Tessa Plum  wrote:

> On 2020/4/2 7:09 下午, Klaus Darilion wrote:
> >
> > So my advice: use a name server which can fill your upstream bandwith
> > (NSD, Knot ...). And for volumetric attacks use a commercial DDoS
> > mitigation provider which filters your traffic (ie. buy the service from
> > your ISP or from a remote DDoS mitigation provider which announces your
> > prefixes on demand.)
>
> That's really what I am looking for.
> We don't mind to buy a commercial mitigation device to prevent such type
> of attacks.
>
> Thank you.
> Tessa
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 7:35 下午, Stephane Bortzmeyer wrote:

You said you are managing DNS for your university and your concern
for secondary DNS is privacy. I'm not sure what exactly the privacy
concerns are.

RFC 7626.

Also, it may raise issues about integrity/trust/etc. In that case,
DNSSEC certainly helps a lot.



That's right.

University has generally some private research projects who have their 
domain names, but university won't let others see these domain names 
unless the projects have got public.


Regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

On Thu, 2 Apr 2020 at 20:58, Tessa Plum  wrote:

> On 2020/4/2 5:39 下午, Ray Bellis wrote:
> > If it's an authoritative server, turn on Response Rate Limiting (RRL) if
> > it's BIND, or the equivalent feature if is isn't.
>
> Yes they are authoritative servers.
> Does RRL work based on IP addr? but the requesting IP seems spoofed.
>
> Is the spoofed IPs randomly generated?

Considering your privacy concern , you can try the appoarch to increase the
bandwidth and harden the name server with cluster using OSPF ECMP (
https://archive.nanog.org/meetings/nanog34/presentations/abley.nameservers.pdf
)

Davey
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 7:28 下午, Stephane Bortzmeyer wrote:

BCP38 is Good,*but*  it protects others against you. So, to be
protected, you need the*others*  to implement it.


Ah OK.
So BCP38 is useless for my case. Others don't care if I am meeting the 
attack or not.


regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 7:09 下午, Klaus Darilion wrote:


So my advice: use a name server which can fill your upstream bandwith 
(NSD, Knot ...). And for volumetric attacks use a commercial DDoS 
mitigation provider which filters your traffic (ie. buy the service from 
your ISP or from a remote DDoS mitigation provider which announces your 
prefixes on demand.)


That's really what I am looking for.
We don't mind to buy a commercial mitigation device to prevent such type 
of attacks.


Thank you.
Tessa
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 6:43 下午, Klaus Darilion wrote:


So what was the bottleneck? I.e. if you use PowerDNS with DB backend you 
quite early hit the limit with random subdomains, which are not a 
problem if you use NSD for example. To mitigation such traffic patterns 
for example we use dnsdist with 2 backends, PowerDNS for nomarl zones 
and NSD for zones which are quite often under attack.


Hi

the bottleneck seems be the bandwidth of server (server has only 1Gbps 
on its public interface).

b/c at that time the server totally has no response.
Even ssh can't get logined.

Thanks.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

On Thu, 2 Apr 2020 at 19:38, Stephane Bortzmeyer  wrote:

>
> RFC 7626.
>
> Also, it may raise issues about integrity/trust/etc. In that case,
> DNSSEC certainly helps a lot.
>

OK. I need more sense of privacy :)

Davey
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 5:39 下午, Ray Bellis wrote:

If it's an authoritative server, turn on Response Rate Limiting (RRL) if
it's BIND, or the equivalent feature if is isn't.


Yes they are authoritative servers.
Does RRL work based on IP addr? but the requesting IP seems spoofed.

Thanks
Tessa
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 05:39:48PM +0800,
 Davey Song  wrote 
 a message of 111 lines which said:

> You said you are managing DNS for your university and your concern
> for secondary DNS is privacy. I'm not sure what exactly the privacy
> concerns are.

RFC 7626.

Also, it may raise issues about integrity/trust/etc. In that case,
DNSSEC certainly helps a lot.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 11:05:48AM +0100,
 Tony Finch  wrote 
 a message of 30 lines which said:

> > ACLs in the server are not enough, you also need ingress filtering
> > on the borders of your network, to prevent packets claiming to be
> > from your network to get inside.
> 
> That kind of ingress filtering protects you against DDoSing
> yourself, which maybe the rest of the Internet isn't too bothered
> about :-)

I'm not sure I understand you.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Klaus Darilion]

Indeed - I only wanted to comment on the rate limiting. It is not that I 
argue against rate limiting, but that admins should be aware when it 
actually helps, and when not. Sorry, when my email seemed a bit harshly.


We also used rate limiting with dnsdist, but due to the mentioned 
problems we switched to high performance backends for the zones which 
are under constant attack.


regards
Klaus


Am 02.04.2020 um 13:22 schrieb Frank Louwers:

That's very selective cutting of my sentence Klaus

On 2 Apr 2020, at 13:09, Klaus Darilion > wrote:


Am 02.04.2020 um 09:15 schrieb Frank Louwers:

dnsdist allows you to do general ratelimiting/blocking


Ratelimiting is often not the correct choice.

If the source IP is random (which is usually the case with spoofed 
source IP addresses), a rate limiting based on source IP is not useful.


If the query-name is random (which is usually the case with spoofed 
source IP addresses), a rate limiting based on qname is not useful.


If the qname is always the same, or at least within the same zone, you 
could do rate limiting for that zone, but this limits all queries, 
attack queries and legitim queries. Hence, you create a DoS for that 
zone, but at least avoid collateral damage to other zones hosted on 
that name server.


So my advice: use a name server which can fill your upstream bandwith 
(NSD, Knot ...). And for volumetric attacks use a commercial DDoS 
mitigation provider which filters your traffic (ie. buy the service 
from your ISP or from a remote DDoS mitigation provider which 
announces your prefixes on demand.)


regards
Klaus

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net 


https://lists.dns-oarc.net/mailman/listinfo/dns-operations




___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 03:06:17PM +0800,
 Tessa Plum  wrote 
 a message of 18 lines which said:

> I never knew BCP38 before. I will try to study it.

BCP38 is Good, *but* it protects others against you. So, to be
protected, you need the *others* to implement it.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

On Thu, 2 Apr 2020 at 18:22, Jim Reid  wrote:

>
> RRL won’t help with the volume of incoming queries.

Exactly!


> It will however reduce the volume of outgoing responses which may well be
> DoS’ing another innocent victim.
>
 Agree

Davey
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Frank Louwers]

That's very selective cutting of my sentence Klaus

> On 2 Apr 2020, at 13:09, Klaus Darilion  > wrote:
> 
> Am 02.04.2020 um 09:15 schrieb Frank Louwers:
>> dnsdist allows you to do general ratelimiting/blocking
> 
> Ratelimiting is often not the correct choice.
> 
> If the source IP is random (which is usually the case with spoofed source IP 
> addresses), a rate limiting based on source IP is not useful.
> 
> If the query-name is random (which is usually the case with spoofed source IP 
> addresses), a rate limiting based on qname is not useful.
> 
> If the qname is always the same, or at least within the same zone, you could 
> do rate limiting for that zone, but this limits all queries, attack queries 
> and legitim queries. Hence, you create a DoS for that zone, but at least 
> avoid collateral damage to other zones hosted on that name server.
> 
> So my advice: use a name server which can fill your upstream bandwith (NSD, 
> Knot ...). And for volumetric attacks use a commercial DDoS mitigation 
> provider which filters your traffic (ie. buy the service from your ISP or 
> from a remote DDoS mitigation provider which announces your prefixes on 
> demand.)
> 
> regards
> Klaus
> 
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net 
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

But Tessa Plum are asking for help when they were under attack with a lot
of UDP requests flooding to the servers.

When a patient with flu asking for help, but his doctor only suggest him to
mask himself avoid he inffectiing others. Wearing masks is generally good
for public but not a cure for that patient.

Given 20Gbps attack, not a huge one, I still think sharing the load among
multiple servers make sense. If possible, some advanced anti-ddos
techniques are available.  There is no cost-free solution for DoS.

Davey

On Thu, 2 Apr 2020 at 18:22, Ray Bellis  wrote:

>
> The OP described a spoofed-source amplification attack.
>
> They are not the "victim", but the unwilling participant.
>
> RRL is the correct solution for this class of attack.
>
> Ray
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Klaus Darilion]

Am 02.04.2020 um 09:15 schrieb Frank Louwers:

dnsdist allows you to do general ratelimiting/blocking


Ratelimiting is often not the correct choice.

If the source IP is random (which is usually the case with spoofed 
source IP addresses), a rate limiting based on source IP is not useful.


If the query-name is random (which is usually the case with spoofed 
source IP addresses), a rate limiting based on qname is not useful.


If the qname is always the same, or at least within the same zone, you 
could do rate limiting for that zone, but this limits all queries, 
attack queries and legitim queries. Hence, you create a DoS for that 
zone, but at least avoid collateral damage to other zones hosted on that 
name server.


So my advice: use a name server which can fill your upstream bandwith 
(NSD, Knot ...). And for volumetric attacks use a commercial DDoS 
mitigation provider which filters your traffic (ie. buy the service from 
your ISP or from a remote DDoS mitigation provider which announces your 
prefixes on demand.)


regards
Klaus

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Klaus Darilion]

Am 02.04.2020 um 05:51 schrieb Tessa Plum:

Hello Paul

We were under some attack like UDP flood to the authority servers, there 
were a lot of UDP requests flooding to the servers. The traffic size was 
about 20Gbps last time as I have said in last message. The clients seem 
using spoofed IP addresses.


Was it a) just some DNS traffic filling the upstream bandwidth, or was 
it b) legitim DNS requests to existing domains (or ie random subdomains)?


For a) you can use any DDoS Mitigation used for any service.

For b) you need some advanced techniques, ie. filtering with dnsdist, or 
if you can detect some pattern to identify the DDoS packets, you can use 
BPF filter to filter out such traffic bevor hitting your name server.


So what was the bottleneck? I.e. if you use PowerDNS with DB backend you 
quite early hit the limit with random subdomains, which are not a 
problem if you use NSD for example. To mitigation such traffic patterns 
for example we use dnsdist with 2 backends, PowerDNS for nomarl zones 
and NSD for zones which are quite often under attack.


regards
Klaus
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Ray Bellis]

On 02/04/2020 11:10, Davey Song wrote:
> I'm very confused that why people on the list are suggesting RRL (even
> BCP38) to the victim of DoS attack? If I remember correctly, the goal of
> both RRL and BCP38 is to reduce the chance of participating the attack
> as a innocent helper.
> 
> In the introduce of RRL (https://kb.isc.org/docs/aa-01000)  , it goes :
> "RRL helps mitigate DNS denial-of-service attacks by reducing the rate
> at which authoritative servers respond to high volumes of malicious
> queries. "  
> 
> Please correct me .

The OP described a spoofed-source amplification attack.

They are not the "victim", but the unwilling participant.

RRL is the correct solution for this class of attack.

Ray

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tony Finch]

Stephane Bortzmeyer  wrote:
> On Thu, Apr 02, 2020 at 03:06:49AM +,
>  Paul Vixie  wrote
>  a message of 29 lines which said:
>
> > to keep your own recursive servers from amplifying spoofed-source
> > attacks, you need ACL's that make it unreachable outside your
> > specific client base.
>
> ACLs in the server are not enough, you also need ingress filtering on
> the borders of your network, to prevent packets claiming to be from
> your network to get inside.

That kind of ingress filtering protects you against DDoSing yourself,
which maybe the rest of the Internet isn't too bothered about :-)

You ALSO need ACLs on all the crappy consumer routers to stop their DNS
forwarders from being used in an attack. And BCP38. Both of these are not
as common as they should be :-(

You can configure your authoritative servers to be less attrative for use
in DDoS attacks: as well as RRL, configure minimal responses, minimal ANY,
roll to DNSSEC algorithm 13 instead of RSA (all help to keep response
sizes small), and set your UDP size limit to less than one MTU (to reduce
packet count amplification).

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
no one shall be enslaved by poverty, ignorance, or conformity
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Davey Song]

<>

The intuitive solution against the DoS attack is to scale your system wiith
mulitple servers in the globe. You can either develop global
anycast instance as Paul suggested or select and operate secondary DNS
servers documented in RFC2182/BCP16.

There are many secondary DNS providers available. They also
provide anti-DDoS-attack solutions if you want to purchase. It 's really
helpful if you run important Internet services.

If you want to focus on your business/applicaiton rather than DNS
operation. You can simplely use managed DNS services in many DNS providers.
It's cheep  and efficient.

You said you are managing DNS for your university and your concern for
secondary DNS is privacy. I'm not sure what exactly the privacy concerns
are. But I think if you still can have one name server localted inside your
university for local/private name space and use another Name server with
secondary DNS for publich availlable name space. People can help if you
provide more information in your case.

Hope it helps you.

Davey

On Thu, 2 Apr 2020 at 10:20, Tessa Plum  wrote:

> Hello
>
> May I ask if there are any solutions for DDoS mitigation of DNS?
> Both commercial or free solutions could be considered.
>
> Thanks.
>
> Tessa
> https://plum.ovh/
>
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Ray Bellis]

On 02/04/2020 10:12, Tessa Plum wrote:

> All the packages were DNS requests, some queries like 'dig domain.com any'.
> but their IP address seems spoofed.
> A request from the fake address to our nameserver, but nameserver try
> its best to reply to this unreal address.

If it's a recursive server, apply an ACL so that only expected clients
can query.

If it's an authoritative server, turn on Response Rate Limiting (RRL) if
it's BIND, or the equivalent feature if is isn't.

Ray

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 05:12:29PM +0800,
 Tessa Plum  wrote 
 a message of 11 lines which said:

> All the packages were DNS requests, some queries like 'dig domain.com any'.
> but their IP address seems spoofed.

In that case, yes, RRL would help.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 3:19 下午, Stephane Bortzmeyer wrote:

DNS or another type?


Stephane,

All the packages were DNS requests, some queries like 'dig domain.com any'.
but their IP address seems spoofed.
A request from the fake address to our nameserver, but nameserver try 
its best to reply to this unreal address.


Thanks.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 11:51:05AM +0800,
 Tessa Plum  wrote 
 a message of 37 lines which said:

> We were under some attack like UDP flood to the authority servers,

DNS or another type?

> The traffic size was about 20Gbps

Note that for DNS traffic, the useful metric is often
packets-per-second, not-bytes-per second. In most cases, that's what
kills the server.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Frank Louwers]

> 
>> May I ask if there are any solutions for DDoS mitigation of DNS?
> 
> All solutions that were mentioned here are correct but incomplete:
> there is no general solution against dDoS, because "it depends". There
> are many types of dDoS. You will need several tools in your toolbox,
> and someone knowledgeable to choose among them.

I completely agree. However, some tools have a large toolbox built-in and 
reduce the need for other tools. Eg: dnsdist allows you to do general 
ratelimiting/blocking, but allows you to build your own rules as well.

Note that I can buy a great, expensive and high-quality garage toolbox, but 
that that doesn't mean I can repair my own car. The toolbox is what the 
knowledgeable expert needs to fix the problem...

Kind Regards and best of luck fending of the bad people...

Frank Louwers
DNS Consultant @ Kiwazo.be ___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 03:06:49AM +,
 Paul Vixie  wrote 
 a message of 29 lines which said:

> to keep your own recursive servers from amplifying spoofed-source
> attacks, you need ACL's that make it unreachable outside your
> specific client base.

ACLs in the server are not enough, you also need ingress filtering on
the borders of your network, to prevent packets claiming to be from
your network to get inside.

> to keep your own servers of whatever kind from being ddos'd into
> congestion loss, you need massive overprovisioning including both
> local and global anycast.

If the congestion is on the link, yes, you are right. If it is on the
server, filtering solutions may be sufficient if there is an easy way
to sort out the bad traffic from the good one, and if they are faster
than the name server (Netfilter on Linux is fast, for instance.)
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

On 2020/4/2 12:25 下午, Mark Andrews wrote:

You use all the mechanisms available to you.

Traceback.  Getting BCP38 installed at the sites emitting spoofed traffic help 
yourself and everyone else.  In many cases this is coming from compromised 
machines.

You enable/tune response rate filtering.

You use DNS COOKIES and encourage your clients to use DNS COOKIES.  This helps 
sort the wheat from the chaff.

You talk to you local politicians about mandating BCP38 deployment in your 
country.  BCP 38 is 20 years old next month so there is unless one is actually 
operating 20 year equipment there is no excuse for not having deployed BCP38 in 
you network.  This needs to see directors of ISPs sitting in gaol for not 
deploying BCP 38.



Thanks. I never knew BCP38 before. I will try to study it.

regards.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Wed, Apr 01, 2020 at 07:35:35PM -0700,
 Fred Morris  wrote 
 a message of 10 lines which said:

> Depends on what you mean. You might look at "response rate limiting" in for
> instance BIND. -- FWM

RRL protects people against you (when your name server is used as a
reflector) but not really you against the dDoS.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Stephane Bortzmeyer]

On Thu, Apr 02, 2020 at 10:14:14AM +0800,
 Tessa Plum  wrote 
 a message of 14 lines which said:

> May I ask if there are any solutions for DDoS mitigation of DNS?

All solutions that were mentioned here are correct but incomplete:
there is no general solution against dDoS, because "it depends". There
are many types of dDoS. You will need several tools in your toolbox,
and someone knowledgeable to choose among them.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Thursday, 2 April 2020 03:51:05 UTC Tessa Plum wrote:
> Hello Paul
> 
> We were under some attack like UDP flood to the authority servers, there
> were a lot of UDP requests flooding to the servers. The traffic size was
> about 20Gbps last time as I have said in last message. The clients seem
> using spoofed IP addresses.
> 
> Thanks.
> Tessa

turn on DNS RRL. many name servers have it; none have it as their default.

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Mark Andrews]

You use all the mechanisms available to you.

Traceback.  Getting BCP38 installed at the sites emitting spoofed traffic help 
yourself and everyone else.  In many cases this is coming from compromised 
machines.

You enable/tune response rate filtering.

You use DNS COOKIES and encourage your clients to use DNS COOKIES.  This helps 
sort the wheat from the chaff.

You talk to you local politicians about mandating BCP38 deployment in your 
country.  BCP 38 is 20 years old next month so there is unless one is actually 
operating 20 year equipment there is no excuse for not having deployed BCP38 in 
you network.  This needs to see directors of ISPs sitting in gaol for not 
deploying BCP 38.

Mark

> On 2 Apr 2020, at 14:51, Tessa Plum  wrote:
> 
> Hello Paul
> 
> We were under some attack like UDP flood to the authority servers, there were 
> a lot of UDP requests flooding to the servers. The traffic size was about 
> 20Gbps last time as I have said in last message. The clients seem using 
> spoofed IP addresses.
> 
> Thanks.
> Tessa
> 
> 
> Paul Vixie wrote:
>> On Thursday, 2 April 2020 02:14:14 UTC Tessa Plum wrote:
>>> Hello
>>> 
>>> May I ask if there are any solutions for DDoS mitigation of DNS?
>>> Both commercial or free solutions could be considered.
>>> 
>>> Thanks.
>>> 
>>> Tessa
>>> https://plum.ovh/
>> to keep your own authority servers from amplifying spoofed-source attacks, 
>> you
>> need response rate limiting, available in bind9, dnsdist, nsd, (any others?)
>> to keep your own recursive servers from amplifying spoofed-source attacks, 
>> you
>> need ACL's that make it unreachable outside your specific client base.
>> to keep your own servers of whatever kind from being ddos'd into congestion
>> loss, you need massive overprovisioning including both local and global
>> anycast. you may also need something like akamai's "clean feed" filtering.
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Tessa Plum]

Hello Paul

We were under some attack like UDP flood to the authority servers, there 
were a lot of UDP requests flooding to the servers. The traffic size was 
about 20Gbps last time as I have said in last message. The clients seem 
using spoofed IP addresses.


Thanks.
Tessa


Paul Vixie wrote:

On Thursday, 2 April 2020 02:14:14 UTC Tessa Plum wrote:

Hello

May I ask if there are any solutions for DDoS mitigation of DNS?
Both commercial or free solutions could be considered.

Thanks.

Tessa
https://plum.ovh/


to keep your own authority servers from amplifying spoofed-source attacks, you
need response rate limiting, available in bind9, dnsdist, nsd, (any others?)

to keep your own recursive servers from amplifying spoofed-source attacks, you
need ACL's that make it unreachable outside your specific client base.

to keep your own servers of whatever kind from being ddos'd into congestion
loss, you need massive overprovisioning including both local and global
anycast. you may also need something like akamai's "clean feed" filtering.


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Paul Vixie]

On Thursday, 2 April 2020 02:14:14 UTC Tessa Plum wrote:
> Hello
> 
> May I ask if there are any solutions for DDoS mitigation of DNS?
> Both commercial or free solutions could be considered.
> 
> Thanks.
> 
> Tessa
> https://plum.ovh/

to keep your own authority servers from amplifying spoofed-source attacks, you 
need response rate limiting, available in bind9, dnsdist, nsd, (any others?)

to keep your own recursive servers from amplifying spoofed-source attacks, you 
need ACL's that make it unreachable outside your specific client base.

to keep your own servers of whatever kind from being ddos'd into congestion 
loss, you need massive overprovisioning including both local and global 
anycast. you may also need something like akamai's "clean feed" filtering.

-- 
Paul


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] solutions for DDoS mitigation of DNS

[Fred Morris]

Depends on what you mean. You might look at "response rate limiting" in 
for instance BIND. -- FWM


On Thu, 2 Apr 2020, Tessa Plum wrote:

May I ask if there are any solutions for DDoS mitigation of DNS?
Both commercial or free solutions could be considered.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations