Re: The DDOS attack on DYN & RRL ?

[Dave Warren]

On Tue, Nov 1, 2016, at 07:45, Ben Croswell wrote:
> The other option being having a master owned by your company and then
> setting both external providers to secondary from your master. You to
> maintain control over data and hqve diversity.



I use this approach here, it's proven to be very robust. Not only is the
internal master well hidden to all but the secondaries, but if it does
get directly targeted by a DDoS it won't impact your slaves at all.
Obviously if your company is the target there probably isn't much you
can do unless you have a very substantial anti-DDoS budget, but in the
case of a DNS neighbour being the target, diversifying your DNS across
2-3 larger providers will ensure that you stay up.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[MURTARI, JOHN]

Folks,
Saw something in a previous posting that should be corrected:

>  The sticking point  seems to be that most DNS providers don't allow zone 
> transfers from 
>  their servers The customers of Dyn are in the same situation.

Actually from personal experience just a few days ago with DYN.  On 
their GUI it is very to setup your own slave servers.  They allow you to enter 
the IPs of machines that should be allowed to perform transfers and you can 
also designate the machine to receive NOTIFY messages.  Tested it and it works 
great!

Best regards!
John


John Murtari - jm5...@att.com
Ciberspring
office: 315-944-0998
cell: 315-430-2702

___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Matthew Seaman]

On 2016/11/01 14:45, Ben Croswell wrote:
> The other option being having a master owned by your company and then
> setting both external providers to secondary from your master. You to
> maintain control over data and hqve diversity.

Agreed.  This works well -- it's what we do.

Cheers,

Matthew




signature.asc
Description: OpenPGP digital signature
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Barry Margolin]

In article ,
 Ben Croswell  wrote:

> The other option being having a master owned by your company and then
> setting both external providers to secondary from your master. You to
> maintain control over data and hqve diversity.

Good point, although that means maintaining another service on your own 
infrastructure.

Another thing that makes it hard for many companies to diversify their 
DNS providers is that they make use of DNS-based load balancing and 
failure (e.g. Amazon's Route 54, Akamai's Global Traffic Management). 
These services can't easily update each other.

-- 
Barry Margolin
Arlington, MA
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Ben Croswell]

The other option being having a master owned by your company and then
setting both external providers to secondary from your master. You to
maintain control over data and hqve diversity.

On Nov 1, 2016 10:42 AM, "Barry Margolin"  wrote:

> In article ,
>  Ben Croswell  wrote:
>
> > I think what we see as a result of this attack is DNS provider diversity
> > being the new buzz phrase. The same as not relying on a single ISP link i
> > see more people using multiple DNS providers.
> > The size of these attacks will grow as IoT continues to grow. It makes
> > sense to have diverse providers to ensure your domains are serviceable
> if a
> > provider gets attacked.
>
> My boss asked me to look into this after the attack. The sticking point
> seems to be that most DNS providers don't allow zone transfers from
> their servers. We currently get our auth DNS from SoftLayer, the hosting
> provider for our primary web, application, and database servers. I
> contacted them to find out if it's possible to enable zone transfers to
> a third party slave service, they said no; they suggested that we simply
> set up both services as masters, which would mean we'd have to update
> them independently (or write our own scripts that make use of each
> service's API). The customers of Dyn are in the same situation.
>
> Maybe last week's incident will prompt enough big customers to demand
> this that they'll change their policies.
>
> --
> Barry Margolin
> Arlington, MA
> ___
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Barry Margolin]

In article ,
 Ben Croswell  wrote:

> I think what we see as a result of this attack is DNS provider diversity
> being the new buzz phrase. The same as not relying on a single ISP link i
> see more people using multiple DNS providers.
> The size of these attacks will grow as IoT continues to grow. It makes
> sense to have diverse providers to ensure your domains are serviceable if a
> provider gets attacked.

My boss asked me to look into this after the attack. The sticking point 
seems to be that most DNS providers don't allow zone transfers from 
their servers. We currently get our auth DNS from SoftLayer, the hosting 
provider for our primary web, application, and database servers. I 
contacted them to find out if it's possible to enable zone transfers to 
a third party slave service, they said no; they suggested that we simply 
set up both services as masters, which would mean we'd have to update 
them independently (or write our own scripts that make use of each 
service's API). The customers of Dyn are in the same situation.

Maybe last week's incident will prompt enough big customers to demand 
this that they'll change their policies.

-- 
Barry Margolin
Arlington, MA
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Moritz Muller]

My co-authors and I wrote a paper about the events at the DNS root servers on 
2015-11-30.
On this date, the root servers received a high number of queries (but by far 
not as many as Dyn) and since most of the Root letters were using anycast, we 
were able to observe how this had an impact on their reachability.
One of our takeaways was, that more DNS anycast site did have an impact on the 
reachability.

http://www.isi.edu/%7ejohnh/PAPERS/Moura16a.pdf

Moritz

> On 31 Oct 2016, at 22:39, Jim Popovitch  wrote:
> 
> On Mon, Oct 31, 2016 at 12:21 PM, Tony Finch  wrote:
>> Jim Popovitch  wrote:
>>> 
>>> It seems to me that anycast is probably much worse in the Mirai botnet
>>> scenario unless each node is pretty much as robust as a traditional
>>> unicast node.
>> 
>> This blog post is a pretty good intro to how anycast can help with DDoS
>> mitgation, though I think Cloudflare are overstating how unique they are -
>> there are other older DNS services that distribute load over large anycast
>> clouds of commodity hardware.
>> 
>> https://blog.cloudflare.com/how-cloudflares-architecture-allows-us-to-scale-to-stop-the-largest-attacks/
>> 
> 
> Thanks for linking that Tony.   The take-away that I get from that
> article is that CF can deal with DDoS because of link capacity in each
> POP, and/or re-route legitimate traffic via BGP.   The principle
> reason they can do this is because their main biz involves packets
> larger than those traditionally seen with DNS.  The comments in that
> article mention 10 TB of capacity, how's that compare to any of the
> capacities of the various DNS providers?
> 
> -Jim P.
> ___
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
> from this list
> 
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Jim Popovitch]

On Mon, Oct 31, 2016 at 12:21 PM, Tony Finch  wrote:
> Jim Popovitch  wrote:
>>
>> It seems to me that anycast is probably much worse in the Mirai botnet
>> scenario unless each node is pretty much as robust as a traditional
>> unicast node.
>
> This blog post is a pretty good intro to how anycast can help with DDoS
> mitgation, though I think Cloudflare are overstating how unique they are -
> there are other older DNS services that distribute load over large anycast
> clouds of commodity hardware.
>
> https://blog.cloudflare.com/how-cloudflares-architecture-allows-us-to-scale-to-stop-the-largest-attacks/
>

Thanks for linking that Tony.   The take-away that I get from that
article is that CF can deal with DDoS because of link capacity in each
POP, and/or re-route legitimate traffic via BGP.   The principle
reason they can do this is because their main biz involves packets
larger than those traditionally seen with DNS.  The comments in that
article mention 10 TB of capacity, how's that compare to any of the
capacities of the various DNS providers?

-Jim P.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Ben Croswell]

I think what we see as a result of this attack is DNS provider diversity
being the new buzz phrase. The same as not relying on a single ISP link i
see more people using multiple DNS providers.
The size of these attacks will grow as IoT continues to grow. It makes
sense to have diverse providers to ensure your domains are serviceable if a
provider gets attacked.

On Oct 31, 2016 12:25 PM, "Matthew Seaman" 
wrote:

> On 2016/10/31 16:09, Barry Margolin wrote:
> > I heard that the impact of the attack was even narrower than just the
> > US, it was mostly eastern US. That suggests some things about the
> > granularity of Dyn's anycast network and the distribution of the Mirai
> > botnet.
>
> There were actually three attacks on the same day.  The first (about
> 12:00 UTC) affected pretty much just the Eastern USA, and we saw little
> beyond some raised RTTs in Europe.  The second (about 16:00UTC) took out
> all the Dyn POPs in the USA and affected their European POP.  The third
> (around 18:00UTC) ... was pretty much a non-event.  Dyn had mitigated
> the attacks pretty effectively by that point.
>
> Cheers,
>
> Matthew
>
>
>
> ___
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Matthew Seaman]

On 2016/10/31 16:09, Barry Margolin wrote:
> I heard that the impact of the attack was even narrower than just the 
> US, it was mostly eastern US. That suggests some things about the 
> granularity of Dyn's anycast network and the distribution of the Mirai 
> botnet.

There were actually three attacks on the same day.  The first (about
12:00 UTC) affected pretty much just the Eastern USA, and we saw little
beyond some raised RTTs in Europe.  The second (about 16:00UTC) took out
all the Dyn POPs in the USA and affected their European POP.  The third
(around 18:00UTC) ... was pretty much a non-event.  Dyn had mitigated
the attacks pretty effectively by that point.

Cheers,

Matthew




signature.asc
Description: OpenPGP digital signature
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Tony Finch]

Jim Popovitch  wrote:
>
> It seems to me that anycast is probably much worse in the Mirai botnet
> scenario unless each node is pretty much as robust as a traditional
> unicast node.

This blog post is a pretty good intro to how anycast can help with DDoS
mitgation, though I think Cloudflare are overstating how unique they are -
there are other older DNS services that distribute load over large anycast
clouds of commodity hardware.

https://blog.cloudflare.com/how-cloudflares-architecture-allows-us-to-scale-to-stop-the-largest-attacks/

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/  -  I xn--zr8h punycode
Hebrides, Bailey: Northwesterly 5 or 6, occasionally 7 at first in north
Bailey. Very rough at first in north Bailey, otherwise moderate or rough. Rain
at first in Hebrides, otherwise showers. Good, occasionally poor at first in
Hebrides.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Barry Margolin]

In article ,
 Jim Popovitch  wrote:

> On Mon, Oct 31, 2016 at 11:27 AM, Matthew Seaman
>  wrote:
> > On 2016/10/31 14:53, Jim Popovitch wrote:
> >> On Mon, Oct 31, 2016 at 10:25 AM, Matthew Seaman
> >>  wrote:
> >>> This despite the fact that Dyn has a global anycast network with
> >>> plenty of bandwidth, points of presence all round the world and
> >>> each POP contains a bunch of top-of-the-line servers.
> >>
> >> It seems to me that anycast is probably much worse in the Mirai botnet
> >> scenario unless each node is pretty much as robust as a traditional
> >> unicast node.
> >
> > I couldn't really say whether unicast is more or less resistant to this
> > sort of attack -- I'd guess either way it would be down to the capacity
> > at each individual node.
> >
> > It was Dyn's USA POPs that bore the brunt of the attack, presumably
> > because most of the Mirai bots were located in the USA.  Even so, it
> > still caused us plenty of grief in Europe.  Apparently the effects were
> > fairly minimal in the Far East.
> >
> 
> That makes one wonder if the EU Anycast nodes are reliant on the USA
> node(s).  I have no insights (and even less DNS knowledge) but it
> makes one wonder if there's a fundamental design flaw in anycast DNS
> that relies on one or more nodes... is anycast DNS really just
> distributed cache DNS?

"Anycast" just means that a single public IP address is routed to 
different POPs depending on where the source is. So if you query 4.2.2.1 
or 8.8.8.8 from the US, you'll go to a US nameserver; if you query them 
from Europe, you'll go to a European server.

While 4.2.2.1 and 8.8.8.8 are caching DNS, the same can be done with 
authoritative DNS, and that's what was attacked in the Dyn case (I'm not 
even sure if Dyn offers caching DNS).

I heard that the impact of the attack was even narrower than just the 
US, it was mostly eastern US. That suggests some things about the 
granularity of Dyn's anycast network and the distribution of the Mirai 
botnet.

-- 
Barry Margolin
Arlington, MA
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Jim Popovitch]

On Mon, Oct 31, 2016 at 11:27 AM, Matthew Seaman
 wrote:
> On 2016/10/31 14:53, Jim Popovitch wrote:
>> On Mon, Oct 31, 2016 at 10:25 AM, Matthew Seaman
>>  wrote:
>>> This despite the fact that Dyn has a global anycast network with
>>> plenty of bandwidth, points of presence all round the world and
>>> each POP contains a bunch of top-of-the-line servers.
>>
>> It seems to me that anycast is probably much worse in the Mirai botnet
>> scenario unless each node is pretty much as robust as a traditional
>> unicast node.
>
> I couldn't really say whether unicast is more or less resistant to this
> sort of attack -- I'd guess either way it would be down to the capacity
> at each individual node.
>
> It was Dyn's USA POPs that bore the brunt of the attack, presumably
> because most of the Mirai bots were located in the USA.  Even so, it
> still caused us plenty of grief in Europe.  Apparently the effects were
> fairly minimal in the Far East.
>

That makes one wonder if the EU Anycast nodes are reliant on the USA
node(s).  I have no insights (and even less DNS knowledge) but it
makes one wonder if there's a fundamental design flaw in anycast DNS
that relies on one or more nodes... is anycast DNS really just
distributed cache DNS?

-Jim P.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Matthew Seaman]

On 2016/10/31 14:53, Jim Popovitch wrote:
> On Mon, Oct 31, 2016 at 10:25 AM, Matthew Seaman
>  wrote:
>> This despite the fact that Dyn has a global anycast network with
>> plenty of bandwidth, points of presence all round the world and
>> each POP contains a bunch of top-of-the-line servers.
> 
> It seems to me that anycast is probably much worse in the Mirai botnet
> scenario unless each node is pretty much as robust as a traditional
> unicast node.

I couldn't really say whether unicast is more or less resistant to this
sort of attack -- I'd guess either way it would be down to the capacity
at each individual node.

It was Dyn's USA POPs that bore the brunt of the attack, presumably
because most of the Mirai bots were located in the USA.  Even so, it
still caused us plenty of grief in Europe.  Apparently the effects were
fairly minimal in the Far East.

Cheers,

Matthew






signature.asc
Description: OpenPGP digital signature
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Jim Popovitch]

On Mon, Oct 31, 2016 at 10:25 AM, Matthew Seaman
 wrote:
> This despite the fact that Dyn has a global anycast network with
> plenty of bandwidth, points of presence all round the world and
> each POP contains a bunch of top-of-the-line servers.

It seems to me that anycast is probably much worse in the Mirai botnet
scenario unless each node is pretty much as robust as a traditional
unicast node.

-Jim P.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: The DDOS attack on DYN & RRL ?

[Matthew Seaman]

On 10/31/16 12:41, MURTARI, JOHN wrote:
> God only knows, the DDOS hackers are probably on this listbut I
> have to ask what protections DYN had in place before the attack
> occurred.  RRL has been promoted as some protection against these
> types of attacks.  If they had it in place, did it help or was the
> pure volume of traffic the real issue?

Having been burned by the DDoS I can tell you that 'RRL' functionality
was pretty much irrelevant in this case.  This was not using DNS servers
as traffic amplifiers (which is what RRL mitigates against).

This was using millions of insecure IoT devices -- frequently web cams
-- to generate a massive overkill-level traffic surge -- lots of DNS
lookups -- that simply overwhelmed Dyn's servers.  This despite the fact
that Dyn has a global anycast network with plenty of bandwidth, points
of presence all round the world and each POP contains a bunch of
top-of-the-line servers.

Surviving DDoS is all about having more capacity available than your
attackers can fill up[*].  These Mirai botnets have upped the ante by a
wide margin.  I suspect that the DDoS protection companies, the big DNS
service providers, the TLD and the root operators are quietly but
franticly working on plans to beef up their defenses...

Cheers,

Matthew

[*] Even by proxy: anti-DDoS companies essentially have network capacity
available for hire as well as some pretty fancy traffic filtering
techniques.



signature.asc
Description: OpenPGP digital signature
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

The DDOS attack on DYN & RRL ?

[MURTARI, JOHN]

Folks,
God only knows, the DDOS hackers are probably on this 
listbut I have to ask what protections DYN had in place before the attack 
occurred.  RRL has been promoted as some protection against these types of 
attacks.  If they had it in place, did it help or was the pure volume of 
traffic the real issue?

I know companies are loathe to share info with competitors, but 
the wolves are out there and we sheep need to communicate a bit with each other.

I think ISC does have secure channels with large corporate 
users.  Is something in place to share attack info, protections that were used? 
  Might be a good idea.

Best regards!


John Murtari - jm5...@att.com
Ciberspring
office: 315-944-0998
cell: 315-430-2702

___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users