Re: Dyn DDoS this AM?

[Mike Hammett]

Side note: I asked Mikrotik and they accepted the feature request of changing 
their uRPF setting from being universal on the machine to being per-interface 
(as the kernel supports). That would make it easier for Mikrotik 
end-user-facing routers to block crap right at the edge, allowing for strict 
facing customer and loose elsewhere. They haven't implemented it yet, but they 
accepted the request. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Alexander Lyamin" <l...@qrator.net> 
To: "Ronald F. Guilmette" <r...@tristatelogic.com> 
Cc: "NANOG list" <nanog@nanog.org> 
Sent: Tuesday, October 25, 2016 3:29:56 AM 
Subject: Re: Dyn DDoS this AM? 

Yeah, it sucked to be a Dyn customer that day. However, if you had a 
backup dns provider, it wasnt that bad. 

You do realize that collateral effect scale is a property of a target and 
not attack? 

My point was that implementing MANRS, while isn't covering all of the 
spectrum of the attacks that made news this autumn will make at least some 
of them if not impossible, but harder to execute. 

And as I said - its work in progress. 

P.S. Jared Mauch notes regarding uRPF underperformance are correct, but it 
only shows how rarely its actually used in a real life. uRPF is more then 
feasible in terms of algorithmical complexity, and this means that bugs can 
be dealed with. 



On Tue, Oct 25, 2016 at 7:30 AM, Ronald F. Guilmette <r...@tristatelogic.com> 
wrote: 

> 
> In message 

Re: Dyn DDoS this AM?

[Alexander Lyamin]

Yeah, it sucked to be a Dyn customer that day.  However, if you had a
backup dns provider, it wasnt that bad.

You do realize that collateral effect scale  is a property of a target and
not attack?

My point was that implementing MANRS, while isn't covering all of the
spectrum of the attacks that made news this autumn will make at least some
of them if not impossible, but harder to execute.

And as I said - its work in progress.

P.S.  Jared Mauch notes regarding uRPF underperformance are correct, but it
only shows how rarely its actually used in a real life.  uRPF is more then
feasible in terms of algorithmical complexity, and this means that bugs can
be dealed with.



On Tue, Oct 25, 2016 at 7:30 AM, Ronald F. Guilmette 
wrote:

>
> In message 

Re: Dyn DDoS this AM?

[Ronald F. Guilmette]

In message 

Re: Dyn DDoS this AM?

[Suzanne Woolf]

> On Oct 24, 2016, at 12:06 PM, Eitan Adler  wrote:
> 
> On 24 October 2016 at 01:25, LHC  wrote:
>> All this TTL talk makes me think.
>> 
>> Why not have two ttls - a 'must-recheck' (does not expire the record but 
>> forces a recheck; updates record if server replies & serial has incremented) 
>> and a 'must-delete' (cache will be stale at this point)?
> 
> If clients can't get one TTL correct what makes you think they will
> get a more complicated two TTL system correct?
> 

….To say nothing of resolvers that simply ignore server-side TTLs and set their 
own. 

For instance, 
https://www.icann.org/en/system/files/files/rssac-003-root-zone-ttls-21aug15-en.pdf
 

 “RSSAC 003: RSSAC Report on Root Zone TTLs” will tell you far more than you 
really want to know about TTLs and caching behavior, and some of it is specific 
to the root zone, but one of the key observations is "Root zone TTLs appear to 
not matter to most clients.”

Modern large-scale DNS is a fairly complex system. Speculating from here about 
how it behaved under attack in someone else’s network is interesting, and I 
look forward to more information from Dyn as they feel they can share it— but 
DDoS is a big enough fact of life for them and everyone else that if there was 
a simple answer, I think someone would be making a fortune on it already, or at 
least have filed the patents.


Suzanne
(speaking for myself)

Re: Dyn DDoS this AM?

[Wayne Bouchard]

See, that's the thing...

The key to victory here is to defeat the robots. Take away the
anonymity of proxies and trojan amplifiers and enforcement gets a lot
easier. Sadly, this war doesn't seem likely to be won anytime soon.
Especially since there are State entities using (and even deploying) a
number of these systems for use against other States and businesses
and/or financial mechanisms. So rather than help the community solve
the problem (for their own good, no less!), it is in their interests
to perpetuate it.

-Wayne

On Fri, Oct 21, 2016 at 05:37:08PM -0400, Alain Hebert wrote:
> Just a FYI,
> 
> That "horrific trend" has been happening since some techie got
> dissed on an IRC channel over 20 years ago.
> 
> He used a bunch of hosted putters to ICMP flood the IRC server.
> 
> Whatever the community is behind, until the carriers decide to wise
> up this will keep happening, that is without talking about the
> industries being developed around DDoSes events.
> 
> Enjoy your weekend. ( I ain't on call anymore anyway =D )
> 
> -
> Alain Hebertaheb...@pubnix.net   
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443
> 
> On 10/21/16 11:52, Brian Davies via NANOG wrote:
> > +1!
> >
> > Well said, Patrick.
> >
> > B
> >
> > On Friday, October 21, 2016, Patrick W. Gilmore  wrote:
> >
> >> I cannot give additional info other than what???s been on ???public 
> >> media???.
> >>
> >> However, I would very much like to say that this is a horrific trend on
> >> the Internet. The idea that someone can mention a DDoS then get DDoS???ed 
> >> Can
> >> Not Stand. See Krebs??? on the Democratization of Censorship. See lots of
> >> other things.
> >>
> >> To Dyn and everyone else being attacked:
> >> The community is behind you. There are problems, but if we stick together,
> >> we can beat these miscreants.
> >>
> >> To the miscreants:
> >> You will not succeed. Search "churchill on the beaches???. It???s a bit
> >> melodramatic, but it???s how I feel at this moment.
> >>
> >> To the rest of the community:
> >> If you can help, please do. I know a lot of you are thinking ???what can I
> >> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
> >> that doesn???t help Mirai, but it still helps. There are many other things
> >> you can do as well.
> >>
> >> But a lot of it is just willingness to help. When someone asks you to help
> >> trace an attack, do not let the request sit for a while. Damage is being
> >> done. Help your neighbor. When someone???s house is burning, your current
> >> project, your lunch break, whatever else you are doing is almost certainly
> >> less important. If we stick together and help each other, we can - we WILL
> >> - win this war. If we are apathetic, we have already lost.
> >>
> >>
> >> OK, enough motivational speaking for today. But take this to heart. Our
> >> biggest problem is people thinking they cannot or do not want to help.
> >>
> >> --
> >> TTFN,
> >> patrick
> >>
> >>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  >> > wrote:
> >>> Does anyone have any additional details? Seems to be over now, but I'm
> >> very
> >>> curious about the specifics of such a highly impactful attack (and it's
> >>> timing following NANOG 68)...
> >>>
> >>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
> >> twitter-spotify-reddit/
> >>> --
> >>> @ChrisGrundemann
> >>> http://chrisgrundemann.com
> >>

---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/

Re: Dyn DDoS this AM?

[Jared Mauch]

On Fri, Oct 21, 2016 at 12:30:44PM -0400, Alain Hebert wrote:
> Rofl,
> 
> Yeah good luck with that... 15+ years later and most of the actors
> that could fix that, for the planete, still refuses to do anything.
> 
> Now you can start the usual circular discussion that goes nowhere
> after 3 days...
> 
> PS: yeah usual BCP38 rant... but its friday.

Not all attacks are BCP38 related. :-)

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Dyn DDoS this AM?

[Jared Mauch]

On Mon, Oct 24, 2016 at 02:38:58PM -0400, Alain Hebert wrote:
> And its not the last time the big Tier(s) will refuse to do anything
> beside dropping the fault to the CPE vendors.

I can say that we had to drop uRPF for technical reasons,
namely not enough people ask their vendors about it so it is:

a) not tested (at all)
b) not performance rated
c) lacks simple fixes

The people who have the purchasing power are not the tier-1
carriers regardless.  We push as hard as we can and end up with the
compromises as a result.

- Jared

Re: Dyn DDoS this AM?

[Alain Hebert]

And its not the last time the big Tier(s) will refuse to do anything
beside dropping the fault to the CPE vendors.

People love circles.

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 10/24/16 14:12, Alexander Lyamin wrote:
> Its not a first time we have and large scale DDoS incident.
> Its not a first time we have  (a kind of) knee-jerk reaction.
>
> I think its a right time to direct community attention to this  document
>
> https://www.routingmanifesto.org/manrs/
>
> It's  work in progress. But its a good start.
>
>
>
> On Fri, Oct 21, 2016 at 5:48 PM, Patrick W. Gilmore 
> wrote:
>
>> I cannot give additional info other than what’s been on “public media”.
>>
>> However, I would very much like to say that this is a horrific trend on
>> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
>> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
>> other things.
>>
>> To Dyn and everyone else being attacked:
>> The community is behind you. There are problems, but if we stick together,
>> we can beat these miscreants.
>>
>> To the miscreants:
>> You will not succeed. Search "churchill on the beaches”. It’s a bit
>> melodramatic, but it’s how I feel at this moment.
>>
>> To the rest of the community:
>> If you can help, please do. I know a lot of you are thinking “what can I
>> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
>> that doesn’t help Mirai, but it still helps. There are many other things
>> you can do as well.
>>
>> But a lot of it is just willingness to help. When someone asks you to help
>> trace an attack, do not let the request sit for a while. Damage is being
>> done. Help your neighbor. When someone’s house is burning, your current
>> project, your lunch break, whatever else you are doing is almost certainly
>> less important. If we stick together and help each other, we can - we WILL
>> - win this war. If we are apathetic, we have already lost.
>>
>>
>> OK, enough motivational speaking for today. But take this to heart. Our
>> biggest problem is people thinking they cannot or do not want to help.
>>
>> --
>> TTFN,
>> patrick
>>
>>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann 
>> wrote:
>>> Does anyone have any additional details? Seems to be over now, but I'm
>> very
>>> curious about the specifics of such a highly impactful attack (and it's
>>> timing following NANOG 68)...
>>>
>>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
>> twitter-spotify-reddit/
>>> --
>>> @ChrisGrundemann
>>> http://chrisgrundemann.com
>>
>

Re: Dyn DDoS this AM?

[Alexander Lyamin]

Its not a first time we have and large scale DDoS incident.
Its not a first time we have  (a kind of) knee-jerk reaction.

I think its a right time to direct community attention to this  document

https://www.routingmanifesto.org/manrs/

It's  work in progress. But its a good start.



On Fri, Oct 21, 2016 at 5:48 PM, Patrick W. Gilmore 
wrote:

> I cannot give additional info other than what’s been on “public media”.
>
> However, I would very much like to say that this is a horrific trend on
> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
> other things.
>
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together,
> we can beat these miscreants.
>
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit
> melodramatic, but it’s how I feel at this moment.
>
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I
> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
> that doesn’t help Mirai, but it still helps. There are many other things
> you can do as well.
>
> But a lot of it is just willingness to help. When someone asks you to help
> trace an attack, do not let the request sit for a while. Damage is being
> done. Help your neighbor. When someone’s house is burning, your current
> project, your lunch break, whatever else you are doing is almost certainly
> less important. If we stick together and help each other, we can - we WILL
> - win this war. If we are apathetic, we have already lost.
>
>
> OK, enough motivational speaking for today. But take this to heart. Our
> biggest problem is people thinking they cannot or do not want to help.
>
> --
> TTFN,
> patrick
>
> > On Oct 21, 2016, at 10:55 AM, Chris Grundemann 
> wrote:
> >
> > Does anyone have any additional details? Seems to be over now, but I'm
> very
> > curious about the specifics of such a highly impactful attack (and it's
> > timing following NANOG 68)...
> >
> > https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
> twitter-spotify-reddit/
> >
> > --
> > @ChrisGrundemann
> > http://chrisgrundemann.com
>
>


-- 

Alexander Lyamin

CEO | Qrator * Labs*

office: 8-800--LAB (522)

mob: +7-916-9086122

skype: melanor9

mailto:  l...@qrator.net

Re: Dyn DDoS this AM?

[Eitan Adler]

On 24 October 2016 at 01:25, LHC  wrote:
> All this TTL talk makes me think.
>
> Why not have two ttls - a 'must-recheck' (does not expire the record but 
> forces a recheck; updates record if server replies & serial has incremented) 
> and a 'must-delete' (cache will be stale at this point)?

If clients can't get one TTL correct what makes you think they will
get a more complicated two TTL system correct?


-- 
Eitan Adler

Re: Dyn DDoS this AM?

[LHC]

All this TTL talk makes me think.

Why not have two ttls - a 'must-recheck' (does not expire the record but forces 
a recheck; updates record if server replies & serial has incremented) and a 
'must-delete' (cache will be stale at this point)?

On October 23, 2016 3:42:58 PM PDT, Mark Andrews  wrote:
>
>In message

Re: Dyn DDoS this AM?

[LHC]

All this TTL talk makes me think.

Why not have two ttls - a 'must-recheck' (does not expire the record but forces 
a recheck; updates record if server replies & serial has incremented) and a 
'must-delete' (cache will be stale at this point)?

On October 23, 2016 3:42:58 PM PDT, Mark Andrews  wrote:
>
>In message

Re: Dyn DDoS this AM?

[Mark Andrews]

In message 

Re: Dyn DDoS this AM?

[Rob Szarka]

On 10/21/2016 7:34 PM, Keenan Tims wrote:
I don't have a horse in this race, and haven't used it in anger, but 
Netflix released denominator to attempt to deal with some of these 
issues:


https://github.com/Netflix/denominator

Their goal is to support the highest common denominator of features 
among the supported providers,


Maybe that helps someone.


Sadly, it looks like the project is stalled: 
.


--
Rob Szarka
http://szarka.org/

Re: Dyn DDoS this AM?

[Masood Ahmad Shah]

>
> > On Oct 21, 2016, at 6:35 PM, Eitan Adler  wrote:
> >
> > [...]
> >
> > In practice TTLs tend to be ignored on the public internet. In past
> > research I've been involved with browser[0] behavior was effectively
> > random despite the TTL set.
> >
> > [0] more specifically, the chain of DNS resolution and caching down to
> > the browser.
>
>
> Yes, but that it can be both better and worse than your TTLs does not mean
> that you can ignore properly working implementations.
>
> If the other end device chain breaks you that's their fault and out of
> your control.  If your own settings break you that's your fault.
>

+1 to what George wrote that we should make efforts to improve our part of
the network. There are ISPs that ignore TTL settings and only update their
cached records every two to three days or even more (particularly the
smaller ones). OTOH, this results in your DNS data being inconsistent but
it’s very common to cache DNS records at multiple levels. It's an effort
that everyone needs to contribute to.


>
> Sent from my iPhone

Re: Dyn DDoS this AM?

[Daniel Ankers]

On 22 October 2016 at 16:40, marcel.duregards--- via NANOG 
wrote:

> What about BCP38+84 on 30 tier-1 instead of asking/hoping 55k others
> autonomous-system having good filters in place ?


The originating ISPs are in a far better position to check that traffic
isn't from spoofed address ranges than transit networks are.  The best
thing to do is to ask EVERY network to do what they can, not just the few
biggest ones.

Any size ISP can be hit by and hurt by DDoS attacks, so every size ISP
should be doing what they can to make sure they are not either the source
or the victim of those attacks.

Dan

Re: Dyn DDoS this AM?

[marcel.duregards--- via NANOG]

Patrick,

We are client of 3 tier1. On our netflow collector, we can observe that
RFC1918 sources ip traffic is entering our AS via 2 of those tier-1.
Yes, 2 bigs tier-1 allow private ip traffic coming from their networks,
clients, peerings to reach others customers, via Internet link, on
public ip.Of course this traffic is dropped on our BGP borders as we
are filtering. But it's still filling the pipe, and this is still
INVALID/UNNAUTHORIZED traffic.

We wrote to them to verify if customers are technically allowed to send
RFC1918 traffic over their backbone, and if we are also allowed to do
so. And the answer was really evasive like :"contractually you're are
not allowed".

So now tell me WTF BCP38 will provide you when tier1 does not care at
all and does not maintain basic filtering to/from their customers.
And then they try to sell you their anti ddos services, because you know
DDOS it sucks. Big joke.

What about BCP38+84 on 30 tier-1 instead of asking/hoping 55k others
autonomous-system having good filters in place ?

--
Marcel

On 21.10.2016 17:48, Patrick W. Gilmore wrote:
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well.

Re: Dyn DDoS this AM?

[Ken Chase]

(Inband signalling - bad except for BGP?)

General comment: why are we blaming the client devices for the lack of security?

This is like Microsoft villifying linux in the late 90s because "there's no
restrictions on use or packet crafting on the client side" - of course there
isn't, in Windows either -- cant trust the client side, ever. Check out online
gaming, so many h4x 'n bots.

Let's stop trying to fix the clients, there'll always be bad actors/crappy 
coding.

Let's fix the networks. 

Pay-to-play? People are sensitive in the pocketbooks. NetCoin or something to
purchase dataflows? I dont know. Also sounds terrible. ("That's an internet
tax!!!111"). But Something Must Be Done[tm], by us, soon, or we'll be
dealing with govt cures which will be worse than the disease.

Regulating devices will never happen. Have you checked out world trade
regulations?  The US can't get Chinese firms to stop shipping
deadly-to-the-touch chemwep/drug carfentanil, how we gonna enforce security
standards on COTS electronics? (More govt soln's/approvals too. Fear.)

We have control of the networks. Lets do something.

(cant find the carfentanil story on nytimes anymore, pulled?
http://www.nytimes.com/aponline/2016/10/07/world/asia/ap-as-china-chemical-weapons.html
 )

/kc


On Sat, Oct 22, 2016 at 04:54:47PM +0200, Mikael Abrahamsson said:
  >On Sat, 22 Oct 2016, Alexander Maassen wrote:
  >
  >>Remember ping packets containing +++ATH0 ?
  >
  >THat only worked because of patents:
  >
  >https://en.wikipedia.org/wiki/Time_Independent_Escape_Sequence
  >
  >Inband signaling is bad, mmmkay?
  >
  >-- 
  >Mikael Abrahamssonemail: swm...@swm.pp.se

--
Ken Chase - Guelph Canada

Re: Dyn DDoS this AM?

[Florian Weimer]

* Randy Bush:

> anyone who relies on a single dns provider is just asking for stuff such
> as this.

Blaming the victim isn't helpful.  And without end-user-visible
changes, most of the victims would still depend on Verisign as a
single provider for a critical part of their DNS service.

Re: Dyn DDoS this AM?

[Mikael Abrahamsson]

On Sat, 22 Oct 2016, Alexander Maassen wrote:


Remember ping packets containing +++ATH0 ?


THat only worked because of patents:

https://en.wikipedia.org/wiki/Time_Independent_Escape_Sequence

Inband signaling is bad, mmmkay?

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Dyn DDoS this AM?

[Alexander Maassen]

Remember ping packets containing +++ATH0 ?
Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- 
Peplink Certified Engineer

 Oorspronkelijk bericht Van: Alain Hebert <aheb...@pubnix.net> 
Datum: 21-10-16  23:37  (GMT+01:00) Aan: nanog@nanog.org Onderwerp: Re: Dyn 
DDoS this AM? 
    Just a FYI,

    That "horrific trend" has been happening since some techie got
dissed on an IRC channel over 20 years ago.

    He used a bunch of hosted putters to ICMP flood the IRC server.

    Whatever the community is behind, until the carriers decide to wise
up this will keep happening, that is without talking about the
industries being developed around DDoSes events.

    Enjoy your weekend. ( I ain't on call anymore anyway =D )

-
Alain Hebert    aheb...@pubnix.net   
PubNIX Inc.    
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.net    Fax: 514-990-9443

On 10/21/16 11:52, Brian Davies via NANOG wrote:
> +1!
>
> Well said, Patrick.
>
> B
>
> On Friday, October 21, 2016, Patrick W. Gilmore <patr...@ianai.net> wrote:
>
>> I cannot give additional info other than what’s been on “public media”.
>>
>> However, I would very much like to say that this is a horrific trend on
>> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
>> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
>> other things.
>>
>> To Dyn and everyone else being attacked:
>> The community is behind you. There are problems, but if we stick together,
>> we can beat these miscreants.
>>
>> To the miscreants:
>> You will not succeed. Search "churchill on the beaches”. It’s a bit
>> melodramatic, but it’s how I feel at this moment.
>>
>> To the rest of the community:
>> If you can help, please do. I know a lot of you are thinking “what can I
>> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
>> that doesn’t help Mirai, but it still helps. There are many other things
>> you can do as well.
>>
>> But a lot of it is just willingness to help. When someone asks you to help
>> trace an attack, do not let the request sit for a while. Damage is being
>> done. Help your neighbor. When someone’s house is burning, your current
>> project, your lunch break, whatever else you are doing is almost certainly
>> less important. If we stick together and help each other, we can - we WILL
>> - win this war. If we are apathetic, we have already lost.
>>
>>
>> OK, enough motivational speaking for today. But take this to heart. Our
>> biggest problem is people thinking they cannot or do not want to help.
>>
>> --
>> TTFN,
>> patrick
>>
>>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com
>> <javascript:;>> wrote:
>>> Does anyone have any additional details? Seems to be over now, but I'm
>> very
>>> curious about the specifics of such a highly impactful attack (and it's
>>> timing following NANOG 68)...
>>>
>>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
>> twitter-spotify-reddit/
>>> --
>>> @ChrisGrundemann
>>> http://chrisgrundemann.com
>>

Re: Dyn DDoS this AM?

[George William Herbert]

Oh god, you invoked @popehat ...

[dyndds and its customers sue XiongMai, the OEM integrators, and Does 
1-10,000,000 who own the devices for neglegence?...]

Sent from my iPhone

> On Oct 21, 2016, at 8:29 PM, Chris Woodfield  wrote:
> 
> As a Twitter network  engineer (and the guy Patrick let camp out in your 
> hotel room all day) - thank you for this. Whoever was behind this just poked 
> a hornet’s nest. 
> 
> “Govern yourselves accordingly”.
> 
> -C
> 
> (Obviously speaking for myself, not my employer…)
> 
>> On Oct 21, 2016, at 10:48 AM, Patrick W. Gilmore  wrote:
>> 
>> I cannot give additional info other than what’s been on “public media”.
>> 
>> However, I would very much like to say that this is a horrific trend on the 
>> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
>> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
>> things.
>> 
>> To Dyn and everyone else being attacked:
>> The community is behind you. There are problems, but if we stick together, 
>> we can beat these miscreants.
>> 
>> To the miscreants:
>> You will not succeed. Search "churchill on the beaches”. It’s a bit 
>> melodramatic, but it’s how I feel at this moment.
>> 
>> To the rest of the community:
>> If you can help, please do. I know a lot of you are thinking “what can I 
>> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, 
>> that doesn’t help Mirai, but it still helps. There are many other things you 
>> can do as well.
>> 
>> But a lot of it is just willingness to help. When someone asks you to help 
>> trace an attack, do not let the request sit for a while. Damage is being 
>> done. Help your neighbor. When someone’s house is burning, your current 
>> project, your lunch break, whatever else you are doing is almost certainly 
>> less important. If we stick together and help each other, we can - we WILL - 
>> win this war. If we are apathetic, we have already lost.
>> 
>> 
>> OK, enough motivational speaking for today. But take this to heart. Our 
>> biggest problem is people thinking they cannot or do not want to help.
>> 
>> -- 
>> TTFN,
>> patrick
>> 
>>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  
>>> wrote:
>>> 
>>> Does anyone have any additional details? Seems to be over now, but I'm very
>>> curious about the specifics of such a highly impactful attack (and it's
>>> timing following NANOG 68)...
>>> 
>>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>>> 
>>> -- 
>>> @ChrisGrundemann
>>> http://chrisgrundemann.com
> 

Re: Dyn DDoS this AM? - dns

[alvin nanog]

On 10/21/16 at 03:21pm, David Birdsong wrote:
> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
> > anyone who relies on a single dns provider is just asking for stuff such
> > as this.

:-)

> I'd love to hear how others are handling the overhead of managing two dns
> providers.

in my view of ( automated ) dns managment:

Only on the one "master" dns server, make your DNS changes, update the 
serial number for example.com changes and reload the new update zone
file ... notifications goes out to all known slave DNS servers ..

For all the other authorized DNS servers, they should all automatically 
update itself ... magic all dns servers are in sync ...

some folks don't like "master" DNS server vs slaves .. i donno why not ..

but, you do have to configure your "master dns server" properly to 
only allow only authorized slaves access to their dns reccords

similarly, slave DNS servers should only update from it's recognized
master dns server

there should be zero isues with managing 2 dns server or 100 dns servers

before downloading new dns info, Man-in-the-Middle tests with OpenSSL 
certs should be done to confirm the other end is in fact who you think
it is that you're going to be sending dns info to or receiving from

c ya
alvin
http://DDoS-Mitigator.net 

Re: Dyn DDoS this AM?

[Chris Woodfield]

As a Twitter network  engineer (and the guy Patrick let camp out in your hotel 
room all day) - thank you for this. Whoever was behind this just poked a 
hornet’s nest. 

“Govern yourselves accordingly”.

-C

(Obviously speaking for myself, not my employer…)

> On Oct 21, 2016, at 10:48 AM, Patrick W. Gilmore  wrote:
> 
> I cannot give additional info other than what’s been on “public media”.
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things.
> 
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants.
> 
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment.
> 
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well.
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost.
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help.
> 
> -- 
> TTFN,
> patrick
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very
>> curious about the specifics of such a highly impactful attack (and it's
>> timing following NANOG 68)...
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>> 
>> -- 
>> @ChrisGrundemann
>> http://chrisgrundemann.com
> 

Re: Dyn DDoS this AM?

[Yang Yu]

On Fri, Oct 21, 2016 at 11:45 AM, Patrick W. Gilmore  wrote:
> My guess is you should track anything to as33517.

And AS15135?

Re: Dyn DDoS this AM?

[George William Herbert]

> On Oct 21, 2016, at 6:35 PM, Eitan Adler  wrote:
> 
> [...]
> 
> In practice TTLs tend to be ignored on the public internet. In past
> research I've been involved with browser[0] behavior was effectively
> random despite the TTL set.
> 
> [0] more specifically, the chain of DNS resolution and caching down to
> the browser.


Yes, but that it can be both better and worse than your TTLs does not mean that 
you can ignore properly working implementations.

If the other end device chain breaks you that's their fault and out of your 
control.  If your own settings break you that's your fault.


Sent from my iPhone

Re: Dyn DDoS this AM?

[Eitan Adler]

On 21 October 2016 at 18:12, Jean-Francois Mezei
 wrote:
> On 2016-10-21 18:45, david raistrick wrote:
>
>> switch too..).   setting TTLs that make sense for a design that supports
>> change is also easy.
>
> Cuts both ways. Had Twitter had TTLs of say 7 days, vast majority
> wouldn't notice an outage of a few hours because their local cache wa
> still valid.

In practice TTLs tend to be ignored on the public internet. In past
research I've been involved with browser[0] behavior was effectively
random despite the TTL set.

[0] more specifically, the chain of DNS resolution and caching down to
the browser.


-- 
Eitan Adler

Re: Dyn DDoS this AM?

[Jean-Francois Mezei]

On 2016-10-21 18:45, david raistrick wrote:

> switch too..).   setting TTLs that make sense for a design that supports
> change is also easy.

Cuts both ways. Had Twitter had TTLs of say 7 days, vast majority
wouldn't notice an outage of a few hours because their local cache wa
still valid.

It does prevent one from reacting quickly to emergencies.

Re: Dyn DDoS this AM?

[Brett Frankenberger]

On Fri, Oct 21, 2016 at 05:11:34PM -0700, Crist Clark wrote:
>
> Given the scale of these attacks, whether having two providers does any
> good may be a crap shoot.
> 
> That is, what if the target happens to share the same providers you do?
> Given the whole asymmetry of resources that make this a problem in the
> first place, the attackers probably have the resources to take out multiple
> providers.
> 
> Having multiple providers may reduce your chance of being collateral damage
> (and I'd also still worry more about the more mundane risks of a single
> provider, maintenance or upgrade gone bad, business risks, etc., than these
> sensational ones), but multiple providers likely won't save you if you are
> the actual target of the attack.

Good, perfect, enemy, etc.

How many sites were down today?  How many were the intended target?

 -- Brett

Re: Dyn DDoS this AM?

[Crist Clark]

Given the scale of these attacks, whether having two providers does any
good may be a crap shoot.

That is, what if the target happens to share the same providers you do?
Given the whole asymmetry of resources that make this a problem in the
first place, the attackers probably have the resources to take out multiple
providers.

Having multiple providers may reduce your chance of being collateral damage
(and I'd also still worry more about the more mundane risks of a single
provider, maintenance or upgrade gone bad, business risks, etc., than these
sensational ones), but multiple providers likely won't save you if you are
the actual target of the attack.


On Fri, Oct 21, 2016 at 4:45 PM, Måns Nilsson <mansa...@besserwisser.org>
wrote:

> Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:37:09AM +0200
> Quoting Niels Bakker (ni...@bakker.net):
> > * mansa...@besserwisser.org (Måns Nilsson) [Sat 22 Oct 2016, 01:27
> CEST]:
> > >Also, do not fall in the "short TTL for service agility" trap.
> >
> > Several CDNs, Akamai among them, do use short TTLs for this exact reason.
> > Server load is constantly monitored and taken into account when crafting
> DNS
> > replies.
>
> But the problem is that this trashes caching, and DNS does not work
> without caches. At least not if you want it to survive when the going
> gets tough.
>
> If we're going to solve this we need to innovate beyond the pathetic
> CNAME chains that todays managed DNS services make us use, and get truly
> distributed load-balancing decision-making (which only will work if you
> give it sensible data; a single CNAME is not sensible data) all the way
> out in the client application.
>
> --
> Måns Nilsson primary/secondary/besserwisser/machina
> MN-1334-RIPE +46 705 989668
> Well, I'm INVISIBLE AGAIN ... I might as well pay a visit to the LADIES
> ROOM ...
>

Re: Dyn DDoS this AM?

[Josh Reynolds]

Ah, disregard. I see what you're saying now.

Yes, I can see how that would be problematic.

On Oct 21, 2016 6:40 PM, "Josh Reynolds"  wrote:

> Ansible would be a decent start.
>
> On Oct 21, 2016 5:26 PM, "David Birdsong"  wrote:
>
>> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>>
>> > anyone who relies on a single dns provider is just asking for stuff such
>> > as this.
>> >
>> > randy
>> >
>>
>> I'd love to hear how others are handling the overhead of managing two dns
>> providers. Every time we brainstorm on it, we see it as blackhole of eng
>> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
>> entire delegation over.
>>
>

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:37:09AM +0200 
Quoting Niels Bakker (ni...@bakker.net):
> * mansa...@besserwisser.org (Måns Nilsson) [Sat 22 Oct 2016, 01:27 CEST]:
> >Also, do not fall in the "short TTL for service agility" trap.
> 
> Several CDNs, Akamai among them, do use short TTLs for this exact reason.
> Server load is constantly monitored and taken into account when crafting DNS
> replies.

But the problem is that this trashes caching, and DNS does not work
without caches. At least not if you want it to survive when the going
gets tough. 

If we're going to solve this we need to innovate beyond the pathetic
CNAME chains that todays managed DNS services make us use, and get truly
distributed load-balancing decision-making (which only will work if you
give it sensible data; a single CNAME is not sensible data) all the way
out in the client application. 

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Well, I'm INVISIBLE AGAIN ... I might as well pay a visit to the LADIES
ROOM ...


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Josh Reynolds]

Ansible would be a decent start.

On Oct 21, 2016 5:26 PM, "David Birdsong"  wrote:

> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>
> > anyone who relies on a single dns provider is just asking for stuff such
> > as this.
> >
> > randy
> >
>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.
>

Re: Dyn DDoS this AM?

[Keenan Tims]

I don't have a horse in this race, and haven't used it in anger, but 
Netflix released denominator to attempt to deal with some of these issues:


https://github.com/Netflix/denominator

Their goal is to support the highest common denominator of features 
among the supported providers,


Maybe that helps someone.

Keenan

On 2016-10-21 16:19, Niels Bakker wrote:

The point of outsourcing DNS isn't just availability of static
hostnames, it's the added services delivered, like returning different
answers based on source of the question, even monitoring your
infrastructure (or it reporting load into the DNS management system).

That is very hard to replicate with two DNS providers.


-- Niels.

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:19:24AM +0200 
Quoting Niels Bakker (niels=na...@bakker.net):

> The point of outsourcing DNS isn't just availability of static hostnames,
> it's the added services delivered, like returning different answers based on
> source of the question, even monitoring your infrastructure (or it reporting
> load into the DNS management system).
> 
> That is very hard to replicate with two DNS providers.

Surely, it must be better to use a singular service that is provably
easy to take out. The advantages are overwhelming.

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Yow!  Are we wet yet?


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Fri, Oct 21, 2016 at 03:21:20PM -0700 
Quoting David Birdsong (da...@imgix.com):
> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush <ra...@psg.com> wrote:
> 
> > anyone who relies on a single dns provider is just asking for stuff such
> > as this.
> >
> > randy
> 
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.

The fault is giving up the primary for an API connection. Sure, it is
tempting. We do, however, need to push the "application-integrated"
DNS vendors harder. They need to give their customers more choice in
how the DNS is populated. 

They also very much need to let people with above-mentioned
"application-integrated" needs add third party DNS providers in the mix.
This diversity capability is what makes DNS resilient. Monocultures have
suboptimal survivability in the long run.

Adding DNS providers when you control the primary is completely
painless. With EDNS0 there's lots of room for insanely large NS RRSETs. 

Also, do not fall in the "short TTL for service agility" trap. 

Besides, what Randy wrote. 

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Hold the MAYO & pass the COSMIC AWARENESS ...


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Niels Bakker]

anyone who relies on a single dns provider is just asking for 
stuff such as this.
I'd love to hear how others are handling the overhead of managing 
two dns providers.


* ra...@psg.com (Randy Bush) [Sat 22 Oct 2016, 00:28 CEST]:
good question.  staying in-band, hidden primary comes to mind.  but 
i am sure clever minds can come up with more clever schemes.


The point of outsourcing DNS isn't just availability of static 
hostnames, it's the added services delivered, like returning different 
answers based on source of the question, even monitoring your 
infrastructure (or it reporting load into the DNS management system).


That is very hard to replicate with two DNS providers.


-- Niels.

Re: Dyn DDoS this AM?

[joel jaeggli]

On 10/21/16 3:21 PM, David Birdsong wrote:
> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>
>> anyone who relies on a single dns provider is just asking for stuff such
>> as this.
>>
>> randy
>>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.

Not all the ones you might choose based on scale support axfr... That's
a bit of a problem for the most traditional approach to this., of those 
that do it's straight-forward to use one as the master for another, or
use a hidden master. Your own master may have demonstrably lower
availability then one or the other of your providers. getting two well
considered choices to play nice with each other isn't that hard.





signature.asc
Description: OpenPGP digital signature

Re: Dyn DDoS this AM?

[david raistrick]

On Fri, Oct 21, 2016 at 6:21 PM, David Birdsong  wrote:

>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.
>


with the usual caveats - and I dont have any projects that currently need
this but have in the past - pretty much every major dns provider allows you
to ship them a full zone in some form or fashion.   The effort to pull and
ship a zone should be fairly minimal in and of itself.

mixing your public zone providers in your authoritative NS records is also
easy - and, depending on your registrar of choice, should be easy to manage
changing those (including having non-public mirrors maintained that you can
switch too..).   setting TTLs that make sense for a design that supports
change is also easy.

the real developmental and architectural challenges are around what to do
if the APIs you use to talk to your "primary" disappear and you need to
consume them (creating new host entries, updating loadbalancer pools,
whatever.  we all have different and sometimes very diverse use cases for
dns.).

one approach - as randy suggested - is to switch to a purely hidden and
self managed primary - which might mean running your own API stack in front
of it to control whatever you need to control and change.   this doesnt
need to be a "real" dns server in todays world - the days of BIND style
zone transfers are generally long gone anyway when you hit these scales and
levels of intra complexity.then your zone-replication components that
ship zone updates to your various external providers are shipping from the
same place.

at least in that case it's fully within your control - but dev time and
complexity definitely comes into play.

if your infra can survive internally without dns change control for the
extent of an outage, that could be much easier to manage.

anyway, random and incomplete thoughts - time ran out, work calls.


...david

Re: Dyn DDoS this AM?

[Nick Hilliard]

Patrick W. Gilmore wrote:
> Our biggest problem is people thinking they cannot or do not want to
> help.

Our biggest problem is that if the Internet community does not handle
problems like this, governments and regulators may decide to intervene.
 If they do this in the wrong way, it will turn one major headache into two.

Nick

Re: Dyn DDoS this AM?

[Randy Bush]

>> anyone who relies on a single dns provider is just asking for stuff such
>> as this.
> I'd love to hear how others are handling the overhead of managing two dns
> providers.

good question.  staying in-band, hidden primary comes to mind.  but i am
sure clever minds can come up with more clever schemes.

randy

Re: Dyn DDoS this AM?

[David Birdsong]

On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:

> anyone who relies on a single dns provider is just asking for stuff such
> as this.
>
> randy
>

I'd love to hear how others are handling the overhead of managing two dns
providers. Every time we brainstorm on it, we see it as blackhole of eng
effort WRT to keeping them in sync and and then waiting for TTLs to cut an
entire delegation over.

Re: Dyn DDoS this AM?

[Randy Bush]

> amen.
>> anyone who relies on a single dns provider is just asking for stuff
>> such as this.

part of the problem is that we think of it as attack surface when, in
fact, it usually has more than two dimensions.

randy

Re: Dyn DDoS this AM?

[Mehmet Akcin]

amen.

On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:

> anyone who relies on a single dns provider is just asking for stuff such
> as this.
>
> randy
>

Re: Dyn DDoS this AM?

[Andrew Fried]

The brutal reality in todays world is that anyone that relies on the
Internet is just asking for stuff like this.  No service is safe.

Andrew


Andrew Fried
andrew.fr...@gmail.com

On 10/21/16 5:58 PM, Randy Bush wrote:
> anyone who relies on a single dns provider is just asking for stuff such
> as this.
> 
> randy
> 

Re: Dyn DDoS this AM?

[Randy Bush]

anyone who relies on a single dns provider is just asking for stuff such
as this.

randy

Re: Dyn DDoS this AM?

[Alain Hebert]

Just a FYI,

That "horrific trend" has been happening since some techie got
dissed on an IRC channel over 20 years ago.

He used a bunch of hosted putters to ICMP flood the IRC server.

Whatever the community is behind, until the carriers decide to wise
up this will keep happening, that is without talking about the
industries being developed around DDoSes events.

Enjoy your weekend. ( I ain't on call anymore anyway =D )

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 10/21/16 11:52, Brian Davies via NANOG wrote:
> +1!
>
> Well said, Patrick.
>
> B
>
> On Friday, October 21, 2016, Patrick W. Gilmore  wrote:
>
>> I cannot give additional info other than what’s been on “public media”.
>>
>> However, I would very much like to say that this is a horrific trend on
>> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
>> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
>> other things.
>>
>> To Dyn and everyone else being attacked:
>> The community is behind you. There are problems, but if we stick together,
>> we can beat these miscreants.
>>
>> To the miscreants:
>> You will not succeed. Search "churchill on the beaches”. It’s a bit
>> melodramatic, but it’s how I feel at this moment.
>>
>> To the rest of the community:
>> If you can help, please do. I know a lot of you are thinking “what can I
>> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
>> that doesn’t help Mirai, but it still helps. There are many other things
>> you can do as well.
>>
>> But a lot of it is just willingness to help. When someone asks you to help
>> trace an attack, do not let the request sit for a while. Damage is being
>> done. Help your neighbor. When someone’s house is burning, your current
>> project, your lunch break, whatever else you are doing is almost certainly
>> less important. If we stick together and help each other, we can - we WILL
>> - win this war. If we are apathetic, we have already lost.
>>
>>
>> OK, enough motivational speaking for today. But take this to heart. Our
>> biggest problem is people thinking they cannot or do not want to help.
>>
>> --
>> TTFN,
>> patrick
>>
>>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann > > wrote:
>>> Does anyone have any additional details? Seems to be over now, but I'm
>> very
>>> curious about the specifics of such a highly impactful attack (and it's
>>> timing following NANOG 68)...
>>>
>>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
>> twitter-spotify-reddit/
>>> --
>>> @ChrisGrundemann
>>> http://chrisgrundemann.com
>>

Re: Dyn DDoS this AM?

[Brian Davies via NANOG]

+1!

Well said, Patrick.

B

On Friday, October 21, 2016, Patrick W. Gilmore  wrote:

> I cannot give additional info other than what’s been on “public media”.
>
> However, I would very much like to say that this is a horrific trend on
> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
> other things.
>
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together,
> we can beat these miscreants.
>
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit
> melodramatic, but it’s how I feel at this moment.
>
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I
> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
> that doesn’t help Mirai, but it still helps. There are many other things
> you can do as well.
>
> But a lot of it is just willingness to help. When someone asks you to help
> trace an attack, do not let the request sit for a while. Damage is being
> done. Help your neighbor. When someone’s house is burning, your current
> project, your lunch break, whatever else you are doing is almost certainly
> less important. If we stick together and help each other, we can - we WILL
> - win this war. If we are apathetic, we have already lost.
>
>
> OK, enough motivational speaking for today. But take this to heart. Our
> biggest problem is people thinking they cannot or do not want to help.
>
> --
> TTFN,
> patrick
>
> > On Oct 21, 2016, at 10:55 AM, Chris Grundemann  > wrote:
> >
> > Does anyone have any additional details? Seems to be over now, but I'm
> very
> > curious about the specifics of such a highly impactful attack (and it's
> > timing following NANOG 68)...
> >
> > https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
> twitter-spotify-reddit/
> >
> > --
> > @ChrisGrundemann
> > http://chrisgrundemann.com
>
>

Re: Dyn DDoS this AM?

[Steve Meuse]

On Fri, Oct 21, 2016 at 12:09 PM, Roland Dobbins  wrote:

> On 21 Oct 2016, at 23:01, Mike Hammett wrote:
>
> > Are there sites that can test your BCP38\84 compliance?
>
> 


Quick note: If anyone has this installed already on OSX, bring up the
console and see if it's still running. I discovered (while watching the
NANOG preso) that mine had an issue and was failing silently. Re-installing
the new version fixed the issue.

The funny part of the story, looking through the logs to see which networks
I roamed on that were spoofable, the only positive hit was for the NANOG
conference network in Chicago :)

-Steve

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

On Oct 21, 2016, at 12:40 PM, David Hubbard  
wrote:
> 
> Do we know the attack destinations so we can watch transit traffic destined 
> for it to help sources that may be unaware?

My guess is you should track anything to as33517.

-- 
TTFN,
patrick

Re: Dyn DDoS this AM?

[David Hubbard]

Do we know the attack destinations so we can watch transit traffic destined for 
it to help sources that may be unaware?

David

RE: Dyn DDoS this AM?

[Brandon Ross]

On Fri, 21 Oct 2016, rar wrote:


Anyone want a quick consulting gig helping us configure BCP38 and BCP84?

Configurations is all cisco
Edge routers connect to Verizon, Level 3 Fiber
Each Edge router talks to two BGP routers.

$150/hour, I'm guessing it is only an hour for somebody to explain, and 
guide us through the configuration, but OK if longer.


Sure, we'll do it.

That rate is quite a bit less than our normal retail rate, but in the 
spirit that Patrick posted about, Network Utility Force will be happy to 
provide you or any other operator resources at that rate to help configure 
BCP38 and BCP84.


Anyone serious about that, email me privately at br...@netuf.net and we'll 
put paperwork together.


--
Brandon Ross  Yahoo & AIM:  BrandonNRoss
Voice:  +1-404-635-6667ICQ:  2269442
Signal Secure SMS:  +1-404-644-9628  Skype:  brandonross
Schedule a meeting:  http://www.doodle.com/bross

Re: Dyn DDoS this AM?

[Alain Hebert]

Rofl,

Yeah good luck with that... 15+ years later and most of the actors
that could fix that, for the planete, still refuses to do anything.

Now you can start the usual circular discussion that goes nowhere
after 3 days...

PS: yeah usual BCP38 rant... but its friday.

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 10/21/16 12:12, Patrick W. Gilmore wrote:
> Attack has re-started. This is the time, folks. Rally the troops, offer help, 
> watch your flow.
>
> STOP THIS NOW.
>

Re: Dyn DDoS this AM?

[Seth Mattinen]

On 10/21/16 09:05, Matthew Black wrote:

LA Times: Why sites like Twitter and Spotify were down for East Coast users 
this morning
http://www.latimes.com/business/la-fi-tn-dyn-attack-20161021-snap-story.html



I actually can't resolve twitter.com this morning and I'm west coast. 
None of the four listed DNS servers are responding.


twitter.com.172800  IN  NS  ns1.p34.dynect.net.
twitter.com.172800  IN  NS  ns2.p34.dynect.net.
twitter.com.172800  IN  NS  ns3.p34.dynect.net.
twitter.com.172800  IN  NS  ns4.p34.dynect.net.

Trace routes seem to point towards San Jose or Palo Alto or Los Angeles.

~Seth

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

https://www.caida.org/projects/spoofer/ 
<https://www.caida.org/projects/spoofer/>

-- 
TTFN,
patrick

> On Oct 21, 2016, at 12:01 PM, Mike Hammett <na...@ics-il.net> wrote:
> 
> Are there sites that can test your BCP38\84 compliance? I'm okay, but 
> interested in what I can share to raise awareness. 
> 
> 
> 
> 
> - 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> Midwest-IX 
> http://www.midwest-ix.com 
> 
> - Original Message -
> 
> From: "Patrick W. Gilmore" <patr...@ianai.net> 
> To: "NANOG list" <nanog@nanog.org> 
> Sent: Friday, October 21, 2016 10:48:21 AM 
> Subject: Re: Dyn DDoS this AM? 
> 
> I cannot give additional info other than what’s been on “public media”. 
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things. 
> 
> To Dyn and everyone else being attacked: 
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants. 
> 
> To the miscreants: 
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment. 
> 
> To the rest of the community: 
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well. 
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost. 
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help. 
> 
> -- 
> TTFN, 
> patrick 
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com> 
>> wrote: 
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very 
>> curious about the specifics of such a highly impactful attack (and it's 
>> timing following NANOG 68)... 
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>>  
>> 
>> -- 
>> @ChrisGrundemann 
>> http://chrisgrundemann.com 
> 

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

Attack has re-started. This is the time, folks. Rally the troops, offer help, 
watch your flow.

STOP THIS NOW.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 11:48 AM, Patrick W. Gilmore  wrote:
> 
> I cannot give additional info other than what’s been on “public media”.
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things.
> 
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants.
> 
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment.
> 
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well.
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost.
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help.
> 
> -- 
> TTFN,
> patrick
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann > > wrote:
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very
>> curious about the specifics of such a highly impactful attack (and it's
>> timing following NANOG 68)...
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>>  
>> 
>> 
>> -- 
>> @ChrisGrundemann
>> http://chrisgrundemann.com
> 

Re: Dyn DDoS this AM?

[Roland Dobbins]

On 21 Oct 2016, at 23:01, Mike Hammett wrote:

> Are there sites that can test your BCP38\84 compliance?



---
Roland Dobbins 

Re: Dyn DDoS this AM?

[Alexander Maassen]

Feel free to feed me with attack sources. Once those companies notice their 
precious mail does not arrive at clients. They will attempt to fix things. Sad 
but true.

Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- 
Peplink Certified Engineer

 Oorspronkelijk bericht Van: "Patrick W. Gilmore" 
<patr...@ianai.net> Datum: 21-10-16  17:48  (GMT+01:00) Aan: NANOG list 
<nanog@nanog.org> Onderwerp: Re: Dyn DDoS this AM? 
I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com> wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm very
> curious about the specifics of such a highly impactful attack (and it's
> timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
> 
> -- 
> @ChrisGrundemann
> http://chrisgrundemann.com

RE: Dyn DDoS this AM?

[Matthew Black]

LA Times: Why sites like Twitter and Spotify were down for East Coast users 
this morning
http://www.latimes.com/business/la-fi-tn-dyn-attack-20161021-snap-story.html




-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Chris Grundemann
Sent: Friday, October 21, 2016 7:56 AM
To: nanog@nanog.org
Subject: Dyn DDoS this AM?

Does anyone have any additional details? Seems to be over now, but I'm very
curious about the specifics of such a highly impactful attack (and it's
timing following NANOG 68)...

https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/

-- 
@ChrisGrundemann
http://chrisgrundemann.com

RE: Dyn DDoS this AM?

[rar]

Anyone want a quick consulting gig helping us configure BCP38 and BCP84?

Configurations is all cisco
Edge routers connect to Verizon, Level 3 Fiber
Each Edge router talks to two BGP routers.

$150/hour,  I'm guessing it is only an hour for somebody to explain, and guide 
us through the configuration, but OK if longer.

Thanks.


Bob Roswell
brosw...@syssrc.com
410-771-5544 ext 4336

Computer Museum Highlights

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Patrick W. Gilmore
Sent: Friday, October 21, 2016 11:48 AM
To: NANOG list <nanog@nanog.org>
Subject: Re: Dyn DDoS this AM?

I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

--
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com> wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm 
> very curious about the specifics of such a highly impactful attack 
> (and it's timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotif
> y-reddit/
> 
> --
> @ChrisGrundemann
> http://chrisgrundemann.com

Re: Dyn DDoS this AM?

[Mike Hammett]

Are there sites that can test your BCP38\84 compliance? I'm okay, but 
interested in what I can share to raise awareness. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Patrick W. Gilmore" <patr...@ianai.net> 
To: "NANOG list" <nanog@nanog.org> 
Sent: Friday, October 21, 2016 10:48:21 AM 
Subject: Re: Dyn DDoS this AM? 

I cannot give additional info other than what’s been on “public media”. 

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things. 

To Dyn and everyone else being attacked: 
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants. 

To the miscreants: 
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment. 

To the rest of the community: 
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well. 

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost. 


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help. 

-- 
TTFN, 
patrick 

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com> wrote: 
> 
> Does anyone have any additional details? Seems to be over now, but I'm very 
> curious about the specifics of such a highly impactful attack (and it's 
> timing following NANOG 68)... 
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>  
> 
> -- 
> @ChrisGrundemann 
> http://chrisgrundemann.com 

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm very
> curious about the specifics of such a highly impactful attack (and it's
> timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
> 
> -- 
> @ChrisGrundemann
> http://chrisgrundemann.com