Re: Death of the Internet, Film at 11

[Ronald F. Guilmette]

Laszlo Hanyecz wrote:

>What does BCP38 have to do with this?

Your're right.  That's not specifically related to *this* attack.  Nobody
needs to spoof anything when you've got a zillion fire hoses just lying
around where any 13 year old can command them from the TRS 80 in his mom's
basement.  (I've seen different estimates today.  One said there's about
a half million of these things, but I think I saw where Dyn itself put
the number of unique IPs in the attack at something like ten million.)

I just threw out BCP 38 as an example of something *very* minimal that
the collective Internet, if it had any brains, would have made de rigueur 
for everyone ten+ years ago.  BCP 38 is something that I personally view
as a "no brainer", that is already widely accepted as being necessary,
and yet is a critical security step that some (many?) are still resisting.
So, it's like "Well, if the Internet-at-large can't even do *this* simple
and relatively non-controversial thing, then we haven't got a prayer in
hell of ever seeing a world-wide determined push to find and neutralize
all of these bloody damn stupid CCTV things.  And when the day comes when
somebody figures out how to remotely pop a default config Windoze XP
box... boy oh boy, will *that* be a fun day... NOT!  Because we're not
ready.  Nobody's ready.  Except maybe DoD, and I'm not even taking bets
on that one."

I didn't intend to focus on BCP 38.  Everybody knows that's only one
thing, designed to deal with just one part of the overall problem.  The
overall problem, in my view, is the whole mindset which says "Oh, we
just connect the wires.  Everything else is somebody else's problem."

Ok, so this mailing list is a list of network operators.  Swell.  Every
network operator who can do so, please raise your hand if you have
*recently* scanned you own network and if you can -honestly- attest
that you have taken all necessary steps to insure that none of the
numerous specific types of CCVT thingies that Krebs and others identified
weeks or months ago as being fundamentally insecure can emit a single
packet out onto the public Internet.

And, cue the crickets...

Recent events, like the Krebs DDoS and the even bigger OVH DDoS, and
today's events make it perfectly clear to even the most blithering of
blithering idiots that network operators, en mass, have to start scanning
their own networks for insecurities.  And you'd all better get on that,
not next fiscal year or even next quarter, but right effing now, because
the next major event is right around the corner.  And remember, *you*
may not be scanning your networks for easily pop'able boxes, but as we
should all be crystal clear on by now, that *does not* mean that nobody
else is doing so.


Regards,
rfg


P.S.  The old saying is that idle hands are the devil's playground.  In
the context of the various post-invasion insurgancies, etc., in Iraq, is
is often mentioned that it was a somewhat less than a brilliant move for
the U.S. to have disbanded the Iraq army, thereby leaving large numbers
of trained young men on the streets with no jobs and nothing to do.

To all of the network operators who think that (or argue that) it will
be too expensive to hire professionals to come in an do the work to
scan your networks for known vulnerabilities, I have a simple suggestion.
Go down to your local high school, find the schmuck who teaches the
kids about computers, and ask him for the name of his most clever student.
Then hire that student and put him to work, scanning your network.

As in Iraq, it will be *much* better to have capable young men inside the
tent, pissing out, rather than the other way around.

Re: Dyn DDoS this AM?

[Chris Woodfield]

As a Twitter network  engineer (and the guy Patrick let camp out in your hotel 
room all day) - thank you for this. Whoever was behind this just poked a 
hornet’s nest. 

“Govern yourselves accordingly”.

-C

(Obviously speaking for myself, not my employer…)

> On Oct 21, 2016, at 10:48 AM, Patrick W. Gilmore  wrote:
> 
> I cannot give additional info other than what’s been on “public media”.
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things.
> 
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants.
> 
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment.
> 
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well.
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost.
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help.
> 
> -- 
> TTFN,
> patrick
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very
>> curious about the specifics of such a highly impactful attack (and it's
>> timing following NANOG 68)...
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>> 
>> -- 
>> @ChrisGrundemann
>> http://chrisgrundemann.com
> 

Re: Death of the Internet, Film at 11

[Mike Hammett]

Block one type of attack enough times and you've accomplished something. 
Because script kiddies are taking advantage of published exploits doesn't mean 
we stop setting passwords on things. You have to protect from them all. 

No, no collateral damage. We discussed this a couple weeks ago and there was no 
credible evidence of collateral damage. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Laszlo Hanyecz"  
To: nanog@nanog.org 
Sent: Friday, October 21, 2016 7:52:42 PM 
Subject: Re: Death of the Internet, Film at 11 


On 2016-10-22 00:39, Ronald F. Guilmette wrote: 
> P.S. To all of you Ayn Rand devotees out there who still vociferously 
> argue that it's nobody else's business how you monitor or police your 
> "private" networks, and who still refuse to take even minimalist steps 
> (like BCP 38), congratulations. 

What does BCP38 have to do with this? All that does is block one 
specific type of attack (and cause a lot of collateral damage). The IoT 
devices do not need to spoof addresses - they can just generate attack 
traffic directly. This is even better, because you can't cut those 
eyeball addresses off - those are the same addresses your target 
audience is using. If you cut off the eyeball networks there's not much 
point to running an internet business website anymore. 

-Laszlo 

Re: Dyn DDoS this AM?

[Yang Yu]

On Fri, Oct 21, 2016 at 11:45 AM, Patrick W. Gilmore  wrote:
> My guess is you should track anything to as33517.

And AS15135?

Re: Death of the Internet, Film at 11

[Randy Bush]

>>> What does BCP38 have to do with this?
>> nothing technical, as these iot attacks are not spoofed.
>> think of it as a religion.
> I'm going to save this e-mail forever!

no extra charge

we deploy it more than most.  we talk about it less than most.  and
every time something untoward happens on the internet, we do not tell
everyone that they should deploy bcp38, iltering, origin validation,
dnssec, ipv6, ...

talk is cheap.

Re: Death of the Internet, Film at 11

[Paul Ferguson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 10/21/2016 8:08 PM, Randy Bush wrote:

>> What does BCP38 have to do with this?
> 
> nothing technical, as these iot attacks are not spoofed.
> 
> think of it as a religion.
> 

I'm going to save this e-mail forever!

Cheers,

- - ferg


- -- 
Paul Ferguson
ICEBRG.io, Seattle USA
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlgK2ukACgkQKJasdVTchbJDywD/frHeNpPnlwT1ddgh4kZyi5MJ
YkH5lbx41an0WNpg3NAA/043VNnfKK5JQ7+dCsXyx8LEno8aIoIPvIvPGsWyjY50
=HMfV
-END PGP SIGNATURE-

Re: Death of the Internet, Film at 11

[Randy Bush]

> What does BCP38 have to do with this?

nothing technical, as these iot attacks are not spoofed.

think of it as a religion.

Re: Dyn DDoS this AM?

[George William Herbert]

> On Oct 21, 2016, at 6:35 PM, Eitan Adler  wrote:
> 
> [...]
> 
> In practice TTLs tend to be ignored on the public internet. In past
> research I've been involved with browser[0] behavior was effectively
> random despite the TTL set.
> 
> [0] more specifically, the chain of DNS resolution and caching down to
> the browser.


Yes, but that it can be both better and worse than your TTLs does not mean that 
you can ignore properly working implementations.

If the other end device chain breaks you that's their fault and out of your 
control.  If your own settings break you that's your fault.


Sent from my iPhone

Re: Dyn DDoS this AM?

[Eitan Adler]

On 21 October 2016 at 18:12, Jean-Francois Mezei
 wrote:
> On 2016-10-21 18:45, david raistrick wrote:
>
>> switch too..).   setting TTLs that make sense for a design that supports
>> change is also easy.
>
> Cuts both ways. Had Twitter had TTLs of say 7 days, vast majority
> wouldn't notice an outage of a few hours because their local cache wa
> still valid.

In practice TTLs tend to be ignored on the public internet. In past
research I've been involved with browser[0] behavior was effectively
random despite the TTL set.

[0] more specifically, the chain of DNS resolution and caching down to
the browser.


-- 
Eitan Adler

Re: Dyn DDoS this AM?

[Jean-Francois Mezei]

On 2016-10-21 18:45, david raistrick wrote:

> switch too..).   setting TTLs that make sense for a design that supports
> change is also easy.

Cuts both ways. Had Twitter had TTLs of say 7 days, vast majority
wouldn't notice an outage of a few hours because their local cache wa
still valid.

It does prevent one from reacting quickly to emergencies.

Re: Dyn DDoS this AM?

[Brett Frankenberger]

On Fri, Oct 21, 2016 at 05:11:34PM -0700, Crist Clark wrote:
>
> Given the scale of these attacks, whether having two providers does any
> good may be a crap shoot.
> 
> That is, what if the target happens to share the same providers you do?
> Given the whole asymmetry of resources that make this a problem in the
> first place, the attackers probably have the resources to take out multiple
> providers.
> 
> Having multiple providers may reduce your chance of being collateral damage
> (and I'd also still worry more about the more mundane risks of a single
> provider, maintenance or upgrade gone bad, business risks, etc., than these
> sensational ones), but multiple providers likely won't save you if you are
> the actual target of the attack.

Good, perfect, enemy, etc.

How many sites were down today?  How many were the intended target?

 -- Brett

Re: Death of the Internet, Film at 11

[James Downs]

> On Oct 21, 2016, at 17:39, Ronald F. Guilmette  wrote:

> P.S.  To all of you Ayn Rand devotees out there who still vociferously
> argue that it's nobody else's business how you monitor or police your
> "private" networks, and who still refuse to take even minimalist steps

What does Ayn Rand have to do with it? She would hardly countenance 
incompetence.

Re: Death of the Internet, Film at 11

[Paul Ferguson]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 10/21/2016 5:52 PM, Laszlo Hanyecz wrote:

> 
> On 2016-10-22 00:39, Ronald F. Guilmette wrote:
>> P.S.  To all of you Ayn Rand devotees out there who still
>> vociferously argue that it's nobody else's business how you
>> monitor or police your "private" networks, and who still refuse
>> to take even minimalist steps (like BCP 38), congratulations.
> 
> What does BCP38 have to do with this?  All that does is block one 
> specific type of attack (and cause a lot of collateral damage).
> The IoT devices do not need to spoof addresses - they can just
> generate attack traffic directly.  This is even better, because you
> can't cut those eyeball addresses off - those are the same
> addresses your target audience is using.  If you cut off the
> eyeball networks there's not much point to running an internet
> business website anymore.
> 

Don't let the perfect be the enemy of the good.

- - ferg (BCP38 instigator)


- -- 
Paul Ferguson
ICEBRG.io, Seattle USA
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlgKunAACgkQKJasdVTchbJJCQD+N6cosKffmfTqERBJ8q3pX+20
jY/FQvzUuKoy+iY3C4wA/2qKV01Z0e16BQ0/030euhCCmTUW0jut+Hp8xyWrVKkN
=+oT7
-END PGP SIGNATURE-

Re: Death of the Internet, Film at 11

[Laszlo Hanyecz]

On 2016-10-22 00:39, Ronald F. Guilmette wrote:

P.S.  To all of you Ayn Rand devotees out there who still vociferously
argue that it's nobody else's business how you monitor or police your
"private" networks, and who still refuse to take even minimalist steps
(like BCP 38), congratulations.


What does BCP38 have to do with this?  All that does is block one 
specific type of attack (and cause a lot of collateral damage).  The IoT 
devices do not need to spoof addresses - they can just generate attack 
traffic directly.  This is even better, because you can't cut those 
eyeball addresses off - those are the same addresses your target 
audience is using.  If you cut off the eyeball networks there's not much 
point to running an internet business website anymore.


-Laszlo

Death of the Internet, Film at 11

[Ronald F. Guilmette]

VICTOR LASZLO:  If we stop fighing our enemies, the world will die.
RICK BLAINE:  Well, what of it?  It will be out of its misery.

  -- From the movie "Casablanca" (1942)

Sorry, but some days I just can't help thinking to myself "Oh well,
as much fun as it has been, this whole lab experiment called The
Internet was never really going last or stand the test of time anyway."

The problem isn't the technology.  It's the politics.  It's fragility
by design.

Oh!  And by the way, one news source that I was just reading a few minutes
ago stated that all of the carnage at Dyn today was caused by something
on the order of just 1/10th of the known CCTV bots out there.

And I'm thinking, like, "Gee!  I guess that we ought to count ourselves
as lucky that whoever was running this thing, for whatever reason, just
didn't much feel like firing up the whole entire bloody thing today.
Otherwise, you know, we might have REALLY had a problem." :-)


Regards,
rfg


P.S.  To all of you Ayn Rand devotees out there who still vociferously
argue that it's nobody else's business how you monitor or police your
"private" networks, and who still refuse to take even minimalist steps
(like BCP 38), congratulations.  Via your inaction and self-centered
intransigence you have today moved us all one step closer to the day
when the relevant decisions will be taken out of your hands.  You are
succeding brilliantly at creating the exact thing that you most abhor,
i.e. government control.

Clemenceau said that war is too important to be left to the generals.
Well, guess what?  The Internet is too important to be left to the
[[fill in the blank]].  It has already begun...

https://krebsonsecurity.com/2016/10/europe-to-push-new-security-rules-amid-iot-mess/

This is just the first timid step.  A few more days like today and more,
much more, will follow.

Re: Dyn DDoS this AM?

[Crist Clark]

Given the scale of these attacks, whether having two providers does any
good may be a crap shoot.

That is, what if the target happens to share the same providers you do?
Given the whole asymmetry of resources that make this a problem in the
first place, the attackers probably have the resources to take out multiple
providers.

Having multiple providers may reduce your chance of being collateral damage
(and I'd also still worry more about the more mundane risks of a single
provider, maintenance or upgrade gone bad, business risks, etc., than these
sensational ones), but multiple providers likely won't save you if you are
the actual target of the attack.


On Fri, Oct 21, 2016 at 4:45 PM, Måns Nilsson 
wrote:

> Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:37:09AM +0200
> Quoting Niels Bakker (ni...@bakker.net):
> > * mansa...@besserwisser.org (Måns Nilsson) [Sat 22 Oct 2016, 01:27
> CEST]:
> > >Also, do not fall in the "short TTL for service agility" trap.
> >
> > Several CDNs, Akamai among them, do use short TTLs for this exact reason.
> > Server load is constantly monitored and taken into account when crafting
> DNS
> > replies.
>
> But the problem is that this trashes caching, and DNS does not work
> without caches. At least not if you want it to survive when the going
> gets tough.
>
> If we're going to solve this we need to innovate beyond the pathetic
> CNAME chains that todays managed DNS services make us use, and get truly
> distributed load-balancing decision-making (which only will work if you
> give it sensible data; a single CNAME is not sensible data) all the way
> out in the client application.
>
> --
> Måns Nilsson primary/secondary/besserwisser/machina
> MN-1334-RIPE +46 705 989668
> Well, I'm INVISIBLE AGAIN ... I might as well pay a visit to the LADIES
> ROOM ...
>

Re: Dyn DDoS this AM?

[Josh Reynolds]

Ah, disregard. I see what you're saying now.

Yes, I can see how that would be problematic.

On Oct 21, 2016 6:40 PM, "Josh Reynolds"  wrote:

> Ansible would be a decent start.
>
> On Oct 21, 2016 5:26 PM, "David Birdsong"  wrote:
>
>> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>>
>> > anyone who relies on a single dns provider is just asking for stuff such
>> > as this.
>> >
>> > randy
>> >
>>
>> I'd love to hear how others are handling the overhead of managing two dns
>> providers. Every time we brainstorm on it, we see it as blackhole of eng
>> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
>> entire delegation over.
>>
>

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:37:09AM +0200 
Quoting Niels Bakker (ni...@bakker.net):
> * mansa...@besserwisser.org (Måns Nilsson) [Sat 22 Oct 2016, 01:27 CEST]:
> >Also, do not fall in the "short TTL for service agility" trap.
> 
> Several CDNs, Akamai among them, do use short TTLs for this exact reason.
> Server load is constantly monitored and taken into account when crafting DNS
> replies.

But the problem is that this trashes caching, and DNS does not work
without caches. At least not if you want it to survive when the going
gets tough. 

If we're going to solve this we need to innovate beyond the pathetic
CNAME chains that todays managed DNS services make us use, and get truly
distributed load-balancing decision-making (which only will work if you
give it sensible data; a single CNAME is not sensible data) all the way
out in the client application. 

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Well, I'm INVISIBLE AGAIN ... I might as well pay a visit to the LADIES
ROOM ...


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Josh Reynolds]

Ansible would be a decent start.

On Oct 21, 2016 5:26 PM, "David Birdsong"  wrote:

> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>
> > anyone who relies on a single dns provider is just asking for stuff such
> > as this.
> >
> > randy
> >
>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.
>

Re: Dyn DDoS this AM?

[Keenan Tims]

I don't have a horse in this race, and haven't used it in anger, but 
Netflix released denominator to attempt to deal with some of these issues:


https://github.com/Netflix/denominator

Their goal is to support the highest common denominator of features 
among the supported providers,


Maybe that helps someone.

Keenan

On 2016-10-21 16:19, Niels Bakker wrote:

The point of outsourcing DNS isn't just availability of static
hostnames, it's the added services delivered, like returning different
answers based on source of the question, even monitoring your
infrastructure (or it reporting load into the DNS management system).

That is very hard to replicate with two DNS providers.


-- Niels.

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Sat, Oct 22, 2016 at 01:19:24AM +0200 
Quoting Niels Bakker (niels=na...@bakker.net):

> The point of outsourcing DNS isn't just availability of static hostnames,
> it's the added services delivered, like returning different answers based on
> source of the question, even monitoring your infrastructure (or it reporting
> load into the DNS management system).
> 
> That is very hard to replicate with two DNS providers.

Surely, it must be better to use a singular service that is provably
easy to take out. The advantages are overwhelming.

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Yow!  Are we wet yet?


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Måns Nilsson]

Subject: Re: Dyn DDoS this AM? Date: Fri, Oct 21, 2016 at 03:21:20PM -0700 
Quoting David Birdsong (da...@imgix.com):
> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
> 
> > anyone who relies on a single dns provider is just asking for stuff such
> > as this.
> >
> > randy
> 
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.

The fault is giving up the primary for an API connection. Sure, it is
tempting. We do, however, need to push the "application-integrated"
DNS vendors harder. They need to give their customers more choice in
how the DNS is populated. 

They also very much need to let people with above-mentioned
"application-integrated" needs add third party DNS providers in the mix.
This diversity capability is what makes DNS resilient. Monocultures have
suboptimal survivability in the long run.

Adding DNS providers when you control the primary is completely
painless. With EDNS0 there's lots of room for insanely large NS RRSETs. 

Also, do not fall in the "short TTL for service agility" trap. 

Besides, what Randy wrote. 

-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
Hold the MAYO & pass the COSMIC AWARENESS ...


signature.asc
Description: Digital signature

Re: Dyn DDoS this AM?

[Niels Bakker]

anyone who relies on a single dns provider is just asking for 
stuff such as this.
I'd love to hear how others are handling the overhead of managing 
two dns providers.


* ra...@psg.com (Randy Bush) [Sat 22 Oct 2016, 00:28 CEST]:
good question.  staying in-band, hidden primary comes to mind.  but 
i am sure clever minds can come up with more clever schemes.


The point of outsourcing DNS isn't just availability of static 
hostnames, it's the added services delivered, like returning different 
answers based on source of the question, even monitoring your 
infrastructure (or it reporting load into the DNS management system).


That is very hard to replicate with two DNS providers.


-- Niels.

Re: Dyn DDoS this AM?

[joel jaeggli]

On 10/21/16 3:21 PM, David Birdsong wrote:
> On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:
>
>> anyone who relies on a single dns provider is just asking for stuff such
>> as this.
>>
>> randy
>>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.

Not all the ones you might choose based on scale support axfr... That's
a bit of a problem for the most traditional approach to this., of those 
that do it's straight-forward to use one as the master for another, or
use a hidden master. Your own master may have demonstrably lower
availability then one or the other of your providers. getting two well
considered choices to play nice with each other isn't that hard.





signature.asc
Description: OpenPGP digital signature

Re: Dyn DDoS this AM?

[david raistrick]

On Fri, Oct 21, 2016 at 6:21 PM, David Birdsong  wrote:

>
> I'd love to hear how others are handling the overhead of managing two dns
> providers. Every time we brainstorm on it, we see it as blackhole of eng
> effort WRT to keeping them in sync and and then waiting for TTLs to cut an
> entire delegation over.
>


with the usual caveats - and I dont have any projects that currently need
this but have in the past - pretty much every major dns provider allows you
to ship them a full zone in some form or fashion.   The effort to pull and
ship a zone should be fairly minimal in and of itself.

mixing your public zone providers in your authoritative NS records is also
easy - and, depending on your registrar of choice, should be easy to manage
changing those (including having non-public mirrors maintained that you can
switch too..).   setting TTLs that make sense for a design that supports
change is also easy.

the real developmental and architectural challenges are around what to do
if the APIs you use to talk to your "primary" disappear and you need to
consume them (creating new host entries, updating loadbalancer pools,
whatever.  we all have different and sometimes very diverse use cases for
dns.).

one approach - as randy suggested - is to switch to a purely hidden and
self managed primary - which might mean running your own API stack in front
of it to control whatever you need to control and change.   this doesnt
need to be a "real" dns server in todays world - the days of BIND style
zone transfers are generally long gone anyway when you hit these scales and
levels of intra complexity.then your zone-replication components that
ship zone updates to your various external providers are shipping from the
same place.

at least in that case it's fully within your control - but dev time and
complexity definitely comes into play.

if your infra can survive internally without dns change control for the
extent of an outage, that could be much easier to manage.

anyway, random and incomplete thoughts - time ran out, work calls.


...david

Re: Dyn DDoS this AM?

[Nick Hilliard]

Patrick W. Gilmore wrote:
> Our biggest problem is people thinking they cannot or do not want to
> help.

Our biggest problem is that if the Internet community does not handle
problems like this, governments and regulators may decide to intervene.
 If they do this in the wrong way, it will turn one major headache into two.

Nick

Re: Dyn DDoS this AM?

[Randy Bush]

>> anyone who relies on a single dns provider is just asking for stuff such
>> as this.
> I'd love to hear how others are handling the overhead of managing two dns
> providers.

good question.  staying in-band, hidden primary comes to mind.  but i am
sure clever minds can come up with more clever schemes.

randy

Re: Dyn DDoS this AM?

[David Birdsong]

On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:

> anyone who relies on a single dns provider is just asking for stuff such
> as this.
>
> randy
>

I'd love to hear how others are handling the overhead of managing two dns
providers. Every time we brainstorm on it, we see it as blackhole of eng
effort WRT to keeping them in sync and and then waiting for TTLs to cut an
entire delegation over.

Re: Dyn DDoS this AM?

[Randy Bush]

> amen.
>> anyone who relies on a single dns provider is just asking for stuff
>> such as this.

part of the problem is that we think of it as attack surface when, in
fact, it usually has more than two dimensions.

randy

Re: Dyn DDoS this AM?

[Mehmet Akcin]

amen.

On Fri, Oct 21, 2016 at 2:58 PM, Randy Bush  wrote:

> anyone who relies on a single dns provider is just asking for stuff such
> as this.
>
> randy
>

Re: Dyn DDoS this AM?

[Andrew Fried]

The brutal reality in todays world is that anyone that relies on the
Internet is just asking for stuff like this.  No service is safe.

Andrew


Andrew Fried
andrew.fr...@gmail.com

On 10/21/16 5:58 PM, Randy Bush wrote:
> anyone who relies on a single dns provider is just asking for stuff such
> as this.
> 
> randy
> 

Re: Dyn DDoS this AM?

[Randy Bush]

anyone who relies on a single dns provider is just asking for stuff such
as this.

randy

Re: MPLS in the campus Network?

[David Bass]

This is exactly what we are recommending and building for our customers in that 
space. Most of the time the university network acts as a provider, so to me it 
only makes sense to use that type of tech.  The biggest problem then is 
support, which could be something they are unwilling or unable to overcome. 

> On Oct 21, 2016, at 1:45 PM, Leo Bicknell  wrote:
> 
> In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis 
> wrote:
>> In a campus network the challenge becomes extending subnets across your
>> core. You may have a college that started in one building with their own
>> /24, but now have offices and labs in other buildings. They want to stay on
>> the same network, but that's not feasible with the routed core setup
>> without some other technology overlay. We end up not being able to extend
>> the L2 like we did in the past and today we modify router ACL's to allow
>> communications. If you already have hundreds of vlans spanned across the
>> network, it's hard to get a campus to migrate to the routed core. I think
>> this may be one of Marks challenge, correct me if I'm wrong please.
> 
> FWIW, if I had to solve the "college across buildings with common
> access control" problem I would create MPLS L3 VPN's, one subnet
> per building (where it is a VLAN inside of a building), with a
> "firewall in the cloud" somewhere to get between VLAN's with all
> of the policy in one place.
> 
> No risk of the L2 across buildings mess, including broadcast and
> multicast issues at L2.  All tidy L3 routing.  Can use a real
> firewall between L3 VPN instances to get real policy tools (AV, URL
> Filtering, Malware detection, etc) rather than router ACL's.  Scales
> to huge sizes because it's all L3 based.
> 
> Combine with 802.1x port authentication and NAC, and in theory every
> L3 VPN could be in every building, with each port dynamically assigning
> the VLAN based on the user's login!  Imagine never manually configuring
> them again.  Write a script that makes all the colleges (20? 40? 60?)
> appear in every building all attached to their own MPLS VPN's, and
> then the NAC handles port assignment.
> 
> -- 
> Leo Bicknell - bickn...@ufp.org
> PGP keys at http://www.ufp.org/~bicknell/

Re: MPLS in the campus Network?

[James R Cutler]

On Oct 21, 2016, at 4:18 PM, Youssef Ghorbal  wrote, 
in part:
> 
> Until people start complaining they can no more auto discover their
> Time Capsule left in the other building whereas their colleagues in
> the other building can etc etc. All fancy discover protocols breaks
> without L2 continuity !


Minor Correction:  Correctly configured*, an Airport Extreme basestation (Time 
Capsule or not) does not require L2 connectivity to discover. In fact, Wide 
Area is used for discovery of many services not necessarily reachable by L2 
connectivity. Apple’s Back to My Mac service is one example.

*Apple’s "Back to My Mac” Wide Area Bonjour is enabled on an Airport 
basestation by entering appropriate Apple ID and password data in the Base 
Station tab as accessed by Airport Utility.


James R. Cutler
james.cut...@consultant.com
PGP keys at http://pgp.mit.edu

Re: Dyn DDoS this AM?

[Alain Hebert]

Just a FYI,

That "horrific trend" has been happening since some techie got
dissed on an IRC channel over 20 years ago.

He used a bunch of hosted putters to ICMP flood the IRC server.

Whatever the community is behind, until the carriers decide to wise
up this will keep happening, that is without talking about the
industries being developed around DDoSes events.

Enjoy your weekend. ( I ain't on call anymore anyway =D )

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 10/21/16 11:52, Brian Davies via NANOG wrote:
> +1!
>
> Well said, Patrick.
>
> B
>
> On Friday, October 21, 2016, Patrick W. Gilmore  wrote:
>
>> I cannot give additional info other than what’s been on “public media”.
>>
>> However, I would very much like to say that this is a horrific trend on
>> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
>> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
>> other things.
>>
>> To Dyn and everyone else being attacked:
>> The community is behind you. There are problems, but if we stick together,
>> we can beat these miscreants.
>>
>> To the miscreants:
>> You will not succeed. Search "churchill on the beaches”. It’s a bit
>> melodramatic, but it’s how I feel at this moment.
>>
>> To the rest of the community:
>> If you can help, please do. I know a lot of you are thinking “what can I
>> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
>> that doesn’t help Mirai, but it still helps. There are many other things
>> you can do as well.
>>
>> But a lot of it is just willingness to help. When someone asks you to help
>> trace an attack, do not let the request sit for a while. Damage is being
>> done. Help your neighbor. When someone’s house is burning, your current
>> project, your lunch break, whatever else you are doing is almost certainly
>> less important. If we stick together and help each other, we can - we WILL
>> - win this war. If we are apathetic, we have already lost.
>>
>>
>> OK, enough motivational speaking for today. But take this to heart. Our
>> biggest problem is people thinking they cannot or do not want to help.
>>
>> --
>> TTFN,
>> patrick
>>
>>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann > > wrote:
>>> Does anyone have any additional details? Seems to be over now, but I'm
>> very
>>> curious about the specifics of such a highly impactful attack (and it's
>>> timing following NANOG 68)...
>>>
>>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
>> twitter-spotify-reddit/
>>> --
>>> @ChrisGrundemann
>>> http://chrisgrundemann.com
>>

Re: MPLS in the campus Network?

[Youssef Ghorbal]

> FWIW, if I had to solve the "college across buildings with common
> access control" problem I would create MPLS L3 VPN's, one subnet
> per building (where it is a VLAN inside of a building), with a
> "firewall in the cloud" somewhere to get between VLAN's with all
> of the policy in one place.
>
> No risk of the L2 across buildings mess, including broadcast and
> multicast issues at L2.  All tidy L3 routing.  Can use a real
> firewall between L3 VPN instances to get real policy tools (AV, URL
> Filtering, Malware detection, etc) rather than router ACL's.  Scales
> to huge sizes because it's all L3 based.

Until people start complaining they can no more auto discover their
Time Capsule left in the other building whereas their colleagues in
the other building can etc etc. All fancy discover protocols breaks
without L2 continuity !
Welcome to the campus network nightmare :)
For now, there is no perfect solution ! either you cope with L2 hell
or users inconvenience (and yes people tend to think that the campus
network is expected to work as their home network)

I've also stumbled upon some "Building Automation and Control
Networks" (BACnet/IP for instance) where each building has some
automats that all needs to be in the same network segment.

> Combine with 802.1x port authentication and NAC, and in theory every
> L3 VPN could be in every building, with each port dynamically assigning
> the VLAN based on the user's login!  Imagine never manually configuring
> them again.  Write a script that makes all the colleges (20? 40? 60?)
> appear in every building all attached to their own MPLS VPN's, and
> then the NAC handles port assignment.

Here again, it's perfect until you start coping with old stuff, all
fancy new ethernet capable "things" or scientific/industrial
equipments. The "802.1x what ? it's plug'n play man !" attitude.

(my experience is with research institutes/academy kind of campuses)

Youssef Ghorbal

Re: Dyn DDoS this AM?

[Brian Davies via NANOG]

+1!

Well said, Patrick.

B

On Friday, October 21, 2016, Patrick W. Gilmore  wrote:

> I cannot give additional info other than what’s been on “public media”.
>
> However, I would very much like to say that this is a horrific trend on
> the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can
> Not Stand. See Krebs’ on the Democratization of Censorship. See lots of
> other things.
>
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together,
> we can beat these miscreants.
>
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit
> melodramatic, but it’s how I feel at this moment.
>
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I
> do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure,
> that doesn’t help Mirai, but it still helps. There are many other things
> you can do as well.
>
> But a lot of it is just willingness to help. When someone asks you to help
> trace an attack, do not let the request sit for a while. Damage is being
> done. Help your neighbor. When someone’s house is burning, your current
> project, your lunch break, whatever else you are doing is almost certainly
> less important. If we stick together and help each other, we can - we WILL
> - win this war. If we are apathetic, we have already lost.
>
>
> OK, enough motivational speaking for today. But take this to heart. Our
> biggest problem is people thinking they cannot or do not want to help.
>
> --
> TTFN,
> patrick
>
> > On Oct 21, 2016, at 10:55 AM, Chris Grundemann  > wrote:
> >
> > Does anyone have any additional details? Seems to be over now, but I'm
> very
> > curious about the specifics of such a highly impactful attack (and it's
> > timing following NANOG 68)...
> >
> > https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-
> twitter-spotify-reddit/
> >
> > --
> > @ChrisGrundemann
> > http://chrisgrundemann.com
>
>

Weekly Routing Table Report

[Routing Analysis Role Account]

This is an automated weekly mailing describing the state of the Internet
Routing Table as seen from APNIC's router in Japan.

The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG,
SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG.

Daily listings are sent to bgp-st...@lists.apnic.net

For historical data, please see http://thyme.rand.apnic.net.

If you have any comments please contact Philip Smith .

Routing Table Report   04:00 +10GMT Sat 22 Oct, 2016

Report Website: http://thyme.rand.apnic.net
Detailed Analysis:  http://thyme.rand.apnic.net/current/

Analysis Summary


BGP routing table entries examined:  615166
Prefixes after maximum aggregation (per Origin AS):  220426
Deaggregation factor:  2.79
Unique aggregates announced (without unneeded subnets):  299813
Total ASes present in the Internet Routing Table: 55044
Prefixes per ASN: 11.18
Origin-only ASes present in the Internet Routing Table:   36336
Origin ASes announcing only one prefix:   15328
Transit ASes present in the Internet Routing Table:6525
Transit-only ASes present in the Internet Routing Table:166
Average AS path length visible in the Internet Routing Table:   4.3
Max AS path length visible:  39
Max AS path prepend of ASN ( 55644)  31
Prefixes from unregistered ASNs in the Routing Table:56
Unregistered ASNs in the Routing Table:  15
Number of 32-bit ASNs allocated by the RIRs:  15849
Number of 32-bit ASNs visible in the Routing Table:   12183
Prefixes from 32-bit ASNs in the Routing Table:   49137
Number of bogon 32-bit ASNs visible in the Routing Table:   247
Special use prefixes present in the Routing Table:0
Prefixes being announced from unallocated address space:353
Number of addresses announced to Internet:   2829985572
Equivalent to 168 /8s, 174 /16s and 39 /24s
Percentage of available address space announced:   76.4
Percentage of allocated address space announced:   76.4
Percentage of available address space allocated:  100.0
Percentage of address space in use by end-sites:   98.3
Total number of prefixes smaller than registry allocations:  200615

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:   156347
Total APNIC prefixes after maximum aggregation:   42794
APNIC Deaggregation factor:3.65
Prefixes being announced from the APNIC address blocks:  170253
Unique aggregates announced from the APNIC address blocks:69829
APNIC Region origin ASes present in the Internet Routing Table:5173
APNIC Prefixes per ASN:   32.91
APNIC Region origin ASes announcing only one prefix:   1143
APNIC Region transit ASes present in the Internet Routing Table:946
Average APNIC Region AS path length visible:4.3
Max APNIC Region AS path length visible: 37
Number of APNIC region 32-bit ASNs visible in the Routing Table:   2439
Number of APNIC addresses announced to Internet:  759730500
Equivalent to 45 /8s, 72 /16s and 145 /24s
APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911, 45056-46079, 55296-56319,
   58368-59391, 63488-64098, 64297-64395, 131072-137529
APNIC Address Blocks 1/8,  14/8,  27/8,  36/8,  39/8,  42/8,  43/8,
49/8,  58/8,  59/8,  60/8,  61/8, 101/8, 103/8,
   106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8,
   116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8,
   123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8,
   163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8,
   203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8,
   222/8, 223/8,

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:185253
Total ARIN prefixes after maximum aggregation:89516
ARIN Deaggregation factor: 2.07
Prefixes being announced from the ARIN address blocks:   191091
Unique aggregates announced from the ARIN address blocks: 88732
ARIN Region origin ASes present in the Internet Routing Table:16165
ARIN Prefixes per ASN:11.82

Re: MPLS in the campus Network?

[Leo Bicknell]

In a message written on Fri, Oct 21, 2016 at 12:02:24PM -0500, Javier Solis 
wrote:
> In a campus network the challenge becomes extending subnets across your
> core. You may have a college that started in one building with their own
> /24, but now have offices and labs in other buildings. They want to stay on
> the same network, but that's not feasible with the routed core setup
> without some other technology overlay. We end up not being able to extend
> the L2 like we did in the past and today we modify router ACL's to allow
> communications. If you already have hundreds of vlans spanned across the
> network, it's hard to get a campus to migrate to the routed core. I think
> this may be one of Marks challenge, correct me if I'm wrong please.

FWIW, if I had to solve the "college across buildings with common
access control" problem I would create MPLS L3 VPN's, one subnet
per building (where it is a VLAN inside of a building), with a
"firewall in the cloud" somewhere to get between VLAN's with all
of the policy in one place.

No risk of the L2 across buildings mess, including broadcast and
multicast issues at L2.  All tidy L3 routing.  Can use a real
firewall between L3 VPN instances to get real policy tools (AV, URL
Filtering, Malware detection, etc) rather than router ACL's.  Scales
to huge sizes because it's all L3 based.

Combine with 802.1x port authentication and NAC, and in theory every
L3 VPN could be in every building, with each port dynamically assigning
the VLAN based on the user's login!  Imagine never manually configuring
them again.  Write a script that makes all the colleges (20? 40? 60?)
appear in every building all attached to their own MPLS VPN's, and
then the NAC handles port assignment.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpdxSVr3MRkH.pgp
Description: PGP signature

Re: MPLS in the campus Network?

[Javier Solis]

Our campus started off with L2 vlans spanning through the core, but we
migrated to routing in the core and moved our many spanning tree/broadcast
domains to the edge of buildings fronted by redundant routing with ecmp to
a redundant core utilizing ospf.

In a campus network the challenge becomes extending subnets across your
core. You may have a college that started in one building with their own
/24, but now have offices and labs in other buildings. They want to stay on
the same network, but that's not feasible with the routed core setup
without some other technology overlay. We end up not being able to extend
the L2 like we did in the past and today we modify router ACL's to allow
communications. If you already have hundreds of vlans spanned across the
network, it's hard to get a campus to migrate to the routed core. I think
this may be one of Marks challenge, correct me if I'm wrong please.

With that said, what are the best options to be able to cost effectively
scale without using vlans and maintaining a routed core? What technology
would someone suggest (mpls, vxlan,etc) to be the best possible solution?

Thank you to the participants in the discussion. I always enjoy reading
comments posted.

-Javier

On Oct 21, 2016 11:46 AM, "Mark Tinka"  wrote:

>
>
> On 21/Oct/16 16:19, Marian Ďurkovič wrote:
>
> >
> > Much easier to setup, operate & maintain than MPLS and obviously much
> > lower cost. Based on 6-months production experience, my recommendation
> > would be to stay away from MPLS in the campus.
>
> I'd be curious to hear what MPLS-specific issues you faced in the 6
> months you had to operate such a network.
>
> Been running IP/MPLS Core, Edge and Access networks for over 15 years,
> and apart from bugs which affect any protocol or feature implementation,
> I can't say it has been a nightmare to operate to the point of not
> recommending it.
>
> I have far fewer words to say about STP, although - I'll admit - I've
> never run TRILL.
>
> Mark.
>

Re: Dyn DDoS this AM?

[Steve Meuse]

On Fri, Oct 21, 2016 at 12:09 PM, Roland Dobbins  wrote:

> On 21 Oct 2016, at 23:01, Mike Hammett wrote:
>
> > Are there sites that can test your BCP38\84 compliance?
>
> 


Quick note: If anyone has this installed already on OSX, bring up the
console and see if it's still running. I discovered (while watching the
NANOG preso) that mine had an issue and was failing silently. Re-installing
the new version fixed the issue.

The funny part of the story, looking through the logs to see which networks
I roamed on that were spoofable, the only positive hit was for the NANOG
conference network in Chicago :)

-Steve

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

On Oct 21, 2016, at 12:40 PM, David Hubbard  
wrote:
> 
> Do we know the attack destinations so we can watch transit traffic destined 
> for it to help sources that may be unaware?

My guess is you should track anything to as33517.

-- 
TTFN,
patrick

Re: MPLS in the campus Network?

[Mark Tinka]

On 21/Oct/16 16:19, Marian Ďurkovič wrote:

>
> Much easier to setup, operate & maintain than MPLS and obviously much
> lower cost. Based on 6-months production experience, my recommendation
> would be to stay away from MPLS in the campus.

I'd be curious to hear what MPLS-specific issues you faced in the 6
months you had to operate such a network.

Been running IP/MPLS Core, Edge and Access networks for over 15 years,
and apart from bugs which affect any protocol or feature implementation,
I can't say it has been a nightmare to operate to the point of not
recommending it.

I have far fewer words to say about STP, although - I'll admit - I've
never run TRILL.

Mark.

Re: Dyn DDoS this AM?

[David Hubbard]

Do we know the attack destinations so we can watch transit traffic destined for 
it to help sources that may be unaware?

David

RE: Dyn DDoS this AM?

[Brandon Ross]

On Fri, 21 Oct 2016, rar wrote:


Anyone want a quick consulting gig helping us configure BCP38 and BCP84?

Configurations is all cisco
Edge routers connect to Verizon, Level 3 Fiber
Each Edge router talks to two BGP routers.

$150/hour, I'm guessing it is only an hour for somebody to explain, and 
guide us through the configuration, but OK if longer.


Sure, we'll do it.

That rate is quite a bit less than our normal retail rate, but in the 
spirit that Patrick posted about, Network Utility Force will be happy to 
provide you or any other operator resources at that rate to help configure 
BCP38 and BCP84.


Anyone serious about that, email me privately at br...@netuf.net and we'll 
put paperwork together.


--
Brandon Ross  Yahoo & AIM:  BrandonNRoss
Voice:  +1-404-635-6667ICQ:  2269442
Signal Secure SMS:  +1-404-644-9628  Skype:  brandonross
Schedule a meeting:  http://www.doodle.com/bross

Re: Dyn DDoS this AM?

[Alain Hebert]

Rofl,

Yeah good luck with that... 15+ years later and most of the actors
that could fix that, for the planete, still refuses to do anything.

Now you can start the usual circular discussion that goes nowhere
after 3 days...

PS: yeah usual BCP38 rant... but its friday.

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 10/21/16 12:12, Patrick W. Gilmore wrote:
> Attack has re-started. This is the time, folks. Rally the troops, offer help, 
> watch your flow.
>
> STOP THIS NOW.
>

Re: Dyn DDoS this AM?

[Seth Mattinen]

On 10/21/16 09:05, Matthew Black wrote:

LA Times: Why sites like Twitter and Spotify were down for East Coast users 
this morning
http://www.latimes.com/business/la-fi-tn-dyn-attack-20161021-snap-story.html



I actually can't resolve twitter.com this morning and I'm west coast. 
None of the four listed DNS servers are responding.


twitter.com.172800  IN  NS  ns1.p34.dynect.net.
twitter.com.172800  IN  NS  ns2.p34.dynect.net.
twitter.com.172800  IN  NS  ns3.p34.dynect.net.
twitter.com.172800  IN  NS  ns4.p34.dynect.net.

Trace routes seem to point towards San Jose or Palo Alto or Los Angeles.

~Seth

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

https://www.caida.org/projects/spoofer/ 


-- 
TTFN,
patrick

> On Oct 21, 2016, at 12:01 PM, Mike Hammett  wrote:
> 
> Are there sites that can test your BCP38\84 compliance? I'm okay, but 
> interested in what I can share to raise awareness. 
> 
> 
> 
> 
> - 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> Midwest-IX 
> http://www.midwest-ix.com 
> 
> - Original Message -
> 
> From: "Patrick W. Gilmore"  
> To: "NANOG list"  
> Sent: Friday, October 21, 2016 10:48:21 AM 
> Subject: Re: Dyn DDoS this AM? 
> 
> I cannot give additional info other than what’s been on “public media”. 
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things. 
> 
> To Dyn and everyone else being attacked: 
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants. 
> 
> To the miscreants: 
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment. 
> 
> To the rest of the community: 
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well. 
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost. 
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help. 
> 
> -- 
> TTFN, 
> patrick 
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  
>> wrote: 
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very 
>> curious about the specifics of such a highly impactful attack (and it's 
>> timing following NANOG 68)... 
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>>  
>> 
>> -- 
>> @ChrisGrundemann 
>> http://chrisgrundemann.com 
> 

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

Attack has re-started. This is the time, folks. Rally the troops, offer help, 
watch your flow.

STOP THIS NOW.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 11:48 AM, Patrick W. Gilmore  wrote:
> 
> I cannot give additional info other than what’s been on “public media”.
> 
> However, I would very much like to say that this is a horrific trend on the 
> Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
> Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
> things.
> 
> To Dyn and everyone else being attacked:
> The community is behind you. There are problems, but if we stick together, we 
> can beat these miscreants.
> 
> To the miscreants:
> You will not succeed. Search "churchill on the beaches”. It’s a bit 
> melodramatic, but it’s how I feel at this moment.
> 
> To the rest of the community:
> If you can help, please do. I know a lot of you are thinking “what can I do?" 
> There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
> doesn’t help Mirai, but it still helps. There are many other things you can 
> do as well.
> 
> But a lot of it is just willingness to help. When someone asks you to help 
> trace an attack, do not let the request sit for a while. Damage is being 
> done. Help your neighbor. When someone’s house is burning, your current 
> project, your lunch break, whatever else you are doing is almost certainly 
> less important. If we stick together and help each other, we can - we WILL - 
> win this war. If we are apathetic, we have already lost.
> 
> 
> OK, enough motivational speaking for today. But take this to heart. Our 
> biggest problem is people thinking they cannot or do not want to help.
> 
> -- 
> TTFN,
> patrick
> 
>> On Oct 21, 2016, at 10:55 AM, Chris Grundemann > > wrote:
>> 
>> Does anyone have any additional details? Seems to be over now, but I'm very
>> curious about the specifics of such a highly impactful attack (and it's
>> timing following NANOG 68)...
>> 
>> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>>  
>> 
>> 
>> -- 
>> @ChrisGrundemann
>> http://chrisgrundemann.com
> 

Re: Dyn DDoS this AM?

[Roland Dobbins]

On 21 Oct 2016, at 23:01, Mike Hammett wrote:

> Are there sites that can test your BCP38\84 compliance?



---
Roland Dobbins 

Re: Dyn DDoS this AM?

[Alexander Maassen]

Feel free to feed me with attack sources. Once those companies notice their 
precious mail does not arrive at clients. They will attempt to fix things. Sad 
but true.

Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- 
Peplink Certified Engineer

 Oorspronkelijk bericht Van: "Patrick W. Gilmore" 
 Datum: 21-10-16  17:48  (GMT+01:00) Aan: NANOG list 
 Onderwerp: Re: Dyn DDoS this AM? 
I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm very
> curious about the specifics of such a highly impactful attack (and it's
> timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
> 
> -- 
> @ChrisGrundemann
> http://chrisgrundemann.com

RE: Dyn DDoS this AM?

[Matthew Black]

LA Times: Why sites like Twitter and Spotify were down for East Coast users 
this morning
http://www.latimes.com/business/la-fi-tn-dyn-attack-20161021-snap-story.html




-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Chris Grundemann
Sent: Friday, October 21, 2016 7:56 AM
To: nanog@nanog.org
Subject: Dyn DDoS this AM?

Does anyone have any additional details? Seems to be over now, but I'm very
curious about the specifics of such a highly impactful attack (and it's
timing following NANOG 68)...

https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/

-- 
@ChrisGrundemann
http://chrisgrundemann.com

RE: Dyn DDoS this AM?

[rar]

Anyone want a quick consulting gig helping us configure BCP38 and BCP84?

Configurations is all cisco
Edge routers connect to Verizon, Level 3 Fiber
Each Edge router talks to two BGP routers.

$150/hour,  I'm guessing it is only an hour for somebody to explain, and guide 
us through the configuration, but OK if longer.

Thanks.


Bob Roswell
brosw...@syssrc.com
410-771-5544 ext 4336

Computer Museum Highlights

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Patrick W. Gilmore
Sent: Friday, October 21, 2016 11:48 AM
To: NANOG list 
Subject: Re: Dyn DDoS this AM?

I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

--
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm 
> very curious about the specifics of such a highly impactful attack 
> (and it's timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotif
> y-reddit/
> 
> --
> @ChrisGrundemann
> http://chrisgrundemann.com

Re: Dyn DDoS this AM?

[Mike Hammett]

Are there sites that can test your BCP38\84 compliance? I'm okay, but 
interested in what I can share to raise awareness. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Patrick W. Gilmore"  
To: "NANOG list"  
Sent: Friday, October 21, 2016 10:48:21 AM 
Subject: Re: Dyn DDoS this AM? 

I cannot give additional info other than what’s been on “public media”. 

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things. 

To Dyn and everyone else being attacked: 
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants. 

To the miscreants: 
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment. 

To the rest of the community: 
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well. 

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost. 


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help. 

-- 
TTFN, 
patrick 

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote: 
> 
> Does anyone have any additional details? Seems to be over now, but I'm very 
> curious about the specifics of such a highly impactful attack (and it's 
> timing following NANOG 68)... 
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
>  
> 
> -- 
> @ChrisGrundemann 
> http://chrisgrundemann.com 

Re: Dyn DDoS this AM?

[Patrick W. Gilmore]

I cannot give additional info other than what’s been on “public media”.

However, I would very much like to say that this is a horrific trend on the 
Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not 
Stand. See Krebs’ on the Democratization of Censorship. See lots of other 
things.

To Dyn and everyone else being attacked:
The community is behind you. There are problems, but if we stick together, we 
can beat these miscreants.

To the miscreants:
You will not succeed. Search "churchill on the beaches”. It’s a bit 
melodramatic, but it’s how I feel at this moment.

To the rest of the community:
If you can help, please do. I know a lot of you are thinking “what can I do?" 
There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that 
doesn’t help Mirai, but it still helps. There are many other things you can do 
as well.

But a lot of it is just willingness to help. When someone asks you to help 
trace an attack, do not let the request sit for a while. Damage is being done. 
Help your neighbor. When someone’s house is burning, your current project, your 
lunch break, whatever else you are doing is almost certainly less important. If 
we stick together and help each other, we can - we WILL - win this war. If we 
are apathetic, we have already lost.


OK, enough motivational speaking for today. But take this to heart. Our biggest 
problem is people thinking they cannot or do not want to help.

-- 
TTFN,
patrick

> On Oct 21, 2016, at 10:55 AM, Chris Grundemann  wrote:
> 
> Does anyone have any additional details? Seems to be over now, but I'm very
> curious about the specifics of such a highly impactful attack (and it's
> timing following NANOG 68)...
> 
> https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/
> 
> -- 
> @ChrisGrundemann
> http://chrisgrundemann.com

Dyn DDoS this AM?

[Chris Grundemann]

Does anyone have any additional details? Seems to be over now, but I'm very
curious about the specifics of such a highly impactful attack (and it's
timing following NANOG 68)...

https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/

-- 
@ChrisGrundemann
http://chrisgrundemann.com

Re: MPLS in the campus Network?

[Marian Ďurkovič]

> Compared to MPLS, a L2 solution with 100 Gb/s interfaces between
> core switches and a 10G connection for each buildings looks so much
> cheaper. But we worry about future trouble using Trill, SPB, or other
> technologies, not only the "open" ones, but specifically the proprietary
> ones based on central controller and lots of magic (some colleagues feel
> the debug nightmare are garanteed).
> 
> If you had to make such a choice recently, did you choose an MPLS design
> even at lower speed ?

A year ago we built NREN backbone using TRILL instead of MPLS.
40 POPs, no central controller, RFC standardized TRILL protocol
i.e. L2 routing using IS-IS, no STP.

See my recent presentation at  http://md.bts.sk/sanet-100g-2.pdf 
for more details.

Much easier to setup, operate & maintain than MPLS and obviously much
lower cost. Based on 6-months production experience, my recommendation
would be to stay away from MPLS in the campus.


  With kind regards,

 M.