Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Large Hadron Collider]

Sorry florian. Meant to put it to list.


On 2016-10-09 12:25 PM, Large Hadron Collider wrote:



On 2016-10-09 04:20 AM, Florian Weimer wrote:

* Eliot Lear:


Not my end goal.  My end goal is that consumers have a means to limit
risk in their home environments, and service providers have a means to
deliver that to them.

They already have, with today's technology.  It's just not a
mass-market business.  Consumers either have to educate themselves
(which is not that hard), and service providers need to provide actual
service, instead charging a fee for access to a computer system.

There is little interest in this, however.  There's a comparable
business case for providing managed PCs to consumers, and I'm not sure
if any such companies are still left.
I'd wager that after the Indian tech support fucks, they've went like 
"too risky"


But yeah there's a good case. If I had it in me I'd hire a bunch of 
people to manage consumers' managed PCs.



I'm not convinced that expected traffic profiles are the right answer.
We already have that in the server hosting market, and it does
constraint the types of services you can run on hosted servers (for
the hosting providers who does this).  I'm wary of the network putting
severe constraints on application architecture, way beyond what is
dictated by current technology.  NAT more or less killed servers on
consumer networks, and this kind of traffic profiling has begun to
kill clients on server networks.

The whole point of MUD is to leave control in the hands of those who
have developed and have to support Things.  It is not simply for the SP
to decide what traffic is ok, or to charge more for it, but to respect
the wishes of the developers.  That may be sufficient to stop a lot of
bad things from happening to a lot of Things.

Nobody respects what developers want, otherwise we wouldn't have any
shipping products at all.

What I'm trying to say: Cutting corners is more often a
non-development decision.  If you can ship today without any security,
or at some unknowable date in the future, with additional security
features whose impact may not matter, things usually head for the
earlier shipping date.

I used to be frustrated by such decisions, but over the past few
years, I've come to realize that most of us have so little data on the
effectiveness of security features that mandates for them are
essentially arbitrary.


And again, this is the wrong way to look at it.  The consumer should
always get final say.  They're the customer.  This is a chance for the
manufacturer of the device they're using to explain how the device is
supposed to behave on the network.

If we want to make consumers to make informed decisions, they need to
learn how things work up to a certain level.  And then current
technology already works.

(Sorry that I'm not inclined to read upon the specs—I do wonder how
this an improvement over UPnP.)

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Florian Weimer]

* Eliot Lear:

> Not my end goal.  My end goal is that consumers have a means to limit
> risk in their home environments, and service providers have a means to
> deliver that to them.

They already have, with today's technology.  It's just not a
mass-market business.  Consumers either have to educate themselves
(which is not that hard), and service providers need to provide actual
service, instead charging a fee for access to a computer system.

There is little interest in this, however.  There's a comparable
business case for providing managed PCs to consumers, and I'm not sure
if any such companies are still left.

>> I'm not convinced that expected traffic profiles are the right answer.
>> We already have that in the server hosting market, and it does
>> constraint the types of services you can run on hosted servers (for
>> the hosting providers who does this).  I'm wary of the network putting
>> severe constraints on application architecture, way beyond what is
>> dictated by current technology.  NAT more or less killed servers on
>> consumer networks, and this kind of traffic profiling has begun to
>> kill clients on server networks.
>
> The whole point of MUD is to leave control in the hands of those who
> have developed and have to support Things.  It is not simply for the SP
> to decide what traffic is ok, or to charge more for it, but to respect
> the wishes of the developers.  That may be sufficient to stop a lot of
> bad things from happening to a lot of Things.

Nobody respects what developers want, otherwise we wouldn't have any
shipping products at all.

What I'm trying to say: Cutting corners is more often a
non-development decision.  If you can ship today without any security,
or at some unknowable date in the future, with additional security
features whose impact may not matter, things usually head for the
earlier shipping date.

I used to be frustrated by such decisions, but over the past few
years, I've come to realize that most of us have so little data on the
effectiveness of security features that mandates for them are
essentially arbitrary.

> And again, this is the wrong way to look at it.  The consumer should
> always get final say.  They're the customer.  This is a chance for the
> manufacturer of the device they're using to explain how the device is
> supposed to behave on the network.

If we want to make consumers to make informed decisions, they need to
learn how things work up to a certain level.  And then current
technology already works.

(Sorry that I'm not inclined to read upon the specs—I do wonder how
this an improvement over UPnP.)

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Stephen Satchell]

On 09/28/2016 12:33 AM, Eliot Lear wrote:

It's not just consumers that need to understand this.  Manufacturers of
Things are right now on a steep learning curve.  Consider that
thermostat, for just a moment.  In The Gold Old Days, before it had a
network interface, the manufacturer cared about a handful of things like
at what temperature to turn the heat or A/C on maybe with some
adjustments for time of day or day or week.  And that was it.  That is
their domain of expertise.  Not security.

Now the Internet looks like a new shiny object that promises to provide
some cool new world capabilities, like letting people adjust the temp
while they're away, or using weather forecasts to manage hysteresis
effects.  And so, the manufacturer initially thinks, we'll add an
interface to the product, and a little bit of code, and we're done.  Now
the manufacturer has stepped outside their domain of expertise, and
doesn't have a full understanding of the risks that need to be
addressed.  We as experts in this domain can help by informing
manufacturers of those risks.


Many manufacturers will outsource the Internet portion of their product 
to a software provider, not build from scratch "in house".  The people 
we really need to get to are the ones that *provide* those packages the 
manufacturers use.


In the case of embedded Linux solutions, the discussion need only be 
about what knobs to turn, and how far.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eliot Lear]

It's not just consumers that need to understand this.  Manufacturers of
Things are right now on a steep learning curve.  Consider that
thermostat, for just a moment.  In The Gold Old Days, before it had a
network interface, the manufacturer cared about a handful of things like
at what temperature to turn the heat or A/C on maybe with some
adjustments for time of day or day or week.  And that was it.  That is
their domain of expertise.  Not security.

Now the Internet looks like a new shiny object that promises to provide
some cool new world capabilities, like letting people adjust the temp
while they're away, or using weather forecasts to manage hysteresis
effects.  And so, the manufacturer initially thinks, we'll add an
interface to the product, and a little bit of code, and we're done.  Now
the manufacturer has stepped outside their domain of expertise, and
doesn't have a full understanding of the risks that need to be
addressed.  We as experts in this domain can help by informing
manufacturers of those risks.

Eliot


On 9/27/16 6:05 PM, Patrick W. Gilmore wrote:
> On Sep 27, 2016, at 11:49 AM, Roland Dobbins  wrote:
>> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:
>>> All the more reason to educate people TODAY on why having vulnerable 
>>> devices is a Very Bad Idea.
>> Yes, but how do they determine that a given device is vulnerable?
> Easy: Can you ping it? It’s vulnerable.
>
> :-)
>
> Hey, I said we would have to educate them. I did not say how that would 
> happen.
>




signature.asc
Description: OpenPGP digital signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Alexander Maassen]

If those where in fact non-spoofed sources, then i am surely interested in 
getting that list in order to put it into my dnsbl (dronebl). So if someone has 
it, or can tell me who to contact. Feel free to provide me with it offlist.
Especially if this botnet uses one of the many irc networks (like undernet) 
that is utilizing the dnsbl list and the cc is harbored there, it might help. 
Also, most 'admins' only start realizing something is wrong in their network 
once their precious bizmail won't arrive at clients because their infected ip 
is listed and the remote mx refuses the mail because of the listing.

Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- 
Peplink Certified Engineer

 Oorspronkelijk bericht Van: Hugo Slabbert <h...@slabnet.com> 
Datum: 26-09-16  05:54  (GMT+01:00) Aan: "John R. Levine" <jo...@iecc.com> Cc: 
nanog@nanog.org Onderwerp: Re: Krebs on Security booted off Akamai network 
after DDoS attack proves pricey 

On Sun 2016-Sep-25 17:01:55 -0400, John R. Levine <jo...@iecc.com> wrote:

>>https://www.internetsociety.org/sites/default/files/01_5.pdf
>>
>>The attack is triggered by a few spoofs somewhere in the world. It is not
>>feasible to stop this.
>
>That paper is about reflection attacks.  From what I've read, this was 
>not a reflection attack.  The IoT devices are infected with botware 
>which sends attack traffic directly.  Address spoofing is not particularly 
>useful for controlling botnets.  

But that's not only remaining use of source address spoofing in direct 
attacks, no?  Even if reflection and amplification are not used, spoofing 
can still be used for obfuscation.

>For example, the Conficker botnet generated pseudo-random domain names 
>where the bots looked for control traffic.
>
>>Please see https://www.ietf.org/rfc/rfc6561.txt
>
>Uh, yes, we're familiar with that.  We even know the people who wrote 
>it. It could use an update for IoT since I get the impression that in 
>many cases the only way for a nontechnical user to fix the infection 
>is to throw the device away.
>
>Regards,
>John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
>Please consider the environment before reading this e-mail. https://jl.ly

-- 
Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
pgp key: B178313E   | also on Signal

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 28 Sep 2016, at 0:18, Brielle Bruns wrote:

> I call shenanigans on providers not seeing their unruly users.

I was talking about the users, not the ISPs.

---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message , Jared Mauch 
writes:
>
> > On Sep 27, 2016, at 12:43 AM, Mark Andrews  wrote:
> >
> > Why not?  You call a washing machine mechanic when the washing
> > machine plays up.  This is not conceptually different.
>
> Mark,
>
> Your logic is infallible here, but the equivalencies are not.  If I
> drive on the road and it’s bumpy, I would complain to the road people,
> but some people will take their car to the shop and says it shakes.
>
> When you are a toll-free call away from a complaint, often this barrier
> of proof is quite high.  I recall something that Vijay said when he was
> still at AOL, if the customer phones in for support they lost all profit
> on the customer for the lifetime of the customer.
>
> Given that most people make decisions based on lowest cost (which isn’t
> always lowest or best due to marketing, promos, etc) the barrier for
> burden
> of proof is set such that a carrier must prove to a non-technical user
> it’s
> their fault.
>
> This proof is tough, not impossible, but look at your EDNS project, the
> problems are real and often can’t be easily addressed.

Actually, EDNS shows they can be addressed.

Firewall vendors are changing the defaults to allow through all
packets that match the test classes.

DNS vendors are fixing their products to properly handle packets
with EDNS extension.

DNS hosters are fixing their deployed firewalls and servers.

Soon I'll be asking, my local opposition MP if she can ask why the
DNS servers for *.gov.au aren't compliant with the standard after
reporting to the DNS operators that they are broken.  I suspect she
will have fun with having more material to fling around.

I'm having to reduce the parallelism of the test runs because
the packets are being answered.

Fixing EDNS is basically a education issue.  Raise the awareness
until it becomes something one can't ignore.  Go look at the TLD
graphs.  Almost all the TLD operators have fixed their firewalls /
servers.

If Microsoft and Go Daddy fix their servers most of the incorrect
echoing EDNS options and EDNS flags will disappear.  Both have been
informed.  Microsoft about 2 years ago when we let them know that
their servers have issues with EDNS, this included both the servers
they ship in Windows and the servers answering DNS queries for
Microsoft domains.  They where reminded a year ago.  Go Daddy was
informed very recently (via email).

Note that COOKIE is echoed.  Also you can't report this to Microsoft
using the email address listed below which was designed for reporting
issues like this.  Microsoft wants you to create a account or use
twitter (which also requires an account to be created).

You will note the DNS COOKIES are on by default.  BIND 9.11.0 will
be sending its queries with a DNS COOKIE option present.  All your
servers need to cope.

% dig boimi.gov. @ns1-06.azure-dns.com soa

; <<>> DiG 9.11.0rc2 <<>> boimi.gov. @ns1-06.azure-dns.com soa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54172
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
; COOKIE: ddcbdd73de5d5ef8 (echoed)
;; QUESTION SECTION:
;boimi.gov. IN  SOA

;; ANSWER SECTION:
boimi.gov.  3600IN  SOA ns1-06.azure-dns.com. 
azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300

;; ADDITIONAL SECTION:
ns1-06.azure-dns.com.   3600IN  A   40.90.4.6

;; Query time: 141 msec
;; SERVER: 40.90.4.6#53(40.90.4.6)
;; WHEN: Wed Sep 28 07:11:15 EST 2016
;; MSG SIZE  rcvd: 152

% 


% dig microsoft.com @ns1.msft.net

; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 7450
;; flags: qr rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 5a294c21d4ac66a3 (echoed)
;; QUESTION SECTION:
;microsoft.com. IN  A

;; Query time: 269 msec
;; SERVER: 2620:0:30::53#53(2620:0:30::53)
;; WHEN: Wed Sep 28 07:05:34 EST 2016
;; MSG SIZE  rcvd: 54

% dig microsoft.com @ns1.msft.net +nocookie

; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net +nocookie
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26221
;; flags: qr aa rd; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;microsoft.com. IN  A

;; ANSWER SECTION:
microsoft.com.  3600IN  A   23.96.52.53
microsoft.com.  3600IN  A   191.239.213.197
microsoft.com.  3600IN  A   104.40.211.35

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jared Mauch]

> On Sep 27, 2016, at 10:48 AM, Brielle Bruns  wrote:
> 
> You start cutting off users or putting them into a walled garden until they 
> fix their machines, and they will start caring.

Wait until the user who claims perfection gets on the phone, etc.

We had a network outage that caused a customer to go down, but they couldn’t 
sort it as they had two links to our network.

Turns out they didn’t advertise the same routes on both links, and when I 
explained this to them they got very quiet on the phone.  (It was a conference 
call with a lot of senior leadership on both sides involved).  I asked if they 
wanted to correct the error so I could validate that it was fixed and they 
quickly went away.  The tone before that point of the call was quite different.

Residential customers can be far worse.  I’ve heard of users clicking “I 
cleaned my stuff” to get out of the portal nearly 200x because they are trained 
to “Click to accept” everything else.  This in no way shocks me, nor can 
everyone understand the complex error messages that computers provide.  I 
myself end up searching for obscure error messages often due to hardware or 
software failures in $dayjob as well as supporting friends and family around me.

End-users can be a unique problems to work with.

- jared

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mike Hammett]

We can't teach other network operators the value of IPv6. Good luck teaching a 
consumer anything other than cat videos (and now recipes - unrelated to the 
former). 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Brielle Bruns" <br...@2mbit.com> 
To: nanog@nanog.org 
Sent: Tuesday, September 27, 2016 10:46:39 AM 
Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey 

On 9/27/16 9:35 AM, Roland Dobbins wrote: 
> On 27 Sep 2016, at 21:48, Brielle Bruns wrote: 
> 
>> You start cutting off users or putting them into a walled garden until 
>> they fix their machines, and they will start caring. 
> 
> It's important to keep in mind that in the not-so-distant future, their 
> 'machines' will include every article of clothing they own, every can of 
> soda in their refrigerator, ever major (and many minor) components of 
> their automobiles, every blade in their windowshades, etc. 
> 


I don't see how this is a problem exactly? If people want to buy 
devices that connect to their home network, they need to be aware of 
what these devices can do, and it is their responsibility. 

Better to teach them _now_ rather then later. 

If Timmy Numbnuts doesn't understand that plugging in a random device he 
found at Goodwill to his network could potentially carry liabilities, 
then he will keep doing it. 

I point to the current trend of parents watching and smiling, doing 
nothing as their kids destroy people's stores and restaurants. ISPs are 
literally doing the exact same thing when it comes to coddling their 
customers. 

-- 
Brielle Bruns 
The Summit Open Source Development Group 
http://www.sosdg.org / http://www.ahbl.org 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jared Mauch]

> On Sep 27, 2016, at 12:43 AM, Mark Andrews  wrote:
> 
> Why not?  You call a washing machine mechanic when the washing
> machine plays up.  This is not conceptually different. 

Mark,

Your logic is infallible here, but the equivalencies are not.  If I
drive on the road and it’s bumpy, I would complain to the road people,
but some people will take their car to the shop and says it shakes.

When you are a toll-free call away from a complaint, often this barrier
of proof is quite high.  I recall something that Vijay said when he was
still at AOL, if the customer phones in for support they lost all profit
on the customer for the lifetime of the customer.  

Given that most people make decisions based on lowest cost (which isn’t
always lowest or best due to marketing, promos, etc) the barrier for burden
of proof is set such that a carrier must prove to a non-technical user it’s
their fault.

This proof is tough, not impossible, but look at your EDNS project, the
problems are real and often can’t be easily addressed.

- Jared

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mike Hammett]

"who from my experience tend to be the least 
experienced and network knowledgeable people running a customer network" 


Also most likely to have built their network from scratch out of pure need 
(perhaps for themselves) rather than someone cashing in on a trend. No offense 
meant (though surely someone took it) either way. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Brielle Bruns" <br...@2mbit.com> 
To: nanog@nanog.org 
Sent: Tuesday, September 27, 2016 9:48:24 AM 
Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey 

On 9/26/16 10:05 PM, Roland Dobbins wrote: 
> +1 for this capability in CPE. 
> 
> OTOH, it will be of no use whatsoever to the user. Providing the user 
> with access to anomalous traffic feeds won't help, either. 
> 
> Users aren't going to call in some third-party service/support company, 
> either. 

You start cutting off users or putting them into a walled garden until 
they fix their machines, and they will start caring. 

This will only work if all providers including cable, DSL and *shudders* 
WISPs (hate to be blunt, but who from my experience tend to be the least 
experienced and network knowledgeable people running a customer network) 
do it so customer's can't just switch networks and 'make the problem go 
away'. 

I use escalating price increases and delays in service/repair time on 
some of my consulting customers who do things I warned them to be more 
careful about. 

It takes time, but when $cost starts to become prohibitive, they stop 
and think. And the ones that never learn... Well, that's more $$$ in 
my pocket for the effort that I would normally charge otherwise. 

-- 
Brielle Bruns 
The Summit Open Source Development Group 
http://www.sosdg.org / http://www.ahbl.org 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eygene Ryabinkin]

Sun, Sep 25, 2016 at 05:57:42PM -0400, Patrick W. Gilmore wrote:
> Remember University of Wisconsin vs. D-Link and their hard-coded
> NTP server address?

UW vs Netgear and Poul-Henning Kamp vs D-Link, both on NTP stuff?
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brielle Bruns]

On 9/27/16 11:18 AM, Brielle Bruns wrote:

On 9/27/16 10:05 AM, Roland Dobbins wrote:

I point to the current trend of parents watching and smiling, doing
nothing as their kids destroy people's stores and restaurants.  ISPs
are literally doing the exact same thing when it comes to coddling
their customers.


They can *see* the unruly children, but *choose* to ignore them.  That's
the difference.


I call shenanigans on providers not seeing their unruly users.  They
have no problems with bandwidth caps, or doing the dirty work of
copyright police, both of which require some level of network monitoring
per customer.


Or even better example - the providers who monetize customer 
browsing/shopping habits using third party network device?  Or 
SiteFinder like services with their DNS?


Providers have no problems monitoring, intercepting, mangling.

I know people here will say, "Well, I'm not like that/don't believe in 
that!", but we're not talking about technical decisions - but decisions 
made by people who's job is to squeeze as much profit out of the 
customers as possible.


There is no financial incentive or penalty for preventing/limiting your 
customers from harming others.  If there was, I don't think we'd be 
having this discussion.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brielle Bruns]

On 9/27/16 10:05 AM, Roland Dobbins wrote:

I point to the current trend of parents watching and smiling, doing
nothing as their kids destroy people's stores and restaurants.  ISPs
are literally doing the exact same thing when it comes to coddling
their customers.


They can *see* the unruly children, but *choose* to ignore them.  That's
the difference.


I call shenanigans on providers not seeing their unruly users.  They 
have no problems with bandwidth caps, or doing the dirty work of 
copyright police, both of which require some level of network monitoring 
per customer.






Keep in mind, most of the folks on this list are not representative of
the average consumer in terms of the skill-sets which are relevant in
this problem space.


I'm very well aware of that.  I've got a fairly decent sized user base 
that I consult for, including small biz and end users, so I'm painfully 
aware of how...  less then spectacular consumers are when it comes to 
even the most basic computer tasks.



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Dale W. Carder]

Thus spake Patrick W. Gilmore (patr...@ianai.net) on Sun, Sep 25, 2016 at 
05:57:42PM -0400:
> On Sep 25, 2016, at 5:50 PM, ryan landry  wrote:
> > On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews  wrote:
> 
> >> This is such a golden opportunity for each of you to find compromised
> >> hosts on your network or your customer's network.  The number of
> >> genuine lookups of the blog vs the number of botted machine would
> >> make it almost certain that anything directed at the blog is a
> >> compromised machine.  A phone call to the customer / further analysis
> >> would reduce the false positive rate.
> >> 
> >> Mark
> >> 
> >> 
> > i wish you luck with that. explaining to grandma that her samsung smart tv
> > has been rooted and needs to be updated should be good fun.
> > 
> > for isp's it's a resourcing vs revenue problem. always has been. always
> > will be. far more inclined to hold liable the folks that are churning out
> > terribly dangerous cpe / IoT(shit). surely some regulatory body is looking
> > into this.
> 
> Yeah, ‘cause that was so successful in the past.
> 
> Remember University of Wisconsin vs. D-Link and their hard-coded NTP server 
> address?

Interestingly, this was just recently looked at again for the Internet of 
Things 
Software Update Workshop (IoTSU).  See:
http://pages.cs.wisc.edu/~plonka/iotsu/IoTSU_2016_paper_25.pdf

3,564 devices still remain.

best,
Dale

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Peter Beckman]

On Tue, 27 Sep 2016, Brielle Bruns wrote:

I don't see how this is a problem exactly?  If people want to buy devices 
that connect to their home network, they need to be aware of what these 
devices can do, and it is their responsibility.


 I understand that is what you want. What you might like. What we all would
 like. People taking responsibility for their impact on others.

 Unfortunately people plug things in, and if they work for them, they don't
 even think about how what they are doing might affect anyone else. In some
 cases, they don't even care. They've got soccer games and work and TV
 shows and kids and family. Who has time to become an expert in Internet
 security?

 Google is doing a great job of annoying or alerting customers to potential
 issues, such as the red lock icon on their email, indicating that the
 email was sent unencrypted. The user gets worried (oooh, a red lock, that
 must be bad, I'm going to yell at someone to fix it for me) and the
 service provider jumps to improve the Internet, ideally.

 FreeBSD updated their default config so you have to proactively remove
 email encryption.

 If we are truly worried about IoT and consumers contributing to the
 downfall of the Internet, force the consumer router manufacturers and third
 party firmware folks to implement whatever is necessary to make filters
 and blocking the default. 90%+ of consumers don't change any settings,
 beyond the SSID and Wifi Password, and those who do might take the
 responsibility you want.

 Get the ISPs to realize that secure-by-default consumer routers that they
 distribute saves them millions/billions of dollars annually in customer
 service and security personnel. Secure-by-default routers means
 cost-savings. Get ISPs to pressure manufacturers to implement measures to
 protect their own network and the Internet from the non-network-admin consumer.

 We tech folk need to do this for the Internet citizens who don't know,
 don't care, or don't have time to mess with it.

If Timmy Numbnuts doesn't understand that plugging in a random device he 
found at Goodwill to his network could potentially carry liabilities, then he 
will keep doing it.


 Timmy Numbnuts needs to be protected from himself, so when he plugs in
 that device, it doesn't do any harm to anyone but his own network. He'd
 have to proactively turn off features or filters on his Router in order to
 harm others.

I point to the current trend of parents watching and smiling, doing nothing 
as their kids destroy people's stores and restaurants.  ISPs are literally 
doing the exact same thing when it comes to coddling their customers.


 Automation and default configs means customers don't have to do anything,
 nor think about it. They are protected both FROM harm from the Internet
 and FROM harming the Internet, at least by default.

Beckman
---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Keith Stokes]

Assuming all devices are vulnerable isn't a bad start.

--

Keith Stokes

> On Sep 27, 2016, at 11:04 AM, Roland Dobbins  wrote:
> 
>> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:
>> 
>> All the more reason to educate people TODAY on why having vulnerable devices 
>> is a Very Bad Idea.
> 
> Yes, but how do they determine that a given device is vulnerable?
> 
> ---
> Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 22:49, Florian Weimer wrote:

Most people over here have at least two providers of water and 
Internet (although the second one is perhaps sufficient for brushing

your teeth, but certainly not for a shower or a bath).


That's not a common arrangement in much of the world, however.  
Especially the Internet part.


;>

---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Patrick W. Gilmore]

On Sep 27, 2016, at 11:49 AM, Roland Dobbins  wrote:
> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:

>> All the more reason to educate people TODAY on why having vulnerable devices 
>> is a Very Bad Idea.
> 
> Yes, but how do they determine that a given device is vulnerable?

Easy: Can you ping it? It’s vulnerable.

:-)

Hey, I said we would have to educate them. I did not say how that would happen.

-- 
TTFN,
patrick

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 22:46, Brielle Bruns wrote:

I point to the current trend of parents watching and smiling, doing 
nothing as their kids destroy people's stores and restaurants.  ISPs 
are literally doing the exact same thing when it comes to coddling 
their customers.


They can *see* the unruly children, but *choose* to ignore them.  That's 
the difference.


Keep in mind, most of the folks on this list are not representative of 
the average consumer in terms of the skill-sets which are relevant in 
this problem space.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:

All the more reason to educate people TODAY on why having vulnerable 
devices is a Very Bad Idea.


Yes, but how do they determine that a given device is vulnerable?

---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Florian Weimer]

* Roland Dobbins:

> On 27 Sep 2016, at 12:17, Sam Silvester wrote:
>
>> or call their electricity retailer/distributer
>
> This is the problematic case that is, unfortunately, the default.
>
> People tend to view anything related to 'the Internet' as a utility,
> and for consumers and SMBs, they typically have a single provider,
> just as they typically do for electricity and water.

Most people over here have at least two providers of water and
Internet (although the second one is perhaps sufficient for brushing
your teeth, but certainly not for a shower or a bath).

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brielle Bruns]

On 9/27/16 9:35 AM, Roland Dobbins wrote:

On 27 Sep 2016, at 21:48, Brielle Bruns wrote:


You start cutting off users or putting them into a walled garden until
they fix their machines, and they will start caring.


It's important to keep in mind that in the not-so-distant future, their
'machines' will include every article of clothing they own, every can of
soda in their refrigerator, ever major (and many minor) components of
their automobiles, every blade in their windowshades, etc.




I don't see how this is a problem exactly?  If people want to buy 
devices that connect to their home network, they need to be aware of 
what these devices can do, and it is their responsibility.


Better to teach them _now_ rather then later.

If Timmy Numbnuts doesn't understand that plugging in a random device he 
found at Goodwill to his network could potentially carry liabilities, 
then he will keep doing it.


I point to the current trend of parents watching and smiling, doing 
nothing as their kids destroy people's stores and restaurants.  ISPs are 
literally doing the exact same thing when it comes to coddling their 
customers.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Alan Buxey]

hi,

>From: NANOG <nanog-boun...@nanog.org> on behalf of Mike Hammett 
><na...@ics-il.net>
>Sent: 27 September 2016 16:30
>Cc: nanog@nanog.org
>Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
>proves pricey
>
>You must not support end users.


haha...i read that wrong. I read it as a command, rather than an observation! 
;-)


alan

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 12:17, Sam Silvester wrote:


or call their electricity retailer/distributer


This is the problematic case that is, unfortunately, the default.

People tend to view anything related to 'the Internet' as a utility, and 
for consumers and SMBs, they typically have a single provider, just as 
they typically do for electricity and water.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Patrick W. Gilmore]

On Sep 27, 2016, at 11:35 AM, Roland Dobbins  wrote:
> On 27 Sep 2016, at 21:48, Brielle Bruns wrote:

>> You start cutting off users or putting them into a walled garden until they 
>> fix their machines, and they will start caring.
> 
> It's important to keep in mind that in the not-so-distant future, their 
> 'machines' will include every article of clothing they own, every can of soda 
> in their refrigerator, ever major (and many minor) components of their 
> automobiles, every blade in their windowshades, etc.

All the more reason to educate people TODAY on why having vulnerable devices is 
a Very Bad Idea.

If we try to teach them when a 6-pack of Coke is a DDoS source, we will have 
lost.

-- 
TTFN,
patrick

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 21:48, Brielle Bruns wrote:

You start cutting off users or putting them into a walled garden until 
they fix their machines, and they will start caring.


It's important to keep in mind that in the not-so-distant future, their 
'machines' will include every article of clothing they own, every can of 
soda in their refrigerator, ever major (and many minor) components of 
their automobiles, every blade in their windowshades, etc.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mike Hammett]

You must not support end users. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mark Andrews" <ma...@isc.org> 
To: "Roland Dobbins" <rdobb...@arbor.net> 
Cc: nanog@nanog.org 
Sent: Monday, September 26, 2016 11:43:36 PM 
Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey 


In message <b796c128-afdf-45a1-b5af-c29bff06e...@arbor.net>, Roland Dobbins wri 
tes: 
> 
> On 27 Sep 2016, at 6:58, Christopher Morrow wrote: 
> 
> > wouldn't something as simple as netflow/sflow/ipfix synthesized on the 
> > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good 
> > enough' to present a pretty clear UI to the user? 
> 
> +1 for this capability in CPE. 
> 
> OTOH, it will be of no use whatsoever to the user. Providing the user 
> with access to anomalous traffic feeds won't help, either. 
> 
> Users aren't going to call in some third-party service/support company, 
> either. 

Why not? You call a washing machine mechanic when the washing 
machine plays up. This is not conceptually different. 

> It call comes down to the network operator, one way or another. There's 
> no separation in the public mind of 'my network' from 'the Internet' 
> that is analogous to the separation between 'the power company' and 'the 
> electrical wiring in my house/apartment' (and even in that space, the 
> conceptual separation often isn't present). 

Actually I don't believe that. They do know what machines they 
have have connected to their home network. Boxes don't magically 
connect. Every machine was explictly connected. 

Mark 

> --- 
> Roland Dobbins <rdobb...@arbor.net> 
-- 
Mark Andrews, ISC 
1 Seymour St., Dundas Valley, NSW 2117, Australia 
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Sam Silvester]

On Tue, Sep 27, 2016 at 1:35 PM, Roland Dobbins  wrote:

> It call comes down to the network operator, one way or another.  There's
> no separation in the public mind of 'my network' from 'the Internet' that
> is analogous to the separation between 'the power company' and 'the
> electrical wiring in my house/apartment' (and even in that space, the
> conceptual separation often isn't present).
>
>
Not sure I agree with this. To my knowledge, when somebody loses power,
they go out and check circuit breakers and stuff, then either call an
electrician (if a breaker doesn't stay on or the like), or call their
electricity retailer/distributer. I'm not talking about IT / technically
savvy people either.

Now, I appreciate what you are saying though - end users are
(generalisation incoming, and I am not having a go / being a dick toward
end users) non-technical, busy and not willing to spend money on experts to
help out. They don't understand that their ISP is not responsible / in
control end to end etc, but yeah - not the best analogy above.

As a second comment...I think there is something also to be considered in
Mark's thoughts.

NAT obviously breaks visibility from a network operator's perspective. As
far as we can see, once a user is sending something flagged as abuse, the
best we can tell is the public IPv4 address. This sucks, as it basically
means suspend the user, who gets shitty as a result, and costs money and
time on the phone to helpdesk as a result.

In IPv6, it's not the case that all traffic is sourced from the same public
IP, which is interesting, especially if the network operator's abuse desk
has appropriate tooling to be able to marry that up to a device (probably
with the end user involved of course, but maybe with less effort).

I do also like the idea of IPv4 CPE having a menu displaying DHCP client
ID, in/out bps/pps counters, especially if that is able to be exposed to
the ISP helpdesk / abuse desk if needed. It's a nice to have, but not sure
it'd ever get meaningful deployment in a timeframe that makes it useful.

Food for thought.

Sam

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brielle Bruns]

On 9/26/16 10:05 PM, Roland Dobbins wrote:

+1 for this capability in CPE.

OTOH, it will be of no use whatsoever to the user.  Providing the user
with access to anomalous traffic feeds won't help, either.

Users aren't going to call in some third-party service/support company,
either.


You start cutting off users or putting them into a walled garden until 
they fix their machines, and they will start caring.


This will only work if all providers including cable, DSL and *shudders* 
WISPs (hate to be blunt, but who from my experience tend to be the least 
experienced and network knowledgeable people running a customer network) 
do it so customer's can't just switch networks and 'make the problem go 
away'.


I use escalating price increases and delays in service/repair time on 
some of my consulting customers who do things I warned them to be more 
careful about.


It takes time, but when $cost starts to become prohibitive, they stop 
and think.   And the ones that never learn...  Well, that's more $$$ in 
my pocket for the effort that I would normally charge otherwise.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jared Mauch]

> On Sep 26, 2016, at 7:58 PM, Christopher Morrow  
> wrote:
> 
> On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews  wrote:
> 
>> 
>> Giving them real time access to the anomalous traffic log feed for
>> their residence would also help.  They or the specialist they bring
>> in will be able to use that to trace back the problem.
>> 
>> 
> wouldn't this work better as a standard bit of CPE software capability?
> wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE
> and kept for ~30mins (just guessing) in a circular buffer be 'good enough'
> to present a pretty clear UI to the user?
> 
> ip/mac/vendor sending (webtraffic|email|probes) to destination-name
> [checkbox]
> 
> 
> 
> select those youd' like to block [clickhere]
> 
> This really doesn't seem hard, to present in a fairly straight forward
> manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something
> similar to this approach... but on the other hand:
>  "At least my ISP isn't snooping on all my traffic"

The UBNT Edgerouter series has this.  You can get fancy graphs and application
breakdown.

Scroll down and check the images:

https://help.ubnt.com/hc/en-us/articles/204951104-EdgeMAX-Deep-Packet-Inspection-Engine-for-EdgeRouter

You can see the hosts that are doing traffic and the destinations.

They even have a model that takes a SFP so you can use it as CPE for FTTH.

- Jared

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eliot Lear]

On 9/27/16 1:19 PM, Florian Weimer wrote:
> * Eliot Lear:
>
>> As some on this thread know, I've been working with the folks who make
>> light bulbs and switches.  They fit a certain class of device that is
>> not general purpose, but rather are specific in nature.  For those
>> devices it is possible for the manufacturers to inform the network what
>> the communication pattern of the device is designed to be.  This may be
>> extraordinarily broad or quite narrow, depending on the device. 
>> Conveniently, the technology for describing much of this dates back to
>> the paleolithic era in the form of access lists.  That is what
>> manufacturer usage descriptions are about.  (Yep- MUD.  There go my
>> marketing credentials). They're slightly abstracted for adaptation to
>> local deployments.   There's a draft and we authors are soliciting comments.
> What's the end goal here?  Charge extra if I'm connecting a TV instead
> of a light bulb?

Not my end goal.  My end goal is that consumers have a means to limit
risk in their home environments, and service providers have a means to
deliver that to them.

>
> I'm not convinced that expected traffic profiles are the right answer.
> We already have that in the server hosting market, and it does
> constraint the types of services you can run on hosted servers (for
> the hosting providers who does this).  I'm wary of the network putting
> severe constraints on application architecture, way beyond what is
> dictated by current technology.  NAT more or less killed servers on
> consumer networks, and this kind of traffic profiling has begun to
> kill clients on server networks.

The whole point of MUD is to leave control in the hands of those who
have developed and have to support Things.  It is not simply for the SP
to decide what traffic is ok, or to charge more for it, but to respect
the wishes of the developers.  That may be sufficient to stop a lot of
bad things from happening to a lot of Things.

>
>> The service providers has a strong role to play here, since they drive
>> standards for connectivity.  Having some access to residential CPE in
>> particular for this purpose, I believe, is very helpful because by doing
>> so not only can SPs protect others, but can also provide lateral
>> protection within the home.  As I wrote upthread, if consumers come to
>> see smart lightbulbs not just as useful, but also as a concern, they may
>> wish their SPs to help them.  That's the internalizing of an externality
>> that I see possible, and maybe even probable over time.
> Most service providers appear to be struggling to maintain up-to-date
> software on their CPEs (and their infrastructure), and following
> recommended configuration practices as they evolve.  This does not
> give me confidence that they could perform device management at
> consumer scale.

It's NOT device management.  Is network management.
>
> Do we know how much the average consumer trusts their ISP?  Would they
> want their ISP to control the digital (and increasingly, physical)
> doors in their home?  My own outlook is rather biased, unfortunately.
>
And again, this is the wrong way to look at it.  The consumer should
always get final say.  They're the customer.  This is a chance for the
manufacturer of the device they're using to explain how the device is
supposed to behave on the network.  Example: how many times have you
heard of some device leaving an extra service like SSH lying around? 
And would you really want that thermostat talking to ALL of the Internet
or just locally + its cloud service?  The manufacturer probably has a
view about that, and I bet it aligns with the consumer and with the
enterprise...

Eliot



signature.asc
Description: OpenPGP digital signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Florian Weimer]

* Eliot Lear:

> As some on this thread know, I've been working with the folks who make
> light bulbs and switches.  They fit a certain class of device that is
> not general purpose, but rather are specific in nature.  For those
> devices it is possible for the manufacturers to inform the network what
> the communication pattern of the device is designed to be.  This may be
> extraordinarily broad or quite narrow, depending on the device. 
> Conveniently, the technology for describing much of this dates back to
> the paleolithic era in the form of access lists.  That is what
> manufacturer usage descriptions are about.  (Yep- MUD.  There go my
> marketing credentials). They're slightly abstracted for adaptation to
> local deployments.   There's a draft and we authors are soliciting comments.

What's the end goal here?  Charge extra if I'm connecting a TV instead
of a light bulb?

I'm not convinced that expected traffic profiles are the right answer.
We already have that in the server hosting market, and it does
constraint the types of services you can run on hosted servers (for
the hosting providers who does this).  I'm wary of the network putting
severe constraints on application architecture, way beyond what is
dictated by current technology.  NAT more or less killed servers on
consumer networks, and this kind of traffic profiling has begun to
kill clients on server networks.

> The service providers has a strong role to play here, since they drive
> standards for connectivity.  Having some access to residential CPE in
> particular for this purpose, I believe, is very helpful because by doing
> so not only can SPs protect others, but can also provide lateral
> protection within the home.  As I wrote upthread, if consumers come to
> see smart lightbulbs not just as useful, but also as a concern, they may
> wish their SPs to help them.  That's the internalizing of an externality
> that I see possible, and maybe even probable over time.

Most service providers appear to be struggling to maintain up-to-date
software on their CPEs (and their infrastructure), and following
recommended configuration practices as they evolve.  This does not
give me confidence that they could perform device management at
consumer scale.

Do we know how much the average consumer trusts their ISP?  Would they
want their ISP to control the digital (and increasingly, physical)
doors in their home?  My own outlook is rather biased, unfortunately.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Florian Weimer]

* Mark Andrews:

> Dear customer,
>we are seeing  traffic coming from your network.
>
> If you need help isolating the source of the traffic here are a few
> companies in your city that can help you.
>
>   
>
> This is not a exhaustive list.
>
> Support

We already had the problem in the past that customer notifications for
compromised machines (with confirmed loss of user data, not just
sourcing unexpected packets) looked like advertisements for antivirus
products.  To most customers, this looks like just another scam.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eliot Lear]

John,

On 9/27/16 2:13 AM, John R. Levine wrote:
>> Therein lies the problem if the traffic does not look anomalous I
>> suppose. But even if it does look unusual, ISPs would be asking
>> consumers to trash/update/turn off a lot of devices in time – like
>> when every home has 10s or 100s of these devices.
>> ISP: Dear customer, looks like one of your light switches is sending
>> spam.
>> Customer: Which one? I have 25 light switches. And 25 smart bulbs.
>> And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and…
>
> That's why turning them off has to be mandatory if the ISP can't
> mitigate the traffic in real time.

As some on this thread know, I've been working with the folks who make
light bulbs and switches.  They fit a certain class of device that is
not general purpose, but rather are specific in nature.  For those
devices it is possible for the manufacturers to inform the network what
the communication pattern of the device is designed to be.  This may be
extraordinarily broad or quite narrow, depending on the device. 
Conveniently, the technology for describing much of this dates back to
the paleolithic era in the form of access lists.  That is what
manufacturer usage descriptions are about.  (Yep- MUD.  There go my
marketing credentials). They're slightly abstracted for adaptation to
local deployments.   There's a draft and we authors are soliciting comments.

The service providers has a strong role to play here, since they drive
standards for connectivity.  Having some access to residential CPE in
particular for this purpose, I believe, is very helpful because by doing
so not only can SPs protect others, but can also provide lateral
protection within the home.  As I wrote upthread, if consumers come to
see smart lightbulbs not just as useful, but also as a concern, they may
wish their SPs to help them.  That's the internalizing of an externality
that I see possible, and maybe even probable over time.

By the way, this isn't just about deliberate attacks.  Ask Raul Rojas
who built an IoT-based concept house and then had it taken down by a
failing lightbulb.[2]

Eliot

[1] https://tools.ietf.org/html/draft-ietf-opsawg-mud-00
[2]
http://fusion.net/story/55026/this-guys-light-bulb-ddosed-his-entire-smart-house/

>
> Sorry, but something in your house is attacking strangers.  Once you
> figure out what  it is, here's a handy list of links to the ongoing
> class action suits against the manufacturers.
>
> Regards,
> John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly
>




signature.asc
Description: OpenPGP digital signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 12:31, Jason Hofmann wrote:

It probably was a tough sell to get people to realize they were fully 
responsible for their in-home wiring, but optional "inside wire 
maintenance plans" made that clear while also adding to providers' 
coffers.  Perhaps something similar would work here.


Concur that this is the least-improbable model, absolutely.

But keep in mind that subscriptions/services for in-home wiring were 
(and are) also a tiny percentage of the user base.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 12:14, Mark Andrews wrote:

I'm yet to see a set top box, DVR, TV, games console, phone, etc. that 
didn't require selecting the WiFi SSID or require you to plug

in a ethernet cable.


I've 'seen' tens of millions of them, worldwide.

You're generalizing your particular connection/personal provisioning 
model.


As I said, they don't magically connect to the network.  Someone did 
something to permit them to connect.


That someone quite often isn't the end-user.  And as noted previously in 
this thread, even when users themselves do this, they promptly forget 
how they did it, lose the documentation, etc.


Why do you think people are incapable of calling in someone to help 
them fix a known issue.


1.  Because they demonstrably don't.

2.  Because it's not perceived as a 'computer problem' - it's perceived 
as an 'Internet problem', and the 'Internet technician' = the broadband 
access operator's help-desk.


3.  Going along with the line of reasoning you've expressed, it seems 
that the user should call a 'lightbulb technician' when his 
Internet-enabled lightbulb is causing a problem.  Do you really think 
that's realistic?


4.  In most cases, the user won't have any idea which connected device 
is causing the problem.  Expecting the user to determine this by 
trial-and-error is unrealistic; most people don't even understand how to 
troubleshoot electrical problems by trial-and-error, much less 
Internet-related problems.


You are a self-selected specialist, and understand all these things and 
have a DIY attitude, because you're an expert in this field.  Most 
people aren't experts in this field.


Ask yourself how many people set up and use 2FA for any online service 
which supports it, on their own initiative (i.e., not having a bank ship 
them a pre provisioned dongle).  The number of people capable of doing 
this troubleshooting for themselves is roughly equivalent to the number 
of people who've successfully set up 2FA on their own initiative.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message , Roland Dobbins 
writes:
> On 27 Sep 2016, at 11:43, Mark Andrews wrote:
> 
> > Why not?  You call a washing machine mechanic when the washing machine 
> > plays up.  This is not conceptually different.
> 
> Washing machines aren't a utility.  Internet is viewed as a utility.
> 
> > Actually I don't believe that.  They do know what machines they have 
> > have connected to their home network.  Boxes don't magically
> > connect.  Every machine was explictly connected.
> 
> First of all, not every devices was explicitly connected by the user.  
> Think set-top boxes/DVRs.

I'm yet to see a set top box, DVR, TV, games console, phone, etc.
that didn't require selecting the WiFi SSID or require you to plug
in a ethernet cable.  As I said, they don't magically connect to
the network.  Someone did something to permit them to connect.

> Secondly, users connect things an then don't think about them, don't 
> remember credentials, had a horrible ordeal (from their perspective) 
> 
> Thirdly, expecting users to troubleshoot which of their devices is 
> emanating bad traffic is unrealistic.

Which is why there are computer technitions.  If you have a fault
with a fan you call a electrian.  If you have a problem with a
toilet you call a plumber.  Why do you think people are incapable
of calling in someone to help them fix a known issue.

> The only effective consumer remediation efforts we've seen to date have 
> been broadband access ISPs proactively scanning their customer networks 
> and contacting them when exploitable devices and compromised PCs have 
> been found.  Although it's a lot of work, that kind of thing can be done 
> for CPE broadband routers; it can't be done for the things sitting 
> behind those devices, which are doing NAT/firewalling.  The partial 
> exception is PCs, because everyone thinks of those when they think of 
> 'the Internet'.
> 
> And the fact that even their lightbulbs are being connected now - i.e., 
> the huge proliferation of connected devices - militates against user 
> troubleshooting, as well.
> 
> ---
> Roland Dobbins 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 11:43, Mark Andrews wrote:

Why not?  You call a washing machine mechanic when the washing machine 
plays up.  This is not conceptually different.


Washing machines aren't a utility.  Internet is viewed as a utility.

Actually I don't believe that.  They do know what machines they have 
have connected to their home network.  Boxes don't magically

connect.  Every machine was explictly connected.


First of all, not every devices was explicitly connected by the user.  
Think set-top boxes/DVRs.


Secondly, users connect things an then don't think about them, don't 
remember credentials, had a horrible ordeal (from their perspective) 
connecting said devices and then promptly forgot how to administer them.


Thirdly, expecting users to troubleshoot which of their devices is 
emanating bad traffic is unrealistic.


The only effective consumer remediation efforts we've seen to date have 
been broadband access ISPs proactively scanning their customer networks 
and contacting them when exploitable devices and compromised PCs have 
been found.  Although it's a lot of work, that kind of thing can be done 
for CPE broadband routers; it can't be done for the things sitting 
behind those devices, which are doing NAT/firewalling.  The partial 
exception is PCs, because everyone thinks of those when they think of 
'the Internet'.


And the fact that even their lightbulbs are being connected now - i.e., 
the huge proliferation of connected devices - militates against user 
troubleshooting, as well.


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message , Roland Dobbins wri
tes:
> 
> On 27 Sep 2016, at 6:58, Christopher Morrow wrote:
> 
> > wouldn't something as simple as netflow/sflow/ipfix synthesized on the 
> > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good 
> > enough' to present a pretty clear UI to the user?
> 
> +1 for this capability in CPE.
> 
> OTOH, it will be of no use whatsoever to the user.  Providing the user 
> with access to anomalous traffic feeds won't help, either.
> 
> Users aren't going to call in some third-party service/support company, 
> either.

Why not?  You call a washing machine mechanic when the washing
machine plays up.  This is not conceptually different. 

> It call comes down to the network operator, one way or another.  There's 
> no separation in the public mind of 'my network' from 'the Internet' 
> that is analogous to the separation between 'the power company' and 'the 
> electrical wiring in my house/apartment' (and even in that space, the 
> conceptual separation often isn't present).

Actually I don't believe that.  They do know what machines they
have have connected to their home network.  Boxes don't magically
connect.  Every machine was explictly connected.

Mark

> ---
> Roland Dobbins 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Roland Dobbins]

On 27 Sep 2016, at 6:58, Christopher Morrow wrote:

wouldn't something as simple as netflow/sflow/ipfix synthesized on the 
CPE and kept for ~30mins (just guessing) in a circular buffer be 'good 
enough' to present a pretty clear UI to the user?


+1 for this capability in CPE.

OTOH, it will be of no use whatsoever to the user.  Providing the user 
with access to anomalous traffic feeds won't help, either.


Users aren't going to call in some third-party service/support company, 
either.


It call comes down to the network operator, one way or another.  There's 
no separation in the public mind of 'my network' from 'the Internet' 
that is analogous to the separation between 'the power company' and 'the 
electrical wiring in my house/apartment' (and even in that space, the 
conceptual separation often isn't present).


---
Roland Dobbins 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John R. Levine]

Therein lies the problem if the traffic does not look anomalous I suppose. But 
even if it does look unusual, ISPs would be asking consumers to 
trash/update/turn off a lot of devices in time – like when every home has 10s 
or 100s of these devices.
ISP: Dear customer, looks like one of your light switches is sending spam.
Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart 
TVs, and 3 smart thermostats, and 6 cameras, and…


That's why turning them off has to be mandatory if the ISP can't mitigate 
the traffic in real time.


Sorry, but something in your house is attacking strangers.  Once you 
figure out what  it is, here's a handy list of links to the ongoing class 
action suits against the manufacturers.


Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Christopher Morrow]

On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews  wrote:

>
> Giving them real time access to the anomalous traffic log feed for
> their residence would also help.  They or the specialist they bring
> in will be able to use that to trace back the problem.
>
>
wouldn't this work better as a standard bit of CPE software capability?
wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE
and kept for ~30mins (just guessing) in a circular buffer be 'good enough'
to present a pretty clear UI to the user?

ip/mac/vendor sending (webtraffic|email|probes) to destination-name
[checkbox]



select those youd' like to block [clickhere]

This really doesn't seem hard, to present in a fairly straight forward
manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something
similar to this approach... but on the other hand:
  "At least my ISP isn't snooping on all my traffic"

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message <20160926234142.6e7705515...@rock.dv.isc.org>, Mark Andrews writes:
> 
> In message <03dc1038-024a-4d9f-ac5b-3e88cdf56...@cable.comcast.com>, 
> "Livingood, Jason" writes:
> > On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews"  > ma...@isc.org> wrote:
> > > A good ISP would be informing their customers that they are seeing
> > anomalous traffic.
> >
> > Therein lies the problem if the traffic does not look anomalous I
> > suppose. But even if it does look unusual, ISPs would be asking consumers
> > to trash/update/turn off a lot of devices in time  like when every home
> > has 10s or 100s of these devices.
> > ISP: Dear customer, looks like one of your light switches is sending spam.
> > Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3
> > smart TVs, and 3 smart thermostats, and 6 cameras, and
> >
> > ;-)
> >
> > Jason
> >
> 
> Dear customer,
>we are seeing  traffic coming from your network.
> 
> If you need help isolating the source of the traffic here are a few
> companies in your city that can help you.
> 
>   
> 
> This is not a exhaustive list.
> 
> Support
> 
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Giving them real time access to the anomalous traffic log feed for
their residence would also help.  They or the specialist they bring
in will be able to use that to trace back the problem.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message <03dc1038-024a-4d9f-ac5b-3e88cdf56...@cable.comcast.com>, 
"Livingood, Jason" writes:
> On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews"  ma...@isc.org> wrote:
> > A good ISP would be informing their customers that they are seeing
> anomalous traffic.
>
> Therein lies the problem if the traffic does not look anomalous I
> suppose. But even if it does look unusual, ISPs would be asking consumers
> to trash/update/turn off a lot of devices in time  like when every home
> has 10s or 100s of these devices.
> ISP: Dear customer, looks like one of your light switches is sending spam.
> Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3
> smart TVs, and 3 smart thermostats, and 6 cameras, and
>
> ;-)
>
> Jason
>

Dear customer,
 we are seeing  traffic coming from your network.

If you need help isolating the source of the traffic here are a few
companies in your city that can help you.



This is not a exhaustive list.

Support

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Livingood, Jason]

On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews"  wrote:
> A good ISP would be informing their customers that they are seeing anomalous 
> traffic.

Therein lies the problem if the traffic does not look anomalous I suppose. But 
even if it does look unusual, ISPs would be asking consumers to 
trash/update/turn off a lot of devices in time – like when every home has 10s 
or 100s of these devices. 
ISP: Dear customer, looks like one of your light switches is sending spam.
Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart 
TVs, and 3 smart thermostats, and 6 cameras, and… 

;-)
 
Jason

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Livingood, Jason]

On 9/25/16, 5:57 PM, "NANOG on behalf of Patrick W. Gilmore" 
 wrote:
> Yeah, ‘cause that was so successful in the past.
>  Remember University of Wisconsin vs. D-Link and their hard-coded NTP server 
> address?

Ha! Yeah, an oldie but a goodie. Anyway, maybe this time will be different? 
(I’m an optimist.) FWIW, a few of the list members here are working on a BITAG 
paper on this – which will likely get some traction in policy/regulatory 
circles once completed. A simple paper certainly won’t turn the tide but if the 
paper is finished soon it presents an opportunity to get a bit wider notice & 
impact than it perhaps otherwise would. 

Jason

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message <20160926155649.14061.qm...@ary.lan>, "John Levine" writes:
> >>That paper is about reflection attacks.  From what I've read, this was 
> >>not a reflection attack.  The IoT devices are infected with botware 
> >>which sends attack traffic directly.  Address spoofing is not particularly 
> >>useful for controlling botnets.  
> >
> >But that's not only remaining use of source address spoofing in direct 
> >attacks, no?  Even if reflection and amplification are not used, spoofing 
> >can still be used for obfuscation.
> 
> I agree that it would be nice if more networks did ingress filtering,
> but if you're expecting a major decrease in evil, you will be
> disappointed.
> 
> At this point it's mostly useful for identifying the guilty or
> negligent parties afterwards.
> 
> R's,
> John

Actually for ISP's that do it, it can becomes a source of intelligence
on where the compromised / misconfigured machines are.  A good ISP
would be informing their customers that they are seeing anomalous
traffic.

With IPv6 there is likely to be more of this traffic visible as
home NATs wont be masking it.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John Levine]

>>That paper is about reflection attacks.  From what I've read, this was 
>>not a reflection attack.  The IoT devices are infected with botware 
>>which sends attack traffic directly.  Address spoofing is not particularly 
>>useful for controlling botnets.  
>
>But that's not only remaining use of source address spoofing in direct 
>attacks, no?  Even if reflection and amplification are not used, spoofing 
>can still be used for obfuscation.

I agree that it would be nice if more networks did ingress filtering,
but if you're expecting a major decrease in evil, you will be
disappointed.

At this point it's mostly useful for identifying the guilty or
negligent parties afterwards.

R's,
John

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Royce Williams]

On Mon, Sep 26, 2016 at 7:23 AM, Mark Milhollan  wrote:
>
> On Sun, 25 Sep 2016, Stephen Satchell wrote:
>
> >Yeah, right.  I looked at BCP38.info, and there is very little concrete
> >information.
>
> Yeah, it's pretty naked.  But how-to isn't the usual stumbling block, as
> has been pointed out in this thread there needs to be the will to spend
> resources setting it up and thus committing future resources to
> maintenance.

Actually, a clear set of how-tos is the best way to quickly convey --
to busy geeks, and to their busy managers and PMs -- how much that
resource spend would probably be.

Royce

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Milhollan]

On Sun, 25 Sep 2016, Stephen Satchell wrote:

>Yeah, right.  I looked at BCP38.info, and there is very little concrete
>information.  

Yeah, it's pretty naked.  But how-to isn't the usual stumbling block, as 
has been pointed out in this thread there needs to be the will to spend 
resources setting it up and thus committing future resources to 
maintenance.

>I've been slogging through the two RFCs, 2827 and 3794, and find
>it tough sledding to extract the nuggets to put into my firewall and routing
>table.  One of the more interesting new additions to my systems is this, to the
>routing tables:

A list of martian addresses is useful to avoid sending to or accepting 
from weird places but it isn't useful for BCP38 purposes, of ensuring a 
node only uses address(es) assigned to it as the source address in the 
packets it creates/sends.

BGP38 checking is not done by the node itself, though that is not 
entirely unreasonable.  Enforcement is performed by the network as close 
to the point of the node's attachment as possible -- failures should be 
discarded or perhaps returned as prohibited and possibly even sampled 
for use by network staff to work on remediation.  In some cases it's 
very simple and effective to just filter toward your upstream(s), i.e., 
allow this area's addresses and drop/reject/log the rest.

>(Has this been published anywhere before?  I haven't found any yet.)

Cymru has lists in various formats and levels of (de)aggregation and 
detail that you can easily turn into those commands, though there's no 
martians-only lists for IPv6.  You might even use one of the "fullbogon" 
lists to block at a very detailed level if you have sufficient resources 
or tools that keep needs light, e.g., ipset.

>In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY filtering
>system -- BSD, Linux, Cisco.

For some it's just uRPF, strict or loose as your needs demand, which 
their router can already perform, e.g.,

  interface your-template/or/range-of-interfaces/etc
ip verify unicast reverse-path
 or ip verify unicast source reachable-via rx
 or ip verify unicast source reachable-via any

Which for Linux is controlled by the net.ipv4.conf.if.rp_filter sysctl 
key where "if" can be "default", "all" or a specific interface name and 
a setting of 1 does strict checking while 2 does loose.

Or there's always plain old packet filters, with varying degrees of ease 
or annoyance, as tightly (per customer applied to incoming packets 
received on their interface) or simply (leaving the pop) as you please 
and makes sense.


/mark

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John Kristoff]

On Sun, 25 Sep 2016 22:59:15 +
Stephen Satchell  wrote:

> In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY 
> filtering system -- BSD, Linux, Cisco.

There is some here for integrating Team Cymru's bogon BGP service into
various router platforms:

  

John

Re: BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ]

[Vincent Bernat]

 ❦ 26 septembre 2016 09:14 CEST, valdis.kletni...@vt.edu :

>> Linux:
>> From /etc/sysctl.conf:
>>
>> # Uncomment the next two lines to enable Spoof protection (reverse-path=20
>> # filter)
>> # Turn on Source Address Verification in all interfaces to
>> # prevent some spoofing attacks
>> net.ipv4.conf.default.rp_filter=1
>> net.ipv4.conf.all.rp_filter=1

Only "all" is needed since the kernel will use the max of all and the
current interface value.

>> Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a
>> thing on Linux.
>
> See net/ipv6/netfilter/ip6t_rpfilter.c
>
> Also, note that a lot of net.ipv4.conf variables also apply to ipv6 (though
> checking the source tree, this isn't one of them, unless it's via a  macro 
> that
> some quick grepping didn't find...)

Yes, it doesn't apply. In Linux, there is no such thing as feature
parity for IPv6. davem said in the past that he didn't want this feature
in IPv6 and was planning to remove it in IPv4 (but I think this will
never happen):
 http://www.spinics.net/lists/netdev/msg166280.html

I am using this instead (assuming ip46tables is iptables + ip6tables):

ip46tables -t raw -N RPFILTER
ip46tables -t raw -A RPFILTER -m rpfilter -j RETURN
iptables   -t raw -A RPFILTER -d 255.255.255.255 -p udp --sport bootpc --dport 
bootps -j RETURN
ip6tables  -t raw -A RPFILTER -m rpfilter --accept-local -m addrtype --dst-type 
MULTICAST -j DROP
ip46tables -t raw -A RPFILTER -m limit --limit 5/s --limit-burst 5 \
   -j NFLOG --nflog-group 99 \
   --nflog-prefix "NF: rpfilter: "
ip46tables -t raw -A RPFILTER -j DROP
ip46tables -t raw -A PREROUTING -j RPFILTER
-- 
Use data arrays to avoid repetitive control sequences.
- The Elements of Programming Style (Kernighan & Plauger)

Re: BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ]

[Valdis . Kletnieks]

On Sun, 25 Sep 2016 21:19:31 -0700, Hugo Slabbert said:

> Linux:
> From /etc/sysctl.conf:
>
> # Uncomment the next two lines to enable Spoof protection (reverse-path=20
> # filter)
> # Turn on Source Address Verification in all interfaces to
> # prevent some spoofing attacks
> net.ipv4.conf.default.rp_filter=1
> net.ipv4.conf.all.rp_filter=1
>
> Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a
> thing on Linux.

See net/ipv6/netfilter/ip6t_rpfilter.c

Also, note that a lot of net.ipv4.conf variables also apply to ipv6 (though
checking the source tree, this isn't one of them, unless it's via a  macro that
some quick grepping didn't find...)


pgptJL_xNvOlh.pgp
Description: PGP signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eliot Lear]

Hi Ryan,


On 9/25/16 11:50 PM, ryan landry wrote:

> for isp's it's a resourcing vs revenue problem. always has been. 

Sure.  The question is whether IoT can make a change in consumer
attitudes.  Riek, Bohme, et al have been working on this [1].  And there
is earlier work as well.  What that earlier work shows, by the way, is
that if someone suffers a loss, or even if they know someone who suffers
a loss, they'll become considerably more risk averse towards Internet
technology, to one extent or another.  The Riek analysis doesn't really
take into account IoT, by the way.  It just looks at losses.  But I
think the logic is likely to hold as IoT creates more risks.  The
question is whether the impact will increase, and whether those losses
will motivate market opportunities for SPs.  I think there's a good
chance of that if the solution doesn't involve a vast amount of work on
the consumer's part.

Eliot
[1] "Estimating the costs of consumer-facing cybercrime: A tailored
instrument and representative data for six EU countries/",
/http://weis2016.econinfosec.org/wp-content/uploads/sites/2/2016/05/WEIS_2016_paper_54-2.pdf
 


signature.asc
Description: OpenPGP digital signature

BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ]

[Hugo Slabbert]

On Sun 2016-Sep-25 15:59:15 -0700, Stephen Satchell  wrote:


On 09/25/2016 07:32 AM, Jay R. Ashworth wrote:

From: "Jay Farrell via NANOG" 

And of course Brian Krebs has a thing or two to say, not the least is which
to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Well, given how few contributions we've gotten at bcp38.info in the last,
what, 4 years, yeah, I guess so...



Yeah, right.  I looked at BCP38.info, and there is very little 
concrete information.  I've been slogging through the two RFCs, 2827 
and 3794, and find it tough sledding to extract the nuggets to put 
into my firewall and routing table.  One of the more interesting new 
additions to my systems is this, to the routing tables:



### snip ###


In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY 
filtering system -- BSD, Linux, Cisco.


I am guilty of not yet contributing cookbook-type info to BCP38.info, but:

Cisco:
http://www.bcp38.info/index.php/HOWTO:Cisco points at 
http://www.cisco.com/c/en/us/about/security-center/unicast-reverse-path-forwarding.html


Juniper:
https://www.juniper.net/documentation/en_US/junos14.2/topics/usage-guidelines/interfaces-configuring-unicast-rpf.html
http://www.juniper.net/documentation/en_US/junos15.1/topics/topic-map/unicast-rpf.html

Linux:
From /etc/sysctl.conf:

# Uncomment the next two lines to enable Spoof protection (reverse-path 
# filter)

# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a 
thing on Linux.


For a belt-and-suspenders approach:
If you're running an edge network and not transiting traffic for any other 
AS, consider using your assigned aggregates prefix lists to filter on 
egress on your edge for anything not sourced from those aggregates.


I'm curious as to the deployment scope and experiences of various sizes of 
networks in deploying the following:


1.  Strict uRPF on customer-facing ports on edge networks

2.  Source address filtering on upstream edge egress based on assigned 
aggregates


3.  Destination address filtering on upstream edge ingress based on 
assigned aggregates


--
Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
pgp key: B178313E   | also on Signal


signature.asc
Description: PGP signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Hugo Slabbert]

On Sun 2016-Sep-25 17:01:55 -0400, John R. Levine  wrote:


https://www.internetsociety.org/sites/default/files/01_5.pdf

The attack is triggered by a few spoofs somewhere in the world. It is not
feasible to stop this.


That paper is about reflection attacks.  From what I've read, this was 
not a reflection attack.  The IoT devices are infected with botware 
which sends attack traffic directly.  Address spoofing is not particularly 
useful for controlling botnets.  


But that's not only remaining use of source address spoofing in direct 
attacks, no?  Even if reflection and amplification are not used, spoofing 
can still be used for obfuscation.


For example, the Conficker botnet generated pseudo-random domain names 
where the bots looked for control traffic.



Please see https://www.ietf.org/rfc/rfc6561.txt


Uh, yes, we're familiar with that.  We even know the people who wrote 
it. It could use an update for IoT since I get the impression that in 
many cases the only way for a nontechnical user to fix the infection 
is to throw the device away.


Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly


--
Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
pgp key: B178313E   | also on Signal


signature.asc
Description: PGP signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John R. Levine]

It’s safe to ignore the silent minority that cannot really tell what is 
happening in most cases, but that doesn’t mean it “works” for any standard I 
would consider valid.


Huh.  So you're saying Bill Woodcock doesn't have the skills to see how 
his traffic is failing?


Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Owen DeLong]

Assuming all transit providers your packets may traverse on the way to all of 
your
customers is the kind of thing that leads to me quoting Mr. Bush…

“I encourage my competitors to try this.”

Owen

> On Sep 25, 2016, at 6:32 PM, Mark Andrews  wrote:
> 
> 
> In message , Owen DeLong 
> writes:
>> 
>>> On Sep 24, 2016, at 8:47 AM, John Levine  wrote:
>>> 
> Well...by anycast, I meant BGP anycast, spreading the "target"
> geographically to a dozen or more well connected/peered origins.  At
>> that
> point, your ~600G DDoS might only be around
 
 anycast and tcp? the heck you say! :)
>>> 
>>> People who've tried it say it works fine.  Routes don't flap that often.
>> 
>> It’s not just about route flap.
>> 
>> Imagine the following. For any two any cast points A,B, one can draw a
>> simple Venn diagram of two circles with equal radii overlapping to form
>> an OGIVE.
>> 
>> Consider that everyone in the nonintersecting portion of circle A will
>> reach server A without issue.
>> Likewise, everyone in the nonintersecting portion of circle B will reach
>> server B without issue.
>> However, for some subset of those within the OGIVE, it’s entirely likely
>> that they will, instead, be broken by ECMP to both A and B.
>> 
>> Here’s where it gets tricky…
>> 
>> The people running A and B are unlikely to ever know because of the
>> layers between the end user trapped in the OGIVE and the people running A
>> and B. Most likely, the end users will suffer in silence or go to another
>> website for their needs. If this is a small enough fraction of users,
>> then it won’t be statistically noticeable drop in overall traffic and A,B
>> may never know. For those few end-users that may actually attempt to
>> resolve the issue in some meaningful way, most likely they will call
>> their ISP rather than the administrators of A,B and if their ISP does
>> anything, rather than bug A,B, they will most likely simple make routing
>> more deterministic for this site for this end-user.
>> 
>> This is the nature of any cast and how any cast problems with TCP get
>> solved (or don’t in most cases).
>> 
>> It’s safe to ignore the silent minority that cannot really tell what is
>> happening in most cases, but that doesn’t mean it “works” for any
>> standard I would consider valid.
>> 
>> Owen
> 
> Which is why sane operators carefully choose where they deploy ECMP
> or include hashes of the source and destination addresses into the
> distribution of traffic over otherwise equal paths.
> 
> Mark
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

In message , Owen DeLong 
writes:
>
> > On Sep 24, 2016, at 8:47 AM, John Levine  wrote:
> >
> >>> Well...by anycast, I meant BGP anycast, spreading the "target"
> >>> geographically to a dozen or more well connected/peered origins.  At
> that
> >>> point, your ~600G DDoS might only be around
> >>
> >> anycast and tcp? the heck you say! :)
> >
> > People who've tried it say it works fine.  Routes don't flap that often.
>
> It’s not just about route flap.
>
> Imagine the following. For any two any cast points A,B, one can draw a
> simple Venn diagram of two circles with equal radii overlapping to form
> an OGIVE.
>
> Consider that everyone in the nonintersecting portion of circle A will
> reach server A without issue.
> Likewise, everyone in the nonintersecting portion of circle B will reach
> server B without issue.
> However, for some subset of those within the OGIVE, it’s entirely likely
> that they will, instead, be broken by ECMP to both A and B.
>
> Here’s where it gets tricky…
>
> The people running A and B are unlikely to ever know because of the
> layers between the end user trapped in the OGIVE and the people running A
> and B. Most likely, the end users will suffer in silence or go to another
> website for their needs. If this is a small enough fraction of users,
> then it won’t be statistically noticeable drop in overall traffic and A,B
> may never know. For those few end-users that may actually attempt to
> resolve the issue in some meaningful way, most likely they will call
> their ISP rather than the administrators of A,B and if their ISP does
> anything, rather than bug A,B, they will most likely simple make routing
> more deterministic for this site for this end-user.
>
> This is the nature of any cast and how any cast problems with TCP get
> solved (or don’t in most cases).
>
> It’s safe to ignore the silent minority that cannot really tell what is
> happening in most cases, but that doesn’t mean it “works” for any
> standard I would consider valid.
>
> Owen

Which is why sane operators carefully choose where they deploy ECMP
or include hashes of the source and destination addresses into the
distribution of traffic over otherwise equal paths.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Owen DeLong]

> On Sep 24, 2016, at 8:47 AM, John Levine  wrote:
> 
>>> Well...by anycast, I meant BGP anycast, spreading the "target"
>>> geographically to a dozen or more well connected/peered origins.  At that
>>> point, your ~600G DDoS might only be around
>> 
>> anycast and tcp? the heck you say! :)
> 
> People who've tried it say it works fine.  Routes don't flap that often.

It’s not just about route flap.

Imagine the following. For any two any cast points A,B, one can draw a simple 
Venn diagram of two circles with equal radii overlapping to form an OGIVE.

Consider that everyone in the nonintersecting portion of circle A will reach 
server A without issue.
Likewise, everyone in the nonintersecting portion of circle B will reach server 
B without issue.
However, for some subset of those within the OGIVE, it’s entirely likely that 
they will, instead, be broken by ECMP to both A and B.

Here’s where it gets tricky…

The people running A and B are unlikely to ever know because of the layers 
between the end user trapped in the OGIVE and the people running A and B. Most 
likely, the end users will suffer in silence or go to another website for their 
needs. If this is a small enough fraction of users, then it won’t be 
statistically noticeable drop in overall traffic and A,B may never know. For 
those few end-users that may actually attempt to resolve the issue in some 
meaningful way, most likely they will call their ISP rather than the 
administrators of A,B and if their ISP does anything, rather than bug A,B, they 
will most likely simple make routing more deterministic for this site for this 
end-user.

This is the nature of any cast and how any cast problems with TCP get solved 
(or don’t in most cases).

It’s safe to ignore the silent minority that cannot really tell what is 
happening in most cases, but that doesn’t mean it “works” for any standard I 
would consider valid.

Owen

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Stephen Satchell]

On 09/25/2016 07:32 AM, Jay R. Ashworth wrote:

From: "Jay Farrell via NANOG" 
> And of course Brian Krebs has a thing or two to say, not the least is which
> to push for BCP38 (good luck with that, right?).
>
> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Well, given how few contributions we've gotten at bcp38.info in the last,
what, 4 years, yeah, I guess so...



Yeah, right.  I looked at BCP38.info, and there is very little concrete 
information.  I've been slogging through the two RFCs, 2827 and 3794, 
and find it tough sledding to extract the nuggets to put into my 
firewall and routing table.  One of the more interesting new additions 
to my systems is this, to the routing tables:


ip route add blackhole 0.0.0.0/32   # broadcast (deprecated)
ip route add blackhole 0.0.0.0/8# “this”
ip route add blackhole 10.0.0.0/8   # private network
ip route add blackhole 100.64.0.0/10# carrier-grade NAT
ip route add blackhole 127.0.0.0/8  # loopback
ip route add blackhole 169.254.0.0/16   # link-local
ip route add blackhole 172.16.0.0/12# private network
ip route add blackhole 192.0.0.0/24 # IANA special purpose registry
ip route add blackhole 192.0.2.0/24 # TEST-NET
ip route add blackhole 192.88.99.0/24   # 6to4 anycast relay (optional)
ip route add blackhole 192.168.0.0/16   # private network
ip route add blackhole 198.18.0.0/15# inter-network testing
ip route add blackhole 198.51.100.0/24  # TEST-NET-2
ip route add blackhole 203.0.113.0/24   # TEST-NET-3
ip route add blackhole 224.0.0.0/4  # Multicast
(all the implied routes from interface definitions that are inserted 
automatically into the routing table by the operating system)

ip route add  via  dev 
ip route add  via  dev 

(Has this been published anywhere before?  I haven't found any yet.)

Notice I've omitted 255.255.255.255, because I have quite a bit of 
software that uses this broadcast address, and it needs to get onto my 
intranets; I just have to be sure that the edge link doesn't let it out 
to the world.  One such program that uses broadcast is Dropbox.


This routing table addition forms the nucleus from which I can derive 
most of the source-address input packet filtering (adding 
255.255.255.255), as well as destination-address output filtering with 
code, instead of having to compile the rules by hand.


Then there are other packet filters:
* TCP “Lamp test/XMAS” et. al.  (mask, set)
--tcp-flags FIN,SYN,RST,PSH,ACK,URG   NONE
--tcp-flags FIN,SYN,RST,ACK,URG   FIN,SYN,RST,ACK,URG
--tcp-flags FIN,SYN,RST,PSH,ACK,URG   FIN,PSH,URG
--tcp-flags FIN,ACK  FIN
--tcp-flags PSH,ACK  PSH
--tcp-flags URG,ACK  URG
--tcp-flags SYN,FIN  SYN,FIN
--tcp-flags SYN,RST  SYN,RST
--tcp-flags FIN,RST  FIN,RST
* DNS flood
* DNS amplification
* NTP flood
* NTP amplification
* ICMP flood
* IGRP flood
* protocol/port “small services” connections

And I'm just getting started.  It helps that I have a few books talking 
about the design of firewalls that discuss some of these filters.


For DNS amplification, I'm using these IPTABLES rules:


# Limit UDP DNS "root NS" queries (must come before generic stream ACCEPT)
#  
01210001020001
-A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string 
"|01210001020001|" --algo bm --from 20 --to 1550 -m recent 
--set --name dnsrootquery --rsource
-A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string 
"|01210001020001|" --algo bm --from 20 --to 1550 -m recent --rcheck 
--seconds 20 --hitcount 3 --name dnsrootquery --rsource -m comment --comment "UDP DNS ROOT 
flood" -j DROP


(Another possibility would be to read the DNS hints file for all the 
root servers, and impose rate-limiting to those IP addresses...but I 
didn't think of that before and so it's not part of the current DNS 
server firewall.  But the filters above seem to work quite well -- it's 
been a long time since my ISP upstream's uplink has been nailed up solid.)


In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY 
filtering system -- BSD, Linux, Cisco.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Nick Hilliard]

Baldur Norddahl wrote:
> The sad thing is that if we boot out grandma they will just switch to one
> of our competors and the TV will still be a bot. You can't win.

Good thing the smart TV / other IoT manufacturers have taken the
responsible approach and have committed to providing lifetime software
updates for all the Internet-connected devices they manufacture.

Nick

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Baldur Norddahl]

> i wish you luck with that. explaining to grandma that her samsung smart tv
> has been rooted and needs to be updated should be good fun.

The sad thing is that if we boot out grandma they will just switch to one
of our competors and the TV will still be a bot. You can't win.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Patrick W. Gilmore]

On Sep 25, 2016, at 5:50 PM, ryan landry  wrote:
> On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews  wrote:

>> This is such a golden opportunity for each of you to find compromised
>> hosts on your network or your customer's network.  The number of
>> genuine lookups of the blog vs the number of botted machine would
>> make it almost certain that anything directed at the blog is a
>> compromised machine.  A phone call to the customer / further analysis
>> would reduce the false positive rate.
>> 
>> Mark
>> 
>> 
> i wish you luck with that. explaining to grandma that her samsung smart tv
> has been rooted and needs to be updated should be good fun.
> 
> for isp's it's a resourcing vs revenue problem. always has been. always
> will be. far more inclined to hold liable the folks that are churning out
> terribly dangerous cpe / IoT(shit). surely some regulatory body is looking
> into this.

Yeah, ‘cause that was so successful in the past.

Remember University of Wisconsin vs. D-Link and their hard-coded NTP server 
address?

-- 
TTFN,
patrick

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[ryan landry]

On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews  wrote:

>
> This is such a golden opportunity for each of you to find compromised
> hosts on your network or your customer's network.  The number of
> genuine lookups of the blog vs the number of botted machine would
> make it almost certain that anything directed at the blog is a
> compromised machine.  A phone call to the customer / further analysis
> would reduce the false positive rate.
>
> Mark
>
>
i wish you luck with that. explaining to grandma that her samsung smart tv
has been rooted and needs to be updated should be good fun.

for isp's it's a resourcing vs revenue problem. always has been. always
will be. far more inclined to hold liable the folks that are churning out
terribly dangerous cpe / IoT(shit). surely some regulatory body is looking
into this.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mark Andrews]

This is such a golden opportunity for each of you to find compromised
hosts on your network or your customer's network.  The number of
genuine lookups of the blog vs the number of botted machine would
make it almost certain that anything directed at the blog is a
compromised machine.  A phone call to the customer / further analysis
would reduce the false positive rate.

Mark

In message 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John R. Levine]

https://www.internetsociety.org/sites/default/files/01_5.pdf

The attack is triggered by a few spoofs somewhere in the world. It is not
feasible to stop this.


That paper is about reflection attacks.  From what I've read, this was not 
a reflection attack.  The IoT devices are infected with botware which 
sends attack traffic directly.  Address spoofing is not particularly 
useful for controlling botnets.  For example, the Conficker botnet 
generated pseudo-random domain names where the bots looked for control 
traffic.



Please see https://www.ietf.org/rfc/rfc6561.txt


Uh, yes, we're familiar with that.  We even know the people who wrote it. 
It could use an update for IoT since I get the impression that in many 
cases the only way for a nontechnical user to fix the infection is to 
throw the device away.


Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Alexander Lyamin]

This time around its not about spoofing.

I presume this is development of the same botnet/worm that we seen day2 of
Shellshock public disclosure - its was pretty hightech - golang,
arm/mips/x86 support, multiple attack vectors - inlcuding (surprisingly)
very effective password guessing.
It counted  ~100k heads on day2,  and i suppose they did grew quite a bit.


Thats part of a problem why cause that much havoc - they do have real IP
addresses and reasonably well conected - so they can wreck a havoc in
bandwidth and tcp stack.

They most likely do not have enough resources to do Full Browser Stack,
thats why I think  L7 capabilities of the botnet will be very basic.



On Sun, Sep 25, 2016 at 7:00 PM, John Kristoff  wrote:

> On Sun, 25 Sep 2016 14:36:18 +
> Ca By  wrote:
>
> > As long as their is one spoof capable network on the net, the problem
> will
> > not be solved.
>
> This is not strictly true.  If it could be determined where a large
> bulk of the spoofing came from, public pressure could be applied.  This
> may not have been the issue in this case, but in many amplification and
> reflection attacks, the originating spoof-enabled networks were from a
> limited set of networks.  De-peering, service termination, shaming, etc
> could have an effect.
>
> John
>



-- 

Alexander Lyamin

CEO | Qrator * Labs*

office: 8-800--LAB (522)

mob: +7-916-9086122

skype: melanor9

mailto:  l...@qrator.net

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brandon Butterworth]

> From deles...@gmail.com  Sun Sep 25 20:26:56 2016
> Sorry you don't understand how multinational companies and
> peering agreements work

Right, thanks for letting me know.

> nor any of the relationships my past networks would of had with akamai

I don't care what yours were in the past, if peering agreements
don't account for ddos maybe it's time they changed to do so.

ddos protection is getting like anti virus where it's a big enough
business the companies selling protection would like it to carry on
forever (noting you have a stake in ddos don't take it personally)

brandon

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[jim deleskie]

Brandon,

 Sorry you don't understand how multinational companies and peering agreements 
work, nor any of the relationships my past networks would of had with akamai.  
But be confident in the fact none of your concerns would of been an issue and 
it certainly wasn't because decisions were made with out all aspects being 
taken into play

-jim

  Original Message  
From:bran...@rd.bbc.co.uk
Sent:September 25, 2016 3:16 PM
To:cb.li...@gmail.com; deles...@gmail.com
Cc:nanog@nanog.org; j...@aharp.iorc.depaul.edu
Subject:Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey

> From: jim deleskie 
> Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and
> medium sized networks.  What does it cost and what do we make doing it,
> over rules what is "good for the internet" every time it came up.

"nice network you have there, shame if something were to happen to it"

one day they may be the target themselves then they can explain to
shareholders their part in enabling so much business disruption

Sadly it seems there will always be an exploding Pinto on the internet

Perhaps Akamai could present them with a bill for unwanted traffic
as they're monetising ddos they may as well charge both sides and
having dropped Krebs due to the disruption a court may agree damages
too.

brandon

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brandon Butterworth]

> From: jim deleskie 
> Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and
> medium sized networks.  What does it cost and what do we make doing it,
> over rules what is "good for the internet" every time it came up.

"nice network you have there, shame if something were to happen to it"

one day they may be the target themselves then they can explain to
shareholders their part in enabling so much business disruption

Sadly it seems there will always be an exploding Pinto on the internet

Perhaps Akamai could present them with a bill for unwanted traffic
as they're monetising ddos they may as well charge both sides and
having dropped Krebs due to the disruption a court may agree damages
too.

brandon

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Chris Woodfield]

> On Sep 24, 2016, at 7:47 AM, John Levine  wrote:
> 
>>> Well...by anycast, I meant BGP anycast, spreading the "target"
>>> geographically to a dozen or more well connected/peered origins.  At that
>>> point, your ~600G DDoS might only be around
>> 
>> anycast and tcp? the heck you say! :)
> 
> People who've tried it say it works fine.  Routes don't flap that often.
> 

There are a number of companies terminating anycasted TCP endpoints without 
issue. It’s not exactly turnkey, but it’s hardly black magic either. 

Here’s Nick Holt @Microsoft presenting their experience: 
https://www.youtube.com/watch?v=40MONHHF2BU 
 

-Chris

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Eliot Lear]

Has anyone stopped to consider what a gift these hackers gave all of
us?  They exposed their capabilities and nobody got hurt.  We all had a
notion as to what sort of attacks were possible in theory.  Now we have
reality.  Business being what it is, customers may not be interested in
others' security, but IoT being what it is, they might be interested in
their own: in this instance, as I understand it, cameras were involved. 
If a camera could be used to attack someone else, it could be used to
invade the privacy of the owner.  If consumers come to see that as a
threat, that'd be a good first step to internalizing what was an
externality.  At that point you can sell something.

Big if, though.

Eliot



On 9/25/16 7:00 PM, John Kristoff wrote:
> On Sun, 25 Sep 2016 14:36:18 +
> Ca By  wrote:
>
>> As long as their is one spoof capable network on the net, the problem will
>> not be solved.
> This is not strictly true.  If it could be determined where a large
> bulk of the spoofing came from, public pressure could be applied.  This
> may not have been the issue in this case, but in many amplification and
> reflection attacks, the originating spoof-enabled networks were from a
> limited set of networks.  De-peering, service termination, shaming, etc
> could have an effect.
>
> John
>




signature.asc
Description: OpenPGP digital signature

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Sunday, September 25, 2016, jim deleskie  wrote:

> Sorry but you are mistaken. I've worked at Sr. levels for several LARGE
> and medium sized networks.
>
> mazel tov


>
> What does it cost and what do we make doing it, over rules what is "good
> for the internet" every time it came up.
>
>
100% agree

Thats why i want to see a pie chart of attribution. Charter had this,
vz had that, and so on.

Headline reads "xyz isp totally hacked network overrun with bots takes down
journalists...FCC and DHS demand heads role ... congress yells at ceo...
investors dump stock"

Perhaps release the article to the brass first, with an alternate ate
 headline "xyz isp seriously commit to security partners to secure critical
infrastructure "

You have 2 weeks to pick the story


>
> On Sun, Sep 25, 2016 at 2:27 PM, Ca By  > wrote:
>
>> On Sunday, September 25, 2016, John Kristoff > > wrote:
>>
>> > On Sun, 25 Sep 2016 14:36:18 +
>> > Ca By >  >
>> wrote:
>> >
>> > > As long as their is one spoof capable network on the net, the problem
>> > will
>> > > not be solved.
>> >
>> > This is not strictly true.  If it could be determined where a large
>> > bulk of the spoofing came from, public pressure could be applied.  This
>> > may not have been the issue in this case, but in many amplification and
>> > reflection attacks, the originating spoof-enabled networks were from a
>> > limited set of networks.  De-peering, service termination, shaming, etc
>> > could have an effect.
>> >
>> > John
>> >
>>
>> Ok, sorry for the not being exact. I am trying to be practical.
>>
>> My point is, a lot of access networks will respond to public pressure if
>> the data is exposed on the offending real ips of the iot crap, and they
>> will enforce their AUP.
>>
>> We have seen comcast do just that, on this list a few months back. That
>> path has legs.
>>
>> Google also blocks service to certain hacked networks as well, we have
>> seen
>> that on this list too. That is an interesting angle in the krebs case.
>> Will
>> google block service to folks sharing ip with the iot  ddos mess ?
>>
>
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[jim deleskie]

Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and
medium sized networks.  What does it cost and what do we make doing it,
over rules what is "good for the internet" every time it came up.

On Sun, Sep 25, 2016 at 2:27 PM, Ca By  wrote:

> On Sunday, September 25, 2016, John Kristoff  wrote:
>
> > On Sun, 25 Sep 2016 14:36:18 +
> > Ca By > wrote:
> >
> > > As long as their is one spoof capable network on the net, the problem
> > will
> > > not be solved.
> >
> > This is not strictly true.  If it could be determined where a large
> > bulk of the spoofing came from, public pressure could be applied.  This
> > may not have been the issue in this case, but in many amplification and
> > reflection attacks, the originating spoof-enabled networks were from a
> > limited set of networks.  De-peering, service termination, shaming, etc
> > could have an effect.
> >
> > John
> >
>
> Ok, sorry for the not being exact. I am trying to be practical.
>
> My point is, a lot of access networks will respond to public pressure if
> the data is exposed on the offending real ips of the iot crap, and they
> will enforce their AUP.
>
> We have seen comcast do just that, on this list a few months back. That
> path has legs.
>
> Google also blocks service to certain hacked networks as well, we have seen
> that on this list too. That is an interesting angle in the krebs case. Will
> google block service to folks sharing ip with the iot  ddos mess ?
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Sunday, September 25, 2016, John Levine  wrote:

> >> Yeh, bcp38 is not a viable solution.
>
> Krebs said this DDoS came from insecure IoT devices, of which there
> are a kazillion, with the numbers growing every day.  Why would they
> need to spoof IPs?  How would BCP38 help?
>
> R's,
> John
>

Worth reading to level set

 https://www.internetsociety.org/sites/default/files/01_5.pdf

The attack is triggered by a few spoofs somewhere in the world. It is not
feasible to stop this.

The attack traffic that blows up to 600gbs is from traceable iot crap , the
victim knows who is sending the packers (iot crap) and the access network
(comcast, att ...) has the AUP authority to shut it down.

One by one.

Or automated.

Please see https://www.ietf.org/rfc/rfc6561.txt

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Sunday, September 25, 2016, John Kristoff  wrote:

> On Sun, 25 Sep 2016 14:36:18 +
> Ca By > wrote:
>
> > As long as their is one spoof capable network on the net, the problem
> will
> > not be solved.
>
> This is not strictly true.  If it could be determined where a large
> bulk of the spoofing came from, public pressure could be applied.  This
> may not have been the issue in this case, but in many amplification and
> reflection attacks, the originating spoof-enabled networks were from a
> limited set of networks.  De-peering, service termination, shaming, etc
> could have an effect.
>
> John
>

Ok, sorry for the not being exact. I am trying to be practical.

My point is, a lot of access networks will respond to public pressure if
the data is exposed on the offending real ips of the iot crap, and they
will enforce their AUP.

We have seen comcast do just that, on this list a few months back. That
path has legs.

Google also blocks service to certain hacked networks as well, we have seen
that on this list too. That is an interesting angle in the krebs case. Will
google block service to folks sharing ip with the iot  ddos mess ?

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John Levine]

>> Yeh, bcp38 is not a viable solution.

Krebs said this DDoS came from insecure IoT devices, of which there
are a kazillion, with the numbers growing every day.  Why would they
need to spoof IPs?  How would BCP38 help?

R's,
John

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John Kristoff]

On Sun, 25 Sep 2016 14:36:18 +
Ca By  wrote:

> As long as their is one spoof capable network on the net, the problem will
> not be solved.

This is not strictly true.  If it could be determined where a large
bulk of the spoofing came from, public pressure could be applied.  This
may not have been the issue in this case, but in many amplification and
reflection attacks, the originating spoof-enabled networks were from a
limited set of networks.  De-peering, service termination, shaming, etc
could have an effect.

John

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mike Hammett]

You don't need complete adoption to reduce the attacks. If ASes representing 
25% of the current spoofed traffic implemented BCP38, then guess what, there's 
25% less of an attack. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Ca By" <cb.li...@gmail.com> 
To: "Jay R. Ashworth" <j...@baylink.com> 
Cc: "North American Network Operators' Group" <nanog@nanog.org> 
Sent: Sunday, September 25, 2016 10:13:24 AM 
Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey 

On Sunday, September 25, 2016, Jay R. Ashworth <j...@baylink.com> wrote: 

> - Original Message - 
> > From: "Ca By" <cb.li...@gmail.com <javascript:;>> 
> 
> > On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org 
> <javascript:;>> 
> > wrote: 
> > 
> >> And of course Brian Krebs has a thing or two to say, not the least is 
> which 
> >> to push for BCP38 (good luck with that, right?). 
> >> 
> >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ 
> > 
> > Yeh, bcp38 is not a viable solution. 
> > 
> > As long as their is one spoof capable network on the net, the problem 
> will 
> > not be solved. While bcp38 is a true bcp, it is not a solution. It will 
> > not, and has not, moved the needle. 
> 
> No; things which are not implemented anywhere generally don't move the 
> needle. 
> 
> 
It is implemented many places in fact. 


> You're confusing cause and effect here, I think. 
> 
> 
I will argue you are confused. 


> You give no evidence that *pervasive implementation of 38* would *not* move 
> the needle, and that's where we are right now: we do not have anything that 
> looks like "pervasive implementation". 
> 
> *Ten* people could solve this problem. Tomorrow. 
> 
> The chief engineers of the top 10 US eyeball providers could simply sit 
> down 
> and say "let's go do this thing". And better than 80% of the potential 
> sources 
> would just vanish off the face of the internet. 
> 
> 
Assume every network in the usa implements bcp38. 

This simply means no spoofs source from usa. Every packet is sent from the 
usa using a valid origin. 

Assume also 50% of networks in Europe and Asia and the Southern Hemisphere 
do bcp38 too. 

Great. 

The result is the needle has not moved at all. 

CC nodes in the non bcp38 locations will send spoofed packets destinations 
is comcast and att with a source of krebs. 

Result? Comcast and att cpe responds with crap to krebs. Ddos success 
despite bcp38 in all of usa. 





> Do I need to go do research, and name these 10 people? :-) 
> 
> Cheers, 
> -- jra 
> -- 
> Jay R. Ashworth Baylink 
> j...@baylink.com <javascript:;> 
> Designer The Things I Think RFC 
> 2100 
> Ashworth & Associates http://www.bcp38.info 2000 Land 
> Rover DII 
> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 
> 1274 
> 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Sunday, September 25, 2016, Jay R. Ashworth  wrote:

> - Original Message -
> > From: "Ca By" >
>
> > On Sunday, September 25, 2016, Jay Farrell via NANOG  >
> > wrote:
> >
> >> And of course Brian Krebs has a thing or two to say, not the least is
> which
> >> to push for BCP38 (good luck with that, right?).
> >>
> >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/
> >
> > Yeh, bcp38 is not a viable solution.
> >
> > As long as their is one spoof capable network on the net, the problem
> will
> > not be solved. While bcp38 is a true bcp, it is not a solution. It will
> > not, and has not, moved the needle.
>
> No; things which are not implemented anywhere generally don't move the
> needle.
>
>
It is implemented many places in fact.


> You're confusing cause and effect here, I think.
>
>
I will argue you are confused.


> You give no evidence that *pervasive implementation of 38* would *not* move
> the needle, and that's where we are right now: we do not have anything that
> looks like "pervasive implementation".
>
> *Ten* people could solve this problem.  Tomorrow.
>
> The chief engineers of the top 10 US eyeball providers could simply sit
> down
> and say "let's go do this thing".  And better than 80% of the potential
> sources
> would just vanish off the face of the internet.
>
>
Assume every network in the usa implements bcp38.

This simply means no spoofs source from usa. Every packet is sent from the
usa using a valid origin.

Assume also 50% of networks in Europe and Asia and the Southern Hemisphere
do bcp38 too.

Great.

The result is the needle has not moved at all.

CC nodes in the non bcp38 locations will send spoofed packets destinations
is comcast and att with a source of krebs.

Result?  Comcast and att cpe responds with crap to krebs. Ddos success
despite bcp38 in all of usa.





> Do I need to go do research, and name these 10 people?  :-)
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com 
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land
> Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jay R. Ashworth]

- Original Message -
> From: "Ca By" 

> On Sunday, September 25, 2016, Jay Farrell via NANOG 
> wrote:
> 
>> And of course Brian Krebs has a thing or two to say, not the least is which
>> to push for BCP38 (good luck with that, right?).
>>
>> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/
> 
> Yeh, bcp38 is not a viable solution.
> 
> As long as their is one spoof capable network on the net, the problem will
> not be solved. While bcp38 is a true bcp, it is not a solution. It will
> not, and has not, moved the needle.

No; things which are not implemented anywhere generally don't move the needle.

You're confusing cause and effect here, I think.

You give no evidence that *pervasive implementation of 38* would *not* move
the needle, and that's where we are right now: we do not have anything that 
looks like "pervasive implementation".

*Ten* people could solve this problem.  Tomorrow.

The chief engineers of the top 10 US eyeball providers could simply sit down
and say "let's go do this thing".  And better than 80% of the potential sources
would just vanish off the face of the internet.

Do I need to go do research, and name these 10 people?  :-)

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Mike Hammett]

I've heard people say doing BCP38 is hard for big networks and it is if you do 
it at your provider\peering edges. It's easier if done at the customer edge. 
Simply don't allow the traffic onto your network to start with. 

Limit the spoofing attacks to just a single random ASN. How much smaller is the 
attack than it is now with hundreds or thousands of them? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Ca By" <cb.li...@gmail.com> 
To: "Jay Farrell" <jay...@jayfar.com> 
Cc: "North American Network Operators' Group" <nanog@nanog.org> 
Sent: Sunday, September 25, 2016 9:36:18 AM 
Subject: Re: Krebs on Security booted off Akamai network after DDoS attack 
proves pricey 

On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org> 
wrote: 

> And of course Brian Krebs has a thing or two to say, not the least is which 
> to push for BCP38 (good luck with that, right?). 
> 
> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ 
> 
> 

Yeh, bcp38 is not a viable solution. 

As long as their is one spoof capable network on the net, the problem will 
not be solved. While bcp38 is a true bcp, it is not a solution. It will 
not, and has not, moved the needle. 

A solution is aggregating the telemetry of source IP addresses in the 
botnet and assigning blame and liability to the owners of the IP addresses 
/ host ASN. 

The networks can then use AUP to shutdown the bot members. 

As where http://openntpproject.org/ was a proactive approach, Kreb's data 
can be reactive approach. And since the data is evidence of a crime, the 
network operators can enforce the AUP. The attack did happen. This ip was 
involved. Remediation is required. 




>From there, the host ASN can 

> On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth <j...@baylink.com 
> <javascript:;>> wrote: 
> 
> > - Original Message - 
> > > From: "Jay Farrell via NANOG" <nanog@nanog.org <javascript:;>> 
> > 
> > > And of course on windows ipconfig /flushdns 
> > > 
> > > Still I had to wait for my corporate caching servers to update; I think 
> > the 
> > > TTL on the old A record was an hour. 
> > 
> > Are big eyeball networks still flooring A record TTLs on resolution? 
> > 
> > Cheers, 
> > -- jra 
> > -- 
> > Jay R. Ashworth Baylink 
> > j...@baylink.com <javascript:;> 
> > Designer The Things I Think RFC 
> > 2100 
> > Ashworth & Associates http://www.bcp38.info 2000 Land 
> > Rover DII 
> > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 
> > 1274 
> > 
> 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Sunday, September 25, 2016, Jay Farrell via NANOG 
wrote:

> And of course Brian Krebs has a thing or two to say, not the least is which
> to push for BCP38 (good luck with that, right?).
>
> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/
>
>

Yeh, bcp38 is not a viable solution.

As long as their is one spoof capable network on the net, the problem will
not be solved. While bcp38 is a true bcp, it is not a solution. It will
not, and has not, moved the needle.

A solution is aggregating the telemetry of source IP addresses in the
botnet and assigning blame and liability to the owners of the IP addresses
/ host ASN.

The networks can then use AUP to shutdown the bot members.

As where http://openntpproject.org/ was a proactive approach, Kreb's data
can be reactive approach. And since the data is evidence of a crime, the
network operators can enforce the AUP. The attack did happen. This ip was
involved. Remediation is required.




>From there, the host ASN can

> On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth  > wrote:
>
> > - Original Message -
> > > From: "Jay Farrell via NANOG" >
> >
> > > And of course on windows ipconfig /flushdns
> > >
> > > Still I had to wait for my corporate caching servers to update; I think
> > the
> > > TTL on the old A record was an hour.
> >
> > Are big eyeball networks still flooring A record TTLs on resolution?
> >
> > Cheers,
> > -- jra
> > --
> > Jay R. Ashworth  Baylink
> > j...@baylink.com 
> > Designer The Things I Think   RFC
> > 2100
> > Ashworth & Associates   http://www.bcp38.info  2000 Land
> > Rover DII
> > St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> > 1274
> >
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jay R. Ashworth]

- Original Message -
> From: "Jay Farrell via NANOG" 

> And of course Brian Krebs has a thing or two to say, not the least is which
> to push for BCP38 (good luck with that, right?).
> 
> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Well, given how few contributions we've gotten at bcp38.info in the last,
what, 4 years, yeah, I guess so...

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jay Farrell via NANOG]

And of course Brian Krebs has a thing or two to say, not the least is which
to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth  wrote:

> - Original Message -
> > From: "Jay Farrell via NANOG" 
>
> > And of course on windows ipconfig /flushdns
> >
> > Still I had to wait for my corporate caching servers to update; I think
> the
> > TTL on the old A record was an hour.
>
> Are big eyeball networks still flooring A record TTLs on resolution?
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land
> Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Ca By]

On Saturday, September 24, 2016, Justin Paine via NANOG 
wrote:

>
> DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157
> IN A 130.211.45.45
>
> On Google now.
>
>
Next question.

Will google use the information from the telemetry, rumored to be webcams,
to defang the bot ?  Like post an informative message that the source
network is hosting a bad actor (same nat ipv4, /25, ... ). Or , work with
the access networks (Comcast, rcs/rds, china telecom) to disconnect and
manage the abusers ?


>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jay Farrell via NANOG]

And of course on windows ipconfig /flushdns

Still I had to wait for my corporate caching servers to update; I think the
TTL on the old A record was an hour.

On Sat, Sep 24, 2016 at 9:51 PM, Jared Mauch  wrote:

>
> > On Sep 24, 2016, at 9:28 PM, Justin Paine via NANOG 
> wrote:
> >
> >
> > DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com
> 157 IN A 130.211.45.45
>
>
> I recommend running this command (or similar):
>
> rndc flushname krebsonsecurity.com
>
> if you still see 127.0.0.1
>
> - Jared

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jared Mauch]

> On Sep 24, 2016, at 9:28 PM, Justin Paine via NANOG  wrote:
> 
> 
> DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN 
> A 130.211.45.45


I recommend running this command (or similar):

rndc flushname krebsonsecurity.com

if you still see 127.0.0.1

- Jared

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Justin Paine via NANOG]

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 
130.211.45.45

On Google now. 

 
Justin Paine 
Head of Trust & Safety 
CloudFlare Inc. 
PGP: BBAA 6BCE 3305 7FD6 6452 711557B6 0114 DE0B 314D




On Sat, Sep 24, 2016 at 2:17 PM -0700, "Brett Watson"  
wrote:











>> 
> that's not the one I was thinking of, this is:
>  
> 
> which references your presentation, nice! and is about J-root, not K-root,
> but mentions Lorenzo's work on K-root studies... In anycase, both seem to
> say that 'tcp anycast works fine' (inside some set of parameters).
> 

Right… and we’ve known this since about… ? 1996?

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Brett Watson]

>> 
> that's not the one I was thinking of, this is:
>  
> 
> which references your presentation, nice! and is about J-root, not K-root,
> but mentions Lorenzo's work on K-root studies... In anycase, both seem to
> say that 'tcp anycast works fine' (inside some set of parameters).
> 

Right… and we’ve known this since about… ? 1996?

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Christopher Morrow]

On Sat, Sep 24, 2016 at 2:43 PM, Niels Bakker 

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Niels Bakker]

* morrowc.li...@gmail.com (Christopher Morrow) [Sat 24 Sep 2016, 18:55 CEST]:
boy, it'd sure be nice if there were some 'science' and 
'measurement' behind such statements.

Didn't k-root do some anycast studies ~8-10 years back?


Not k-root but CacheFly 2006: 
https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf



-- Niels.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Christopher Morrow]

On Sat, Sep 24, 2016 at 12:28 PM, Bill Woodcock  wrote:

>
> > On Sep 24, 2016, at 7:47 AM, John Levine  wrote:
> >
> >>> Well...by anycast, I meant BGP anycast, spreading the "target"
> >>> geographically to a dozen or more well connected/peered origins.  At
> that
> >>> point, your ~600G DDoS might only be around
> >>
> >> anycast and tcp? the heck you say! :)
> >
> > People who've tried it say it works fine.
>
> It’s worked fine for 28 years, for me.
>
>
>

boy, it'd sure be nice if there were some 'science' and 'measurement'
behind such statements.
Didn't k-root do some anycast studies ~8-10 years back?

-chris
(note I'm totally a believer in anycast for tcp in the 'right'
circumstances, but often it feels like talking to climate-change-deniers
when proffering it as a solution)

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Bill Woodcock]

> On Sep 24, 2016, at 7:47 AM, John Levine  wrote:
> 
>>> Well...by anycast, I meant BGP anycast, spreading the "target"
>>> geographically to a dozen or more well connected/peered origins.  At that
>>> point, your ~600G DDoS might only be around
>> 
>> anycast and tcp? the heck you say! :)
> 
> People who've tried it say it works fine.

It’s worked fine for 28 years, for me.

-Bill






signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[John Levine]

>> Well...by anycast, I meant BGP anycast, spreading the "target"
>> geographically to a dozen or more well connected/peered origins.  At that
>> point, your ~600G DDoS might only be around
>
>anycast and tcp? the heck you say! :)

People who've tried it say it works fine.  Routes don't flap that often.

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Christopher Morrow]

On Fri, Sep 23, 2016 at 10:13 PM, Jon Lewis  wrote:

> On Fri, 23 Sep 2016, Christopher Morrow wrote:
>
> On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis  wrote:
>>
>> On Fri, 23 Sep 2016, Patrick W. Gilmore wrote:
>>>
>>> Is CloudFlare able to filter Layer 7 these days? I was under the
>>>
 impression CloudFlare was not able to do that.

 There have been a lot of rumors about this attack. Some say reflection,
 others say Layer 7, others say .. other stuff. If it is Layer 7, how are
 you going to ÿÿstep in front of the cannonÿÿ? Would you just pass
 through
 all the traffic?


>>> Anycast + load balancers + high powered varnish?
>>>
>>>
>>> notionally (because I have been paying zero attention to this) jon's
>> suggesting:
>>  1) setup a crapload of nginx/squid/etc configured tightly for things to
>> be accessed behind them
>>  2) ecmp to them across several layers (assume 32 ecmp at each layer, call
>> it 4 layers get craploads of machines running)
>>  3) change over the dns
>>  4) profit--
>>
>> eh? If you can eat the PPS, you can spray across enough tcp listeners, you
>> can weed out the chaff and start filtering in the 'application'... perhaps
>> also run a 'low bandwidth' version of the target site...
>>
>> hey look, we invented prolexic.
>>
>
> Well...by anycast, I meant BGP anycast, spreading the "target"
> geographically to a dozen or more well connected/peered origins.  At that
> point, your ~600G DDoS might only be around


anycast and tcp? the heck you say! :)


> 50G per site, and at that level, filtering the obvious crap gets much more
> reasonable.  Then, doing the layer 7 scrubbing of the less obvious crap is
> more easily dealt with than a single site receiving 600G of attack traffic.
>
>
sure, yes.


> I haven't actually done this (specifically for DDoS mitigation)...just
> speculating as to how it might easily be done given sufficient resources.
> The trouble is, the attackers have virtually unlimited bandwidth, and
> aren't constrained by having to pay for the bandwidth.
>
>
sounds like you got it all sorted out...


>
> --
>  Jon Lewis, MCP :)   |  I route
>  |  therefore you are
> _ http://www.lewis.org/~jlewis/pgp for PGP public key_
>

Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

[Jon Lewis]

On Fri, 23 Sep 2016, Christopher Morrow wrote:


On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis  wrote:


On Fri, 23 Sep 2016, Patrick W. Gilmore wrote:

Is CloudFlare able to filter Layer 7 these days? I was under the

impression CloudFlare was not able to do that.

There have been a lot of rumors about this attack. Some say reflection,
others say Layer 7, others say .. other stuff. If it is Layer 7, how are
you going to ÿÿstep in front of the cannonÿÿ? Would you just pass through
all the traffic?



Anycast + load balancers + high powered varnish?



notionally (because I have been paying zero attention to this) jon's
suggesting:
 1) setup a crapload of nginx/squid/etc configured tightly for things to
be accessed behind them
 2) ecmp to them across several layers (assume 32 ecmp at each layer, call
it 4 layers get craploads of machines running)
 3) change over the dns
 4) profit--

eh? If you can eat the PPS, you can spray across enough tcp listeners, you
can weed out the chaff and start filtering in the 'application'... perhaps
also run a 'low bandwidth' version of the target site...

hey look, we invented prolexic.


Well...by anycast, I meant BGP anycast, spreading the "target" 
geographically to a dozen or more well connected/peered origins.  At that 
point, your ~600G DDoS might only be around 50G per site, and at that 
level, filtering the obvious crap gets much more reasonable.  Then, doing 
the layer 7 scrubbing of the less obvious crap is more easily dealt with 
than a single site receiving 600G of attack traffic.


I haven't actually done this (specifically for DDoS mitigation)...just 
speculating as to how it might easily be done given sufficient resources. 
The trouble is, the attackers have virtually unlimited bandwidth, and 
aren't constrained by having to pay for the bandwidth.


--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_