Re: Open Resolver Problems

[na...@mitteilung.com]

Am 27.03.2013 00:04, schrieb Alain Hebert:


 We're on it here...

 Been using the work of http://bindguard.activezone.de/ to watch it =D

 There is a lot of targets... kinda hard to figure out the goal...

-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443




A new export of my blocking list is now available. Over 1500 bad hosts 
are now in my bogon configuration and every day it be more and more ...

See here: http://bindguard.activezone.de/bogon-list.txt

In the last 10 weeks are 749 new blocked hosts  [ result of a small 
environment ] :-(


Mar 28 07:40:56 BindGuard: Stats (PID 34874): 761 entries, 525851 
updates, 885070 ignored, 749 blocked, 1486207 events parsed.
Mar 28 07:40:56 BindGuard: Used Memory: 0.24 MByte (Index: 131072 Bytes, 
Free: 15624 / Entries: 125670 Bytes)
Mar 28 07:40:56 BindGuard: Version 0.69, Uptime: 10 weeks, 6 days, 23 
hours, 12 minutes, 38 seconds


Regards,
Markus

alexandria cable cutters?

[Randy Bush]

nyt reports capture of scuba divers attempting to cut telecom egypt
undersea fiber.


http://www.nytimes.com/aponline/2013/03/27/world/middleeast/ap-ml-egypt-internet.html

randy

Re: BCP38 - Internet Death Penalty

[Dobbins, Roland]

On Mar 28, 2013, at 11:42 AM, Paul Ferguson wrote:

> Actually, I do know someone who is in the "digital insurance" (for lack of a 
> better term) business, and although I just met them a few weeks ago, somehow 
> I get the feeling  that it is a growth industry.

I think this concept applies to traditional insurers, as well as newer 
tech-focused insurers, as they're all potentially on the hook for payouts due 
to business-disrupting events of all sorts - including DDoS-induced losses.

Perhaps your friend can provide some insight and/or pointers on this general 
topic?  Ideally, it would be a Good Thing to engage with the various actuarial 
organizations in order to brief them at properly-calibrated level of detail in 
order to educate them on this topic, followed by engagement with specific 
insurers.

---
Roland Dobbins  // 

  Luck is the residue of opportunity and design.

   -- John Milton

Re: BCP38 - Internet Death Penalty

[Paul Ferguson]

On Wed, Mar 27, 2013 at 9:18 PM, Dobbins, Roland  wrote:

>
> On Mar 28, 2013, at 6:01 AM, Mark Andrews wrote:
>
>> Secondly you reduce your legal liability.
>
> IANAL, but this has yet to be proven, AFAIK.
>
> One approach that hasn't been tried, to my knowledge, is educating the 
> insurance companies about how they can potentially reduce *their* liability 
> for payouts by requiring that real, actionable security BCPs such as 
> BCP38/84, running closed resolvers, implementing iACLs, et. al. are 
> implemented by those they insure.
>
> Does anyone have insight into examples of how insurance policies have been 
> paid out as a result of losses stemming from availability-related security 
> events?
>
> Another approach is educating the 'risk management' and 'business continuity' 
> communities about the risks and how to mitigate them, and how doing so 
> enhances business continuity.
>

Funny you should mention it.

Actually, I do know someone who is in the "digital insurance" (for
lack of a better term) business, and although I just met them a few
weeks ago, somehow I get the feeling  that it is a growth industry.
I'm semi --> :-)

- ferg


-- 
"Fergie", a.k.a. Paul Ferguson
 fergdawgster(at)gmail.com

Re: BCP38 - Internet Death Penalty

[Dobbins, Roland]

On Mar 28, 2013, at 6:01 AM, Mark Andrews wrote:

> Secondly you reduce your legal liability.

IANAL, but this has yet to be proven, AFAIK.

One approach that hasn't been tried, to my knowledge, is educating the 
insurance companies about how they can potentially reduce *their* liability for 
payouts by requiring that real, actionable security BCPs such as BCP38/84, 
running closed resolvers, implementing iACLs, et. al. are implemented by those 
they insure.

Does anyone have insight into examples of how insurance policies have been paid 
out as a result of losses stemming from availability-related security events?

Another approach is educating the 'risk management' and 'business continuity' 
communities about the risks and how to mitigate them, and how doing so enhances 
business continuity.

---
Roland Dobbins  // 

  Luck is the residue of opportunity and design.

   -- John Milton

Re: Open Resolver Problems

[Valdis . Kletnieks]

On Wed, 27 Mar 2013 16:59:16 -0500, Jack Bates said:
> On 3/27/2013 4:49 PM, Tony Finch wrote:
> > Jack Bates  wrote:
> >
> >> 3) BCP38 (in spirit)
> > That should be deployed as well as RRL.
> >
> > Tony.
>
> If BCP38 was properly deployed, what would be the purpose of RRL outside
> of misbehaving clients or direct attacks against that one server?

It would be quite sufficient just for the misbehaving clients aspect.



pgpcmH74KAbr4.pgp
Description: PGP signature

Re: Line cut in Mediterranean?

[Steven Bellovin]

The BBC has a similar story: 
http://www.bbc.co.uk/news/world-middle-east-21963100

On Mar 27, 2013, at 6:41 PM, Neil J. McRae  wrote:

> Via renesys
> 
> http://www.washingtonpost.com/world/middle_east/egypt-naval-forces-capture-3-scuba-divers-trying-to-sabotage-undersea-internet-cable/2013/03/27/dd2975ec-9725-11e2-a976-7eb906f9ed9b_story.html
> 
> 
> Sent from my iPhone
> 
> On 27 Mar 2013, at 21:53, "Neil J. McRae" 
> mailto:n...@domino.org>> wrote:
> 
> quite a few EU to India cables are impacted right now 4/7 down.
> 
> Sent from my iPad
> 
> On 27 Mar 2013, at 18:14, "Aftab Siddiqui" 
> mailto:aftab.siddi...@gmail.com>> wrote:
> 
> Well, it's not just SMW4 outage, we've been witnessing serious issues on
> IMEWE for couple of weeks now and this outages just made it worse.
> So, right now most of the traffic taking east bound routes.
> Who needs DDoS at this stage, these links are already chocked up :)
> 
> Maybe it was because of this: Global Internet Slows after 'biggest attack
> in history'
> http://www.bbc.co.uk/news/technology-21954636
> 
> 
> --
> Regards,
> 
> Aftab A. Siddiqui
> 
> 
> 
> 
> 


--Steve Bellovin, https://www.cs.columbia.edu/~smb

Re: BCP38 - Internet Death Penalty

[Arturo Servin]

I am afraid you are right.

It is going to cost us money and time, but unfortunately I do not see
another way out.

/as

On 3/27/13 6:19 PM, Paul Ferguson wrote:
> As I mentioned on another list earlier today, let's face it -- this is
> going to require a large-scale, very public, and probably multi-year
> education & awareness effort (as if 13+ years isn't enough already!).

Re: BCP38 - Internet Death Penalty

[Jimmy Hess]

On 3/26/13, Dobbins, Roland  wrote:
> On Mar 26, 2013, at 9:51 PM, Jay Ashworth wrote:

Perhaps you should reframe your strategy as "security problem",  and
show how providers have implemented BCP38,  how it is such a common
practice,  that not implementing BCP38  may fall short of the minimum
standard of due care  required to avoid liability,
in case your network is abused to launch an attack..  Incurs
possible legal risks that should be reviewd by lawyers,  due to
possible liability in facilitating a DoS attack.

That may be better at persuading your CEOs of large SPs than "It's
just good engineering";  it's not that following BCP38 is just
excellent practice. It's also that ignoring BCP38 in some
circumstances might be extremely poor, even negligent practice.


Possibly
Develop an industry certification/accreditation based on network
engineering practices,  and make it potentially so that service
providers want to carry it.  Then their marketing people can display
their "See our network is more secure and reliable"  logo on the
website,  and pressure other networks to seek 3rd party qualification;
 include  BCP38 as one of  several criteria,   "designed to help
reduce the degree of malicious activity, unmitigated DoS incidents,
instability,  or poor/inconsistent user experience".

If enough networks carry some sort of mark of quality, then maybe it
becomes meaningful as a tool persuasion:  there may be a smaller
quantity of demand for the purchase of services from networks that
don't carry it, unless they compensate by lowering their price.


While you're at it, include as recommended practices,
and provide multiple levels of  "Verified good network neighbor"  status:

  o 3rd party audited practices with regards to responsiveness and
cooperation by contacts to address  abuse and connectivity issues.

  o Requirement the network have a policy of assisting with the
mitigation of attack traversing any peers or customers,   through
required extensive network information sharing.

  o Truthful representation of service in all marketing materials.

  o  No "banned" internet protocols or ports, (e.g. "Our network
doesn't allow SSH protocol");  no NAT'ing by the SP.

  o A no-spamming policy,  a  no-repeated-failed-login policy, a
no port scanning policy, a no DoS policy that includes requirement the
SP investigate spam or other complaints
and take sufficient actions to disable offending hosts or
networks, or ensure
further spam is blocked..

  o Appropriate filtering  of incoming bgp announcements.
  o Accurate WHOIS information, listing the actual contact, no 3rd
party or intermediary
 for number resources, domains, etc.
  o Easily accessible and responsive technical and abuse contacts
for all services.


  o Not subverting or altering DNS query responses, or other
packets, as they
 cross the network;  for example,  not offering name lookup servers that
 claim to provide DNS service,  but covertly rewrite or capture
 NXDOMAIN or other responses,  sending an altered response instead.


>> Do the engineering heads at the top 10 tier-1/2 carriers carry enough
>> water to make that sale to the CEOs?

> Unfortunately, no - else it would've come to pass quite some time ago.
-- 
-JH

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Avi Freedman]

An important question...

I recall a peering panel at an ISPCON in 1996 when the current 
Peering Badguys, BBN, were represented by John, who listened
to a ton of bitching for an hour about the unfairness of it all and
said (paraphrasing)...

"I understand you all have your opinions and desires but I just want
 to point out one thing.  It is now 1996, 2 years after the widespread
 adoption of the web, and in every city in the US there are at least
 two ISPs happily providing unlimited {dialup} access for under $20/mo.
 What do you think we'd have if it were run or regulated by the government?"

Luckily, many bureaucrats and politicians in our government do
understand that.  And so far The Community has been able to put
pressure on international bodies and other governments don't 
have the clout.  Hopefully that remains the case for some time.

Avi

> In general, governments have avoided regulating various aspects of
> the Internet, in part because of lack of understanding and in part
> because the community keeps telling them that trying to regulate
> won't work because of its decentralized nature.  As the Internet
> becomes increasingly important to each country's economy and its
> citizens, the status quo is not likely to continue. 
> 
> The real question is, when governments do decide to try and help
> "improve the Internet", who will they be listening to, and will
> the operator community have spoken with a clear enough voice in
> these matters on what actually would make for an improvement?
> 
> FYI,
> /John

Re: BCP38 - Internet Death Penalty

[Mark Andrews]

In message 
, Jason Ackley writes:
> 
> On Wed, Mar 27, 2013 at 4:19 PM, Paul Ferguson wrote:
> 
> 
> > Some people are going to have to step and add a few thousand more
> > frequent flier miles and get out to various geographic constituencies,
> > at various events, and start talking about this. And we need a lot
> > more people on board. Nation & international campaigns, etc.
> >
> >
> Agree 100%.
> 
> One thing that I will mention is a subtle issue that needs more thought.  I
> think one of the challenges for this is answering the question of 'How does
> this make it better for my network on day one?'   . Well , it doesn't for
> the majority of impact that people may be seeing.

Firstly you protect your customers from your other customer machines
that are compromised.

Secondly you reduce your legal liability.

Third you can use it to improve the reputation of your network.

> For example - Let us say someone is not currently running a fully
> BCP38-compliant network (shame on them, blah blah). They can do the
> remaining work to fix this in XXX hours at YYY cost.
> 
> The issue for them may be that they are the *destination* of the attacks
> that leverage non-BCP38 compliant networks. So even after investing XXX
> hours and YYY dollars it doesn't 'cure' the majority of the problems for
> them related to spoofing. So any spend on this is not a 'fix' as much as it
> is a 'fix for others'. (which certainly still needs to be done , don't get
> me wrong!)
> 
> Spoofing remains a problem until *everyone*  gets it done or there is
> enough motivation to get it done without benefit to your own network.

It's a reducing problem as more networks filter.

> If Network_Zed is the last to go BCP38-complaint in 2023 , then the rest of
> the internet is still vulnerable to the nasty items that can take place
> from the Network_Zed network until that time.

Or the rest of the world can just shun Network_Zed.
 
> Accepting that I think we need to find ways to make it where it stays on
> the radar - as has been suggested via walls of shame, RIR pressure, etc.
> Perhaps 'spoofing fees' etc ?
> 
> I hate to have an approach of 'fix this or I will hit you with this stick!'
> - but this has got to stop..
> 
> OK, back to my hole watching all the presumably spoofed incoming traffic
> that happens to be on udp/53 and looking for ANY? isc.org :-)

Which you can chase back to offending sources and complain to them about.
 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Open Resolver Problems

[Joe Abley]

On 2013-03-27, at 17:59, Jack Bates  wrote:

> DNS is UDP for a reason.

Not a great reason, as it turns out. But hindsight is 20/20.

> The infrastructure to switch it to TCP is prohibitive and completely destroys 
> the anycast mechanisms.

No.


Joe

Re: Open Resolver Problems

[Tony Finch]

Jack Bates  wrote:
>
> If BCP38 was properly deployed, what would be the purpose of RRL outside of
> misbehaving clients or direct attacks against that one server?

If fictional scenario, irrelevant answer. Given the current situation,
efforts to deploy both RRL and BCP38 in parallel will reduce the mess we
are in. Let's race to see who gets to full deployment first.

> The infrastructure to switch it to TCP is prohibitive and completely
> destroys the anycast mechanisms.

Yeah, anycast for HTTP doesn't work at all. Just ask CloudFlare.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[John Curran]

On Mar 27, 2013, at 6:25 PM, Rich Kulawiec  wrote:

> Or worse, before some government somewhere decides to "solve" this
> problem for a value of "solved" involving (shudder) legislation.

In general, governments have avoided regulating various aspects of 
the Internet, in part because of lack of understanding and in part
because the community keeps telling them that trying to regulate
won't work because of its decentralized nature.  As the Internet
becomes increasingly important to each country's economy and its 
citizens, the status quo is not likely to continue.  

The real question is, when governments do decide to try and help
"improve the Internet", who will they be listening to, and will 
the operator community have spoken with a clear enough voice in 
these matters on what actually would make for an improvement?

FYI,
/John

Re: Line cut in Mediterranean?

[Neil J. McRae]

Via renesys

http://www.washingtonpost.com/world/middle_east/egypt-naval-forces-capture-3-scuba-divers-trying-to-sabotage-undersea-internet-cable/2013/03/27/dd2975ec-9725-11e2-a976-7eb906f9ed9b_story.html


Sent from my iPhone

On 27 Mar 2013, at 21:53, "Neil J. McRae" 
mailto:n...@domino.org>> wrote:

quite a few EU to India cables are impacted right now 4/7 down.

Sent from my iPad

On 27 Mar 2013, at 18:14, "Aftab Siddiqui" 
mailto:aftab.siddi...@gmail.com>> wrote:

Well, it's not just SMW4 outage, we've been witnessing serious issues on
IMEWE for couple of weeks now and this outages just made it worse.
So, right now most of the traffic taking east bound routes.
Who needs DDoS at this stage, these links are already chocked up :)

Maybe it was because of this: Global Internet Slows after 'biggest attack
in history'
http://www.bbc.co.uk/news/technology-21954636


--
Regards,

Aftab A. Siddiqui

Re: BCP38 - Internet Death Penalty

[Warren Bailey]

I think the media fire about this will enlighten many c level executives. After 
that, it's a matter of them saying "go do this". You can't get any traction if 
there isn't a perceived issue, from what I've seen anyways. I still think the 
ipv4 to 6 transition will require media outlets running special coverage on the 
end of the Internet because we broke it by not addressing issues. I've shouted 
from roof tops on various occasions, only to hear months later about how we 
should have seen something coming. Get CNN to run Nanog has a solution and 
watch the hordes gather. People slow down on the freeway to see an accident, 
they'll slow down long enough to see what's happened and drive off. When their 
house is on fire, it's a completely different story.


>From my Android phone on T-Mobile. The first nationwide 4G network.



 Original message 
From: John Curran 
Date: 03/27/2013 3:33 PM (GMT-08:00)
To: Mark Andrews 
Cc: NANOG 
Subject: Re: BCP38 - Internet Death Penalty


On Mar 27, 2013, at 4:54 PM, Mark Andrews 
 wrote:
>> Umm... How many North American ISP's/datacenters/web hosting firms were
>> aware of the BCP 38 development as it was on-going, and participated in
>> some manner in its review?  ...
>
> I'd say enough were aware. :-)
>
> 8. Acknowledgments
>
>   The North American Network Operators Group (NANOG) [5] group as a
>   whole deserves special credit for openly discussing these issues and
>   actively seeking possible solutions. Also, thanks to Justin Newton
>   [Priori Networks] and Steve Bielagus [IronBridge Networks].  for
>   their comments and contributions.

Mark -

  That's plenty of consideration for voluntary efforts (which is what
  we've tried to date in various forums with rather limited success...)

  Whether that's sufficient notice and consideration on which to base
  mandatory requirements from a public policy perspective is not clear.

  Frankly, I would suggest that NANOG document a best common operating
  practice (BCOP) based on BCP38 (written at a somewhat higher level
  which describes what types of connections ingress filtering it applies
  to, e.g. consumer edge, business, transit, etc.; whether it should
  be just a customer default or an absolute requirement, etc.), and
  then holding an approval process to make the result a NANOG BCOP...
  

  If this were done in a fairly formal manner, the result would be closer
  to the prior example (National Fire Protection Association code) and
  would far more convincing both in aiding governments to pick up this
  cause in the region, as well as encouraging similar efforts elsewhere.

FYI,
/John

Verizon Wireless security contact needed

[nick hatch]

Hi all,

I just discovered a somewhat-exigent issue which affects
confidentiality for Verizon Wireless customers. (PSTN / Voice)

I'm failing at trying to find a Verizon Wireless security contact
through normal means. If someone can provide a contact off-list it
would be much appreciated.

Thanks,

-Nick

Re: BCP38 - Internet Death Penalty

[John Curran]

On Mar 27, 2013, at 4:54 PM, Mark Andrews 
 wrote:
>> Umm... How many North American ISP's/datacenters/web hosting firms were 
>> aware of the BCP 38 development as it was on-going, and participated in 
>> some manner in its review?  ...
> 
> I'd say enough were aware. :-)
> 
> 8. Acknowledgments
> 
>   The North American Network Operators Group (NANOG) [5] group as a
>   whole deserves special credit for openly discussing these issues and
>   actively seeking possible solutions. Also, thanks to Justin Newton
>   [Priori Networks] and Steve Bielagus [IronBridge Networks].  for
>   their comments and contributions.

Mark - 
 
  That's plenty of consideration for voluntary efforts (which is what
  we've tried to date in various forums with rather limited success...)

  Whether that's sufficient notice and consideration on which to base
  mandatory requirements from a public policy perspective is not clear.
  
  Frankly, I would suggest that NANOG document a best common operating 
  practice (BCOP) based on BCP38 (written at a somewhat higher level 
  which describes what types of connections ingress filtering it applies
  to, e.g. consumer edge, business, transit, etc.; whether it should 
  be just a customer default or an absolute requirement, etc.), and 
  then holding an approval process to make the result a NANOG BCOP...
  

  If this were done in a fairly formal manner, the result would be closer
  to the prior example (National Fire Protection Association code) and 
  would far more convincing both in aiding governments to pick up this 
  cause in the region, as well as encouraging similar efforts elsewhere.

FYI,
/John

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Rich Kulawiec]

On Wed, Mar 27, 2013 at 12:30:43PM -0700, Paul Ferguson wrote:
> Consider this a call-to-arms, in all aspects. Please.

+1

No.  Not enough.  +10.

But...our collective track record in responding in a timely and effective
fashion to such calls is not very good.  Twenty years ago we could have
killed spam.  Ten years ago we could have killed botnets.  We didn't
do either (despite *numerous* warnings of how bad it would get --
warnings dismissed as unduly pessimistic at the time, now viewed as
naively optimistic) and in part because we didn't...now we have this.
There are entire business sectors which now exist just to make up for
our failure to do those things when we had the chance.  And while
there are good and smart people in those doing some good and smart
things, all those sectors are really doing are (a) costing us a ton
of money and (b) helping us tread water.

I suggest we fix these problems before we wind up creating yet another
market for yet another several billion dollars that could be better used
on making forward progress.

Or worse, before some government somewhere decides to "solve" this
problem for a value of "solved" involving (shudder) legislation.

--rsk

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Seth Mattinen]

On 3/27/13 2:46 PM, Warren Bailey wrote:

Wasn't there a ton of drama with the SpamHaus guys a year or so ago
regarding RBL's on NANOG?




There's always someone who publicly flips out over being listed by a 
major DNSBL at least once a year.


~Seth

Re: BCP38 needs advertising

[Alain Hebert]

Noted.

But today's contribution by Eric M. Caroll might end up on the front
page =D.

I got the domains...  Now I just need a few free hours to setup
something useful.

As always, don't be shy to drop me contribution offlist.   

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 03/27/13 14:31, Paul Ferguson wrote:
> But of course. :-)
>
> Also, just saw this:
>
> http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
>
> - ferg
>
> On Wed, Mar 27, 2013 at 11:02 AM, Arturo Servin  
> wrote:
>
>> And do not forget 
>>
>> http://tools.ietf.org/html/bcp38
>>
>> :)
>>
>> -as
>>
>>
>> On 3/27/13 2:17 PM, Paul Ferguson wrote:
>>> Please reference:
>>>
>>> http://openresolverproject.org/
>>> http://spoofer.csail.mit.edu/
>>> http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack
>>>
>>> ...and anything else to raise the awareness level.
>>>
>>> Thanks,
>>>
>>> - ferg (co-perpetrator of BCP38)  :-)
>>>
>>> On Wed, Mar 27, 2013 at 9:48 AM, Alain Hebert  wrote:
>>>
 bcp38.org coming soon =D

 -
 Alain Hebertaheb...@pubnix.net
 PubNIX Inc.
 50 boul. St-Charles
 P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
 Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

 On 03/27/13 11:20, Jack Bates wrote:
> Outside of needing more details and examples, BCP38 could use more
> advertising.
>
> The best option, if they would accept it, is to have all RIRs mention
> BCP38 as well as require that mention of BCP38 be included in all IP
> justification requests to customers (so that those who receive
> netblocks from their ISPs are also aware of it).
>
> For ARIN, at least, having it mentioned in the attestation process
> wouldn't be a bad idea. At least someone of management would be aware
> of it.
>
> The only issue is see concerning the RIRs is that they may object to
> it being out of scope to their duties. However, informing people of
> something is not requiring implementation of something. On the other
> hand, we know that there are a great number of networks that don't
> participate with the community at large and may have no idea about
> BCP38 and why it is important.
>
>
> Jack
>
>
>
>
>

>>>
>>>
>
>

Re: BCP38 - Internet Death Penalty

[Jason Ackley]

On Wed, Mar 27, 2013 at 4:19 PM, Paul Ferguson wrote:


> Some people are going to have to step and add a few thousand more
> frequent flier miles and get out to various geographic constituencies,
> at various events, and start talking about this. And we need a lot
> more people on board. Nation & international campaigns, etc.
>
>
Agree 100%.

One thing that I will mention is a subtle issue that needs more thought.  I
think one of the challenges for this is answering the question of 'How does
this make it better for my network on day one?'   . Well , it doesn't for
the majority of impact that people may be seeing.

For example - Let us say someone is not currently running a fully
BCP38-compliant network (shame on them, blah blah). They can do the
remaining work to fix this in XXX hours at YYY cost.

The issue for them may be that they are the *destination* of the attacks
that leverage non-BCP38 compliant networks. So even after investing XXX
hours and YYY dollars it doesn't 'cure' the majority of the problems for
them related to spoofing. So any spend on this is not a 'fix' as much as it
is a 'fix for others'. (which certainly still needs to be done , don't get
me wrong!)

Spoofing remains a problem until *everyone*  gets it done or there is
enough motivation to get it done without benefit to your own network.

If Network_Zed is the last to go BCP38-complaint in 2023 , then the rest of
the internet is still vulnerable to the nasty items that can take place
from the Network_Zed network until that time.

Accepting that I think we need to find ways to make it where it stays on
the radar - as has been suggested via walls of shame, RIR pressure, etc.
Perhaps 'spoofing fees' etc ?

I hate to have an approach of 'fix this or I will hit you with this stick!'
- but this has got to stop..

OK, back to my hole watching all the presumably spoofed incoming traffic
that happens to be on udp/53 and looking for ANY? isc.org :-)


--
jason

Re: BCP38 - Internet Death Penalty

[Valdis . Kletnieks]

On Wed, 27 Mar 2013 14:19:05 -0700, Paul Ferguson said:

> And there may even be some stick approaches to accompany the carrot,
> but some awareness is going to have to happen.
>
> Sing it from the mountain tops.

http://www.sans.org/dosstep/roadmap.php

Note the date.  Note the list of recommendations. Note the flat
spot on my forehead from continual pounding against vertical surfaces.

:)


pgptPkleId7ch.pgp
Description: PGP signature

Re: Open Resolver Problems

[Jack Bates]

On 3/27/2013 4:49 PM, Tony Finch wrote:

Jack Bates  wrote:


3) BCP38 (in spirit)

That should be deployed as well as RRL.

Tony.


If BCP38 was properly deployed, what would be the purpose of RRL outside 
of misbehaving clients or direct attacks against that one server?


We already know the fix for spoofing. Trying to tweak every service that 
spoofing effectively takes advantage of will not be a winning game. 
Sending legitimate clients to TCP is also a losing game. DNS is UDP for 
a reason. The infrastructure to switch it to TCP is prohibitive and 
completely destroys the anycast mechanisms.



Jack

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Neil J. McRae]

that article is absolute rubbish. take with large pinch of salt, rockstar in 
hamster outfit type nonsense. 

$dayjob didn't lose any traffic during the period, some guys where affected 
because of the lottery of being on the same switch as couldfare.

regards,
Neil.

On 27 Mar 2013, at 18:45, "Jay Ashworth"  wrote:

> http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
> 
> Yes: 120 gigabits/second, primarily of DNS amplification traffic.
> 
> Still think it's optional to implement BCP38 pervasively?
> 
> Cheers,
> -- jra
> -- 
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
> St Petersburg FL USA   #natog  +1 727 647 1274
> 
> 

Re: Open Resolver Problems

[Tony Finch]

Jack Bates  wrote:

> You'll also find that [DNS RRL] serves little purpose.

In my experience it works extremely well. Yes it is possible to work
around it, but you still need to stop the attacks that are happening now.
It is good to make the attacker's job harder.

> 1) tcp

RRL pushes legitimate clients to TCP if they get muddled up with attack
traffic.

> 2) require all requests to pad out to maximum response

I expect that is as easy to deploy as BCP38, IPv6, and DNSSEC.

> 3) BCP38 (in spirit)

That should be deployed as well as RRL.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

Re: Line cut in Mediterranean?

[Neil J. McRae]

quite a few EU to India cables are impacted right now 4/7 down.

Sent from my iPad 

On 27 Mar 2013, at 18:14, "Aftab Siddiqui"  wrote:

> Well, it's not just SMW4 outage, we've been witnessing serious issues on
> IMEWE for couple of weeks now and this outages just made it worse.
> So, right now most of the traffic taking east bound routes.
> Who needs DDoS at this stage, these links are already chocked up :)
> 
>> Maybe it was because of this: Global Internet Slows after 'biggest attack
>> in history'
>> http://www.bbc.co.uk/news/technology-21954636
> 
> 
> -- 
> Regards,
> 
> Aftab A. Siddiqui
> 

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Warren Bailey]

Wasn't there a ton of drama with the SpamHaus guys a year or so ago
regarding RBL's on NANOG?



On 3/27/13 2:54 PM, "Scott Weeks"  wrote:

>
>--- b...@herrin.us wrote:
>From: William Herrin 
>
>According to the New York Times it was 300 gbps and Cyberbunker was the
>bad guy.
>http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becom
>es-internet-snarling-attack.html?pagewanted=all&_r=0
>-
>
>Got a link that we don't have to allow cookies and have to create an
>account to read?
>
>
>
>I found it using startpage.com's proxy and pasted it below for
>others that don't want to create accounts and all:
>
>
>A squabble between a group fighting spam and a Dutch company that hosts
>Web sites said to be sending spam has escalated into one of the largest
>computer attacks on the Internet, causing widespread congestion and
>jamming crucial infrastructure around the world, John Markoff and Nicole
>Perlroth write on Wednesday in The New York Times.
>
>Millions of ordinary Internet users have experienced delays in services
>like Netflix or could not reach a particular Web site for a short time.
>However, for the Internet engineers who run the global network, the
>problem is more worrisome. The attacks are becoming increasingly
>powerful, and computer security experts worry that if they continue to
>escalate, people may not be able to reach basic Internet services, like
>e-mail and online banking.
>
>The dispute started when the spam-fighting group, called Spamhaus, added
>the Dutch company Cyberbunker to its blacklist, which is used by e-mail
>providers to weed out spam. Cyberbunker, named for its headquarters, a
>five-story former NATO bunker, offers hosting services to any Web site
>³except child porn and anything related to terrorism,² according to its
>Web site.
>
>A spokesman for Spamhaus, which is based in Europe, said the attacks
>began on March 19, but had not stopped the group from distributing its
>blacklist.
>
>Patrick Gilmore, chief architect at Akamai Networks, a digital content
>provider, said Spamhaus¹s role was to generate a list of Internet
>spammers. Of Cyberbunker, he added: ³These guys are just mad. To be
>frank, they got caught. They think they should be allowed to spam.²
>
>Mr. Gilmore said that the attacks, which are generated by swarms of
>computers called botnets, concentrate data streams that are larger than
>the Internet connections of entire countries. He likened the technique,
>which uses a long-known flaw in the Internet¹s basic plumbing, to using a
>machine gun to spray an entire crowd when the intent is to kill one
>person. The so-called distributed denial of service, or DDoS, attacks
>have reached previously unknown magnitudes, growing to a data stream of
>300 billion bits per second.
>
>Questioned about the attacks, Sven Olaf Kamphuis, an Internet activist
>who said he was a spokesman for the attackers, said in an online message
>that, ³We are aware that this is one of the largest DDoS attacks the
>world had publicly seen.² Mr. Kamphuis said Cyberbunker was retaliating
>against Spamhaus for ³abusing their influence.²
>
>
>scott

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Scott Weeks]

"...Sven Olaf Kamphuis, an Internet activist who 
said he was a spokesman for the attackers..."


I wonder is he'll ever post here again as he has in the past.  It
probably would not go well for him if he did...

scott

Re: Open Resolver Problems

[Tony Finch]

Jack Bates  wrote:
>
> Tracking the clients would be a huge dataset and be especially complicated in
> clusters.

The memory usage is guite manageable: for the BIND patch it is at most
40-80 bytes (for 32 or 64 bit machines) per request per second. You're
doing well if you need a megabyte. There's no need to get complicated with
clusters: it's enough of an improvement just to track rates per server.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

Re: Open Resolver Problems

[Tony Finch]

Joe Abley  wrote:
>
> My assessment is that the implementations I have seen are ready for
> production use, but I think it's understandable given the moving
> goalpoasts that some vendors have not yet promoted the code to be
> included in stable releases.

It is in the current stable release of NSD 3.2.15 though it is a
build-time option. It is in the current release candidate of knot DNS
1.2.0-rc4. It will be in BIND-9.10 which has not yet reached public beta.

Our servers have been abused as reflectors, and we're using the BIND RRL
patch with versions 9.8 and 9.9 to stop the attack traffic.

There are other interim options such as using firewall rate limiting
which is worse than RRL because it is much more likely to hurt legitimate
queries. For example,
http://www.bortzmeyer.org/rate-limiting-dns-open-resolver.html

Or you can use a configuration add-on such as bindguard.
http://bindguard.activezone.de

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Scott Weeks]

--- b...@herrin.us wrote:
From: William Herrin 

According to the New York Times it was 300 gbps and Cyberbunker was the bad guy.
http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0
-

Got a link that we don't have to allow cookies and have to create an account to 
read?



I found it using startpage.com's proxy and pasted it below for
others that don't want to create accounts and all:


A squabble between a group fighting spam and a Dutch company that hosts Web 
sites said to be sending spam has escalated into one of the largest computer 
attacks on the Internet, causing widespread congestion and jamming crucial 
infrastructure around the world, John Markoff and Nicole Perlroth write on 
Wednesday in The New York Times.

Millions of ordinary Internet users have experienced delays in services like 
Netflix or could not reach a particular Web site for a short time. However, for 
the Internet engineers who run the global network, the problem is more 
worrisome. The attacks are becoming increasingly powerful, and computer 
security experts worry that if they continue to escalate, people may not be 
able to reach basic Internet services, like e-mail and online banking.

The dispute started when the spam-fighting group, called Spamhaus, added the 
Dutch company Cyberbunker to its blacklist, which is used by e-mail providers 
to weed out spam. Cyberbunker, named for its headquarters, a five-story former 
NATO bunker, offers hosting services to any Web site “except child porn and 
anything related to terrorism,” according to its Web site.

A spokesman for Spamhaus, which is based in Europe, said the attacks began on 
March 19, but had not stopped the group from distributing its blacklist.

Patrick Gilmore, chief architect at Akamai Networks, a digital content 
provider, said Spamhaus’s role was to generate a list of Internet spammers. Of 
Cyberbunker, he added: “These guys are just mad. To be frank, they got caught. 
They think they should be allowed to spam.”

Mr. Gilmore said that the attacks, which are generated by swarms of computers 
called botnets, concentrate data streams that are larger than the Internet 
connections of entire countries. He likened the technique, which uses a 
long-known flaw in the Internet’s basic plumbing, to using a machine gun to 
spray an entire crowd when the intent is to kill one person. The so-called 
distributed denial of service, or DDoS, attacks have reached previously unknown 
magnitudes, growing to a data stream of 300 billion bits per second.

Questioned about the attacks, Sven Olaf Kamphuis, an Internet activist who said 
he was a spokesman for the attackers, said in an online message that, “We are 
aware that this is one of the largest DDoS attacks the world had publicly 
seen.” Mr. Kamphuis said Cyberbunker was retaliating against Spamhaus for 
“abusing their influence.” 


scott

Re: BCP38 - Internet Death Penalty

[Paul Ferguson]

On Wed, Mar 27, 2013 at 1:54 PM, Mark Andrews  wrote:
>
> In message <8da1853ce466b041b104c1caee00b3748fa4e...@chaxch01.corp.arin.net>, 
> John Curran writes:

>>
>> Umm... How many North American ISP's/datacenters/web hosting firms were
>> aware of the BCP 38 development as it was on-going, and participated in
>> some manner in its review?  The IETF is a wonderful organization, but it
>> is not exactly overflowing with representation from the service provider
>> community.  Also, given that you really need these practices picked up
>> on a global basis, repeat the above question with "Global" rather than
>> North American...
>
> I'd say enough were aware. :-)
>
> 8. Acknowledgments
>
>The North American Network Operators Group (NANOG) [5] group as a
>whole deserves special credit for openly discussing these issues and
>actively seeking possible solutions. Also, thanks to Justin Newton
>[Priori Networks] and Steve Bielagus [IronBridge Networks].  for
>their comments and contributions.
>

I think the core group of backbone engineers, and to some degree
others at the periphery of the DFZ, understand the issues.

And yes, I think it gets back to an education problem on "why you should care".

As I mentioned on another list earlier today, let's face it -- this is
going to require a large-scale, very public, and probably multi-year
education & awareness effort (as if 13+ years isn't enough already!).

How long did it take to get some movement on open SMTP relays? You get the idea.

Some people are going to have to step and add a few thousand more
frequent flier miles and get out to various geographic constituencies,
at various events, and start talking about this. And we need a lot
more people on board. Nation & international campaigns, etc.

And there may even be some stick approaches to accompany the carrot,
but some awareness is going to have to happen.

Sing it from the mountain tops.

- ferg

-- 
"Fergie", a.k.a. Paul Ferguson
 fergdawgster(at)gmail.com

Re: BCP38 - Internet Death Penalty

[Mark Andrews]

In message <8da1853ce466b041b104c1caee00b3748fa4e...@chaxch01.corp.arin.net>, 
John Curran writes:
> On Mar 27, 2013, at 10:23 AM, Jay Ashworth  wrote:
> 
> > Indeed, but I have an even better example of how that's already done, 
> that
> > is probably pertinent.
> > 
> 
> > The National Electric Code is assimilated law now, I think, in every
> > state in the US.  It is promulgated by the National Fire Protection 
> > Association, which is a standards organization originally started by 
> > insurors to reduce their exposure and expenses; by professional 
> consensus,
> > they have become, effectively, a lawmaking body.
> > 
> > So they're not the government, they're subject-matter experts, 
> technically
> > competent to have opinions, and their opinions are assimilated into 
> > controlling law.
> 
> Indeed... a perfect example of industry self-regulation supplemented by
> a mandatory legal framework referencing the best practice documents.
> 
> > Is BCP38 *not* well enough though out even for large and medium sized 
> > carriers to adopt as contractual language, ...
> 
> You're back to discussing voluntary mechanisms in the above, but 
> your example is a case where compliance is due to legislation at 
> both federal and state levels.
> 
> > much less for FCC or someone to
> > impose upon them?  If so, we should work on it further.
> 
> Umm... How many North American ISP's/datacenters/web hosting firms were 
> aware of the BCP 38 development as it was on-going, and participated in 
> some manner in its review?  The IETF is a wonderful organization, but it
> is not exactly overflowing with representation from the service provider 
> community.  Also, given that you really need these practices picked up
> on a global basis, repeat the above question with "Global" rather than
> North American...  

I'd say enough were aware. :-)

8. Acknowledgments

   The North American Network Operators Group (NANOG) [5] group as a
   whole deserves special credit for openly discussing these issues and
   actively seeking possible solutions. Also, thanks to Justin Newton
   [Priori Networks] and Steve Bielagus [IronBridge Networks].  for
   their comments and contributions.


> If you actually want to see certain technical practices made mandatory
> for Internet service providers globally, then you need a development 
> process for those practices which fairly robust to insure that the 
> practices are equally germane and reasonable to many different service 
> providers in many different parts of the world.  It's actually relatively
> easy to get governments to require compliance with accepted technical
> practices for the Internet, the problem has been we've never taken the
> effort to produce best practices with sufficient rigor than any given
> government can know that it has the necessary backing (of many of the
> service providers in its domain) needed to actually require compliance.
> 
> (With regard to the FCC, there is some question as to whether or not 
> their mandate would allow establishing required practices for ISPs...
> To the extent that VoIP traffic is being carried, this is far more 
> likely to be possible, and hence folks should be aware of various
> activities such as the CSRIC "Communications Security, Reliability 
> and Interoperability Council", which develops recommendations that
> could effectively become requirements on Internet Service Providers
> in some contexts. 
>  nteroperability-council-iii>
> Noe that the problem with this sort of approach is that having dozens 
> of countries all developing their own specific technical best practices 
> is most likely to cumulatively interact in ways impossible to comply 
> with...  Hence, the need for clear global technical best practices,
> through which countries with a particular desire to "improve the
> state of the Internet" can channel their legislative desires...)
> 
> FYI,
> /John

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Warren Bailey]

At least they compared it to a traffic jam. ;)


>From my Android phone on T-Mobile. The first nationwide 4G network.



 Original message 
From: Huasong Zhou 
Date: 03/27/2013 1:00 PM (GMT-08:00)
To: sur...@mauigateway.com,nanog@nanog.org
Subject: Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"


Try this one:
http://www.bbc.co.uk/news/technology-21954636

On 3/27/13 3:55 PM, "Scott Weeks"  wrote:

>
>
>--- b...@herrin.us wrote:
>From: William Herrin 
>
>According to the New York Times it was 300 gbps and Cyberbunker was the
>bad guy.
>http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becom
>es-internet-snarling-attack.html?pagewanted=all&_r=0
>-
>
>
>Got a link that we don't have to allow cookies and have to create an
>account to read?
>
>scott
>

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Huasong Zhou]

Try this one:
http://www.bbc.co.uk/news/technology-21954636

On 3/27/13 3:55 PM, "Scott Weeks"  wrote:

>
>
>--- b...@herrin.us wrote:
>From: William Herrin 
>
>According to the New York Times it was 300 gbps and Cyberbunker was the
>bad guy.
>http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becom
>es-internet-snarling-attack.html?pagewanted=all&_r=0
>-
>
>
>Got a link that we don't have to allow cookies and have to create an
>account to read?
>
>scott
>

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[William Herrin]

On Wed, Mar 27, 2013 at 3:55 PM, Scott Weeks  wrote:
> According to the New York Times it was 300 gbps and Cyberbunker was the bad 
> guy.
> http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0
> -
>
> Got a link that we don't have to allow cookies and have to create an account 
> to read?

>From the New York Times?

It didn't ask me for an account. YMMV. I suspect you will need cookies.

-Bill


-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Enforcing Source Integrity: BCP38 and Open Resolver Problems

[Eric M. Carroll]

The root cause of high scale directed amplification attacks is the failure
to assure the integrity of the source IP address. This failure leads to a
large set of directed amplification attack vectors.

BCP38 was written in 2000, coming up on its 13th anniversary. This root
cause, and various methodologies & technologies to resolve it, have been an
ongoing discussion since back to the 90s.

The failure to enforce this BCP or the related technological mechanisms to
force implementation is the root cause of why the Internet cannot always
trust source addresses and why these attack vectors persist. Until the ISP
community gets serious about forcing the integrity of source addresses
throughout its topology, various flavours of attack whose root cause is the
spoofed source addresses will continue.

Yes, it is not easy to do because it is a transitive trust issue, linked to
topology and address management policy. Yes it would be easier if there was
a magic bullet to validate source addresses built into the architecture.
But there is not, the architecture is what it is. If every step of the
chain enforced the integrity of source addresses, this risk would be
resolved. There are multiple different steps that could be taken, including
law enforcement, statute, contractual, policy, process and technological
mechanisms.

Every ISP and content providers' business model is threatened by this
vulnerability. Every attack drives up operational expenses for everyone.
Opportunity costs of missed sales and impacted business are everywhere. It
is a pure tragedy of the commons - for lack of enforcement, the whole
system is threatened in scale.

This problem cannot be allowed to rest at the edges simply by pointing at
the current amplification vector. Yesterday it was something different.
Tomorrow it will different again. The constant is the rising scale of the
Internet and resulting increase in scale of the attack and its
corresponding economic impact. The root cause is not today's Google issue.

The ISP community has the power to enforce this through policy and
technological means. Whether it has the will and ability to self-organize
and enforce is a different issue (and also, a long standing one).

The discussion needs to be not just about the edge issue of the day. It
needs to be about what forum, and what means can be used to enforce this
integrity. Post-9/11 the ISP community has significantly more hammers in
its arsenal now that it did in May 2000. Perhaps NANOG is not the right
forum to discuss, but if not, what is? This is truly an operational threat
to the whole community. Leadership needs to come from the largest
providers, not just from the smallest.

Today the threat is rogue data centres hosting spammers trying to game the
system, tolerated by their up stream providers. Does this really need to be
a hostile state or quasi-state actor deliberately threatening the
infrastructure before serious coordinated action is taken?

We really do know better.

Eric Carroll

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Scott Weeks]

--- b...@herrin.us wrote:
From: William Herrin 

According to the New York Times it was 300 gbps and Cyberbunker was the bad guy.
http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0
-


Got a link that we don't have to allow cookies and have to create an account to 
read?

scott

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Paul Ferguson]

On Wed, Mar 27, 2013 at 12:18 PM, Joshua Goldbard  wrote:

> That was a really big attack.
>
> The scary part is that it's all DNS reflection, meaning the attackers only 
> need 3Gbps of bandwidth to generate 300Gbps of DDoS.
>
> Imagine if they compromised some of the medium sized corporate networks along 
> with these Botnets. I don't know if the exchanges could hold up against 1Tbps 
> of DDoS, and the difference between 300 and 1000Gbps is not a lot.
>
> While I'm excited that CloudFlare is doing such a good job bringing this to 
> the attention of the masses I can't help but feel that this is essentially a 
> time bomb. If this attack was an order of magnitude larger, things might be 
> very different.
>

Consider this a call-to-arms, in all aspects. Please.

- ferg


-- 
"Fergie", a.k.a. Paul Ferguson
 fergdawgster(at)gmail.com

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Warren Bailey]

As cyberbunker stops killing spamhaus and goes after Gilmore.. I think
these are the guys who used to colo HavenCo after they burnt their
platform down? I'm not sure how I feel about Cloudflare comparing being
packeted to a nuclear bomb? After the packeting drys up, is there really
total devastation? Seems to me it would better to compare it to something
like a giant traffic jam
(http://en.wikipedia.org/wiki/China_National_Highway_110_traffic_jam) not
miles of land completely wiped out with zero hope of salvage? Unless cisco
has implemented a mechanism to melt a router when the traffic exceeds
100gbps? ;)

On 3/27/13 12:22 PM, "William Herrin"  wrote:

>On Wed, Mar 27, 2013 at 3:09 PM, Warren Bailey
> wrote:
>> Is someone pissed off at Spamhaus, or was the intention to packet them
>>so
>> hard their entire network ceased to exist so they can no longer offer
>> DROP/RBL/xyz service?
>
>According to the New York Times it was 300 gbps and Cyberbunker was the
>bad guy.
>
>http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becom
>es-internet-snarling-attack.html?pagewanted=all&_r=0
>
>-Bill
>

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Jordan Michaels]

You won't care "who" until the target is you. ;)

Warm Regards,
Jordan Michaels

On 03/27/2013 12:09 PM, Warren Bailey wrote:

Seldom do hax0r nations target things without some type of
"justification". I don't really care who is being internet murdered, I
care why.

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[William Herrin]

On Wed, Mar 27, 2013 at 3:09 PM, Warren Bailey
 wrote:
> Is someone pissed off at Spamhaus, or was the intention to packet them so
> hard their entire network ceased to exist so they can no longer offer
> DROP/RBL/xyz service?

According to the New York Times it was 300 gbps and Cyberbunker was the bad guy.

http://www.nytimes.com/2013/03/27/technology/internet/online-dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0

-Bill

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Joshua Goldbard]

That was a really big attack.

The scary part is that it's all DNS reflection, meaning the attackers only need 
3Gbps of bandwidth to generate 300Gbps of DDoS.

Imagine if they compromised some of the medium sized corporate networks along 
with these Botnets. I don't know if the exchanges could hold up against 1Tbps 
of DDoS, and the difference between 300 and 1000Gbps is not a lot.

While I'm excited that CloudFlare is doing such a good job bringing this to the 
attention of the masses I can't help but feel that this is essentially a time 
bomb. If this attack was an order of magnitude larger, things might be very 
different.

Cheers,
Joshua

Sent from my iPhone

On Mar 27, 2013, at 12:10 PM, "Warren Bailey" 
 wrote:

> Is someone pissed off at Spamhaus, or was the intention to packet them so
> hard their entire network ceased to exist so they can no longer offer
> DROP/RBL/xyz service?
> 
> Seldom do hax0r nations target things without some type of
> "justification". I don't really care who is being internet murdered, I
> care why. 
> 
> It's probably the same people who have been posting news articles from
> Ashworth's email.
> 
> On 3/27/13 11:44 AM, "Jay Ashworth"  wrote:
> 
>> http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
>> 
>> Yes: 120 gigabits/second, primarily of DNS amplification traffic.
>> 
>> Still think it's optional to implement BCP38 pervasively?
>> 
>> Cheers,
>> -- jra
>> -- 
>> Jay R. Ashworth  Baylink
>> j...@baylink.com
>> Designer The Things I Think   RFC
>> 2100
>> Ashworth & Associates http://baylink.pitas.com 2000 Land
>> Rover DII
>> St Petersburg FL USA   #natog  +1 727 647
>> 1274
>> 
>> 
> 
> 
> 

Re: BCP38 - Internet Death Penalty

[Saku Ytti]

On (2013-03-27 11:05 -0500), Jack Bates wrote:

> I'm not arguing that the process can't be done. The problem is,
> there are a number of networks that don't know it needs to be done
> and why, or they don't know how to do it. There are a number of
> networks that have no concept of scripting changes into their
> routers.

Exactly. If we target BCP38 at last-mile we have 0 hope to achieve
sufficient coverage to make spoofing attacks less practical than HTTP GET
from unspoofed address.

I think we should educate tier2 operators who offer transit to tier3. It's
most practical place for BCP38. tier1<->tier2 can't do it, strict IRR
prefix-filtering is not practical. tier2<->tier3 can do it, it's practical
to do strict BGP prefix-filter.

If you are doing strict BGP prefix-filter, it's either very easy to
generate ACL while at it or 0 work in say JunOS, as you can just use same
prefix-list for firewall filter. 



Open recursors may have been discussion point pre-DNSSEC world, post DNSSEC
world it's easy enough to find large RRs from arbitrary authorative server,
that is, even if you'd close all open recursors problem would not go away.

-- 
  ++ytti

Re: Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Warren Bailey]

Is someone pissed off at Spamhaus, or was the intention to packet them so
hard their entire network ceased to exist so they can no longer offer
DROP/RBL/xyz service?

Seldom do hax0r nations target things without some type of
"justification". I don't really care who is being internet murdered, I
care why. 

It's probably the same people who have been posting news articles from
Ashworth's email.

On 3/27/13 11:44 AM, "Jay Ashworth"  wrote:

>http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
>
>Yes: 120 gigabits/second, primarily of DNS amplification traffic.
>
>Still think it's optional to implement BCP38 pervasively?
>
>Cheers,
>-- jra
>-- 
>Jay R. Ashworth  Baylink
>j...@baylink.com
>Designer The Things I Think   RFC
>2100
>Ashworth & Associates http://baylink.pitas.com 2000 Land
>Rover DII
>St Petersburg FL USA   #natog  +1 727 647
>1274
>
>

Re: Open Resolver Problems

[Joe Abley]

On 2013-03-27, at 14:52, Jared Mauch  wrote:

> I am very concerned about examples such as this possibly being implemented by 
> a well intentioned sysadmin or neteng type without understanding their query 
> load and patterns.  bind with the rrl patch does log when things are 
> happening.  While the data is possible to extract from iptables, IMHO it's 
> not quite as easy to audit as a syslog.

For an authoritative-only server, people can expect coarse rate-limits such as 
those quoted earlier with iptables to give false positives and to reject 
legitimate queries. RRL is far safer.

For a recursive server, I agree you need a much better understanding of your 
traffic patterns before you try something like the iptables example. Dropping 
queries from your own clients' stub resolvers has an immediate support cost. 
You *really* don't want false positives, there.


Joe

Re: BCP38 - Internet Death Penalty

[Jay Ashworth]

- Original Message -
> From: "Valdis Kletnieks" 

> On Wed, 27 Mar 2013 10:51:35 -0500, Jack Bates said:
> > They are not, and I can think of quite a few people who would stare
> > blankly at you for making such a statement. Of course, I can think
> > of plenty of people who we'd like to see implementing BCP38 concepts
> > that would need you to define ingress and egress.
> 
> Of course, the fact they don't understand ingress and egress are *totally*
> unrelated to the fact they can't figure out how to deploy BCP38, correct?

Oh, Jeezus; do we need to start seeing geek licenses[1] before we'll turn
up your BGP session or Transit link now?

Have we really sunk that low?[2]

Cheers,
-- jra
[1] Seriously; I think we need these; for escalating to Tier 3 support if
nothing else.[3]
[2] No, I don't actually want an answer to this; I'm depressed enough.
[3] "Hello, Road Runner business support!" "Yes, my Geek License number is 
G 17135, and I need to speak to your backbone BGP coordinator." "I'll 
transfer you right now, sir."
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA   #natog  +1 727 647 1274

Re: Open Resolver Problems

[Jared Mauch]

On Mar 27, 2013, at 11:54 AM, Owen DeLong  wrote:

> It's been available in linux for a long time, just not in BIND…
> 
> Here is a working ip6tales example:
> 
> -A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m udp -p udp 
> --dport 53 -j ACCEPT
> -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m udp 
> -p udp --dport 53 -j ACCEPT
> -A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m tcp -p tcp 
> --dport 53 -j ACCEPT
> -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m tcp 
> -p tcp --dport 53 -j ACCEPT
> -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit 
> --limit 30/minute --limit-burst 90 -j ACCEPT
> -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit 
> --limit 30/minute --limit-burst 90 -j ACCEPT
> 
> YMMV and you may wish to provide tighter limits (less than 30 QPM or a burst 
> of <90).


I am very concerned about examples such as this possibly being implemented by a 
well intentioned sysadmin or neteng type without understanding their query load 
and patterns.  bind with the rrl patch does log when things are happening.  
While the data is possible to extract from iptables, IMHO it's not quite as 
easy to audit as a syslog.

- Jared

Re: BCP38 - Internet Death Penalty

[Valdis . Kletnieks]

On Wed, 27 Mar 2013 10:51:35 -0500, Jack Bates said:

> They are not, and I can think of quite a few people who would stare
> blankly at you for making such a statement. Of course, I can think of
> plenty of people who we'd like to see implementing BCP38 concepts that
> would need you to define ingress and egress.

Of course, the fact they don't understand ingress and egress are *totally*
unrelated to the fact they can't figure out how to deploy BCP38, correct?


pgpE00q9ub6UQ.pgp
Description: PGP signature

Re: Open Resolver Problems

[Marco Davids]

Op 27-03-13 16:54, Owen DeLong schreef:
> It's been available in linux for a long time, just not in BIND…

Not entirely true:

http://www.redbarn.org/dns/ratelimits
>
> Here is a working ip6tales example:
>
Tricky...

There is also the 'hashlimit' module (at least for v4, not sure about
v6), that may be a better approach, because it works on a 'per ip
address'-basis.

See https://lists.isc.org/pipermail/bind-users/2012-July/088223.html for
some inspiration of how it may be of value.

--
Marco

On Mar 27, 2013, at 6:47 AM, William Herrin  wrote:
>> On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka  wrote:
>>> Authoritative DNS servers need to implement rate limiting. (a client
>>> shouldn't query you twice for the same thing within its TTL).
>> Right now that's a complaint for the mainstream software authors, not
>> for the system operators. When the version of Bind in Debian Stable
>> implements this feature, I'll surely turn it on.
>>
>> Regards,
>> Bill Herrin
>>
>>
>> -- 
>> William D. Herrin  her...@dirtside.com  b...@herrin.us
>> 3005 Crane Dr. .. Web: 
>> Falls Church, VA 22042-3004
>


-- 
Marco Davids




smime.p7s
Description: S/MIME-cryptografische ondertekening

Cloudflare, and the 120Gbps DDOS "that almost broke the Internet"

[Jay Ashworth]

http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet

Yes: 120 gigabits/second, primarily of DNS amplification traffic.

Still think it's optional to implement BCP38 pervasively?

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA   #natog  +1 727 647 1274

Re: BCP38 needs advertising

[Paul Ferguson]

But of course. :-)

Also, just saw this:

http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet

- ferg

On Wed, Mar 27, 2013 at 11:02 AM, Arturo Servin  wrote:

>
> And do not forget 
>
> http://tools.ietf.org/html/bcp38
>
> :)
>
> -as
>
>
> On 3/27/13 2:17 PM, Paul Ferguson wrote:
>> Please reference:
>>
>> http://openresolverproject.org/
>> http://spoofer.csail.mit.edu/
>> http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack
>>
>> ...and anything else to raise the awareness level.
>>
>> Thanks,
>>
>> - ferg (co-perpetrator of BCP38)  :-)
>>
>> On Wed, Mar 27, 2013 at 9:48 AM, Alain Hebert  wrote:
>>
>>> bcp38.org coming soon =D
>>>
>>> -
>>> Alain Hebertaheb...@pubnix.net
>>> PubNIX Inc.
>>> 50 boul. St-Charles
>>> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
>>> Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443
>>>
>>> On 03/27/13 11:20, Jack Bates wrote:
 Outside of needing more details and examples, BCP38 could use more
 advertising.

 The best option, if they would accept it, is to have all RIRs mention
 BCP38 as well as require that mention of BCP38 be included in all IP
 justification requests to customers (so that those who receive
 netblocks from their ISPs are also aware of it).

 For ARIN, at least, having it mentioned in the attestation process
 wouldn't be a bad idea. At least someone of management would be aware
 of it.

 The only issue is see concerning the RIRs is that they may object to
 it being out of scope to their duties. However, informing people of
 something is not requiring implementation of something. On the other
 hand, we know that there are a great number of networks that don't
 participate with the community at large and may have no idea about
 BCP38 and why it is important.


 Jack





>>>
>>>
>>
>>
>>



-- 
"Fergie", a.k.a. Paul Ferguson
 fergdawgster(at)gmail.com

Re: Line cut in Mediterranean?

[Aftab Siddiqui]

Well, it's not just SMW4 outage, we've been witnessing serious issues on
IMEWE for couple of weeks now and this outages just made it worse.
So, right now most of the traffic taking east bound routes.
Who needs DDoS at this stage, these links are already chocked up :)

> Maybe it was because of this: Global Internet Slows after 'biggest attack
> in history'
> http://www.bbc.co.uk/news/technology-21954636
>
>


-- 
Regards,

Aftab A. Siddiqui

Re: BCP38 needs advertising

[Arturo Servin]

And do not forget 

http://tools.ietf.org/html/bcp38

:)

-as


On 3/27/13 2:17 PM, Paul Ferguson wrote:
> Please reference:
> 
> http://openresolverproject.org/
> http://spoofer.csail.mit.edu/
> http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack
> 
> ...and anything else to raise the awareness level.
> 
> Thanks,
> 
> - ferg (co-perpetrator of BCP38)  :-)
> 
> On Wed, Mar 27, 2013 at 9:48 AM, Alain Hebert  wrote:
> 
>> bcp38.org coming soon =D
>>
>> -
>> Alain Hebertaheb...@pubnix.net
>> PubNIX Inc.
>> 50 boul. St-Charles
>> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
>> Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443
>>
>> On 03/27/13 11:20, Jack Bates wrote:
>>> Outside of needing more details and examples, BCP38 could use more
>>> advertising.
>>>
>>> The best option, if they would accept it, is to have all RIRs mention
>>> BCP38 as well as require that mention of BCP38 be included in all IP
>>> justification requests to customers (so that those who receive
>>> netblocks from their ISPs are also aware of it).
>>>
>>> For ARIN, at least, having it mentioned in the attestation process
>>> wouldn't be a bad idea. At least someone of management would be aware
>>> of it.
>>>
>>> The only issue is see concerning the RIRs is that they may object to
>>> it being out of scope to their duties. However, informing people of
>>> something is not requiring implementation of something. On the other
>>> hand, we know that there are a great number of networks that don't
>>> participate with the community at large and may have no idea about
>>> BCP38 and why it is important.
>>>
>>>
>>> Jack
>>>
>>>
>>>
>>>
>>>
>>
>>
> 
> 
> 

Re: Line cut in Mediterranean?

[Huasong Zhou]

Maybe it was because of this: Global Internet Slows after 'biggest attack
in history'
http://www.bbc.co.uk/news/technology-21954636



Huasong Zhou
Associate

Kalorama Group, LLC
1000 Potomac Street, NW, Suite 350
Washington, D.C. 20007
Mobile: +1 763 221 6784
Email: huas...@kalorama.com
www.kalorama.com 




On 3/27/13 12:38 PM, "Anurag Bhatia"  wrote:

>Yes smw4 issues across Egypt.
>
>In India (and Pakistan also) services are badly impacted.
>
>
>Here in India most of traffic from major networks is going via East Asia
>route and we are experiencing latency of over 700ms with US and Europe
>from
>last few hours.
>
>On Wed, Mar 27, 2013 at 6:50 PM, James Smith
>wrote:
>
>>
>> Thanks for the quick responses, great information!
>>
>> > From: thepacketmas...@hotmail.com
>> > To: nanog@nanog.org
>> > Subject: Line cut in Mediterranean?
>> > Date: Wed, 27 Mar 2013 08:49:10 -0400
>> >
>> >
>> > Getting reports from a third party vendor that there's been a line cut
>> in the Mediterranean that is affecting some Internet traffic.  Anyone
>>have
>> any details?
>> >
>>
>>
>
>
>
>-- 
>
>Anurag Bhatia
>anuragbhatia.com
>
>Linkedin  |
>Twitter
>Skype: anuragbhatia.com

Re: BCP38 needs advertising

[Paul Ferguson]

Please reference:

http://openresolverproject.org/
http://spoofer.csail.mit.edu/
http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack

...and anything else to raise the awareness level.

Thanks,

- ferg (co-perpetrator of BCP38)  :-)

On Wed, Mar 27, 2013 at 9:48 AM, Alain Hebert  wrote:

> bcp38.org coming soon =D
>
> -
> Alain Hebertaheb...@pubnix.net
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443
>
> On 03/27/13 11:20, Jack Bates wrote:
>> Outside of needing more details and examples, BCP38 could use more
>> advertising.
>>
>> The best option, if they would accept it, is to have all RIRs mention
>> BCP38 as well as require that mention of BCP38 be included in all IP
>> justification requests to customers (so that those who receive
>> netblocks from their ISPs are also aware of it).
>>
>> For ARIN, at least, having it mentioned in the attestation process
>> wouldn't be a bad idea. At least someone of management would be aware
>> of it.
>>
>> The only issue is see concerning the RIRs is that they may object to
>> it being out of scope to their duties. However, informing people of
>> something is not requiring implementation of something. On the other
>> hand, we know that there are a great number of networks that don't
>> participate with the community at large and may have no idea about
>> BCP38 and why it is important.
>>
>>
>> Jack
>>
>>
>>
>>
>>
>
>



-- 
"Fergie", a.k.a. Paul Ferguson
 fergdawgster(at)gmail.com

Cisco Security Advisory: Cisco IOS Software Zone-Based Policy Firewall Session Initiation Protocol Inspection Denial of Service Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Zone-Based Policy Firewall Session Initiation
Protocol Inspection Denial of Service Vulnerability

Advisory ID: cisco-sa-20130327-cce

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

Cisco IOS Software contains a memory leak vulnerability that could be
triggered through the processing of malformed Session Initiation
Protocol (SIP) messages. Exploitation of this vulnerability could
cause an interruption of services. Only devices that are configured
for SIP inspection are affected by this vulnerability.

Cisco has released free software updates that address this
vulnerability. There are no workarounds for devices that must run SIP
inspection.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-cce

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd0ACgkQUddfH3/BbTo1NQD+JTLByafJPlfucXQ7tGEHnYy5
vVv944CH2/B0vC3+AHUA/Aw9dc2MzCzkrKELNu9FQDBFkr5lIhdY9i942xPDfHKQ
=6IL2
-END PGP SIGNATURE-

Cisco Security Advisory: Cisco IOS Software Internet Key Exchange Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Internet Key Exchange Vulnerability

Advisory ID: cisco-sa-20130327-ike

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

The Cisco IOS Software Internet Key Exchange (IKE) feature contains a
denial of service (DoS) vulnerability.

Cisco has released free software updates that address this
vulnerability. Workarounds that mitigate this vulnerability are not
available.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-ike

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd0ACgkQUddfH3/BbTovwQD8DwYcxZks8h9lxLcC9YX0Stal
GfVltUM7jduv3M2tsQgBAIdGU+jBhC8Ct4i/0idzEkoX6o8TAK3EbcUqZt9QjK6F
=Viuu
-END PGP SIGNATURE-

Re: BCP38 needs advertising

[Alain Hebert]

bcp38.org coming soon =D

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 03/27/13 11:20, Jack Bates wrote:
> Outside of needing more details and examples, BCP38 could use more
> advertising.
>
> The best option, if they would accept it, is to have all RIRs mention
> BCP38 as well as require that mention of BCP38 be included in all IP
> justification requests to customers (so that those who receive
> netblocks from their ISPs are also aware of it).
>
> For ARIN, at least, having it mentioned in the attestation process
> wouldn't be a bad idea. At least someone of management would be aware
> of it.
>
> The only issue is see concerning the RIRs is that they may object to
> it being out of scope to their duties. However, informing people of
> something is not requiring implementation of something. On the other
> hand, we know that there are a great number of networks that don't
> participate with the community at large and may have no idea about
> BCP38 and why it is important.
>
>
> Jack
>
>
>
>
>

Cisco Security Advisory: Cisco IOS Software Resource Reservation Protocol Denial of Service Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Resource Reservation Protocol Denial of Service
Vulnerability

Advisory ID: cisco-sa-20130327-rsvp

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

The Resource Reservation Protocol (RSVP) feature in Cisco IOS Software
and Cisco IOS XE Software contains a vulnerability when used on a
device that has Multiprotocol Label Switching with Traffic Engineering
(MPLS-TE) enabled. Successful exploitation of the vulnerability could
allow an unauthenticated, remote attacker to cause a reload of the
affected device. Repeated exploitation could result in a sustained
denial of service (DoS) condition.

Cisco has released free software updates that address this
vulnerability. There are no workarounds available to mitigate this
vulnerability.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-rsvp

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd4ACgkQUddfH3/BbTqZ+AD/SPWuHu+4uf/xKA+RAbRbCZxd
H9SFakcWJIPsy9TYjBABAI6/LmnQ9FrB1PHcVABckjYOnB+9JUd03ynxrsFPzIQS
=W+Lt
-END PGP SIGNATURE-

Cisco Security Advisory: Cisco IOS Software Network Address Translation Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Network Address Translation Vulnerability

Advisory ID: cisco-sa-20130327-nat

Revision 1.0

For Public Release 2013 March 27 10:00  UTC (GMT)
+-

Summary
===

The Cisco IOS Software implementation of the virtual routing and
forwarding (VRF) aware network address translation (NAT) feature
contains a vulnerability when translating IP packets that could allow
an unauthenticated, remote attacker to cause a denial of service (DoS)
condition.

Cisco has released free software updates that address this
vulnerability. Workarounds that mitigate this vulnerability are not
available.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-nat

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd0ACgkQUddfH3/BbTrndAD/Qxm/suF3S/US+6bDND+/OKB3
9KpBW/wUPVC2+87IFRQBAIXFrAjFqnbmmBAKFEVZztVhRN1TlOW9JL7mKd6SXwZw
=jAQM
-END PGP SIGNATURE-

Cisco Security Advisory: Cisco IOS Software Smart Install Denial of Service Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Smart Install Denial of Service Vulnerability

Advisory ID: cisco-sa-20130327-smartinstall

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

The Smart Install client feature in Cisco IOS Software contains a
vulnerability that could allow an unauthenticated, remote attacker to
cause a denial of service (DoS) condition on an affected device.

Affected devices that are configured as Smart Install clients are
vulnerable.

Cisco has released free software updates that address this
vulnerability. There are no workarounds for devices that have the
Smart Install client feature enabled.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-smartinstall

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd4ACgkQUddfH3/BbToUsAD+NSDtaCAvOzfjmsqhxVZN6Uy+
ceAxXTPCp6M0n8yGk0sA/1uJk8CWE1yjCtTu1IDGX8K/SUvWFEUi0pqFyKfKVFEa
=eRMY
-END PGP SIGNATURE-

Cisco Security Advisory: Cisco IOS Software Protocol Translation Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software Protocol Translation Vulnerability

Advisory ID: cisco-sa-20130327-pt

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

The Cisco IOS Software Protocol Translation (PT) feature contains a
vulnerability that could allow an unauthenticated, remote attacker to
cause a denial of service (DoS) condition.

Cisco has released free software updates that address this
vulnerability. Workarounds that mitigate this vulnerability are
available.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-pt

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd4ACgkQUddfH3/BbTr/hQEAhB32OjahAaNFUbeYsZloNqCX
C9JHEqRP4k4Y27LcWZUA+wTwW0yKpKzQ9+ZDvaWYiXtL1iSvOhlSjS178A3kMIhb
=JlLG
-END PGP SIGNATURE-

Cisco Security Advisory: Cisco IOS Software IP Service Level Agreement Vulnerability

[Cisco Systems Product Security Incident Response Team]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Cisco IOS Software IP Service Level Agreement Vulnerability

Advisory ID: cisco-sa-20130327-ipsla

Revision 1.0

For Public Release 2013 March 27 16:00  UTC (GMT)
+-

Summary
===

The Cisco IOS Software implementation of the IP Service Level
Agreement (IP SLA) feature contains a vulnerability in the validation
of IP SLA packets that could allow an unauthenticated, remote attacker
to cause a denial of service (DoS) condition.

Cisco has released free software updates that address this
vulnerability. Mitigations for this vulnerability are available.

This advisory is available at the following link:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20130327-ipsla

Note: The March 27, 2013, Cisco IOS Software Security Advisory bundled
publication includes seven Cisco Security Advisories. All advisories
address vulnerabilities in Cisco IOS Software. Each Cisco IOS Software
Security Advisory lists the Cisco IOS Software releases that correct
the vulnerability or vulnerabilities detailed in the advisory as well
as the Cisco IOS Software releases that correct all Cisco IOS Software
vulnerabilities in the March 2013 bundled publication.

Individual publication links are in "Cisco Event Response: Semiannual
Cisco IOS Software Security Advisory Bundled Publication" at the
following link:
http://www.cisco.com/web/about/security/intelligence/Cisco_ERP_mar13.html
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org

iF4EAREIAAYFAlFQcd0ACgkQUddfH3/BbTp8QwD+IPK7Dzz7B0uga/FtZKjYU9XC
ik2D1EIVMDWcFNYovn8A/i2M+COtgQr9j/7CuMRdNfnAoA65JOxRHu4NTW7cdZoo
=w51Y
-END PGP SIGNATURE-

Re: Line cut in Mediterranean?

[Anurag Bhatia]

Yes smw4 issues across Egypt.

In India (and Pakistan also) services are badly impacted.


Here in India most of traffic from major networks is going via East Asia
route and we are experiencing latency of over 700ms with US and Europe from
last few hours.

On Wed, Mar 27, 2013 at 6:50 PM, James Smith wrote:

>
> Thanks for the quick responses, great information!
>
> > From: thepacketmas...@hotmail.com
> > To: nanog@nanog.org
> > Subject: Line cut in Mediterranean?
> > Date: Wed, 27 Mar 2013 08:49:10 -0400
> >
> >
> > Getting reports from a third party vendor that there's been a line cut
> in the Mediterranean that is affecting some Internet traffic.  Anyone have
> any details?
> >
>
>



-- 

Anurag Bhatia
anuragbhatia.com

Linkedin  |
Twitter
Skype: anuragbhatia.com

Re: BCP38 - Internet Death Penalty

[Jack Bates]

On 3/27/2013 10:40 AM, William Herrin wrote:
Build a web page where a downstream can set the filters on his 
interface at his convenience. Apply some basic sanity checks against 
wide-open. Worry about small lies from a forensic after-the-fact 
perspective. This problem has a trivial technology-only solution.


Well, that would definitely be easier than the RR updates I have to do.

I'm not arguing that the process can't be done. The problem is, there 
are a number of networks that don't know it needs to be done and why, or 
they don't know how to do it. There are a number of networks that have 
no concept of scripting changes into their routers.


Implementing BCP38 isn't a technology issue as much as an education 
issue. The BCP provides a brief methodology to accomplish its goals. It 
doesn't mention other methods or point to resources that an uneducated 
person may need. The problem might not be with BCP38. Perhaps it is as 
detailed as it needs to be from an IETF standpoint. However, it is not 
enough information to cultivate the changes we need.



Jack

Re: Open Resolver Problems

[Owen DeLong]

It's been available in linux for a long time, just not in BIND…

Here is a working ip6tales example:

-A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m udp -p udp 
--dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m udp 
-p udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -s 2620:0:930::/48 -m state --state NEW -m tcp -p tcp 
--dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m tcp 
-p tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit 
--limit 30/minute --limit-burst 90 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit 
--limit 30/minute --limit-burst 90 -j ACCEPT

YMMV and you may wish to provide tighter limits (less than 30 QPM or a burst of 
<90).

Owen





On Mar 27, 2013, at 6:47 AM, William Herrin  wrote:

> On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka  wrote:
>> Authoritative DNS servers need to implement rate limiting. (a client
>> shouldn't query you twice for the same thing within its TTL).
> 
> Right now that's a complaint for the mainstream software authors, not
> for the system operators. When the version of Bind in Debian Stable
> implements this feature, I'll surely turn it on.
> 
> Regards,
> Bill Herrin
> 
> 
> -- 
> William D. Herrin  her...@dirtside.com  b...@herrin.us
> 3005 Crane Dr. .. Web: 
> Falls Church, VA 22042-3004

Re: BCP38 - Internet Death Penalty

[Jack Bates]

On 3/27/2013 10:25 AM, Mark Andrews wrote:

Technologies change.  Concepts rarely do.  BCP38 is technology neutral.
If we follow that, we should just state "Don't allow spoofed IP 
Addresses!" and leave it to the individual to figure it out. BCP38 
leaves that premise by mentioning ingress filtering as well as 
mentioning some issues with DHCP. If nothing else, it should have an 
extensive appendix to point people to the correct documentation for 
implementation.



EGRESS filters are just INGRESS filters applied a couple of hops later.


They are not, and I can think of quite a few people who would stare 
blankly at you for making such a statement. Of course, I can think of 
plenty of people who we'd like to see implementing BCP38 concepts that 
would need you to define ingress and egress.


Fact: Ignorance is abound on the Internet, even in the running of networks.

If you want a solid change, we're going to have to educate people; 
especially those who are not on NANOG, don't know about the IETF, and 
have never heard of an RFC or BCP.



Jack

Re: BCP38 - Internet Death Penalty

[William Herrin]

On Wed, Mar 27, 2013 at 11:02 AM, Jack Bates  wrote:
> It's also not a bad idea for an ISP to deploy EGRESS filters if they do not
> offer BGP Transit services.

Nor is it a bad idea for their upstream to inquire as to whether the
downstream offers BGP transit services and apply INGRESS filters if
they do not.

> This way they are not depending on their transit
> providers to handle spoof protection and they cover their entire network
> regardless of last mile ingress filtering. This doesn't generally work well
> when doing transit services of any size due to the number of egress filter
> updates you'd have to issue, but it is great for the small/medium ISP.

Build a web page where a downstream can set the filters on his
interface at his convenience. Apply some basic sanity checks against
wide-open. Worry about small lies from a forensic after-the-fact
perspective. This problem has a trivial technology-only solution.

Regards,
Bill Herrin


-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: BCP38 - Internet Death Penalty

[John Curran]

On Mar 27, 2013, at 10:23 AM, Jay Ashworth  wrote:

> Indeed, but I have an even better example of how that's already done, that
> is probably pertinent.
> 

> The National Electric Code is assimilated law now, I think, in every
> state in the US.  It is promulgated by the National Fire Protection 
> Association, which is a standards organization originally started by 
> insurors to reduce their exposure and expenses; by professional consensus,
> they have become, effectively, a lawmaking body.
> 
> So they're not the government, they're subject-matter experts, technically
> competent to have opinions, and their opinions are assimilated into 
> controlling law.

Indeed... a perfect example of industry self-regulation supplemented by
a mandatory legal framework referencing the best practice documents.

> Is BCP38 *not* well enough though out even for large and medium sized 
> carriers to adopt as contractual language, ...

You're back to discussing voluntary mechanisms in the above, but 
your example is a case where compliance is due to legislation at 
both federal and state levels.

> much less for FCC or someone to
> impose upon them?  If so, we should work on it further.

Umm... How many North American ISP's/datacenters/web hosting firms were 
aware of the BCP 38 development as it was on-going, and participated in 
some manner in its review?  The IETF is a wonderful organization, but it
is not exactly overflowing with representation from the service provider 
community.  Also, given that you really need these practices picked up
on a global basis, repeat the above question with "Global" rather than
North American...  

If you actually want to see certain technical practices made mandatory
for Internet service providers globally, then you need a development 
process for those practices which fairly robust to insure that the 
practices are equally germane and reasonable to many different service 
providers in many different parts of the world.  It's actually relatively
easy to get governments to require compliance with accepted technical
practices for the Internet, the problem has been we've never taken the
effort to produce best practices with sufficient rigor than any given
government can know that it has the necessary backing (of many of the
service providers in its domain) needed to actually require compliance.

(With regard to the FCC, there is some question as to whether or not 
their mandate would allow establishing required practices for ISPs...
To the extent that VoIP traffic is being carried, this is far more 
likely to be possible, and hence folks should be aware of various
activities such as the CSRIC "Communications Security, Reliability 
and Interoperability Council", which develops recommendations that
could effectively become requirements on Internet Service Providers
in some contexts. 

Noe that the problem with this sort of approach is that having dozens 
of countries all developing their own specific technical best practices 
is most likely to cumulatively interact in ways impossible to comply 
with...  Hence, the need for clear global technical best practices,
through which countries with a particular desire to "improve the
state of the Internet" can channel their legislative desires...)

FYI,
/John

Re: BCP38 - Internet Death Penalty

[Mark Andrews]

In message <515309ec.4070...@brightok.net>, Jack Bates writes:
> On 3/27/2013 9:23 AM, Jay Ashworth wrote:
> > Is BCP38 *not* well enough though out even for large and medium sized 
> > carriers to adopt as contractual language, much less for FCC or 
> > someone to impose upon them? If so, we should work on it further.
> 
> BCP38 could definitely use some work. It is correct as a general 
> concept. It does not go into depth of the different available 
> technologies and how they might be of use. For example, dhcp is nice, 
> but it usually requires uRPF (sometimes with exceptions) depending on 
> the vendor. If BGP filters are being applied, it is usually not hard to 
> apply packet filtering according to the same route filters. Some NSPs 
> use traditional ingress filtering, while others have uRPF enabled with 
> exception lists. Some require that you send all networks, but set 
> communities for networks you don't want routed yet allowed via uRPF 
> (which usually means anyone connected to the same router as you will 
> still route your way).

Technologies change.  Concepts rarely do.  BCP38 is technology neutral.
 
> It's also not a bad idea for an ISP to deploy EGRESS filters if they do 
> not offer BGP Transit services. This way they are not depending on their 
> transit providers to handle spoof protection and they cover their entire 
> network regardless of last mile ingress filtering. This doesn't 
> generally work well when doing transit services of any size due to the 
> number of egress filter updates you'd have to issue, but it is great for 
> the small/medium ISP.

EGRESS filters are just INGRESS filters applied a couple of hops later.
 
> Jack
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

BCP38 needs advertising

[Jack Bates]

Outside of needing more details and examples, BCP38 could use more 
advertising.


The best option, if they would accept it, is to have all RIRs mention 
BCP38 as well as require that mention of BCP38 be included in all IP 
justification requests to customers (so that those who receive netblocks 
from their ISPs are also aware of it).


For ARIN, at least, having it mentioned in the attestation process 
wouldn't be a bad idea. At least someone of management would be aware of it.


The only issue is see concerning the RIRs is that they may object to it 
being out of scope to their duties. However, informing people of 
something is not requiring implementation of something. On the other 
hand, we know that there are a great number of networks that don't 
participate with the community at large and may have no idea about BCP38 
and why it is important.



Jack

Re: Open Resolver Problems

[Mark Andrews]

In message <51530632.3020...@brightok.net>, Jack Bates writes:
> On 3/27/2013 9:34 AM, William Herrin wrote:
> > On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates  wrote:
> >>
> >> Tracking the clients would be a huge dataset and be especially complicated
> >> in clusters. They'd be better off at detecting actual attack vectors rather
> >> than rate limiting.
> > I count this among the several reasons I intend to wait until a
> > solution has been accepted into the bind mainline.
> >
> >
> You'll also find that it serves little purpose. The only 2 methods for 
> stopping DNS amplification to my knowledge are:
> 
> 1) tcp
> 
> 2) require all requests to pad out to maximum response
> 
> 3) BCP38 (in spirit)

4) a variant of dns cookies see the draft by Donald Eastlake III
   if you can accept a couple of extra bytes in the reply to a
   non cookie query

* minimal amplification to spoofed queries.
* remove the need to randomise source ports.
* state is stored in the stubs / recursives serves about
  the upstream.
* works with recursive servers and authoritative servers

hash (server secret + client differentiator + time) -> crypto hash

query
[code = 0 (8 bits), extended id (64 bits), client differentiator (64 
bits),
 server time (32 bits), crypto hash ]

response
[code (8 bit), extended id (64 bits), client differentiator' (64 bits),
 server time' (32 bits), crypto hash' ]

A query is only accepted if the presented client differentiator
and server time along with the server secret give the
presented hash and not too much time has elasped otherwise
code is set to 1 and the completed option is returned.

clients record everything after the extended id to send with
future queries.

client differentiator, server time and crypto hash may be missing
on the initial query.

> The first has latency, load, and connection limitations. It is just too 
> expensive.
> 
> The second would stop amplification, however, it will not stop botnets 
> using them in attempts to hide the bot nodes in a very effective manner. 
> It's also unlikely that we'd ever see it implemented.
> 
> The only effective fix is still BCP38 (in spirit).
> 
> 
> Jack
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: BCP38 - Internet Death Penalty

[Jack Bates]

On 3/27/2013 9:23 AM, Jay Ashworth wrote:
Is BCP38 *not* well enough though out even for large and medium sized 
carriers to adopt as contractual language, much less for FCC or 
someone to impose upon them? If so, we should work on it further.


BCP38 could definitely use some work. It is correct as a general 
concept. It does not go into depth of the different available 
technologies and how they might be of use. For example, dhcp is nice, 
but it usually requires uRPF (sometimes with exceptions) depending on 
the vendor. If BGP filters are being applied, it is usually not hard to 
apply packet filtering according to the same route filters. Some NSPs 
use traditional ingress filtering, while others have uRPF enabled with 
exception lists. Some require that you send all networks, but set 
communities for networks you don't want routed yet allowed via uRPF 
(which usually means anyone connected to the same router as you will 
still route your way).


It's also not a bad idea for an ISP to deploy EGRESS filters if they do 
not offer BGP Transit services. This way they are not depending on their 
transit providers to handle spoof protection and they cover their entire 
network regardless of last mile ingress filtering. This doesn't 
generally work well when doing transit services of any size due to the 
number of egress filter updates you'd have to issue, but it is great for 
the small/medium ISP.



Jack

Re: Open Resolver Problems

[Jack Bates]

On 3/27/2013 9:34 AM, William Herrin wrote:

On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates  wrote:


Tracking the clients would be a huge dataset and be especially complicated
in clusters. They'd be better off at detecting actual attack vectors rather
than rate limiting.

I count this among the several reasons I intend to wait until a
solution has been accepted into the bind mainline.


You'll also find that it serves little purpose. The only 2 methods for 
stopping DNS amplification to my knowledge are:


1) tcp

2) require all requests to pad out to maximum response

3) BCP38 (in spirit)

The first has latency, load, and connection limitations. It is just too 
expensive.


The second would stop amplification, however, it will not stop botnets 
using them in attempts to hide the bot nodes in a very effective manner. 
It's also unlikely that we'd ever see it implemented.


The only effective fix is still BCP38 (in spirit).


Jack

Re: Open Resolver Problems

[William Herrin]

On Wed, Mar 27, 2013 at 10:00 AM, Jack Bates  wrote:
> On 3/27/2013 8:47 AM, William Herrin wrote:
>> Right now that's a complaint for the mainstream software authors, not
>> for the system operators. When the version of Bind in Debian Stable
>> implements this feature, I'll surely turn it on.
>
> Tracking the clients would be a huge dataset and be especially complicated
> in clusters. They'd be better off at detecting actual attack vectors rather
> than rate limiting.

I count this among the several reasons I intend to wait until a
solution has been accepted into the bind mainline.


Regards,
Bill Herrin



-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: BCP38 - Internet Death Penalty

[Jay Ashworth]

- Original Message -
> From: "John Curran" 

> On Mar 26, 2013, at 10:51 AM, Jay Ashworth  wrote:
> > The problem here is, of course, one of externalities and the Common
> > Good, hard sales to make in a business environment.
> 
> 
> "Common Good" situations are readily dealt with, but generally not on a
> voluntary basis. You establish how the resource is to be managed, and
> then you put in place mechanisms to deal with enforcement. The problem
> is that en-force-ment contains "force", which is something that we
> really don't want anyone (or set of anyones) using except "governments"
> (which in our social constructs are the only ones supposed to be telling one
> party under penalty of force not to do something otherwise in its best
> interest.)

Basically, though there are lots of other things that are close.

> The problem, of course, with asking governments for help is that the
> output often does not resemble anything related to the original problem
> statement and can make the situation worse...

Yup, a problem which is indeed perennial, and which we mentioned in an
earlier set of exchanges.

> If we had truly global operational best practices for the Internet (ones
> that went through a fairly well-defined policy development process which
> included multiple operator forums from around the globe) then you might
> have a solid chance of producing output which avoided the various anti-
> competitive aspects, and yet were a reasonable basis for governments
> to then step up and indicate should be required for ISPs in their
> operations.
> It wouldn't take very many governments asking "How do I reduce SPAM
> and DDoS attacks?" and hearing back "Please require ISPs to adhere to this
> Best Common Operating Practice Foo-01" before it became common
> practice.

Indeed, but I have an even better example of how that's already done, that
is probably pertinent.

The National Electric Code is assimilated law now, I think, in every
state in the US.  It is promulgated by the National Fire Protection 
Association, which is a standards organization originally started by 
insurors to reduce their exposure and expenses; by professional consensus,
they have become, effectively, a lawmaking body.

So they're not the government, they're subject-matter experts, technically
competent to have opinions, and their opinions are assimilated into 
controlling law.

Is BCP38 *not* well enough though out even for large and medium sized 
carriers to adopt as contractual language, much less for FCC or someone to
impose upon them?  If so, we should work on it further.

If not, do carriers need to be threatened with its imposition to get them
to adopt it contractually?

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA   #natog  +1 727 647 1274

Re: Open Resolver Problems

[Jack Bates]

On 3/27/2013 8:47 AM, William Herrin wrote:

On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka  wrote:

Authoritative DNS servers need to implement rate limiting. (a client
shouldn't query you twice for the same thing within its TTL).

Right now that's a complaint for the mainstream software authors, not
for the system operators. When the version of Bind in Debian Stable
implements this feature, I'll surely turn it on.




Tracking the clients would be a huge dataset and be especially 
complicated in clusters. They'd be better off at detecting actual attack 
vectors rather than rate limiting. However, there are enough nodes out 
there to easily spread a trickle to avoid individual detections. You 
don't want to DOS your amplifier, after all. It also wouldn't be hard to 
rotate through different requests to defeat the "rate limits".



Jack

Re: Open Resolver Problems

[Jared Mauch]

On Mar 27, 2013, at 8:47 AM, Nick Hilliard  wrote:

> then use a vpn and/or provide that service to your users.  Sure, hotels and
> public access wifi does all sorts of stupid and obnoxious stuff, but the
> way to work around this is not by hardwiring your dns to some open resolver.

I've been in many a hotel where 4.2.2.1 is reachable with ttl=1

You must use a VPN or something else to get around places like that.

The hotel I'm typing from right now is even more broken..

Jareds-MacBook-Air:~ jared$ ping 4.2.2.1
PING 4.2.2.1 (4.2.2.1): 56 data bytes
64 bytes from 4.2.2.1: icmp_seq=0 ttl=53 time=17.159 ms
64 bytes from 4.2.2.1: icmp_seq=0 ttl=53 time=17.181 ms (DUP!)
64 bytes from 4.2.2.1: icmp_seq=1 ttl=53 time=16.787 ms
64 bytes from 4.2.2.1: icmp_seq=1 ttl=53 time=17.156 ms (DUP!)
64 bytes from 4.2.2.1: icmp_seq=2 ttl=53 time=22.056 ms
64 bytes from 4.2.2.1: icmp_seq=2 ttl=53 time=22.081 ms (DUP!)
^C

- Jared

Re: Open Resolver Problems

[Joe Abley]

On 2013-03-27, at 09:47, William Herrin  wrote:

> On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka  wrote:
>> Authoritative DNS servers need to implement rate limiting. (a client
>> shouldn't query you twice for the same thing within its TTL).
> 
> Right now that's a complaint for the mainstream software authors, not
> for the system operators. When the version of Bind in Debian Stable
> implements this feature, I'll surely turn it on.

RRL is a moving target, although a promising one.

There are currently three implementations of RRL which all behave slightly 
differently. There is active discussion between the vendors who have 
implemented RRL, and between early adopters and the vendors. The specification 
is not yet stable, and changes in the functionality and the rate-limiting 
behaviour continue to be made.

My assessment is that the implementations I have seen are ready for production 
use, but I think it's understandable given the moving goalpoasts that some 
vendors have not yet promoted the code to be included in stable releases.

As an operator, I understand the benefits of using packaged, stable releases of 
code. However, we also have a responsibility to deal with operational problems 
in a timely way. I think it's worth considering that it may well be worth 
deviating from internal policies about code deployment in this instance; the 
benefits of doing so can be substantial, and the costs of doing so (especially 
if we expect them to be time-limited) are not that high.


Joe

Re: Open Resolver Problems

[William Herrin]

On Tue, Mar 26, 2013 at 10:07 PM, Tom Paseka  wrote:
> Authoritative DNS servers need to implement rate limiting. (a client
> shouldn't query you twice for the same thing within its TTL).

Right now that's a complaint for the mainstream software authors, not
for the system operators. When the version of Bind in Debian Stable
implements this feature, I'll surely turn it on.

Regards,
Bill Herrin


-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: BCP38 - Internet Death Penalty

[William Herrin]

On Tue, Mar 26, 2013 at 9:18 PM, Jay Ashworth  wrote:
>> From: "William Herrin" 
>> Indeed. But it isn't achievable. $Random_SOHO will continue to be
>> hacked on a regular basis. He doesn't have someone working for him
>> with the skill to prevent it. Further victimizing him with a game of
>> whack-a-mole helps nobody.
>>
>> Besides, his failings aren't important to us. What's important to us
>> is that his failings don't impact us. We achieve that by insisting
>> that his ISP not leak his forged packets on to the public Internet. It
>> would be nice if his ISP didn't accept the forged packets at all, but
>> that's really not our problem and we don't need to make it our
>> business.
>
> It's possible I badly misunderstand how things like unicast-rpf work,
> Bill.

No, you're pretty close. The technology known as unicast-rpf works
anywhere there's a choke point where traffic to and from a given set
of IP addresses has no other candidate paths.


> When I say "drop forged traffic coming from...", *who I mean is 'his ISP'*,
> as you suggest in the next graf.  I don't see that there's anyway to *know*
> packets have a forged address anywhere north of the edge of the lowest tier
> IAP the connection is served from.

RPF is but a subset of source address filtering strategies, the one
where the source address filtering always mirrors the routing table.

As a origin-only BGP AS, I know exactly what sources I expect to leave
my system. And my ISPs know what source addresses I've told them to
expect. RPF won't work reliably on those links, but they can be
configured manually or with a tool to accept only the source addresses
I've declared.

If I declare 0.0.0.0/0, a reasonable ISP understands that I'm lying.
If I declare a particular /24 and two days later the ISP gets a trace
request from the apparent registrant (who isn't me) it's not hard to
infer that I may be a bad actor.


> Did I miss something?  Or was I simply unclear?

The problem with RPF is that understanding where you can and can't
employ it is part of a senior network engineer's skill set. But the
trivial network architectures where it can be applied are often handed
off to a junior network engineer.

The skill set is available primarily in the places where it can't be used.


> ...which you would detect ... how?  *Those* aggregator networks have
> no contractual reason to know what's a valid address coming to them,
> unlike the networks to which end sites attach directly.

They most certainly do have a valid contractual reason to know what
routes and source addresses their origin-only AS customer intends to
send them and the responsible transit networks already do filter those
links.


> Filtering based on routes doesn't help; that's *destination address*, not
> source, no?

Except for the special case where RPF is usable, that's correct.


> Yes, filtering your peers, or even transit customers, is effectively
> impossible; it has to be done where addresses are handed out.

Filtering that subset of your customers which consists of non-BGP
speakers and BGP origin-only networks is neither impossible nor
particularly hard. Harder than "rpf on", but not hard. Even if they
lie about what addresses to expect, they're not left with carte
blanche to impersonate any address at all.

The more transit BGP systems which do this, the smaller the spoofing
problem becomes. And there are few enough _transit_ BGP systems (less
than 10,000) that they can be realistically and usefully held
accountable for a failure to do so.


>> I don't care about about pissing them off. I care about pissing off my
>> customers. My customers will be pissed off if they can't reach their
>> customers and suppliers. Who will often be hosted by the target of the
>> IDP. But will much more rarely be the target of the IDP.
>
> Ok; I apologies; I have laid the bike down in the sandy corner at
> this point.  Huh?

My leeway with the CEO ends when I lose customers. So does yours.

For an IDP to be acceptable, the Penalty part has to be something
painful to the offender without pissing off my customers. Cutting him
off the network pisses off my customers who can no longer reach his
customers. It's why no one wins a peering battle with Cogent: Cogent
is willing to take the heat.

Deep sixing the packets to his particular web site is far more
tenable, especially when paired with an explanation of the credible
threat he poses to my customers' networks.


>> Which is A-OK because if we're applying more than 1 or 2 IDPs in a
>> year to folks who weren't intentionally bad actors then we're doing it
>> wrong. It shouldn't ever "scale."
>
> Yes, but you can't measure such a panel on output, you have to measure
> it on *input*, no?

Not particularly. There's nothing wrong with picking the worst current
offenders and letting the rest slide with a warning to clean up their
act or be next on the list.


Regards,
Bill Herrin



-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 

Re: Open Resolver Problems

[Alain Hebert]

Little bit of fun with http://bindguard.activezone.de/

This little example with an open resolver with only 200 queries a
minute...

The following list show the # of queries made followed by the query
in question.

False positive:

69.x.x.x
2 a1.mzstatic.com IN A +
2 a1001.phobos.apple.com IN A +
1153 a.root-servers.net IN A +

^- 1153 root queries under 10s... from a small office...

Old uncontrolled botnet:

5.x.x.141
1020 isc.org IN ANY +ED
64.x.x.22
1440 isc.org IN ANY +ED
64.x.x.82
1075 isc.org IN ANY +ED
64.x.x.50
1011 isc.org IN ANY +ED
64.x.x.242
1103 isc.org IN ANY +ED

This result come from my cheap scripts(tm) and bindguard.

If anyone wish to try it I can provide you with some support files
and examples.

Just contact me offlist.  PS: It will be later today...

Enjoy today's drama.

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

RE: Line cut in Mediterranean?

[James Smith]

Thanks for the quick responses, great information!

> From: thepacketmas...@hotmail.com
> To: nanog@nanog.org
> Subject: Line cut in Mediterranean?
> Date: Wed, 27 Mar 2013 08:49:10 -0400
> 
> 
> Getting reports from a third party vendor that there's been a line cut in the 
> Mediterranean that is affecting some Internet traffic.  Anyone have any 
> details?
> 
  

Re: Line cut in Mediterranean?

[sthaug]

> Getting reports from a third party vendor that there's been a line cut in the 
> Mediterranean that is affecting some Internet traffic.  Anyone have any 
> details?

See the outages list:

https://puck.nether.net/pipermail/outages/2013-March/005386.html

Steinar Haug, Nethelp consulting, sth...@nethelp.no

Re: Line cut in Mediterranean?

[Nick Hilliard]

On 27/03/2013 12:49, James Smith wrote:
> Getting reports from a third party vendor that there's been a line cut
> in the Mediterranean that is affecting some Internet traffic.  Anyone
> have any details?

smw4 is down, off the north coast of egypt:

> http://www.itnewsafrica.com/2013/03/seacom-suffers-another-cable-disruption/

Nick

Re: Line cut in Mediterranean?

[Pierre Emeriaud]

Hello James,

2013/3/27 James Smith :
>
> Getting reports from a third party vendor that there's been a line cut in the 
> Mediterranean that is affecting some Internet traffic.  Anyone have any 
> details?
>

SMW4 : http://www.seacom.mu/news/article-140/seacom-outage-08-40-gmt/

"SEACOM can confirm that at 06:20 GMT 27 March, the SMW4 cable system
suffered a cable cut off the coast of Egypt. Earlier this morning,
SEACOM had restored all services on both SMW4 and IMEWE cable systems.

SEACOM is currently in the process of prioritizing and reallocating
already available capacity for customers and sourcing further capacity
to re-establish full restoration.

We will update further as information becomes available."



regards,
pierre.

Re: BCP38 - Internet Death Penalty

[William Herrin]

On Tue, Mar 26, 2013 at 10:40 PM, Mark Andrews  wrote:
> Surveying which connections are open to address spoofing may or may
> not be a criminal activity.  It all depends on intent of the person
> gathering the data.

Such is the nature of law. When a dead body shows up shot, intent
(fancily called "mens rea") is the difference between murder,
manslaughter and self-defense.

Its true of source address spoofing. Its true of collusion and
anti-trust as well.

Regards,
Bill Herrin

-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Line cut in Mediterranean?

[James Smith]

Getting reports from a third party vendor that there's been a line cut in the 
Mediterranean that is affecting some Internet traffic.  Anyone have any details?
  

Re: Open Resolver Problems

[Nick Hilliard]

On 27/03/2013 12:40, Rich Kulawiec wrote:
> It's necessary because many operations are screwing with DNS results in
> order to advance/suppress political agendas, impose their moral code
> via censorship, profit via redirection to search portals, etc.  If we
> could actually trust that J. Random Hotel would not do so, then yes,
> whatever DNS servers are assigned via DHCP would suffice.

then use a vpn and/or provide that service to your users.  Sure, hotels and
public access wifi does all sorts of stupid and obnoxious stuff, but the
way to work around this is not by hardwiring your dns to some open resolver.

Nick

Re: Open Resolver Problems

[Rich Kulawiec]

On Wed, Mar 27, 2013 at 11:20:54AM +, Nick Hilliard wrote:
> I'm struggling to understand why it's necessary to hard-code dns servers
> into the ip networking configuration of a portable device.  By definition,
> these devices will already have dhcp enabled.

It's necessary because many operations are screwing with DNS results in
order to advance/suppress political agendas, impose their moral code
via censorship, profit via redirection to search portals, etc.  If we
could actually trust that J. Random Hotel would not do so, then yes,
whatever DNS servers are assigned via DHCP would suffice.

(Let me caveat this by saying that I don't have a problem with screwing
with DNS results for operational reasons, e.g., I think refusing to send
DNS queries into DROP-listed space is a good security practice.)

---rsk

Re: Open Resolver Problems

[Alain Hebert]

Same ol' same ol'

(at least since I started this around '93 =D)

On 03/26/13 22:25, Jon Lewis wrote:
> On Tue, 26 Mar 2013, Matthew Petach wrote:
>
>> The concern Valdis raised about securing recursives while still
>> being able to issue static nameserver IPs to mobile devices
>> is an orthogonal problem to Owen putting rate limiters on
>> the authoritative servers for he.net.  If we're all lighting up
>> pitchforks and raising torches, I'd kinda like to know at which
>> castle we're going to go throw pitchforks.
>
> BCP38.  As you can see from the wandering conversation, there are many
> attack vectors that hinge on the ability to spoof the source address,
> and thereby misdirect responses to your DDoS target.  BCP38 filtering
> stops them all.  Or, we can ignore BCP38 for several more years, go on
> a couple years crusade against open recursive resolvers, then against
> non-rate-limited authoratative servers, default public RO SNMP
> communities, etc.
>
> --
>  Jon Lewis, MCP :)   |  I route
>  |  therefore you are
> _ http://www.lewis.org/~jlewis/pgp for PGP public key_
>
IP Spoofing still exists because of lazy Peers...

Same as the ability to hijack a subnet with BGP...  ( *wave* DoD
from 2-3 weeks ago )

But, as always, its our responsibility to kill our infrastructure,
was IRC Servers in the past, now DNS Servers...

Just for those lazy Peers to not HAVE to fix their broken setup.

Same ol', same ol'.

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

Re: Open Resolver Problems

[Alain Hebert]

Well,
   
On 03/27/13 07:20, Nick Hilliard wrote:
> On 26/03/2013 14:21, valdis.kletni...@vt.edu wrote:
>> And if you get a recursive lookup for www.ebay.com from a hotel network,
> I'm struggling to understand why it's necessary to hard-code dns servers
> into the ip networking configuration of a portable device.  By definition,
> these devices will already have dhcp enabled.
>
> Nick
I can only talk as a very small outfit, too often our roaming/office
customers have issues with their local DNS server (in embedded devices
like their $60 linksys) or the local telco (ACME Cable Company)...

Having them using our DNS servers save a lot of time and a massive
amount of money and headache... same idea as offering them our outgoing
mail server (with authentication of course) instead of letting them deal
with their Telco's.

Mind you none of my 2 servers are a problem because they where
always rate-limited...  But a few of my clients where for an amount
around ~80Mbps of amps.

And got fixed within the hour.

Now about the struggling about BCP38...

-
Alain Hebertaheb...@pubnix.net   
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

Re: BCP38 - Internet Death Penalty

[John Curran]

On Mar 26, 2013, at 10:51 AM, Jay Ashworth  wrote:
> The problem here is, of course, one of externalities and the Common Good,
> hard sales to make in a business environment.


"Common Good" situations are readily dealt with, but generally not on a 
voluntary basis.  You establish how the resource is to be managed, and 
then you put in place mechanisms to deal with enforcement.  The problem 
is that en-force-ment contains "force", which is something that we really
don't want anyone (or set of anyones) using except "governments" (which in 
our social constructs are the only ones supposed to be telling one party 
under penalty of force not to do something otherwise in its best interest.)
The problem, of course, with asking governments for help is that the output
often does not resemble anything related to the original problem statement
and can make the situation worse...

If you setup an economic system where folks do the right thing because
it is in their own interest, that's one option that doesn't involve 
government, but we know that is hard to do on technical grounds alone...
A group of commercial entities that work together to setup a system
which strongly encourages others to follow particular practices does 
indeed need to worry about Matt Petach's list of statutes, and exercise 
extreme care in its processes less the result be more about some form of 
market control and less about common good management. Net result is that 
management of a common good without some form of government involvement 
quickly gets quite challenging.  

If we had truly global operational best practices for the Internet (ones 
that went through a fairly well-defined policy development process which 
included multiple operator forums from around the globe) then you might
have a solid chance of producing output which avoided the various anti-
competitive aspects, and yet were a reasonable basis for governments to 
then step up and indicate should be required for ISPs in their operations.
It wouldn't take very many governments asking "How do I reduce SPAM and 
DDoS attacks?" and hearing back "Please require ISPs to adhere to this
Best Common Operating Practice Foo-01" before it became common practice.

FYI,
/John

Re: Open Resolver Problems

[Nick Hilliard]

On 26/03/2013 14:21, valdis.kletni...@vt.edu wrote:
> And if you get a recursive lookup for www.ebay.com from a hotel network,

I'm struggling to understand why it's necessary to hard-code dns servers
into the ip networking configuration of a portable device.  By definition,
these devices will already have dhcp enabled.

Nick