Re: [c-nsp] ACL to block udp/0?

2023-12-06 Thread Dobbins, Roland via cisco-nsp

On Dec 6, 2023, at 17:46, Gert Doering  wrote:

I'd argue that the DNS folks recommend using EDNS0 with 1232 bytes, which
works just fine to avoid fragments...

Of course, the last true Internet flag day was in 1994, flag days aren’t 
possible anymore, & this is far from universally implemented. ;>

I know you know this, just stating it for the record. Concur 100% otherwise, of 
course.




Roland Dobbins 

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] ACL to block udp/0?

2023-12-06 Thread Dobbins, Roland via cisco-nsp


On Dec 6, 2023, at 04:45, Gert Doering via cisco-nsp 
 wrote:

deny ipv4 any any fragments

This is approach is generally contraindicated, as it tends to break EDNS0, & 
DNSSEC along with it.

If the target is a broadband access network, you can use flow telemetry to 
measure normal rates of non-initial fragments destined for it (said rates are 
generally minimal). You can then implements a QoS policy to police down 
non-initial fragments in excess of the rate you’ve decided upon, ensuring that 
you leave some headroom for normal variations in traffic rates.

It would be a good idea to exempt the well-known, well-run open resolvers like 
Google DNS, Quad9, OpenDNS, et. al. from this policy, as well as your own 
on-net resolvers.

If the target is a downstream transit customer, something sitting in an IDC, 
etc., more research & nuance in terms of tACLs, policies, & rates is likely 
necessary.



Roland Dobbins 

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Netflow vs SNMP

2023-10-02 Thread Dobbins, Roland via cisco-nsp



On 2 Oct 2023, at 17:10, Hank Nussbacher 
mailto:h...@interall.co.il>> wrote:

cache timeout inactive 15
Kentik recommends 15s:

This is an old, out-of-date recommendation from Cisco should be retired.

5s is plenty of time for inactive flows.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Netflow vs SNMP

2023-10-02 Thread Dobbins, Roland via cisco-nsp



On 2 Oct 2023, at 13:13, Hank Nussbacher via cisco-nsp 
mailto:cisco-nsp@puck.nether.net>> wrote:

Does this make sense to go 1:1 which will only increase the number of Netflow 
record to export?  Everyone that does 1:1000 or 1:1 sampling, do you also 
seen a discrepancy between Netflow stats vs SNMP stats?

No and no.

Ensure that the active flow timer is set to 60s, that the inactive flow timer 
is set to 5s, and that the NetFlow capture/analysis system is configured with 
those values.

For SNMP, ensure that the counter tabulation values are set to 60s/1m, and that 
the SNMP polling/analysis system is configured with those values.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Nexus Architecture question

2021-06-02 Thread Dobbins, Roland



> On Jun 2, 2021, at 20:46, Drew Weaver  wrote:
> 
> The reason I am asking is because I've noticed that no matter what I do I 
> cannot seem to "close" the BGP port by using CoPP.

iACLs can accomplish the goal, yes?

---
roland.dobb...@netscout.com
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] tcp intercept on IOS-XE?

2021-03-15 Thread Dobbins, Roland


> On 15 Mar 2021, at 14:08, Bryan Holloway  wrote:
> 
> Care to elaborate?

Under any kind of load, it tends to send the RP up through 100%, which causes 
routing adjacencies to be lost.

I tried to get this misfeature deprecated when I was at Cisco; sadly, marketing 
pressure kept it in the software, even though it’s iatrogenic in nature.


Roland Dobbins 



___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] tcp intercept on IOS-XE?

2021-03-15 Thread Dobbins, Roland



On Mar 14, 2021, at 14:10, h...@interall.co.il wrote:

We are trying to implement tcp intercept on some brand new ASR1009x running 
IOS-XE 16.12.5 yet nothing is seen (sometimes).

TCP Intercept is a self-DoS waiting to happen.  Strongly suggest not doing this.




Roland Dobbins 
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Netflow/Sflow for "irrelevant" traffic?

2020-07-30 Thread Dobbins, Roland


> On 30 Jul 2020, at 18:48, Drew Weaver  wrote:
> 
> I'm just curious mostly but has anyone found a platform that has high enough 
> sflow/netflow "resolution" that it picks up things like single pings, or 
> broadcast traffic, or other very low volume traffic?

I think what you’re looking for is gear which supports 1:1 flow telemetry at 
the interface speeds/densities you require.


Roland Dobbins 



___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] UDP/0 ACL IOSXR issue?

2019-02-08 Thread Dobbins, Roland


On 9 Feb 2019, at 3:02, Bryan Holloway wrote:

> I suspect you are right. Saku made the same suggestion off-line.

Concur that these are likely non-initial fragments.  Don't just block 
all non-initial fragments willy-nill, or you'll break EDNS0.

If the targeted networks are endpoint networks within your span of 
administrative control, or endpoint networks of your direct 
end-customers, consider using flow telemetry analysis to profile the 
rates of non-initial UDP fragments normally seen destined for those 
networks.  You can add some headroom, and then use QoS at your edge to 
police down the non-initial fragments to a relatively low rate; this 
won't break anything during normal operations, and will eat a 
considerable amount of attack volume from UDP reflection/amplification 
attacks which generate non-initial fragments.

Be sure to exempt your own (and customers') recursive DNS farms from 
this policy, as well as well-known/well-run open DNS recursors such as 
Google DNS, OpenDNS, CloudFlare, et. al.

And be sure to exempt traffic that's just traversing your network on its 
way to some topologically-distant downstream network with which you have 
no direct relationship, as well.

QPPB can be used to propagate these polices if you've a significant 
number of peering/transit edge routers.


Roland Dobbins 
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IKEv2 unknown connections

2019-01-03 Thread Dobbins, Roland
On 3 Jan 2019, at 16:58, Robert Hass wrote:

> How I can check which IP is trying constantly connect via IKEv2 to my 
> router ?

Use flow telemetry to look for incoming traffic directed to your router 
on UDP/4500?

You could also use a classification ACL.  Or if your circumstances 
permit, just use an iACL to deny this traffic from all but designated 
IPs.


Roland Dobbins 
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] ACL TCAM LOU exhaustion on 7600 running 15.1 code

2014-05-05 Thread Dobbins, Roland

On May 6, 2014, at 6:25 AM, Mack McBride mack.mcbr...@viawest.com wrote:

 One other note is that the acl compiler will attempt to expand acls for range 
 commands provided there aren't
 too many ports in the range.  This can cause TCAM exhaustion rather than LOU 
 exhaustion.

sh fm sum has been helpful to me in this scenario in the past . . .

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Cisco to support flow spec?

2014-05-04 Thread Dobbins, Roland

On May 4, 2014, at 1:26 PM, Justin M. Streiner strei...@cluebyfour.org wrote:

 I remember hearing a while ago that Cisco was working on a competing 
 technology to flowspec, but no details were available at that time.

You're thinking of this, which they didn't implement properly, and eventually 
EoLed because they'd made a dog's breakfast of it:

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_data_tidp_tms/configuration/12-4t/sec-tidp-tms-12-4t-book/sec-tidp-tms-eol.html

They're now (finally!) implementing RFC5575 flowspec.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] ASA 5520 icmp error inspection not functioning after upgrade

2014-05-04 Thread Dobbins, Roland

On May 4, 2014, at 11:16 AM, vinny_abe...@dell.com wrote:

 I've always allowed echo-reply in the outside interface as well as 
 ttl-exceeded in the access-list applied to it.

You should also allow ICMP type-3/code-4, or you're breaking PMTU-D.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IP Options Drop

2014-04-21 Thread Dobbins, Roland

On Apr 21, 2014, at 4:37 PM, Saku Ytti s...@ytti.fi wrote:

 It's RP only, it's cross-platform feature. That is RP receives IP options 
 like it normally does, but will always drop them.

Does Sup2T/DFC4 drop options on the linecard?  How about ip options ignore?

Note to OP:  traceroute uses options . . . . you're far better off 
rate-limiting the punted packets than dropping them.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IP Options Drop

2014-04-21 Thread Dobbins, Roland

On Apr 21, 2014, at 5:47 PM, Saku Ytti s...@ytti.fi wrote:

 While some traceroute programs do support IP options, it's very rare for 
 people to use IP options while traceroute.

Network engineers tend to use traceroute in this manner more than most . . .

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IP Options Drop

2014-04-21 Thread Dobbins, Roland

On Apr 21, 2014, at 11:26 PM, Saku Ytti s...@ytti.fi wrote:

 As ACL match work, you could do it in iACL and then you're only left with own 
 customers attacking you.

iACLs should be applied at all edges of the network, including customer 
aggregation edges, IDC distribution edges, et. al.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IP Options Drop

2014-04-21 Thread Dobbins, Roland

On Apr 22, 2014, at 1:28 AM, Phil Mayers p.may...@imperial.ac.uk wrote:

 I'm wondering if other people stop because of ACL sizes too, if you have 
 large numbers of interfaces in non-aggregatable ranges e.g. customer-owned 
 space?

This is why having a rationalized address plan is important for security 
purposes; also, customers should use the operator's space for p2p transit links.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] IP Options Drop

2014-04-18 Thread Dobbins, Roland

On Apr 18, 2014, at 11:40 PM, Robert Williams rob...@custodiandc.com wrote:

 The lab kit is running 15.1(2)SY1 in the tests shown above.

What Sup/linecards?

Are you sure your Sup/linecards support evaluating options as a classifier?  If 
they don't, then even though the options keyword shows up in the ACL, it's just 
textual, and you're actually doing a deny ip any any.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] management access BCP?

2014-03-27 Thread Dobbins, Roland

On Mar 14, 2014, at 9:10 PM, Gustav UHLANDER gustav.ulan...@steria.se wrote:

 What we've done is find some other ISP in the same colo that we knew from 
 some common event, and just throw two cat5 cables over the wall - one has a 
 /29 from their IP space, one has a /29 from our IP space, and we both get 
 nicely independent OOB-over-IP.
 
 Of course you might want to connect that to a device that is seriously 
 hardened, and such :-) - not to a speaks telnet-only and has direct access 
 to all your consoles boxes.

+1

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] BGP session going down during DDOS

2014-03-09 Thread Dobbins, Roland

On Mar 10, 2014, at 2:41 AM, redscorpion69 redscorpio...@gmail.com wrote:

 Filters don't allow BGP sessions to our PE router.

You might want to double-check that your iACLs are up-to-date, that you've 
enabled GTSM, that you've enabled CoPP, etc.

What make/model/OS/train/revision/linecard?

 By the way, what IS the best way to defend against this huge amount of 
 traffic? You can't really place policers at the edge of the network, it's
 cumbersome and prone to errors.

Sure you can.  Police down UDP/123 traffic which isn't 76 bytes in size down to 
a 1mb/sec aggregate or thereabouts, or UDP/123 traffic which is greater than 
400 bytes in size down to a 1mb/sec aggregate, or thereabouts.

I prefer the former, but YMMV.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] BGP session going down during DDOS

2014-03-06 Thread Dobbins, Roland

On Mar 7, 2014, at 2:07 AM, redscorpion69 redscorpio...@gmail.com wrote:

 How to make sure this doesn't happen again?

Are you sure the router wasn't attacked directly?  Have you implemented iACLs 
to keep unauthorized traffic off your routers?

Maybe the CE router isn't properly protected and went down, or was simply 
overwhelmed due to traffic volume?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Shapping NTP traffic on 6500/7600

2014-02-27 Thread Dobbins, Roland

On Feb 27, 2014, at 9:50 PM, Brian Turnbow b.turn...@twt.it wrote:

 There is more info here on it

Does that still apply to Sup2T/DFC4?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Shapping NTP traffic on 6500/7600

2014-02-27 Thread Dobbins, Roland

On Feb 27, 2014, at 9:54 PM, Dobbins, Roland rdobb...@arbor.net wrote:

 Does that still apply to Sup2T/DFC4?

Another way to deal with this is to QoS down all non-76-byte UDP/123 traffic 
(i.e., non-timesync traffic) to something like 1mb/sec in aggregate.

Alternatively, someone else suggested policing down all 468-byte UDP/123 
traffic, which is generally the monlist output which is what's seen on the 
reflector/amplifier - target leg.  I prefer the 76-byte match as being more 
comprehensive, but YMMV.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Shapping NTP traffic on 6500/7600

2014-02-27 Thread Dobbins, Roland

On Feb 28, 2014, at 2:51 AM, Randy a...@djlab.com wrote:

 If the primary DDOS payload is non-initial fragments (which I suspect may be 
 the case) it will bypass your ACL unless you match fragments, which may 
 impact other traffic.

Actually, with ntp, it isn't - ntp handles message segmentation on its own at 
layer-7, generating multiple packets for long replies.

Unlike DNS, SNMP, chargen, and other UDP reflection/amplification attacks, we 
don't see non-initial fragments with ntp reflection/amplification attacks.

But it's a good point to keep in mind when dealing with those other attack 
techniques.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Shapping NTP traffic on 6500/7600

2014-02-26 Thread Dobbins, Roland

On Feb 27, 2014, at 6:41 AM, Thomas St-Pierre tstpie...@iweb.com wrote:

  Normal traffic shouldn’t be affected,

It will be crowded out during an attack.

I don't know if you've the ability to match on packet size or not in hardware 
for QoS - if so, UDP/123 packets which *aren't* 76 bytes in length is a good 
classifier, as it leaves timesync ntp traffic alone and squelches everything 
else.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-18 Thread Dobbins, Roland

On Feb 18, 2014, at 2:15 PM, Aaron aar...@gvtc.com wrote:

 Usually nfsen seems to be pretty accurate, is there a reason for that ~40 
 gbps reading during that ntp attack ?

Have you set your active flow timer to 60s/1m, so as to avoid backlogged spikes?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-18 Thread Dobbins, Roland

On Feb 18, 2014, at 5:48 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 AFAIK nfdump uses the start/end time in the flows to calculate pps, so would 
 this matter? Or is it a result of the sampling maths?

It has to do with long flows - flows aren't exported from the router/switch 
until they're terminated.  Be sure your active flow timer is set to 1m/60s, and 
your inactive flow timer set to 5s.

Otherwise, you'll have all these false peaks and valleys from your stats being 
backlogged up to 30m, which is the default for the active flow timer.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-18 Thread Dobbins, Roland

On Feb 18, 2014, at 6:20 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 Aaron reported his netflow was reporting too-high spikes. How would shorter 
 flow timeouts - which equals high-frequency reporting bins/windows at the 
 collector - result in *lower* pps counters?

Because the spikes may be artificial, artifacts of backlogged stats.

 I can only see this being the case of the collector doesn't honour start/end 
 times, and does something dumb like assuming end time == reception time. 
 AFAIK that's not the case with nfdump.

Some do, some don't.  Getting your stats 30m late isn't helpful, in any event.

I've seen artificial spikes in bps/pps take place many, many times due to 
active flow timer misconfiguration, that's why I suggested checking the active 
flow timer.  YMMV.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Change clock, netflow doesn't change

2014-02-17 Thread Dobbins, Roland

On Feb 18, 2014, at 12:05 AM, Joe Loiacono jloia...@csc.com wrote:

 The netflow exports stayed on local time. 

Are you sure it isn't an artifact of your collection/analysis software ignoring 
the flow start/flow end fields in the telemetry export?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-12 Thread Dobbins, Roland

On Feb 12, 2014, at 11:07 PM, Richard Clayton sledge...@gmail.com wrote:

 What is this type of DDoS called?

An ntp reflection/amplification DDoS attack.

 Is the the customer being individually targeted or just the expolitable NTP 
 server?

It sounds as if these are ntpds which are misconfigured and allow level-6/-7 
commands such as monlist to be issued, which produces a significant 
amplification.  The attackers are spoofing the source IPs of their targets, and 
the ntpds 'reply' with unsolicited large, fragmented UDP ntp 'responses'.

Check Jared's compendium for abusable ntpds on your netblocks and those of your 
customers:

http://www.openntpproject.org/

 Are these caused by bots or manually by individuals?

Bots being driven by individuals (when we get to the point where the bots make 
their own targeting decisions for DDoS attacks, things will be interesting, 
indeed, heh).

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-12 Thread Dobbins, Roland

On Feb 13, 2014, at 12:17 AM, SilverTip257 silvertip...@gmail.com wrote:

 I've seen these NTP attacks called DRDoS attacks.
 ( Distributed Reflection Denial of Service )

This is a marketing term used by some commercial organizations to try and 
pretend that these attacks are something new and different (they aren't).  
Security professionals don't use this term.

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-12 Thread Dobbins, Roland

On Feb 13, 2014, at 2:10 AM, Joe Loiacono jloia...@csc.com wrote:

 This is port 123 exclusively going and coming right? 

No - the reflector/amplifier - target leg will be sourced from UDP/123, but 
will be destined for the port of the attackers' choice.  We see a lot of 
UDP/123 - UDP/80, UDP/123 - UDP/123, and UDP/123 - UDP/foo.  

UDP/80 is probably the most popular, but I just got off a call where it was in 
fact all UDP/123 - UDP/123 as you indicated.

Also, that's just for the initial fragments.  The bulk of the traffic on the 
reflector/amplifier - target leg is non-initial UDP fragments.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] NTP DDoS

2014-02-11 Thread Dobbins, Roland

On Feb 12, 2014, at 6:46 AM, omar parihuana omar.parihu...@gmail.com wrote:

 I've just put an ACL in order to block NTP outbound traffic.

You should look at the ntp sources, find out which allow monlist, et. al. (see 
http://www.openntpproject.org/), then work to remediate those specific ntpds. 
 Blocking ntp traffic wholesale is something which might make sense in an 
emergency as you describe, for a  brief time, but which shouldn't be done any 
longer than is absolutely necessary.

btw, you don't need NBAR to detect/classify this traffic - regular NetFlow will 
do.  NBAR eats up a lot more resources on your box.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T netflow problems

2014-02-05 Thread Dobbins, Roland

On Feb 5, 2014, at 6:58 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 That can't be. Sup2T has operationally useful netflow. I read it 
 somewhere...

That means it isn't broken by design, as in pre-EARL8 Sups  DFCs.  It doesn't 
mean there can't be bugs . . .

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T netflow problems

2014-02-05 Thread Dobbins, Roland

On Feb 5, 2014, at 7:34 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 If that's the case, do please spell them out.

Real sampling, thereby avoiding mls table overflow, with the resultant 
nondeterministic skew of statistics; seeing stats on dropped traffic; getting 
TCP flags so as to classify things like SYN-floods; and so forth.

Things pre-EARL8 Sups/DFCs simply didn't support, but which are vital for even 
the most basic traffic engineering, peering analysis, and troubleshooting 
purposes, much less security.

NetFlow on pre-EARL8 Sups/DFCs was a disaster; many folks who thought they were 
getting accurate statistics weren't due to mls table overflow, and all the 
above caveats regarding basic functionality really made it not very useful.

I can't tell you the number of folks I've talked to for the last 13 years or so 
who thought their own NetFlow stats were fine from EARL6 and EARL7, until they 
realized they weren't getting what they thought they were getting, and they 
couldn't even trust that due to mls table overflow.  I'm glad you were happy 
with EARL7 NetFlow, but that's a minority view, in my experience (both when I 
worked for Cisco and afterwards).

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T 15.1(SY1) inline packet traffic crash

2014-02-05 Thread Dobbins, Roland

On Feb 6, 2014, at 3:23 AM, Charles Spurgeon c.spurg...@austin.utexas.edu 
wrote:

 TAC advised a reload on 15.1(2)SY1, which we have done.

Hopefully, this has been reported to PSIRT?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Cisco 6503 Sup2T Engine block outbound TCP or UDP Port traffic

2014-02-01 Thread Dobbins, Roland

On Feb 2, 2014, at 11:28 AM, Joseph Hardeman jwharde...@gmail.com wrote:

 I know how to NULL route IP 's but I don't know if there is a way to block or 
 deny traffic based on destination port's also based on IP ranges.

http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a00800c9470.shtml

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] default route for internet

2014-01-24 Thread Dobbins, Roland

On Jan 25, 2014, at 2:11 AM, Michael Sprouffske msprouff...@yahoo.com wrote:

 My primary site is advertising default-originate so all my other sites can 
 get to the internet.  How would I go about advertising 2 internet circuits in 
 the event my primary site lost internet?

Why don't you originate default from a point in your iBGP mesh which has full 
tables?  That way, you're active/active, instead of having a 'failover' 
scenario.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] EARL_L2_ASIC-SP-4-DBUS_HDR_ERR

2014-01-15 Thread Dobbins, Roland

On Jan 15, 2014, at 10:25 PM, Sukumar Subburayan (sukumars) 
sukum...@cisco.com wrote:

 Error in the DBUS (data bus) header indicates that you have had hardware.

Or that you need to re-seat the linecard(s)/RP(s) in question . . .

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-03 Thread Dobbins, Roland

On Jan 3, 2014, at 4:15 PM, Eugeniu Patrascu eu...@imacandi.net wrote:

 Maybe I should try this again: what I said was that on the recursive 
 resolvers dedicated for your clients you can add an extra layer of protection 
 in terms of dropping fake responses targeted at those servers by the means of 
 a local firewall setup on each box, not on a gateway like box.

I understand; this is a Very Bad Idea, for the reasons mentioned previously.

 What I'm arguing against is the idea of rate limiting a service just because 
 it might be attacked and have your customers play the lottery with their 
 queries and try again if their packets are lost due to rate limiting.

RRL is intended for authoritative servers, absolutely.  There are other 
mechanisms which can be used to protect recursive servers from abuse from the 
customer side, starting with S/RTBH.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-03 Thread Dobbins, Roland

On Jan 3, 2014, at 6:57 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 It would be interesting to see some real-world numbers on this, to see if it 
 is a win or not. As I say, my gut says no, but gut != proof ;o)

I've seen it melt enough times in real life that I get a sinking feeling in my 
gut every time someone mentions it.

;

 It seems to me that any system which can detect bad replies would be better 
 thresholding them into short-lived stateless blocks, rather than short-lived 
 stateful.

This matches my experience, concur 100%.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-02 Thread Dobbins, Roland

On Jan 2, 2014, at 4:22 PM, Eugeniu Patrascu eu...@imacandi.net wrote:

 FWIW, if you run your customers recursive resolvers on a Linux/*BSD box you 
 can setup iptables/pf in such a way that you only allow queries from 
 customers and from your resolver to the internet in a stateful way and deny 
 unrelated incoming responses and still have the same performance levels.

Until someone DDoSes the box from one end or the other, taking down both 
authoritative service and recursive service at one fell swoop.

That's one of the many reasons one's DNS ought to look something like this:

https://app.box.com/s/72bccbac1636714eb611

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-02 Thread Dobbins, Roland

On Jan 2, 2014, at 8:09 PM, Gert Doering g...@greenie.muc.de wrote:

 I would strongly recommend *against* doing stateful anything in front of a 
 DNS server.  It won't serve a useful function (as unbound etc. are
 quite good in recognizing real responses vs. fake), but serves as an 
 additional chokepoint which might run into overload far before your
 servers die.

Concur 100%:

https://app.box.com/s/a3oqqlgwe15j8svojvzl

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2014-01-02 Thread Dobbins, Roland

On Jan 3, 2014, at 12:17 AM, Mack McBride mack.mcbr...@viawest.com wrote:

 The big problem with DDoS is pipe filling anyway, not CPU load.

That's entirely subjective and varies from attack to attack, FYI.

And to be carify, the issue with putting stateful anything in front of servers 
isn't primarily CPU load (although it certainly can be a factor), but rather 
state-table memory exhaustion.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-02 Thread Dobbins, Roland

On Jan 3, 2014, at 12:32 AM, Eugeniu Patrascu eu...@imacandi.net wrote:

 With modern machines (from a few years back) you can track a lot of 
 connections effortlessly.

I think you don't understand the scale of even small DDoS attacks in terms of 
state-tracking.

Stateful devices put in front of servers which are then DDoSed go down, taking 
down everything behind those stateful devices.  I've seen 3mb/sec of spoofed 
SYN-flood take down a 20gb/sec stateful firewall; I've seen 10kpps of HOIC take 
down a 10gb/sec load-balancer.

This isn't theoretical or speculative.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2014-01-01 Thread Dobbins, Roland

On Jan 1, 2014, at 7:21 PM, Gert Doering g...@greenie.muc.de wrote:

 Which is why Paul Vixie's RRL or Lutz Donnerhacke's dampening patches for 
 BIND exist.

Yes, hence 'not all; directly-spoofed ANY attacks and the like, which don't 
involve open recursors, are the exception'.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2014-01-01 Thread Dobbins, Roland

On Jan 1, 2014, at 7:27 PM, Gert Doering g...@greenie.muc.de wrote:

 Abusing authoritatives is not the exception, and has not been for over a 
 year.

It is 'the exception' in the context of my previous reply, which had to do with 
abuse of open recursors.  Agree 100% it isn't rare; that wasn't what I meant, 
apologies for being unclear.

Direct abuse of authoritative servers using spoofed ANY queries and other 
spoofed queries intended to generate large responses goes back many years, 
absolutely.  But we still see lots of attacks utilizing open recursors; direct 
abuse of authoritative servers hasn't superseded or eliminated the use of open 
recursors, attacks leveraging open recursors take place every day, as you know.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2013-12-31 Thread Dobbins, Roland

On Jan 1, 2014, at 4:13 AM, Mack McBride mack.mcbr...@viawest.com wrote:

 Recursive servers have to be able to receive responses from anywhere on the 
 internet.

Hence 'external resolvers', mentioned in my post.

https://app.box.com/s/72bccbac1636714eb611

 Nor can RTBH stop a true DDoS.

S/RTBH can, up to the point that the number of sources becomes unmanageable.  
Hence, 'other mechanisms', mentioned in my post.

  That is the 'distributed' part that is the first D. Nor will it stop a 
 reflection attack, which is even more damaging because then you are blocking 
 important authoritative DNS servers.

Again, hence 'other mechanisms', mentioned in my post.

Also, if you're on the designated-target leg of a DNS reflection/amplification 
attack, in most (not all; directly-spoofed ANY attacks and the like, which 
don't involve open recursors, are the exception) cases, you're receiving 
traffic from open recursors, not authoritative severs, and the sources you end 
up blocking are open recursors, not authoritative servers.

If your external resolvers are open recursors and are being abused, then you 
need to remediate them.

 As an ISP operator, I can tell you that your solution will only work for 
 someone whose customers can't leave for another provider.

As someone who's worked with many ISPs to successfully mitigated many extremely 
large-scale and complex DNS-related DDoS attacks, including 100gb/sec+ 
reflection/amplification attacks, I can assure you that I do in fact understand 
the issues involved and how to deal with them.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-31 Thread Dobbins, Roland

On Jan 1, 2014, at 5:26 AM, Mack McBride mack.mcbr...@viawest.com wrote:

 Why would a company that requires maximum uptime want to depend on our DNS 
 servers when they have bandwidth from a number of other companies?

The recommendation in question is intended for consumer broadband access 
operators only, not for transit operators.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-30 Thread Dobbins, Roland

On Dec 31, 2013, at 2:19 AM, Mike mike-cisconspl...@tiedyenetworks.com wrote:

 Not true. I've seen more than 600mbps of traffic and, while not in the league 
 of what you see, is still a sizable total of my transit and we kept chunking 
 along.

This is a pretty trivial amount of traffic; also, note that attacking the boxes 
directly will likely yield results pretty easily, as well.  

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-30 Thread Dobbins, Roland

On Dec 31, 2013, at 1:27 AM, Mack McBride mack.mcbr...@viawest.com wrote:

 Phishing has little to do with DNS per se.

Some does, actually.

 BUT, forcing customers to use your DNS results in the possibility of all of 
 your customers suffering in a DDoS situation where your DNS servers are 
 targeted.

If your first-line recursive DNS servers are configured correctly, then they 
can't be DDoSed directly from outside your network, and it's easy enough to 
squelch attacks originating from within your network via S/RTBH or other 
mitigation mechanisms.  There are mitigation mechanisms to protect the upper 
tier of external resolvers which feed the first-tier resolvers, as well.

What part of allowing Google DNS and OpenDNS by default wasn't clear?

Also, note that policies can be altered, if circumstances warrant.  But any 
network operator which doesn't have the capability defend its own recursive DNS 
servers from DDoS attacks should take steps to implement S/RTBH, et. al.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-29 Thread Dobbins, Roland

On Dec 29, 2013, at 2:00 AM, MIke mike-cisconspl...@tiedyenetworks.com wrote:

 Open internet. I don't want to dictate to anyone which port numbers or 
 protocols they are limited in using, and I want to impose only the absolute 
 minimum of controls in order to deliver as much of an unfiltered / 
 unrestricted service as I can.

Causing users to use your recursors by default, plus Open DNS and Google DNS, 
and with an opt-out proviso, does not in any way inhibit their ability to 
access the Internet, while doing so materially contributes to the security of 
your user base.

 That may be a wonderful design goal in theory, but our 'transit edge' as you 
 call it, is a pair of linux boxen that do not have any effective interface 
 for implementing policies or mitigation tools. 

In that case, unfortunately, nothing you do is going to matter, anyways, as 
even a very small DDoS attack can take those boxes down completely.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-29 Thread Dobbins, Roland

On Dec 29, 2013, at 5:18 PM, Gert Doering g...@greenie.muc.de wrote:

 I might be a bit extreme on this, but I highly value the end-to-end 
 communication nature of the Internet,

Again, causing users to utilize your recursors by default, plus Open DNS and 
Google DNS, and with an opt-out proviso for 'advanced' users, does not in any 
way inhibit their ability to access the Internet, while implementing such a 
policy materially contributes to the security of your user base.

I used to dread the day that a user would end up successfully suing a consumer 
broadband network operator due to a compromise which could've been avoided by 
implementing sensible, non-intrusive policies such as this one, as I thought 
that such a verdict would likely set a very bad precedent (literally) and do 
far more harm than good.  More and more, though, I'm coming around to the view 
that some sizable damage awards are the only thing that will motivate consumer 
broadband network operators to use common sense and stop throwing up specious 
objections to entirely reasonable, lightweight default policies which do not in 
any way, shape, form, or fashion impact 'the end-to-end communication nature of 
the Internet', yet which provide welcome protections for average consumer users 
while at the same time not limiting more 'advanced' users.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2013-12-29 Thread Dobbins, Roland

On Dec 29, 2013, at 7:21 PM, Gert Doering g...@greenie.muc.de wrote:

 And that is where we differ.  You find it OK to limit the protocol du jour to 
 what users do not need, I do not agree to it.  Even if I agree with
 you that most users would not notice.

I'm not proposing blocking DNS.  I'm proposing a default policy for consumer 
broadband users which assumes that they'll use the DNS recursors provided by 
the broadband network operator, unless the use chooses to opt-out.

 in reasonable countries, ISPs are protected from charges for traffic they 
 transport *unless* they start messing with it - if you start filtering 
 traffic for protocol X, but leave through the evil packets for protocol 
 Z, you're *way* more likely to be made liable for it.

Again, this isn't the same thing.  Nobody's talking about blocking the DNS.

Here's the risk that I see for network operators, moving forward, if they don't 
implement sensible, low-impact default (with the ability to opt-out, which 
would include indemnification) policies of this nature to protect their user 
bases:

1.  Consumer user X ends up getting phished/compromised, attacker empties 
his bank account, maxes his credit cards, applies for new credit cards in the 
user's name but delivered to another mailing address under the control of the 
attacker or his minions, etc.

2.  User X ends up suing the bank(s) and credit card issuer(s) in question, 
alleging that those entities didn't take reasonable security precautions, and 
are now liable for all the actual and punitive damages claimed by user X as he 
struggles to get his money back, clear his credit history, etc.

3.  Liability insurance companies for the bank(s) and credit card issuer(s) 
in question turn around and sue the network operator for damages based upon 
negligence, alleging that reasonable and practical security policies which 
could've potentially prevented this fraud from being possible weren't 
implemented.  They might sue software vendors - OS vendors, foundations 
providing open-source Web browsers, and so forth, as well.

4.  Politicians/regulators get wind of this, and pile on.

A little bit of prudence now could obviate a whole lot of financial hurt and 
heavy-handed legislation/regulation, later.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2013-12-29 Thread Dobbins, Roland

On Dec 29, 2013, at 7:47 PM, Mark Tinka mark.ti...@seacom.mu wrote:

 The majority of (phishing) attacks have nothing to do with the network, with 
 the exception of having the network transport those packets to the user's 
 computing device.

Yes, but those that do, which replace the user-configured DNS settings with DNS 
settings of the attacker's choice, not to mention the possibility of 
cache-poisoning of poorly-maintained random DNS recursors on the Internet, 
would apply.

Also, I haven't even touched on availability - the whole open DNS recursor 
problem, and various remedies for it, which can and should include default 
policies for consumer broadband network operators which include anti-spoofing 
as close to the customer as possible as well as only allowing users who request 
it the ability to send DNS queries to DNS servers elsewhere on the Internet.

 Do they now sue Apple or Samsung for not detecting the  spurious e-mail? Do 
 they sue Google for not including  protection within Android? Do they sue 
 Dell for manufacturing and selling bundled hardware/software without adequate 
 protection? Do they sue the regulator for not enacting (and enforcing) policy 
 that protects the end user?  Do they sue Cisco, Juniper, ALU, Huawei, e.t.c., 
 for not providing protection in their network-based devices? 

My main concern in this particular discussion is with attacks which depend upon 
perturbing the intended destinations of specific traffic, because that's the 
primary risk to network operators.

Yes, I believe all the examples you cite are coming, although the network 
infrastructure vendors can point to features which they do in fact provide, but 
which some operators do not enable, out of ignorance, apathy, or a desire to 
avoid opex.  

There's simply too much money and too much advantage for politicians/regulators 
for the present state of affairs to continue much longer.  The imposiiton of 
sales taxes on goods/services procured via the Internet are a good indicator of 
the general trend.

 If we start down this path, at what point are we satisfied that the customer 
 is reasonably protected 
 from all possible attack vectors?

Network operators should concern themselves with network traffic destination 
manipulation in the network infrastructure and with ancillary supporting 
services they offer which affect same - namely, DNS - and with availability.

 How do customers and operators delineate lines between which responsibility 
 lies in view of those attack vectors?

Assets and services under the control of and offered by network operators are 
quite clearly the responsibility of network operators.   That's my primary 
concern; the software developers/vendors are another matter, entirely.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2013-12-27 Thread Dobbins, Roland

On Dec 27, 2013, at 10:55 AM, Mike mike-cisconspl...@tiedyenetworks.com wrote:

Can anyone suggest how we might tighten this up and either have a seperate 
 rate limit list or somehow exclude my small list of resolver IP's from the 
 above limiting?

Using any QoS mechanism, let alone an old, obsolete, unmaintained one like CAR, 
to deal with DDoS isn't a good idea - programmatically-generated attack traffic 
can 'crowd out' legitimate traffic.  

Why are you allowing DNS responses from outside your network to your 
subscribers at all, excepting Google DNS, OpenDNS, and anything specifically 
arranged for specific customers (the assumption is that you're running a 
consumer broadband access network)?

Also, you should have sufficient layer-3 hierarchy in your network to have the 
ability deploy policies/mitigation tools at your transit edge which do not 
affect your customer aggregation edge, and vice versa.  If you don't currently 
have separation of these topological roles, then implementing same should be a 
priority.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] rate limit dns

2013-12-27 Thread Dobbins, Roland

On Dec 27, 2013, at 4:50 PM, Gert Doering g...@greenie.muc.de wrote:

 I'd terminate my contract if my ISP would take away the ability to query 
 foreign DNS servers (usually done to troubleshoot things), to run 
 traceroutes, to ping stuff, etc.

Neither you nor I are typical broadband access customers; the overwhelming 
majority of broadband access customers have no need to use DNS servers beyond 
the recursive DNS servers provided by their ISPs and/or Google DNS or OpenDNS, 
and in fact are exposed to danger in the form of various malware which changes 
the recursive DNS settings on their computers by unfettered DNS access.  
Unrestricted recursive DNS access is in fact inimical to the overwhelming 
majority of users.

Exceptions should always be granted for 'advanced' users who want to utilize 
DNS servers outside their broadband operator's own network, and these cases can 
be accommodated in a scalable manner via automation; but the overall security 
posture of the Internet as a whole and of any given ISP's broadband users 
specifically would be greatly increased if the default policy were to limit 
recursive DNS service to the DNS recursors provided by the broadband operator 
and a few reputable services like Google and OpenDNS.

Another side-effect would be to somewhat ameliorate the effect of DNS 
reflection/amplification attacks.  Broadband operators would still need to work 
with their transits/peers to 'push back' the attack traffic, but this would 
still be better than trying to use ill-suited QoS mechanisms to deal with it.

The bit about traceroutes and pings is a red herring, of course.  I'm not 
suggesting that they ought to be restricted, apart from aggregate policing of 
the relevant 'inbound' traffic in order to minimize the effects of abuse.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] rate limit dns

2013-12-27 Thread Dobbins, Roland

On Dec 27, 2013, at 7:00 PM, Peter Rathlev pe...@rathlev.dk wrote:

 Most people on this list might not be typical access customers -- they might 
 be running their own resolver to get proper DNSSEC -- but that
 still doesn't make it okay for an ISP to do things most of their customers 
 wouldn't notice

Of course it does, when those things don't inhibit the ability of customers to 
do the things they want to do.

ISPs should run their own DNSSEC-capable recursive resolvers, anyways.

The only folks who need a more open policy are those who're capable of figuring 
out what default policies are in place and of requesting to opt out.

 Think Phorm et cetera. It's a slippery slope.

No.  This is not a valid analogy, nor is it a slippery slope from implementing 
reasonable access policies to actively tampering with DNS responses.

 Though I'm not unsympathetic to your arguments, one could actually use the 
 same reason to block port 80/tcp; much malware comes in this way.

Straw man, hyperbole, not a valid analogy.

 I'm with Gert here; we don't need to further inhibit the concept of 
 end-to-end and any kind of filtering needs to be considered very
 closely.


Making the default limiting recursive DNS to locally-operated recursors plus 
OpenDNS and Google DNS, plus the option to opt out for 'advanced users' *is* 
'considered very closely'.

 Inbound filtering of DNS responses should at most be a _temporary_ measure to 
 combat a specific DoS attempt.

For consumer broadband access networks, this is a harmful and counterproductive 
stance.  People's credit is being ruined, and their ability to access Internet 
resources is being impeded.  A default policy limiting recursive DNS service to 
local recursors plus Google DNS and OpenDNS, with the option for 'advanced' 
users to opt out is perfectly reasonable, does not break the Internet, and is 
no more of a 'slippery slope' than is anti-spoofing.

Note that these reasonable, well-thought-out, default policies (with opt-out 
provisions) are mooted for endpoint networks like consumer broadband access 
networks, *not* for transit networks.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] sampled v9 netflow

2013-12-24 Thread Dobbins, Roland

On Dec 24, 2013, at 6:05 PM, Nikolay Shopik sho...@inblock.ru wrote:

 flow-sampler-map is only apply to hardware routing platforms.

Thanks for coming back and posting this - good reference info!

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T interface ACL limitations

2013-12-21 Thread Dobbins, Roland

On Dec 22, 2013, at 7:52 AM, Łukasz Bromirski luk...@bromirski.net wrote:

 ACLs are good for basic sanity checks and segmenting the traffic for ports 
 (L4+). BGP scales way better for L3 than them and it’s faster
 and way easier to dynamically update the entries.

Concur 100%.

ACLs are a network access policy enforcement tool.

S/RTBH is a DDoS reaction/mitigation tool.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Sup2T interface ACL limitations

2013-12-20 Thread Dobbins, Roland

On Dec 20, 2013, at 8:36 PM, Phil Mayers p.may...@imperial.ac.uk wrote:

 Personally, I wouldn't do what you're doing - a 100k ACE ACL is just asking 
 for trouble.

Concur 100%.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Traffic Monitoring Question

2013-12-17 Thread Dobbins, Roland

On Dec 17, 2013, at 2:36 PM, Mark Tinka mark.ti...@seacom.mu wrote:

 Flow analysis tells what kind of traffic it is, where it's coming from, where 
 it's going to.

Flow telemetry tells us what SNMP telemetry tells us, plus the above.

It's useful to have SNMP telemetry from interfaces to cross-check against flow 
telemetry, and there are many other things one can monitor via SNMP, of course. 
 But overall, flow telemetry is infinitely more useful for monitoring network 
traffic than SNMP telemetry.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Traffic Monitoring Question

2013-12-17 Thread Dobbins, Roland

On Dec 17, 2013, at 8:29 PM, Mark Tinka mark.ti...@seacom.mu wrote:

 /it's just coincidence that you work with Arbor

Whatever point you think you're making with this sort of remark, a simple 
search of the list archives will demonstrate I've consistently advocated the 
use of flow telemetry and open-source flow telemetry collection/analysis 
systems over the years, irrespective of wherever I happen to be employed at any 
given point in time.

It's all too easy to confuse cause and effect, one supposes.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Traffic Monitoring Question

2013-12-17 Thread Dobbins, Roland

On Dec 17, 2013, at 10:26 PM, Mark Tinka mark.ti...@seacom.mu wrote:

 The two are detached, in my mind, but I can see how some might not have seen 
 that way, hence...

My apologies for being so dense as to fail to understand that you meant what 
you said literally, heh (and I was a bit puzzled, given that we've known one 
another for quite some time).

;

Many thanks for hitting me with the clue-bat, heh.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] Sup2T interface ACL limitations

2013-12-16 Thread Dobbins, Roland

On Dec 16, 2013, at 9:26 PM, Rolf Hanßen n...@rhanssen.de wrote:

 Maybe it works if I use an ACL with 100k entries but it takes a minute to 
 install.

In what topological situation do you need 100K entries?  Unless you're a very 
large wholesale transit network trying to enforce anti-spoofing for downstreams 
of your downstreams, do you really need that many entries?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T interface ACL limitations

2013-12-16 Thread Dobbins, Roland

On Dec 10, 2013, at 12:38 AM, Rolf Hanßen n...@rhanssen.de wrote:

 I am thinking about dropping some (mainly ddos) traffic on the outside 
 network borders with ACLs.

ACLs don't work well as a DDoS reaction mechanism.  They're good for protecting 
your network infrastructure:

https://app.box.com/s/osk4po8ietn1zrjjmn8b

S/RTBH is much better as a DDoS reaction mechanism:

https://app.box.com/s/xznjloitly2apixr5xge

All the caveats folks have noted about ACLs hold true.  

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T interface ACL limitations

2013-12-16 Thread Dobbins, Roland

On Dec 17, 2013, at 6:33 AM, Rolf Hanßen n...@rhanssen.de wrote:

 My fear is that somebody creates blackholes in my network with spoofed source 
 IPs.

Nobody can create blackhole routes on your network than you - or else you have 
much bigger problems, heh.

This issue applies to S/RTBH or any other mitigation mechanism.  You whitelist 
things which oughtn't ever to be blocked, and then vet sources/destinations 
before blackholing them.

 I think this is a potential damage amplifier and may cause much bigger impact 
 than a flooding itself could ever do.

If you misuse it, sure.  But this is not a new technique, it's been in use for 
many years.  Nothing's perfect; the idea is to exercise caution.

 I could black/whitelist something like 8.8.8.8, but I think there is no 
 chance to build a list that will ever be sufficient for blackholing
 sources.

Other operators can do it - why can't you?

 I furthermore think I will run into problems as soon as I block anything from 
 source xy in the complete network, i.e. also for customers that do
 not want their traffic to be filtered at all.

If attack traffic is ingressing your network, then you've every right and in 
fact responsibility to mitigate it, let it cause issues for your network and 
your customers.  There are many mitigation mechanisms; S/RTBH is just one of 
them.

You can set up S/RTBH with communities, controlling where traffic is dropped - 
all edges, all peers, all customers, some edges, some peers, some customers, 
etc.

You can also build a mitigation center and divert traffic headed for attack 
targets into that mitigation center. You can enable S/RTBH on the coreward 
interfaces of the mitigation center gateway, and drop traffic via S/RTBH there 
- i.e., only for the specific target under attack.  There are other things you 
can do there, too:

http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html

None of this stuff is theoretical; they're proven techniques.  Rather than 
making the perfect the enemy of the merely good, it might be a good idea to 
assemble a toolbox of various tools, and utilize them if and as necessary.

ACLs aren't feasible for DDoS mitigation for many reasons.  S/RTBH is a very 
useful tool to have in the toolbox.  All tools must be used with caution and 
good sense, let we bruise our thumbs.

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Data Center Core Switches

2013-11-30 Thread Dobbins, Roland

On Nov 30, 2013, at 11:45 PM, madu...@gmail.com wrote:

 The above spec could apply to juniper, cisco, hp, xtreme ...etc, any
 recommendation should I add/adjust to my  TECHNICAL SPECIFICATION

It's hard for strangers who don't know your actual requirements to make 
recommendations, beyond generalities like good NetFlow support, uRPF, et. al.  
Also, there's nothing in your list related specifically to IPv6, required 
IPv6-IPv4 feature parity, and so forth.

In the general performance range you specify, maybe the Catalyst 6800 would be 
something to consider?

But you can't develop a network architecture and determine your needs based 
upon a laundry-list of general specs.  It sounds as if you should consider 
working with an experienced network architect and/or integrator to help 
determine your actual requirements, first.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Data Center Core Switches

2013-11-30 Thread Dobbins, Roland

On Dec 1, 2013, at 2:52 AM, Eugeniu Patrascu eu...@imacandi.net wrote:

 He's baiting you to do his work for him.

Concur - which is why I gently suggested that lists of specs won't cut it, that 
it's necessary to engage SMEs in order to make proper decisions.

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] sampled v9 netflow

2013-11-29 Thread Dobbins, Roland

On Nov 29, 2013, at 4:24 PM, Nikolay Shopik sho...@inblock.ru wrote:

 flow monitor IPv4
 record netflow ipv4 original-input
 exporter AS-STATS
 cache timeout active 300

 flow monitor IPv6
 record netflow ipv6 original-input
 exporter AS-STATS
 cache timeout active 300

Your active timers should be 60s, and your inactive timers should be 5s.  With 
this 5-minute active timer, all your stats will be back-loaded and not 
representative of actual traffic.

 interface GigabitEthernet0/1
 ip address 10.10.112.143 255.255.255.0
 ipv6 address 2001:db8:20:101::99/64
 ip flow monitor IPv4 input
 ipv6 flow monitor IPv6 input
 flow-sampler SM
 end

Try this on your interface:

interface GigabitEthernet0/1
ip address 10.10.112.143 255.255.255.0
ipv6 address 2001:db8:20:101::99/64
ip flow monitor IPv4 sampler SM input
ipv6 flow monitor IPv6 sampler SM input
end


---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] sampled v9 netflow

2013-11-29 Thread Dobbins, Roland

On Nov 29, 2013, at 5:49 PM, Nikolay Shopik sho...@inblock.ru wrote:

 So this is apply to any sampled netflow?

Any NetFlow, sampled or non-sampled.

Set the active timer to 60s, and the inactive timer to 5s.

 c7201(config-if)#ip flow monitor IPv4 sampler SM input
  ^
 % Invalid input detected at '^' marker.

Try this:

c7201(config-if)#ip flow monitor IPv4 sampler ?

or

c7201(config-if)#ip flow monitor IPv4 ?

and see what options it gives.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] sampled v9 netflow

2013-11-29 Thread Dobbins, Roland

On Nov 29, 2013, at 4:24 PM, Nikolay Shopik sho...@inblock.ru wrote:

 flow-sampler-map SM
 mode random one-out-of 100

flow-sampler-map SM
mode random 1 out-of 100

Try that with this:

interface GigabitEthernet0/1
ip address 10.10.112.143 255.255.255.0
ipv6 address 2001:db8:20:101::99/64
ip flow monitor IPv4 sampler SM input
ipv6 flow monitor IPv6 sampler SM input
end

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] sampled v9 netflow

2013-11-29 Thread Dobbins, Roland

On Nov 29, 2013, at 6:09 PM, Nikolay Shopik sho...@inblock.ru wrote:

 Even when you draw graph in 5 min interval?

Yes.  Your graphs will be way off unless you bring the active timer down to 
60s, irrespective of graph resolution.

You should also consider graphing with 1-minute intervals, it's more 
operationally useful.

 As lowering values means more flows exported per min so more load.

NDE is going to consume a very low CPU percentage, normally single-digit; if 
you're so close to the edge that NDE load matters, you have bigger problems.

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Effect of simultaneous TCP sessions on bandwidth

2013-11-10 Thread Dobbins, Roland

On Nov 10, 2013, at 1:09 PM, Youssef Bengelloun-Zahr yous...@720.fr wrote:

 Yes, sorry but I forgot to mention that we have activated every possible TCP 
 extension on servers in order to support latency effects over long WAN
 distances.

Due to the nature of TCP, delay-product has throughput effects even if 
everything has been optimized, stack- and application-wise.

That being said, have you been able to check the layer-2 stats on each of the 
ports in the chain, and verified that speed/duplex settings are either 
auto-negotiating properly or are hand-configured in matching configurations on 
each side of each link?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Effect of simultaneous TCP sessions on bandwidth

2013-11-10 Thread Dobbins, Roland

On Nov 11, 2013, at 5:32 AM, Octavio Alvarez alvar...@alvarezp.ods.org wrote:

 If you are trying to saturate the link in both directions each of the 
 acknowledge packets will compete against the other stream and will have a 
 hard time reaching back.

I first read this thread when I was half-asleep, stupid me - concur 100%, the 
performance is probably ACK-constrained in this scenario due to the traffic in 
the 'opposite' direction, as you wisely note.

It's easy to test this proposition - start the first session in direction A - 
B, and then the second in direction B, and measure the results.  Then start the 
first session in direction B - A, and the second session in direction A, and 
measure the results.  

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] vs Netscaler loadbalancer

2013-11-04 Thread Dobbins, Roland

On Nov 5, 2013, at 12:57 PM, Arne Larsen / Region Nordjylland a...@rn.dk 
wrote:

 Citrix's is full of endless configurations samples and how to setup, none 
 describes how the hardware, or at least I can't find it.

Normally, when buying an expensive piece of equipment like this, a service 
contract is also purchased which entitles the organization to technical support 
from the vendor.  Have you talked to Citrix?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] TAC hits a new record level of aggravation...

2013-11-02 Thread Dobbins, Roland

On Nov 3, 2013, at 7:29 AM, Justin M. Streiner strei...@cluebyfour.org wrote:

 It would be great if Cisco focus-group tested these 'enhancements' before 
 rolling them out, and knock it off with the Java nonsense.

They've been going in this direction for the last 10 years - it's doubtful that 
anything's going to change.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] TAC hits a new record level of aggravation...

2013-11-02 Thread Dobbins, Roland

On Nov 3, 2013, at 12:08 PM, Jeff Kell jeff-k...@utc.edu wrote:

 If enough of us complain... maybe.

Plenty of people inside and outside of Cisco have complained vociferously, to 
no avail.  It's unlikely to change.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T - poor netflow performance

2013-10-18 Thread Dobbins, Roland

On Oct 18, 2013, at 12:13 PM, Rolf Hanßen n...@rhanssen.de wrote:

 ip flow monitor monitorname input

 ip flow monitor monitorname output

If you're collecting both ingress and egress NetFlow on the same interface, 
this could be contributing to your issues - Cisco do not recommend doing this 
due to overflow issues (which could lead to punting).

Sampler configuration is covered in the Flexible NetFlow Command Reference for 
15.x on cisco.com. 

And again, input ifindex should be obtained via 'match', not 'collect', in 
order to ensure that it's a key field.

-
Roland Dobbins rdobb...@arbor.net



___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T - poor netflow performance

2013-10-17 Thread Dobbins, Roland

On Oct 17, 2013, at 7:06 PM, Rolf Hanßen n...@rhanssen.de wrote:

 For example a box exporting something to a Peakflow SP for dos recognition.
 I recognized that starting a random-source-ip flood over my box even could
 make the cli freeze.

This is not normal.

What does your per-interface config look like?

Are you sampling?

What linecards are you using?  Are they DFC4s or CFC linecards?

Just as an aside, it would be advisable not to use the collect verb for the 
input interface, but rather to use the match verb in order to use input ifindex 
as a key field.  'Collect' is for non-key fields.

-
Roland Dobbins rdobb...@arbor.net



___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] creation of flows for acl-deined traffic - Sup2T

2013-09-23 Thread Dobbins, Roland

On Sep 23, 2013, at 9:57 PM, Jiri Prochazka wrote:

 Is this doable?

It's not a good idea, as you lose visibility into traffic which you're 
blocking, but which is still chewing up link capacity and pummeling your boxen.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] 6500 real world (sampled) netflow

2013-09-03 Thread Dobbins, Roland

On Sep 3, 2013, at 4:34 AM, Jon Lewis wrote:

 Having used it exactly for that, I disagree and am curious why you say 
 it's useless. 

Because in any Internet-facing environment with any kind of traffic diversity, 
it's non-deterministically skewed.

So, in a network environment of any scale, you can't actually know whether or 
not a given source or destination is sending/receiving unusual volumes of 
traffic, as you don't know what is usual.  

It can't be relied upon in production environments.  fprobe or somesuch on a 
tap is more useful, although one loses the ifindex information.

Upgrading to Sup2T/DFC4 is the best option, if possible.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] 6500 real world (sampled) netflow

2013-09-02 Thread Dobbins, Roland

On Sep 3, 2013, at 3:41 AM, Peter Rathlev wrote:

 Though Sup720 Netflow has many limitations, the OP's use case is one that it 
 can actually help with. 

No, it isn't - he won't be able to detect anomalies reliably nor will he be 
able to characterize floods, because the statistics are non-determinstically 
skewed.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] 6500 real world (sampled) netflow

2013-09-01 Thread Dobbins, Roland

On Sep 1, 2013, at 7:57 AM, Randy wrote:

 It would only be used for detecting inbound UDP floods and other high PPS 
 anomalies so there is no need for full flows or even much details, just ip 
 src/dst. 

It's useless for this or any other application because of the limitations of 
the EARL7.  NetFlow isn't useful on 6500s until you get to Sup2T/DFC4.

Also, there's no such thing as packet-sampled control of flow creation - i.e., 
'sampled NetFlow' - on pre-Sup2T/DFC4 6500s.  There's output flow sampling, 
which simply serves to make the non-determinisically-skewed, completely 
unreliable statistics even worse.

Don't waste your time.  Upgrade, or use probes on taps until you can upgrade.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

The basis of optimism is sheer terror.

  -- Oscar Wilde


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] 6500, 7600 or ASR

2013-08-29 Thread Dobbins, Roland

On Aug 29, 2013, at 3:28 PM, CiscoNSP List wrote:

 Netflow

If you go with 6500 or 7600, be sure to use Sup2T and DFC4 in order to get 
usable NetFlow, uRPF, and ACLs.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] New Catalyst 6k chassis

2013-06-27 Thread Dobbins, Roland

On Jun 27, 2013, at 5:36 PM, Tom Hill wrote:

 I'm quite annoyed that there aren't any newer line cards announced that take 
 advantage of faster SerDes rates (as your existing X6800/X6900 
 series line cards will not run any faster) but then I seem to recall 6500-E 
 came a short while before PFC4/EARL8 was announced too.

Be sure to push the vendor on to provide a real-world performance envelope, 
including actual 64-byte packet throughput on edge interfaces with various 
features such as ACLs, uRPF, and CoPP enabled.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] New Catalyst 6k chassis

2013-06-26 Thread Dobbins, Roland

On Jun 27, 2013, at 3:40 AM, Gert Doering wrote:

 But can cisco afford to have three quite similar product lines,
 that are expensive to maintain? 

Cisco isn't really a unitary company, it's a loose confederation of semi-feudal 
fifedoms, each with its own PL.   Effectively, they're separate companies 
utilizing a common branding/marketing framework and shared administrative 
resources.

So, there is no 'Cisco' per se to afford or not afford to keep separate product 
lines going.  As long as each makes a profit, and there are no other overriding 
financial considerations such as an acquisition or somesuch, Cisco will 
continue to have X number of product lines in various spaces which overlap and 
even compete with one another.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] New Catalyst 6k chassis

2013-06-26 Thread Dobbins, Roland

On Jun 27, 2013, at 10:10 AM, Justin M. Streiner wrote:

 It just seems like the new 6k is positioned to poach prospective customers 
 from the (arguably) higher-margin Nexus 7k product line.

Not 'just seems' - 'is'.  Just as the new fixed-config one is positioned to 
poach prospective customers from the 4xxx-series.

;

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Linecard GSR12000 reboot

2013-06-12 Thread Dobbins, Roland

On Jun 12, 2013, at 2:03 PM, PlaWanSai RMUTT CPE IX wrote:

 Please help to check if you can give any advice will be appreciated.

Looks like CSCtr88610 or related, possibly.  Check with your Cisco account team 
and/or TAC.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Equivalent of ip multicast boundary on N7k for blocking data packets?

2013-06-05 Thread Dobbins, Roland

On Jun 4, 2013, at 4:54 AM, Phil Mayers wrote:

 including that you don't need to write both ingress and egress ACLs. Though I 
 suppose the latter are more flexible.

Egress ACLs are generally considered to be a Bad Thing, as they allow 
potentially undesirable packets past the port/linecard ASICs before dropping 
them on egress.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Need help with IPv6 CoPP

2013-05-07 Thread Dobbins, Roland

On May 7, 2013, at 5:17 PM, Nick Hilliard wrote:

 It's a more sensible idea to filter protocol 89 to your core address ranges 
 using an iACL and then permit all 89 in the CoPP policy.

Concur 100%.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Need help with IPv6 CoPP

2013-05-06 Thread Dobbins, Roland

On May 6, 2013, at 7:49 PM, Rolf Hanßen wrote:

 I am trying to configure IPv6 CoPP and could use some help with several 
 issues.

I know this isn't what you're asking, but if you haven't done so already, 
you'll get more benefit from iACLs, GTSM, re-coloring at your edges, et. al. 
first, then worrying about CoPP.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Need help with IPv6 CoPP

2013-05-06 Thread Dobbins, Roland

On May 6, 2013, at 11:11 PM, Rogelio Gamino wrote:

 At that stage, neighbors agree on Master/Slave relationship before moving to 
 exchange DBD's.

Unless you're doing OSPF with an external organization and anticipate an attack 
(either deliberate or inadvertent) from the adjacent router(s), why not leave 
OSPF out of it entirely, and instead concentrate on traffic which is 
layer-3-agile?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] 7600 IOS Recommendation

2013-05-01 Thread Dobbins, Roland

On Oct 12, 2009, at 12:23 PM, Matt Hill wrote:

 We have a 7600 whose sole purpose is to do Anomaly Guard/Detector Work.  It 
 has some EBGP peering and is in the traffic path.

You should run whatever IOS Cisco say supports the AGM/ADM (keeping in mind 
that the Guard/Detector are EoS/EoL, and ensuring that there are no issues with 
support in more recent software revisions).

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Netflow collector location

2013-04-20 Thread Dobbins, Roland

On Apr 20, 2013, at 1:10 PM, CiscoNSP List wrote:

 Thanks for the reply - data is used for billing (So it is critical)hence 
 my concern with lost flows.

NetFlow should be exported via your OOB/DCN. 

If you're losing packets on your OOB/DCN, you've bigger problems than worrying 
about a few lost flows.

;

Some degree of centralization or regionalization is standard in most flow 
telemetry collection/analysis deployments.  UDP vs. TCP isn't an issue in 
well-designed, well-operated networks.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] uRPF Core Internet Routers

2013-04-16 Thread Dobbins, Roland

On Apr 17, 2013, at 12:37 AM, Antonio Soares wrote:

 Now my question, is it appropriate to use uRPF loose mode on Core Routers
 (Full Routing Tables) ?

It's an edge technology - use loose-mode on your peering/transit interfaces, 
use strict-mode whenever possible on your customer aggregation/IDC access edges.

You can also use ACLs, IP Source Verify, Cable Source-Verify, PACLs, VACLs, et. 
al. as appropriate/necessary.

In terms of XR platforms, be sure you have the appropriate Engine cards in 
12000s.  CRS, ASR9K should support it just fine on about anything.

Of course, verify with your Cisco account team, test in lab with appropriate 
OS/train/release/throttle/feature mix before deploying, and so forth.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] uRPF Core Internet Routers

2013-04-16 Thread Dobbins, Roland

On Apr 17, 2013, at 8:42 AM, Lee wrote:

 But the IPv4 address space is close to all allocated, so enabling it for IPv4 
 doesn't seem like a huge win.  

This is incorrect, and is actually harmful misinformation.

The value of antispoofing has nothing to do with allocated address space 
percentages.  It has everything to do with removing the ability to launch 
high-volume reflection/amplification DDoS attacks, spoofed SYN-floods, et. al.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] DDoS + ME3600/ME3800

2013-03-28 Thread Dobbins, Roland

On Mar 28, 2013, at 3:54 PM, Lukasz Bromirski wrote:

 uRPF is coming, both for IPv4 and IPv6. Unfortunately, it's still
 in the future, not now.

It appears that Source Guard may be available now, depending upon the image 
being run:

http://www.cisco.com/en/US/docs/switches/metro/me3400/software/release/12.2_25_ex/configuration/guide/swdhcp82.html#wp1204202

ACLs are also available, in the interim.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T - poor netflow performance

2013-03-27 Thread Dobbins, Roland

On Mar 27, 2013, at 6:42 PM, Jiri Prochazka wrote:

 As soon as I limit usage to 70% (both Sup and linecards), no flows are 
 dropped, but the box is still dying.

Are your linecards all either DFC4 or CFC?

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


Re: [c-nsp] Sup2T - poor netflow performance

2013-03-27 Thread Dobbins, Roland

On Mar 27, 2013, at 7:50 PM, Mikael Abrahamsson wrote:

  For Internet peering router at 10GE with typical eyeball traffic my 
 opinion is that 6500/7600 doesn't have working netflow.

Sup2T/DFC4 fixes these issues, as well as the uRPF mode limitation and weird 
ACL threshold limitation.

The problem the OP is experiencing is likely a result of configuration issues, 
lots of punted traffic generating flows, or a bug.  The EARL8 ASIC solves all 
the previous issues associated with 6500/7600 NetFlow.

---
Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com

  Luck is the residue of opportunity and design.

   -- John Milton


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


  1   2   3   >