Re: FlowSpec
On 23 Apr 2020, at 22:57, Denys Fedoryshchenko wrote: In general operators don't like flowspec Its increasing popularity tens to belie this assertion. Yes, you're right that avoiding overflowing the TCAM is very important. But as Rich notes, a growing number of operators are in fact using flowspec within their own networks, when it's appropriate. Smart network operators tend to do quite a bit of lab testing, prototyping, PoCs, et. al. against the very specific combinations of platforms/linecards/ASICs/OSes/trains/revisions before generally deploying new features and functionality; this helps ameliorate many concerns. Also, don't forget about S/RTBH. It's generally confined to within an operator's own span of administrative control for some of the same reasons as flowspec (not generally TCAM, per se, but concerns about giving Customer A the ability to interfere with Customer B's traffic, and the difficulty of implementing such constraints). It can be an option worth exploring, in many circumstances. Roland Dobbins
Re: UDP/123 policers & status
On 21 Mar 2020, at 4:58, Hal Murray wrote: I don't want to start a flame war, but why isn't BCP 38 widely deployed? Can somebody give me a pointer to a talk at NANOG or such? What fraction of the world does implement BCP 38? I'd also be interested in general background info on DDoS. Who is DDoS-ing whom and/or why? Is this gamers trying to get an advantage on a competitor? Bad guys making a test run to see if the server can be used for a real run? Is DDoS software widely available on the dark web? ... Answers to all of these questions are readily available via search engines, searching the archive of this and related listservs, searching the presentation archives of NANOG/RIPE/APRICOT/APNIC/AusNOG/NZNOG/LACNIC, et. al. My archive of public presentations is available here: <https://app.box.com/s/4h2l6f4m8is6jnwk28cg> These topics are well-understood and -documented, and a bit of research can help bring one up to speed on them pretty quickly. ---- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 1 Sep 2018, at 1:43, Hugo Slabbert wrote: Generally on the TCP side you can try SYN or ACK floods, but you're not going to get an amplified reflection. Actually, TCP reflection/amplification has been on the increase; the attacker is guaranteed at least 4:1 amplification in most circumstances, the number of reflectors/amplifiers is for all practical purposes infinite, and they're mostly legitimate, non-broken services/applications. And as always, it's important to note that with all reflection/amplification attacks, the root of the issue is the lack of universal source-address validation (SAV). Without the ability to spoof, there would be no reflection/amplification attacks. --- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 1 Sep 2018, at 1:20, Lotia, Pratik M wrote: Arbor report mentions volumetric attacks using DNS, NTP form 75+% of the attacks. I'm well aware of what's mentioned in the Arbor report, thanks! ;> Then QoSing certain ports and protocols is the best way to start with. The point is that when applying broad policies of this nature, one must be very conservative, else one can cause larger problems on a macro scale. Internet ateriosclerosis is a significant issue. --- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 1 Sep 2018, at 1:35, Aaron Gould wrote: I may mark internet-sourced-udp with a certain marking dscp/exp so that as it travels through my internet network, it will be the first to get dropped (? Wred ? work well for udp?) during congestion when an attack gets through You can use flow telemetry analysis to look at the UDP non-initial fragments destined for any access networks under your control; you'll likely see that they comprise a tiny portion of the overall traffic mix, and they're most commonly associated with large DNS answers. Once you've determined the numbers, you can police down the non-initial fragments destined for the access networks you control (don't do this on transit traffic!) to whatever small percentage makes the most sense, with a bit of extra headroom. 1% of link bandwidth works for several operators. In that QoS policy, you exempt well-known/well-run open DNS recursor farms like Google DNS, OpenDNS, et. al. (and possibly your own, depending on your topology, etc.), and any other legitimate source CIDRs which generate appreciable amounts of non-initial fragments. When a reflection/amplification attack which involves non-initial fragments hits, the QoS policy will sink a significant proportion of the attack. It doesn't help with your peering links, but keeps the traffic off your core and off the large network(s). Again, don't apply this across-the-board; only do it for access networks within your span of administrative control. * btw, what can you experts tell me about tcp-based volumetric attacks... TCP reflection/amplification. --- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 31 Aug 2018, at 23:53, Lotia, Pratik M wrote: Instead of rtbh I would suggest blocking/rate limiting common ports used in DDoS attacks. This isn't an 'instead of', it's an 'in addition to'. And it must be done judiciously; many operators doing this have concentrated on common port-pairs observed in UDP reflection/amplification attacks. It's important to understand that any kind of packet of any protocol/ports (if such concepts apply on the protocol in question) can be used to launch DDoS attacks. We've many tools in the toolbox, and should use them in a situationally-appropriate manner. And when we're using techniques like QoSing down certain ports/protocols, we must err on the side of caution, lest we cause larger problems than the attacks themselves. --- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 31 Aug 2018, at 22:15, Hugo Slabbert wrote: I would love an upstream that accepts flowspec routes to get granular about drops and to basically push "stateless ACLs" upstream. <https://pc.nanog.org/static/published/meetings/NANOG71/1447/20171003_Levy_Operationalizing_Isp_v2.pdf> ------- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 31 Aug 2018, at 16:33, Ryan Hamel wrote: From experience, sflows are horribly inaccurate for DDoS detection, since the volume could disrupt the control plane and render the process useless, thus not giving data to the external system to act upon it. On the contrary, flow telemetry in general works quite well for DDoS detection/classification/traceback, and is widely utilized for such purposes; it has been for many years. I'm not a big fan of s/Flow comparatively speaking, but it and NetFlow, IPFIX, et. al. have proven themselves over the years, assuming that the flow export parameters on the exporting devices are configured correctly, and the collection/analysis systems are configured optimally. Flow telemetry is management-plane, not control-plane. Implementing network infrastructure self-protection BCPs such as iACLs is definitely recommended in general. --- Roland Dobbins
Re: automatic rtbh trigger using flow data
On 31 Aug 2018, at 6:47, Aaron Gould wrote: I'm really surprised that you all are doing this based on source ip, simply because I thought the distribution of botnet members around the world we're so extensive that I never really thought it possible to filter based on sources, i Using S/RTBH to drop attack sources has been a valid and useful mitigation tactic for close to 20 years. Any kind of modern router scales up to large numbers of sources; and note that S/RTBH isn't limited to /32s. It's discussed in this .pdf preso: <https://app.box.com/s/xznjloitly2apixr5xge> --- Roland Dobbins
Re: tcp md5 bgp attacks?
On 15 Aug 2018, at 9:27, Randy Bush wrote: my theory is that, as the attacks were mitigated the attackers moved on to other things. With regards to BGP, the MD5 thing was promulgated to counter what was a largely theoretical threat. iACLs, and later GTSM and CoPP and LPTS and so forth really obviated the need for it. For IGPs, MD5 was belt-and-suspenders against someone deliberately or accidentally bringing up a new router and manipulating traffic internally. Passiving the IGP on non-core links was the BCP, but often was honored in the breach; pushing an additional feature for 'security' purposes got some folks' attention when the passiving BCP was ignored. We still see DDoS attacks against routers, of course. But the goal there is disruption of availability, not trying to move traffic onto some alternate path which would somehow benefit the attacker. --- Roland Dobbins
Re: tcp md5 bgp attacks?
On 15 Aug 2018, at 6:28, Grant Taylor via NANOG wrote: > Is there something that I've missed the boat on? No - it's a belt-and-suspenders sort of thing, along with GTSM. --- Roland Dobbins
Re: SP security knowledge build up
On 23 Jul 2018, at 22:30, Compton, Rich A wrote: Barry Greene's site has some good info on ISP security as well: Here're some more: <https://app.box.com/s/4h2l6f4m8is6jnwk28cg> This book is also highly recommended: <https://www.amazon.com/Router-Security-Strategies-Networking-Technology-ebook/dp/B0051TM5L2/> --- Roland Dobbins
Re: Attacks on BGP Routing Ranges
On 18 Apr 2018, at 18:03, Ryan Hamel wrote: Could you explain how this can resolve my issue? I am not sure how this would work. You should have iACLs and GTSM enabled, as noted previously. Ideally, the link should come from an unadvertised range, or a range which is sunk to null0 at the edge, as Job indicated. If the link is numbered from a range assigned to your peer, they should have iACLs in place to prevent that range being packeted. If the link is numbered from your own range, you should ask your peer to add that range to their iACLs, as well. This .pdf preso discusses infrastructure self-protection concepts: <https://app.box.com/s/osk4po8ietn1zrjjmn8b> --- Roland Dobbins <rdobb...@arbor.net>
Re: New Active Exploit: memcached on port 11211 UDP & TCP being exploited for reflection attacks
On 28 Feb 2018, at 5:26, Ca By wrote: Just udp. This Arbor Threat Summary discusses the TCP issue, as well, FWIW: <https://www.arbornetworks.com/blog/asert/memcached-reflection-amplification-description-ddos-attack-mitigation-recommendations/> 'It should also be noted that memcached priming queries can also be directed towards TCP/11211 on abusable memcached servers. TCP is not currently considered a high-risk memcached reflection/amplification transport as TCP queries cannot be reliably spoofed.' We also recommend implementing situationally-appropriate network access policies at the IDC edge which disallow unwanted UDP/11211 as well as TCP/11211 from reaching abusable memcached deployments. --- Roland Dobbins <rdobb...@arbor.net>
Re: BCP38/84 and DDoS ACLs
On 27 May 2017, at 0:19, Roland Dobbins wrote: > <https://app.box.com/s/ko8lk4vlh1835p36na3u> This is the correct URI for the first preso, apologies: <https://app.box.com/s/osk4po8ietn1zrjjmn8b> ------- Roland Dobbins <rdobb...@arbor.net>
Re: BCP38/84 and DDoS ACLs
On 27 May 2017, at 0:54, valdis.kletni...@vt.edu wrote: > I'll go out on a limb and suggest that except for a very basic home/SOHO > network, "You may need" should be "You will probably need". Concur, heh. ----------- Roland Dobbins <rdobb...@arbor.net>
Re: BCP38/84 and DDoS ACLs
On 26 May 2017, at 22:39, Graham Johnston wrote: I am looking for information regarding standard ACLs that operators may be using at the internet edge of their network, on peering and transit connections, These .pdf presos may be of interest: <https://app.box.com/s/ko8lk4vlh1835p36na3u> <https://app.box.com/s/xznjloitly2apixr5xge> They talk about iACL and tACL design philosophy. What traffic you should permit/deny on your network is, of course, situationally-specific. Depends on what kind of network it is, what servers/services/applications/users you have, et. al. You may need one set of ACLs at the peering/transit edge, and other, more specific ACLs, at the IDC distribution gateway, customer aggregation gateway, et. al. --- Roland Dobbins <rdobb...@arbor.net>
Re: Consumer networking head scratcher
On 2 Mar 2017, at 9:55, Oliver O'Boyle wrote: Currently, I have 3 devices connected. :) You could have one or more botted machines launching outbound DDoS attacks, potentially filling up the NAT translation table and/or getting squelched by your broadband access provider with layer-4 granularity. And the boxes themselves could be churning away due to being compromised (look at CPU and memory stats over time). Aggressive horizontal scanning is often a hallmark of botted machines, and it can interrupt normal network access on the botted hosts themselves. I don't actually think that's the case, given the symptomology you report, but just wanted to put it out there for the list archive. What about DNS issues? Are you sure that you really have a networking issue, or are you having intermittent DNS resolution problems caused by flaky/overloaded/attacked recursivs, EDNS0 problems (i.e., filtering on DNS responses > 512 bytes), or TCP/53 blockage? Different host OSes/browsers/apps exhibit differing re-query characteristics. Are the Windows boxes and the other boxes set to use the same recursors? Can you resolve DNS requests during the outages? Are your boxes statically-addressed, or are they using DHCP? Periodically-duplicate IPs can cause intermittent symptoms, too. If you're using the consumer router as a DHCP server, DHCP-lease nonsense could be a contributing factor. Are the Windows boxes running some common application/service which updates and/or churns periodically? Are they members of a Windows workgroup? All kinds of strange name-resolution stuff goes on with Windows-specific networking. Also, be sure to use -n with traceroute. tcptraceroute is useful, too. netstat -rn should work on Windows boxes, IIRC. --- Roland Dobbins <rdobb...@arbor.net>
Re: Software for network modelling / documentation / GIS
On 24 Feb 2017, at 10:31, Israel G. Lugo wrote: Does anyone know of something similar to this exist in commodity software, outside of custom solutions developed for a specific network? FWIW, I'm pretty sure Visio has been able to snmpwalk for many years. Some NMSes have this sort of capability, too. --- Roland Dobbins <rdobb...@arbor.net>
Re: Distributed Object Architecture versus DNS
On 7 Jan 2017, at 14:22, Joly MacFie wrote: > Blind backlash from IoT DDoS? Looming billions of rf tagged items? None of this has anything to do with this 'DOA' thing, though. --- Roland Dobbins <rdobb...@arbor.net>
Re: Distributed Object Architecture versus DNS
On 7 Jan 2017, at 10:15, Joly MacFie wrote: They are worried (as I understand it) 1) that it could be an ITU end run to grab back numbering, 2) it could be abused by bad actors such as repressive governments who want to use it for digital id. Based on seemingly cyclical statements of this nature, I've been waiting for the ITU to impose GOSIP or whatever on us for the last ~30 years or so - but so far, nothing much has happened in that regard. Is there actually a reason to suspect that this time it will be any different? --- Roland Dobbins <rdobb...@arbor.net>
Re: Distributed Object Architecture versus DNS
On 7 Jan 2017, at 4:15, Stephenson, Ryan M CIV DISA IE (US) wrote: Does anyone have any information about DOA versus DNS. My understanding of 'DOA' is that it's a general category of software architecture (think CORBA) and nothing to do with name resolution or any sort of directory services, per se. Can you provide more context? --- Roland Dobbins <rdobb...@arbor.net>
Re: [Tier1 ISP] : Vulnerable to a new DDoS amplification attack
On 22 Dec 2016, at 23:56, Tom Beecher wrote: What he did was send 1500 byte ICMP packets with a max TTL at an IP address that is not reachable due to a routing loop. Same here. Here's some context I sent him: <https://www.usenix.org/legacy/events/imc05/tech/full_papers/xia/xia_html/imc05-paper-128-final.html> <http://nanog.org/meetings/nanog36/presentations/xia.pdf> <https://youtu.be/cWF4p5EuvQk> Note related discussion of mitigation tactics here (e.g., TTL-based filtering via tACLs): <http://www.cisco.com/c/en/us/about/security-center/ttl-expiry-attack.html> ------- Roland Dobbins <rdobb...@arbor.net>
Re: [Tier1 ISP] : Vulnerable to a new DDoS amplification attack
On 22 Dec 2016, at 20:27, Jean | ddostest.me via NANOG wrote: the already known Layer 4 amp DDoS like dns, ntp, ssdp, snmp These are layer-7 reflection/amplification attacks - i.e., application-layer - *not* layer-4. --- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 20 Dec 2016, at 12:18, Laurent Dumont wrote: > As a student in the field, this is the kind of stuff I live for! ;) <https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Notable_cases> ------- Roland Dobbins <rdobb...@arbor.net>
Re: Prepending with another ASN you don't own
On 17 Dec 2016, at 0:13, Job Snijders wrote: There are providers who inspect the AS_PATH's contents and make decisions to reject (ignore) a route announcement or not based on the presence of certain values. +1 --- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 16:40, Roland Dobbins wrote: Looking at the source IP distribution, does a significant proportion of the larger query base seem to originate out-of-region? And are do they appear to be mostly broadband access networks, or . . . ? --- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 6:20, Allan Liska wrote: FWIW The US server has seen a spike in traffic, but I have not seen a similar spike on the EMEA server. Looking at the source IP distribution, does a significant proportion of the larger query base seem to originate out-of-region? --- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 10:17, Roland Dobbins wrote: <http://pages.cs.wisc.edu/~plonka/netgear-sntp/> Over on nznog, Cameron Bradley posited that this may be related to a TR-069/-064 Mirai variant, which makes use of a 'SetNTPServers' exploit. Perhaps one of them is actually setting timeservers? This SANS writeup details the SOAP strings: <https://isc.sans.edu/forums/diary/Port+7547+SOAP+Remote+Code+Execution+Attack+Against+DSL+Modems/21759> --- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 10:16, Roland Dobbins wrote: > <http://pages.cs.wisc.edu/~plonka/netgear-sntp/> ------- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 10:09, Dan Drown wrote: This seems more like "someone pushed out bad firmware" rather than something malicious. Everything old is new again . . . ------- Roland Dobbins <rdobb...@arbor.net>
Re: Recent NTP pool traffic increase
On 16 Dec 2016, at 5:45, Jose Gerardo Perales Soto wrote: We've recently experienced a traffic increase on the NTP queries to NTP pool project (pool.ntp.org) servers. Do you have flow telemetry, which provides a lot more information than basic pps/bps stats? Are you seeing normal timesync queries, or lots of level-6/level-7 admin command attempts? --- Roland Dobbins <rdobb...@arbor.net>
Re: Favorite Speed Test Systems
On 5 Dec 2016, at 21:50, Graham Johnston wrote: What is your preferred one and why? <http://testmy.net/> Thorough, reasonable teat methodology, allows one to store history, decent range of test servers worldwide. --- Roland Dobbins <rdobb...@arbor.net>
Re:
On 2 Dec 2016, at 22:31, Christopher Morrow wrote: > that statement seems ... hard to prove. Paging Geoff Huston to the white courtesy phone . . . ;> --- Roland Dobbins <rdobb...@arbor.net>
Re: Spitballing IoT Security
On 30 Oct 2016, at 7:32, Ronald F. Guilmette wrote: you don't need to be either an omnious "state actor" or even SPECTER to assemble a truly massive packet weapon. I agree: <https://www.arbornetworks.com/blog/asert/how-to-become-an-internet-supervillain-in-three-easy-steps/> ;> Two kids with a modest amount of knowledge and a lot of time on their hands can do it from their mom's basement. And indeed have done so, many times. The *entire purpose* of Mirai is DDoS-for-hire - it's a foundation for so-called 'booter/stresser' services. So, the various articles about how these botnets 'might' be for sale are uninformed - they're *for rent*, that's their raison d'être. And renting them is cheap. The economic and resource asymmetries highly favor the attackers. All the speculation about how 'state actors' are somehow 'learning how to take down the Internet' is equally uninformed. State actors already know how to do this, they don't need to 'learn' or 'test' anything. DDoS attacks are the Great Equalizer; when it comes to DDoS, nation-states are just another player. ------- Roland Dobbins <rdobb...@arbor.net>
Re: How to find all of an ISP's ASNs
On 26 Oct 2016, at 0:41, Gary Baribault wrote: > other than the two local major ISPs (keeping last Friday in mind!) . . . why would you want to expose them to the public Internet at all? There are many, many reasons not to do so. --- Roland Dobbins <rdobb...@arbor.net>
Re: Dyn DDoS this AM?
On 21 Oct 2016, at 23:01, Mike Hammett wrote: > Are there sites that can test your BCP38\84 compliance? <https://www.caida.org/projects/spoofer/> ------- Roland Dobbins <rdobb...@arbor.net>
Re: MPLS in the campus Network?
On 20 Oct 2016, at 23:32, Mark Tinka wrote: Some requirements call for Ethernet transport as opposed to IP. Sure - but it's probably worth revisiting the origins of those requirements, and whether there are better alternatives. --- Roland Dobbins <rdobb...@arbor.net>
Re: MPLS in the campus Network?
On 20 Oct 2016, at 23:17, Mark Tinka wrote: especially looking at how much Layer 2 traffic you're hauling around. And I'd definitely recommend figuring out why that's being done so broadly today, and working to reduce its prevalence and scope, moving forward. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 28 Sep 2016, at 0:18, Brielle Bruns wrote: > I call shenanigans on providers not seeing their unruly users. I was talking about the users, not the ISPs. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 22:49, Florian Weimer wrote: Most people over here have at least two providers of water and Internet (although the second one is perhaps sufficient for brushing your teeth, but certainly not for a shower or a bath). That's not a common arrangement in much of the world, however. Especially the Internet part. ;> --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 22:46, Brielle Bruns wrote: I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers. They can *see* the unruly children, but *choose* to ignore them. That's the difference. Keep in mind, most of the folks on this list are not representative of the average consumer in terms of the skill-sets which are relevant in this problem space. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote: All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. Yes, but how do they determine that a given device is vulnerable? --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 12:17, Sam Silvester wrote: or call their electricity retailer/distributer This is the problematic case that is, unfortunately, the default. People tend to view anything related to 'the Internet' as a utility, and for consumers and SMBs, they typically have a single provider, just as they typically do for electricity and water. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 21:48, Brielle Bruns wrote: You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. It's important to keep in mind that in the not-so-distant future, their 'machines' will include every article of clothing they own, every can of soda in their refrigerator, ever major (and many minor) components of their automobiles, every blade in their windowshades, etc. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 12:31, Jason Hofmann wrote: It probably was a tough sell to get people to realize they were fully responsible for their in-home wiring, but optional "inside wire maintenance plans" made that clear while also adding to providers' coffers. Perhaps something similar would work here. Concur that this is the least-improbable model, absolutely. But keep in mind that subscriptions/services for in-home wiring were (and are) also a tiny percentage of the user base. ------- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 12:14, Mark Andrews wrote: I'm yet to see a set top box, DVR, TV, games console, phone, etc. that didn't require selecting the WiFi SSID or require you to plug in a ethernet cable. I've 'seen' tens of millions of them, worldwide. You're generalizing your particular connection/personal provisioning model. As I said, they don't magically connect to the network. Someone did something to permit them to connect. That someone quite often isn't the end-user. And as noted previously in this thread, even when users themselves do this, they promptly forget how they did it, lose the documentation, etc. Why do you think people are incapable of calling in someone to help them fix a known issue. 1. Because they demonstrably don't. 2. Because it's not perceived as a 'computer problem' - it's perceived as an 'Internet problem', and the 'Internet technician' = the broadband access operator's help-desk. 3. Going along with the line of reasoning you've expressed, it seems that the user should call a 'lightbulb technician' when his Internet-enabled lightbulb is causing a problem. Do you really think that's realistic? 4. In most cases, the user won't have any idea which connected device is causing the problem. Expecting the user to determine this by trial-and-error is unrealistic; most people don't even understand how to troubleshoot electrical problems by trial-and-error, much less Internet-related problems. You are a self-selected specialist, and understand all these things and have a DIY attitude, because you're an expert in this field. Most people aren't experts in this field. Ask yourself how many people set up and use 2FA for any online service which supports it, on their own initiative (i.e., not having a bank ship them a pre provisioned dongle). The number of people capable of doing this troubleshooting for themselves is roughly equivalent to the number of people who've successfully set up 2FA on their own initiative. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 11:43, Mark Andrews wrote: Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different. Washing machines aren't a utility. Internet is viewed as a utility. Actually I don't believe that. They do know what machines they have have connected to their home network. Boxes don't magically connect. Every machine was explictly connected. First of all, not every devices was explicitly connected by the user. Think set-top boxes/DVRs. Secondly, users connect things an then don't think about them, don't remember credentials, had a horrible ordeal (from their perspective) connecting said devices and then promptly forgot how to administer them. Thirdly, expecting users to troubleshoot which of their devices is emanating bad traffic is unrealistic. The only effective consumer remediation efforts we've seen to date have been broadband access ISPs proactively scanning their customer networks and contacting them when exploitable devices and compromised PCs have been found. Although it's a lot of work, that kind of thing can be done for CPE broadband routers; it can't be done for the things sitting behind those devices, which are doing NAT/firewalling. The partial exception is PCs, because everyone thinks of those when they think of 'the Internet'. And the fact that even their lightbulbs are being connected now - i.e., the huge proliferation of connected devices - militates against user troubleshooting, as well. --- Roland Dobbins <rdobb...@arbor.net>
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
On 27 Sep 2016, at 6:58, Christopher Morrow wrote: wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE and kept for ~30mins (just guessing) in a circular buffer be 'good enough' to present a pretty clear UI to the user? +1 for this capability in CPE. OTOH, it will be of no use whatsoever to the user. Providing the user with access to anomalous traffic feeds won't help, either. Users aren't going to call in some third-party service/support company, either. It call comes down to the network operator, one way or another. There's no separation in the public mind of 'my network' from 'the Internet' that is analogous to the separation between 'the power company' and 'the electrical wiring in my house/apartment' (and even in that space, the conceptual separation often isn't present). --- Roland Dobbins <rdobb...@arbor.net>
Re: PlayStationNetwork blocking of CGNAT public addresses
On 21 Sep 2016, at 15:37, Baldur Norddahl wrote: Which means we may ignore it instead. . . . copy/paste or awk/sed or whatever isn't an option? If not, have you requested a) separate notifications per source and/or b) a more textual-manipulation-friendly format? Unless they're sending .gifs or something, surely this might be possible, yes? It seems within the realm of possibility this sort of response - or lack thereof - could result in some gaming network operators becoming a bit jaded. And perhaps some customers, too. --- Roland Dobbins <rdobb...@arbor.net>
Re: PlayStationNetwork blocking of CGNAT public addresses
On 16 Sep 2016, at 20:38, Simon Lockhart wrote: Unless we know what to look for, it's hard to detect and stop it. It's not just application-layer stuff - they're subject to all sorts of attacks. Screening out the obvious stuff would certainly help. The main issue is a dearth of engagement of clueful folks in the global operational community. Some gaming-oriented networks are well-represented; others are not, sadly. --- Roland Dobbins <rdobb...@arbor.net>
Re: PlayStationNetwork blocking of CGNAT public addresses
On 16 Sep 2016, at 20:12, Simon Lockhart wrote: Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. --- Roland Dobbins <rdobb...@arbor.net>
Re: EVERYTHING about Booters (and CloudFlare)
On 29 Jul 2016, at 20:34, J. Oquendo wrote: Because someone breaking AUPs and TOS is not enough. The AUP, the TOS, and the RFP are the most powerful security tools any network operator has at their disposal - assuming they've invested some time and effort in crafting them, and in ensuring they can be enforced. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thinking Methodically about building a PoC
On 13 Jun 2016, at 8:52, Kasper Adel wrote: > 2) Do some planning and research first. This. --- Roland Dobbins <rdobb...@arbor.net>
Re: AW: AW: Verizon and Level3 DNS flush
On Jun 2, 2016, at 3:42 PM, Jürgen Jaritsch <jjarit...@anexia-it.com> wrote: > it IS expected behavior that traffic will switch over to the new DNS. Altering routing and/or adding capacity/capabilities to the existing infrastructure is generally better, whenever possible, due to the cache-flushing challenges you're now experiencing. Sometimes it isn't possible, of course. ------- Roland Dobbins <rdobb...@arbor.net>
Re: AW: Verizon and Level3 DNS flush
On Jun 2, 2016, at 1:24 AM, Jürgen Jaritsch <jjarit...@anexia-it.com> wrote: > and that's the reason why we had to move over to a new NS set. Which the attackers (or their attack tools) will immediately discern, & shift their targeting accordingly. Playing games like this with addressing seldom, if ever, accomplishes anything useful in terms of successfully defending against DDoS attacks. ------- Roland Dobbins <rdobb...@arbor.net>
Re: Turning Off IPv6 for Good (was Re: Netflix VPN detection - actual engineer needed)
On 2 Jun 2016, at 10:47, Paul Ferguson wrote: There is an epic lesson here. I'm just not sure what it is. :-) That Netflix offering free streaming to everyone over IPv6 (after fixing their VPN detection) would be the most effective way to convince end-users to demand IPv6 service from their ISPs? ;> --- Roland Dobbins <rdobb...@arbor.net>
Re: NIST NTP servers
On 11 May 2016, at 8:59, Mel Beckman wrote: My point is, when the fix is so cheap, why put up with this risk at all? Time and Position Spoofing With Open Source Projects. [.pdf link] <https://www.blackhat.com/docs/eu-15/materials/eu-15-Kang-Is-Your-Timespace-Safe-Time-And-Position-Spoofing-Opensourcely-wp.pdf> Just keep in mind, *nothing* is perfect. --- Roland Dobbins <rdobb...@arbor.net>
Re: BGP FlowSpec
On 3 May 2016, at 5:38, Martin Bacher wrote: Let the packets come is not the message. That was *precisely* the message which was spoken to me directly by a large regional CONUS ISP in mid-2003 or thereabouts. I know this; I was there. And it was the wrong message, as that particular ISP found out a couple of weeks later when their network was knocked flat and they lost customers because of it. A bit of schadenfreude might not have been out of place, for the less-charitably inclined. or remark and/or rate-limit the particular flows with nearly, of course not for the customer under attack, the same result. This is almost always a Bad Idea, because the programmatically-generated attack traffic ends up 'crowding out' the legitimate traffic. For some attacks which are obviously out-of-profile with regards to the attack targets, this isn't as much of a concern; some large network operators are doing this with regards to common UDP reflection/amplification traffic (but they're being careful about it). And that still doesn't address the issue of high-volume traffic choking peering/transit links, of course. But that does not imply that all upstream ISPs are filtering out attacks by default for customers which are not paying for that. Nobody here has said that. But some beneficiary collateral effects of this nature do show up, from time to time. This is at least my interpretation from reading the various available DDoS reports and research papers. You should probably be aware that you are likely conversing directly with the authors of/contributors to some of those very reports and research papers in this thread (depending on which reports and papers you mean), and that the people with whom you are interacting routinely mitigate DDoS attacks on the public Internet as part of their normal work routine - and have done so for many years. For many of us, this is not a theoretical discussion; and it would probably be a good idea to keep in mind that our contributions to this thread aren't based upon reading various reports and research papers, but rather upon our actions which generate the data and experiential observations upon which such reports and research papers are based. --- Roland Dobbins <rdobb...@arbor.net>
Re: BGP FlowSpec
On 3 May 2016, at 4:51, jim deleskie wrote:
I was going to avoid this thread because I've never been a huge fan of
Flowspec for my own reasons.
Flowspec is an extremely useful tool, IMHO - not only for direct,
layer-4-granular mitigation leveraging linecard ASICs, but for more
granular and selective diversion into mitigation centers, as well. And
its value is growing with increased platform support. It isn't perfect
(nothing is), and operators must be aware of its performance/scalability
envelope on a given platform, but it's a great tool to have in the
toolbox.
I can say I, nor any of my peers ( in any sense of that word) that I
have known, have wanted to keep "bad " traffic on our networks so we
can bill for it.
+1!
I ran into this situation precisely twice early in the 'oughts ("Let the
packets come!" was the quote which stood out in my mind); those
espousing it pretty quickly changed their tunes once their networks had
been knocked flat a couple of times.
;>
-------
Roland Dobbins <rdobb...@arbor.net>
Re: BGP FlowSpec
On 2 May 2016, at 20:16, Martin Bacher wrote: However, Tier 1s and most probably also some of the Tier 2s may not want to offer it to customers because they are loosing money if less traffic is sent downstream on IP-Transit links. I will go a step further than Danny's comments and state that this is categorically and demonstrably untrue. Many of the quite large 'Tier-1' and 'Tier-2' (using the old terminology) operators on this list offer commercial DDoS mitigation services making use of technologies like D/RTBH, S/RTBH, IDMS, et. al. due to customer demand. They need these capabilities in order to defend their own properties and assets, and they are also offering them to end-customers who want and need them. In point of fact, it's becoming difficult to find one which *doesn't* offer this type of service. There were a couple of situations in the first half of the first decade of this millennium where operators took this attitude. But they changed their tunes pretty rapidly once they themselves were impacted, and once they started losing customers because they couldn't and wouldn't protect them. And as Danny notes, these technologies are all tools in the toolbox. NFV and 'SDN' have tremendous potential to make it a lot easier to bring mitigation resources to bear in a dynamic and optimal fashion within single spans of administrative control; and there are standards-based efforts underway to provide for a higher degree of automation, increased rapidity of response, and interoperability in both inter- and intra-network DDoS mitigation scenarios. --- Roland Dobbins <rdobb...@arbor.net>
Re: BGP FlowSpec
On 30 Apr 2016, at 19:56, Pierre Lamy wrote: > to null out the destination rather than the source. <https://tools.ietf.org/html/rfc5635> ------- Roland Dobbins <rdobb...@arbor.net>
Re: Why the US Government has so many data centers
On 13 Mar 2016, at 3:03, George Herbert wrote: > It's a symptom of trying to save a few cents at the risk of dollars. Concur 100%. Not to mention the related security issues. --- Roland Dobbins <rdobb...@arbor.net>
Re: Why the US Government has so many data centers
On 12 Mar 2016, at 0:03, Sean Donelan wrote: The U.S. Government has an odd defintion of what is a data center, which ends up with a lot of things no rational person would call a data center. There's also a case to be made that governmental organizations really oughtn't to have servers just lying around in random rooms, and that those rooms are de facto government data centers, whether those who're responsible for said rooms/servers know it or not . . . --- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 16:59, Pavel Odintsov wrote: I have only one question. Why you against sFLOW protocol telemetry with so huge passion ? :) Because I've had very poor experiences with it. And it doesn't seem to scale very well. Actually, sflow is not so popular as netflow. But to be honest it's pretty young standard in compare with netflow and it implements slightly different approach. sFlow has been around for a while, though. It isn't new. So IX could not use netflow even if they want. This depends upon the devices utilized - there are actually some devices which can export layer-2 NetFlow. There are other issues with NetFlow as it's currently generally implemented which are also concerns with IX scenarios, FYI. I will leave it as an exercise for you to find out what they are. But you vote for "sflow is weird protocol and you should avoid it". My view is that it's generally better to use NetFlow or IPFIX, where and when possible. How IX could monitor traffic if they haven't netflow? So if they follow your recommendations they should drop idea about traffic monitoring at all :) Straw man. I never said that nor implied it. If sFlow is all that's available, then of course operators can and should use it. But actually in modern network world every technology has applicable usage and it's not good idea to avoid it just because your religion (I'm speaking about netflow religion) prohibit it for you. It isn't 'religion'. It's based upon the fact that a) my experiences with sFlow have been suboptimal and b) sFlow isn't generally available on large routers used at network edges. Actually you are writing this email from company email and I could conclude it's Arbor vision and is not your own. No, that is incorrect. I speak only for myself. And as I previously noted, Arbor products support sFlow, and have for many years; I'm just not a big fan of it. Could you clarify it? I just did. Could I use your vision as Arbor's vision in public speeches / presentations? No, you may not, per the above. Arbor is telemetry-neutral; we aim to support all relevant telemetry formats in line with the expressed needs of our customers. And that includes sFlow. These trollish, passive-aggressive rhetorical tactics grow wearisome. I will not reply any further to this thread, so as to avoid further spamming the list. ------- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 15:53, Pavel Odintsov wrote: It's not about default. It's about minimal possible. To my knowledge, there has never been a Cisco router which only allowed an active flow timer value of 180s, which wasn't user-configurable. I would appreciate the details of any such router. For Mikrotik routers same issue. Minimal possible timeout is 60 / 60. And impossible to decrease it. As we've seen already from another poster in this thread, that isn't the case. Also so much routers could not do enough accurate netflow without additional (and very expensive) line cards just for netflow generation. I believe you're referring to PICs on Juniper routers, yes? Or perhaps the requirement for E3 or E5 linecards on Cisco 12Ks? Or maybe DFCs on Cisco 6500s/7600s? Or possibly M-series linecards on Cisco N7Ks (which are switches, of course)? TANSTAAFL. OK, we could handle some sort of SYN flood. As noted previously, this is indeed the case. But what about 20 Gbps http flood with valid requests when each customer are real (and not spoofied) and they are sending huge post requests and hang on connection? Attacks of this nature generally leave a 'wake' or 'contrail' which is pretty easily spotted if one's statistical anomaly detection routines are optimal. Actually it could wait for active/inactive timeout and you will get bad news from ops guys about network downtime. As a network ops guy, I can assure you that you are incorrect, largely because you don't seem to understand the interplay of active flow timer, inactive flow timer, NetFlow cache size, NetFlow cache FIFOing, and normal flow cache baselines. But sflow will handle it with flying colors without delay. NetFlow handles it with flying colors without delay. What about destination http host detection with netflow? Could it extract "host" header from netflow? And drop only part of traffic to our own host? Of course not, for classical flow telemetry templates - but that's when one drops from the macroanlytical to the microanalytical. And flow telemetry doesn't 'drop' anything. For some reason, you don't mention Flexible NetFlow at all. It's true that it's taken a while to become practical to use (back when the then-Cisco NetFlow PM asked me to create the CLI grammar and syntax for FNF, I noted that it wouldn't take off until there was a decent control-plane interface for creating, configuring, and tearing down dynamic flow caches, as well as some degree of ASIC support on larger platforms), but now that the various 'SDN'-type provisioning mechanisms are being implemented, and now that at least partial FNF is supported to varying degrees on various ASICs, this will hopefully change. Netflow haven't any information about http headers but sflow has. See above. This isn't necessary, and it isn't possible at scale with s/Flow, either. What about same issue for dns flood when somebody flood out some certain host? You could detect this attack with netflow. But you could not extract information about certain type of DDoS attack and attacked domain. There's no need to do this with flow telemetry. Once the attack has been detected/classified/traced back, one drops to the microanalytical for situationally-appropriate mitigation. When we speaking about "very rough" DDoS attack mitigation and filtering we could use netflow. Not just 'very rough', see above. But when we are really care about network stability, customer service SLA and ability to filter malicious traffic with perfect precision we should use sflow. This is demonstrably incorrect. Many of the largest networks in the world successfully utilize NetFlow telemetry for all these purposes; they have for many years, and will continue to do so. [And, btw, nothing has 'perfect precision'.] That doesn't mean that NetFlow (or IPFIX) is perfect, and it doesn't mean that all implementations are perfect, and it doesn't mean that the ability to get more information about traffic via FNF or IPFIX EE mechanisms isn't desirable. But you are simply wrong about the utility of NetFlow and/or IPFIX with classical flow templates. I really like to hear feedback about my vision. See above. ------- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 15:12, Pavel Odintsov wrote: Looks like you haven't so much field experience with sflow. I could help and offer some real field experience below. I've already recounted my real-world operational experience with NetFlow. I have my own netflow collector implementation for netflow v5, netflow v9 and IPFIX (just check my repository Coding something and using something operationally are two different things. I'm not a coder, but I've used NetFlow operationally since 1998, primarily on Cisco platforms (some Junipers, but I don't know a lot about Juniper boxes). So you know about Mirkotik implementation of netflow (they have minimum possible active and inactive timeout - 60 seconds) ? Yes. That does not equate to a 60s delay in detection/classifying/tracing back a SYN-flood, or anything else. Or what about old Cisco routers which support only 180 seconds as active timeouts? I think you're referring to the *default* value for the active flow timer, which can of course be altered. Could they offer affordable time for telemetry delivery? Yes, because there has never been any such router, and also because cache size and other tunable parameters, as well as FIFOing out of flows when the cache is full, guarantees that very few flows of the type seen in DDoS traffic hang around in the cache for any appreciable length of time. Because not all netflow implementations are OK. And definitely some netflow implementations are broken. You can search the archives on this list and see my previous detailed explanation of NetFlow caveats on Cisco 6500/7600 with EARL6 and EARL7 ASICs. Your statements about it taking an inordinately long time to detect/classify/traceback SYN-floods and other types of DDoS attacks utilizing NetFlow implementations (with the exceptions of crippled implementations like the aforementioned EARL6/EARL7 and pre-Sup7 Cisco 4500) are simply untrue. --- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 14:41, Pavel Odintsov wrote: Could you describe they in details? Inconsistent stats, lack of ifindex information. But netflow __could__ delay telemetry up to 30 seconds (in case of huge syn/syn-ack flood for example) and you network will experience downtime. This is incorrect, and reflects an inaccurate understanding of how NetFlow/IPFIX actually works, in practice. It's often repeated by those with little or no operational experience with NetFlow/IPFIX. --- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 14:26, Pavel Odintsov wrote: From my own experience sflow should be selected if you are interested in internal packet payload (for dpi / ddos detection) or you need fast reaction time on some actions (ddos is best example). This does not match my experience. In particular, the implied canard about flow telemetry being inadequate for timely DDoS detection/classification/traceback grows tiresome, as it's used for that purpose every day, and works quite well. If one is also using an IDMS-type device to mitigate DDoS traffic, the device sees the whole packet, anyways. --- Roland Dobbins <rdobb...@arbor.net>
Re: sFlow vs netFlow/IPFIX
On 29 Feb 2016, at 6:26, Baldur Norddahl wrote: > Around here they are currently voting on a law that will require unsampled > 1:1 netflow on all data in an ISP network with more than 100 users. That's interesting, given that most larger routers don't support 1:1. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 8:06, Keith Medcalf wrote: Consumer Narrowband Access Networks use these protocols all the time. Most broadband access customers do not actively use these protocols, themselves, with the partial exception of SIP. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 7:59, John Levine wrote: I think that most if not all of the consumer over the top VoIP phones like Vonage use SIP. That's true. One would hope that they're not globally reachable, however. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 7:23, John Levine wrote: The VoIP phones sure use SIP. True, but how prevalent are 'bare' SIP phones vs. VoIP systems utilized by remote workers via VPNs? --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 4:03, John Levine wrote: A certain number of us work from home and connect to headquarters with a VPN. and have SIP phones, you know. Not typically via/requiring the protocols you mentioned. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 22:52, Jay Nugent wrote: Customers regularly use various VPN protocols from GRE, SIT, and IPIP, monitoring protocols such as SNMP, as well as RTP and SIP (where we spend the bulk of our time troubleshooting). Not so on consumer broadband access networks, which are what's being discussed in this thread. It's a different story for transit operators. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 0:25, Anthony Junk wrote: There is so much arrogance in these posts saying that these things should be blocked because it's best or because it's negligible. I think there's a lack of comprehension on the part of those who don't run large networks and/or who aren't responsible for maintaing network availability for their customer base. Blocking or QoSing down port-pairs at the peering/transit edge is all too often the least worst option these operators have Please tell me again about my need for a business connection. Caveat emptor. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 0:16, Brielle Bruns wrote: You can't do anything about idiots buying a pro-sumer/professional device like an EdgeRouter and misconfiguring it, but Linksys/Cisco, D-Link, Netgear, etc that are targeted towards home users should be held to the fire for that kind of screw up. Also, see this article: <http://arstechnica.com/security/2016/02/asus-lawsuit-puts-entire-industry-on-notice-over-shoddy-router-security/> and this .pdf preso: <https://app.box.com/s/rblnddlhda44giwfa8hy> --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 27 Feb 2016, at 0:16, Brielle Bruns wrote: UDP is a fun protocol - stateless, so blocking a DST of 53/UDP to the customer also will block responses to recursive queries that originate from SRC 53/UDP. Which are relatively rare, these days. Any device doing this by default is likely running out-of-date software that is abusable in multiple ways. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 23:44, Blake Hudson wrote: Jason, how do you propose to block SSDP without also blocking legitimate traffic as well (since SSDP uses a port > 1024 and is used as part of the ephemeral port range on some devices) ? I'm not Jason, but blocking specific port-pairs such as UDP/80 ---> UDP/1900 and UDP/443 ---> UDP/1900 solves close to 90% of the problem, as UDP/80 and UDP/443 are the most common destination ports leveraged in this type of attack. For an explanation of how UDP reflection/amplification attacks work, see this .pdf preso: <https://app.box.com/s/r7an1moswtc7ce58f8gg> ------- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 23:15, Mike Hammett wrote: I think you'd be hard pressed to find more than a tenth of a percent of people attempt to run their own DNS server. You'll find a heck of a lot more of them doing so unknowingly, because they're running misconfigured, abusable CPE devices which can be leveraged by attackers to launch DNS reflection/amplification attacks. Note that outbound/crossbound DDoS attacks can have just as much of a negative impact on availability as inbound DDoS attacks; even more, when multiple attackers are abusing the same reflectors/amplifiers (which is often the case). And even that small tenth of a percent who're deliberately running their own DNS servers can end up inadvertently causing disruption if they're running those DNS servers as open recursors. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 23:02, Damian Menscher via NANOG wrote: What I'd much rather see Comcast do is use their netflow to trace the source of the spoofed packets (one of their peers or transit providers, no doubt) and strongly encourage (using their legal or PR team as needed) them to trace back and stop the spoofing. These approaches aren't necessarily mutually exclusive, as most flow telemetry implementations still report on blocked traffic from exporting devices. Keeping the network up and available for the vast majority of the customer base trumps all other considerations. DNS queries should not typically be directed towards consumer broadband access netblocks, period; and when they cause operational problems due to abusable CPE being, well, abused, immediate remediation action(s) must be taken. To do otherwise would be irresponsible. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 20:17, Nick Hilliard wrote: If you block packets with udp src port=53 towards customers, you will also block legitimate return traffic if the customers run their own DNS servers or use opendns / google dns / etc. Actually, what they're talking about is blocking packets *destined* for UDP/53 on broadband access networks, not *sourced from*. --- Roland Dobbins <rdobb...@arbor.net>
Re: Thank you, Comcast.
On 26 Feb 2016, at 10:52, Paras Jha wrote: You don't typically see DNS amplified floods coming from home ISPs. Actually, it's quite common, as a lot of CPE have abusable DNS forwarders running on their public interfaces. DNS, SSDP, and SNMP reflection/amplification quite commonly emanate from broadband access networks due to abusable CPE. Others, as well, of course, but those are generally the most prevalent. --- Roland Dobbins <rdobb...@arbor.net>
Re: UDP Amplification DDoS - Help!
On 9 Feb 2016, at 9:50, mike.l...@gmail.com wrote: Sounds like there is a compromised host downstream of the 1G that is reporting back it's source IP and that is why changing the IP doesn't help. It's much more likely that the attacker is just following the DNS changes. --- Roland Dobbins <rdobb...@arbor.net>
Re: UDP Amplification DDoS - Help!
On 9 Feb 2016, at 6:14, Mitch Dyer wrote: I'm hoping someone with some experience on this topic would be able to shed some light on a better way to attack this or would be willing to confirm that we are simply SOL without prolonged assistance from the upstream carrier. Take a look at this .pdf preso: <https://app.box.com/s/r7an1moswtc7ce58f8gg> Who's the upstream? Is it the sole upstream You may well not be speaking to the right folks there, the ones who can provide assistance. Also note that there are multiple overlay MSSPs who can potentially help, as well, apart from the immediate upstream. --- Roland Dobbins <rdobb...@arbor.net>
Re: Netflix NOC? VPN Mismarked?
On 29 Jan 2016, at 0:05, Crane, Todd wrote: > Imagine the issues if EoL'ed and EoS'ed those iPads. Um, I think they are . . . --- Roland Dobbins <rdobb...@arbor.net>
Re: Netflix stuffing data on pipe
On 30 Dec 2015, at 19:42, Matt Hoppes wrote: I'm seeing switch buffers getting overrun by what appears to be Netflix traffic coming in at rates faster than the subscribers throttled speeds. By what mechanism is the throttling accomplished? QoS on routers, or some kind of middlebox, or . . . ? --- Roland Dobbins <rdobb...@arbor.net>
Re: John McAfee: Massive DDoS attack on the internet was from smartphone botnet on popular app
On 13 Dec 2015, at 0:23, Jim Shankland wrote: Am I missing something, or is an even distribution of originating IP addresses virtually impossible *without* using spoofing? If his remarks were reported correctly, they are incorrect. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 8 Dec 2015, at 14:24, Joe Morgan wrote: At the point in time we blackholed our ip we were seeing 20+Gbps. These two presos discuss extortion DDoS and UDP reflection/amplification attacks, specifically - it isn't necessary to resort to D/RTBH to deal with these attacks: <https://app.box.com/s/776tkb82634ewvzvp26nnout6v4ij39q> <https://app.box.com/s/r7an1moswtc7ce58f8gg> --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 10 Dec 2015, at 13:21, Joe Morgan wrote: We have custom in house software that watches the traffic flows from our edge routers and automatically blackholes any ip getting targeted. Suggest you take a look at the presos I posted earlier and look into S/RTBH, flowspec, some limited QoS, and some preemptive ACLs so that you aren't forced into completing the DDoS. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 9 Dec 2015, at 11:46, Jean-Francois Mezei wrote: Since the OP mentioned a "ransom" demand (aka: extortion), should law enforcement be contacted in such cases ? Yes. Is there any experience doing this ? Yes. Are they any help ? Operationally, no. Investigatively, possibly. In North america, would that mean FBI in USA and RCMP in Canada Yes. or local police force which then escalates to proper law enforcement agency ? If you're asking about US and/or Canada, the relevant national LEA generally applies. In other jurisdictions, it's situationally-specific. ------- Roland Dobbins <rdobb...@arbor.net>
Re: Questions regarding equipment for a large LAN event
On 7 Dec 2015, at 13:41, Laurent Dumont wrote: > I appreciate any input on the matter! 1. cisco-nsp is a better list for this type of question. 2. The ASR9K is an edge router, not an access switch. 3. Why not just ask Cisco, for starters? --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 4 Dec 2015, at 9:34, alvin nanog wrote: all that tcpdump jibberish Is entirely unnecessary, as well as being completely impractical on a network of any size. Reasonable network access policies for the entities under attack plus flow telemetry collection/analysis, S/RTBH, and/or flowspec are a good start, along with this: <http://www.merit.edu/mail.archives/nanog/msg03776.html> This business of attempting to use packet captures for everything is the equivalent of your doctor attempting to diagnose the reason you're running a fever by using an electron microscope. Start with the BCPs, then move to the macroanalytical. Only dip into the microanalytical when required, and even then, do so very selectively. --- Roland Dobbins <rdobb...@arbor.net>
Re: Staring Down the Armada Collective
On 4 Dec 2015, at 9:28, Lyndon Nerenberg wrote: Are we perhaps, finally, reaching the cusp where everyone has realized that if we all, collectively, tell the rodents to f*** off, they just might? By my very rough and subjective guesstimate, extortion is the motivation behind ~15% of all DDoS attacks, FYI. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 3 Dec 2015, at 22:26, Nick Hilliard wrote: > If you believe that someone who issues a ransom threat will stop if you pay > them off, you're smoking crack. +1 These attacks aren't rocket-science to defend against. OP, ping me 1:1. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 3 Dec 2015, at 15:15, halp us wrote: Based on certain details that I can't reveal here, we believe the magnitude of the upcoming attack may be in the several hundred Gbps. They lie. The largest attacks we've seen from these threat actors are in the ~60gb/sec range - which is nothing to shake a stick at, mind. Many times, they don't follow through. But you're right to be prepared. See these two presos: <https://app.box.com/s/2kpbqfdl1ko3qhfhe4y8ekd1rvj24vfd> <https://app.box.com/s/r7an1moswtc7ce58f8gg> I would really appreciate help in a few areas (primarily with certain provider contacts/intros) so we can execute our strategy (which I can't reveal here for obvious reasons). All this super-secret squirrel stuff doesn't help, it's actually a hindrance. The short answer is 'upstream ACLs'. Nevertheless, contact me 1:1 and I'll work to hook you up with the right folks. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 3 Dec 2015, at 22:04, Josh Reynolds wrote: > None of those names you just mentioned have made the international news. Of course they have. --- Roland Dobbins <rdobb...@arbor.net>
Re: Ransom DDoS attack - need help!
On 4 Dec 2015, at 2:38, Dovid Bender wrote: > The last I spoke with NTT they said the largest they ever saw was > 300GB That wasn't DD4BC or Armada Collective. --- Roland Dobbins <rdobb...@arbor.net>
Re: strategies to mitigate DNS amplification attacks in ISP network
On 2 Dec 2015, at 0:14, Roland Dobbins wrote: Until the happy day when we've achieved universal source-address validation arrives, various combinations of the above. I forgot to mention RRL on authoritative servers, apologies. --- Roland Dobbins <rdobb...@arbor.net>
Re: strategies to mitigate DNS amplification attacks in ISP network
On 1 Dec 2015, at 23:59, Martin T wrote: What are the common practices to mitigate DNS amplification attacks in ISP network? Situationally-appropriate network access policies instantiated as ACLs on hardware-based routers/layer-3 switches in IDCs, on customer aggregation routers, in mitigation centers, etc. S/RTBH. flowspec. IDMS (full disclosure, I work for a vendor of such systems). See this .pdf preso: <https://app.box.com/s/r7an1moswtc7ce58f8gg> Statefulness is out, as you indicate. QoS is out, as you indicated (e.g., legitimate traffic is 'crowded out' by programmatically-generated attack traffic). The real solution to this entire problem set is source-address validation, as you indicate. Until the happy day when we've achieved universal source-address validation arrives, various combinations of the above. --- Roland Dobbins <rdobb...@arbor.net>
