Re: Cogent NOC
I think people are just going to see a traceroute determining packet loss and not going to read the rest of what happened. Just going to shortcut to an answer. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Randy" <a...@djlab.com> To: "Nanog" <nanog@nanog.org> Sent: Wednesday, December 14, 2016 1:16:41 PM Subject: Cogent NOC Hi all, Anyone beyond front line support at cogento on list? Nanog is the last place I'd look for assistance but it seems support over at cogentco is not nearly what it used to be. Example MTR to cogen't own website (support doesn't utilize or understand MTR at all apparently): Host Loss% Snt Last Avg Best Wrst StDev 1. x.x.x.x 0.0% 196 0.5 11.7 0.3 186.8 35.2 2. x.x.x.x 0.0% 196 0.6 10.2 0.4 226.3 36.2 3. 38.88.249.209 0.0% 196 0.9 1.1 0.7 17.7 1.2 4. te0-0-2-3.nr13.b023801-0.iad01.atl 0.0% 196 1.0 1.0 0.8 2.0 0.1 5. te0-0-0-1.rcr22.iad01.atlas.cogent 2.0% 196 2.1 1.9 1.0 3.3 0.4 6. be2961.ccr41.iad02.atlas.cogentco. 2.6% 196 1.8 2.1 1.1 3.8 0.5 7. be2954.rcr21.iad03.atlas.cogentco. 2.6% 196 2.0 2.3 1.2 9.4 0.7 8. be2952.agr11.iad03.atlas.cogentco. 0.5% 196 2.7 2.6 1.5 6.8 0.6 9. cogentco.com 4.1% 196 2.1 2.0 1.0 16.8 1.1 Pretty much the same to anywhere. Packet loss begins at rcr22.iad01 and propagates all the way down the line. Worse during peak hours, gone late at night. After three days of no email response for my ticket, I called and after an hour of my life I want back, front line support cannot reproduce the loss. Final conclusion: "Your host is dropping packets". -- ~Randy
Re: Favorite Speed Test Systems
A lot of people can't differentiate between what the test is testing, a bad test and connectivity issues producing bad results on an otherwise good test. I'd say that most of the time, it's the last category. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, December 5, 2016 12:42:56 PM Subject: Re: Favorite Speed Test Systems Right, it's mostly ISPs that don't understand the BGP world or how speedtests work. I think, you, Paul and myself were the only ones participating that really knew. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, December 5, 2016 10:28:22 AM Subject: Re: Favorite Speed Test Systems There was an afmug thread about this exact issue several months ago. On Dec 5, 2016 9:57 AM, "Mike Hammett" < na...@ics-il.net > wrote: Ah, this is the first I've heard of slow fast.com performance with someone actually connected to them. Usually it's an ISP that's a few AS hops away from Netflix. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Josh Reynolds" < j...@kyneticwifi.com > To: "Steven Miano" < mian...@gmail.com > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, December 5, 2016 9:51:30 AM Subject: Re: Favorite Speed Test Systems A lot of people have crappy performance to those. For example, from a 10G server to fast.com I was pulling around 9Mbps up/down. 1 hop away from a Netflix open connect appliance. On Dec 5, 2016 9:49 AM, "Steven Miano" < mian...@gmail.com > wrote: > fast.com is a dead fast/simple download result page. > > ...also with a huge customer base - it is often closer to > speedtest..net|com than some of those others. > > There is also a speedtest-cli available on Linux/MacOS (via Brew). > > On Mon, Dec 5, 2016 at 9:50 AM, Graham Johnston < johnst...@westmancom.com > > wrote: > > > For many years we have had a local instance of the Ookla speedtest.net > on > > our network, and while it is pretty good some other tests seem include > more > > detailed results. > > > > I am aware of the following speedtest systems that an operator can likely > > have a local instance of: > > > > * Speedtest.net > > > > * Sourceforge.net/speedtest > > > > * Dslreports.com/speedtest > > > > Are there others? What is your preferred one and why? > > > > Thanks, > > Graham > > > > > > > -- > Miano, Steven M. > http://stevenmiano.com >
Re: Favorite Speed Test Systems
Right, it's mostly ISPs that don't understand the BGP world or how speedtests work. I think, you, Paul and myself were the only ones participating that really knew. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, December 5, 2016 10:28:22 AM Subject: Re: Favorite Speed Test Systems There was an afmug thread about this exact issue several months ago. On Dec 5, 2016 9:57 AM, "Mike Hammett" < na...@ics-il.net > wrote: Ah, this is the first I've heard of slow fast.com performance with someone actually connected to them. Usually it's an ISP that's a few AS hops away from Netflix. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Josh Reynolds" < j...@kyneticwifi.com > To: "Steven Miano" < mian...@gmail.com > Cc: "NANOG" < nanog@nanog.org > Sent: Monday, December 5, 2016 9:51:30 AM Subject: Re: Favorite Speed Test Systems A lot of people have crappy performance to those. For example, from a 10G server to fast.com I was pulling around 9Mbps up/down. 1 hop away from a Netflix open connect appliance. On Dec 5, 2016 9:49 AM, "Steven Miano" < mian...@gmail.com > wrote: > fast.com is a dead fast/simple download result page. > > ...also with a huge customer base - it is often closer to > speedtest..net|com than some of those others. > > There is also a speedtest-cli available on Linux/MacOS (via Brew). > > On Mon, Dec 5, 2016 at 9:50 AM, Graham Johnston < johnst...@westmancom.com > > wrote: > > > For many years we have had a local instance of the Ookla speedtest.net > on > > our network, and while it is pretty good some other tests seem include > more > > detailed results. > > > > I am aware of the following speedtest systems that an operator can likely > > have a local instance of: > > > > * Speedtest.net > > > > * Sourceforge.net/speedtest > > > > * Dslreports.com/speedtest > > > > Are there others? What is your preferred one and why? > > > > Thanks, > > Graham > > > > > > > -- > Miano, Steven M. > http://stevenmiano.com >
Re: Favorite Speed Test Systems
Ah, this is the first I've heard of slow fast.com performance with someone actually connected to them. Usually it's an ISP that's a few AS hops away from Netflix. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Steven Miano" <mian...@gmail.com> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, December 5, 2016 9:51:30 AM Subject: Re: Favorite Speed Test Systems A lot of people have crappy performance to those. For example, from a 10G server to fast.com I was pulling around 9Mbps up/down. 1 hop away from a Netflix open connect appliance. On Dec 5, 2016 9:49 AM, "Steven Miano" <mian...@gmail.com> wrote: > fast.com is a dead fast/simple download result page. > > ...also with a huge customer base - it is often closer to > speedtest..net|com than some of those others. > > There is also a speedtest-cli available on Linux/MacOS (via Brew). > > On Mon, Dec 5, 2016 at 9:50 AM, Graham Johnston <johnst...@westmancom.com> > wrote: > > > For many years we have had a local instance of the Ookla speedtest.net > on > > our network, and while it is pretty good some other tests seem include > more > > detailed results. > > > > I am aware of the following speedtest systems that an operator can likely > > have a local instance of: > > > > * Speedtest.net > > > > * Sourceforge.net/speedtest > > > > * Dslreports.com/speedtest > > > > Are there others? What is your preferred one and why? > > > > Thanks, > > Graham > > > > > > > -- > Miano, Steven M. > http://stevenmiano.com >
Re: Facebook Geo Routing Issues
I'm in Chicago and I saw mine going to Miami as well (per rDNS). Haven't looked into it at all. I did see a video where they said they occasionally purposely give people less than ideal facilities to test connectivity. Maybe that process buggered up? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "John Cenile" <jcenile1...@gmail.com> To: nanog@nanog.org Sent: Wednesday, November 16, 2016 7:48:25 PM Subject: Facebook Geo Routing Issues Hello, Does anybody have a contact I could use at Facebook to get a routing issue resolved? Some of our networks are being routed to Miami, rather than using the much closer PoP of Sydney, and it's obviously causing significant performance issues when browsing Facebook.
Re: Port 2323/tcp
Probably best to go with A) what we could do in the best of situations and B) what the rest will do. Some of us are last mile networks and *DO* care. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mel Beckman" <m...@beckman.org> To: l...@satchell.net Cc: nanog@nanog.org Sent: Wednesday, November 16, 2016 11:25:34 AM Subject: Re: Port 2323/tcp It's pretty much part of the IBR now. And what can a provider do, really? It's not likely he will expend much effort blocking customers. Maybe we should all start filtering 2323? -mel via cell > On Nov 16, 2016, at 11:53 AM, Stephen Satchell <l...@satchell.net> wrote: > > I've been seeing a lot of rejections in my logs for 2323/tcp. According > to the Storm Center, this is what the Mirai botnet scanner uses to look > for other target devices. > > Is it worthwhile to report sightings to the appropriate abuse addresses? > (That assumes there *is* an abuse address associated with the IPv4 > address that is the source.) Would administrations receiving these > notices do anything with them? > > Alternatively, is there anyone collecting this information from people > like me to expose the IP addresses of possible infections? > > I am toying with the idea of setting up a honey-pot, but I'm so far > behind with $DAYJOB that such a project will have to wait a bit. > > I want to be a good net citizen. I also want to make sure I'm not > wasting my time. > > Today's crop: > >> 1.34.169.183 >> 12.221.236.2 >> 14.138.22.12 >> 14.169.142.30 >> 14.174.71.158 >> 14.177.197.101 >> 31.168.146.33 >> 31.168.212.174 >> 36.71.224.179 >> 36.72.253.206 >> 37.106.18.86 >> 42.115.187.189 >> 42.117.254.248 >> 42.119.228.222 >> 43.225.195.180 >> 46.59.6.249 >> 49.114.192.91 >> 58.11.238.146 >> 58.186.231.59 >> 59.8.136.21 >> 59.49.191.4 >> 59.57.68.56 >> 59.126.35.47 >> 59.126.242.70 >> 59.127.104.67 >> 59.127.242.8 >> 60.251.125.125 >> 61.219.165.38 >> 73.84.152.194 >> 78.179.113.148 >> 78.186.61.30 >> 78.189.169.142 >> 78.226.222.234 >> 79.119.74.255 >> 81.16.8.193 >> 81.101.233.14 >> 81.214.121.43 >> 81.214.134.133 >> 81.214.137.197 >> 82.77.68.189 >> 83.233.40.141 >> 85.96.202.199 >> 85.99.121.41 >> 85.238.103.111 >> 86.121.225.48 >> 87.251.252.22 >> 88.249.224.167 >> 89.122.87.239 >> 89.151.128.198 >> 90.177.91.201 >> 92.53.52.235 >> 92.55.231.90 >> 94.31.239.178 >> 94.254.41.152 >> 94.255.162.90 >> 95.78.245.54 >> 95.106.34.92 >> 95.161.236.182 >> 96.57.103.19 >> 101.0.43.13 >> 108.203.68.245 >> 110.55.108.215 >> 110.136.233.10 >> 112.133.69.176 >> 112.165.93.130 >> 112.186.42.216 >> 113.5.224.110 >> 113.161.64.11 >> 113.169.18.153 >> 113.171.98.158 >> 113.172.4.204 >> 113.183.204.112 >> 113.188.44.246 >> 114.32.28.219 >> 114.32.87.32 >> 114.32.189.5 >> 114.34.29.167 >> 114.34.170.10 >> 114.35.153.123 >> 114.226.53.133 >> 115.76.127.118 >> 116.73.65.248 >> 116.100.170.92 >> 117.0.7.77 >> 117.1.26.234 >> 117.195.254.3 >> 118.32.44.99 >> 118.42.15.21 >> 118.43.112.120 >> 118.100.64.159 >> 118.163.191.208 >> 119.199.160.207 >> 119.202.78.47 >> 120.71.215.81 >> 121.129.203.22 >> 121.178.104.129 >> 121.180.53.143 >> 122.117.245.28 >> 123.9.72.86 >> 123.16.78.77 >> 123.23.49.149 >> 123.24.108.10 >> 123.24.250.187 >> 123.25.74.209 >> 123.27.159.13 >> 123.240.245.72 >> 124.66.99.251 >> 124.131.28.38 >> 125.166.193.206 >> 125.227.138.132 >> 138.204.203.66 >> 171.97.245.221 >> 171.224.7.147 >> 171.226.20.220 >> 171.232.118.93 >> 171.248.210.120 >> 171.249.223.213 >> 171.250.26.209 >> 173.56.21.67 >> 175.138.81.130 >> 175.203.202.232 >> 175.207.137.139 >> 175.211.251.156 >> 177.207.49.108 >> 177.207.67.170 >> 177.223.52.193 >> 178.222.246.96 >> 179.4.140.63 >> 179.235.55.39 >> 179.253.163.107 >> 180.73.117.62 >> 180.254.224.10 >> 182.37.156.98 >> 182.180.80.75 >> 182.180.123.43 >> 183.46.49.216 >> 183.144.245.235 >> 186.19.48.158 >> 186.69.170.130 >> 186.219.1.156 >> 187.104.248.17 >> 187.211.63.51 >> 188.209.153.15 >> 189.101.220.244 >> 189.234.9.147 >> 191.103.35.250 >> 191.180.198.31 >> 191.249.21.41 >> 196.207.83.23 >> 197.224.37.108 >> 201.243.225.103 >> 210.178.250.121 >> 211.7.146.51 >> 211.216.202.191 >> 213.5.216.213 >> 213.14.195.100 >> 213.170.76.149 >> 217.129.243.48 >> 218.161.121.178 >> 218.186.43.224 >> 220.85.169.133 >> 220.132.111.124 >> 220.133.24.142 >> 220.133.198.71 >> 220.133.234.229 >> 220.134.132.200 >> 220.134.193.133 >> 220.135.64.43 >> 221.145.147.78 >> 221.159.105.17 >> 221.167.64.53 >> 222.254.238.188 >> 223.154.223.159 >
Re: Dyn DDoS this AM?
Side note: I asked Mikrotik and they accepted the feature request of changing their uRPF setting from being universal on the machine to being per-interface (as the kernel supports). That would make it easier for Mikrotik end-user-facing routers to block crap right at the edge, allowing for strict facing customer and loose elsewhere. They haven't implemented it yet, but they accepted the request. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Alexander Lyamin" <l...@qrator.net> To: "Ronald F. Guilmette" <r...@tristatelogic.com> Cc: "NANOG list" <nanog@nanog.org> Sent: Tuesday, October 25, 2016 3:29:56 AM Subject: Re: Dyn DDoS this AM? Yeah, it sucked to be a Dyn customer that day. However, if you had a backup dns provider, it wasnt that bad. You do realize that collateral effect scale is a property of a target and not attack? My point was that implementing MANRS, while isn't covering all of the spectrum of the attacks that made news this autumn will make at least some of them if not impossible, but harder to execute. And as I said - its work in progress. P.S. Jared Mauch notes regarding uRPF underperformance are correct, but it only shows how rarely its actually used in a real life. uRPF is more then feasible in terms of algorithmical complexity, and this means that bugs can be dealed with. On Tue, Oct 25, 2016 at 7:30 AM, Ronald F. Guilmette <r...@tristatelogic.com> wrote: > > In message
Re: Spitballing IoT Security
Oh, yeah, list e-mail usually just gets skimmed through. No time for reading in detail or links. ;-) Sorry. :-\ - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Hugo Slabbert" <h...@slabnet.com> To: "Mike Hammett" <na...@ics-il.net> Cc: nanog@nanog.org Sent: Monday, October 24, 2016 5:21:48 PM Subject: Re: Spitballing IoT Security It's possible you might have wanted to read the link for the context that pointed this out as sarcastic hyperbole, though the text as-is could (unfortunately) have been read as serious. -- Hugo Slabbert | email, xmpp/jabber: h...@slabnet.com pgp key: B178313E | also on Signal On Mon 2016-Oct-24 17:17:43 -0500, Mike Hammett <na...@ics-il.net> wrote: >There's a buffer overrun in some software, so let's just remove all passwords >(and keys), since they can get in anyway. > > > > > >Just pointing out flawed logic. > > > > >- >Mike Hammett >Intelligent Computing Solutions >http://www.ics-il.com > >Midwest-IX >http://www.midwest-ix.com > >- Original Message - > >From: "J. Oquendo" <joque...@e-fensive.net> >To: "Steve Mikulasik" <steve.mikula...@civeo.com> >Cc: nanog@nanog.org >Sent: Monday, October 24, 2016 3:53:25 PM >Subject: Re: Spitballing IoT Security > >On Mon, 24 Oct 2016, Steve Mikulasik wrote: > >> if we automatically blackholed those IPs as they get updated it could put a >> big dent in the effectiveness of Zeus. >> > >That would involve someone lifting a finger and implement >a config change. Much easier to implement BCP38 or was it >RFC 4732? Would never work the moment someone has to lift >a finger. > >/* >I think I'll change my position on BCP38. It's pointless to try >blocking spoofed source addresses because: > >* It doesn't solve every single problem >* It means more effort for service providers >* It requires more CPU processing power >* Using it will generate smarter "black hats". > >https://www.nanog.org/mailinglist/mailarchives/old_archive/2004-10/msg00132.html > > >*/ > > >-- >=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ >J. Oquendo >SGFA, SGFE, C|EH, CNDA, CHFI, OSCP, CPT, RWSP, GREM > >"Where ignorance is our master, there is no possibility of >real peace" - Dalai Lama > >0B23 595C F07C 6092 8AEB 074B FC83 7AF5 9D8A 4463 >https://pgp.mit.edu/pks/lookup?op=get=0xFC837AF59D8A4463 >
Re: Spitballing IoT Security
There's a buffer overrun in some software, so let's just remove all passwords (and keys), since they can get in anyway. Just pointing out flawed logic. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "J. Oquendo" <joque...@e-fensive.net> To: "Steve Mikulasik" <steve.mikula...@civeo.com> Cc: nanog@nanog.org Sent: Monday, October 24, 2016 3:53:25 PM Subject: Re: Spitballing IoT Security On Mon, 24 Oct 2016, Steve Mikulasik wrote: > if we automatically blackholed those IPs as they get updated it could put a > big dent in the effectiveness of Zeus. > That would involve someone lifting a finger and implement a config change. Much easier to implement BCP38 or was it RFC 4732? Would never work the moment someone has to lift a finger. /* I think I'll change my position on BCP38. It's pointless to try blocking spoofed source addresses because: * It doesn't solve every single problem * It means more effort for service providers * It requires more CPU processing power * Using it will generate smarter "black hats". https://www.nanog.org/mailinglist/mailarchives/old_archive/2004-10/msg00132.html */ -- =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP, CPT, RWSP, GREM "Where ignorance is our master, there is no possibility of real peace" - Dalai Lama 0B23 595C F07C 6092 8AEB 074B FC83 7AF5 9D8A 4463 https://pgp.mit.edu/pks/lookup?op=get=0xFC837AF59D8A4463
Re: Death of the Internet, Film at 11
A support call to an end-user serving ISP takes how long to ROI? That wouldn't make sense. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Keith Medcalf" <kmedc...@dessus.com> To: "Ronald F. Guilmette" <r...@tristatelogic.com> Cc: "NANOG" <nanog@nanog.org> Sent: Sunday, October 23, 2016 8:39:52 PM Subject: Re: Death of the Internet, Film at 11 Why would the provider want to do anything? They suuport (make money from) their cudtomers. And the more traffic the send/receive, the more money the providers make. Wouldn't surprise me if the providers were selling access to their customers networks to the botherders so they could make money from both ends. --- Sent from Samsung Mobile Original message From: "Ronald F. Guilmette" <r...@tristatelogic.com> Date:2016-10-23 17:20 (GMT-07:00) To: Cc: nanog@nanog.org Subject: Re: Death of the Internet, Film at 11
Re: Death of the Internet, Film at 11
Thanks for the link. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ray Van Dolson" <rvandol...@esri.com> To: "Mike Hammett" <na...@ics-il.net> Cc: nanog@nanog.org Sent: Saturday, October 22, 2016 5:35:50 PM Subject: Re: Death of the Internet, Film at 11 https://urldefense.proofpoint.com/v2/url?u=http-3A__hub.dyn.com_dyn-2Dblog_dyn-2Dstatement-2Don-2D10-2D21-2D2016-2Dddos-2Dattack=DQIBAg=n6-cguzQvX_tUIrZOS_4Og=5PqhtPogDeswmEQMQZk1IQ=6rpDhHbntFiyuuA6uUxOIVfEwHY13H9SH6zBwx93OBE=QIsYvf_c8f_VWuMbYe7DbF58d1UqsbxJBEjf8CYotcc= On Sat, Oct 22, 2016 at 04:48:01PM -0500, Mike Hammett wrote: > Until Dyn says or someone says Dyn said, everything is assumed. > > - Original Message - > > From: "Peter Baldridge" <petebaldri...@gmail.com> > To: "Jean-Francois Mezei" <jfmezei_na...@vaxination.ca> > Cc: nanog@nanog.org > Sent: Saturday, October 22, 2016 4:45:13 PM > Subject: Re: Death of the Internet, Film at 11 > > On Sat, Oct 22, 2016 at 1:47 PM, Jean-Francois Mezei > <jfmezei_na...@vaxination.ca> wrote: > > Generic question: > > > > The media seems to have concluded it was an "internet of things" that > > caused this DDoS. > > > > I have not seen any evidence of this. Has this been published by an > > authoritative source or is it just assumed? > > Flashpoint[0], krebs[1], arstechnica[2]. I'm not sure what credible > looks like unless they release a packet but this is probably > consensus. > > > Has the type of device involved been identified? > > routers and cameras with shitty firmware [3] > > > Is it more plausible that those devices were "hacked" in the OEM > > firmware and sold with the "virus" built-in ? That would explain the > > widespread attack. > > The source code has been released. krebs [4], code [5] > > > Also, in cases such as this one, while the target has managed to > > mitigate the attack, how long would such an attack typically continue > > and require blocking ? > This is an actual question that hasn't been answered. > > > Since the attack seemed focused on eastern USA DNS servers, would it be > > fair to assume that the attacks came mostly from the same region (aka: > > devices installed in eastern USA) ? (since anycast would point them to > > that). > > Aren't heat maps just population graphs? > > > BTW, normally, if you change the "web" password on a "device", it would > > also change telnet/SSH/ftp passwords. > > Seems like no one is doing either.
Re: Death of the Internet, Film at 11
Until Dyn says or someone says Dyn said, everything is assumed. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Peter Baldridge" <petebaldri...@gmail.com> To: "Jean-Francois Mezei" <jfmezei_na...@vaxination.ca> Cc: nanog@nanog.org Sent: Saturday, October 22, 2016 4:45:13 PM Subject: Re: Death of the Internet, Film at 11 On Sat, Oct 22, 2016 at 1:47 PM, Jean-Francois Mezei <jfmezei_na...@vaxination.ca> wrote: > Generic question: > > The media seems to have concluded it was an "internet of things" that > caused this DDoS. > > I have not seen any evidence of this. Has this been published by an > authoritative source or is it just assumed? Flashpoint[0], krebs[1], arstechnica[2]. I'm not sure what credible looks like unless they release a packet but this is probably consensus. > Has the type of device involved been identified? routers and cameras with shitty firmware [3] > Is it more plausible that those devices were "hacked" in the OEM > firmware and sold with the "virus" built-in ? That would explain the > widespread attack. The source code has been released. krebs [4], code [5] > Also, in cases such as this one, while the target has managed to > mitigate the attack, how long would such an attack typically continue > and require blocking ? This is an actual question that hasn't been answered. > Since the attack seemed focused on eastern USA DNS servers, would it be > fair to assume that the attacks came mostly from the same region (aka: > devices installed in eastern USA) ? (since anycast would point them to > that). Aren't heat maps just population graphs? > BTW, normally, if you change the "web" password on a "device", it would > also change telnet/SSH/ftp passwords. Seems like no one is doing either. [0] https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns-ddos-attacks/ [1] https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-todays-massive-internet-outage/ [2] http://arstechnica.com/security/2016/10/double-dip-internet-of-things-botnet-attack-felt-across-the-internet/ [3] https://blog.sucuri.net/2016/09/iot-home-router-botnet-leveraged-in-large-ddos-attack.html [4] https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-released/ [5] https://github.com/jgamblin/Mirai-Source-Code -- Pete Baldridge 206.992.2852
Re: Death of the Internet, Film at 11
It's also generally counter to them being available outside of that network. (web and proprietary interfaces needed, SSH and telnet not). That's also not much I can do as a network operator. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Chris Boyd" <cb...@gizmopartners.com> To: "Elizabeth Zwicky via NANOG" <nanog@nanog.org> Sent: Saturday, October 22, 2016 11:42:05 AM Subject: Re: Death of the Internet, Film at 11 > On Oct 22, 2016, at 7:34 AM, Mike Hammett <na...@ics-il.net> wrote: > > "taken all necessary steps to insure that none of the numerous specific types > of CCVT thingies that Krebs and others identified" > > Serious question... how? Putting them behind a firewall without general Internet access seems to work for us. We have a lot of cheap IP cameras in our facility and none of them can reach the net. But this is probably a bit beyond the capabilities of the general home user. —Chris
Re: Death of the Internet, Film at 11
Not trolling in the least. I'm genuinely trying my best to help the greater community. Agreed on ShadowServer. I get their reports and I recommend others do the same. Oh, okay, I responded to someone that said: = Every network operator who can do so, please raise your hand if you have *recently* scanned you own network and if you can -honestly- attest that you have taken all necessary steps to insure that none of the numerous specific types of CCVT thingies that Krebs and others identified weeks or months ago as being fundamentally insecure can emit a single packet out onto the public Internet. = That's the direction I was heading. How can I as a network operator seek out and eliminate the sources of these attacks? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Brandon Butterworth" <bran...@rd.bbc.co.uk> To: na...@ics-il.net Cc: nanog@nanog.org Sent: Saturday, October 22, 2016 10:02:42 AM Subject: Re: Death of the Internet, Film at 11 > From nanog-boun...@nanog.org Sat Oct 22 15:51:34 2016 > If they are easy to trace, then it should be easy for you to > tell me how to find them on my network. Not sure if you're trolling now, apologies if what I wrote wasn't clear. If you did want to find them before they attack then you could scan for them, the miscreants already did and easily found them. For some attack vectors there are services that are doing it for you, see the excellent https://www.shadowserver.org/wiki/pmwiki.php/Involve/GetReportsOnYourNetwork > The addresses being known to them doesn't help me at all clean > up my network or help other networks clean up theirs. Did you read my whole mail? The suggestion is people who get attacked tell the ISPs of the devices doing the attacking > It would be rather difficult for me (and I'm sure many other operators) > to distinguish normal Dyn traffic from DDoS Dyn traffic. I was not suggesting you try and guess, I was suggesting you be given data from actual attacks. brandon
Re: Death of the Internet, Film at 11
If they are easy to trace, then it should be easy for you to tell me how to find them on my network. The addresses being known to them doesn't help me at all clean up my network or help other networks clean up theirs. It would be rather difficult for me (and I'm sure many other operators) to distinguish normal Dyn traffic from DDoS Dyn traffic. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Brandon Butterworth" <bran...@rd.bbc.co.uk> To: na...@ics-il.net Cc: nanog@nanog.org Sent: Saturday, October 22, 2016 9:41:52 AM Subject: Re: Death of the Internet, Film at 11 > "their" Whose addresses are known The "CCVT thingies" you refer to. Unlike spoof attacks these are easy to trace > and who are they known to? Those who were attacked by them or worked on mitigation of the attack. If not this time then they should next time as there will be a next time. > Some work can produce Dyn allocations, I suppose. Indeed, that is what I was saying brandon
Re: Death of the Internet, Film at 11
"their" Whose addresses are known and who are they known to? I certainly don't know the addresses of anyone involved. Some work can produce Dyn allocations, I suppose. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Brandon Butterworth" <bran...@rd.bbc.co.uk> To: na...@ics-il.net Cc: nanog@nanog.org Sent: Saturday, October 22, 2016 9:22:55 AM Subject: Re: Death of the Internet, Film at 11 > From: Mike Hammett <na...@ics-il.net> > "taken all necessary steps to insure that none of the numerous specific types > of CCVT thingies that Krebs and others identified" > > Serious question... how? Well their addresses are now known so one way would be for each ISP to drop traffic from them. If people don't fix them why should these devices stay on the net? If say Comcast has a million of them it might be tricky to scale but not impossible It'd take a bit of effort and care to aggregate and disseminate the data to each responsible AS, there'd be risk of bad guys getting the data and false positives/people spoofing to attack others. They'd also be building a tool that some might try to hijack for other purposes. None of that is an excuse to do nothing as is usually the result with any suggested measure that involves doing work to fix a problem I know ISPs generaly don't want the support calls but they'll end up with them and a legislative burden with commerial liability if they don't sort it out themselves. brandon
Re: Death of the Internet, Film at 11
"taken all necessary steps to insure that none of the numerous specific types of CCVT thingies that Krebs and others identified" Serious question... how? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ronald F. Guilmette" <r...@tristatelogic.com> To: nanog@nanog.org Sent: Saturday, October 22, 2016 12:53:42 AM Subject: Re: Death of the Internet, Film at 11 Laszlo Hanyecz wrote: >What does BCP38 have to do with this? Your're right. That's not specifically related to *this* attack. Nobody needs to spoof anything when you've got a zillion fire hoses just lying around where any 13 year old can command them from the TRS 80 in his mom's basement. (I've seen different estimates today. One said there's about a half million of these things, but I think I saw where Dyn itself put the number of unique IPs in the attack at something like ten million.) I just threw out BCP 38 as an example of something *very* minimal that the collective Internet, if it had any brains, would have made de rigueur for everyone ten+ years ago. BCP 38 is something that I personally view as a "no brainer", that is already widely accepted as being necessary, and yet is a critical security step that some (many?) are still resisting. So, it's like "Well, if the Internet-at-large can't even do *this* simple and relatively non-controversial thing, then we haven't got a prayer in hell of ever seeing a world-wide determined push to find and neutralize all of these bloody damn stupid CCTV things. And when the day comes when somebody figures out how to remotely pop a default config Windoze XP box... boy oh boy, will *that* be a fun day... NOT! Because we're not ready. Nobody's ready. Except maybe DoD, and I'm not even taking bets on that one." I didn't intend to focus on BCP 38. Everybody knows that's only one thing, designed to deal with just one part of the overall problem. The overall problem, in my view, is the whole mindset which says "Oh, we just connect the wires. Everything else is somebody else's problem." Ok, so this mailing list is a list of network operators. Swell. Every network operator who can do so, please raise your hand if you have *recently* scanned you own network and if you can -honestly- attest that you have taken all necessary steps to insure that none of the numerous specific types of CCVT thingies that Krebs and others identified weeks or months ago as being fundamentally insecure can emit a single packet out onto the public Internet. And, cue the crickets... Recent events, like the Krebs DDoS and the even bigger OVH DDoS, and today's events make it perfectly clear to even the most blithering of blithering idiots that network operators, en mass, have to start scanning their own networks for insecurities. And you'd all better get on that, not next fiscal year or even next quarter, but right effing now, because the next major event is right around the corner. And remember, *you* may not be scanning your networks for easily pop'able boxes, but as we should all be crystal clear on by now, that *does not* mean that nobody else is doing so. Regards, rfg P.S. The old saying is that idle hands are the devil's playground. In the context of the various post-invasion insurgancies, etc., in Iraq, is is often mentioned that it was a somewhat less than a brilliant move for the U.S. to have disbanded the Iraq army, thereby leaving large numbers of trained young men on the streets with no jobs and nothing to do. To all of the network operators who think that (or argue that) it will be too expensive to hire professionals to come in an do the work to scan your networks for known vulnerabilities, I have a simple suggestion. Go down to your local high school, find the schmuck who teaches the kids about computers, and ask him for the name of his most clever student. Then hire that student and put him to work, scanning your network. As in Iraq, it will be *much* better to have capable young men inside the tent, pissing out, rather than the other way around.
Re: Death of the Internet, Film at 11
Block one type of attack enough times and you've accomplished something. Because script kiddies are taking advantage of published exploits doesn't mean we stop setting passwords on things. You have to protect from them all. No, no collateral damage. We discussed this a couple weeks ago and there was no credible evidence of collateral damage. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Laszlo Hanyecz" <las...@heliacal.net> To: nanog@nanog.org Sent: Friday, October 21, 2016 7:52:42 PM Subject: Re: Death of the Internet, Film at 11 On 2016-10-22 00:39, Ronald F. Guilmette wrote: > P.S. To all of you Ayn Rand devotees out there who still vociferously > argue that it's nobody else's business how you monitor or police your > "private" networks, and who still refuse to take even minimalist steps > (like BCP 38), congratulations. What does BCP38 have to do with this? All that does is block one specific type of attack (and cause a lot of collateral damage). The IoT devices do not need to spoof addresses - they can just generate attack traffic directly. This is even better, because you can't cut those eyeball addresses off - those are the same addresses your target audience is using. If you cut off the eyeball networks there's not much point to running an internet business website anymore. -Laszlo
Re: Dyn DDoS this AM?
Are there sites that can test your BCP38\84 compliance? I'm okay, but interested in what I can share to raise awareness. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Patrick W. Gilmore" <patr...@ianai.net> To: "NANOG list" <nanog@nanog.org> Sent: Friday, October 21, 2016 10:48:21 AM Subject: Re: Dyn DDoS this AM? I cannot give additional info other than what’s been on “public media”. However, I would very much like to say that this is a horrific trend on the Internet. The idea that someone can mention a DDoS then get DDoS’ed Can Not Stand. See Krebs’ on the Democratization of Censorship. See lots of other things. To Dyn and everyone else being attacked: The community is behind you. There are problems, but if we stick together, we can beat these miscreants. To the miscreants: You will not succeed. Search "churchill on the beaches”. It’s a bit melodramatic, but it’s how I feel at this moment. To the rest of the community: If you can help, please do. I know a lot of you are thinking “what can I do?" There is a lot you can do. BCP38 & BCP84 instantly come to mind. Sure, that doesn’t help Mirai, but it still helps. There are many other things you can do as well. But a lot of it is just willingness to help. When someone asks you to help trace an attack, do not let the request sit for a while. Damage is being done. Help your neighbor. When someone’s house is burning, your current project, your lunch break, whatever else you are doing is almost certainly less important. If we stick together and help each other, we can - we WILL - win this war. If we are apathetic, we have already lost. OK, enough motivational speaking for today. But take this to heart. Our biggest problem is people thinking they cannot or do not want to help. -- TTFN, patrick > On Oct 21, 2016, at 10:55 AM, Chris Grundemann <cgrundem...@gmail.com> wrote: > > Does anyone have any additional details? Seems to be over now, but I'm very > curious about the specifics of such a highly impactful attack (and it's > timing following NANOG 68)... > > https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twitter-spotify-reddit/ > > > -- > @ChrisGrundemann > http://chrisgrundemann.com
Re: Coherent CWDM 40G QSFP
Apparently I just remembered the big transport platforms using coherent 40G and 100G and assumed there was a cheap variant, but there isn't. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Tim Durack" <tdur...@gmail.com> To: "Mike Hammett" <na...@ics-il.net>, "nanog list" <nanog@nanog.org> Sent: Tuesday, October 18, 2016 9:28:02 PM Subject: Re: Coherent CWDM 40G QSFP Not aware of ACO/DCO in QSFP form factor. Inphi is doing 100G QSFP28 PAM4 DWDM for MS. Probably the best you will see for a while. On Tue, Oct 18, 2016 at 4:50 PM Mike Hammett < na...@ics-il.net > wrote: Does anyone make a coherent CWDM 40G QSFP? I thought so, but the first couple places I checked, I struck out at. This would be for a passive mux\MROADM. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: 18 years ago today - rfc 2468
"or you haven't read enough RFCs" so for those of us that aren't masochists ;-) I did get my summary last year at NANOG, though. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Wayne Bouchard" <w...@typo.org> To: "Patrick W. Gilmore" <patr...@ianai.net> Cc: "NANOG list" <nanog@nanog.org> Sent: Wednesday, October 19, 2016 2:44:31 AM Subject: Re: 18 years ago today - rfc 2468 And for those of you who you don't recognize his name, either you aren't old enough or you haven't read enough RFCs, though his contributions go wayyy beyond that. It is fair to say he is very much one of the cadre of personell who quite literally built the internet that so many of the rest now take for granted. On Sat, Oct 15, 2016 at 09:21:01AM -0400, Patrick W. Gilmore wrote: > We do. > > Thank you for reminding us. And thanks to Dr. Postel for making what we do > possible. > > -- > TTFN, > patrick > > > On Oct 15, 2016, at 9:19 AM, Rodney Joffe <rjo...@centergate.com> wrote: > > > > To be clear - Oct 16. Which has just tolled in the APAC region. For most of > > you it will be tomorrow. But no matter. You get the point. > > > >> On Oct 15, 2016, at 9:08 AM, Rodney Joffe <rjo...@centergate.com> wrote: > >> > >> How time flies > --- Wayne Bouchard w...@typo.org Network Dude http://www.typo.org/~web/
Coherent CWDM 40G QSFP
Does anyone make a coherent CWDM 40G QSFP? I thought so, but the first couple places I checked, I struck out at. This would be for a passive mux\MROADM. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: Two BGP peering sessions on single Comcast Fiber Connection?
It really seems like it's a grave oversight to *NOT* support multiple BGP sessions. I drop to two routers for that same reason, I can do maintenance on one, while the other carries traffic. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Poublon" <mpoub...@secantnet.net> To: "rar" <r...@syssrc.com>, nanog@nanog.org Sent: Thursday, October 13, 2016 2:04:29 PM Subject: Re: Two BGP peering sessions on single Comcast Fiber Connection? I started a thread around the same topic back on 10/16 of 2014. A Comcast engineer (who ultimately spoke to the national product manager) came back after discussing and said the same thing "We don't support that". I got a slightly longer explanation of: In a nutshell, when we design a product we do it to accommodate the most typical customer cases. Given that the design includes a single fiber path and thus the fiber path and device that terminates on either end each are a single point of failure, adding extra BGP sessions doesn’t seem to add value in the typical failure scenarios. In order to achieve the simplest and most scalable solution to address the market, we rely on narrowing the possible combinations of parameters. I explained to them that their interpretation prevents me from being able to do concurrent maintenance on my side (single router reboot/upgrade, etc). Never got anywhere with it though. I'm still interested in having this set up, but have given up on it ever really coming to reality. Luckily ALL of my other providers were more than happy to set up an extra session. If anyone from Comcast is listening, there is customer demand for this. It's not about making it better for Comcast, it's about allowing customers to have more flexibility. Mike Poublon /Senior Datacenter Network Engineer/ *Secant Technologies* 6395 Technology Ave. Suite A Kalamazoo, MI 49009 On 10/13/2016 1:48 PM, rar wrote: > After a many month wait, we were ready to turn up our BGP peering sessions on > a new Comcast fiber connection. > > With our other providers (Level 3 and Verizon) we have edge routers that > directly connect between the provider's on premise connection and our primary > and a backup core routers. Each core router has a multihop BGP session with > the provider's BGP router. The goal is to keep the single BGP router from > being a single point of failure. > > Comcast said they could not support two separate BGP peering sessions on the > same circuit. Does anyone have any counter examples? We used to have this > setup with Comcast 5+ years ago, but now they say they can't support it. > > > Bob Roswell > brosw...@syssrc.com<mailto:brosw...@syssrc.com> > 410-771-5544 ext 4336 > > Computer Museum Highlights<http://museum.syssrc.com/> >
Re: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos
I like putting a switch in front so then I can run two routers behind and get a /29 from the upstream. I can then do router maintenance, upgrades, etc. without taking the circuit down. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Pedro" <piotr.1...@interia.pl> To: nanog@nanog.org Sent: Friday, September 30, 2016 2:42:37 PM Subject: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos Hello, I have some idea to put switch before bgp router in order to terminate isp 10G uplinks on switch, not router. Main reason is that could be some kind of 1st level of defence against ddos, second reason, less important, save cost of router ports, do many port mirrors. I think about N3K-C3064PQ or Juniper ex4500 because there are quite cheap and a lot of on Ebay. I would like on nexus or juniper try use some feature: - limit udp, icmp, bum packets (bandwith,pps) at ingress tagged port or vlan - create counters: passed and dropped packets, best way to get this counters via snmp oid, sent snmp traps, syslog etc in order to monitor or even as a action shut down port - port mirror from many ports/vlans to multiple port (other anty ddos solutions) - limited bgp but with flowspec to comunicate with another anty ddos devices I'm also wondering how this feature above impact on cpu/whole switch. It can be some performance degradation ot all of this feature are done in hardware, with wirespeeed ? Which model will better to do this ? Thanks for any advice, Pedro --- Ta wiadomość została sprawdzona na obecność wirusów przez oprogramowanie antywirusowe Avast. https://www.avast.com/antivirus
Re: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos
That sort of thing has never bothered me much. If the platform is so great, surely it'll last more than a few years. What's the MTBF on these things? Decades? Better power performance, newer features, higher capacities sure are all great reasons to get newer hardware. EOL isn't. Don't too many of you adopt that strategy, though. I still want my source of cheap EOL hardware. :-) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Matt Freitag" <mlfre...@mtu.edu> To: "Saku Ytti" <s...@ytti.fi> Cc: "nanog list" <nanog@nanog.org> Sent: Friday, September 30, 2016 3:50:25 PM Subject: Re: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos Pedro, Please also keep in mind that the Juniper EX4500 is an end of life product. Soon you won't be able to get Juniper to support you. That's why there are so many for so cheap on eBay. Matt Freitag Network Engineer I Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.it.mtu.edu/ On Fri, Sep 30, 2016 at 4:06 PM, Saku Ytti <s...@ytti.fi> wrote: > On 30 September 2016 at 22:42, Pedro <piotr.1...@interia.pl> wrote: > > Hey Pedro, > > > I have some idea to put switch before bgp router in order to terminate > isp > > 10G uplinks on switch, not router. Main reason is that could be some > kind of > > 1st level of defence against ddos, second reason, less important, save > cost > > of router ports, do many port mirrors. > > I don't understand your rationale, unless your router is software box, > but as it has 10G interface, probably not. > Your router should be able to limit packets in HW, likely with better > counter and filtering options than cheap switch. > > -- > ++ytti >
Re: BCP38 adoption "incentives"?
IPv6? Is that common in CMTSes or just in certain ones? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Wesley George" <wesgeo...@puck.nether.net> To: "Mike Hammett" <na...@ics-il.net> Cc: nanog@nanog.org Sent: Wednesday, September 28, 2016 10:08:00 AM Subject: Re: BCP38 adoption "incentives"? At least as far as cable is concerned, there is already configuration on the CMTS (e.g. https://www.cisco.com/c/en/us/support/docs/broadband-cable/cable-security/20691-source-verify.html ) that rejects things not coming from the assigned address, and AFAIK, it's best practice to enable it for more reasons than attack prevention. However... most residential IPv4 traffic lives behind a NATing CPE. The CPE will either: a) drop anything sourced from addresses not part of the configured LAN prefix b) NAT everything regardless of its source c) NAT things from its configured LAN, but bridge/forward anything else A and C result in spoofed traffic being dropped, either at the CPE or the CMTS. Same is true if the CPE itself has been compromised and is sending spoofed traffic. B results in it no longer being spoofed traffic, meaning that it defuses reflection attacks (the source address is no longer your attack target's address) but if it's raw packet floods, the attack still works but is now traceable back to its source. The behavior of a specific CPE is largely dependent on its raw source materials. Many CPE cheap plastic routers are built from a few common reference architectures from the chipset makers (Broadcom, Intel, etc) and then modified and adapted to brand their UI with the name silk-screened on the plastic, add features to distinguish one cheap plastic router from another, etc. Reasonably recent linux-based kernels do some of A by themselves, may even do things like RPF check, TCP sequence number window check, state comparison, so unless the CPE vendor defeats it when they adapt it for their use, it mostly works. Devices built to captive standards (i.e. purpose-built for Cable, DSL providers) could have specific guidance about which behavior is the correct one, but that may or may not affect what happens to the ones that show up at your favorite big box retailer. --Wes George, who has learned a thing or two about cable, but is speaking only for himself. On Sep 27, 2016, at 4:51 PM, Mike Hammett < na...@ics-il.net > wrote: They don't need to manage the router. The raw DSL modem, cable modem, etc. can watch the packets and see what's assigned. This would need new hardware, but it's not like this is happening quickly any other way. Yes, there are some consumer purchased DSL routers and cable routers, but doing what you can with what you can. FWIW, I believe most American ISPs *DO* manage their end-user routers. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Andrew White" < andrew.whi...@charter.com > To: "Mike Hammett" < na...@ics-il.net > Cc: nanog@nanog.org Sent: Tuesday, September 27, 2016 3:44:35 PM Subject: RE: BCP38 adoption "incentives"? Hi Mike, This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. Andrew NB: My personal opinion and not official communiqué of Charter. Andrew White Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber andrew.whi...@charter.com Systems Engineer III, DAS DNS group Charter Communications 12405 Powerscourt Drive, St. Louis, MO 63131 -----Original Message- From: NANOG [ mailto:nanog-boun...@nanog.org ] On Behalf Of Mike Hammett Sent: Tuesday, September 27, 2016 3:33 PM Cc: nanog@nanog.org Subject: Re: BCP38 adoption "incentives"? It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Stephen Satchell" < l...@satchell.net > To: nanog@nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large
Re: BCP38 adoption "incentives"?
They don't need to manage the router. The raw DSL modem, cable modem, etc. can watch the packets and see what's assigned. This would need new hardware, but it's not like this is happening quickly any other way. Yes, there are some consumer purchased DSL routers and cable routers, but doing what you can with what you can. FWIW, I believe most American ISPs *DO* manage their end-user routers. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Andrew White" <andrew.whi...@charter.com> To: "Mike Hammett" <na...@ics-il.net> Cc: nanog@nanog.org Sent: Tuesday, September 27, 2016 3:44:35 PM Subject: RE: BCP38 adoption "incentives"? Hi Mike, This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. Andrew NB: My personal opinion and not official communiqué of Charter. Andrew White Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber andrew.whi...@charter.com Systems Engineer III, DAS DNS group Charter Communications 12405 Powerscourt Drive, St. Louis, MO 63131 -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mike Hammett Sent: Tuesday, September 27, 2016 3:33 PM Cc: nanog@nanog.org Subject: Re: BCP38 adoption "incentives"? It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Stephen Satchell" <l...@satchell.net> To: nanog@nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".)
Re: BCP38 adoption "incentives"?
It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Stephen Satchell" <l...@satchell.net> To: nanog@nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".)
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
We can't teach other network operators the value of IPv6. Good luck teaching a consumer anything other than cat videos (and now recipes - unrelated to the former). - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Brielle Bruns" <br...@2mbit.com> To: nanog@nanog.org Sent: Tuesday, September 27, 2016 10:46:39 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On 9/27/16 9:35 AM, Roland Dobbins wrote: > On 27 Sep 2016, at 21:48, Brielle Bruns wrote: > >> You start cutting off users or putting them into a walled garden until >> they fix their machines, and they will start caring. > > It's important to keep in mind that in the not-so-distant future, their > 'machines' will include every article of clothing they own, every can of > soda in their refrigerator, ever major (and many minor) components of > their automobiles, every blade in their windowshades, etc. > I don't see how this is a problem exactly? If people want to buy devices that connect to their home network, they need to be aware of what these devices can do, and it is their responsibility. Better to teach them _now_ rather then later. If Timmy Numbnuts doesn't understand that plugging in a random device he found at Goodwill to his network could potentially carry liabilities, then he will keep doing it. I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
"who from my experience tend to be the least experienced and network knowledgeable people running a customer network" Also most likely to have built their network from scratch out of pure need (perhaps for themselves) rather than someone cashing in on a trend. No offense meant (though surely someone took it) either way. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Brielle Bruns" <br...@2mbit.com> To: nanog@nanog.org Sent: Tuesday, September 27, 2016 9:48:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On 9/26/16 10:05 PM, Roland Dobbins wrote: > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. This will only work if all providers including cable, DSL and *shudders* WISPs (hate to be blunt, but who from my experience tend to be the least experienced and network knowledgeable people running a customer network) do it so customer's can't just switch networks and 'make the problem go away'. I use escalating price increases and delays in service/repair time on some of my consulting customers who do things I warned them to be more careful about. It takes time, but when $cost starts to become prohibitive, they stop and think. And the ones that never learn... Well, that's more $$$ in my pocket for the effort that I would normally charge otherwise. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
You must not support end users. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mark Andrews" <ma...@isc.org> To: "Roland Dobbins" <rdobb...@arbor.net> Cc: nanog@nanog.org Sent: Monday, September 26, 2016 11:43:36 PM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey In message <b796c128-afdf-45a1-b5af-c29bff06e...@arbor.net>, Roland Dobbins wri tes: > > On 27 Sep 2016, at 6:58, Christopher Morrow wrote: > > > wouldn't something as simple as netflow/sflow/ipfix synthesized on the > > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good > > enough' to present a pretty clear UI to the user? > > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different. > It call comes down to the network operator, one way or another. There's > no separation in the public mind of 'my network' from 'the Internet' > that is analogous to the separation between 'the power company' and 'the > electrical wiring in my house/apartment' (and even in that space, the > conceptual separation often isn't present). Actually I don't believe that. They do know what machines they have have connected to their home network. Boxes don't magically connect. Every machine was explictly connected. Mark > --- > Roland Dobbins <rdobb...@arbor.net> -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Request for comment -- BCP38
I would assume that on a broadband grade connection it shouldn't work unless you have a niche player and proper LOA. I would assume that on a BGP level circuit that it would work, again, given proper documentation (LOAs, IRRDB entry, etc.). IRRDBs make this wonderfully easier. By default, deny. Allow whatever is in the IRRDB entry. $250 for manual changes. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Hugo Slabbert" <h...@slabnet.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "John Levine" <jo...@iecc.com>, nanog@nanog.org Sent: Monday, September 26, 2016 11:21:55 AM Subject: Re: Request for comment -- BCP38 On Mon 2016-Sep-26 11:15:11 -0500, Mike Hammett <na...@ics-il.net> wrote: >> >>- Original Message - >> >>From: "John Levine" <jo...@iecc.com> >>To: nanog@nanog.org >>Sent: Monday, September 26, 2016 11:04:33 AM >>Subject: Re: Request for comment -- BCP38 >> >>>If you have links from both ISP A and ISP B and decide to send traffic out >>>ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >>>drop that traffic on the floor. There is no automated or scalable way for >>>ISP A to distinguish this "legitimate" use from spoofing; unless you >>>consider it scalable for ISP A to maintain thousands if not more >>>"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >>>of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >>>allocated to them by other ISPs? >> >>I gather the usual customer response to this is "if you don't want our >>$50K/mo, I'm sure we can find another ISP who does." >> >>From the conversations I've had with ISPs, the inability to manage >>legitimate traffic from dual homed customer networks is the most >>significant bar to widespread BCP38. I realize there's no way to do >>it automatically now, but it doesn't seem like total rocket science to >>come up with some way for providers to pass down a signed object to >>the customer routers that the routers can then pass back up to the >>customer's other providers. >> >>R's, >>John >> >>PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. >> >Are you talking BGP level customers or individual small businesses' >broadband service? I myself am talking about the latter and included the option of PI space to cover that (although I guess at some point this can be made fly with PA space from another provider if both providers are willing enough to play ball), though from the $50/mo figure John listed, I'm assuming he's talking about the latter. Do people really expect to be able to do this on residential or small business broadband networks? I can't remember any time in recent memory where I assumed I could set a source address to any IP I fancy and have that packet successfully make its way through the SP's network. > >- >Mike Hammett >Intelligent Computing Solutions >http://www.ics-il.com > >Midwest-IX >http://www.midwest-ix.com -- Hugo Slabbert | email, xmpp/jabber: h...@slabnet.com pgp key: B178313E | also on Signal
Re: Request for comment -- BCP38
Are you talking BGP level customers or individual small businesses' broadband service? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "John Levine" <jo...@iecc.com> To: nanog@nanog.org Sent: Monday, September 26, 2016 11:04:33 AM Subject: Re: Request for comment -- BCP38 >If you have links from both ISP A and ISP B and decide to send traffic out >ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >drop that traffic on the floor. There is no automated or scalable way for >ISP A to distinguish this "legitimate" use from spoofing; unless you >consider it scalable for ISP A to maintain thousands if not more >"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >allocated to them by other ISPs? I gather the usual customer response to this is "if you don't want our $50K/mo, I'm sure we can find another ISP who does." >From the conversations I've had with ISPs, the inability to manage legitimate traffic from dual homed customer networks is the most significant bar to widespread BCP38. I realize there's no way to do it automatically now, but it doesn't seem like total rocket science to come up with some way for providers to pass down a signed object to the customer routers that the routers can then pass back up to the customer's other providers. R's, John PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle.
Re: Request for comment -- BCP38
The only asymmetric routing broken is when the source isn't in public Internet route-able space. That just leaves those multi-ISP WAN routers that NAT it. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Laszlo Hanyecz" <las...@heliacal.net> To: nanog@nanog.org Sent: Monday, September 26, 2016 10:47:43 AM Subject: Re: Request for comment -- BCP38 On 2016-09-26 15:12, Hugo Slabbert wrote: > > On Mon 2016-Sep-26 10:47:24 -0400, Ken Chase <m...@sizone.org> wrote: > >> This might break some of those badly-behaving "dual ISP" COTS routers >> out there >> that use different inbound from outbound paths since each is the >> fastest of >> either link. > > As it should. > > If you have links from both ISP A and ISP B and decide to send traffic > out ISP A's link sourced from addresses ISP B allocated to you, ISP A > *should* drop that traffic on the floor. There is no automated or > scalable way for ISP A to distinguish this "legitimate" use from > spoofing; unless you consider it scalable for ISP A to maintain > thousands if not more "exception" ACLs to uRPF and BCP38 egress > filters to cover all of the cases of customers X, Y, and Z sourcing > traffic into ISP A's network using IPs allocated to them by other ISPs? > This is a legitimate and interesting use case that is broken by BCP38. The effectiveness of BCP38 at reducing abuse is dubious, but the benefits of asymmetric routing are well understood. Why should everyone have to go out of their way to break this.. it works fine if you just don't mess with it. > If you want to play asymmetry tricks, get some PI space and make > arrangements. If that's outside your wheelhouse, get an ISP that will > sell this to you as a service either with dissimilar links they > provide to you or over-the-top with tunnels etc. > > Playing NAT games with different classes of traffic to e.g. send > traffic type 1 over ISP A and traffic type 2 over ISP B *BUT* using > the corresponding source addresses in each case and having the traffic > return back over the same links is fine and dandy. If you send > traffic into an ISP-provided link using addresses from another > provider, though, that ISP *should* be dropping that traffic. If they > don't, send them here so we can yell at them. > So instead of being able to use simple destination based routes to direct their traffic, like the service provider can, the CPE operator has to learn and implement policy based routing and manage state to juggle each of the IP addresses they are assigned. It's orders of magnitude harder to do this with the current ecosystem of routers/CPEs, than it is to add a destination route. I think stuff like this is one of the reasons why many are hesitant to implement this type of filtering. It makes a specific type of abuse easier to track down *for someone else* but it doesn't help you much and it can cause debugging nightmares when something doesn't work due to filtering. -Laszlo >> I did this manually when I was messing around with multiple broadband >> links on >> a fbsd router years ago, was glad it worked at the time. >> >> /kc >> >> >> On Mon, Sep 26, 2016 at 07:11:42AM -0700, Paul Ferguson said: >> >No -- BCP38 only prescribes filtering outbound to ensure that no >> packets leave your network with IP source addresses which are not >> from within your legitimate allocation. >> > >> > - ferg >> > >> > >> >On September 26, 2016 7:05:49 AM PDT, Stephen Satchell >> <l...@satchell.net> wrote: >> >>Is this an accurate thumbnail summary of BCP38 (ignoring for the >> moment >> >> >> >>the issues of multi-home), or is there something I missed? >> >> >> >>> The basic philosophy of BCP38 boils down to two axioms: >> >>> >> >>> Don't let the "bad stuff" into your router >> >>> Don't let the "bad stuff" leave your router >> >>> >> >>> The original definition of "bad stuff" is limited to source- >> >>> address grooming both inbound and outbound. I've expanded on >> the >> >>> original definition by including rule generation to control >> >>> broadcast address abuse. >> > >> >-- >> >Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> -- >> Ken Chase - m...@sizone.org Toronto Canada >
Re: One Year On: IPv4 Exhaust
ARIN exhausted their last /8 about a year ago. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Paul Thornton" <p...@prt.org> To: nanog@nanog.org Sent: Sunday, September 25, 2016 11:19:01 AM Subject: Re: One Year On: IPv4 Exhaust On 25/09/2016 01:54, Jay R. Ashworth wrote: > One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John > Curran announced that the last IPv4 address block in ARIN's Free Pool had > been assigned. > > How's that been workin' out for everyone? If you'll all indulge a bit of a RIPE-centric reply on this; I've was allocated a /22 from around half-way through 185.169.0.0/16 last week (185 being RIPE's final /8). Assuming that RIPE are allocating sequentially - and I believe they are - This means that they have consumed around 66.5% of their final /8. They started allocating from this in September 2012, which suggests a reasonably low consumption rate but the RIPE final /8 will be exhausted in around two years time. I can't find an equivalent ARIN page of "how much we've allocated from our last /8" - the statistics show that just over 2x /16s worth have been assigned/allocated between January 2016 and July 2016, so a lower rate by some margin than RIPE - but there are of course policy differences at play there. Now the operational question of "How has this affected us" is probably best answered with "We've had to pay real money for IPv4 addresses since then." What may be much more interesting is what happens when the fairly ready supply of IPv4 addresses in the secondary transfer market starts to dry up. Just throwing additional money at the problem will probably not be an effective or viable solution then. I'm sure that Geoff Huston has a much more accurate and colourful set of predictions than my back-of-envelope calculations for those interested! Paul.
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
You don't need complete adoption to reduce the attacks. If ASes representing 25% of the current spoofed traffic implemented BCP38, then guess what, there's 25% less of an attack. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ca By" <cb.li...@gmail.com> To: "Jay R. Ashworth" <j...@baylink.com> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Sunday, September 25, 2016 10:13:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sunday, September 25, 2016, Jay R. Ashworth <j...@baylink.com> wrote: > - Original Message - > > From: "Ca By" <cb.li...@gmail.com <javascript:;>> > > > On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org > <javascript:;>> > > wrote: > > > >> And of course Brian Krebs has a thing or two to say, not the least is > which > >> to push for BCP38 (good luck with that, right?). > >> > >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > > > Yeh, bcp38 is not a viable solution. > > > > As long as their is one spoof capable network on the net, the problem > will > > not be solved. While bcp38 is a true bcp, it is not a solution. It will > > not, and has not, moved the needle. > > No; things which are not implemented anywhere generally don't move the > needle. > > It is implemented many places in fact. > You're confusing cause and effect here, I think. > > I will argue you are confused. > You give no evidence that *pervasive implementation of 38* would *not* move > the needle, and that's where we are right now: we do not have anything that > looks like "pervasive implementation". > > *Ten* people could solve this problem. Tomorrow. > > The chief engineers of the top 10 US eyeball providers could simply sit > down > and say "let's go do this thing". And better than 80% of the potential > sources > would just vanish off the face of the internet. > > Assume every network in the usa implements bcp38. This simply means no spoofs source from usa. Every packet is sent from the usa using a valid origin. Assume also 50% of networks in Europe and Asia and the Southern Hemisphere do bcp38 too. Great. The result is the needle has not moved at all. CC nodes in the non bcp38 locations will send spoofed packets destinations is comcast and att with a source of krebs. Result? Comcast and att cpe responds with crap to krebs. Ddos success despite bcp38 in all of usa. > Do I need to go do research, and name these 10 people? :-) > > Cheers, > -- jra > -- > Jay R. Ashworth Baylink > j...@baylink.com <javascript:;> > Designer The Things I Think RFC > 2100 > Ashworth & Associates http://www.bcp38.info 2000 Land > Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > 1274 >
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
I've heard people say doing BCP38 is hard for big networks and it is if you do it at your provider\peering edges. It's easier if done at the customer edge. Simply don't allow the traffic onto your network to start with. Limit the spoofing attacks to just a single random ASN. How much smaller is the attack than it is now with hundreds or thousands of them? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ca By" <cb.li...@gmail.com> To: "Jay Farrell" <jay...@jayfar.com> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Sunday, September 25, 2016 9:36:18 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org> wrote: > And of course Brian Krebs has a thing or two to say, not the least is which > to push for BCP38 (good luck with that, right?). > > https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > Yeh, bcp38 is not a viable solution. As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle. A solution is aggregating the telemetry of source IP addresses in the botnet and assigning blame and liability to the owners of the IP addresses / host ASN. The networks can then use AUP to shutdown the bot members. As where http://openntpproject.org/ was a proactive approach, Kreb's data can be reactive approach. And since the data is evidence of a crime, the network operators can enforce the AUP. The attack did happen. This ip was involved. Remediation is required. >From there, the host ASN can > On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth <j...@baylink.com > <javascript:;>> wrote: > > > - Original Message - > > > From: "Jay Farrell via NANOG" <nanog@nanog.org <javascript:;>> > > > > > And of course on windows ipconfig /flushdns > > > > > > Still I had to wait for my corporate caching servers to update; I think > > the > > > TTL on the old A record was an hour. > > > > Are big eyeball networks still flooring A record TTLs on resolution? > > > > Cheers, > > -- jra > > -- > > Jay R. Ashworth Baylink > > j...@baylink.com <javascript:;> > > Designer The Things I Think RFC > > 2100 > > Ashworth & Associates http://www.bcp38.info 2000 Land > > Rover DII > > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > > 1274 > > >
Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey
I believe the article says they were being hosted for free. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Grant Ridder" <shortdudey...@gmail.com> To: nanog@nanog.org Sent: Friday, September 23, 2016 12:58:44 PM Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ "Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size." -Grant
Re: CDN Overload?
Thanks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: Martin Hannigan <hanni...@gmail.com> To: Mike Hammett <na...@ics-il.net> Cc: NANOG <nanog@nanog.org> Sent: Thu, 22 Sep 2016 18:29:38 -0500 (CDT) Subject: Re: CDN Overload? Mike, I have the right contact there and I'll flag this thread that way in case they havent already seen it. Best, Martin Hannigan AS 20940 // AS 32787 On Thursday, September 22, 2016, Mike Hammett <na...@ics-il.net> wrote: > Do we have any contacts at Microsoft that we can talk to about this? This > time around, they are the common denominator. I know people have been > complaining about this for longer than Windows 10 has been out, so there > must be some other reasons why other parties we are to blame. > > -Mike HammettIntelligent Computing SolutionsMidwest Internet > ExchangeThe Brothers WISP > > - Original Message - > From: Bruce Curtis <bruce.cur...@ndsu.edu <javascript:;>> > To: Mike Hammett <na...@ics-il.net <javascript:;>> > Cc: Martin Hannigan <hanni...@gmail.com <javascript:;>>, NANOG < > nanog@nanog.org <javascript:;>> > Sent: Thu, 22 Sep 2016 16:28:17 -0500 (CDT) > Subject: Re: CDN Overload? > > > I have seen traffic from Microsoft in Europe to single hosts on our > campus that seemed to be unusually (high bps) and long. > > I don’t recall if the few multiple hosts I noticed this on over time > were only on our campus wifi. > > If not perhaps the common factor is longer latency? Both connects over > wireless and connections from Europe to the US would have longer latency. > > Perhaps this longer latency combined with some other factor is > triggering a but in modern TCP Congestion Control algorithms? > > > > This mentions that there have been bugs in TCP Congestion Control > algorithm implementations. Perhaps there could be other bugs that result > in the descried issue? > > https://www.microsoft.com/en-us/research/wp-content/ > uploads/2016/08/ms_feb07_eval.ppt.pdf > > > I have seen cases on our campus where too small buffers on an ethernet > switch caused a Linux TCP Congestion Control algorithm to act badly > resulting in slower downloads than a simple algorithm that depended on > dropped packets rather than trying to determine window sizes etc. The fix > in that case was to increase the buffer size. Of course buffer bloat is > also known to play havoc with TCP Congestion Control algorithms. Just > wondering if some combination of higher latency and another unknown > variable or just a bug might cause a TCP Congestion Control algorithm to > think it can safely try to increase the transmit rate? > > > > On Sep 21, 2016, at 8:29 PM, Mike Hammett <na...@ics-il.net > <javascript:;>> wrote: > > > > Thanks Marty. I have only experienced this on my network once and it was > directly with Microsoft, so I haven't done much until a couple days ago > when I started this campaign. I don't know if anyone else has brought this > to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > > > > > > - > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > - Original Message - > > > > From: "Martin Hannigan" <hanni...@gmail.com <javascript:;>> > > To: "Mike Hammett" <na...@ics-il.net <javascript:;>> > > Cc: "NANOG" <nanog@nanog.org <javascript:;>> > > Sent: Wednesday, September 21, 2016 8:19:35 PM > > Subject: Re: CDN Overload? > > > > > > > > > > > > Mike, > > > > > > I will forward to the requisite group for a look. Have you brought this > to our attention previously? I don't see anything. If you did, please > forward me the ticket numbers or message(s) (peering@ is best) so wee can > track down and see if someone already has it in queue. > > > > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > > > > Best, > > > > > > Martin Hannigan > > AS 20940 // AS 32787 > > > > > > > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < na...@ics-il.net > <javascript:;> > wrote: > > > > > > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5A > bYUV8CDxGwLSm8/edit?usp=sharing > > > > I have made the anonymized answers public. This will obviously have some > bias
Re: CDN Overload?
Do we have any contacts at Microsoft that we can talk to about this? This time around, they are the common denominator. I know people have been complaining about this for longer than Windows 10 has been out, so there must be some other reasons why other parties we are to blame. -Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP - Original Message - From: Bruce Curtis <bruce.cur...@ndsu.edu> To: Mike Hammett <na...@ics-il.net> Cc: Martin Hannigan <hanni...@gmail.com>, NANOG <nanog@nanog.org> Sent: Thu, 22 Sep 2016 16:28:17 -0500 (CDT) Subject: Re: CDN Overload? I have seen traffic from Microsoft in Europe to single hosts on our campus that seemed to be unusually (high bps) and long. I don’t recall if the few multiple hosts I noticed this on over time were only on our campus wifi. If not perhaps the common factor is longer latency? Both connects over wireless and connections from Europe to the US would have longer latency. Perhaps this longer latency combined with some other factor is triggering a but in modern TCP Congestion Control algorithms? This mentions that there have been bugs in TCP Congestion Control algorithm implementations. Perhaps there could be other bugs that result in the descried issue? https://www.microsoft.com/en-us/research/wp-content/uploads/2016/08/ms_feb07_eval.ppt.pdf I have seen cases on our campus where too small buffers on an ethernet switch caused a Linux TCP Congestion Control algorithm to act badly resulting in slower downloads than a simple algorithm that depended on dropped packets rather than trying to determine window sizes etc. The fix in that case was to increase the buffer size. Of course buffer bloat is also known to play havoc with TCP Congestion Control algorithms. Just wondering if some combination of higher latency and another unknown variable or just a bug might cause a TCP Congestion Control algorithm to think it can safely try to increase the transmit rate? > On Sep 21, 2016, at 8:29 PM, Mike Hammett <na...@ics-il.net> wrote: > > Thanks Marty. I have only experienced this on my network once and it was > directly with Microsoft, so I haven't done much until a couple days ago when > I started this campaign. I don't know if anyone else has brought this to > anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > - > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > - Original Message - > > From: "Martin Hannigan" <hanni...@gmail.com> > To: "Mike Hammett" <na...@ics-il.net> > Cc: "NANOG" <nanog@nanog.org> > Sent: Wednesday, September 21, 2016 8:19:35 PM > Subject: Re: CDN Overload? > > > > > > Mike, > > > I will forward to the requisite group for a look. Have you brought this to > our attention previously? I don't see anything. If you did, please forward me > the ticket numbers or message(s) (peering@ is best) so wee can track down and > see if someone already has it in queue. > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > Best, > > > Martin Hannigan > AS 20940 // AS 32787 > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < na...@ics-il.net > wrote: > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing > > > I have made the anonymized answers public. This will obviously have some bias > to it given that I mostly know fixed wireless operators, but I'm hoping this > gets some good distribution to catch more platforms. > > > > > - > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > - Original Message - > > From: "Mike Hammett" < na...@ics-il.net > > To: "NANOG" < nanog@nanog.org > > Sent: Wednesday, September 21, 2016 9:08:55 AM > Subject: Re: CDN Overload? > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > I have made this into a Google Form to make it easier to track compared to > randomly formatted responses on multiple mailing lists, Facebook Groups, etc. > > > > > - > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > - Original Message - > > From: "Mike Hammett" < na...@ics-il.net > > To: "NANOG" < nanog@nanog.org > > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > > I participate on a few othe
Re: PlayStationNetwork blocking of CGNAT public addresses
If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care. ;-) They know enough to hate NAT. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Alexander Maassen" <outsi...@scarynet.org> Cc: "NANOG" <nanog@nanog.org> Sent: Thursday, September 22, 2016 3:35:01 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer Oorspronkelijk bericht Van: Mark Andrews <ma...@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <li...@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874f-4fa4-b410-b7b66a75f...@mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j...@mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: CDN Overload?
Thanks Marty. I have only experienced this on my network once and it was directly with Microsoft, so I haven't done much until a couple days ago when I started this campaign. I don't know if anyone else has brought this to anyone's attention. I just sent an e-mail to Owen when I saw yours. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Martin Hannigan" <hanni...@gmail.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Wednesday, September 21, 2016 8:19:35 PM Subject: Re: CDN Overload? Mike, I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. Jared alluded to fasttcp a few emails ago. Astute man. Best, Martin Hannigan AS 20940 // AS 32787 On Sep 21, 2016, at 14:30, Mike Hammett < na...@ics-il.net > wrote: https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" < na...@ics-il.net > To: "NANOG" < nanog@nanog.org > Sent: Wednesday, September 21, 2016 9:08:55 AM Subject: Re: CDN Overload? https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" < na...@ics-il.net > To: "NANOG" < nanog@nanog.org > Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: CDN Overload?
https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Wednesday, September 21, 2016 9:08:55 AM Subject: Re: CDN Overload? https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: CDN Overload?
I've had DSL and AE service providers respond with the issues. So far there is not a common element other than CDNs. That's the point of the questions I'm asking, to gather a ton of information and then figure out how to act on it. You're assuming that the CDNs are using an unmolested, vanilla TCP stack. That may not be the case, especially if doing something like Fast TCP. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Baldur Norddahl" <baldur.nordd...@gmail.com> To: nanog@nanog.org Sent: Wednesday, September 21, 2016 9:32:58 AM Subject: Re: CDN Overload? It appears all complaints are from SP doing wireless. I am going to go with a yes and put forth a these that these guys have a common factor somewhere. It could be equipment from a some popular vendor of wireless or maybe some common method to throttle that is popular in the wireless community. I note that while we have slow links we have no throttling or bandwidth management going on except for the buffering that happens in the DSLAM. Also there is no way to cheat. If you send 4 mbps to a 2 mbps DSL it will drop half of the traffic and TCP will not survive that. The CDN would have an effective transfer rate approaching zero for that customer. That seems to be a rather bad business proposal seen from the view if the CDN so they would not do that. The other customers will be unaffected as the DSLAM itself has plenty of capacity. Regards Baldur Den 21. sep. 2016 14.36 skrev "Josh Reynolds" <j...@kyneticwifi.com>: > With so many geographically diverse complaints on many hardware routing > and switching platforms, I'm going to go with a "no". > > On Sep 21, 2016 4:04 AM, "Baldur Norddahl" <baldur.nordd...@gmail.com> > wrote: > >> How come we have never seen this problem? We have a ton of DSL and many >> of those are slow, but no customer complaints about overloaded lines from >> CDN networks. >> >> Could it be that the way you throttle the bandwidth is defect? It is easy >> to blame the other guy but could it be that you are doing it wrong? >> >> Regards, >> >> Badur >> >>
Re: CDN Overload?
https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: CDN Overload?
Likewise, why was it never an issue before and why does it only affect certain types of traffic from certain CDNs? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Baldur Norddahl" <baldur.nordd...@gmail.com> To: nanog@nanog.org Sent: Wednesday, September 21, 2016 4:02:30 AM Subject: Re: CDN Overload? How come we have never seen this problem? We have a ton of DSL and many of those are slow, but no customer complaints about overloaded lines from CDN networks. Could it be that the way you throttle the bandwidth is defect? It is easy to blame the other guy but could it be that you are doing it wrong? Regards, Badur
Re: CDN Overload?
This is what I'm asking of them: = Have you seen a CDN overloading a customer? Help me gather information on the issue. What CDN? What have you identified the traffic to be? What is the access network? Where is the rate limiting done? How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? What is doing the rate limiting? What is the rate-limit set to? Upstream of the rate-limiter, what are you seeing for inbound traffic? One connection or many? How much traffic? How does other traffic behave when exceeding the rate limit? Where is NAT performed? What is doing NAT? Shared NAT or isolated to that customer? Have you done a packet capture before and after the rate limiter? The NAT device? Would you be willing to send a filtered packet capture (only the frames that relate to this CDN) to the CDN if they want it? There have been reports of CDNs sending more traffic than the customer can handle and ignores TCP convention to slow down. Trying to investigate this thoroughly so we can get the CDN to fix their system. Multiple CDNs have been shown to do this. = - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: CDN Overload?
What do most broadband platforms do for rate limiting? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Matthew Walster" <matt...@walster.org> To: "George Skorup" <geo...@cbcast.com> Cc: "nanog list" <nanog@nanog.org> Sent: Tuesday, September 20, 2016 2:44:24 AM Subject: Re: CDN Overload? On 20 Sep 2016 9:14 am, "George Skorup" <geo...@cbcast.com> wrote: > > Now lets move the Windows 10 updates. A 'buried in the sticks' customer on Canopy 900 FSK. 1.5Mbps/384k. Multiple streams from Microsoft and LLNW at the same time. LLNW alone had maybe 10 streams going and was sending at over 15Mbps on average and at worst about 25Mbps... to a 1.5Mbps subscriber. I could throw in a MikroTik queue upstream which only moved the problem as that 15-25Mbps was still hitting backhaul links. And when I have a 100Mbps link going into the site, 25Mbps is a lot. Maybe I'm being naive but this sounds like an issue primarily with buffers. Police rather than shape the traffic, and reduce the burst size, and a lot of this should disappear... M
Re: CDN Overload?
http://www.theregister.co.uk/2016/06/08/is_win_10_ignoring_sysadmins_qos_settings/ This explains the recent situations (well, not really an explanation, but a bit more information from other people). Not so much for the ones going back a year or two. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mike Hammett" <na...@ics-il.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
CDN Overload?
I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Re: PlayStationNetwork blocking of CGNAT public addresses
People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Tom Beecher" <beec...@beecher.cc> To: "Tom Smyth" <tom.sm...@wirelessconnect.eu> Cc: "NANOG" <nanog@nanog.org> Sent: Sunday, September 18, 2016 8:15:08 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.sm...@wirelessconnect.eu> wrote: > Hi Simon, > > as other responders have said it is an inherent issue with NAT in general, > on workaround is to limit the ratio of actual users to an external IPv4 > address, the other thing we have seen from our Abuse contact emails from > PSN, is that malicious activity towards the PSN is often accompanied by > other malicious activities such as SSH brute force outbound and spaming... > > I would suggest that > > 1) limit the ratio of users to an external ipv4 address as much as possible > (which would reduce the impact of one compromised customer bringing down > play time for other clients behind the same nat > > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. > > this is the pain with NAT you have to do extra work in order prevent > infected users interrupting internet connectivity for other innocent > users... > I think you can use simple firewall rules on your edge router to identify > multiple connections to SMTP and SSH in a short period of time.. > > If you do the minimum to detect that abuse then you cant be accused of > invading peoples privacy... (bear in mind obvious false positives) > (Monitoring systems etc) ... > > Hope this helps, > > On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <si...@slimey.org> wrote: > > > All, > > > > We operate an access network with several hundred thousand users. > > Increasingly > > we're putting the users behind CGNAT in order to continue to give them an > > IPv4 > > service (we're all dual-st
Re: PlayStationNetwork blocking of CGNAT public addresses
A network that doesn't support IPv6, yet discriminates against CGNAT? That seems like a promising future. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Simon Lockhart" <si...@slimey.org> To: nanog@nanog.org Sent: Friday, September 16, 2016 8:12:46 AM Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon
Re: Arista unqualified SFP
https://sourceforge.net/p/e1000/mailman/message/28698959/ That or similar doesn't work for that model? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Denys Fedoryshchenko" <de...@visp.net.lb> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG Mailing List" <nanog@nanog.org> Sent: Thursday, August 18, 2016 7:51:13 AM Subject: Re: Arista unqualified SFP Not a case with Intel X*710 new chipset, check is in firmware. Someone hacked it, but ... On 2016-08-18 15:41, Mike Hammett wrote: > Intel does allow DAC of any vendor (assuming they properly identify as > DACs. You can also disable Intel's check in the Linux drivers. > > > > > - > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > - Original Message - > > From: "Mikael Abrahamsson" <swm...@swm.pp.se> > To: "Mark Tinka" <mark.ti...@seacom.mu> > Cc: "nanog list" <nanog@nanog.org> > Sent: Thursday, August 18, 2016 7:32:55 AM > Subject: Re: Arista unqualified SFP > > On Thu, 18 Aug 2016, Mark Tinka wrote: > >> All other vendors, explicitly or silently, adopt the same approach. > > I've heard from people running Intel NICs and HP switches, that this > can't > be turned off there. You run into very interesting problems when you're > trying to use DAC cables between multi vendor. > > Any pointers to how to turn this of on Intel NICs and HP switches?
Re: Arista unqualified SFP
Intel does allow DAC of any vendor (assuming they properly identify as DACs. You can also disable Intel's check in the Linux drivers. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mikael Abrahamsson" <swm...@swm.pp.se> To: "Mark Tinka" <mark.ti...@seacom.mu> Cc: "nanog list" <nanog@nanog.org> Sent: Thursday, August 18, 2016 7:32:55 AM Subject: Re: Arista unqualified SFP On Thu, 18 Aug 2016, Mark Tinka wrote: > All other vendors, explicitly or silently, adopt the same approach. I've heard from people running Intel NICs and HP switches, that this can't be turned off there. You run into very interesting problems when you're trying to use DAC cables between multi vendor. Any pointers to how to turn this of on Intel NICs and HP switches? -- Mikael Abrahamsson email: swm...@swm.pp.se
Re: Zayo Extortion
Try more facts and less emotion. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "HonorFirst Name Ethics via NANOG" <nanog@nanog.org> To: nanog@nanog.org Sent: Saturday, August 13, 2016 11:50:46 AM Subject: Zayo Extortion Question to the NANOG community, Is anyone else being extorted by Zayo? Is Zayo threatening shutdown over bogus and fabricated charges? The purpose of this message to the group is twofold: 1) to share our experience being extorted by Zayo with the community and 2) to understand the depth and extent of Zayo's less than ethical behavior by getting feedback from the community. Abovenet was a great organization with quality service, reasonable prices and nice folks to work with. Since being acquired by Zayo we have seen a significant degradation of service quality and responsiveness which is not unusual from a provider, but Zayo has taken things to a level of low ethics that would make Tony Soprano proud. Most interestingly they seem to identify points where you are dependent on them and threaten a shut down unless you pay them some arbitrary amount. In our case we use multiple Zayo IP, Transport, and Colo Services -- they set their extortion amount at $128,000. A completely arbitrary and fabricated number. They put significant pressure threatening to shut us down by setting their lawyers on us. Our detailed contract breakdowns, invoice and payment spreadsheets, along with all other commonsense and professional efforts were simply disregarded. At one point their lawyers and accounting people had the nerve to say "our accounting system does not track invoice details -- it only shows the total amount due so your numbers mean nothing to us." All the while they relentlessly levied disconnect threats with short timelines such as: "if you don't pay us $128,000 by this Friday, we will shut your operation down." We have had anecdotal feedback that we are not alone in our experience and that there are many more like us. If you and your company have had a similar experience with Zayo, please share it with the group or if like us you are concerned about retaliation from Zayo, please respond privately. If the group shares their experiences the public shaming may drive Zayo to stop operating like mafia thugs. If the problem is as common as we suspect, it may warrant getting the Attorney General involved. In the mean time, I strongly urge anyone already in a relationship with Zayo or considering a relationship to make sure your are well diversified with other more ethical carriers. Otherwise please consider another organization to work with. In our case we were better of with Ransomeware, than Zayo as a vendor! Its cheaper and less damaging A Zayo victim and a NANOG Member
Re: Host.us DDOS attack -and- related conversations
As discussed a few months ago (maybe Christmas time?), Comcast is actively suspending accounts involved in DNS amplification. Certainly on a network like theirs, it's an internal issue as well. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ca By" <cb.li...@gmail.com> To: aheb...@pubnix.net Cc: nanog@nanog.org Sent: Wednesday, August 3, 2016 10:05:04 AM Subject: Re: Host.us DDOS attack -and- related conversations On Wednesday, August 3, 2016, Alain Hebert <aheb...@pubnix.net> wrote: > Well, > > I'm sorry. > > That sound like the CloudFlare argument: You cannot fix the DDoSs > at the source because Elbonia can do it. The only solution is to pay > for protection. > > No. I hate the idea of paying for protection from a cloud or appliance. Elbonia just has the trigger. The loaded gun is the ddos reflector in comcast, cox, vz, and everyone else. > Between you and me, if only Elbonia are left DDoSing at 100Gbps, we > simply de-peer the commercial subnets from that country (leaving the > govt subnets up obviously) and see for them to deal with their trash > ISPs once for all. ( That's how we used to do it early on when the IIRC > flooding started ). > > There are known problematic networks. I have not seen any of them or their facilitating upstreams depeered. I can name 4 networks that source 75% of my attack attack traffic. Comcast was one due to their ssdp reflection, they stopped that now. But still lots of dns attacks from them. Or we keep getting DDoSed for the next 100+ years. > > On that track. > PS: Yes, the fictional country from the Dilbert syndicated cartoons. > > > Swap in your favorite real world country / network that has very real abuse source reputation. > On a humorous note: > > The DDoS protection lobby is our NRA. > > - > Alain Hebert aheb...@pubnix.net > <javascript:;> > PubNIX Inc. > 50 boul. St-Charles > P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 > Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 > > On 08/03/16 10:36, Ca By wrote: > > On Wednesday, August 3, 2016, Alain Hebert <aheb...@pubnix.net > <javascript:;>> wrote: > > > >> Well, > >> > >> > >> Could it be related to the last 2 days DDoS of PokemonGO (which > >> failed) and some other gaming sites (Blizzard and Steam)? > >> > >> > >> And on the subject of CloudFlare, I'm sorry for that CloudFlare > >> person that defended their position earlier this week, but there may be > >> more hints (unverified) against your statements: > >> > >> https://twitter.com/xotehpoodle/status/756850023896322048 > >> > >> That could be explored. > >> > >> > >> On top of which there is hints (unverified) on which is the real bad > >> actor behind that new DDoS service: > >> > >> > >> > >> > http://news.softpedia.com/news/pokemon-go-ddos-attacks-postponed-as-poodlecorp-botnet-suffers-security-breach-506910.shtml > > >> > >> > >> And I quote: > >> > >> "One thing LeakedSource staff spotted was that the first payment > >> recorded in the botnet's control panel was of $1, while payments for the > >> same package plan were of $19.99." > >> > >> ( Paypal payments btw ) > >> > >> > >> There is enough information, and damages, imho, to start looking for > >> the people responsible from a legal standpoint. And hopefully the > >> proper authorities are interested. > >> > >> PS: > >> > >> I will like to take this time to underline the lack of > >> participation from a vast majority of ISPs into BCP38 and the like. We > >> need to keep educating them at every occasion we have. > >> > >> For those that actually implemented some sort of tech against > >> it, you are a beacon of hope in what is a ridiculous situation that has > >> been happening for more than 15 years. > >> > >> > > Bcp38 is not the issue. It is only the trigger, and as long as one > network > > in Elbonia allows spoofs, that one network can marshall 100s of gbs of > > ddos power. Years of telling people to do bcp38 has not worked. > > > > The issue is for you and your neighbor to turn off your reflecting udp > > amplifiers (open dns relay, ssdp, ntp, chargen) and generously block > > obvious ddos traffic. A health
Re: Host.us DDOS attack -and- related conversations
Stopping one vector that makes up the largest of DDoSes certainly isn't a bad thing. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "James Bensley" <jwbens...@gmail.com> To: nanog@nanog.org Sent: Wednesday, August 3, 2016 9:40:17 AM Subject: Re: Host.us DDOS attack -and- related conversations On 3 August 2016 at 15:16, Alain Hebert <aheb...@pubnix.net> wrote: > PS: > > I will like to take this time to underline the lack of > participation from a vast majority of ISPs into BCP38 and the like. We > need to keep educating them at every occasion we have. > > For those that actually implemented some sort of tech against > it, you are a beacon of hope in what is a ridiculous situation that has > been happening for more than 15 years. At the risk of starting a "NANOG war" [1], BCP isn't a magic wand. If I find a zero day in the nasty customised kernels that OVH run on their clients boxes, I only need 300 compromised hosts to send 300Gbps of traffic without spoofing the IP or using amplification attacks [2]. I can rent a server with a 10Gbps connection for 1 hour for a few quid/dollars. I could generate hundreds of Gbps of traffic for about £1000 from legitimate IPs, paid for with stolen card details. How will BCP save you then? Can everyone stop praising it like it was a some magic bullet? James. [1] A pathetic and futile one, so different from the rest. [2] Subsitute OVH for any half decent provider that isn't really oversubscribed.
Re: Host.us DDOS attack -and- related conversations
Doing BCP38 or blocking\shutting off known amplification vectors both require effort and both accomplish the same thing. Of course doing both is best. :-) One provider in "Elbonia" getting through is far more damaging to that provider in Elbonia than the rest of the world, if they were the only ones left. Do many last mile providers implement BCP38 at their CE? Seems like it's better to stop it at the CE than the PE. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Ca By" <cb.li...@gmail.com> To: aheb...@pubnix.net Cc: nanog@nanog.org Sent: Wednesday, August 3, 2016 9:36:09 AM Subject: Re: Host.us DDOS attack -and- related conversations On Wednesday, August 3, 2016, Alain Hebert <aheb...@pubnix.net> wrote: > Well, > > > Could it be related to the last 2 days DDoS of PokemonGO (which > failed) and some other gaming sites (Blizzard and Steam)? > > > And on the subject of CloudFlare, I'm sorry for that CloudFlare > person that defended their position earlier this week, but there may be > more hints (unverified) against your statements: > > https://twitter.com/xotehpoodle/status/756850023896322048 > > That could be explored. > > > On top of which there is hints (unverified) on which is the real bad > actor behind that new DDoS service: > > > > http://news.softpedia.com/news/pokemon-go-ddos-attacks-postponed-as-poodlecorp-botnet-suffers-security-breach-506910.shtml > > > > And I quote: > > "One thing LeakedSource staff spotted was that the first payment > recorded in the botnet's control panel was of $1, while payments for the > same package plan were of $19.99." > > ( Paypal payments btw ) > > > There is enough information, and damages, imho, to start looking for > the people responsible from a legal standpoint. And hopefully the > proper authorities are interested. > > PS: > > I will like to take this time to underline the lack of > participation from a vast majority of ISPs into BCP38 and the like. We > need to keep educating them at every occasion we have. > > For those that actually implemented some sort of tech against > it, you are a beacon of hope in what is a ridiculous situation that has > been happening for more than 15 years. > > Bcp38 is not the issue. It is only the trigger, and as long as one network in Elbonia allows spoofs, that one network can marshall 100s of gbs of ddos power. Years of telling people to do bcp38 has not worked. The issue is for you and your neighbor to turn off your reflecting udp amplifiers (open dns relay, ssdp, ntp, chargen) and generously block obvious ddos traffic. A healthy udp policer is also smart. I suggest taking a baseline of your normal peak udp traffic, and build a policer that drops all udp that is 10x the baseline for bw and pps. Bcp38 is good, but it is not the solution we need to tactically stop attacks. This is not pretty. But it works at keeping your network up. CB - > Alain Hebert aheb...@pubnix.net > <javascript:;> > PubNIX Inc. > 50 boul. St-Charles > P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 > Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 > > On 08/03/16 09:41, Robert Webb wrote: > > Anyone have any additonal info on a DDOS attack hitting host.us? > > > > Woke up to no email this morning and the following from their web site: > > > > > > > > *Following an extortion attempt, HostUS is currently experiencing > sustained > > large-scale DDOS attacks against a number of locations. The attacks were > > measured in one location at 300Gbps. In another location the attacks > > temporarily knocked out the entire metropolitan POP for a Tier-1 > provider. > > Please be patient. We will return soon. Your understanding is > appreciated. > > * > > > > > > >From my monitoring system, looks like my VPS went unavailable around > 23:00 > > EDT last night. > > > > Robert > > > >
ExtremeWare
Can those that ran switches with ExtremeWare on them remember that far back? I've got a Summit 400t-48 and I can't seem figure out how to get DDM information from the SFP. Did they have that ability? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Re: akamai abnormal spike
Several of my WISP colleagues have noticed this behavior (CDN sending way more traffic than the customer's pipe can handle) from (I believe) multiple CDNs. Not sure if it is intention on behalf of the CDN or an error, but it has been on-going for several months if not years. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Blake Hudson" <bl...@ispn.net> To: nanog@nanog.org Sent: Monday, July 18, 2016 8:49:21 AM Subject: Re: akamai abnormal spike We noticed that on the 12th-14th we had multiple subscribers on ~5Mbps subscription rates that were being sent ~50Mbps of data sourced from TCP port 80 (apparently HTTP) from Limelight Networks' servers. The data did appear to be user requested, still not sure why TCP didn't throttle the data rate appropriately. The 50Mbps was distributed across multiple LLNW servers. Makes me wonder if the customer was requesting one batch of data and multiple servers were responding. The issue cleared up on its own and I never was able to perform a full packet capture to investigate. I have not noticed the same behavior from Akamai servers. Clayton Zekelman wrote on 7/18/2016 8:26 AM: > > > We noticed on the 12th and 13th there was a significant up tick in > traffic served from our Akamai servers as well. > > > At 05:37 PM 13/07/2016, eric c wrote: >> Good afternoon, >> >> Has anyone notice any abnormal spike in Akamai trafic in the last 24-48 >> hours compared to other days. I know it was black tuesday yesterday but >> traffic from last month didn't even come close to what we saw from >> Akamai. >> >> We have some caching servers and even notice a spike to them as well. >> >> Limelight even showed up on our network. >> >> thanks >> eric >
Re: Experience on Wanguard for 'anti' DDOS solutions
(I debated starting a new thread, only to have someone point me to previous ones vs. replying to an old post. I thought the latter was less offensive.) Did you find anything else near the price range that didn't have these deficiencies? As an eyeball network, would I have much to worry about regarding non-layer3/4 attacks? "Considering how easy it is to blocklayer 3/4 attacks on your own, their filtering clusters don't offer much value." I am aware of manual ACLs, but are there other automated methods (near this price range) to handle the 3/4 attacks? "it runs out of memory quickly" How much memory are we talking here? Reasonable to mitigate that downside by just stuffing more RAM in the box? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Richard Hesse" <richard.he...@weebly.com> To: "NANOG Mailing List" <nanog@nanog.org> Sent: Friday, August 28, 2015 1:23:01 PM Subject: Re: Experience on Wanguard for 'anti' DDOS solutions We've tried their products off an on for the past 3-4 years. Here are my impressions: * UI stuck in 1999. Can't click zoom, drill down, etc. * Inflexible UI. Want a bandwidth graph with only egress or ingress? Too bad. * Inexpensive. I don't like that it's licensed yearly, but it's not too much money. * Inaccurate flow processing. Do you have iBGP peering sessions between border routers? WANGuard will struggle mightily to correctly classify the traffic as internal or external. * Yes, it runs out of memory quickly during a spoofed SYN flood with many sources. This is due to setting the Top generator to Full. If you just want to mitigate and not have any insight into network data, set this to Extended and you'll be fine. But if you want to use WANGuard/WANSight as a network intelligence tool as well, you need to set the generator to Full and it will fall over. * Doesn't process IPFIX flow data properly. There's an old thread on the j-nsp list about this. Basically their support claims Juniper is broken (which I don't doubt) but then refuses to work around the issue. None of our other flow processing tools have these problems. * Support is responsive at times and is always cranky. I brought them two bonafide bugs in their product that they refused to admit. It got to the point where I asked for my money back and I think someone in sales lit up their support team. I get the feeling that the support team is staffed with employees who really don't like their job or working with customers. A bad combination. * The TAP generators with Myricom cards work well. The docs say you can use SolarFlare for TAPs but they don't work at all. Again, they blame SolarFlare and say that the cards are too complicatedbut fail to update their documentation saying this. * Doesn't support any kind of layer 7 detection or filtering. It's all very rudimentary layer 3-4 stuff. Considering how easy it is to block layer 3/4 attacks on your own, their filtering clusters don't offer much value. * No real scale out solution on the detection side. It's basically scale up your server or use clunky tech like NFS to share out directories across managers. * Works well enough to get you a rough idea of what's going on. It's also decently cheap. We use it as one part of our attack detection toolset. We don't use it for on-site attack mitigation. I'd recommend it if you don't want to use flow data and only want to use it for intelligence on TAP ports. -richard On Mon, Aug 10, 2015 at 6:58 AM, Marcel Duregards <marcel.durega...@yahoo.fr> wrote: > Dear Nogers, > We are currently evaluating some DDOS detection/mitigation solutions. > Do you have any inputs/experiences on Wanguard from Andrisoft, please > ?https://www.andrisoft.com/software/wanguard > Currently we are just interested on the packets/flows sensors with the > console for detection and RTBH trigger. Maybe the packet filtering (for > scrubbing) will come later. > Best Regards,-Marcel Duregards > > >
Re: IPv6 deployment excuses
Are you saying that functional game consoles aren't your problem? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Masataka Ohta" <mo...@necom830.hpcl.titech.ac.jp> To: "Valdis Kletnieks" <valdis.kletni...@vt.edu> Cc: nanog@nanog.org Sent: Monday, July 4, 2016 11:22:59 PM Subject: Re: IPv6 deployment excuses valdis.kletni...@vt.edu wrote: >> A large ISP should just set up usual NAT. In addition, > Thus almost guaranteeing a call to the support desk for each and every single > game console, because the PS3 and PS4 doesn't have a configuration interface > for that, and the XBox probably doesn't either (and if it does, it's probably > something that Joe Sixpack can't do without help). With usual NAT? That is not my problem. >> But, if you want to run a server at fixed IP address >> and port, port forwarding must be static. > > A laudable network design for my competitors. Feel free to deploy it at a > realistic sized ISP and let us know how it works out. Are you saying there is no realistic sized ISP offering fixed IP addresses without NAT? If not, additional setup of static port forwarding on NAT boxes can not be a problem. Masataka Ohta
Re: IPv6 deployment excuses
Security that is too strict will be disabled and be far less effective than proper security measures. Security zealots are often blind to that. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Keith Medcalf" <kmedc...@dessus.com> To: "nanog list" <nanog@nanog.org> Sent: Saturday, July 2, 2016 11:41:48 AM Subject: RE: IPv6 deployment excuses Yes, the default is "on". An exception is added for EVERY SINGLE PIECE of Microsoft Crapware, whether it is needed or not (and in every single case, it is not). And if you turn those exceptions "off", then they are turned back on by Microsoft and their NSA partners for you, without your permission, whenever automatic updates run (and also at other times that I have not determined the trigger). You must continuously check that the firewall (although ON) remains configured as you configured it, or if Microsoft (and their NSA partners) have changed the configuration without your permission. Of course, most people do not bother configuring the firewall and do not wonder why every piece of Crapware has in incoming exception, and do not bother to turn those off (including some on this list apparently). So they will never notice these nefarious doings which have been a hotbed of discussion on the Internet for many years. And this is on the latest distribution of Windows 10 including the upcoming anniversary edition and has been that way since at least the first version of Windows 8. Whether or not Windows 7 also behaves the same way I do not know because I never ran it. > -Original Message- > From: Spencer Ryan [mailto:sr...@arbor.net] > Sent: Saturday, 2 July, 2016 10:08 > To: Keith Medcalf > Cc: North American Network Operators' Group > Subject: RE: IPv6 deployment excuses > > Windows 8 and 10 with the most recent service packs default the firewall > to on with very few inbound exemptions. > > > On Jul 2, 2016 11:38 AM, "Keith Medcalf" <kmedc...@dessus.com> wrote: > > > > > There is no difference between IPv4 and IPv6 when it comes to > > firewalls and reachability. It is worth noting that hosts which > > support IPv6 are typically a lot more secure than older IPv4-only > > hosts. As an example every version of Windows that ships with IPv6 > > support also ships with the firewall turned on by default. > > Just because the firewall is turned on does not mean that it is > configured properly. > > Every version of Windows that ships with IPv6 support also ships > with the Firewall configured in such a fashion that you may as well have > it turned off. > > This is especially true in Windows 8 and later where the firewall is > reconfigured without your permission by Microsoft every time you install > any update whatsoever back to the "totally insecure" default state -- and > there is absolutely no way to fix this other than to check, every single > minute, that the firewall is still configured as you configured it, and > not as Microsoft (and their NSA partners) choose to configure it. > > All versions of Windows 8 and later whether using IPv4 or IPv6 are > completely unsuitable for use on a network attached to the Internet by any > means (whether using NAT or not) that does not include an external (to > Windows) -- ie, in network -- statefull firewall over which Windows, > Microsoft, (and their NSA partners) have no automatic means of control. > If you allow UPnP control of the external statefull firewall from Windows > version 8 or later, you may as well not bother having any firewall at all > because it is not under your control. > > > > >
Re: IPv4 Legacy assignment frustration
<3 name and shame. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Tom Smyth" <tom.sm...@wirelessconnect.eu> To: "Ray Soucy" <r...@maine.edu> Cc: nanog@nanog.org Sent: Thursday, June 23, 2016 10:23:39 AM Subject: Re: IPv4 Legacy assignment frustration Hi Ray, Kraig I think people affected just have to try to put pressure on their isps in the path between the afffected ips and hope for the best... public pressure is probably the only way to get around what I think most of us would agree is a terrible practice... I really hope that we can get rid of this practice as the last crumbs of IPv4 are carved up and re-distributed amongst new and growing isps. perhaps a name and shame project to highlight those isps that block ip ranges constantly and indiscriminately, needless to say the impact such practice has on peoples freedom to communicate, Thanks Tom Smyth On Thu, Jun 23, 2016 at 4:09 PM, Ray Soucy <r...@maine.edu> wrote: > Regardless of whether or not people "should" do this, I think the horse has > already left the barn on this one. I don't see any way of getting people > who decided to filter all of APNIC to make changes. Most of them are > static configurations that they'll never look to update. > > On Wed, Jun 22, 2016 at 12:06 PM, Kraig Beahn <kr...@enguity.com> wrote: > > > The following might add some clarity, depending upon how you look at it: > > > > We, as "core" engineers know better than to use some of the sources > listed > > below, tho, my suspicion is that when an engineer or local IT person, on > an > > edge network starts to see various types of attacks, they play > wack-a-mole, > > based upon outdated or incomplete data, and never think twice about > > revisiting such, as, from their perspective, everything is working just > > fine. > > > > In a networking psychology test, earlier this morning, I wrote to ten > > well-known colleagues that I was fairly confident didn't regularly follow > > the nanog lists. Such individuals comprised of IP and IT engineers for > > which manage various network sizes and enterprises, ultimately posing the > > question of "Where in the world is 150.201.15.7, as we were researching > > some unique traffic patterns". > > > > *Seven out of ten came back with overseas*. Two came back with more > > questions "as the address space appeared to be assigned to APNIC", but > was > > routed domestically. > > > > *One came back with the correct response.* (MORENET) > > > > Two of the queried parties were representative of major networks, one for > > an entire state governmental network with hundreds of thousands of actual > > users and tens of thousands of routers, the other from another major > > university. (Names left out, in the event they see this message later in > > the day or week) > > > > After probing the origin of their responses, I found the following > methods > > or data-sources were used: > > > > -Search Engines - by far, the worst offender. Not necessarily "the > engines" > > at fault, but a result of indexed sites containing inaccurate or outdated > > CIDR lists. > > -User generated forums, such as "Block non-North American Traffic for > > Dummies Like Me > > <https://www.webmasterworld.com/search_engine_spiders/4663915-2-30.htm>" > > (Yes - that's the actual thread name on WebMasterWorld.com, from a Sr. > > Member) > > -Static (or aged) CIDR web-page based lists, usually placed for > advertorial > > generation purposes and rarely up to date or accurate. (usually via SE's > or > > forum referrals) > > -APNIC themselves - A basic SE search resulted in an APNIC page > > < > > > https://www.apnic.net/manage-ip/manage-historical-resources/erx-project/erx-ranges > > > > > > that, > > on it's face, appears to indicate 150.0.0.0/8 is in fact, part of the > > current APNIC range. > > -GitHub BGP Ranking tools: CIRCL / bgp-ranging example > > < > https://github.com/CIRCL/bgp-ranking/blob/master/lib/db_init/ip_del_list> > > (last > > updated on May 16th, 2011, tho an RT lookup > > <http://bgpranking.circl.lu/ip_lookup?ip=150.201.15.7> via the CIRCL > tool > > does shows the appropriate redirect/org) > > -Several routing oriented books and Cisco examples > > < > > > http://www.cisco.com/c/en/us/support/do
Brocade Fabric Help
I asked on the Brocade forum, but it's largely been crickets there. I hoped someone here would have an idea. One switch says: 23 Te 12/0/24 Up ISL segmented,(ESC mismatch, Distributed Config DB)(Trunk Primary) The other switch says: 23 Te 54/0/24 Up ISL segmented,(ESC mismatch, Distributed Config DB)(Trunk Primary) I saw that means, "The DCM Configuration DB is different on both the ends of ISL," but I have no idea how to resolve that. VDX-6720s running 4.1.3b. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com
Re: cross connects and their pound of flesh
Before 100G, you'd need ten cross connects to move 100G. Now you'd need only one. That's a big drop in revenue. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Brandon Butterworth" <bran...@rd.bbc.co.uk> To: br...@pobox.com, d...@temk.in Cc: nanog@nanog.org Sent: Sunday, June 19, 2016 8:55:57 AM Subject: Re: cross connects and their pound of flesh Dave Temkin <d...@temk.in> wrote: > And as colo operators get freaked out over margin compression on the > impending 10->100G conversion (which is happening exponentially faster than > 100->1G & 1G->10G) they'll need to move those levers of spend around > regardless. If they've based their model on extracting profit proportional to technology speed then they've misunderstood Moore's law brandon
Re: cross connects and their pound of flesh
I think that's where the value in a distributed IX comes into play. The more nimble networks can move to different facilities while still maintaining the connectivity. Enough of that happens and pricing pressure comes into play in other parts of the market (space and cross connects). For those of you that operate in many markets, do you see any parallels where one operator has (or had) a hold on the market (Chicago Equinix and Miami Terremark for instance) compared to more diversified markets like NYC (due to a variety of IXes) or Seattle (due to SIX)? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Dave Temkin" <d...@temk.in> To: "Brandon Ross" <br...@pobox.com> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Sunday, June 19, 2016 8:19:16 AM Subject: Re: cross connects and their pound of flesh On Sat, Jun 18, 2016 at 12:54 PM, Brandon Ross <br...@pobox.com> wrote: > > > Value based pricing is all the rage these days, which is why they charge > you so much for cross connects. Exactly. Not that I don't like free cross connects (they're the bees knees, in fact), but at the end of the day, an existing colo operator is not going to go from paid->free cross connects without extracting that pound of flesh (read: sweet sweet 100% pure margin) from somewhere else. Your space and/or power prices will go up to backfill that lost profit. That said, those of us that buy a decent amount of colo prefer to trade in the value of the asset leased/purchased - space & power - as we have real world indexes to tie the underlying cost to for negotiation purposes. And as colo operators get freaked out over margin compression on the impending 10->100G conversion (which is happening exponentially faster than 100->1G & 1G->10G) they'll need to move those levers of spend around regardless. -Dave
Re: NANOG67 - Tipping point of community and sponsor bashing?
I think the popularity of the donation-based IX largely a violent reaction to the over-priced major IX operators in the US. People didn't like what was happening, so went to the polar opposite. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Nick Hilliard" <n...@foobar.org> To: "Dave Temkin" <d...@temk.in> Cc: "NANOG list" <nanog@nanog.org> Sent: Thursday, June 16, 2016 6:45:22 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? Dave Temkin wrote: > They are representative of the most important IXPs to deliver traffic > from in Western Europe. I don't doubt that they are important IXPs for delivering traffic. However, no other IXP in europe (both eastern and western) is doing expansion outside the countries that they operate in, other than three out of the four that you mentioned; none of the member-owned organisations in the region are making large profits or in most cases anything more than marginal profits, and all of them have lower port costs. Also, none of their activities suggest that their marketing budgets are large. These, I think, were the main points of contention you were concerned about. > I would posit that what defines important to me may not be what defines > important to you and the same can be said when you look at how various > "internet" companies look at what's important in their vertical. We're not talking about relative importance; we're talking about whether the problems you identified with the four IXPs named in your talk are representative of problems with the larger IXP community. I cannot find evidence that they are, at least not in the areas that you identified as problems. > Netnod runs a dns root server > system (i.root-servers.net <http://i.root-servers.net>) as well as a > heavy duty time service. > > There are others who do this for no cost and some who do it for > government money. Whether or not my port fees should subsidize this is a > valid question, and was brought up in the Q afterwards. All root operators do this for no charge, but at substantial cost. Running a root dns server system is one of the things what Netnod does because that's one of the things that the organisation is chartered to do. > Regarding the pricing reduction on page 16 of your preso, the US$ and > UK£ are not much different than what they were 5 years ago, but the € > has dropped by 30% against the US$. > > You speak to this below, however if my business is primarily run in USD > (which was the relevant use case presented: I'm a US company deciding if > I should peer in Europe or buy transit) then those currency fluctuations > have a very different impact than if I'm a European company functioning > primarily in local currency. Oh sure, but this is a matter that you need to take up with your financial people. I have no doubt that Netflix employs smart financial people, and that their decisions are the right thing for Netflix. IXPs are going to operate in their local currency and they cannot be held responsible for international currency fluctuations. From this respect, I don't think it's useful to bring this up in a critical context because it's not something that they can influence in any way whatever. > I did purposefully mention SIX as a polar opposite example - there is > definitely a happy medium to be found. This edges into one of the things that is crucial to this discussion, and it was unfortunate that it wasn't explored more. The crux is that there is a substantial cultural difference between how US people view IXPs and how european people view IXPs. As far as I can tell there are, for the most part, two types of IXPs in the US: commercial and co-operative. How they differ from european IXPs is that the commercials are almost all run by the data centres and are tied to those data centres. Most if not all of the co-operative IXPs are to some degree or other financed by donations or sponsorship and the donation types are: cash, equipment and manpower. In europe, there are three types of IXP: commercial, member based and non-member, non-profit. Many of the commercial IXPs are not owned by the data centres (e.g. NL-IX, ECIX, etc). The member-owned IXPs are answerable fully to their membership (e.g. LINX, INEX), and the non-member, non-profit IXPs (Netnod, VIX, etc) provide a service to the community as they see fit, but are not required to answer to the organisations who use them for peering services, even if they are likely to listen to what those organisations say. Crucially, almost all of the european non-profit IXPs are 100% self-funded without donations, sponsorship or subsidisation of manpower. They have offices,
Re: NANOG67 - Tipping point of community and sponsor bashing?
I think a similar point was made at NANOG. A distributed IX will let the market dictate that. Places that are better for people to operate in will see a rise in customers and places that aren't won't. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Eric Kuhnke" <eric.kuh...@gmail.com> To: nanog@nanog.org Sent: Thursday, June 16, 2016 6:17:51 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? > However: exchange port fees are not my biggest enemy today. My cross connect fees have not gone down *at all*. On a proportion basis, cross connect fees have gone from "not mattering" to being an important part of any deployment cost calculation. Why aren't we raising hell about cross connect fees? IMHO we should be, in the spirit of: https://en.wikipedia.org/wiki/Rent_Is_Too_Damn_High_Party Assuming the existence of overhead fiber trays throughput, when you consider the actual cost of a two strand XC between two cages in the same facility: 30 meter SC-SC duplex 9/125 G.657.A1 cable: $11 There should be a community effort to lobby facility managers and colo/IX real estate management that the value of their facility will be greater if XCs are free or nearly free, resulting in higher occupancy and a greater critical mass of carriers, rather than trying to extract revenue from the tenants by $300/mo MRC per fiber pair between two racks. On Thu, Jun 16, 2016 at 4:06 PM, Phil Rosenthal <p...@isprime.com> wrote: > Hello all, > > I wasn't able to attend NANOG this time around, but watched Dave Temkin's > presentation on youtube. > > My comments are: > 1) Over the past 5 years: > My cost for switch/router ports have gone down a lot. > My cost for transit has gone down a lot. > My cost for exchange ports have gone down, but not quite as fast as my > transit and switch/router ports, and this does lead to some value > questions. Dave is right to ask them. > > However: exchange port fees are not my biggest enemy today. My cross > connect fees have not gone down *at all*. On a proportion basis, cross > connect fees have gone from "not mattering" to being an important part of > any deployment cost calculation. Why aren't we raising hell about cross > connect fees? > > 2) Exotic features -- Pvlan, L2VPN, L3VPN have absolutely no purpose on an > exchange. If it could be done 'free' with commodity hardware, then fine -- > but if it translates to requiring Big Expensive Routers instead of a > cheaper but fast switch, this should translate to higher pricing for the > customers requiring these exotic features -- not the customers who just > want a big L2 vlan. > > 3) Remote peering -- This is mostly a question about distance for value. > There is a clear benefit in providing multi-datacenter exchanges within a > metro, and both FL-IX and SIX are doing this with a very good value > proposition. Having the ability to join DECIX Frankfurt from NYC and vice > versa -- again, this is a bizarre service to be offered, and regular users > should not be expected to pay for this. If there is a market for these > services at an unsubsidized price, then fine -- but regular members should > not be subsidizing this service. > > 4) sFlow -- I'm not sure why this is even really a topic. Commodity > hardware does have sFlow capability, and FLIX demonstrates this well. With > that said, for us, it is of extremely limited value. We might check these > graphs to validate measurements of our internal netflow/sflow graphing > systems, but generally, I look at the graphs generated by my exchange > vendors less than once per year per exchange. I am honestly not even sure > if SIX offers this service, as I never had a reason to check. > > 5) Marketting vs Outreach: These things are honestly basically the same > thing, mostly separated by the question of "is it good marketing or not". I > like having more members at the exchanges I am a member of. If it > translates to an additional 3% per year to have an additional 5% of traffic > to new members, I am fine with this. If it translates to an extra 50% of > cost for 5% of additional traffic, I am not fine with it. > > Finally -- there is nothing wrong with asking questions. If you are an > exchange company and you can defend your prices for what you offer, then > there is no problem. If you are an exchange and are mostly just hoping > nobody asks the questions because you won't have any good answers -- well, > I think this is exactly why Dave asked the question. > > Best Regards, > -Phil Rosenthal > > On Jun 16, 2016, at 1:58 PM, Adam
Re: NANOG67 - Tipping point of community and sponsor bashing?
I think that's a very limited mindset. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Zbyněk Pospíchal" <zby...@dialtelecom.cz> To: nanog@nanog.org Sent: Thursday, June 16, 2016 1:19:22 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? Dne 16.06.16 v 17:17 Niels Bakker napsal(a): > * zby...@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]: >> Are you sure they still want them if they have to pay for these >> features separately? >> >> Currently, such luxury functions are increasing costs also for >> networks who don't need/want it. > > sFlow statistics isn't a luxury function. Anything more than plain L2 in an IXP is a kind of luxury. An IXP member with it's own flow collection (or at least mac accounting) can feel they don't need sFlow statistics in an exchange. It's also proven it's possible to run an IXP, including a big one, without sFlow stats. We can say the same about route servers, SLA, customer portals etc. (ok, remote peering is a different case). If IXP members think they have to pay such functionality in their port fees, ok, it's their own decision, but member's opinion "we don't need it and we don't want to pay for it" is rational and plausible. Best Regards, Zbynek
Re: NANOG67 - Tipping point of community and sponsor bashing?
Getting people to show up can be a challenge. I've been asked by members of two midwestern IXes to come to their markets because their existing donation-supported loose and easy IX isn't really doing anything for them. Not arguing models, arguing that what should matter is results. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Seth Mattinen" <se...@rollernet.us> To: nanog@nanog.org Sent: Wednesday, June 15, 2016 2:14:21 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? On 6/15/16 05:37, Mike Hammett wrote: > I agree that the SIX is a fine organization, but the framework of the > organization has little to do with the members getting screwed over. A > non-profit donation-based IX that doesn't produce results could be screwing > its "customers" over more than a MRC-based for-profit IX that does produce. An IX just needs to "produce" a layer 2 peering fabric. That's not a tall order to get results from. Anything beyond that is extra fluff. Some people want to pay more for the fluff, some don't. ~Seth
Re: NANOG67 - Tipping point of community and sponsor bashing?
I agree that the SIX is a fine organization, but the framework of the organization has little to do with the members getting screwed over. A non-profit donation-based IX that doesn't produce results could be screwing its "customers" over more than a MRC-based for-profit IX that does produce. I also think that the individual merits of an organization or business model is pretty astray from the OP's original point (correct or not) about using the NANOG presentation platform for thinly veiled personal agenda. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Eric Kuhnke" <eric.kuh...@gmail.com> To: nanog@nanog.org Sent: Wednesday, June 15, 2016 12:43:13 AM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? Re: Item #3 there, the Google Docs spreadsheet with the IX costs... Scroll all the way down to the bottom in $/Mbps and you will find the SIX. Everyone in the Pacific NW should appreciate the excellent work that the SIX does. It's a nonprofit with transparency in its finances, a health cash reserve for emergencies and new equipment and meets very stringent uptime and reliability requirements. ISP entities and enterprise end users 1000 km away from the SIX in random locations in British Columbia, Montana, Utah and other western US states benefit from it. People who have no idea what an IX is or how it functions have better, faster and lower cost last mile Internet access thanks to their local small ISP that has had the foresight to purchased a transport circuit to Seattle to reach the SIX. It is worth mentioning that the fine people at the NWAX in Portland are working to build on the example set by the SIX, and are a 501(c)6 nonprofit: http://www.nwax.net/ On Tue, Jun 14, 2016 at 1:20 PM, Jared Mauch <ja...@puck.nether.net> wrote: > > > On Jun 14, 2016, at 11:12 AM, Matt Peterson <m...@peterson.org> wrote: > > > > This week at NANOG67, a presentation was given early on that did not > > reflect well for our community at large. > > I think that the data presented was interesting but the style of > the presenter and tone could have been different. It seemed > to be a variant of “The Rent is Too Damn High”[1] while it can > be interesting, there wasn’t a complete talk there IMHO. > > The feedback mechanism for this is honestly the survey[2]. I’m confident > that the PC will take this input seriously and work with presenters > in this regard. > > The IXP cost sheet[3] that is being maintained by Job I think gives an > idea of the peering vs transit costs assuming various bitrates and > list prices. > > The fates of IXPs and their roles will naturally resolve itself through > market economics I suspect. > > - Jared > > - snip - links - snip - > 1 - https://en.wikipedia.org/wiki/Rent_Is_Too_Damn_High_Party > 2 - https://www.nanog.org/meetings/nanog67/survey > 3 - > https://docs.google.com/spreadsheets/d/18ztPX_ysWYqEhJlf2SKQQsTNRbkwoxPSfaC6ScEZAG8/edit#gid=0 >
Re: Equinix IX Port Moves
The second option. Well, there is the first under process too, but the second is the priority at the moment. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Christopher Morrow" <morrowc.li...@gmail.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, June 10, 2016 9:46:17 AM Subject: Re: Equinix IX Port Moves On Fri, Jun 10, 2016 at 10:00 AM, Mike Hammett < na...@ics-il.net > wrote: Who has moved an Equinix IX port? We're told that it's a full cancellation, re-order, re IPs, re-peering, etc. Can anyone lend any input either way on that? there are 2 meanings (at least) to 'move', did you mean: 1) move port from 1G to 10G (or 'change speed') 2) move port from cage/rack1 to cage/rack2 (endpoint move in your space(s) ) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Equinix IX Port Moves
Who has moved an Equinix IX port? We're told that it's a full cancellation, re-order, re IPs, re-peering, etc. Can anyone lend any input either way on that? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Re: Netflix VPN detection - actual engineer needed
(not specifically to Cryptographrix) Anyone that expects any consumer-focused support to be able to address any legal or high level technical situation is a fool for having thought appropriate. These sorts of issues are things you start with Tempkin and others that frequent NOGs and other telecom events. You don't go to the web site support chat to get them to make a change to how they handle IPv6 on their end. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Cryptographrix" <cryptograph...@gmail.com> To: "Mark Felder" <f...@feld.me>, nanog@nanog.org Sent: Tuesday, June 7, 2016 8:55:10 AM Subject: Re: Netflix VPN detection - actual engineer needed As I said to Netflix's tech support - if they advocate for people to turn off IPv6 on their end, maybe Netflix should stop supporting it on their end. It's in the air whether it's just an HE tunnel issue or an IPv6 issue at the moment, and if their tech support is telling people to turn off IPv6, maybe they should just instead remove their records. (or fail back to ipv4 when v6 looks like a tunnel) On Tue, Jun 7, 2016 at 9:22 AM Mark Felder <f...@feld.me> wrote: > > > On Jun 6, 2016, at 22:25, Spencer Ryan <sr...@arbor.net> wrote: > > > > The tunnelbroker service acts exactly like a VPN. It allows you, from any > > arbitrary location in the world with an IPv4 address, to bring traffic > out > > via one of HE's 4 POP's, while completely masking your actual location. > > > > Perhaps Netflix should automatically block any connection that's not from > a known residential ISP or mobile ISP as anything else could be a server > someone is proxying through. It's very easy to get these subnets -- the > spam filtering folks have these subnets well documented. /s > > -- > Mark Felder > f...@feld.me > >
Re: Monitoring system recommendation
I'm not at that scale, but I've seen some fairly impressive performance searching through a friend's NetXMS system with a couple years of verbose syslog and monitoring to go through. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Manuel Marín" <m...@transtelco.net> To: "NANOG" <nanog@nanog.org> Sent: Monday, June 6, 2016 9:18:07 AM Subject: Monitoring system recommendation Dear Nanog community We are currently planning to upgrade our monitoring system (Opsview) due to scalability issues and I was wondering what do you recommend for monitoring 5000 hosts and 35000 services. We would like to use a monitoring system that is compatible with the nagios plugin format, however we are not sure if systems like Icinga/Shinken/Op5 are the way to go. Is someone using systems like Op5 or Icinga2 for monitoring > 5000 hosts? Would you recommend commercial systems like Sevone, Zabbix, etc instead of open source ones? Your input is really appreciated it Thank you and have a great day Regards
Re: Traffic engineering and peering for CDNs
Some rely on performance testing to the client's DNS resolver and if they're not using on-net ones, they'll be directed to use a different CDN node. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Graham Johnston" <johnst...@westmancom.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Monday, June 6, 2016 8:36:43 AM Subject: Traffic engineering and peering for CDNs Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one. Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon? Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend? I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play. Thanks, Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnst...@westmancom.com<mailto:johnst...@westmancom.com> P think green; don't print this email.
Re: Netflix VPN detection - actual engineer needed
It might be a few years yet before the new channels have that much power. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Steve Naslund" <snasl...@medline.com> To: nanog@nanog.org Sent: Friday, June 3, 2016 4:51:38 PM Subject: RE: Netflix VPN detection - actual engineer needed Actually it's time for Netflix to get out of the network transport business and tell the content providers to get over it or not get carried on Netflix. It used to be that Netflix needed content providers, now I am starting to believe it might be the other way around. Netflix might have to take a page from the satellite guys and start calling them out publicly. i.e. "Netflix will no longer be able to provide you with Warner Bros. content because they are dinosaurs that are worried that someone might be watching in the wrong country. We are pleased to offer you content from producers that are not complete morons" As the content producers lose more and more control over the distribution channel they are going to take whatever terms are necessary to get them on Netflix, Apple TV, Comcast, Time Warner, DirecTV and Dish. If you are not on any or all of those platforms, you are going to be dead meat. Who would be hurt worse, Netflix or the movie producer that got seen nowhere on their latest film. To me, this is the last gasp of an industry that lost control of its distribution channel years ago and is still trying to impose that control. Steven Naslund -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mark Andrews Sent: Friday, June 03, 2016 4:28 PM To: Laszlo Hanyecz Cc: nanog@nanog.org Subject: Re: Netflix VPN detection - actual engineer needed It's time for Netflix to offer IPv6 tunnels. That way they can correlate IPv4 and IPv6 addresses. Longest match will result is the correct source address being selected if they do the job correctly. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: Netflix VPN detection - actual engineer needed
As bad as some are in the telecom industry, they don't hold a candle to those in the content industry. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Steve Naslund" <snasl...@medline.com> To: nanog@nanog.org Sent: Friday, June 3, 2016 3:55:43 PM Subject: RE: Netflix VPN detection - actual engineer needed Wifi location depends on a bunch of problematic things. First, your SSID needs to get collected and put in a database somewhere. That itself is a crap shoot. Next, you can stop google (and some other wifi databases) from collecting the data by putting _nomap at the end of your SSID. Lastly, not everyone has wifi or iOS or GPS or whatever location method you can think of. BTW, my apple TV is on a wired Ethernet, not wifi. Point is, for whatever location technology you want to use be it IP, GPS, WiFi location, sextant…..they can be inaccurate and they can be faked and there are privacy concerns with all of them. What the content producers need to figure out is that regionalization DOES NOT WORK ANYMORE! The original point was that they could have different release dates in different areas at different prices and availability. They are going to have to get over it because they will lose the technological arms race. There is no reason you could not beat all of the location systems with a simple proxy. A proxy makes a Netflix connection from an allowed IP, location or whatever and then builds a new video/audio stream out the back end to the client anywhere in the world. Simple to implement and damn near impossible to beat. Ever hear of Slingbox? Steven Naslund Chicago IL From: Cryptographrix [mailto:cryptograph...@gmail.com] Sent: Friday, June 03, 2016 3:42 PM To: Naslund, Steve; nanog@nanog.org Subject: Re: Netflix VPN detection - actual engineer needed Apple TVs get their location indoors using the same method they use for other iOS devices when indoors - wifi ssid/Mac scanning. Non-iOS devices are often capable of this as well. (As someone that spends >67% of his time underground and whose Apple TV requests my location from my underground bedroom and is very accurate) On Fri, Jun 3, 2016 at 4:36 PM Naslund, Steve <snasl...@medline.com<mailto:snasl...@medline.com>> wrote: Their app could request your devices location. Problem is a lot of devices (like TVs, Apple TVs, most DVD player, i.e. device with built in Netflix) don't know where they are and it cannot easily be added (indoor GPS is still difficult/expensive) and even if they could should they be believed. I think the bigger issue is whether any kind of regional controls are enforceable or effective any more. Steven Naslund Chicago IL -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] On Behalf Of Cryptographrix Sent: Friday, June 03, 2016 3:21 PM To: Spencer Ryan Cc: North American Network Operators' Group Subject: Re: Netflix VPN detection - actual engineer needed Come now, content providers really just care that they have access to regional controls more so than their ability to blanket-deny access (ok, minus the MLB who are just insane). And part of those regional controls deal with the accuracy of the location information. If their app can request my device's precise location, it doesn't need to infer my location from my IP any more. As a matter of fact, it's only detrimental to them for it to do so, because of the lack of accuracy from geo databases and the various reasons that people use VPNs nowadays (i.e. for some devices that you can't even turn VPN connections off for - OR in the case of IPv6, when you can't reach a segment of the Internet without it). On Fri, Jun 3, 2016 at 4:17 PM Spencer Ryan <sr...@arbor.net<mailto:sr...@arbor.net>> wrote: > There is a large difference between "the VPN run at your house" and > "Arguably the most popular, free, mostly anonymous tunnel broker service" > > If it were up to the content providers, they probably would block any > IP they saw a VPN server listening on. > > > *Spencer Ryan* | Senior Systems Administrator | > sr...@arbor.net<mailto:sr...@arbor.net> *Arbor > Networks* > +1.734.794.5033 (d) | +1.734.846.2053 (m) > www.arbornetworks.com<http://www.arbornetworks.com> > > On Fri, Jun 3, 2016 at 4:09 PM, Cryptographrix > <cryptograph...@gmail.com<mailto:cryptograph...@gmail.com>> > wrote: > >> I have a VPN connection at my house. There's no way for them to know >> the difference between me using my home network connection from Hong >> Kong or my home network connection from my house. >> >> Are they going to disable connectivity from everywhere they can >> d
Re: IPv6 is better than ipv4
I would be surprised if more than 10% - 20% of networks have received effective marketing on IPv6. Look at how many network operators that don't "get" basic network security alerts like "There is a long since patched vulnerability being actively exploited on the Internet right now. Your equipment will reset to default in 18.5 hours of infection. Please patch now." Equipment resetting to default is a metric crap ton more serious than IPv6 implementation and people don't take that seriously. Think outside of the NANOG bubble. (I *REALLY* hate the way this list replies to the individual and not the list... and doesn't have a bracketed name in the subject.) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Christopher Morrow" <morrowc.li...@gmail.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "nanog list" <nanog@nanog.org> Sent: Thursday, June 2, 2016 12:31:43 PM Subject: Re: IPv6 is better than ipv4 On Thu, Jun 2, 2016 at 1:17 PM, Mike Hammett < na...@ics-il.net > wrote: Yes. REALLY??? I mean REALLY? people that operate networks haven't haven't had beaten into their heads: 1) cgn is expensive 2) there is no more ipv4 (not large amounts for large deployments of new thingies) 3) there really isn't much else except the internet for global networking and reachabilty 4) ipv6 'works' on almost all gear you'd deploy in your network and content side folks haven't had beaten into their heads: 1) ipv6 is where the network is going, do it now so you aren't caught with your pants (proverbial!) down 2) more and more customers are going to have ipv6 and not NAT'd ipv4... you can better target, better identify and better service v6 vs v4 users. 3) adding ipv6 transport really SHOULD be as simple as adding a I figure at this point, in 2016, the reasons aren't "marketing" but either: a) turning the ship is hard (vz's continual lack of v6 on wireline services...) b) can't spend the opex/capex while keeping the current ship afloat c) meh I can't see that 'marketing' is really going to matter... I mean, if you haven't gotten the message now: http://i.imgur.com/8vZOU0T.gif - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Christopher Morrow" < morrowc.li...@gmail.com > To: "Daniel Corbe" < dco...@hammerfiber.com > Cc: nanog@nanog.org Sent: Thursday, June 2, 2016 11:41:33 AM Subject: Re: IPv6 is better than ipv4 On Thu, Jun 2, 2016 at 12:23 PM, Daniel Corbe < dco...@hammerfiber.com > wrote: > Maybe we should let people believe that IPv6 is faster than IPv4 even if > objectively that isn’t true. Perhaps that will help speed along the > adoption process. do we REALLY think it's still just /marketing problem/ that keeps v6 deployment on the slow-boat?
Re: IPv6 is better than ipv4
Yes. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Christopher Morrow" <morrowc.li...@gmail.com> To: "Daniel Corbe" <dco...@hammerfiber.com> Cc: nanog@nanog.org Sent: Thursday, June 2, 2016 11:41:33 AM Subject: Re: IPv6 is better than ipv4 On Thu, Jun 2, 2016 at 12:23 PM, Daniel Corbe <dco...@hammerfiber.com> wrote: > Maybe we should let people believe that IPv6 is faster than IPv4 even if > objectively that isn’t true. Perhaps that will help speed along the > adoption process. do we REALLY think it's still just /marketing problem/ that keeps v6 deployment on the slow-boat?
Re: Global/distributed IXP operators?
Could you define what you mean by a distributed\global IXP? There are plenty of IXPs, but there aren't really global IXPs, those just become networks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Daniel Rohan" <dro...@gmail.com> To: "NANOG" <nanog@nanog.org> Sent: Friday, May 27, 2016 12:47:07 PM Subject: Global/distributed IXP operators? If there are any operators working at distributed/global IXPs and wouldn't mind having their brains picked regarding design questions, would you make yourselves known to me (on or off-list is fine). Thanks, Dan
Re: FlowSpec Support
I read that discussion (and several others going back about two or three years) before I posted this. As an occasional OP on here, I've noticed I get a lot of off-list responses so I obviously wouldn't have seen any of those from other people's threads. I didn't take that observation away from that thread, but maybe I'm dense. ;-) I know it was suggested that they wanted to bill for that sued capacity, but that was debunked. I know DDoS services were mentioned, but I didn't see a clear line drawn to that's why it isn't happening... nor confirmed. Also, what's big? Listed on the Baker's Dozen? Wide-spread POPs on six continents? Showing up on 50 IXPs? 1k IPv4 adjacencies? A medium sized network that does FlowSpec could be vastly more useful to you than a large network that doesn't. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Saturday, May 28, 2016 5:41:38 PM Subject: Re: FlowSpec Support There was just a recent discussion about this. None of the big upstreams support it because they are all too busy selling their own DDoS mitigation services :) On May 28, 2016 5:38 PM, "Mike Hammett" < na...@ics-il.net > wrote: I know support (from customers) is limited among networks. I know it isn't on all hardware, but does appear to be on at least a couple platforms from the major router vendors. It is supported on an increasing number of DDoS appliances and software packages. What all networks support receiving BGP FlowSpec information from customers and acting upon it? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
FlowSpec Support
I know support (from customers) is limited among networks. I know it isn't on all hardware, but does appear to be on at least a couple platforms from the major router vendors. It is supported on an increasing number of DDoS appliances and software packages. What all networks support receiving BGP FlowSpec information from customers and acting upon it? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Re: B5-Lite
I know it'll result in the air interface coming down on the M series, but verify your noise with the AirView tool. I've grown to not trust the noise floor measurement. 40 MHz at that supposed amount of SNR should be rocking almost double what you're getting. With the V and H chains that far apart, alignment might be off. What are your CCQ, AMC and AMQ numbers? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Jared Mauch" <ja...@puck.nether.net> To: "Mike Hammett" <na...@ics-il.net> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Tuesday, May 17, 2016 10:29:57 AM Subject: Re: B5-Lite I’m seeing -61 (63/67 V/H) with floor at -101 right now with the XW PowerBeam 400 w/ 40mhz. The speeds are “Ok” but getting beyond 60Mb/s is hard as the CPU maxes in a bridged setup. Doesn’t seem to have any issues with the wireless rate during load, so perhaps it’s not doing offload to the chipset right? The goal is to improve capacity in the interim while some strategic fiber is deployed for some areas. A pair of B5s or AF5X would likely work out but would rather spend that on fiber. - Jared > On May 17, 2016, at 11:06 AM, Mike Hammett <na...@ics-il.net> wrote: > > I think there is some information missing on your longer link. Did you still > have appropriate signal? Was there noise? > > I have a B5 link that's about 2 miles that's rocking full data rate and a B5c > one that's going about 4 miles at full data rate. My 8 mile B5c link is less > than full data rate due to interference. > > > > > - > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > > > Midwest Internet Exchange > http://www.midwest-ix.com > > > - Original Message - > > From: "Hal Ponton" <h...@buzcom.net> > To: "Matt Hoppes" <mattli...@rivervalleyinternet.net> > Cc: "North American Network Operators' Group" <nanog@nanog.org> > Sent: Saturday, May 14, 2016 7:31:10 AM > Subject: Re: B5-Lite > > We've deployed 2 B5 links into production, the newer firmware seems to have > fixed the issues we saw in the links when we first tested them. > > We have a very rural customer where two hops are needed around the site. > We're lucky in that we had two 80MHz channels free. We see around 350Mbps > both ways actual throughput on both links. > > However, these links are short est. 200mtrs when we had tested these on > longer links their performance was awful, on a 40MHz channel we saw 20Mbps. > > For our longer links that need a bit more throughput than a Rocket M5 we > either use Licensed radios or the AF5X which works very well. > > Regards, > > Hal Ponton > > Senior Network Engineer > > Buzcom / FibreWiFi > >> On 14 May 2016, at 11:07, Matt Hoppes <mattli...@rivervalleyinternet.net> >> wrote: >> >> Jared - why not go to Ubiquiti AC gear if you need some more speed and >> something more modern? >> >>> On May 14, 2016, at 01:43, Eric C. Miller <e...@ericheather.com> wrote: >>> >>> B5c is the only product that I've had much success with from Mimosa. >>> >>> The B5Lite is a cheap plastic shell and, and it performs like it too. >>> >>> If you have UBNT gear now, Mimosa is a good next step, but I'd strongly >>> recommend that you stear away from the lite and go with the B5c. We use >>> them with rocket dishes. You just need the RP-SMA to N cables. >>> >>> >>> Eric Miller, CCNP >>> Network Engineering Consultant >>> >>> >>> >>> -Original Message- >>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jared Mauch >>> Sent: Friday, May 13, 2016 7:06 PM >>> To: North American Network Operators' Group <nanog@nanog.org> >>> Subject: B5-Lite >>> >>> Anyone deployed this radio in production in the US? I’m curious to hear >>> from people who are using it, looking at replacing some UBNT hardware with >>> it on some PTP links, going from the M-series class devices to something >>> more modern. >>> >>> Thanks, >>> >>> - Jared >
Re: B5-Lite
I think there is some information missing on your longer link. Did you still have appropriate signal? Was there noise? I have a B5 link that's about 2 miles that's rocking full data rate and a B5c one that's going about 4 miles at full data rate. My 8 mile B5c link is less than full data rate due to interference. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Hal Ponton" <h...@buzcom.net> To: "Matt Hoppes" <mattli...@rivervalleyinternet.net> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Saturday, May 14, 2016 7:31:10 AM Subject: Re: B5-Lite We've deployed 2 B5 links into production, the newer firmware seems to have fixed the issues we saw in the links when we first tested them. We have a very rural customer where two hops are needed around the site. We're lucky in that we had two 80MHz channels free. We see around 350Mbps both ways actual throughput on both links. However, these links are short est. 200mtrs when we had tested these on longer links their performance was awful, on a 40MHz channel we saw 20Mbps. For our longer links that need a bit more throughput than a Rocket M5 we either use Licensed radios or the AF5X which works very well. Regards, Hal Ponton Senior Network Engineer Buzcom / FibreWiFi > On 14 May 2016, at 11:07, Matt Hoppes <mattli...@rivervalleyinternet.net> > wrote: > > Jared - why not go to Ubiquiti AC gear if you need some more speed and > something more modern? > >> On May 14, 2016, at 01:43, Eric C. Miller <e...@ericheather.com> wrote: >> >> B5c is the only product that I've had much success with from Mimosa. >> >> The B5Lite is a cheap plastic shell and, and it performs like it too. >> >> If you have UBNT gear now, Mimosa is a good next step, but I'd strongly >> recommend that you stear away from the lite and go with the B5c. We use them >> with rocket dishes. You just need the RP-SMA to N cables. >> >> >> Eric Miller, CCNP >> Network Engineering Consultant >> >> >> >> -Original Message- >> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jared Mauch >> Sent: Friday, May 13, 2016 7:06 PM >> To: North American Network Operators' Group <nanog@nanog.org> >> Subject: B5-Lite >> >> Anyone deployed this radio in production in the US? I’m curious to hear from >> people who are using it, looking at replacing some UBNT hardware with it on >> some PTP links, going from the M-series class devices to something more >> modern. >> >> Thanks, >> >> - Jared
Re: Mobile providers in the US for backup access
*shrugs* Seems to work here, though if Ting uses T-Mo and Sprint, I suppose Ting's more likely to have a good signal. I don't expect much support on a $6 mobile wireless service. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Owen DeLong" <o...@delong.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Wednesday, April 20, 2016 1:42:44 PM Subject: Re: Mobile providers in the US for backup access I had horrible experience when I tried to use Freedom POP many years ago. Their customer service is awful and completely uncooperative. Their equipment did not work well in my environment at all. I would not wish them on my worst enemy. Owen > On Apr 20, 2016, at 1:35 PM, Mike Hammett <na...@ics-il.net> wrote: > > I'd look at FreedomPOP's Netgear 341U. $20 - $50 NRC, single digit MRC for > low usage. > > > > > - > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > > > Midwest Internet Exchange > http://www.midwest-ix.com > > > - Original Message - > > From: "Dovid Bender" <do...@telecurve.com> > To: "NANOG" <nanog@nanog.org> > Sent: Wednesday, April 20, 2016 1:16:56 PM > Subject: Mobile providers in the US for backup access > > A while ago some people mentioned that some US carriers have basic internet > plans for backup access to their equipment. A few questions: > 1) Do they give you a public IP per connection or do you tunnel back to a > central location and then connect via the tunnel? > 2) Which carriers offer this and what kind of devices do you use to > connect? Is it simply a GSM card on a "MyFi" like device? We have lots of > Pi's out there that we want backup access to. > 3) Can you send off list contacts and pricing that you have gotten in the > past? > > TIA. > > Dovid
Re: Mobile providers in the US for backup access
I'd look at FreedomPOP's Netgear 341U. $20 - $50 NRC, single digit MRC for low usage. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Dovid Bender" <do...@telecurve.com> To: "NANOG" <nanog@nanog.org> Sent: Wednesday, April 20, 2016 1:16:56 PM Subject: Mobile providers in the US for backup access A while ago some people mentioned that some US carriers have basic internet plans for backup access to their equipment. A few questions: 1) Do they give you a public IP per connection or do you tunnel back to a central location and then connect via the tunnel? 2) Which carriers offer this and what kind of devices do you use to connect? Is it simply a GSM card on a "MyFi" like device? We have lots of Pi's out there that we want backup access to. 3) Can you send off list contacts and pricing that you have gotten in the past? TIA. Dovid
Re: 10G-capable customer router recommendations?
If you were on FB, the TBW page would be a great venue. ;-) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Andrew Thrift" <and...@networklabs.co.nz> Cc: "NANOG" <nanog@nanog.org> Sent: Saturday, April 16, 2016 9:12:13 AM Subject: Re: 10G-capable customer router recommendations? You might ask Normis about that :) It has nothing to do with fastpath, and isn't scheduled to be fixed until 7.x when many features are rewritten to take advantage of multiple tile cores. Currently each port is pinned to a single cpu (affinity) due to latency and performance reasons - but yes there are drawbacks when your per core clock is still in 1GHz territory. If you want to talk more about this, we can discuss.offlist or on the Mikrotik forum. On Apr 16, 2016 12:51 AM, "Andrew Thrift" <and...@networklabs.co.nz> wrote: > This has not been the case for at least a year now. > > Most Mikrotik routers now support FastPath/FastTrack. This is kind of > like CEF in Cisco land. > > http://wiki.mikrotik.com/wiki/Manual:Fast_Path > > http://wiki.mikrotik.com/wiki/Manual:Wiki/Fasttrack > On 16/04/2016 10:07 am, "Josh Reynolds" <j...@kyneticwifi.com> wrote: > >> Can't do more than 1Gbps per flow. Not suitable for this application. >> On Apr 15, 2016 5:03 PM, <mike.l...@gmail.com> wrote: >> >> > Check out the Mikrotik Cloud Core routers, they make them with SFP+ >> > support now. I have one of them with 10g deployed right now. >> > >> > -Mike >> > >> > > On Apr 15, 2016, at 14:52, Aaron <aa...@wholesaleinternet.net> wrote: >> > > >> > > Not a lot of 10G capable CPEs out there. For our 10G residential >> > customers we install Brocade ICXs. >> > > >> > > Aaron >> > > >> > > >> > >> On 4/15/2016 3:18 PM, David Sotnick wrote: >> > >> Hello masters of the Internet, >> > >> >> > >> I was recently asked to set up networking at a VIP's home where he >> has >> > >> Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port >> > on a >> > >> Comcast-supplied Juniper ACX-2100 router. >> > >> >> > >> Which customer router would you suggest for such a setup? It needs >> to do >> > >> IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that >> > also >> > >> supports IPv6). >> > >> >> > >> The customer pays for "2Gb" service (Comcast caps this at 2G+10% = >> > 2.2Gbps) >> > >> and would like to get what he pays for (*cough*) by having the >> ability >> > to >> > >> stream two 1Gbps streams (or at least achieve > 1.0Gbps). >> > >> >> > >> I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel >> to >> > the >> > >> customer switch, or replace the AV-integrator-installed Cisco >> SG300-52P >> > >> (Cisco switch with e.g. an EX-3300 with 10Gb uplinks). >> > >> >> > >> Thanks in advance for your suggestions. >> > >> >> > >> -Dave >> > > >> > > -- >> > > >> > > Aaron Wendel >> > > Chief Technical Officer >> > > Wholesale Internet, Inc. (AS 32097) >> > > (816)550-9030 >> > > http://www.wholesaleinternet.com >> > > >> > > >> > >> >
Re: 10G-capable customer router recommendations?
CCRs do firewalling and NAT just great. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Chris Knipe" <sav...@savage.za.org> To: "Josh Reynolds" <j...@kyneticwifi.com> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, April 15, 2016 5:11:54 PM Subject: Re: 10G-capable customer router recommendations? On Sat, Apr 16, 2016 at 12:04 AM, Josh Reynolds <j...@kyneticwifi.com> wrote: > Can't do more than 1Gbps per flow. Not suitable for this application. > On Apr 15, 2016 5:03 PM, <mike.l...@gmail.com> wrote: > > > Check out the Mikrotik Cloud Core routers, they make them with SFP+ > > support now. I have one of them with 10g deployed right now. > > > > -Mike > Also it falls pretty much flat on it's face the moment you do anything useful in terms of firewalling / NATing.
Re: 10G-capable customer router recommendations?
I'm glad you're in Missouri and not in my area. :-) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Mike Hammett" <na...@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, April 15, 2016 8:32:17 PM Subject: Re: 10G-capable customer router recommendations? If I were sold a $400/mo+ service that had a limitation like that, I would be very unhappy. To each their own. On Apr 15, 2016 8:29 PM, "Mike Hammett" < na...@ics-il.net > wrote: The CCRs' primary weaknesses are full tables and 1 gigabit cap per flow. Neither is likely to be an issue for this residential use case. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Josh Reynolds" < j...@kyneticwifi.com > To: "Filip Hruska" < f...@fhrnet.eu > Cc: "NANOG" < nanog@nanog.org > Sent: Friday, April 15, 2016 5:12:35 PM Subject: Re: 10G-capable customer router recommendations? As much as I enjoy Mikrotik products and respect my friends and peers who use them, until ROS 7.x the CCR is a "gimped" product. On Apr 15, 2016 5:10 PM, "Filip Hruska" < f...@fhrnet.eu > wrote: > Hi, > > I would also vote for Mikrotik products; IMHO this looks perfect for this > situation. > > http://routerboard.com/CCR1009-8G-1S-1SplusPC > > > > On 04/16/2016 12:01 AM, mike.l...@gmail.com wrote: > >> Check out the Mikrotik Cloud Core routers, they make them with SFP+ >> support now. I have one of them with 10g deployed right now. >> >> -Mike >> >> On Apr 15, 2016, at 14:52, Aaron < aa...@wholesaleinternet.net > wrote: >>> >>> Not a lot of 10G capable CPEs out there. For our 10G residential >>> customers we install Brocade ICXs. >>> >>> Aaron >>> >>> >>> On 4/15/2016 3:18 PM, David Sotnick wrote: >>>> Hello masters of the Internet, >>>> >>>> I was recently asked to set up networking at a VIP's home where he has >>>> Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port >>>> on a >>>> Comcast-supplied Juniper ACX-2100 router. >>>> >>>> Which customer router would you suggest for such a setup? It needs to do >>>> IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that >>>> also >>>> supports IPv6). >>>> >>>> The customer pays for "2Gb" service (Comcast caps this at 2G+10% = >>>> 2.2Gbps) >>>> and would like to get what he pays for (*cough*) by having the ability >>>> to >>>> stream two 1Gbps streams (or at least achieve > 1.0Gbps). >>>> >>>> I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel to >>>> the >>>> customer switch, or replace the AV-integrator-installed Cisco SG300-52P >>>> (Cisco switch with e.g. an EX-3300 with 10Gb uplinks). >>>> >>>> Thanks in advance for your suggestions. >>>> >>>> -Dave >>>> >>> >>> -- >>> >>> Aaron Wendel >>> Chief Technical Officer >>> Wholesale Internet, Inc. (AS 32097) >>> (816)550-9030 >>> http://www.wholesaleinternet.com >>> >>> >>> >>
Re: 10G-capable customer router recommendations?
Conversely, the UI is Mikrotik's big draw. :-) Being or not being like CIsco has zero bearing on me. Assuming the commands do what they say they'll do, any platform with tab complete is fine. :-) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Ken Chase" <m...@sizone.org> To: "NANOG" <nanog@nanog.org> Sent: Friday, April 15, 2016 7:24:56 PM Subject: Re: 10G-capable customer router recommendations? Does that lanner even do SFP+? Dont see it listed in the specs. Looks like 4210 has 2x SFP+, though their 'performance' level products look more in line with 'useful'. http://www.lannerinc.com/products/x86-network-appliances/x86-rackmount-appliances/fw-8877 As for the microtics, wonky user interface, so very unciscolike (i guess thats my problem - but the GUI thing feels like a toy), but for their midrange models I found their bgp convergence times pretty poor on their low end cpus... What do you put on the lanner? OpenBGPd? Quagga? Also looking for a 10G solution here, low power (than a full ASR stack..) is my goal for 5-6 full bgp feeds. /kc On Fri, Apr 15, 2016 at 07:45:39PM -0400, Michael Brown said: >Not *exactly* what you're asking for, but a Lanner appliance >(???http://www.lannerinc.com/products/network-appliances/x86-rackmount-network-appliances/nca-5210) > might suit your needs. > >M. > >?? Original Message ?? >From: David Sotnick >Sent: Friday, April 15, 2016 16:19 >To: NANOG >Subject: 10G-capable customer router recommendations? > >Hello masters of the Internet, > >I was recently asked to set up networking at a VIP's home where he has >Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port on a >Comcast-supplied Juniper ACX-2100 router. > >Which customer router would you suggest for such a setup? It needs to do >IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that also >supports IPv6). > >The customer pays for "2Gb" service (Comcast caps this at 2G+10% = 2.2Gbps) >and would like to get what he pays for (*cough*) by having the ability to >stream two 1Gbps streams (or at least achieve > 1.0Gbps). > >I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel to the >customer switch, or replace the AV-integrator-installed Cisco SG300-52P >(Cisco switch with e.g. an EX-3300 with 10Gb uplinks). > >Thanks in advance for your suggestions. > >-Dave Ken Chase - m...@sizone.org
Re: 10G-capable customer router recommendations?
The CCRs' primary weaknesses are full tables and 1 gigabit cap per flow. Neither is likely to be an issue for this residential use case. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Josh Reynolds" <j...@kyneticwifi.com> To: "Filip Hruska" <f...@fhrnet.eu> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, April 15, 2016 5:12:35 PM Subject: Re: 10G-capable customer router recommendations? As much as I enjoy Mikrotik products and respect my friends and peers who use them, until ROS 7.x the CCR is a "gimped" product. On Apr 15, 2016 5:10 PM, "Filip Hruska" <f...@fhrnet.eu> wrote: > Hi, > > I would also vote for Mikrotik products; IMHO this looks perfect for this > situation. > > http://routerboard.com/CCR1009-8G-1S-1SplusPC > > > > On 04/16/2016 12:01 AM, mike.l...@gmail.com wrote: > >> Check out the Mikrotik Cloud Core routers, they make them with SFP+ >> support now. I have one of them with 10g deployed right now. >> >> -Mike >> >> On Apr 15, 2016, at 14:52, Aaron <aa...@wholesaleinternet.net> wrote: >>> >>> Not a lot of 10G capable CPEs out there. For our 10G residential >>> customers we install Brocade ICXs. >>> >>> Aaron >>> >>> >>> On 4/15/2016 3:18 PM, David Sotnick wrote: >>>> Hello masters of the Internet, >>>> >>>> I was recently asked to set up networking at a VIP's home where he has >>>> Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port >>>> on a >>>> Comcast-supplied Juniper ACX-2100 router. >>>> >>>> Which customer router would you suggest for such a setup? It needs to do >>>> IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that >>>> also >>>> supports IPv6). >>>> >>>> The customer pays for "2Gb" service (Comcast caps this at 2G+10% = >>>> 2.2Gbps) >>>> and would like to get what he pays for (*cough*) by having the ability >>>> to >>>> stream two 1Gbps streams (or at least achieve > 1.0Gbps). >>>> >>>> I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel to >>>> the >>>> customer switch, or replace the AV-integrator-installed Cisco SG300-52P >>>> (Cisco switch with e.g. an EX-3300 with 10Gb uplinks). >>>> >>>> Thanks in advance for your suggestions. >>>> >>>> -Dave >>>> >>> >>> -- >>> >>> Aaron Wendel >>> Chief Technical Officer >>> Wholesale Internet, Inc. (AS 32097) >>> (816)550-9030 >>> http://www.wholesaleinternet.com >>> >>> >>> >>
Re: Connecting rural providers: ethernet to large city or nearby transit
Get backhaul to somewhere useful. Do not buy from the incumbent. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Jean-Francois Mezei" <jfmezei_na...@vaxination.ca> To: Nanog@nanog.org Sent: Tuesday, April 12, 2016 11:51:38 PM Subject: Connecting rural providers: ethernet to large city or nearby transit Generic question. Say you have a municipal provider in small town where the municipality won the subsidy over the incumbent to deploy broadband. The easiest is for the town's ISP to buy transit from the incumbent. But incumbent will not be interested in offering competitive pricing. As a sanity check, would a rural ISP come out ahead getting an ethernet link to large city where cheaper transit is available as well as peering to offload a lot of traffic, or would buying transit at higher price locally end up being better ? Is the difference between the two small, or orders of magnitudes cheaper to go one way or the other ? context: in order to provide affordable backhaul to towns, the CRTC *might consider regulation. The Chairman used a key word today "market failure" indicating they are ready to listen to arguments on this.
Re: Telco Systems
I know of a WISP in Puerto Rico that loves them. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Colton Conor" <colton.co...@gmail.com> To: "NANOG" <nanog@nanog.org> Sent: Tuesday, April 12, 2016 8:07:44 AM Subject: Telco Systems Does anyone use Telco Systems Carrier Ethernet & MPLS Aggregation Switches? I have heard good things about them. Overall, the saying is they price 10G ethernet switches at 1G ethernet pricing. It looks like they support MPLS. http://www.telco.com/index.php?page=product-category=ethernet-mpls-aggregation
Re: GeoIP database issues and the real world consequences
So they launch exhaustive and expensive searches of lakes instead? :-) - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Steve Mikulasik" <steve.mikula...@civeo.com> To: nanog@nanog.org Sent: Monday, April 11, 2016 12:34:35 PM Subject: RE: GeoIP database issues and the real world consequences Just so everyone is clear, Maxmind is changing their default locations. " Now that I’ve made MaxMind aware of the consequences of the default locations it’s chosen, Mather says they’re going to change them. They are picking new default locations for the U.S. and Ashburn, Virginia that are in the middle of bodies of water, rather than people’s homes."
Re: Stop IPv6 Google traffic
That is the problem with some of these companies. They've gotten just as cocky and arrogant as the incumbent telco providers and won't actually tell you what you're doing wrong, but will punish you for doing wrong. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Max Tulyev" <max...@netassist.ua> To: nanog@nanog.org Sent: Sunday, April 10, 2016 9:27:53 AM Subject: Re: Stop IPv6 Google traffic The problem is IPv6-enabled customers complaints see captcha, and Google NOC refuses to help solve it saying like find out some of your customer violating some of our policy. As you can imagine, this is not possible. So, the working solutions is either correctly cut IPv6 to Google, or cut all IPv6 (which I don't want to do). On 10.04.16 17:17, Mike Hammett wrote: > I think the group wants to know what problem you're trying to solve. > Obviously if you block something, there will be a timeout in getting to it. > > What is broken that you're trying to fix by blackholing them? > > > > > - > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > > > Midwest Internet Exchange > http://www.midwest-ix.com > > > - Original Message - > > From: "Max Tulyev" <max...@netassist.ua> > To: nanog@nanog.org > Sent: Sunday, April 10, 2016 9:07:47 AM > Subject: Re: Stop IPv6 Google traffic > > Customers see timeouts if I blackhole Google network. I looking for > alternatives (other than stop providing IPv6 to customers at all). > > On 10.04.16 16:50, valdis.kletni...@vt.edu wrote: >> On Sun, 10 Apr 2016 16:29:39 +0300, Max Tulyev said: >> >>> I need to stop IPv6 web traffic going from our customers to Google >>> without touching all other IPv6 and without blackhole IPv6 Google >>> network (this case my customers are complaining on long timeouts). >>> >>> What can you advice for that? >> >> Umm.. fix the reasons why they're seeing timeouts? :) >> >> Have you determined why the timeouts are happening? >> > > >
Re: Stop IPv6 Google traffic
I think the group wants to know what problem you're trying to solve. Obviously if you block something, there will be a timeout in getting to it. What is broken that you're trying to fix by blackholing them? - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Max Tulyev" <max...@netassist.ua> To: nanog@nanog.org Sent: Sunday, April 10, 2016 9:07:47 AM Subject: Re: Stop IPv6 Google traffic Customers see timeouts if I blackhole Google network. I looking for alternatives (other than stop providing IPv6 to customers at all). On 10.04.16 16:50, valdis.kletni...@vt.edu wrote: > On Sun, 10 Apr 2016 16:29:39 +0300, Max Tulyev said: > >> I need to stop IPv6 web traffic going from our customers to Google >> without touching all other IPv6 and without blackhole IPv6 Google >> network (this case my customers are complaining on long timeouts). >> >> What can you advice for that? > > Umm.. fix the reasons why they're seeing timeouts? :) > > Have you determined why the timeouts are happening? >
Re: Microwave link capacity
A lot of new gear is gigabit. The current price\performance leader is SIAE's ALFOPlus2. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Jean-Francois Mezei" <jfmezei_na...@vaxination.ca> To: Nanog@nanog.org Sent: Monday, April 4, 2016 12:28:41 PM Subject: Microwave link capacity In a context of providing rural communities with modern broadband. Reading some tells me that Microwave links can be raised to 1gbps. How common is that ? I assume that cell phone towers have modern microwave links (when not directly on fibre). What sort of capacity would typically be provided ? And in the case of a remote village/town served by microwave originally designed to handle just phone calls, how difficult/expensive is it to upgrade to 1gbps or higher capacity ? Just a change of radio ? or radio and antenna, keeping only the tower ? (keeping spectrum acquisition out of discussion as that is a whole other ball game).
Re: Microwave link capacity
You might be better served with the lists over at wispa.org. Not saying the people here don't have the answers, but that's what those guys do. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com - Original Message - From: "Jean-Francois Mezei" <jfmezei_na...@vaxination.ca> To: Nanog@nanog.org Sent: Monday, April 4, 2016 12:28:41 PM Subject: Microwave link capacity In a context of providing rural communities with modern broadband. Reading some tells me that Microwave links can be raised to 1gbps. How common is that ? I assume that cell phone towers have modern microwave links (when not directly on fibre). What sort of capacity would typically be provided ? And in the case of a remote village/town served by microwave originally designed to handle just phone calls, how difficult/expensive is it to upgrade to 1gbps or higher capacity ? Just a change of radio ? or radio and antenna, keeping only the tower ? (keeping spectrum acquisition out of discussion as that is a whole other ball game).
