Re: Texas ERCOT power shortages (again) April 13
Can we keep this mailing list free of politics please? Being for or against renewable energy has nothing to do with network operations. - Mike Bolitho On Wed, Apr 14, 2021, 6:31 AM Izaac wrote: > On Wed, Apr 14, 2021 at 06:54:55AM +0200, Mark Tinka wrote: > > So looks like ERCOT have 32,000MW of capacity offline for maintenance and > > repairs, which they claim is not unusual for this time of the year as > they > > gear up for the summer. So generation capacity was only 50,000MW, while > > demand was 49,000MW. 1,000MW in reserve is right on the nose. Solar > > production was also down by 3,000MW due to cloudy skies. > > Sorry guys, I bought 1210MW for impulse delivery, which very briefly ate > that reserve. I can assure you that the next four days of sunny skies > will regenerate it, though. > > In unrelated news, the Rangers got me on an 88MPH speeding ticket. > Anyone know a decent traffic attorney that accepts payment in lotto > tickets? > > -- > . ___ ___ . . ___ > . \/ |\ |\ \ > . _\_ /__ |-\ |-\ \__ >
Re: Ip space Dilemma
Have you written a state legislature yet? Reach out to your representative's offices and let them know. That's part of their job, constituent services. Since they are state government websites, they will have a little power. - Mike Bolitho On Tue, Mar 9, 2021, 7:15 AM Justin Wilson (Lists) wrote: > Folks, > We have an IP block I have asked about help on a few times on > here. This is a block we received from ARIN in June of 2020. We have > several state networks here in Indiana dropping this traffic at their > firewalls. I have been working with them since we discovered this issue in > September. I am not getting anywhere with them and was finally told we > were not a priority. > > I am at the point I need to give the space back because it is > unusable to the ISP customers. Does anyone have any creative ideas on how > to fix this? > > > > Justin Wilson > j...@mtin.net > > — > https://j2sw.com - All things jsw (AS209109) > https://blog.j2sw.com - Podcast and Blog > >
Re: Re Parler
> > And now, with prejudice, I'm requesting that this thread get moderated, > before > anyone *else* volunteers to jump off a bridge. List admins, for real. This has run its course just like I said it would several days ago. It is 100% speculative, has nothing to do with network operations, and requires actual lawyers with access to the case information and witnesses to figure out what's going on. And as Jay said, it's getting stupid. - Mike Bolitho On Thu, Jan 14, 2021 at 5:00 PM Jay R. Ashworth wrote: > - Original Message - > > From: "Mel Beckman" > > > John, > > > > What’s your point? Are you saying that it’s OK for an ISP to break > antitrust > > laws for a political cause? > > No, Mel. > > In very short, he's saying that criminal sedition and armed insurrection > *are > not political causes*, and I am adding that hitching your star to that > wagon > may shorten your career as much as it's shortening the careers of the > people > who were in Washington. > > And now, with prejudice, I'm requesting that this thread get moderated, > before > anyone *else* volunteers to jump off a bridge. > > Cheers, > -- jra > -- > Jay R. Ashworth Baylink > j...@baylink.com > Designer The Things I Think RFC > 2100 > Ashworth & Associates http://www.bcp38.info 2000 Land > Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > 1274 >
Re: Parler
It has nothing to do with networking. Their decision was necessarily political. If you can specifically bring up an issue, beyond speculative, on how their new chosen CDN is somehow now causing congestion or routing issues on the public internet, then great. But as of now, that isn't even a thing. It's just best to leave it alone because it will devolve into chaos. - Mike Bolitho On Sun, Jan 10, 2021, 6:54 AM wrote: > Why? This is extremely relevant to network operators and is not political > at all. > > On Jan 10, 2021, at 8:51 AM, Mike Bolitho wrote: > > > Can we please not go down this rabbit hole on here? List admins? > > - Mike Bolitho > > On Sun, Jan 10, 2021, 1:26 AM William Herrin wrote: > >> Anybody looking for a new customer opportunity? It seems Parler is in >> search of a new service provider. Vendors need only provide all the >> proprietary AWS APIs that Parler depends upon to function. >> >> >> https://www.washingtonpost.com/technology/2021/01/09/amazon-parler-suspension/ >> >> Regards, >> Bill HErrin >> >
Re: Parler
Can we please not go down this rabbit hole on here? List admins? - Mike Bolitho On Sun, Jan 10, 2021, 1:26 AM William Herrin wrote: > Anybody looking for a new customer opportunity? It seems Parler is in > search of a new service provider. Vendors need only provide all the > proprietary AWS APIs that Parler depends upon to function. > > > https://www.washingtonpost.com/technology/2021/01/09/amazon-parler-suspension/ > > Regards, > Bill HErrin >
Re: WhatsApp's New Policy Has...
Zoom bought Keybase. Keybase also has a bit of technical overhead that prevents casual users from adopting. It's why my group chats are migrating to Signal. Having non-tech friends generate key strings and all that... definitely not going to happen. - Mike Bolitho On Fri, Jan 8, 2021 at 12:17 PM Richard Porter wrote: > Has anyone considered or used Keybase? >
Re: Centurylink having a bad morning?
That's all we can do. Thankfully I work for an org that understands this and has *at least* two fully redundant circuits. Sometimes a third smaller carrier if we can prove that it is diverse, but that isn't the case very often. - Mike Bolitho On Mon, Aug 31, 2020 at 7:35 AM Tomas Lynch wrote: > Maybe we are idealizing these so-called tier-1 carriers and we, tier-ns, > should treat them as what they really are: another AS. Accept that they are > going to fail and do our best to mitigate the impact on our own networks, > i.e. more peering. > > On Mon, Aug 31, 2020 at 9:54 AM Martijn Schmidt via NANOG > wrote: > >> At this point you don't even know whether it's a human error (example: >> generating a flowspec rule for port TCP/179), a filtering issue (example: >> accepting a flowspec rule for port TCP/179), or a software issue (example: >> certain flowspec update crashes the BGP daemon). And in the third scenario >> I think that at least some portion of the blame shifts from the carrier to >> its vendors, assuming the thing that crashed was not a home-grown BGP >> implementation. >> >> With the route optimizer incidents - because let's face it, Honest >> Networker is on the money as usual >> https://honestnetworker.net/2020/08/06/as10990-routing/ - there is >> really no excuse for any tier-1 carrier, they should at the very least have >> strict prefix-list based filtering in place for customer-facing EBGP >> sessions. In those cases it's much easier to state who's not taking care of >> their proverbial lawn. >> >> Best regards, >> Martijn >> >> On 8/31/20 3:25 PM, Tom Beecher wrote: >> >> https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ >> >> >> I definitely found Mr. Prince's writing about yesterday's events >> fascinating. >> >> Verizon makes a mistake with BGP filters that allows a secondary mistake >> from leaked "optimizer" routes to propagate, and Mr. Prince takes every >> opportunity to lob large chunks of granite about how terrible they are. >> >> L3 allows an erroneous flowspec announcement to cause massive global >> connectivity issues, and Mr. Prince shrugs and says "Incidents happen." >> >> >> >> >> >> On Mon, Aug 31, 2020 at 1:15 AM Hank Nussbacher >> wrote: >> >>> On 30/08/2020 20:08, Baldur Norddahl wrote: >>> >>> >>> https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ >>> >>> Sounds like Flowspec possibly blocking tcp/179 might be the cause. >>> >>> But that is Cloudflare speculation. >>> >>> Regards, >>> Hank >>> Caveat: The views expressed above are solely my own and do not express >>> the views or opinions of my employer >>> >>> An outage is what it is. I am not worried about outages. We have >>> multiple transits to deal with that. >>> >>> It is the keep announcing prefixes after withdrawal from peers and >>> customers that is the huge problem here. That is killing all the effort and >>> money I put into having redundancy. It is sabotage of my network after I >>> cut the ties. I do not want to be a customer at an outlet who has a system >>> that will do that. Luckily we do not currently have a contract and now they >>> will have to convince me it is safe for me to make a contract with them. If >>> that is impossible I guess I won't be getting a contract with them. >>> >>> But I disagree in that it would be impossible. They need to make a good >>> report telling exactly what went wrong and how they changed the design, so >>> something like this can not happen again. The basic design of BGP is such >>> that this should not happen easily if at all. They did something unwise. >>> Did they make a route reflector based on a database or something? >>> >>> Regards, >>> >>> Baldur >>> >>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho >>> wrote: >>> >>>> Exactly. And asking that they somehow prove this won't happen again is >>>> impossible. >>>> >>>> - Mike Bolitho >>>> >>>> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver >>>> wrote: >>>> >>>>> I’m not defending them but I am sure it isn’t intentional. >>>>> >>>>> >>>>> >>>>> *From:* NANOG *On >>>>> Behalf Of *Baldur Norddahl >>>>> *Sent:* Sunday, August 30, 2020 9:28 AM >>&
Re: Does anyone actually like CenturyLink?
Maybe we should start an "Uptime mailing list" ha! But yeah, when things are working well nobody talks about it. The CTL network is very large. However, it's clear their blast radius mentality isn't real great. We saw this yesterday. We saw this Dec 2018. Global outages shouldn't be a thing. - Mike Bolitho On Mon, Aug 31, 2020, 6:31 AM Tom Beecher wrote: > I've never heard a single positive word about them >> > > There is rarely much in the way of emails/messages sent about things when > they work well. > > On Sun, Aug 30, 2020 at 11:03 AM Ross Tajvar wrote: > >> I've never heard a single positive word about them, and I've had my fair >> share of issues myself (as an indirect customer). But it seems that lots of >> people put them in their transit blend. Other than lack of options, why >> would anyone use them? To me, it just seems like asking for trouble...but >> maybe I'm missing something? >> >
Re: Centurylink having a bad morning?
Exactly. And asking that they somehow prove this won't happen again is impossible. - Mike Bolitho On Sun, Aug 30, 2020, 8:10 AM Drew Weaver wrote: > I’m not defending them but I am sure it isn’t intentional. > > > > *From:* NANOG *On Behalf > Of *Baldur Norddahl > *Sent:* Sunday, August 30, 2020 9:28 AM > *To:* nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > How is that acceptable behaviour? I shall remember never to make a > contract with these guys until they can prove that they won't advertise my > prefixes after I pull them. Under any circumstances. > > > > søn. 30. aug. 2020 15.14 skrev Joseph Jenkins >: > > Finally got through on their support line and spoke to level1. The only > thing the tech could say was it was an issue with BGP route reflectors and > it started about 3am(pacific). They were still trying to isolate the issue. > I've tried failing over my circuits and no go, the traffic just dies as L3 > won't stop advertising my routes. > > > > On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG > wrote: > > Hello, > > > > Woke up this morning to a bunch of reports of issues with connectivity had > to shut down some Level3/CTL connections to get it to return to normal. > > > > As of right now their support portal won’t load: > https://www.centurylink.com/business/login/ > > > > Just wondering what others are seeing. > > > >
Re: Does anyone actually like CenturyLink?
This is on the Legacy Level 3 side which is generally much better and still largely separate (according to my former co-workers). So say what you want about CTL but this is an LVLT issue. - Mike Bolitho On Sun, Aug 30, 2020, 8:03 AM Ross Tajvar wrote: > I've never heard a single positive word about them, and I've had my fair > share of issues myself (as an indirect customer). But it seems that lots of > people put them in their transit blend. Other than lack of options, why > would anyone use them? To me, it just seems like asking for trouble...but > maybe I'm missing something? >
Re: Question on BlackBox or Commworks
We currently use BlackBox and they use their own techs where I'm at (Phoenix). We also used them extensively when I worked for Level 3 several years ago. As with anything, your experience with them will vary largely by location and can even vary within a market. I have dealt with some awesome BlackBox techs. And I have dealt with some really bad BlackBox techs. The guys I work with on a regular basis in Phoenix are great. - Mike Bolitho On Wed, Jul 29, 2020 at 10:05 AM Joseph Jenkins wrote: > Do you know or have experience with either company? Do they have their own > techs are they just bidding out for local techs in the area? I have work > that needs to be done all across the US and just trying to look for some > options. >
Re: Contact at Ubiquiti Networks?
>>Your or my pet bug may never get fixed, based on market demand. That’s simply capitalism, not low quality. No, that's low quality, full stop. Bugs need to be fixed in software that you are selling. I bought a product and I expect it to work. If they are going to tout themselves as enterprise grade, which they do (*Narrator: They're not*), then they need to fix bugs in their production software. - Mike Bolitho On Tue, May 26, 2020 at 6:44 AM Mel Beckman wrote: > I deploy Ubiquiti equipment quite a bit, both in WLANs and WISP > distribution networks. It’s excellent quality at a dirt cheap price. As > with all software-based products, there will be bugs. Your or my pet bug > may never get fixed, based on market demand. That’s simply capitalism, not > low quality. None of us can afford to pay for perfection, because it would > never ship. > > I deployed 400 HP-Aruba APs at SFO, and that installation requires a > full-time network engineer to manage the system. I’ve deployed many more > times that in Unifi APs and they run perfectly well with only periodic > software updates to accommodate new client device types. Unifi is 75% > cheaper than Aruba, for essentially the same result. > > -mel > > > On May 26, 2020, at 6:29 AM, Matt Hoppes < > mattli...@rivervalleyinternet.net> wrote: > > > > Except, you could argue they are exceling. Stocks are going up up up, > and folks buy the product. > > > > I really wish stock holders would ask the proper questions in the > quarterly calls. > > > >> On 5/26/20 8:53 AM, Mike Hammett wrote: > >> That is a big problem. In terms of their UniFi product line, there are > no reasonable alternatives. > >> Upper management is the biggest problem. They have severe ADD. > >> A ton of companies have these kinds of issues. They just plain don't > hire enough people in the right areas to really excel. > >> - > >> Mike Hammett > >> Intelligent Computing Solutions <http://www.ics-il.com/> > >> <https://www.facebook.com/ICSIL>< > https://plus.google.com/+IntelligentComputingSolutionsDeKalb>< > https://www.linkedin.com/company/intelligent-computing-solutions>< > https://twitter.com/ICSIL> > >> Midwest Internet Exchange <http://www.midwest-ix.com/> > >> <https://www.facebook.com/mdwestix>< > https://www.linkedin.com/company/midwest-internet-exchange>< > https://twitter.com/mdwestix> > >> The Brothers WISP <http://www.thebrotherswisp.com/> > >> <https://www.facebook.com/thebrotherswisp>< > https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > >> > >> *From: *"Ben P" > >> *To: *"Mike Hammett" > >> *Cc: *"j k" , "NANOG list" > >> *Sent: *Tuesday, May 26, 2020 5:01:36 AM > >> *Subject: *Re: Contact at Ubiquiti Networks? > >> Agree 1000% with the sentiments expressed by Mike. > >> Unfortunately despite much research I’ve been unable to find a suitable > replacement vendor. All the other vendors seem to want to ram > cloud-management down your throat which I absolutely do not want. My > network, my control, not under the auspices of someone else’s magic cloud. > >>On 25 May 2020, at 21:21, Mike Hammett >><mailto:na...@ics-il.net>> wrote: > >>The company has mostly fallen apart. Their sales are going up, but > >>their responsiveness and customer support have been declining over > >>the last five years. > >>- > >>Mike Hammett > >>Intelligent Computing Solutions <http://www.ics-il.com/> > >><https://www.facebook.com/ICSIL>< > https://plus.google.com/+IntelligentComputingSolutionsDeKalb>< > https://www.linkedin.com/company/intelligent-computing-solutions>< > https://twitter.com/ICSIL> > >>Midwest Internet Exchange <http://www.midwest-ix.com/> > >><https://www.facebook.com/mdwestix>< > https://www.linkedin.com/company/midwest-internet-exchange>< > https://twitter.com/mdwestix> > >>The Brothers WISP <http://www.thebrotherswisp.com/> > >><https://www.facebook.com/thebrotherswisp>< > https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > >> > > >>*From:*"j k" mailto:jskl...@gmail.com>> > >>*To:*"NANOG list" mailto:nanog@nanog.org>> > >>*Sent:*Monday, May 25, 2020 3:16:36 PM > >>*Subject:*Contact at Ubiquiti Networks? > >>Does anyone have a good contact at Ubiquity Networks? Finding a > >>pattern I don't like. > >>Joe Klein > >>"inveniet viam, aut faciet"^ --- Seneca's Hercules Furens (Act II, > >>Scene 1) > >>"I never lose. I either win or learn" - Nelson Mandela >
Re: COVID-19 vs. our Networks
> > "It is something that matters, because it has the potential to set a > dangerous precedent." > Can we stop with this talk... around everything? We're literally living through an unprecedented event right now. My 86 year old grandmother said she's never seen anything like this in the US. My friends 94 year old grandmother in Italy said she hasn't seen this since WWII. Nobody is going to say "Well we did this during a global pandemic so we can now do it because we feel like it". People will laugh them out of the room. I live in Phoenix, the mayor shut down bars and restaurants (carryout only) in order to help stop us from becoming Italy. One of our city councilmen was saying the same thing: "This is martial law and sets bad precedent! We must open everything up!" Of course, they then held a closed to the public meeting because city council can't be exposed. The point is, the mayor isn't going to do the same thing in six months on a whim because traffic on the freeway is bad. Thankfully calmer heads prevailed and the rest of the council told him to pound sand, at least for now. Something that keeps happening on this mailing list over the last few weeks is this tendency to try to take the "Moral high ground". And from way up there people are looking at the whole topic from an idealistic point of view like we live in some Network Operators Utopia with perfect conditions where money doesn't exist and we can do whatever we want because there is no upper management. We should be having a practical conversation that sits within the confines of reality. We don't have perfect networks built. We don't have unlimited resources. We are facing a global pandemic. Money is tight. In principle, I agree with what you guys are saying. But in reality, we're going to have to bend our convictions in order to protect populations from COVID-19. You will be changing your tune when your mother is sick and can't get the care she needs because the system is overwhelmed because we (communities, not just network operators) didn't do what was necessary because of some idealistic hard line people drew in the sand. - Mike Bolitho On Fri, Mar 20, 2020 at 7:44 AM Tom Beecher wrote: > It is something that matters, because it has the potential to set a > dangerous precedent. > > If you say "$Service should reduce their bit rates because this is an > emergency!" , I guarantee that exact same argument will be made well after > this crisis has passed with a different definition of "emergency", and > adding on "well it's an emergency to me!". > > Some of the pipes Netflix goes through is also used by other services that >> aren't as adaptable. >> > > And how is that Netflix's responsibility? They have already taken action > to ramp down bitrates when they detect congestion. Why should other > applications be able to say piss off, I don't want to? Didn't we just have > a 10 year net neutrality argument that we're not supposed to want to treat > the bits differently? > > On Fri, Mar 20, 2020 at 10:17 AM Mike Hammett wrote: > >> It's one of those most important things that matters. >> >> The end user likely won't notice the difference between 4k and 720p. They >> also aren't likely to notice the transition from one to the other. >> >> The person on the VPN, VoIP call, video conference, video game, etc. will >> very much notice the congested link, even if it's only a few seconds. >> >> >> Yes, Netflix video is very efficient, if not the most efficient. They're >> also one of if not the largest slingers of bits on the Internet. Small >> changes in usage of such a huge player totally eclipse most other usages on >> the Internet. >> >> https://help.netflix.com/en/node/306 >> >> Netflix recommends 25 megs for Ultra HD, while only 5 megs for HD. That's >> a 5x difference in something people likely won't notice and would make a >> big difference on the additional VPN, VoIP, video conferencing, etc. >> >> >> >> - >> Mike Hammett >> Intelligent Computing Solutions <http://www.ics-il.com/> >> <https://www.facebook.com/ICSIL> >> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> >> <https://www.linkedin.com/company/intelligent-computing-solutions> >> <https://twitter.com/ICSIL> >> Midwest Internet Exchange <http://www.midwest-ix.com/> >> <https://www.facebook.com/mdwestix> >> <https://www.linkedin.com/company/midwest-internet-exchange> >> <https://twitter.com/mdwestix> >> The Brothers WISP <http://www.thebrotherswisp.com/> >> <https://www.facebook.com/thebrotherswisp> >> <https://www.youtube.com/channe
Re: COVID-19 vs. our Networks
*Restoration:* *The repair or returning to service of one or more telecommunications services that have experienced a service outage or are unusable for any reason, including a damaged or impaired telecommunications facility. Such repair or returning to service may be done by patching, rerouting, substitution of component parts or pathways, and other means, as determined necessary by a service vendor.* https://www.cisa.gov/sites/default/files/publications/OEC%20TSP%20Operations%20Guide%20Final%2012062016_FINAL%20508C.pdf My understanding, and what we did while I worked for a Tier I ISP, was that even for degraded circuits we had to do everything in our power to restore to full operations. If capacity is an issue and causes TSP coded DIA circuits to be unusable then that falls under the "any reason" clause of that line. - Mike Bolitho On Thu, Mar 19, 2020 at 10:05 AM Tom Beecher wrote: > Yes, you have said that. I still believe you are incorrect. > > TSP allows priority for turnup of new capacity , and priority restoration > for capacity. There is nothing in the regulations that I can find that > would allow TSP to be used to rectify general internet congestion issues. > > On Thu, Mar 19, 2020 at 12:53 PM Mike Bolitho > wrote: > >> I've said it over and over again, we have TSP and it could easily be used >> to enforce priority to emergency preparedness customers. It's built into >> the language. >> >> - Mike Bolitho >> >> >> On Thu, Mar 19, 2020 at 9:52 AM Tom Beecher wrote: >> >>> EU regulations with such things are vastly different than in the US. >>> >>> On Thu, Mar 19, 2020 at 12:08 PM Mike Bolitho >>> wrote: >>> >>>> I was getting blasted earlier for suggesting streaming services and >>>> gaming DLCs could likely be slowed by government intervention. EU is >>>> currently working with Netflix to do just that. It's currently a strong >>>> suggestion and even a plead but I maintain that we're going to see this >>>> pushed harder in the coming weeks. >>>> >>>> In a statement on Thursday, Breton said that given the unprecedented >>>> situation, streaming platforms, telecom operators and users "all have a >>>> joint responsibility to take steps to ensure the smooth functioning of the >>>> internet during the battle against the virus propagation." >>>> >>>> >>>> https://www.cnn.com/2020/03/19/tech/netflix-internet-overload-eu/index.html >>>> >>>> >>>> - Mike Bolitho >>>> >>>> >>>> On Thu, Mar 19, 2020 at 5:03 AM Mark Tinka >>>> wrote: >>>> >>>>> >>>>> >>>>> On 19/Mar/20 04:35, Scott Weeks wrote: >>>>> > >>>>> > >>>>> > We do about 70-80Gbps at peak over the external >>>>> > BGP links we have and I am not seeing a large >>>>> > increase nor am I seeing it spread out over time. >>>>> > We're an eyeball network plus some really large >>>>> > customers. >>>>> > >>>>> > Anyone else seeing something different? We're >>>>> > now into the 3rd day, so I thought I'd see >>>>> > something change by now. >>>>> >>>>> South Africa and a few other African countries put countries on >>>>> semi-lockdown from about Sunday. >>>>> >>>>> We've seen a 15% increase in peak traffic on our network since the >>>>> 17th. >>>>> >>>>> Mark. >>>>> >>>>
Re: COVID-19 vs. our Networks
I've said it over and over again, we have TSP and it could easily be used to enforce priority to emergency preparedness customers. It's built into the language. - Mike Bolitho On Thu, Mar 19, 2020 at 9:52 AM Tom Beecher wrote: > EU regulations with such things are vastly different than in the US. > > On Thu, Mar 19, 2020 at 12:08 PM Mike Bolitho > wrote: > >> I was getting blasted earlier for suggesting streaming services and >> gaming DLCs could likely be slowed by government intervention. EU is >> currently working with Netflix to do just that. It's currently a strong >> suggestion and even a plead but I maintain that we're going to see this >> pushed harder in the coming weeks. >> >> In a statement on Thursday, Breton said that given the unprecedented >> situation, streaming platforms, telecom operators and users "all have a >> joint responsibility to take steps to ensure the smooth functioning of the >> internet during the battle against the virus propagation." >> >> >> https://www.cnn.com/2020/03/19/tech/netflix-internet-overload-eu/index.html >> >> >> - Mike Bolitho >> >> >> On Thu, Mar 19, 2020 at 5:03 AM Mark Tinka wrote: >> >>> >>> >>> On 19/Mar/20 04:35, Scott Weeks wrote: >>> > >>> > >>> > We do about 70-80Gbps at peak over the external >>> > BGP links we have and I am not seeing a large >>> > increase nor am I seeing it spread out over time. >>> > We're an eyeball network plus some really large >>> > customers. >>> > >>> > Anyone else seeing something different? We're >>> > now into the 3rd day, so I thought I'd see >>> > something change by now. >>> >>> South Africa and a few other African countries put countries on >>> semi-lockdown from about Sunday. >>> >>> We've seen a 15% increase in peak traffic on our network since the 17th. >>> >>> Mark. >>> >>
Re: COVID-19 vs. our Networks
I was getting blasted earlier for suggesting streaming services and gaming DLCs could likely be slowed by government intervention. EU is currently working with Netflix to do just that. It's currently a strong suggestion and even a plead but I maintain that we're going to see this pushed harder in the coming weeks. In a statement on Thursday, Breton said that given the unprecedented situation, streaming platforms, telecom operators and users "all have a joint responsibility to take steps to ensure the smooth functioning of the internet during the battle against the virus propagation." https://www.cnn.com/2020/03/19/tech/netflix-internet-overload-eu/index.html - Mike Bolitho On Thu, Mar 19, 2020 at 5:03 AM Mark Tinka wrote: > > > On 19/Mar/20 04:35, Scott Weeks wrote: > > > > > > We do about 70-80Gbps at peak over the external > > BGP links we have and I am not seeing a large > > increase nor am I seeing it spread out over time. > > We're an eyeball network plus some really large > > customers. > > > > Anyone else seeing something different? We're > > now into the 3rd day, so I thought I'd see > > something change by now. > > South Africa and a few other African countries put countries on > semi-lockdown from about Sunday. > > We've seen a 15% increase in peak traffic on our network since the 17th. > > Mark. >
Re: COVID-19 vs. our Networks
>You're facing essentially the same issue as many in non-healthcare do ; how to best talk to applications in Magic Cloud Land. Reaching the major cloud providers does not require DIA ; they all have presences on the major IXes, and direct peering could be an option too depending on your needs and traffic. I totally agree and 99.999% of the time, congestion on the Internet is a nuisance, not a critical problem. I'm not sitting here complaining that my public internet circuits don't have SLAs or that we run into some packet loss and latency here and there under normal operations. That's obviously to be expected. But this whole topic is around what to do when a once in a lifetime pandemic hits and we're faced with unseen levels of congestion across the country's infrastructure. I mean the thread is titled COVID-19 Vs Our Networks. That's why I brought up the possible application of TSP to tell some of the big CDNs that maybe they should limit 4K streaming or big DLCs during a pandemic. That's it. And yet I'm getting chastised (not necessarily by you) for suggesting that hospitals, governments, water treatment plants, power plants, first responders, etc are actually more important during times like this. - Mike Bolitho On Tue, Mar 17, 2020 at 10:35 AM Tom Beecher wrote: > You're facing essentially the same issue as many in non-healthcare do ; > how to best talk to applications in Magic Cloud Land. Reaching the major > cloud providers does not require DIA ; they all have presences on the major > IXes, and direct peering could be an option too depending on your needs and > traffic. > > I don't mean to be dismissive of the issues you face, I apologize if > that's how it comes off. What you describe is certainly challenging, but I > think that you will have better success with some of the options that are > out there already than hoping for any resolution of intermittent congestion > issues in the wild west of the DFZ. > > On Tue, Mar 17, 2020 at 1:03 PM Mike Bolitho > wrote: > >> >The answer is don't shove application traffic that has tight service >> level requirements onto the public internet at large and expect the same >> performance as private circuits or other SLA protected services. >> >> I keep seeing this over and over again in this long thread. What's your >> suggestion? How does a hospital, with dozens of third party >> applications/devices across multiple cloud platforms do this? >> >> We have two redundant private lines out of each hospital connecting back >> to primary and DR DCs and a metro connecting everything together in each >> region. But for things we do not own that are not hosted locally, what are >> we supposed to do? We have to go out DIA to get there. Everything we own is >> connected via fully SLAed private lines. We have zero issues there. I think >> people vastly underestimate just how much in the healthcare vertical is >> outside of a medical providers control/ownership. >> >> - Mike Bolitho >> >> >> On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher wrote: >> >>> The answer is don't shove application traffic that has tight service >>> level requirements onto the public internet at large and expect the same >>> performance as private circuits or other SLA protected services. >>> >>> >>> >>> On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho >>> wrote: >>> >>>> If an x-ray machine won't work because the Internet is down, I'm not sure >>>>> that is responsible. As inefficient as it may be to have a license server >>>>> on-prem if there is an option to check against one in the public cloud, >>>>> for a medical use-case, that would make more sense to me. >>>> >>>> >>>> Totally agree with you. Unfortunately it's not a problem with the >>>> medical providers, it's a problem with the medical devices. Anybody who >>>> works in the healthcare vertical will tell you just how bad medical devices >>>> are to work with from an IT perspective. And that is part of my >>>> original comments. >>>> >>>> In your case, I am not sure I have an answer for you, unfortunately. >>>>> The public Internet is what it is, mostly best-effort. Your applications >>>>> and use-cases certainly deserve better than that. I'm not sure how to >>>>> achieve that as your industry shoves more and more activity into the >>>>> public >>>>> Internet domain, for one reason or another. >>>> >>>> >>>> I don't know what it's going to take either. A general shift in >>>> mentality from the vend
Re: COVID-19 vs. our Networks
>The answer is don't shove application traffic that has tight service level requirements onto the public internet at large and expect the same performance as private circuits or other SLA protected services. I keep seeing this over and over again in this long thread. What's your suggestion? How does a hospital, with dozens of third party applications/devices across multiple cloud platforms do this? We have two redundant private lines out of each hospital connecting back to primary and DR DCs and a metro connecting everything together in each region. But for things we do not own that are not hosted locally, what are we supposed to do? We have to go out DIA to get there. Everything we own is connected via fully SLAed private lines. We have zero issues there. I think people vastly underestimate just how much in the healthcare vertical is outside of a medical providers control/ownership. - Mike Bolitho On Tue, Mar 17, 2020 at 9:54 AM Tom Beecher wrote: > The answer is don't shove application traffic that has tight service level > requirements onto the public internet at large and expect the same > performance as private circuits or other SLA protected services. > > > > On Tue, Mar 17, 2020 at 11:40 AM Mike Bolitho > wrote: > >> If an x-ray machine won't work because the Internet is down, I'm not sure >>> that is responsible. As inefficient as it may be to have a license server >>> on-prem if there is an option to check against one in the public cloud, >>> for a medical use-case, that would make more sense to me. >> >> >> Totally agree with you. Unfortunately it's not a problem with the medical >> providers, it's a problem with the medical devices. Anybody who works in >> the healthcare vertical will tell you just how bad medical devices are to >> work with from an IT perspective. And that is part of my original comments. >> >> In your case, I am not sure I have an answer for you, unfortunately. The >>> public Internet is what it is, mostly best-effort. Your applications and >>> use-cases certainly deserve better than that. I'm not sure how to achieve >>> that as your industry shoves more and more activity into the public >>> Internet domain, for one reason or another. >> >> >> I don't know what it's going to take either. A general shift in mentality >> from the vendors we use I guess. I'm not sure how you get a bunch of >> medical providers to tell these companies they need to fix their stuff. You >> can't exactly use your wallet to force change either. There are only a >> handful of vendor options out there so there isn't a ton of choice. It's >> not like you can buy one of 50 different models of CT machines or EHR >> systems. >> >> Generally speaking it's not an issue. It's just in crazy times like these >> where, if congestion on the public internet gets too crazy, that certain >> platforms might need to be deemed "unnecessary". Is playing Fortnight a >> right? Is streaming a movie in 4K a right? In cases like San Francisco they >> have decided that leaving your home for anything other than work or medical >> care is no longer a right because you're now infringing on other's rights >> by potentially getting them sick. Maybe 4K Netflix fits into that category >> if you're causing problems for first responders and hospitals trying to >> save lives. >> >> >> - Mike Bolitho >> >> >> On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka wrote: >> >>> >>> >>> On 16/Mar/20 16:54, Carsten Bormann wrote: >>> >>> > I recently had to reschedule an X-ray because the license manager for >>> the X-ray machine was acting up. I don’t think people have a grasp for how >>> much of the medical infrastructure no longer works when the Internet is >>> down. >>> >>> I get this, to some extent. But also, there is a reason hospitals, >>> airports and military installations are either put on special power >>> grids or invest plenty of money in backup power. >>> >>> If an x-ray machine won't work because the Internet is down, I'm not >>> sure that is responsible. As inefficient as it may be to have a license >>> server on-prem if there is an option to check against one in the public >>> cloud, for a medical use-case, that would make more sense to me. >>> >>> Mark. >>> >>
Re: COVID-19 vs. our Networks
> > If an x-ray machine won't work because the Internet is down, I'm not sure > that is responsible. As inefficient as it may be to have a license server > on-prem if there is an option to check against one in the public cloud, > for a medical use-case, that would make more sense to me. Totally agree with you. Unfortunately it's not a problem with the medical providers, it's a problem with the medical devices. Anybody who works in the healthcare vertical will tell you just how bad medical devices are to work with from an IT perspective. And that is part of my original comments. In your case, I am not sure I have an answer for you, unfortunately. The > public Internet is what it is, mostly best-effort. Your applications and > use-cases certainly deserve better than that. I'm not sure how to achieve > that as your industry shoves more and more activity into the public > Internet domain, for one reason or another. I don't know what it's going to take either. A general shift in mentality from the vendors we use I guess. I'm not sure how you get a bunch of medical providers to tell these companies they need to fix their stuff. You can't exactly use your wallet to force change either. There are only a handful of vendor options out there so there isn't a ton of choice. It's not like you can buy one of 50 different models of CT machines or EHR systems. Generally speaking it's not an issue. It's just in crazy times like these where, if congestion on the public internet gets too crazy, that certain platforms might need to be deemed "unnecessary". Is playing Fortnight a right? Is streaming a movie in 4K a right? In cases like San Francisco they have decided that leaving your home for anything other than work or medical care is no longer a right because you're now infringing on other's rights by potentially getting them sick. Maybe 4K Netflix fits into that category if you're causing problems for first responders and hospitals trying to save lives. - Mike Bolitho On Tue, Mar 17, 2020 at 2:22 AM Mark Tinka wrote: > > > On 16/Mar/20 16:54, Carsten Bormann wrote: > > > I recently had to reschedule an X-ray because the license manager for > the X-ray machine was acting up. I don’t think people have a grasp for how > much of the medical infrastructure no longer works when the Internet is > down. > > I get this, to some extent. But also, there is a reason hospitals, > airports and military installations are either put on special power > grids or invest plenty of money in backup power. > > If an x-ray machine won't work because the Internet is down, I'm not > sure that is responsible. As inefficient as it may be to have a license > server on-prem if there is an option to check against one in the public > cloud, for a medical use-case, that would make more sense to me. > > Mark. >
Re: COVID-19 vs. our Networks
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers do, GoToMeeting, O365, VaaS, etc. There's no practical way to engineer your WAN to facilitate dozens of connections to these services. This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of whether or not those entities planned well (I think we did), the government should and will step in if critical services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to utilization and the impact that will have on $aaS products that are interwoven into every single vertical, including entities that fall under TSP, critical national security and emergency preparedness functions, including those areas related to safety, maintenance of law and order, and public health. It's easy for all you guys to sit here and armchair quarterback other people's planning but when things really start to degrade, all bets are off. If you don't believe that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, private schools) and banning people from associating in groups of larger than 50. *The opinions expressed here are my own and do not represent my employer or their views.* - Mike Bolitho On Sun, Mar 15, 2020 at 6:12 PM Stephen Fulton wrote: > In $dayjob I constantly see the lack of understanding of the difference > between what the Internet is and what a path engineered private circuit > is (eg. pseudowire, wave, whatever). The latest fight is over SD-WAN > and those who think it will replace MPLS entirely and they won't need > those expensive routers anymore. But I digress. > > Mark's comment and others like it are the correct approach Mike. If > your private WAN is most critical, then invest in and manage user > complaints about poor Internet service. ISP's, IXP's and CDN's are not > going to twist themselves into knots to solve your problems, even if > someone calls it an emergency. Sorry. > > Stephen > > > On 2020-03-15 02:01, Mark Tinka wrote: > > > > > > On 14/Mar/20 19:14, Mike Bolitho wrote: > > > >> / > >> / > >> > >> I work for a hospital, we ran into some issues last week due to > >> congestion that was totally outside of our control that was off of our > >> WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission > >> critical at the time but it was still disruptive. As more and more > >> people are driven home during this time, more and more people will be > >> using bandwidth intensive streaming and online gaming products. If > >> more and more TSP coded entities are running into issues, ISPs, IXPs, > >> and CDNs will be forced to act. > > > > Hmmh, if that level of priority is required, I'd probably build my own > > network, and not rely on public infrastructure like the Internet. > > > > Mark. >
Re: COVID-19 vs. our Networks
First of all, we use a mixture of layer 2/3 private lines and DIA circuits. You don't know our infrastructure, stop being condescending. It goes against the spirit of this mailing list. Second, yes, the Internet is protected. Both public and private lines. I know this because we have TSP coded circuits and I spent four years at a Tier I ISP servicing TSP coded circuits Third, the trouble we had was a third party service having congestion issues. They are hosted by the same CDN as Call of Duty. The problem was both outside of our control and our third party service's control. The chokepoint was between ISPs/IXPs and the CDN. I've seen this time and time again while working at the aforementioned ISP. Saturated links on ISP/IXP/CDN networks. This is where the TSP code comes in. In this day and age of cloud services, it is financially unfeasible for every company to have a private line to every single cloud provider. That's preposterous to even suggest. - Mike Bolitho On Sat, Mar 14, 2020 at 10:40 AM Clayton Zekelman wrote: > > > The Internet is not a telecommunications service, according to your FCC. > If you want predictability, buy WAN circuits, not Internet circuits. If > your provider is co-mingling Internet and WAN traffic (i.e. circuits with > defined endpoints vs. public Internet or VPN), then you need to talk to > them about their prioritization. > > If you have mission critical applications, put them on mission critical > infrastructure, not the public Internet. > > Oh, that's right - Internet circuits are cheaper than WAN circuits > Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
Re: COVID-19 vs. our Networks
> > *Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. > Who determines what is "mission critical"? Our mission as an ISP is to > deliver Internet to our customers. If they want to play online games or > watch video, who am I to say that isn't critical to THEIR mission?...* *...The last thing we need are a bunch of kids in quarantine that have > NOTHING to do because Mike Bolitho thinks their entertainment isn't part of > the "mission" of the Internet.* We already have that. It's called Telecommunications Service Priority and this is the charge: Telecommunications Service Priority (TSP) is a program that authorizes > national security and emergency preparedness (NS/EP) organizations to > receive priority treatment for vital voice and data circuits or other > telecommunications services. I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act. For more information: https://www.cisa.gov/telecommunications-service-priority-tsp https://www.fcc.gov/general/telecommunications-service-priority https://en.wikipedia.org/wiki/Telecommunications_Service_Priority *These views are my own and do not reflect the opinions or official stances of my employer etc etc.* - Mike Bolitho On Sat, Mar 14, 2020 at 9:27 AM Clayton Zekelman wrote: > > Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. > Who determines what is "mission critical"? Our mission as an ISP is to > deliver Internet to our customers. If they want to play online games or > watch video, who am I to say that isn't critical to THEIR mission? > > The last thing we need are a bunch of kids in quarantine that have NOTHING > to do because Mike Bolitho thinks their entertainment isn't part of the > "mission" of the Internet. > > About the only thing that might be useful is something to smooth out the > big jumps in utilization on game releases - but even that is something that > can be managed by adding capacity. > > To quote Jay Leno - Crunch All You Want, We'll Make More. > > At 12:16 PM 14/03/2020, Mike Bolitho wrote: > > Basically that. It's probably more streaming services that could crowd out > what would be considered "mission critical" infrastructure. Maybe the > Netflixs and Hulus of the world will limit 4K streaming or something along > those lines. Basically cap resolution to 720p for the time being. > > - Mike Bolitho > > > On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert wrote: > > The impact of all these bored school kids on the networks due to gaming > might cause some issues. I know that if I'm working from home and my > videoconferencing slows down because of someones gaming, I'm taking the > necessary action (read, change some rules on my firewall). > > People are welcome to do whatever they want on their own networks. I just > didn't get the suggestion that online gaming services would shut down. Or > were you saying, Mike, that online gaming would crowd out other services > and so "shut down" those other services? > > On Fri., Mar. 13, 2020, 21:42 Owen DeLong wrote: > You don’t have kids, do you… > > They have the attention span off Koi these days. They’ll play most games > for about 15 minutes or so before downloading the next one. (At least > that’s been my observation of behavior among my GF’s daughter and her > friends). > > Owen > > > On Mar 13, 2020, at 20:31 , Darin Steffl wrote: > > Playing games doesn't take much bandwidth. Downloading games does. So as > long as everyone already has their games and there's no updates, playing > the game is typically under 100 kbps which is negligible compared to > streaming video which takes 1 to 25 mbps. > > On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha wrote: > Hi, > > I don't know where y'all live, but here in the SF Bay Area, pretty much > all public and private schools have closed down. My school district (in > Santa Clara County) will be closed until Spring Break. > > The impact of all these bored school kids on the networks due to gaming > might cause some issues. I know that if I'm working from home and my > videoconferencing slows down because of someones gaming, I'm taking the > necessary action (read, change some rules on my firewall). > > Thanks, > > Sabri > > > --
Re: COVID-19 vs. our Networks
Basically that. It's probably more streaming services that could crowd out what would be considered "mission critical" infrastructure. Maybe the Netflixs and Hulus of the world will limit 4K streaming or something along those lines. Basically cap resolution to 720p for the time being. - Mike Bolitho On Sat, Mar 14, 2020 at 1:06 AM Hugo Slabbert wrote: > > The impact of all these bored school kids on the networks due to gaming > might cause some issues. I know that if I'm working from home and my > videoconferencing slows down because of someones gaming, I'm taking the > necessary action (read, change some rules on my firewall). > > People are welcome to do whatever they want on their own networks. I just > didn't get the suggestion that online gaming services would shut down. Or > were you saying, Mike, that online gaming would crowd out other services > and so "shut down" those other services? > > On Fri., Mar. 13, 2020, 21:42 Owen DeLong wrote: > >> You don’t have kids, do you… >> >> They have the attention span of Koi these days. They’ll play most games >> for about 15 minutes or so before downloading the next one. (At least >> that’s been my observation of behavior among my GF’s daughter and her >> friends). >> >> Owen >> >> >> On Mar 13, 2020, at 20:31 , Darin Steffl wrote: >> >> Playing games doesn't take much bandwidth. Downloading games does. So as >> long as everyone already has their games and there's no updates, playing >> the game is typically under 100 kbps which is negligible compared to >> streaming video which takes 1 to 25 mbps. >> >> On Fri, Mar 13, 2020, 8:52 PM Sabri Berisha >> wrote: >> >>> Hi, >>> >>> I don't know where y'all live, but here in the SF Bay Area, pretty much >>> all public and private schools have closed down. My school district (in >>> Santa Clara County) will be closed until Spring Break. >>> >>> The impact of all these bored school kids on the networks due to gaming >>> might cause some issues. I know that if I'm working from home and my >>> videoconferencing slows down because of someones gaming, I'm taking the >>> necessary action (read, change some rules on my firewall). >>> >>> Thanks, >>> >>> Sabri >>> >>> >>> - On Mar 13, 2020, at 4:12 PM, Hugo Slabbert >>> wrote: >>> >>> I think under circumstances like this, I could definitely see some of >>>> the online based games shutting services down. >>> >>> >>> How so? >>> >>> Signed, >>> >>> Someone who works for an online gaming company and has heard nothing of >>> this. >>> >>> -- >>> Hugo Slabbert | email, xmpp/jabber: h...@slabnet.com >>> pgp key: B178313E | also on Signal >>> >>> >>> On Fri, Mar 13, 2020 at 2:52 PM Mike Bolitho >>> wrote: >>> >>>> I think under circumstances like this, I could definitely see some of >>>> the online based games shutting services down. >>>> >>>> - Mike Bolitho >>>> >>>> >>>> On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno wrote: >>>> >>>>> Its already happening in Italy, and now that schools are shutting down >>>>> here as well, its going to get interesting: >>>>> >>>>> https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-strain-network-with-fortnite-marathon >>>>> >>>>> The ultimate traffic test is coming, looking forward to hearing about >>>>> it on this thread. >>>>> >>>>> Maybe its a good time to start a communication channel between content >>>>> providers/gaming companies and ISPs/CDNs. >>>>> >>>>> >>>>> On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 12, 2020 at 3:46 PM g...@1337.io wrote: >>>>>> >>>>>>> With talk of there being an involuntary statewide (WA) and then >>>>>>> national quarantines (house arrest) for multiple weeks, has anyone put >>>>>>> thought into the impacts of this on your networks if/when this comes to >>>>>>> fruition? >>>>>>> >>>>>>> We're already pushing the limits with telecommuters / those that are >>>>>>> WFH, but I can only imagine what things will look like with everyone >>>>>>> stuck >>>>>>> at home for any duration of time. >>>>>>> >>>>>> >>>>>> >>>>>> People will turn to you and every other ISP hoping you keep them >>>>>> online. So besides demand issues, keeping your network up will be >>>>>> important >>>>>> to a whole lot of people. >>>>>> >>>>>> >>>>>> Rubens >>>>>> >>>>>> >>>>> >>> >>
Re: COVID-19 vs. our Networks
I think under circumstances like this, I could definitely see some of the online based games shutting services down. - Mike Bolitho On Fri, Mar 13, 2020 at 2:41 PM Ahmed Borno wrote: > Its already happening in Italy, and now that schools are shutting down > here as well, its going to get interesting: > > > https://www.bloomberg.com/news/articles/2020-03-12/housebound-italian-kids-strain-network-with-fortnite-marathon > > The ultimate traffic test is coming, looking forward to hearing about it > on this thread. > > Maybe its a good time to start a communication channel between content > providers/gaming companies and ISPs/CDNs. > > > On Fri, Mar 13, 2020 at 11:22 AM Rubens Kuhl wrote: > >> >> >> On Thu, Mar 12, 2020 at 3:46 PM g...@1337.io wrote: >> >>> With talk of there being an involuntary statewide (WA) and then national >>> quarantines (house arrest) for multiple weeks, has anyone put thought into >>> the impacts of this on your networks if/when this comes to fruition? >>> >>> We're already pushing the limits with telecommuters / those that are >>> WFH, but I can only imagine what things will look like with everyone stuck >>> at home for any duration of time. >>> >> >> >> People will turn to you and every other ISP hoping you keep them online. >> So besides demand issues, keeping your network up will be important to a >> whole lot of people. >> >> >> Rubens >> >> >
Re: The curious case of 159.174.0.0/16
> > If you always e-mail j...@telco.com instead of n...@telco.com for your > issues, you may end of in a situation where Jake is gone, on vacation, or > simply moved on to accounting. Plus, Jake hates this. He might pretend to be your friend but he's getting paid to do that. Nothing more annoying than having a customer demand to work with Jake when Jake has 20 other things going on and literally anyone else on the team can help you. Once you're known within the right team, it should be easy to get prompt > responses. Exactly. Show the team that you know what you're talking about and that you're not belligerent and people will be more than happy to work with you. - Mike Bolitho On Wed, Jan 29, 2020 at 9:16 AM Sabri Berisha wrote: > - On Jan 29, 2020, at 12:40 AM, Ronald F. Guilmette > r...@tristatelogic.com wrote: > > Hi, > > > (I have a standing policy of never attempting to converse with > unaccountable > > anonymized role accounts. Based on past experience, this is without > > exception an utter waste of my time.) > > In the real world, this should be the exact opposite. People move teams, > leave companies. If you always e-mail j...@telco.com instead of > n...@telco.com for your issues, you may end of in a situation where Jake > is gone, on vacation, or simply moved on to accounting. Once you're known > within the right team, it should be easy to get prompt responses. > > I'm surprised about the lack of response from FT/DT though. > > Thanks, > > Sabri >
Re: Internet services in Antarctica
One of my buddies was a network engineer at Palmer Station for a winter. Let me reach out to him. - Mike Bolitho On Mon, Jan 20, 2020 at 3:15 AM Ask Bjørn Hansen wrote: > Hi, > > I have a hobby project running DNS service to people looking for NTP > public servers. I noticed that the DNS servers apparently get ~5 thousand > queries per day from IPs that the GeoIP database we use claim are in in > Antarctica. It’s less than 0.0001% of the overall DNS queries, but it made > me curious what it’d take to make the service work better there. > > I imagine the internet service is fragmented between the various stations > with each being best connected to a particular country? Does anyone have > contacts there that I could talk to? I imagine (some of?) the stations > would have a local NTP service as part of their compute facilities. > > > Ask > >
Re: power to the internet
I'm pretty sure political bickering is well beyond the scope of the mailing list. Is anyone moderating this? - Mike Bolitho On Thu, Dec 26, 2019, 7:20 AM Tom Beecher wrote: > Same story again different colors. PG making a mint while people get >> screwed >> > > I'm not quite sure that's an accurate statement. > > In 2000-2001, PG got screwed by Enron's market manipulation. ( Good job > those who pushed so hard for deregulation of public utility services! ) > > PG is currently in bankruptcy proceedings, largely as a result of > liabilities from wildfires in 2017 and 2018. Under California's > application of inverse condemnation, a power utility is responsible for any > damage caused by a wildfire if it was determined that their equipment was > part of the cause. This applies even if the utility was in 100% compliance > with all laws and regulations. > > So you have a terrible combination where housing prices in the state are > driving more and more people to build in wildfire prone areas, climate > change is increasing the frequency of weather conditions favorable to > wildfire ignition, and the utility company that is being held financially > liable for damages while at the same time not being allowed by the PUC to > raise capital for infrastructure changes to reduce the chances of > electrical equipment starting such things. > > The answer is easy. Money. If people want a power grid that is safe and > reliable, then the utility should be given the funds to do it via rates and > appropriate tax revenues. They should not be expected to turn profits like > private enterprise. The power grid is for the benefit of all, not just the > financial benefit of those who have equity stakes. > > This situation is the logical extension of 40+ years of America's only > real product ; financial engineering. > > > On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis wrote: > >> >> >> On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin >> wrote: >> >>> On Wed, 25 Dec 2019 at 19:32, Michael Thomas wrote: >>> >>>> On the dark side, this is probably coming to a lot more states and >>>> countries due to climate change. Australia. Sigh. >>>> >>> >>> Do you have a source for this? It would seem that these power issues >>> are rather unique to California not because of some "climate change" >>> bogeyman, but rather because of a failed public policy at the state level. >>> >>> It would also seem that these issues of rolling blackouts aren't even >>> new to California, either, as, apparently, it's already been the norm >>> during 2000/2001: >>> >> >> >> Having lived through the blackouts that was entirely different. 90% Enron >> manipulating the markets. There was plenty of capacity both in transmission >> and generation, but Enron manipulated prices and apparent supply to make >> money and screwed the whole state over. There was just about 2x the >> generating capacity, no real shortage. >> >> This time it’s PG all alone, but still fallout from back then. Too much >> liability and they’ve not maintained the infrastructure and so they decided >> that to reduce the liability costs it’s cheaper to blackout. Same story >> again different colors. PG making a mint while people get screwed (PG >> was mostly at the getting screwed end in 2000-2001) >> >>> >>> * https://en.wikipedia.org/wiki/California_electricity_crisis >>> >>> C. >>> >> -- >> >> "Genius might be described as a supreme capacity for getting its >> possessors >> into trouble of all kinds." >> -- Samuel Butler >> >
Re: Disney+ Streaming
Again, this has gone beyond off-topic for the NANOG list. Please take the discussion elsewhere. -Mike Bolitho On Thu, Nov 28, 2019, 3:52 PM Michael Thomas wrote: > Back in the old days, we had the ultimate in unbundling: you walked up, > got a ticket, and watched the movie. > > In principle it wouldn't be that hard these days to do something similar > with a tremendous reduction in friction. Basically pay-per-view on > steroids. > > My sense is that it would be tremendous failure though: how would a > consumer know how to value different content? Going to a movie is > comparatively a big commitment with plenty of time to decide if you think > it's worth it. Channel surfing, not so much. So maybe we are doomed to some > sort of bundling. > > The big problem is that I don't want to pay for a month of content to > watch one or two shows. And I definitely don't want to pay a month's worth > of content to three dozen providers of which i may only watch a few of > their programs a couple of times a month. Now if you reduced that to, say, > a day pass I might bite, especially if there was no more friction than the > usual channel surfing. > > Mike > On 11/28/19 2:23 PM, Robert Haylock wrote: > > I agree with Brian, this is not unbundling, it's just removing one layer > of distribution; you no longer need the Cable company to play aggregator to > the content distributors, you now buy from them direct (especially true in > the case of HBO and Disney, except ESPN is not yet included). The next > logical large player to enter the global** direct-to-streamer market would > be NBCUniversal, so I'm sure we will soon be preparing for that one too :) > > Rob > > On Fri, 29 Nov 2019 at 06:47, Brian J. Murrell > wrote: > >> On Thu, 2019-11-28 at 10:50 -0800, Owen DeLong wrote: >> > While I agree about the likely outcome, I will point out that >> > consumers have been >> > begging for unbundling for years. >> >> This is not the "unbundling" that consumers have been begging for. >> Rather I would submit that it's actually quite the opposite and much >> more like the bundling that they have been railing against. >> >> The "unbundling" that consumers have been begging for is minimally, the >> ability to buy a single channel for a fair price and not have to take >> 14 other channels of *garbage* with it at 15x the cost one of those >> channels. I say minimally because I suspect that the really savvy >> consumers would actually rather even pay (again, at a fair price) per >> show or episode. >> >> But that's not what's happening with this fragmentation. This >> fragmentation is like the cable company splitting up that "once price >> for all" bundle and putting the pieces into other bundles, each at the >> same cost as that original "all in one" bundle that the consumers were >> originally happy with and saw as fair value. Of course now to continue >> to getting those pieces of the original bundle that they were happy >> with, consumers are having to buy multiples of these new bundles and >> their costs are driving up sharply accordingly. >> >> > This fragmentation of streaming services _IS_ the direct result of >> > that request. >> >> I would submit that that is completely untrue. Do you really think >> Disney pulled out of Netflix and started their own service because >> consumers wanted Disney to unbundle from Netflix? I would suggest that >> that is completely not why. Rather, Disney was not happy to have just >> a piece of the Netflix pie, and decided, as greedy as they are, that >> they would sell their own pies and take the fully monthly subscription >> price. >> >> > It’s unbundled service, exactly what they have been asking for. >> >> Again. No. Not at all. Not even close. Quite the opposite in fact. >> >> The problem with suggesting that this is unbundling is that the cost of >> Netflix didn't reduce when Disney pulled out and Disney (I would bet, I >> haven't actually looked at it's cost) isn't charging the faction of the >> Netflix cost that would be commensurate with their percentage of the >> entire Netflix library. >> >> So there has been no "unbundling" of any sort. Rather it's been an >> exercise of actually creating a new bundling. And I still predict that >> once the reality of this sets in with consumers, they are going to >> reject it and head back to that low (zero) cost means of obtaining >> their media that they used when they were unhappy with the previous >> generation of bundling. >> >> b. >> >>
Re: Level(3) DNS Spoofing All Domains
> > How many of (my) clients have miss-typed something and sent their data, > unknowingly, to a 3rd party host? (Who’s fault would that be?) Yours? They paid you to set up their network properly and you set it up to resolve to Level 3. So if they "unknowingly sent their data" to a third party then it would be your fault. - Mike Bolitho On Tue, Nov 19, 2019 at 11:18 AM Marshall, Quincy wrote: > *On *Tuesday, November 19, 2019 12:49 PM, Mike Bolitho < > mikeboli...@gmail.com> said… > > “This is was my thought as well. People always get up in arms about how > it's "Public DNS!" but it's really not. It's just well known and used > because it's easy to remember” > > > I am not against their “securing” their hosts. It costs them money to > provide the service. I disagree with what they did - Disable the service or > only allow local or on-net resolution. How many of (my) clients have > miss-typed something and sent their data, unknowingly, to a 3rd party > host? (Who’s fault would that be?) > > > > That said I AM a L(3) customer. These IPs were provided when the circuit > was provisioned for NS resolution. Admittedly, they has indicated, this > morning, that we are using the “wrong” Anycast NS and provided a different > set; which functioned the same as the “Public” ones. > > *Lawrence Q. Marshall* > > > > -- > This email has been scanned for email related threats and delivered safely > by Mimecast. > For more information please visit http://www.mimecast.com > -- >
Re: Level(3) DNS Spoofing All Domains
This is was my thought as well. People always get up in arms about how it's "Public DNS!" but it's really not. It's just well known and used because it's easy to remember. - Mike Bolitho On Tue, Nov 19, 2019 at 9:28 AM Ryan, Spencer wrote: > Are you a CL/L3 customer? Those resolvers have only ever been for > “customers” even though they would resolve for anyone. They started > injecting NXDOMAIN redirects a while ago for non-customers. > > > > > > *From:* NANOG *On Behalf Of *Marshall, Quincy > *Sent:* Monday, November 18, 2019 12:45 PM > *Subject:* Level(3) DNS Spoofing All Domains > > > > This message originated outside of NETSCOUT. Do not click links or open > attachments unless you recognize the sender and know the content is safe. > > This is mostly informational and may have already hit this group. My > google-foo failed me if so. > > > > I discovered that the CenturyLink/Level(3) public DNS (4.2.2.2, etc) are > spoofing all domains. If the hostname begins with a “w” and does not exist > in the authoritative zone these hosts will return two Akamai hosts. > > > > [root@localhost ~]# dig +short w3.dummydomaindoesntexist.gov @4.2.2.2 > > 23.202.231.167 > > 23.217.138.108 > > [root@localhost ~]# dig +short w3.dummydomaindoesntexist.net @4.2.2.2 > > 23.202.231.167 > > 23.217.138.108 > > [root@localhost ~]# dig +short w3.dummydomaindoesntexist.com @4.2.2.2 > > 23.202.231.167 > > 23.217.138.108 > > [root@localhost ~]# dig +short w3.dummydomaindoesntexist.org @4.2.2.2 > > 23.202.231.167 > > 23.217.138.108 > > > > My apologies if this is old news. > > > > *Lawrence Q. Marshall* > > > > > -- > > This email has been scanned for email related threats and delivered safely > by Mimecast. > For more information please visit http://www.mimecast.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mimecast.com=DwMFaQ=Hlvprqonr5LuCN9TN65xNw=VfFQaWKwN0L3efDXtkWoSUKlJtu8LJ9Ke5bevkfX6C0=q6vn3t-QWxYOtFEQ5UhCttLDcerYncizhmA0BXauzSg=0udD7os_Gb1eyxuW47ezLZB2f-gk_Ipxso3m4n80kqg=> > -- >
Re: Disney+ Streaming
This has gone well beyond out of scope of the NANOG list. Discussing who watches what kind of content has nothing to do with networking. Can you guys take the conversation elsewhere? - Mike Bolitho On Tue, Nov 12, 2019 at 4:34 PM Matthew Petach wrote: > > My point was that Disney has a lock on much of the content kids love. > > Netflix/HBO/AmazonPrime, not so much. > > So, the new eyeballs aren't going to be from parents watching different > shows, it'll be from parents watching their adult-ish stuff, while the kids > are happily ensconced with Disney+. > > I called out Game of Thrones and Good Omens as shows that are popular with > adults but that aren't terribly family friendly, so you won't be getting > many 12-and-unders watching them. > > That's where the new eyeballs come from. > > Matt > > > On Tue, Nov 12, 2019, 13:17 Mark Andrews wrote: > >> They can already stream different content to multiple devices >> simultaneously. >> All this does is make some content that wasn’t available previously now >> available. >> >> People can really only watch one thing at a time. Net streaming of the >> last mile >> is unlikely to change much. Just where that content is coming from may >> change. >> >> Mark >> >> > On 13 Nov 2019, at 07:53, Matthew Petach wrote: >> > >> > >> > Different target audiences. >> > >> > Now the parents can be watching "Good Omens" or "Game of Thrones" on >> Netflix while the kids are streaming "The Lion King" on Disney+ streaming. >> Instead of the whole family watching one show together, now we have >> segmentation in the marketplace. >> > >> > End result is more total overall bandwidth consumption. >> > >> > Matt >> > >> > >> > On Tue, Nov 12, 2019, 12:38 Brian J. Murrell >> wrote: >> > On Tue, 2019-11-12 at 15:26 -0500, Valdis Klētnieks wrote: >> > > >> > > I can foresee a lot of families subscribing to Netflix *and* Disney+ >> > > because neither one has all the content the family wants to watch. >> > >> > Absolutely. But the time spent watching Disney would *replace* (not be >> > in addition to, or would it? Would Disney's content result in existing >> > streamers watching more hours of streaming than they did before?) >> > Netflix watching. >> > >> > > Has anybody seen a significant drop in total streaming traffic due to >> > > Netflix >> > > users jumping ship to Amazon/Hulu, or are consumers just biting the >> > > bullet, >> > > coughing up the $$, and streaming more total because across the >> > > services >> > > there's more stuff they want to watch? >> > >> > I actually suspect streaming is going to decline (at least in >> > comparison to where it could have grown to) if this streaming service >> > fragmentation continues. >> > >> > I think people are going to reject the idea that they need to subscribe >> > to a dozen streaming services at $10-$20/mo. each and will be driven >> > back the good old "single source" (piracy) they used to use before 1 >> > (or perhaps 2) streaming services kept them happy enough to abandon >> > piracy. >> > >> > The content providers are going to piss in their bed again due to >> > greed. Again. >> > >> > Cheers, >> > b. >> > >> >> -- >> Mark Andrews, ISC >> 1 Seymour St., Dundas Valley, NSW 2117, Australia >> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org >> >> >>
Re: Russian government’s disconnection test
> > I would imagine that the internet is a whole less resilient today in 2019 > than it was back in the day before the cloud takeover. It's far *more* resilient now than it has ever been. More sub-sea cables. Multiple routes across continents. The very fact that there are AWS/Azure/Google Cloud data centers located around the globe makes anything hosted there even more resilient, not less (and for the most part, I still prefer on prem DC so I'm not even pushing "To the cloud!"). - Mike Bolitho On Fri, Nov 1, 2019 at 5:16 PM Constantine A. Murenin wrote: > Unpopular opinion: other countries should do the same. > > If somehow all the transatlantic (and/or transpacific) cables are offline; > will the whole internet outside of the US stop working, too? > > AWS and all the other providers have DCs all over the world, but would > they still work if they can't contact the mothership, and for how long? > (Has any of this ever been tested?) > > I would imagine that the internet is a whole less resilient today in 2019 > than it was back in the day before the cloud takeover. You often can't > even install OSS without an internet connection anymore. Would Golang stop > working? What else? > > Would you and/or your corporation be able to access your own email? All > these things may seem silly, until you actually encounter the situation > where you're offline, and it's too late to do anything. > > C. > > On Fri, 1 Nov 2019 at 18:04, Scott Weeks wrote: > >> >> >> --- sur...@mauigateway.com wrote: >> From: "Scott Weeks" >> >> Anyone got any technical info on how Russia plans to execute >> a disconnection test of the internet? >> >> >> >> Got crickets, so now I have to respond to my own post on >> what I just found out about it. Is that like talking to >> yourself? :) >> >> >> https://www.npr.org/2019/11/01/775366588/russian-law-takes-effect-that-gives-government-sweeping-power-over-internet >> >> "The "sovereign Internet law," as the government calls it, >> greatly enhances the Kremlin's control over the Web. It was >> passed earlier this year and allows Russia's government to >> cut off the Internet completely or from traffic outside >> Russia "in an emergency," as the BBC reported. But some of >> the applications could be more subtle, like the ability to >> block a single post." >> >> "The equipment would conduct what's known as "deep packet >> inspection," an advanced way to filter network traffic. >> >> "Regardless of what the government intends, some experts >> think it would be technically difficult for Russia to >> actually close its network if it wanted to, because of the >> sheer number of its international connections." >> >> "What I found was that there were hundreds of existing >> Internet exchange points in Russia, some of which have >> hundreds of participants...Many of them are international >> network providers, he says, so "basically it's challenging >> — if not impossible, I think — to completely isolate the >> Russian Internet." >> >> Belson says that the requirement for Internet service >> providers to install tracking software will very likely >> also be challenging in practice. He adds that it will be >> difficult to get hundreds of providers to deploy it and >> hard to coordinate that they're all filtering the same >> content. >> >> scott >> >> >> >> >> >
Re: Viability of GNS3 network simulation for testing features/configurations.
EVE-NG is also really good. Just an FYI, GNS3 went through a major refresh about 18 months ago or so and it's so much better now. Either way, you can't go wrong with GNS3 or EVE-NG. - Mike Bolitho On Wed, Oct 16, 2019 at 11:18 AM Aaron Gould wrote: > Oh, forgot the links… > > > > http://www.eve-ng.net/ > > > > http://www.eve-ng.net/documentation/howto-s > > > > > > > > > > *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Aaron Gould > *Sent:* Wednesday, October 16, 2019 1:14 PM > *To:* 'Mike Bolitho'; 'Tom Beecher'; 'Ryland Kremeier' > *Cc:* nanog@nanog.org > *Subject:* RE: Viability of GNS3 network simulation for testing > features/configurations. > > > > I’ve used GNS3 some years ago for a lot of simulation and testing. But, > I’m blown away at how much more I like EVE-NG (emulated virtual environment > next-gen) > > > > I use the community free version… lots of vendor OS support… of which, > I’ve actually work with the following…. > > -XRv > > -IOS virtual > > -vMX > > -vSRX > > -vQFX > > > > …check your in-box for a screen shot of my current environment. > > > > -Aaron > > > > *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Mike Bolitho > *Sent:* Wednesday, October 16, 2019 12:02 PM > *To:* Tom Beecher > *Cc:* > *Subject:* Re: Viability of GNS3 network simulation for testing > features/configurations. > > > > Totally agree with Tom here. It's going to work really well for most > things. But if you're testing code for bugs you NEED to do it on the same > hardware you have in your environment in an actual lab. > > > - Mike Bolitho > > > > > > On Wed, Oct 16, 2019 at 9:56 AM Tom Beecher wrote: > > GNS3 can do a heck of a lot, and the price is definitely right. > > > > I have used it extensively for initial fleshing out of designs or ideas, > protocol nerding, automation interaction testing, etc. There certainly > other tools out there, but being able to visually draw a topology out, > connect the dots, and have an environment to test in about 10 minutes is > very nice. There is an API you can hook into to do some of that for you if > you are so inclined, but that would depend on your use case and resources. > For how I've used it, never been required. > > > > Some of the VMs from vendors can be pretty CPU and/or RAM intensive, so > I've had the best experience running them all on a dedicated server, not > locally. Again, use case dependent. For code testing I would always run the > test set on hardware as well for likely obvious reasons. > > > > If you really get into the weeds with it you can do quite a lot. > > > > On Wed, Oct 16, 2019 at 11:52 AM Ryland Kremeier < > rkreme...@barryelectric.com> wrote: > > Hello, > > > > I’m currently in the process of setting up a near identical network to our > own in GNS3 for testing purposes. Has anyone here tried this before to any > success? We need to buy the Cisco IOSv image to continue with the sim so I > figured I would inquire here first before diving in. > > > > All info is appreciated, > > -- > > Ryland Kremeier > >
Re: Viability of GNS3 network simulation for testing features/configurations.
Totally agree with Tom here. It's going to work really well for most things. But if you're testing code for bugs you NEED to do it on the same hardware you have in your environment in an actual lab. - Mike Bolitho On Wed, Oct 16, 2019 at 9:56 AM Tom Beecher wrote: > GNS3 can do a heck of a lot, and the price is definitely right. > > I have used it extensively for initial fleshing out of designs or ideas, > protocol nerding, automation interaction testing, etc. There certainly > other tools out there, but being able to visually draw a topology out, > connect the dots, and have an environment to test in about 10 minutes is > very nice. There is an API you can hook into to do some of that for you if > you are so inclined, but that would depend on your use case and resources. > For how I've used it, never been required. > > Some of the VMs from vendors can be pretty CPU and/or RAM intensive, so > I've had the best experience running them all on a dedicated server, not > locally. Again, use case dependent. For code testing I would always run the > test set on hardware as well for likely obvious reasons. > > If you really get into the weeds with it you can do quite a lot. > > On Wed, Oct 16, 2019 at 11:52 AM Ryland Kremeier < > rkreme...@barryelectric.com> wrote: > >> Hello, >> >> >> >> I’m currently in the process of setting up a near identical network to >> our own in GNS3 for testing purposes. Has anyone here tried this before to >> any success? We need to buy the Cisco IOSv image to continue with the sim >> so I figured I would inquire here first before diving in. >> >> >> >> All info is appreciated, >> >> -- >> >> Ryland Kremeier >> >
Re: Word Usage (was Re: Elad Cohen)
Everytime you guys change the subject on this pointless thread, you break my filter. Admins can you please take action on this? Enough is enough. -Mike Bolitho On Fri, Sep 20, 2019, 3:21 AM James Downs via NANOG wrote: > For the record: > > Slander is false *spoken* statements. > Libel is false *written* statements. > > HTH, HAND. >
Re: What can ISPs do better? Removing racism out of internet
"I am sure there are many sites like this out there, but could network operators do anything to make these sites “not so easy” to be found, reached, and used to end innocent lives?" As network operators? We shouldn't do anything. The onus falls on the hosting companies. I do not want to go down the slippery slope of deciding what traffic should or should not be allowed on the internet. That process involves traffic sniffing and possibly attempting to break encryption to see what's flowing through the pipes. I'm adamantly against that. If I'm building and maintaining highways, I'm not opening up every single truck to make sure there's nobody being smuggled inside. The trucking company can police what cargo is in their trailers. On Sun, Aug 4, 2019, 8:42 PM Mehmet Akcin wrote: > Ok, two mass shootings, touchy topic, lots of emotions this weekend. Going > straight to the point. > > Most of us who operate internet services believe in not being the > moderator of internet. We provide a service and that’s it. Obviously there > are some established laws around protecting copyrights, and other things > which force us to legally take action and turn things down when reported. > > What can we do better as network operators about hate sites like 8Chan? > > I applaud cloudflare’s (perhaps slightly late) decision on kicking 8chan > off its platform today after El Paso attack. > https://blog.cloudflare.com/terminating-service-for-8chan/ > > I am sure there are many sites like this out there, but could network > operators do anything to make these sites “not so easy” to be found, > reached, and used to end innocent lives? > > Mehmet > > > -- > Mehmet > +1-424-298-1903 >
Re: AS3549 NOC contacts? Another BGP hijack
NOC is 877-453-8353. That will get you the legacy Global Crossing (Level 3) teams. On Fri, Jul 19, 2019, 2:12 PM Dmitry A.Deineka wrote: > Greetings, > > Unfortunately, n...@gblx.net is not accepting emails anymore. Someone from > AS3549 announced one of our network (more specific route) 46.28.67.0/24. > > It's not major impact but it's like that at least RIPE whois has outdated > contact information about responsive persons. > > Can someone kindly share contact email of AS3549 (Centurylink?) NOC or > other direct contacts? > > Regards, > Dmitry > > -- > Dmitry A.Deineka > ITLDC >
Re: Fiber providers - Englewood / Centennial Colorado
Denver is a tough market for diversity's sake. Just about everyone that was there was gobbled up by what is now CenturyLink. -Mike Bolitho On Wed, Jul 17, 2019, 4:12 PM JASON BOTHE via NANOG wrote: > Hi all > > Just curious if you know of any fiber providers other than CL or Zayo in > the Englewood/Centennial area. Having a really tough time finding routes > that avoid the Solarium at Quebec / E Orchard as well as 910 15th St. Seems > there are so many single points of failure and collapsed routes that all > lead to these two locations to get diverse long haul. > > Many thanks. > > J~ >
Re: Reddit down
Working fine in Phoenix on Cox and CenturyLink. -Mike Bolitho On Thu, Jul 11, 2019, 7:16 AM Mark Tinka wrote: > Good for me at my house right now - Johannesburg. > > Fastly are delivering... > > MacBook-Pro-7:~ tinka$ traceroute -I www.reddit.com > traceroute to reddit.map.fastly.net (151.101.173.140), 64 hops max, 72 > byte packets > 1 10.0.32.1 (10.0.32.1) 6.238 ms 5.571 ms 5.425 ms > 2 ae-1-4.pr-01-jnb.za.seacomnet.com (105.16.165.253) 6.902 ms 3.966 > ms 5.166 ms > 3 ae-4-0.pp-01-jnb.za.seacomnet.com (105.16.29.8) 5.466 ms 6.375 ms > 5.811 ms > 4 fastly.ixp.joburg (196.60.8.13) 4.556 ms 5.220 ms 5.085 ms > 5 151.101.173.140 (151.101.173.140) 4.846 ms 4.562 ms 5.395 ms > MacBook-Pro-7:~ tinka$ > > Mark. > > On 11/Jul/19 15:57, Robert Webb wrote: > > I am getting, "Our CDN was unable to reach our servers". > > Looking at down detector, there are lots of folks reporting issues and the > Reddit status page has a high error rate... > > On Thu, Jul 11, 2019 at 9:55 AM Michael Morrison < > michael.d.morri...@me.com> wrote: > >> Reddit main page loads for me in Columbus Ohio on a Level 3 loop. >> >> > On Jul 11, 2019, at 9:51 AM, Robert Webb wrote: >> > >> > Are we having yet another CDN meltdown or is it isolated to just >> reddit? >> > >
Re: Looking for a knowledgeable Level3 and GTT off-list contact
They're not going to do anything unless there is a warrant, especially considering that it's not a customer of theirs. Of course "illegal" traffic is going to flow over Tier I equipment, public internet is public internet. - Mike Bolitho On Wed, Jul 10, 2019 at 6:30 AM ecdhe via NANOG wrote: > Some of your resources are being used for organized crime and attacks (but > not originating from your network). > > I would appreciate an off-list contact if anyone knows of one. Thank you. >
Re: CenturyLink/Level3 feedback
Just out of curiosity, what network are they bringing you up on? - Mike Bolitho On Fri, Jul 5, 2019 at 12:11 PM Stephen Frost wrote: > Greetings, > > I have to admit that I was hoping to be able to report to this list that > CL was able to spin up a new 1G in fairly short order (after all, this > is what they assured me of when discussing it with them...) but it's now > been over a month, with them telling me it'll be another couple weeks > because they need to send a tech out (the wiring and all of the > equipment has been ready to go, though that also took longer than it > should have imv...). > > And this in an already lit building in northern Virginia, not some back > of the woods location, small town, or something going across an ocean. > > Pretty disappointing. > > Thanks, > > * Mike Hammett (na...@ics-il.net) wrote: > > Anything more than a week for things not requiring last mile > construction is ridiculous. > > > > > > > > > > - > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > - Original Message - > > > > From: "JASON BOTHE via NANOG" > > To: "Mehmet Akcin" > > Cc: "nanog" > > Sent: Wednesday, June 5, 2019 9:56:14 AM > > Subject: Re: CenturyLink/Level3 feedback > > > > It’s taking over a year to get waves turned up in EU. I’m currently > willing to wager on what comes up first, them or amazon peering (that’s > taking just as long). After the merger, we have seen Level3 slide into the > CL abyss becoming a pain to deal with. Pricing and ordering has been > outsourced we’ve been told and decisions are no longer at a regional level. > Frustrating at best. > > > > > On Jun 4, 2019, at 09:30, Mehmet Akcin wrote: > > > > > > hi there, > > > > > > Just a general high-level question about Centurylink/Level3 > post-merger, how is your overall experience with CenturyLink? if you could > be sitting with the CEO of the company what is one thing you would ask him > to fix? > > > > > > please keep it high level and general. i intend to pass these to him > and his team in an upcoming meeting. > > > > > > Mehmet > > > > >
Re: CenturyLink/Level 3 combined AS
As usual, that depends. Gotta give us a lot more information than that. -Mike Bolitho On Sat, Jun 8, 2019, 6:10 AM Darin Steffl wrote: > Ok just so simplify things. > > Is Cogent or CenturyLink/L3 better for transit? > > On Fri, Jun 7, 2019, 3:00 PM Brielle Bruns wrote: > >> On 6/7/2019 11:03 AM, Romeo Czumbil wrote: >> > All new CL Internet get's provisioned on AS3356 >> > You would need a strong case for them to put you on AS209 >> >> >> Got provisioned last year on AS209 when they turned up my ent Fiber with >> BGP. >> >> Could depend heavily on what services and where. >> >> -- >> Brielle Bruns >> The Summit Open Source Development Group >> http://www.sosdg.org/ http://www.ahbl.org >> >
Re: FCC Hurricane Michael after-action report
In Florida, especially the panhandle, it's not possible to bury it. The water table is way too high. On Mon, May 13, 2019, 9:47 PM wrote: > This webinar may be of some interest to those in this group: > > > https://www.fcc.gov/small-rural-communications-provider-network-resiliency-webinar > > > > Here’s some additional color commentary on the FCC’s concerns: > > > https://urgentcomm.com/2019/05/10/backhaul-problems-disjointed-recovery-efforts-key-causes-of-unacceptable-extended-wireless-outage-after-hurricane-michael-fcc-report-says/ > > "“Uniti Fiber (Uniti) provides backhaul services to Verizon Wireless in > Bay and Gulf Counties. Uniti indicates it experienced at least 33 separate > fiber cuts during the recovery effort. These fiber cuts included damage to > sections that already had been repaired. Commenters attributed fiber cuts > to debris-removal crews, power-company restorations, and returning > homeowners clearing their property.” > > One of my takeaways from that article was that burying fiber underground > could likely have avoided many/most of these fiber cuts, though I’m not > familiar enough with the terrain to know how feasible that is. > > Frank > > > > *From:* NANOG *On Behalf Of *Mel Beckman > *Sent:* Saturday, May 11, 2019 9:52 AM > *To:* Mike Bolitho > *Cc:* nanog@nanog.org > *Subject:* Re: FCC Hurricane Michael after-action report > > > > This is what I tell outage complainers during natural disasters, such as > the fires in California that recently took out a lot of power and > communications: > > > > “Stop whining about how long it is taking to repair your Internet, your > cell phone service, or your cable TV. You didn’t pay anything extra to > recover from natural disasters, and none of us in the field are getting > paid anything extra to restore your services. > > > > No, we don’t know how long it will take. It takes what it takes. That you > don’t get instant gratification doesn’t make us incompetent. It makes you > ungrateful. > > > > It’s a natural disaster. These are not scheduled. Your outage is nobody’s > fault. We don’t have a duty to mitigate all conceivable failures. > > > > It takes time to repair. We’re not cheating you, or loafing around. We > don’t owe you any special attention because of your status or reputation. > > > > So quit whining and be thankful you’re alive, and hopefully you haven’t > lost too much. Maybe pitch in and help those who have.“ > > > > I also send this to ignorant journalists and grandstanding politicians. > > -mel via cell > > > On May 11, 2019, at 4:29 AM, Mike Bolitho wrote: > > Trying not to get political, here goes... > > > > Something important to keep in mind: The current administration has been > getting slammed for their lack of response in the aftermath of Michael > since the hurricane hit. A lot of that criticism revolves around > communications infrastructure and FEMA's lack of assistance. The current > administration has, time and time again, used federal agencies > (specifically their presidential appointees) to defend the administration's > actions or inactions. I have read the full report and it is more or less a > thinly veiled hit piece. I'm not going to link them here (they are easy > enough to find via Google) but there are several very good articles written > by reputable tech journalists that go into greater detail responding to the > report. Worth checking out. > > > > I say all of that because most of us like to hate on telecom companies > (many times rightly so) but I don't think they are entirely to blame here. > There's nothing Verizon or AT can do if their backhaul is cut by a tree > or some third party clean up crew. The report is a gross oversimplification > of how telecommunication infrastructure works. I think anyone here that has > ever worked a storm like this can attest to the complexity and difficulty > you run into during recovery. Hanlon's Razor and all but this is the FCC > and I would hope they would know better. > > > > Speaking specifically to point 51, it's impossible to coordinate between > the thousands of crews working to clean things up and repair physical > infrastructure after a massive storm like this. Many of the people doing > physical cleanup are volunteers that are fully independent of any governing > body or company. It is not a telco's responsibility to know when and where > those crews are working. Further, even if those crews we're calling in and > letting each telco know exactly where they were, what does that provide > other than an impossibly large and fluid dataset to parse for any > meaningful information. > > > > - Mike
Re: FCC Hurricane Michael after-action report
Trying not to get political, here goes... Something important to keep in mind: The current administration has been getting slammed for their lack of response in the aftermath of Michael since the hurricane hit. A lot of that criticism revolves around communications infrastructure and FEMA's lack of assistance. The current administration has, time and time again, used federal agencies (specifically their presidential appointees) to defend the administration's actions or inactions. I have read the full report and it is more or less a thinly veiled hit piece. I'm not going to link them here (they are easy enough to find via Google) but there are several very good articles written by reputable tech journalists that go into greater detail responding to the report. Worth checking out. I say all of that because most of us like to hate on telecom companies (many times rightly so) but I don't think they are entirely to blame here. There's nothing Verizon or AT can do if their backhaul is cut by a tree or some third party clean up crew. The report is a gross oversimplification of how telecommunication infrastructure works. I think anyone here that has ever worked a storm like this can attest to the complexity and difficulty you run into during recovery. Hanlon's Razor and all but this is the FCC and I would hope they would know better. Speaking specifically to point 51, it's impossible to coordinate between the thousands of crews working to clean things up and repair physical infrastructure after a massive storm like this. Many of the people doing physical cleanup are volunteers that are fully independent of any governing body or company. It is not a telco's responsibility to know when and where those crews are working. Further, even if those crews we're calling in and letting each telco know exactly where they were, what does that provide other than an impossibly large and fluid dataset to parse for any meaningful information. - Mike Bolitho On Thu, May 9, 2019, 4:43 PM Sean Donelan wrote: > > The FCC has released its report and analysis of Hurricane Michael impact > on communications: preparation, effect and recovery. > > > > https://www.fcc.gov/document/fcc-releases-report-communication-impacts-hurricane-michael-0 > > Conclusions and Recommendations > > 51. Backhaul outages loomed large as an impediment to communications > recovery. Uncoordinated post-storm recovery efforts between and among > communications, utility, and debris removal teams created unnecessary > delays to a speedy return to service. Customers who had communications > service restored – only to lose it again almost immediately because of a > fiber cut – provide a clear example of how better cross-sector > coordination could have improved the restoration process. >
Re: Comcast storing WiFi passwords in cleartext?
> > "than the relatively low risk of a database compromise leading to a > miscreant getting ahold of their wireless password and using their access > point as free wifi." > And this is the thing, not only does someone have to 'hack' the database, they also need to drive up to your house and sit in your driveway to get free Internet. Of all the things to worry about, this is way down on my list. >
Re: residential/smb internet access in 2019 - help?
> > Agreedthis is why monopolies are bad and municipal fiber is good. > It's not like municipal fiber has some magic spell to make last mile affordable though. On OP's instance he would run into the same issue and would be paying that five figure amount to bring FTTP. Municipal fiber is only good if you happen to live where a municipality has already buried conduit. I'm not saying we should support monopolistic practices, but "municipal fiber everywhere!" isn't necessarily the answer either. >
