Re: CloudFlare issues?

On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher wrote: > > On 25/06/2019 03:03, Tom Beecher wrote: > > Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do > > not work on 701. My comments are my own opinions only. > > > > Respectfully, I believe Cloudflare’s public comments

Re: CloudFlare issues?

On 25/06/2019 03:03, Tom Beecher wrote: Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701.  My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter

Re: Cost effective time servers

It's about minimizing the impact of the attack vector. And you shouldn't implicitly trust the second alignment either. In a potential spoofing attack, if you trust the GPS for all of the data exclusively, then someone who can spoof your GPS (not as hard/expensive as one would think) can fully

Re: Cost effective time servers

Once upon a time, Forrest Christian (List Account) said: > I would submit that the proper use of a GPS receiver is for alignment > of the start of the second to a more precise value than can be > distributed across an asymmetric network like the Internet. The > actual 'time label' for that

Re: Cost effective time servers

I would submit that the proper use of a GPS receiver is for alignment of the start of the second to a more precise value than can be distributed across an asymmetric network like the Internet. The actual 'time label' for that second doesn't necessarily need to come from GPS at all. For security

Re: Cost effective time servers

Once upon a time, Jay Hennigan said: > The data from GPS includes the offset value from UTC for leap-second > correction. This should be easily included in your time calculation. Not only that, but at least some GPS receivers/protocols notify of pending leap seconds, so software can properly

Re: Cost effective time servers

On 6/21/19 07:57, Quan Zhou wrote: Yep, went through the same route until I figured out that GPS time is a bit ahead of UTC. The data from GPS includes the offset value from UTC for leap-second correction. This should be easily included in your time calculation. It's presently 18 seconds.

Re: CloudFlare issues?

> On Jun 24, 2019, at 9:39 PM, Ross Tajvar wrote: > > > On Mon, Jun 24, 2019 at 9:01 PM Jared Mauch wrote: > > > > > On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > > > > > Maybe I'm in the minority here, but I have higher standards for a T1 than > > > any of the other players

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 9:01 PM Jared Mauch wrote: > > > On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > > > Maybe I'm in the minority here, but I have higher standards for a T1 than any of the other players involved. Clearly several entities failed to do what they should have done, but

Re: CloudFlare issues?

> On Jun 24, 2019, at 8:50 PM, Ross Tajvar wrote: > > Maybe I'm in the minority here, but I have higher standards for a T1 than any > of the other players involved. Clearly several entities failed to do what > they should have done, but Verizon is not a small or inexperienced operation. >

Re: CloudFlare issues?

> On Jun 24, 2019, at 8:03 PM, Tom Beecher wrote: > > Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work > on 701. My comments are my own opinions only. > > Respectfully, I believe Cloudflare’s public comments today have been a real > disservice. This blog

Re: CloudFlare issues?

Maybe I'm in the minority here, but I have higher standards for a T1 than any of the other players involved. Clearly several entities failed to do what they should have done, but Verizon is not a small or inexperienced operation. Taking 8+ hours to respond to a critical operational problem is what

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 08:03:26PM -0400, Tom Beecher wrote: > > You are 100% right that 701 should have had some sort of protection > mechanism in place to prevent this. But do we know they didn???t? Do we know > it was there and just setup wrong? Did another change at another time break > what

Re: CloudFlare issues?

--- beec...@beecher.cc wrote: From: Tom Beecher :: Shouldn’t we be working on facts? Nah, this is NANOG... >;-) :: But this industry is one big ass glass house. What’s that :: thing about stones again? We all have broken windows? :) scott

Re: CloudFlare issues?

Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter today, took every opportunity to say “DAMN

Re: CloudFlare issues?

FYI for the group -- we just published this: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ _ *Justin Paine* Director of Trust & Safety PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D 101 Townsend St., San

Re: CloudFlare issues?

On 24/Jun/19 18:09, Pavel Lunin wrote: > > Hehe, I haven't seen this text before. Can't agree more. > > Get your tie back on Job, nobody listened again. > > More seriously, I see no difference between prefix hijacking and the > so called bgp optimisation based on completely fake announces on >

Re: CloudFlare issues?

On 2019-06-24 20:16, Mark Tinka wrote: > > > On 24/Jun/19 16:11, Job Snijders wrote: > >> >> - deploy RPKI based BGP Origin validation (with invalid == reject) >> - apply maximum prefix limits on all EBGP sessions >> - ask your router vendor to comply with RFC 8212 ('default deny')

Re: Cost effective time servers

Patrick : > On 2019-06-20 20:18, Jay Hennigan wrote: > > If you want to go really cheap and don't value your time, but do value > > knowing the correct time, a GPS receiver with a USB interface and a > > Raspberry Pi would do the trick. > >

Re: CloudFlare issues?

On 24/Jun/19 16:11, Job Snijders wrote: > > - deploy RPKI based BGP Origin validation (with invalid == reject) > - apply maximum prefix limits on all EBGP sessions > - ask your router vendor to comply with RFC 8212 ('default deny') > - turn off your 'BGP optimizers' I cannot

Re: Cost effective time servers

On 21 Jun 2019, at 10:57, Quan Zhou wrote: > Yep, went through the same route until I figured out that GPS time is a bit > ahead of UTC. The clocks on the GPS satellites are set to GPST which I think (I'm not a time geek so this is going to make someone cringe) is UTC without leap seconds or

Re: Cellular backup connections

We deploy routers with Verizon LTE failover - for full functionality, make sure your MTU is 1428 or less, per their specifications. Here's an example doc from Spirent that talks about it. https://support.spirent.com/SC_KnowledgeView?Id=FAQ14556 Alex On Mon, Jun 24, 2019, 7:51 AM Dovid Bender

Re: CloudFlare issues?

From https://www.cloudflarestatus.com/​: Identified - We have identified a possible route leak impacting some Cloudflare IP ranges and are working with the network involved to resolve this. Jun 24, 11:36 UTC Seeing issues in Australia too for some sites that are routing through Cloudflare.

Re: Cost effective time servers

On Thu, 20 Jun 2019 10:39:41 -0400, David Bass wrote: >What are folks using these days for smaller organizations, that need to >dole out time from an internal source? If "internal" means a local NTP server independent of external network resources, the other responses are apposite. If

Re: CloudFlare issues?

>I'd like to point everyone to an op-ed I wrote on the topic of "BGP optimizers": >https://seclists.org/nanog/2017/Aug/318 Hehe, I haven't seen this text before. Can't agree more. Get your tie back on Job, nobody listened again. More seriously, I see no difference between prefix hijacking and

Re: Cost effective time servers

Yep, went through the same route until I figured out that GPS time is a bit ahead of UTC. We simply use a windows NTP server for internal use at work, and I won't recommend doing so, because it went off the rails once for a while despite of having several upstream servers pointed to. Also

Re: Cost effective time servers

On 2019-06-20 20:18, Jay Hennigan wrote: > If you want to go really cheap and don't value your time, but do value > knowing the correct time, a GPS receiver with a USB interface and a > Raspberry Pi would do the trick. https://www.ntpsec.org/white-papers/stratum-1-microserver-howto/ RPi + GPS

AWS 40/100G wholesale Express-Route ?

Hello everyone, I was wondering, is there any way to get more than a 10G port for a PNI with AWS customers ? Right now I'm looking at 4 ridiculously expensive X-Cos (on two locations, so that makes 8) to establish a redundant 40Gbps backhaul, where I have 40/100G ports available. How could we

[NANOG-announce] NANOG’s new website has launched!

Welcome to our all-new web presence, with a community-first approach. Since our original charter was formed, we’ve seen many changes as an organization, and grown substantially: branching off from Merit to become an independent, self-governing organization; growing the NSFNET’s Regional-Techs

Re: Russian Anal Probing + Malware

I chuckle the most at the original twitter post from Greynoise : "We have revoked the benign tag for OpenPortStats[.]com" Did anyone actually think such a thing would be legitimate to start with? :) On Mon, Jun 24, 2019 at 12:26 AM Hank Nussbacher wrote: > On 24/06/2019 00:23, Randy Bush

Re: CloudFlare issues?

24.06.19 19:04, Matthew Walster пише: On Mon, 24 Jun 2019, 16:28 Max Tulyev, > wrote: 1. Why Cloudflare did not immediately announced all their address space by /24s? This can put the service up instantly for almost all places Probably RPKI and that being

Re: Verizon Routing issue

> On Jun 24, 2019, at 11:12 AM, Max Tulyev wrote: > > 24.06.19 17:44, Jared Mauch пише: >>> 1. Why Cloudflare did not immediately announced all their address space by >>> /24s? This can put the service up instantly for almost all places. >> They may not want to pollute the global routing

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 10:41 AM Filip Hruska wrote: > > Verizon is the one who should've noticed something was amiss and dropped > their customer's BGP session. > They also should have had filters and prefix count limits in place, > which would have prevented this whole disaster. > oddly VZ

Re: Verizon Routing issue

24.06.19 17:44, Jared Mauch пише: 1. Why Cloudflare did not immediately announced all their address space by /24s? This can put the service up instantly for almost all places. They may not want to pollute the global routing table with these entries. It has a cost for everyone. If we all did

Re: Verizon Routing issue

> On Jun 24, 2019, at 11:00 AM, ML wrote: > > > On 6/24/2019 10:44 AM, Jared Mauch wrote: >> It was impacting to many networks. You should filter your transits to >> prevent impact from these more specifics. >> >> - Jared >> >> https://twitter.com/jaredmauch/status/1143163212822720513 >>

Re: Verizon Routing issue

On 6/24/2019 10:44 AM, Jared Mauch wrote: It was impacting to many networks. You should filter your transits to prevent impact from these more specifics. - Jared https://twitter.com/jaredmauch/status/1143163212822720513 https://twitter.com/JobSnijders/status/1143163271693963266

Verizon Routing issue

(Updating subject line to be accurate) > On Jun 24, 2019, at 10:28 AM, Max Tulyev wrote: > > Hi All, > > here in Ukraine we got an impact as well! > > Have two questions: > > 1. Why Cloudflare did not immediately announced all their address space by > /24s? This can put the service up

Re: CloudFlare issues?

Verizon is the one who should've noticed something was amiss and dropped their customer's BGP session. They also should have had filters and prefix count limits in place, which would have prevented this whole disaster. As to why any of that didn't happen, who actually knows. Regards, Filip

Re: CloudFlare issues?

Hi All, here in Ukraine we got an impact as well! Have two questions: 1. Why Cloudflare did not immediately announced all their address space by /24s? This can put the service up instantly for almost all places. 2. Why almost all carriers did not filter the leak on their side, but waited

Re: CloudFlare issues?

This is what looked happened: There was a large scale BGP 'leak' incident causing about 20k prefixes for 2400 network (ASNs) to be rerouted through AS396531 (a steel plant) and then on to its transit provider: Verizon (AS701) Start time: 10:34:21 (UTC) End time: 12:37 (UTC) All ASpaths had the

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 08:18:27AM -0400, Tom Paseka via NANOG wrote: > a Verizon downstream BGP customer is leaking the full table, and some more > specific from us and many other providers. It appears that one of the implicated ASNs, AS 33154 "DQE Communications LLC" is listed as customer on

Re: Cellular backup connections

I ran into this problem and Verizon told me that they filter ports 22 and 23 to help stem the tide of IoT attacks on their networks by cellular-connected phone and alarm systems. They said their operational model assumes that all traffic will be encrypted via either SSLVPN or IPSec. I’m using

Re: CloudFlare issues?

This is my final update, I’m going back to bed, wake me up when the internet is working again. https://news.ycombinator.com/item?id=20262316 —— 1230 UTC update We are working with networks around the world and are observing network routes for Google and AWS being leaked at well. On Mon, Jun

Re: CloudFlare issues?

*1204 UTC update* This leak is wider spread that just Cloudflare. *1208 UTC update* Amazon Web Services now reporting external networking problem On Mon, Jun 24, 2019 at 05:18 Tom Paseka wrote: > a Verizon downstream BGP customer is leaking the full table, and some more > specific from us and

Re: CloudFlare issues?

a Verizon downstream BGP customer is leaking the full table, and some more specific from us and many other providers. On Mon, Jun 24, 2019 at 7:56 AM Robbie Trencheny wrote: > *1147 UTC update* Staring at internal graphs looks like global traffic is > now at 97% of expected so impact lessening.

Re: CloudFlare issues?

*1147 UTC update* Staring at internal graphs looks like global traffic is now at 97% of expected so impact lessening. On Mon, Jun 24, 2019 at 04:51 Dovid Bender wrote: > We are seeing issues as well getting to HE. The traffic is going via Alter. > > > > On Mon, Jun 24, 2019 at 7:48 AM Robbie

Re: CloudFlare issues?

We are seeing issues as well getting to HE. The traffic is going via Alter. On Mon, Jun 24, 2019 at 7:48 AM Robbie Trencheny wrote: > From John Graham-Cumming, CTO of Cloudflare, on Hacker News right now: > > This appears to be a routing problem with Level3. All our systems are > running

Re: Cellular backup connections

I am getting the same for SSH and https traffic. It's strange. Where the response is something small like: Moved to this https://63.XX.XX.XX:443/auth.asp;>location. It works But when I try to load pages that are any bigger it fails. Like I said before I assume it's either an issue with the MTU

Re: CloudFlare issues?

>From John Graham-Cumming, CTO of Cloudflare, on Hacker News right now: This appears to be a routing problem with Level3. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're

Re: CloudFlare issues?

On Mon, Jun 24, 2019 at 02:03:47PM +0300, Antonios Chariton wrote: > Yes, traffic from Greek networks is routed through NYC (alter.net > ), and previously it had a 60% packet loss. Now it???s > still via NYC, but no packet loss. This happens in GR-IX Athens, not GR-IX >

Re: Cellular backup connections

Could be wrong on this but direct SSH on the LTE side may possibly be not allowed(filtered) and might just be something you could discuss in a ticket with Verizon. -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic

Re: CloudFlare issues?

Yes, traffic from Greek networks is routed through NYC (alter.net ), and previously it had a 60% packet loss. Now it’s still via NYC, but no packet loss. This happens in GR-IX Athens, not GR-IX Thessaloniki, but the problem definitely exists. Antonis > On 24 Jun 2019, at

CloudFlare issues?

Hello are there any issues with CloudFlare services now? Dmitry Sherman dmi...@interhost.net Interhost Networks Ltd Web: http://www.interhost.co.il fb: https://www.facebook.com/InterhostIL Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157

Re: Cellular backup connections

All, I finally got around to putting in a Verizon LTE connection and the ping times are pretty good. There is the occasional issue however for the most part ping times are < 50 ms. I have another strange issue though. When I try to ssh or connect via the endpoints web interface it fails. If I