Re: Cellular backup connections
On Fri, 28 Dec 2018, Dovid Bender wrote: >I finally got around to setting up a cellular backup device in our new POP. >When SSH'ing in remotely the connection seems rather slow. Perhaps using MOSH can help make the interactive CLI session less annoying. >Verizon they charge $500.00 just to get a public IP and I want to avoid >that if possible. You might look into have it call out / maintain a connection back to your infrastructure. /mark
Re: CenturyLink
On 2018-12-29 7:51 a.m., Matthew Huff wrote: We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement. On one occasion, due to bad firmware or a configuration issue, I have seen GPS stratum 1 diverge from NTP. It was somewhat eye brow raising to the company. My NTP monitored servers were shown to be diverging their GPS/NTP, but after looking at twice or thrice, it was the other way around.
Re: CenturyLink
On Fri, Dec 28, 2018 at 9:24 PM Yang Yu wrote: > On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer > wrote: > > Is this problem also responsible for the 911 outage? If so, the > > post-mortem analysis is not useful only for CenturyLink customers but > > for everyone on the west coast. > > Looks like most time.nist.gov servers (3 x NIST sites on AS49) are > single homed on CenturyLink, anyone noticed NTP issues yesterday? > > https://tf.nist.gov/tf-cgi/servers.cgi NIST could take a hint from the ntp.br project: pool.ntp.br has address 200.186.125.195 => CenturyLink pool.ntp.br has address 200.20.186.76 => RNP (Local academic network) pool.ntp.br has address 200.160.7.193 => NIC.br (Local ccTLD, IX, NTP and other services) pool.ntp.br has address 200.160.7.186 => NIC.br pool.ntp.br has address 200.160.7.209 => NIC.br pool.ntp.br has address 200.160.0.8 => NIC.br (distributed in different buildings, rack clusters, NTP hierarchy) pool.ntp.br has IPv6 address 2001:12ff::8 pool.ntp.br has IPv6 address 2001:12ff:0:7::186 pool.ntp.br has IPv6 address 2001:12ff:0:7::193 (unfortunately there is lack of IPv6 diversity at this point) a.st1.ntp.br 200.160.7.186 2001:12ff:0:7::186 => NIC.br b.st1.ntp.br 201.49.148.135 => STF (Local Supreme Court) c.st1.ntp.br 200.186.125.195 => CenturyLink d.st1.ntp.br 200.20.186.76 => RNP (Rio) a.ntp.br 200.160.0.8 e 2001:12ff::8 => NIC.br b.ntp.br 200.189.40.8 => Globenet Fortaleza (Cable Landing Station) c.ntp.br 200.192.232.8 => RNP (Brasilia) gps.ntp.br 200.160.7.193 e 2001:12ff:0:7::193 => NIC.br Perhaps what happened in NIST was a bad governmental RFP not requiring diversity ? Rubens
Re: CenturyLink
On 12/29/18 6:51 AM, Matthew Huff wrote: > We have two stratum-1 servers synced with GPS and a PTP feed from a provider > that also provides PTP to market data systems, but we still have to monitor > drift between system time and NIST time. Don't ask for the logic behind it, > it's a regulation, not a technical requirement. Having been a participant on Standards Working Groups, I understand completely. Regulations, like Standards, need to be written to apply to as wide a population as possible. Do those regulations dictate how you monitor drift? For example, can you have a system (I would use CentOS) that syncs to the appliance as well as NIST and your inside NTP sources, and use ntpq(8) to read the drift directly?
Re: CenturyLink...is being investigated by the FCC
The telephone companies (I'm looking at YOU Verizon!) are bringing this situation onto the community. I can see the FCC NPRM now: "What percentage of E911 terminations is being serviced over VoIP with carrier-based network switching, or third-party network switching, interfaced to the PSTN? "How many emergency service areas terminate E911 VoIP into an on-premises device like a Cisco voice router with outward-facing T1/E1 cards, or even outward-facing DS0 ports?" This is just the flip side of the problem with VoIP on the consumer side, not being able to easily associated a location with a 911 call without significant help from the calling device. Think cell phones on the one hand, and the ubiquitous Cisco VoIP desk set on the other. (Personal note: I have two copper-based DS0 lines here at my home office. And I'll keep them until Nevada Bell pries them out of my cold, dead hands. Now, those lines do terminate at a neighborhood SONET ring fiber terminal with a battery, but it's not my worry. Fax works fine -- which you can't say for VoIP connections.) On 12/28/18 2:21 PM, Patrick Boyle via NANOG wrote: > Ouch. Feel bad for the guys on the ground at C-link. Not a fun 24 hours. > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Friday, December 28, 2018 3:17 PM, Anne P. Mitchell, Esq. > wrote: > >> And the other latest news is that the FCC is investigating the CenturyLink >> outage: >> >> https://www.theinternetpatrol.com/fcc-investigating-centurylink-outage-says-unacceptable/ >> >>> On Dec 28, 2018, at 3:11 PM, Patrick Boyle via NANOG nanog@nanog.org wrote: >>> Yes, there were 911 services affected. The latest word from C-link as of >>> 1:46PM mountain is that all 911 services are restored where they are the >>> provider. I'm not 100% sure if that's system-wide, or just my area in the >>> northwest, however. >>> Sent with ProtonMail Secure Email. >>> ‐‐‐ Original Message ‐‐‐ >>> On Friday, December 28, 2018 1:03 AM, Stephane Bortzmeyer bortzme...@nic.fr >>> wrote: >>> On Fri, Dec 28, 2018 at 07:07:42AM +, Erik Sundberg esundb...@nitelusa.com wrote a message of 131 lines which said: > CenturyLink will be conducting an extensive post-incident > investigation and root cause analysis to provide follow-up > information to our customers Is this problem also responsible for the 911 outage? If so, the post-mortem analysis is not useful only for CenturyLink customers but for everyone on the west coast. > >
RE: CenturyLink
We have two stratum-1 servers synced with GPS and a PTP feed from a provider that also provides PTP to market data systems, but we still have to monitor drift between system time and NIST time. Don't ask for the logic behind it, it's a regulation, not a technical requirement. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell Sent: Saturday, December 29, 2018 9:34 AM To: nanog@nanog.org Subject: Re: CenturyLink On 12/28/18 3:23 PM, Yang Yu wrote: > On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer > wrote: >> Is this problem also responsible for the 911 outage? If so, the >> post-mortem analysis is not useful only for CenturyLink customers but >> for everyone on the west coast. > > Looks like most time.nist.gov servers (3 x NIST sites on AS49) are > single homed on CenturyLink, anyone noticed NTP issues yesterday? > > https://tf.nist.gov/tf-cgi/servers.cgi > I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't see any issues. I suspect many other operators are in the same situation.
Re: CenturyLink
On 12/28/18 3:23 PM, Yang Yu wrote: > On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer > wrote: >> Is this problem also responsible for the 911 outage? If so, the >> post-mortem analysis is not useful only for CenturyLink customers but >> for everyone on the west coast. > > Looks like most time.nist.gov servers (3 x NIST sites on AS49) are > single homed on CenturyLink, anyone noticed NTP issues yesterday? > > https://tf.nist.gov/tf-cgi/servers.cgi > I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't see any issues. I suspect many other operators are in the same situation.
RE: CenturyLink
Yep. We are required by FINRA to verify that our clocks on our trading systems are within a certain tolerance of NIST time. We are still seeing issues with 3 of the NIST servers this morning. Since NIST is on a skeleton crew anyway due to the government shutdown, I don't expect any resolution shortly. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Yang Yu Sent: Friday, December 28, 2018 6:23 PM To: Stephane Bortzmeyer Cc: nanog@nanog.org Subject: Re: CenturyLink On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer wrote: > Is this problem also responsible for the 911 outage? If so, the > post-mortem analysis is not useful only for CenturyLink customers but > for everyone on the west coast. Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed on CenturyLink, anyone noticed NTP issues yesterday? https://tf.nist.gov/tf-cgi/servers.cgi
