Re: Speaking of NTP...

[Matthew Huff]

Thanks. We have always had a few outliers, but we have never had a large number 
of external NTP have a consistent offset, and not one as big as 10ms. Something 
changed last Friday, probably at some peering point that caused the issue. 
Maybe a symmetric path got created to route around some outage. Maybe some MPLS 
circuit got introduced into the mix that hides the underlying path/latency. 
Glad to know someone else has seen something like this.

Our 3 NTP servers that sync from external sources have at least 5 upstream 
stratum 1 servers and are peered to each other . They have settled on  a sense 
of time that is good within +/i 1 msec of our strata 1 clocks, so all is good, 
but it was a stage occurrence after we had been good for so long. Each of our 
servers are clients of our 2 x strata 1 servers and 3 x strata 2 NTP servers. 
They all look good now.

 


> On Jul 16, 2015, at 3:24 PM, Tony Hain  wrote:
> 
> I have had a consistent 10ms offset on a set of servers for the last 5 years. 
> After extensive one-way tracing, it turns out there is a 20ms asymmetry 
> "within" the Seattle Westin colo between HE & Comcast, causing all the IPv6 
> peers appearing over the HE tunnel to be 10ms offset from everything else. 
> There may be other instances of indirect peering causing a static asymmetric 
> path delay, and NTP will report that as an offset of half of the difference. 
> 
> Tony
> 
> 
>> -Original Message-
>> From: NANOG [mailto:nanog-bounces+alh-ietf=tndh@nanog.org] On
>> Behalf Of Rafael Possamai
>> Sent: Thursday, July 16, 2015 8:53 AM
>> To: Matthew Huff
>> Cc: nanog@nanog.org
>> Subject: Re: Speaking of NTP...
>> 
>> Depending on how exactly you have these servers configured with relation
>> to one another, small variations from one single source can be augmented
>> down the line.
>> 
>> https://en.wikipedia.org/wiki/Propagation_of_uncertainty
>> 
>> 
>> 
>> On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff  wrote:
>> 
>>> We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers
>>> with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external
>>> internet based NTP stratum 1 servers. We monitor our NTP environment
>>> closely, and over the last 10+ years, normally all of our NTP servers
>>> are sync'ed within
>>> +/- 2 msec. Starting last Friday, we started seeing some remote NTP
>>> +servers
>>> with GPS reference consistently offset by 10 msec.
>>> 
>>> Any one else seeing this?
>>> 
>>> 
>>> Matthew Huff | 1 Manhattanville Rd
>>> Director of Operations   | Purchase, NY 10577
>>> OTA Management LLC   | Phone: 914-460-4039
>>> aim: matthewbhuff| Fax:   914-694-5669
>>> 
>>> 
> 

RE: Speaking of NTP...

[Tony Hain]

I have had a consistent 10ms offset on a set of servers for the last 5 years. 
After extensive one-way tracing, it turns out there is a 20ms asymmetry 
"within" the Seattle Westin colo between HE & Comcast, causing all the IPv6 
peers appearing over the HE tunnel to be 10ms offset from everything else. 
There may be other instances of indirect peering causing a static asymmetric 
path delay, and NTP will report that as an offset of half of the difference. 

Tony


> -Original Message-
> From: NANOG [mailto:nanog-bounces+alh-ietf=tndh@nanog.org] On
> Behalf Of Rafael Possamai
> Sent: Thursday, July 16, 2015 8:53 AM
> To: Matthew Huff
> Cc: nanog@nanog.org
> Subject: Re: Speaking of NTP...
> 
> Depending on how exactly you have these servers configured with relation
> to one another, small variations from one single source can be augmented
> down the line.
> 
> https://en.wikipedia.org/wiki/Propagation_of_uncertainty
> 
> 
> 
> On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff  wrote:
> 
> > We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers
> > with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external
> > internet based NTP stratum 1 servers. We monitor our NTP environment
> > closely, and over the last 10+ years, normally all of our NTP servers
> > are sync'ed within
> > +/- 2 msec. Starting last Friday, we started seeing some remote NTP
> > +servers
> > with GPS reference consistently offset by 10 msec.
> >
> > Any one else seeing this?
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff| Fax:   914-694-5669
> >
> >

Re: Speaking of NTP...

[Rafael Possamai]

Depending on how exactly you have these servers configured with relation to
one another, small variations from one single source can be augmented down
the line.

https://en.wikipedia.org/wiki/Propagation_of_uncertainty



On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff  wrote:

> We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers with
> GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet
> based NTP stratum 1 servers. We monitor our NTP environment closely, and
> over the last 10+ years, normally all of our NTP servers are sync'ed within
> +/- 2 msec. Starting last Friday, we started seeing some remote NTP servers
> with GPS reference consistently offset by 10 msec.
>
> Any one else seeing this?
>
> 
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC   | Phone: 914-460-4039
> aim: matthewbhuff| Fax:   914-694-5669
>
>

Re: Speaking of NTP...

[Stephane Bortzmeyer]

On Mon, Jul 13, 2015 at 01:17:01PM +,
 Matthew Huff  wrote 
 a message of 14 lines which said:

> We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers
> with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to
> external internet based NTP stratum 1 servers. We monitor our NTP
> environment closely, and over the last 10+ years, normally all of
> our NTP servers are sync'ed within +/- 2 msec. Starting last Friday,
> we started seeing some remote NTP servers with GPS reference
> consistently offset by 10 msec.

I have no idea but I just wanted to remind people that, for a few
months, RIPE Atlas probes have been able to do NTP queries
 so it may be a
cool way to monitor NTP servers from many points.

Speaking of NTP...

[Matthew Huff]

We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers with GPS 
sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP 
stratum 1 servers. We monitor our NTP environment closely, and over the last 
10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. 
Starting last Friday, we started seeing some remote NTP servers with GPS 
reference consistently offset by 10 msec. 

Any one else seeing this? 


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669