RE: Cost-effectivenesss of highly-accurate clocks for NTP

[Jameson, Daniel]

Give this guy a look,  1RMU form factor, GPS with Rubidium holdover (7 days) if 
you need it very inexpensive,  
http://www.fibrolan.com/FibroLAN/SendFile.asp?DBID=1=1=3979
or
http://www.fibrolan.com/FibroLAN/SendFile.asp?DBID=1=1=3978  if you 
have a SYC-E source
or  If you just need highly accurate NTP without the holdover,  you can do it 
with FBSD,  and a GPS device.
http://www.satsignal.eu/ntp/FreeBSD-GPS-PPS.htm
http://www.brandywinecomm.com/46-products/bus-level-plug-in-boards/212-pcie-1588-universal-timing-card



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
Sent: Monday, May 16, 2016 11:27 AM
To: Lamar Owen
Cc: nanog@nanog.org
Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP

Lamar,

Although VoIP has different technical challenges than TDM, they are all 
surmountable. Usually VoIP problems result from poor network design (e.g., 
mixed traffic with no QoS, Internet transport with no guarantees, etc). Public 
safety networks are generally well designed, don’t use the Internet, and 
support a single type of traffic: audio streams. I’ve done demonstrations of 
VoIP that works well in this environment, and you get the advantages of IP 
routing for redundancy, as opposed to clunky T1 failover mechanisms usually 
based on hardware switches.

But public safety customers are not swayed. TDM works, and it’s the gold 
standard. They don’t want to change, and you can’t make them. They only see 
risk, no reward. BTW, in the TDM model, remote data and audio are usually two 
different systems, which is probably a good idea for redundancy: you might lose 
audio or data, but rarely both.

In any event, I’m only proposing that public safety upgrade their audio systems 
to VoIP (or cellular-style private GSM, which is in essence VoIP). But nobody 
is willing.

 -mel

> On May 16, 2016, at 9:02 AM, Lamar Owen <lo...@pari.edu> wrote:
> 
> On 05/15/2016 01:05 PM, Eric S. Raymond wrote:
>> I'm not used to thinking of IT as a relatively low-challenge environment! 
> 
> I actually changed careers from broadcast engineering to IT to lower my 
> stress level and 'personal bandwidth challenge.'  And, yes, it worked.  In my 
> case, I'm doing IT for radio and optical astronomy, and at least the timing 
> aspect is higher-challenge that most IT environments.
> 
>> You're implicitly suggesting there might be a technical case for replacing 
>> these T1/T3 trunks with some kind of VOIP provisioning less dependent on 
>> accurate time synch. Do you think that's true? 
> 
> While I know the question was directed at Mel specifically, I'll just say 
> from the point of view of a T1 voice trunk customer that I hope to never see 
> it go to a VoIP solution.  VoIP codecs can have some serious latency issues; 
> I already notice the round-trip delay if I try to carry on a conversation 
> between our internal VoIP system and someone on a cell phone.  And this is 
> before codec artifacting (and cascaded codec scrambling) is counted.  Can we 
> please keep straight μ-law (A-law if relevant) lossless DS0 PCM timeslices 
> for trunklines so we get at least one less lossy codec cascade?  Or have you 
> never experimented with what happens when you cascade G.722 with G.729 with 
> G.726 and then G.711 and back?  Calls become mangled gibberish.
> 
> I would find it interesting to see how many carriers are still doing large 
> amounts of SONET, as that is the biggest use-case for high-stability timing.
> 

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

Lamar,

Although VoIP has different technical challenges than TDM, they are all 
surmountable. Usually VoIP problems result from poor network design (e.g., 
mixed traffic with no QoS, Internet transport with no guarantees, etc). Public 
safety networks are generally well designed, don’t use the Internet, and 
support a single type of traffic: audio streams. I’ve done demonstrations of 
VoIP that works well in this environment, and you get the advantages of IP 
routing for redundancy, as opposed to clunky T1 failover mechanisms usually 
based on hardware switches.

But public safety customers are not swayed. TDM works, and it’s the gold 
standard. They don’t want to change, and you can’t make them. They only see 
risk, no reward. BTW, in the TDM model, remote data and audio are usually two 
different systems, which is probably a good idea for redundancy: you might lose 
audio or data, but rarely both.

In any event, I’m only proposing that public safety upgrade their audio systems 
to VoIP (or cellular-style private GSM, which is in essence VoIP). But nobody 
is willing.

 -mel

> On May 16, 2016, at 9:02 AM, Lamar Owen  wrote:
> 
> On 05/15/2016 01:05 PM, Eric S. Raymond wrote:
>> I'm not used to thinking of IT as a relatively low-challenge environment! 
> 
> I actually changed careers from broadcast engineering to IT to lower my 
> stress level and 'personal bandwidth challenge.'  And, yes, it worked.  In my 
> case, I'm doing IT for radio and optical astronomy, and at least the timing 
> aspect is higher-challenge that most IT environments.
> 
>> You're implicitly suggesting there might be a technical case for replacing 
>> these T1/T3 trunks with some kind of VOIP provisioning less dependent on 
>> accurate time synch. Do you think that's true? 
> 
> While I know the question was directed at Mel specifically, I'll just say 
> from the point of view of a T1 voice trunk customer that I hope to never see 
> it go to a VoIP solution.  VoIP codecs can have some serious latency issues; 
> I already notice the round-trip delay if I try to carry on a conversation 
> between our internal VoIP system and someone on a cell phone.  And this is 
> before codec artifacting (and cascaded codec scrambling) is counted.  Can we 
> please keep straight μ-law (A-law if relevant) lossless DS0 PCM timeslices 
> for trunklines so we get at least one less lossy codec cascade?  Or have you 
> never experimented with what happens when you cascade G.722 with G.729 with 
> G.726 and then G.711 and back?  Calls become mangled gibberish.
> 
> I would find it interesting to see how many carriers are still doing large 
> amounts of SONET, as that is the biggest use-case for high-stability timing.
> 

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Lamar Owen]

On 05/15/2016 03:16 PM, Måns Nilsson wrote:
...If you think the IP implementations in IoT devices are naîve, wait 
until you've seen what passes for broadcast quality network 
engineering. Shoving digital audio samples in raw Ethernet frames is 
at least 20 years old, but the last perhaps 5 years has seen some 
progress in actually using IP to carry audio streams. (this is 
close-to-realtime audio, not file transfers, btw.) 


Close to realtime is a true statement.  Using an IP STL 
(studio-transmitter link) has enough latency that the announcer can no 
longer use the air signal as a monitor.


And the security side of things is a pretty serious issue; just ask a 
major IP STL appliance vendor about the recent hijacking of some of 
their customers' IP STL devices yeah, a random intruder on the 
internet hijacked several radio stations' IP STL's and began 
broadcasting their content over the radio.  Not pretty.  I advise any of 
my remaining broadcast clients that if they are going to an IP STL that 
they put in a dedicated point to point IP link without publicly routable 
IP addresses.


Digital audio for broadcast STL's is old tech; we were doing G.722/G.723 
over switched-56 in the early 90's.  But using a public-facing internet 
connection with no firewalling for an IP STL appliance like the Barix 
boxes and the Tieline boxes and similar? That borders on networking 
malpractice.


... But, to try to return to "relevant for NANOG", there are actual 
products requiring microsecond precision being sold. And used. And 
we've found that those products don't have a very good holdover. ... 

Television broadcast is another excellent example of timing needs; thanks.

Valdis mentioned the scariest thing the scariest thing I've seen 
recently?  Windows NT 3.5 being used for a transmitter control system, 
within the past five years.  I will agree with Valdis on the scary 
aspects of the public safety communications Mel mentioned. Thanks, Mel, 
for the educational post.

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Lamar Owen]

On 05/15/2016 01:05 PM, Eric S. Raymond wrote:
I'm not used to thinking of IT as a relatively low-challenge environment! 


I actually changed careers from broadcast engineering to IT to lower my 
stress level and 'personal bandwidth challenge.'  And, yes, it worked.  
In my case, I'm doing IT for radio and optical astronomy, and at least 
the timing aspect is higher-challenge that most IT environments.


You're implicitly suggesting there might be a technical case for 
replacing these T1/T3 trunks with some kind of VOIP provisioning less 
dependent on accurate time synch. Do you think that's true? 


While I know the question was directed at Mel specifically, I'll just 
say from the point of view of a T1 voice trunk customer that I hope to 
never see it go to a VoIP solution.  VoIP codecs can have some serious 
latency issues; I already notice the round-trip delay if I try to carry 
on a conversation between our internal VoIP system and someone on a cell 
phone.  And this is before codec artifacting (and cascaded codec 
scrambling) is counted.  Can we please keep straight μ-law (A-law if 
relevant) lossless DS0 PCM timeslices for trunklines so we get at least 
one less lossy codec cascade?  Or have you never experimented with what 
happens when you cascade G.722 with G.729 with G.726 and then G.711 and 
back?  Calls become mangled gibberish.


I would find it interesting to see how many carriers are still doing 
large amounts of SONET, as that is the biggest use-case for 
high-stability timing.

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[sthaug]

> I was just thing about this WAN jitter issue myself.  I'm wondering how many
> folks put NTP traffic in priority queues?  At least for devices in your
> managed IP ranges.  Seems like that would improve jitter.  Would like to
> hear about others doing this successfully prior to suggesting it for our
> network.

NTP, no. Not worth it. PTP, synched to a suitably accurate source,
absolutely.

Steinar Haug, AS2116

RE: Cost-effectivenesss of highly-accurate clocks for NTP

[Chuck Church]

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Leo Bicknell
Sent: Monday, May 16, 2016 10:28 AM
To: nanog@nanog.org
Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP

>For a typical site, there are two distinct desires from the same NTP
process.

>First, synchronization with the rest of the world, which is generally over
the WAN and the topic that was well discussed in your post.
>I agree completely that the largest factor is WAN jitter.

>Leo Bicknell - bickn...@ufp.org
>PGP keys at http://www.ufp.org/~bicknell/

I was just thing about this WAN jitter issue myself.  I'm wondering how many
folks put NTP traffic in priority queues?  At least for devices in your
managed IP ranges.  Seems like that would improve jitter.  Would like to
hear about others doing this successfully prior to suggesting it for our
network.

Chuck

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Leo Bicknell]

In a message written on Fri, May 13, 2016 at 03:39:27PM -0400, Eric S. Raymond 
wrote:
> According to RFC 5095 expected accuracy of NTP time is "several tens
> of milliseconds." User expectations seem to evolved to on the close
> order of 10ms.  I think it's not by coincidence this is pretty close
> to the jitter in ping times I see when I bounce ICMP off a
> well-provisioned site like (say) google.com through my Verizon FIOS
> connection.

For a typical site, there are two distinct desires from the same
NTP process.

First, syncronization with the rest of the world, which is generally
over the WAN and the topic that was well discussed in your post.
I agree completely that the largest factor is WAN jitter.

The other is local syncronization, insuring multiple servers have
the same time for log analysis purposes and such.  This typically
takes place across a lightly loaded ethernet fabric, perhaps with
small latency (across a compus).  Jitter here is much, much smaller.

Does the limitation of accuracy remain jitter in the low jitter
LAN/Campus enviornment, or does that expose other issues like the
quality of oscellators, OS scheduling, etc?  Or perhaps another
way, is it possible to get say 10's or 100's of nanosecond accuracy
in the lan/campus?

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpE9K0yZ7Yjy.pgp
Description: PGP signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

Joe and Eric,

It's frustrating how far public safety technology lags behind what Industry can 
actually deliver. It's the same in aviation. Institutions are slow to adopt new 
tech due to fears about reliability, and and unwillingness to take any risk at 
all.  So PS and aviation capabilities lag horribly. This is why commercial 
pilots, tired of waiting on the FAA, are buying their own tablets and running 
non-certified navigation tools. And police officers use cellular data 
connection with VPN to query wants and warrants databases. 

-mel beckman

>> On May 15, 2016, at 5:28 PM, joel jaeggli  wrote:
>> 
>>> On 5/15/16 10:05 AM, Eric S. Raymond wrote:
>>> Mel Beckman :
>>> The upshot is that there are many real-world situations where
>>> expensive clock discipline is needed. But IT isn't, I don't think,
>>> one of them, with the exception of private SONET networks (fast
>>> disappearing in the face of metro Ethernet).
>> 
>> Thank you, that was very interesting information.  I'm not used to thinking
>> of IT as a relatively low-challenge environment!
>> 
>> You're implicitly suggesting there might be a technical case for
>> replacing these T1/T3 trunks with some kind of VOIP provisioning less
>> dependent on accurate time synch.  Do you think that's true?
> 
> APCO  and TETRA trunked radio  are mature systems, they do carry data,
> but are somewhat lower bandwidth. Being TDM they are dependent on
> accurate clocks.
> 
> LTE systems are used or envisioned being used for high bandwidth
> applications.
> 
> 
> 

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[joel jaeggli]

On 5/15/16 10:05 AM, Eric S. Raymond wrote:
> Mel Beckman :
>> The upshot is that there are many real-world situations where
>> expensive clock discipline is needed. But IT isn't, I don't think,
>> one of them, with the exception of private SONET networks (fast
>> disappearing in the face of metro Ethernet).
> 
> Thank you, that was very interesting information.  I'm not used to thinking
> of IT as a relatively low-challenge environment!
> 
> You're implicitly suggesting there might be a technical case for
> replacing these T1/T3 trunks with some kind of VOIP provisioning less
> dependent on accurate time synch.  Do you think that's true?

APCO  and TETRA trunked radio  are mature systems, they do carry data,
but are somewhat lower bandwidth. Being TDM they are dependent on
accurate clocks.

LTE systems are used or envisioned being used for high bandwidth
applications.

> 




signature.asc
Description: OpenPGP digital signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Måns Nilsson]

Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP Date: Sun, 
May 15, 2016 at 03:21:02PM + Quoting Mel Beckman (m...@beckman.org):
 
> The upshot is that there are many real-world situations where expensive clock 
> discipline is needed. But IT isn't, I don't think, one of them, with the 
> exception of private SONET networks (fast disappearing in the face of metro 
> Ethernet).

Pro audio is moving to Ethernet (they talk about it, Ethernet, as either
"RJ45" or "Internet"...) and sometimes even to IP in a fairly rapid
pace. If you think the IP implementations in IoT devices are naîve, wait
until you've seen what passes for broadcast quality network engineering.

Shoving digital audio samples in raw Ethernet frames is at least 20 years
old, but the last perhaps 5 years has seen some progress in actually
using IP to carry audio streams. (this is close-to-realtime audio,
not file transfers, btw.)

A lot of audio is sent using codecs like Opus, with SIP as
signalling. That works quite nicely. We've got a smartphone app to do
that, for instance. 

But, this is all mostly floating in terms of absolute sampling
frequency. Digital audio needs a clock to work. In the simple home stereo
case, this is taken care of by listening to the pace samples arrive at,
and using that. But as soon as you are mixing two sources, they need
to be in tune. Something needs to decide what to use as master. In the
smartphone case, we simply buffer some 20-100ms of audio and start playing
back, using our own clock. Then we hope the interview is over before
the buffer is overflowing or drained. Which mostly works.

Inside facilities, when we use the SIP-signalled streams, we usually can
rely on a separate clock distribution. In our specific case, we've bought
country-wide clock distribution that gives us the same sample clock in
all facilities. (Digital TV is mostly built as single frequency networks,
which requires syntonous (at least) transmitters. Thus, it today is
quite easy to find providers of frequency in the broadcast business.)

Now, the Audio Engineering Society has published AES67 which in essence
is multichannel, multicast RTP audio (L48 mediatype, ie. linear 48KHz
24-bit) synchronized by PTP, also multicast.

Now, bear in mind that I wrote _synchronized_, not _syntonized_. 

Up to now, the only thing that mattered to keep track of was
frequency. Since one of the big reasons for AES67 is distributing
sound to several different loudspeakers that can be heard by one
listener simultaneously, the prime example being a stereo pair of active
loudspeakers with one network jack on each, _phase_ matters, as well as
absolute time. (Mostly, telco synchronization mentions absolute time as
phase.) This application requires absolute time, since a mono sound
in our stereo example needs to play back _at the same time_ from both
speakers. Or it ceases to be a mono sound, instead becoming a sound
that is offset in the soundstage by delaying it. 
Most classical stereo recordings are mono in terms of level, but not in
terms of the time domain; since they derive all spatial info from time,
not gain. Like we humans do.

The usual test case is to buy a PTP-aware switch, a PTP Grand Master,
steered by   and build a small LAN, test that Vendor A and
Vendor B can send audio between themselves via this simple network and
call it a day.

That is a nice lab setup. Also very far from what needs to be built in 
order to solve the actual production cases. 

But, to try to return to "relevant for NANOG", there are actual products
requiring microsecond precision being sold. And used. And we've found
that those products don't have a very good holdover. On ranty days I
usually accuse them of having hotglued an Ethernet adapter onto the old
TDM-based audio devices and sent them out to customers with a prayer
and instructions to build an overengineered network to make certain that
PTP always is delivered with zero IPDV.

A lot of strange things are getting network connectors these days. Not
all of them are content with a http connection to some cloud provider.
-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
The PILLSBURY DOUGHBOY is CRYING for an END to BURT REYNOLDS movies!!


signature.asc
Description: Digital signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Valdis . Kletnieks]

On Sun, 15 May 2016 15:21:02 -, Mel Beckman said:
> But a more critical deployment of rubidium clocks is in cash-strapped public
> safety institutions, such as local police dispatch centers. Timing is crucial
> for the squad car communication systems, which these days are all digital,
> based on wireless T1/T3 trunks to remote repeaters. The clocking on these
> trunks can't drift much before voice communication fails due to repeater
> outages. The telecom gear has OXCO clocks that can provide a few hours
> holdover. A rubidium clock onsite provides coverage for longer outages.

That may be the scariest entry on "Things I was not aware of" for quite
some time


pgpC80zoEyR7q.pgp
Description: PGP signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Mel Beckman :
> The upshot is that there are many real-world situations where
> expensive clock discipline is needed. But IT isn't, I don't think,
> one of them, with the exception of private SONET networks (fast
> disappearing in the face of metro Ethernet).

Thank you, that was very interesting information.  I'm not used to thinking
of IT as a relatively low-challenge environment!

You're implicitly suggesting there might be a technical case for
replacing these T1/T3 trunks with some kind of VOIP provisioning less
dependent on accurate time synch.  Do you think that's true?
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

I have deployed rubidium-disciplined clocks at cellular carriers, where money 
is no object (look at your cellphone bill), typically for $20K-$100K for 
redundant clocks. The holdover time on these is supposed to be days, but we've 
never tested that. I think I'd better. 

But a more critical deployment of rubidium clocks is in cash-strapped public 
safety institutions, such as local police dispatch centers. Timing is crucial 
for the squad car communication systems, which these days are all digital, 
based on wireless T1/T3 trunks to remote repeaters. The clocking on these 
trunks can't drift much before voice communication fails due to repeater 
outages. The telecom gear has OXCO clocks that can provide a few hours 
holdover. A rubidium clock onsite provides coverage for longer outages. 

In these installations I have tested GPS outages of up to a week without enough 
clock drift to knock out the Tx links. I've never gone longer than that though, 
so I don't know the actual breaking point. But if you lose that rubidium clock, 
things go south in a less than a day.

The cash-strapping comes into play when municipal bean counters eyeball the 
rubidium clock(s) and start thinking they can get by with a cheaper substitute. 

The upshot is that there are many real-world situations where expensive clock 
discipline is needed. But IT isn't, I don't think, one of them, with the 
exception of private SONET networks (fast disappearing in the face of metro 
Ethernet).

 -mel beckman

> On May 15, 2016, at 3:22 AM, Eric S. Raymond  wrote:
> 
> Baldur Norddahl :
>> Ok how many hours or days of holdover can you expect from quartz,
>> temperature compensated quartz or Rubidium?
> 
> Sorry, I don't have those drift figures handy.  I'm a programmer, not
> a large-site sysadmin - I've never had purchase authority with a
> budget large enough to buy a rubidium-oscillator GPSDO or any other
> device with holdover.  Better to ask Mel Beckman or someone else
> with field experience.
> 
>>Should we calculate holdover as
>> time until drift is more than 1 millisecond, 10 ms or more for NTP
>> applications?
> 
> If you want to go super-accurate, 1ms.  If you want to go cheap, on
> sampling-theory grounds I'd say you want to vary your drift threshold
> from 1 to 5ms (half the expected precision of WAN time, think of it as
> the Nyquist rate) and look for a knee in the cost curve.
> 
>> I am thinking that many available datacenter locations will have poor GPS
>> signal so we can expect signal loss to be common. Some weather patterns
>> might even cause extended GPS signal loss.
> 
> Weather won't do it, usually. Rain and fog and clouds are transparent
> to GPS frequencies. Standing water directly on an antenna can cause
> some attenuation, but with any serious GPS engine made more recently
> than 5 years ago I would be extremely surprised if that lost it
> lock.  The newer ones handle down to 30 feet in ocean water on robot
> submarines.
> -- 
>http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Bruce Simpson :
> On 13/05/16 20:39, Eric S. Raymond wrote:
> >In 2012, nearly three years before being recruited for NTPsec, I
> >solved this problem as part of my work on GPSD.  The key to this
> >solution is an obscure feature of USB, and a one-wire
> >patch to the bog-standard design for generic USB that exploits
> >it.  Technical details on request, but what it comes down to is
> >that with this one weird trick(!) you can mass-produce primary time
> >sources with a jitter bounded by the USB polling interval for
> >about $20 a pop.
> >
> >The USB 1 polling interval is 1ms.
> 
> What about USB 3.1 (assuming the device is not intended to be backwards
> compatible with the polling model) ? I should point out Intel intend to
> retire EHCI/UHCI and implement only xHCI.

Nobody makes GPSes with even USB 2 or 3 yet, and it is unlikely to happen
for a long time.  Cost reasons - USB GPSes are cheap consumer-grade hardware
and the manufacturers care about fractions of a cent on the BOM. 
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Baldur Norddahl :
> Ok how many hours or days of holdover can you expect from quartz,
> temperature compensated quartz or Rubidium?

Sorry, I don't have those drift figures handy.  I'm a programmer, not
a large-site sysadmin - I've never had purchase authority with a
budget large enough to buy a rubidium-oscillator GPSDO or any other
device with holdover.  Better to ask Mel Beckman or someone else
with field experience.

> Should we calculate holdover as
> time until drift is more than 1 millisecond, 10 ms or more for NTP
> applications?

If you want to go super-accurate, 1ms.  If you want to go cheap, on
sampling-theory grounds I'd say you want to vary your drift threshold
from 1 to 5ms (half the expected precision of WAN time, think of it as
the Nyquist rate) and look for a knee in the cost curve.

> I am thinking that many available datacenter locations will have poor GPS
> signal so we can expect signal loss to be common. Some weather patterns
> might even cause extended GPS signal loss.

Weather won't do it, usually. Rain and fog and clouds are transparent
to GPS frequencies. Standing water directly on an antenna can cause
some attenuation, but with any serious GPS engine made more recently
than 5 years ago I would be extremely surprised if that lost it
lock.  The newer ones handle down to 30 feet in ocean water on robot
submarines.
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

> I clearly need three of those maser things for my network.

Gives new meaning to the phrase "Set and forget". :)

 -mel beckman

> On May 14, 2016, at 12:40 PM, Baldur Norddahl  
> wrote:
> 
>> On 13 May 2016 at 23:01, Baldur Norddahl  wrote:
>> 
>> Ok how many hours or days of holdover can you expect from quartz,
>> temperature compensated quartz or Rubidium? Should we calculate holdover as
>> time until drift is more than 1 millisecond, 10 ms or more for NTP
>> applications?
>> 
>> I am thinking that many available datacenter locations will have poor GPS
>> signal so we can expect signal loss to be common. Some weather patterns
>> might even cause extended GPS signal loss.
>> 
>> 
>> 
> I found some data points here: https://en.wikipedia.org/wiki/Crystal_oven
> 
> Assuming that acceptable drift is 10 milliseconds due that being the
> expected accuracy from NTP.
> 
> The common crystal oscillator can be as bad as 1E-4 => holdover time is 2
> minutes.
> TCXO is listed as 1E-6 => holdover time is 3 hours.
> OCXO is listed as multiple values, I will use 1E-7 => holdover time is 1
> day.
> Rubidium is listed as 1E-9 => 3 months
> Caesium is listed as 1E-11 => 30 years
> Hydrogen Maser 1E-15 => 300 millennia
> 
> I clearly need three of those maser things for my network.
> 
> Regards,
> 
> Baldur

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Baldur Norddahl]

On 13 May 2016 at 23:01, Baldur Norddahl  wrote:

> Ok how many hours or days of holdover can you expect from quartz,
> temperature compensated quartz or Rubidium? Should we calculate holdover as
> time until drift is more than 1 millisecond, 10 ms or more for NTP
> applications?
>
> I am thinking that many available datacenter locations will have poor GPS
> signal so we can expect signal loss to be common. Some weather patterns
> might even cause extended GPS signal loss.
>
>
>
I found some data points here: https://en.wikipedia.org/wiki/Crystal_oven

Assuming that acceptable drift is 10 milliseconds due that being the
expected accuracy from NTP.

The common crystal oscillator can be as bad as 1E-4 => holdover time is 2
minutes.
TCXO is listed as 1E-6 => holdover time is 3 hours.
OCXO is listed as multiple values, I will use 1E-7 => holdover time is 1
day.
Rubidium is listed as 1E-9 => 3 months
Caesium is listed as 1E-11 => 30 years
Hydrogen Maser 1E-15 => 300 millennia

I clearly need three of those maser things for my network.

Regards,

Baldur

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Lamar Owen]

On 05/13/2016 03:39 PM, Eric S. Raymond wrote:
Traditionally dedicated time-source hardware like rubidium-oscillator 
GPSDOs is sold on accuracy, but for WAN time service their real draw 
is long holdover time with lower frequency drift that you get from the 
cheap, non-temperature-compensated quartz crystals in your PC. There 
is room for debate about how much holdover you should pay for, but 
you'll at least be thinking more clearly about the problem if you 
recognize that you *should not* buy expensive hardware for accuracy. 
For WAN time service, in that price range, you're wither buying 
holdover and knowing you're doing so or wasting your money. 


Eric,

Thanks for the pointers; nice information.

A cheap way to get a WAN frequency standard is to use a WAN that is 
delivered over something derived from the telco's synchronous network; a 
POS on an OC3 with the clock set to network has an exceptionally stable 
frequency standard available.  Less expensive, get a voice T1 trunk 
delivered (robbed-bit signaled will typically be less expensive than 
PRI) and grab clock from that; tarriffs for RBS T1/fractional T1 around 
here at least are less than an analog POTS line).  Very stable.  The 
plesiochronous digital hierarchy on copper or synchronous digital 
hierarchy/SONET on fiber have cesium clocks behind them, and you can get 
that stability by doing clock recovery on those WAN circuits.  Back when 
this was the most common WAN technology frequency standards were there 
for the taking; Ethernet, on the other hand, not so much.


But a nice catch on using the isochronous nature of USB.  Cheap webcams 
also take advantage of the isochronous transfer mode.  Do note that 
isochronous is often not supported in USB-passthrough for 
virtualization, though.  But you shouldn't use a VM to do timing, 
either. :-)


Now I'm looking for myself one of those Navisys devices you 
mentioned. do any of them have external antenna inputs, say on an 
SMA connector (MCX is in my experience just too fragile) with a bias tee 
to drive phantom to an active antenna?  The quick search I did seemed to 
indicate that the three you mentioned are self-contained with their own 
smart antenna.  External antenna input would be required here, where we 
use timing-grade GPS antennas to feed our Z3816's.  But for straight 
1PPS and GPS timecode, dealing with the Z3816's complexity is overkill.


Thanks again for the info; looking forward to seeing how NTPsec develops.

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Baldur Norddahl]

Den 13. maj 2016 21.40 skrev "Eric S. Raymond" :

> Traditionally dedicated time-source hardware like rubidium-oscillator
> GPSDOs is sold on accuracy, but for WAN time service their real draw
> is long holdover time with lower frequency drift that you get from the
> cheap, non-temperature-compensated quartz crystals in your PC.
>
> There is room for debate about how much holdover you should pay for,
> but you'll at least be thinking more clearly about the problem if
> you recognize that you *should not* buy expensive hardware for
> accuracy.  For WAN time service, in that price range, you're wither
> buying holdover and knowing you're doing so or wasting your money.

Ok how many hours or days of holdover can you expect from quartz,
temperature compensated quartz or Rubidium? Should we calculate holdover as
time until drift is more than 1 millisecond, 10 ms or more for NTP
applications?

I am thinking that many available datacenter locations will have poor GPS
signal so we can expect signal loss to be common. Some weather patterns
might even cause extended GPS signal loss.

Regards

Baldur

Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Mel Beckman :
>Finally, do you want to weigh in on the necessity for highly accurate local RT
>clocks in NTP servers? That seems to be the big bugaboo in cost limiting right
>now.

Yes, this is a topic on which I have some well-developed judgments
due to having collected (and, where practical, tested) a pretty
comprehensive set of figures on components of the NTP error budget.
I've even invented some hardware that simplifies the problem.

The background to my findings is laid out in my "Introduction to Time
Service" HOWTO:

http://www.catb.org/gpsd/time-service-intro.html

I find that an effective way to learn my way into a new application domain
is to first do knowledge capture on the assumptions its experts are
using and then document those. "Introduction to Time Service" was
written to do that and as a white paper for my project management.
Criticism and corrections are, of course, welcome.

In order to discuss the value of accurate clocks intelligently, we need
to split apart two issues: accuracy and availability. Of course we
want the most accurate time our networks can deliver; we also want to
hedge against sporadic or systemic failure of single time sources.

The most important simplification of either issue is that clock
accuracy worth paying for is bounded both by user expectations and
the noise floor defined by our network jitter.

According to RFC 5095 expected accuracy of NTP time is "several tens
of milliseconds." User expectations seem to evolved to on the close
order of 10ms.  I think it's not by coincidence this is pretty close
to the jitter in ping times I see when I bounce ICMP off a
well-provisioned site like (say) google.com through my Verizon FIOS
connection.

It's good rule-of-thumb engineering that if you want to be
metrologically accurate you should pay for precision an order of
magnitude below your feature size *and not more than that*.  Thus,
with a feature size of 10ms the economic sweet spot is a clock with
accuracy about 1ms.

One reason discussions of how to budget for WAN timeservice clocks has
tended to become heated in the past is that nothing inexpensive hit
this sweet spot.  The world was largely divided between cheap time
sources with too much jitter (e.g. GPS in-band data with a wander of
100ms or worse) and expensive high-precision clocks designed for PTP
over Ethernet that deliver three or more orders of magnitude than WAN
time service can actually use.

However...also use the 1PPS signal your GPS engine ships (top of UTC
second accurate to about 50ns) and the picture changes
completely. With that over RS232 your delivered accuracy rises to
single-digit numbers of microseconds, which is two orders of magnitude
below what you need for your 1ms goal.

Now we have a historical problem, though: RS232 and the handshake
lines you could use to signal 1PPS are dying, being replaced by USB.
which doesn't normally bring 1PPS out to where the timeserver OS
can see it.

In 2012, nearly three years before being recruited for NTPsec, I
solved this problem as part of my work on GPSD.  The key to this
solution is an obscure feature of USB, and a one-wire
patch to the bog-standard design for generic USB that exploits
it.  Technical details on request, but what it comes down to is
that with this one weird trick(!) you can mass-produce primary time
sources with a jitter bounded by the USB polling interval for
about $20 a pop.

The USB 1 polling interval is 1ms. Bingo.  We're done.  If we're only
weighting accuracy and not availability, a USB GPS is as much clock as
you need for WAN timeservice *provided it exposes 1PPS*.  These
devices exist, because I designed them and found a shop in Shenzhen
to build them. They're called the Navisys GR-601W, GR-701W, and
GR-801W.

(A viable, only skightly more expensive alternative is to mate a GPS
daughterboard to a hackerboard like the Raspberry Pi and run NTP
service on that.  I'll have much, much more to say about that in a
future post.)

Of course, now we have to talk about availability.  GPS sometimes
loses lock.  There are sporadic and systemic availability risks due to
jamming and system failures like the 2012 hiccup, and extreme
scenarios like a giant meteorite hitting GPSS ground control in
Colorado.

What you should be willing to pay for a hedge against this is
proportional to your 95% confidence estimate of the maximum
outage interval. At the low end, this is simple; put your
antenna on a mast that guarantees unobstructed skyview.  At the high
end it's effectively impossible, in that anything that takes down GNSS
and Galileo permanently (giant meteor impact, war in space, collapse
of the U.S. and Europe) is likely to be in the you-have-much-bigger-
problems than-inaccurate-time department.

Traditionally dedicated time-source hardware like rubidium-oscillator
GPSDOs is sold on accuracy, but for WAN time service their real draw
is long holdover time with lower frequency drift that you get from the
cheap,