Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

Joe and Eric,

It's frustrating how far public safety technology lags behind what Industry can 
actually deliver. It's the same in aviation. Institutions are slow to adopt new 
tech due to fears about reliability, and and unwillingness to take any risk at 
all.  So PS and aviation capabilities lag horribly. This is why commercial 
pilots, tired of waiting on the FAA, are buying their own tablets and running 
non-certified navigation tools. And police officers use cellular data 
connection with VPN to query wants and warrants databases. 

-mel beckman

>> On May 15, 2016, at 5:28 PM, joel jaeggli  wrote:
>> 
>>> On 5/15/16 10:05 AM, Eric S. Raymond wrote:
>>> Mel Beckman :
>>> The upshot is that there are many real-world situations where
>>> expensive clock discipline is needed. But IT isn't, I don't think,
>>> one of them, with the exception of private SONET networks (fast
>>> disappearing in the face of metro Ethernet).
>> 
>> Thank you, that was very interesting information.  I'm not used to thinking
>> of IT as a relatively low-challenge environment!
>> 
>> You're implicitly suggesting there might be a technical case for
>> replacing these T1/T3 trunks with some kind of VOIP provisioning less
>> dependent on accurate time synch.  Do you think that's true?
> 
> APCO  and TETRA trunked radio  are mature systems, they do carry data,
> but are somewhat lower bandwidth. Being TDM they are dependent on
> accurate clocks.
> 
> LTE systems are used or envisioned being used for high bandwidth
> applications.
> 
> 
> 

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[joel jaeggli]

On 5/15/16 10:05 AM, Eric S. Raymond wrote:
> Mel Beckman :
>> The upshot is that there are many real-world situations where
>> expensive clock discipline is needed. But IT isn't, I don't think,
>> one of them, with the exception of private SONET networks (fast
>> disappearing in the face of metro Ethernet).
> 
> Thank you, that was very interesting information.  I'm not used to thinking
> of IT as a relatively low-challenge environment!
> 
> You're implicitly suggesting there might be a technical case for
> replacing these T1/T3 trunks with some kind of VOIP provisioning less
> dependent on accurate time synch.  Do you think that's true?

APCO  and TETRA trunked radio  are mature systems, they do carry data,
but are somewhat lower bandwidth. Being TDM they are dependent on
accurate clocks.

LTE systems are used or envisioned being used for high bandwidth
applications.

> 




signature.asc
Description: OpenPGP digital signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Måns Nilsson]

Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP Date: Sun, 
May 15, 2016 at 03:21:02PM + Quoting Mel Beckman (m...@beckman.org):
 
> The upshot is that there are many real-world situations where expensive clock 
> discipline is needed. But IT isn't, I don't think, one of them, with the 
> exception of private SONET networks (fast disappearing in the face of metro 
> Ethernet).

Pro audio is moving to Ethernet (they talk about it, Ethernet, as either
"RJ45" or "Internet"...) and sometimes even to IP in a fairly rapid
pace. If you think the IP implementations in IoT devices are naîve, wait
until you've seen what passes for broadcast quality network engineering.

Shoving digital audio samples in raw Ethernet frames is at least 20 years
old, but the last perhaps 5 years has seen some progress in actually
using IP to carry audio streams. (this is close-to-realtime audio,
not file transfers, btw.)

A lot of audio is sent using codecs like Opus, with SIP as
signalling. That works quite nicely. We've got a smartphone app to do
that, for instance. 

But, this is all mostly floating in terms of absolute sampling
frequency. Digital audio needs a clock to work. In the simple home stereo
case, this is taken care of by listening to the pace samples arrive at,
and using that. But as soon as you are mixing two sources, they need
to be in tune. Something needs to decide what to use as master. In the
smartphone case, we simply buffer some 20-100ms of audio and start playing
back, using our own clock. Then we hope the interview is over before
the buffer is overflowing or drained. Which mostly works.

Inside facilities, when we use the SIP-signalled streams, we usually can
rely on a separate clock distribution. In our specific case, we've bought
country-wide clock distribution that gives us the same sample clock in
all facilities. (Digital TV is mostly built as single frequency networks,
which requires syntonous (at least) transmitters. Thus, it today is
quite easy to find providers of frequency in the broadcast business.)

Now, the Audio Engineering Society has published AES67 which in essence
is multichannel, multicast RTP audio (L48 mediatype, ie. linear 48KHz
24-bit) synchronized by PTP, also multicast.

Now, bear in mind that I wrote _synchronized_, not _syntonized_. 

Up to now, the only thing that mattered to keep track of was
frequency. Since one of the big reasons for AES67 is distributing
sound to several different loudspeakers that can be heard by one
listener simultaneously, the prime example being a stereo pair of active
loudspeakers with one network jack on each, _phase_ matters, as well as
absolute time. (Mostly, telco synchronization mentions absolute time as
phase.) This application requires absolute time, since a mono sound
in our stereo example needs to play back _at the same time_ from both
speakers. Or it ceases to be a mono sound, instead becoming a sound
that is offset in the soundstage by delaying it. 
Most classical stereo recordings are mono in terms of level, but not in
terms of the time domain; since they derive all spatial info from time,
not gain. Like we humans do.

The usual test case is to buy a PTP-aware switch, a PTP Grand Master,
steered by   and build a small LAN, test that Vendor A and
Vendor B can send audio between themselves via this simple network and
call it a day.

That is a nice lab setup. Also very far from what needs to be built in 
order to solve the actual production cases. 

But, to try to return to "relevant for NANOG", there are actual products
requiring microsecond precision being sold. And used. And we've found
that those products don't have a very good holdover. On ranty days I
usually accuse them of having hotglued an Ethernet adapter onto the old
TDM-based audio devices and sent them out to customers with a prayer
and instructions to build an overengineered network to make certain that
PTP always is delivered with zero IPDV.

A lot of strange things are getting network connectors these days. Not
all of them are content with a http connection to some cloud provider.
-- 
Måns Nilsson primary/secondary/besserwisser/machina
MN-1334-RIPE +46 705 989668
The PILLSBURY DOUGHBOY is CRYING for an END to BURT REYNOLDS movies!!


signature.asc
Description: Digital signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Valdis . Kletnieks]

On Sun, 15 May 2016 15:21:02 -, Mel Beckman said:
> But a more critical deployment of rubidium clocks is in cash-strapped public
> safety institutions, such as local police dispatch centers. Timing is crucial
> for the squad car communication systems, which these days are all digital,
> based on wireless T1/T3 trunks to remote repeaters. The clocking on these
> trunks can't drift much before voice communication fails due to repeater
> outages. The telecom gear has OXCO clocks that can provide a few hours
> holdover. A rubidium clock onsite provides coverage for longer outages.

That may be the scariest entry on "Things I was not aware of" for quite
some time


pgpC80zoEyR7q.pgp
Description: PGP signature

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Mel Beckman :
> The upshot is that there are many real-world situations where
> expensive clock discipline is needed. But IT isn't, I don't think,
> one of them, with the exception of private SONET networks (fast
> disappearing in the face of metro Ethernet).

Thank you, that was very interesting information.  I'm not used to thinking
of IT as a relatively low-challenge environment!

You're implicitly suggesting there might be a technical case for
replacing these T1/T3 trunks with some kind of VOIP provisioning less
dependent on accurate time synch.  Do you think that's true?
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Mel Beckman]

I have deployed rubidium-disciplined clocks at cellular carriers, where money 
is no object (look at your cellphone bill), typically for $20K-$100K for 
redundant clocks. The holdover time on these is supposed to be days, but we've 
never tested that. I think I'd better. 

But a more critical deployment of rubidium clocks is in cash-strapped public 
safety institutions, such as local police dispatch centers. Timing is crucial 
for the squad car communication systems, which these days are all digital, 
based on wireless T1/T3 trunks to remote repeaters. The clocking on these 
trunks can't drift much before voice communication fails due to repeater 
outages. The telecom gear has OXCO clocks that can provide a few hours 
holdover. A rubidium clock onsite provides coverage for longer outages. 

In these installations I have tested GPS outages of up to a week without enough 
clock drift to knock out the Tx links. I've never gone longer than that though, 
so I don't know the actual breaking point. But if you lose that rubidium clock, 
things go south in a less than a day.

The cash-strapping comes into play when municipal bean counters eyeball the 
rubidium clock(s) and start thinking they can get by with a cheaper substitute. 

The upshot is that there are many real-world situations where expensive clock 
discipline is needed. But IT isn't, I don't think, one of them, with the 
exception of private SONET networks (fast disappearing in the face of metro 
Ethernet).

 -mel beckman

> On May 15, 2016, at 3:22 AM, Eric S. Raymond  wrote:
> 
> Baldur Norddahl :
>> Ok how many hours or days of holdover can you expect from quartz,
>> temperature compensated quartz or Rubidium?
> 
> Sorry, I don't have those drift figures handy.  I'm a programmer, not
> a large-site sysadmin - I've never had purchase authority with a
> budget large enough to buy a rubidium-oscillator GPSDO or any other
> device with holdover.  Better to ask Mel Beckman or someone else
> with field experience.
> 
>>Should we calculate holdover as
>> time until drift is more than 1 millisecond, 10 ms or more for NTP
>> applications?
> 
> If you want to go super-accurate, 1ms.  If you want to go cheap, on
> sampling-theory grounds I'd say you want to vary your drift threshold
> from 1 to 5ms (half the expected precision of WAN time, think of it as
> the Nyquist rate) and look for a knee in the cost curve.
> 
>> I am thinking that many available datacenter locations will have poor GPS
>> signal so we can expect signal loss to be common. Some weather patterns
>> might even cause extended GPS signal loss.
> 
> Weather won't do it, usually. Rain and fog and clouds are transparent
> to GPS frequencies. Standing water directly on an antenna can cause
> some attenuation, but with any serious GPS engine made more recently
> than 5 years ago I would be extremely surprised if that lost it
> lock.  The newer ones handle down to 30 feet in ocean water on robot
> submarines.
> -- 
>http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Bruce Simpson :
> On 13/05/16 20:39, Eric S. Raymond wrote:
> >In 2012, nearly three years before being recruited for NTPsec, I
> >solved this problem as part of my work on GPSD.  The key to this
> >solution is an obscure feature of USB, and a one-wire
> >patch to the bog-standard design for generic USB that exploits
> >it.  Technical details on request, but what it comes down to is
> >that with this one weird trick(!) you can mass-produce primary time
> >sources with a jitter bounded by the USB polling interval for
> >about $20 a pop.
> >
> >The USB 1 polling interval is 1ms.
> 
> What about USB 3.1 (assuming the device is not intended to be backwards
> compatible with the polling model) ? I should point out Intel intend to
> retire EHCI/UHCI and implement only xHCI.

Nobody makes GPSes with even USB 2 or 3 yet, and it is unlikely to happen
for a long time.  Cost reasons - USB GPSes are cheap consumer-grade hardware
and the manufacturers care about fractions of a cent on the BOM. 
-- 
http://www.catb.org/~esr/;>Eric S. Raymond

Re: Cost-effectivenesss of highly-accurate clocks for NTP

[Eric S. Raymond]

Baldur Norddahl :
> Ok how many hours or days of holdover can you expect from quartz,
> temperature compensated quartz or Rubidium?

Sorry, I don't have those drift figures handy.  I'm a programmer, not
a large-site sysadmin - I've never had purchase authority with a
budget large enough to buy a rubidium-oscillator GPSDO or any other
device with holdover.  Better to ask Mel Beckman or someone else
with field experience.

> Should we calculate holdover as
> time until drift is more than 1 millisecond, 10 ms or more for NTP
> applications?

If you want to go super-accurate, 1ms.  If you want to go cheap, on
sampling-theory grounds I'd say you want to vary your drift threshold
from 1 to 5ms (half the expected precision of WAN time, think of it as
the Nyquist rate) and look for a knee in the cost curve.

> I am thinking that many available datacenter locations will have poor GPS
> signal so we can expect signal loss to be common. Some weather patterns
> might even cause extended GPS signal loss.

Weather won't do it, usually. Rain and fog and clouds are transparent
to GPS frequencies. Standing water directly on an antenna can cause
some attenuation, but with any serious GPS engine made more recently
than 5 years ago I would be extremely surprised if that lost it
lock.  The newer ones handle down to 30 feet in ocean water on robot
submarines.
-- 
http://www.catb.org/~esr/;>Eric S. Raymond