subject:"NTP Sync Issue Across Tata $Europe$"

The recommendation tends to be the following:

1) Run your GPS-derived NTP appliances, but DO NOT point end-user clients
at it.
2) Run a set of internal NTPd servers, and configure them to pull time from
all of your GPS-derived NTP servers, AND trusted public NTP servers
3) Point your clients at the internal NTPd servers.

Note that it's not entirely unreasonable to go out and buy numerous GPS
appliances, deploy them at multiple locations, and point your NTPd servers
at those.   With enough sites, your NTPd server will skip over any
defective NTP appliance.At some point, using publicly available NTP
sources is redundant unless one wants to mitigate away the risks behind
failure of the GPS system itself.

What I'm advocating against is the seemingly common practice to go buy an
off-the-shelf lower-cost GPS-NTP appliance (under $1K or so), stick an
antenna in a window or maybe on the rooftop, and point all your devices at
that device.This is asking for a failure for reasons I've covered
previously.   Robust time needs multiple time sources in order to mitigate
against any one source being spoofed or jammed.   The more time sources you
incorporate into your solution, the less likely it is that any one of them
going haywire would cause a timing failure.

On Wed, Aug 9, 2023 at 6:12 PM Mel Beckman  wrote:

> Seth,
>
> My point exactly. Use GPS as primary, and have anti-PS back up, and if you
> want automatic fail over, do that in an intermediate server on your site
> that makes a conscious test and decision to fail over to regular NTP
>
> -mel via cell
>
> > On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG 
> wrote:
> >
> > On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
> >> Note that NIST operates a pool of 24 time servers for public use.
> These are spread across four different locations in two different states.
> My understanding is that they all get their time directly from the official
> NIST clocks without GPS or NTP being involved.
> >
> > I used to jump through all the hoops for that but honestly I like the
> appliances better (they are also PTP grandmaster clocks). I can always
> disable the GPS inputs if any of the doom and gloom actually comes to pass.
> >
> > ~Seth
>

-- 
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

ll the extant spoofing only
> tampers with the ephemeris (satellite position) data, not the timing
> stream. That's because hackers have been aiming at navigation, and may not
> have expressed interest in GPS tampering when NTP tampering is so easy 
>
> To spoof GPS clocks, a hacker has to know where the antennas are, and get
> above them in order to inject a signal with the right directionality.
> Commercial GPS clock vendors have implemented various anti-spoofing
> measures that, for example, only accept signals from a certain cone of
> visibility, which faces up. They have other measures too, some of which
> exploit geographic diversity, so if  you can have two or more GPS clocks in
> different hub sites, the clocks will reject spoofing signals.
>
> This seems like a much easier defense than deploying secure NTP (NTS),
> which adds a huge amount of complexity. At least one researcher has shown
> that poluting the existing public NTP pool with enough bogus servers to
> seriously impact network time is trivial (I cited their paper in an earlier
> post on this thread).  A well funded state actor could be laying the
> framework for such an attack as we speak, lying in wait until an
> opportunity to disrupt Internet NTP globally.
>
>-mel
> --
> *From:* NANOG  on behalf of Jay
> Hennigan 
> *Sent:* Wednesday, August 9, 2023 10:58 AM
> *To:* nanog@nanog.org 
> *Subject:* Re: NTP Sync Issue Across Tata (Europe)
>
> On 8/9/23 09:29, Seth Mattinen via NANOG wrote:
>
> > I liked having a WWVB receiver in my mix, but all the hardware
> > appliances (at least those offering OCXO or Rubidium oscillator options)
> > seem to have rejected it in favor of GPS only. I can only conclude that
> > either vendors think options like WWVB are a dead end or there's no
> > demand for GPS alternatives.
>
> Both GPS and WWVB are over-the-air. There has been concern expressed of
> a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or
> spoofing WWVB is a trivial joke.
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Mel Beckman

Seth,

My point exactly. Use GPS as primary, and have anti-PS back up, and if you want 
automatic fail over, do that in an intermediate server on your site that makes 
a conscious test and decision to fail over to regular NTP

-mel via cell

> On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG  wrote:
> 
> On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
>> Note that NIST operates a pool of 24 time servers for public use.These 
>> are spread across four different locations in two different states.  My 
>> understanding is that they all get their time directly from the official 
>> NIST clocks without GPS or NTP being involved.
> 
> I used to jump through all the hoops for that but honestly I like the 
> appliances better (they are also PTP grandmaster clocks). I can always 
> disable the GPS inputs if any of the doom and gloom actually comes to pass.
> 
> ~Seth

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Seth Mattinen via NANOG


On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
Note that NIST operates a pool of 24 time servers for public use.  
  These are spread across four different locations in two different 
states.  My understanding is that they all get their time directly from 
the official NIST clocks without GPS or NTP being involved.




I used to jump through all the hoops for that but honestly I like the 
appliances better (they are also PTP grandmaster clocks). I can always 
disable the GPS inputs if any of the doom and gloom actually comes to pass.


~Seth

Re: NTP Sync Issue Across Tata (Europe)

Note that NIST operates a pool of 24 time servers for public use.   These
are spread across four different locations in two different states.  My
understanding is that they all get their time directly from the official
NIST clocks without GPS or NTP being involved.

You can also request a symmetrical key,  exchanged via paper mail,  for
four of them if you would like to run ntp encryption.

See https://tf.nist.gov/tf-cgi/servers.cgi

You could also add official servers operated by the time labs of other
countries.   A list of many of them are at the end of the pdf at
https://webtai.bipm.org/ftp/pub/tai/annual-reports/bipm-annual-report/TIMESERVICES/timeservices.pdf
 .

On Wed, Aug 9, 2023, 10:30 AM Seth Mattinen via NANOG 
wrote:

> On 8/9/23 2:39 AM, Forrest Christian (List Account) wrote:
> > When GPS is working, time transmission with accuracies of under 1
> > microsecond is common.   This is especially true if the GPS integrates
> > some sort of disciplined oscillator.  Note that this is in excess of
> > what NTPd running on a typical OS can reliably retransmit.
> >
> > BUT..  if I was to choose only one protocol, it would be NTP, not GPS,
> > because of all of the reasons you mention.
> >
> > I find it distressing that sites are relying on GPS only.  I suspect
> > that this a failure to assign proper risk to using GPS.  It's
> > particularly odd when one considers that adding NTP time sources are
> > essentially free and improve robustness and reliability greatly.
> >
>
>
> I liked having a WWVB receiver in my mix, but all the hardware
> appliances (at least those offering OCXO or Rubidium oscillator options)
> seem to have rejected it in favor of GPS only. I can only conclude that
> either vendors think options like WWVB are a dead end or there's no
> demand for GPS alternatives.
>
> Products like the BlueSky GNSS Firewall exist, but not something I've
> thought was as necessary expenditure for my needs (yet). Mouser lists it
> at just under $10k.
>
> Personally I'm just not that comfortable using random unknown platform
> and unknown installation conditions time server pools over the big-I
> internet. I would possibly consider NTP servers operated by entities I
> have peering with.
>
> ~Seth
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Mel Beckman

While GPS spoofing is technically possible, all the extant spoofing only 
tampers with the ephemeris (satellite position) data, not the timing stream. 
That's because hackers have been aiming at navigation, and may not have 
expressed interest in GPS tampering when NTP tampering is so easy 

To spoof GPS clocks, a hacker has to know where the antennas are, and get above 
them in order to inject a signal with the right directionality. Commercial GPS 
clock vendors have implemented various anti-spoofing measures that, for 
example, only accept signals from a certain cone of visibility, which faces up. 
They have other measures too, some of which exploit geographic diversity, so if 
 you can have two or more GPS clocks in different hub sites, the clocks will 
reject spoofing signals.

This seems like a much easier defense than deploying secure NTP (NTS), which 
adds a huge amount of complexity. At least one researcher has shown that 
poluting the existing public NTP pool with enough bogus servers to seriously 
impact network time is trivial (I cited their paper in an earlier post on this 
thread).  A well funded state actor could be laying the framework for such an 
attack as we speak, lying in wait until an opportunity to disrupt Internet NTP 
globally.

   -mel

From: NANOG  on behalf of Jay Hennigan 

Sent: Wednesday, August 9, 2023 10:58 AM
To: nanog@nanog.org 
Subject: Re: NTP Sync Issue Across Tata (Europe)

On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

> I liked having a WWVB receiver in my mix, but all the hardware
> appliances (at least those offering OCXO or Rubidium oscillator options)
> seem to have rejected it in favor of GPS only. I can only conclude that
> either vendors think options like WWVB are a dead end or there's no
> demand for GPS alternatives.

Both GPS and WWVB are over-the-air. There has been concern expressed of
a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or
spoofing WWVB is a trivial joke.

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Chris Adams

Once upon a time, Jay Hennigan  said:
> Both GPS and WWVB are over-the-air. There has been concern expressed
> of a bad actor spoofing or jamming GPS. Comparatively speaking,
> jamming or spoofing WWVB is a trivial joke.

WWVB is not generally useful for precision timing applications, due to
the distance and wave reflections.  Also, from a security point of view,
I have read that it is legal to have your own low-power transmitter on
the WWVB frequency, and there are instructions for doing it with a Pi,
so it would be very cheap and easy to mess with somebody's WWVB signal.
-- 
Chris Adams

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Jay Hennigan


On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

I liked having a WWVB receiver in my mix, but all the hardware 
appliances (at least those offering OCXO or Rubidium oscillator options) 
seem to have rejected it in favor of GPS only. I can only conclude that 
either vendors think options like WWVB are a dead end or there's no 
demand for GPS alternatives.


Both GPS and WWVB are over-the-air. There has been concern expressed of 
a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or 
spoofing WWVB is a trivial joke.


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Seth Mattinen via NANOG


On 8/9/23 2:39 AM, Forrest Christian (List Account) wrote:
When GPS is working, time transmission with accuracies of under 1 
microsecond is common.   This is especially true if the GPS integrates 
some sort of disciplined oscillator.  Note that this is in excess of 
what NTPd running on a typical OS can reliably retransmit.


BUT..  if I was to choose only one protocol, it would be NTP, not GPS, 
because of all of the reasons you mention.


I find it distressing that sites are relying on GPS only.  I suspect 
that this a failure to assign proper risk to using GPS.  It's 
particularly odd when one considers that adding NTP time sources are 
essentially free and improve robustness and reliability greatly.





I liked having a WWVB receiver in my mix, but all the hardware 
appliances (at least those offering OCXO or Rubidium oscillator options) 
seem to have rejected it in favor of GPS only. I can only conclude that 
either vendors think options like WWVB are a dead end or there's no 
demand for GPS alternatives.


Products like the BlueSky GNSS Firewall exist, but not something I've 
thought was as necessary expenditure for my needs (yet). Mouser lists it 
at just under $10k.


Personally I'm just not that comfortable using random unknown platform 
and unknown installation conditions time server pools over the big-I 
internet. I would possibly consider NTP servers operated by entities I 
have peering with.


~Seth

Re: NTP Sync Issue Across Tata (Europe)

2023-08-09 Thread Masataka Ohta


John Gilmore wrote:


 I was also speaking specifically about installing GPS antennas in
 viable places, not using a facility-provided GPS or NTP service.


Am I confused?  Getting the time over a multi-gigabit Internet from a
national time standard agency such as NIST (or your local country's
equivalent) should produce far better accuracy and stability than
relying on locally received GPS signals.


When the (wrong) question is "how to build a stratum 1 server?",
that can not be an answer.


GPS uses very weak radio
signals which are regularly spoofed by all sorts of bad actors:


The question, seemingly, is not "how to build a secure stratum
1 server?".

BTW, the proper question should be "how to obtain secure time?".

Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

When GPS is working, time transmission with accuracies of under 1
microsecond is common.   This is especially true if the GPS integrates some
sort of disciplined oscillator.  Note that this is in excess of what NTPd
running on a typical OS can reliably retransmit.

BUT..  if I was to choose only one protocol, it would be NTP, not GPS,
because of all of the reasons you mention.

I find it distressing that sites are relying on GPS only.  I suspect that
this a failure to assign proper risk to using GPS.  It's particularly odd
when one considers that adding NTP time sources are essentially free and
improve robustness and reliability greatly.

NTP is not without it's risks but the most common server implementation is
specifically designed to be able to discard time sources which are not
telling the truth, provided the server is given enough valid time sources.
Even if a spoofed or misconfigured server is giving the wrong time,  NTPd
will be able to ignore those errant time sources.

 When configured with numerous network time sources and a GPS source,  NTPd
will determine what the correct time should be, and then will use the
higher accuracy GPS source to improve the overall accuracy.  This is more
or less automatic since the latency to the GPS time source will be
essentially zero when compared to a typical network source.

However,  if the GPS source starts lying about the time,  NTPd will start
ignoring it as a potential time source even with the lower latency.
Without having non-GPS sources in your configuration, this essentially free
protection against GPS spoofing is no longer available since it has nothing
to compare it to.

If your network is large enough that you could install multiple GPS
receivers in diverse locations,  then I'd configure all of the NTPd servers
to pull from all of the GPS receivers.  That way you gain additional
redundancy.  I'd still not drop the public trusted NTP servers though.

On Tue, Aug 8, 2023, 2:58 PM John Gilmore  wrote:

> > I was also speaking specifically about installing GPS antennas in
> > viable places, not using a facility-provided GPS or NTP service.
>
> Am I confused?  Getting the time over a multi-gigabit Internet from a
> national time standard agency such as NIST (or your local country's
> equivalent) should produce far better accuracy and stability than
> relying on locally received GPS signals.  GPS uses very weak radio
> signals which are regularly spoofed by all sorts of bad actors:
>
>   https://www.gps.gov/spectrum/jamming/
>
> for all sorts of reasons (like misleading drone navigation):
>
>   https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident
>
> Depending on satnav systems creates a large single point of failure for
> worldwide civilian infrastructure.
>
> Jamming GPS with subtly fake time data near big data centers seems like
> an easy move that would cause all sorts of distributed algorithms to
> start failing in unusual ways.  And in a more serious wartime attack,
> many or most GPS satellites themselves would be destroyed or disabled.
> Yet digital radio modulations like FT8 or DMR rely on tight time
> synchronization among different transmitters.  So do many modern
> cellphone modulations -- not to mention distributed database sync
> algorithms.  Depending on any of these for emergency communications when
> their time comes from GPS, is a recipe for having no communications
> during wars or cyber-wars in which GPS satellites are attacked or
> jammed.  See a longer explanation here:
>
>   https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/
>
> I suspect that even today, if you rely on civilian GPS time near the US
> White House, Pentagon, or other military targets like bases, you will
> discover "anomalies" in the local radio GPS data, compared to what you
> get from an authenticated time standard over NTP.  How reliable is
> civilian GPS time in Ukraine these days?
>
> John
>
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Martin Hannigan


Same here. Latency and the algo work together. Time outs or un-reachable peers 
are network issues.

Paging Randy Bush.

Get Outlook for iOS<https://aka.ms/o0ukef>

From: NANOG  on behalf of John 
Gilmore 
Sent: Tuesday, August 8, 2023 17:01
To: Mel Beckman 
Cc: nanog@nanog.org 
Subject: Re: NTP Sync Issue Across Tata (Europe)

> I was also speaking specifically about installing GPS antennas in
> viable places, not using a facility-provided GPS or NTP service.

Am I confused?  Getting the time over a multi-gigabit Internet from a
national time standard agency such as NIST (or your local country's
equivalent) should produce far better accuracy and stability than
relying on locally received GPS signals.  GPS uses very weak radio
signals which are regularly spoofed by all sorts of bad actors:

  https://www.gps.gov/spectrum/jamming/

for all sorts of reasons (like misleading drone navigation):

  https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident

Depending on satnav systems creates a large single point of failure for
worldwide civilian infrastructure.

Jamming GPS with subtly fake time data near big data centers seems like
an easy move that would cause all sorts of distributed algorithms to
start failing in unusual ways.  And in a more serious wartime attack,
many or most GPS satellites themselves would be destroyed or disabled.
Yet digital radio modulations like FT8 or DMR rely on tight time
synchronization among different transmitters.  So do many modern
cellphone modulations -- not to mention distributed database sync
algorithms.  Depending on any of these for emergency communications when
their time comes from GPS, is a recipe for having no communications
during wars or cyber-wars in which GPS satellites are attacked or
jammed.  See a longer explanation here:

  https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/

I suspect that even today, if you rely on civilian GPS time near the US
White House, Pentagon, or other military targets like bases, you will
discover "anomalies" in the local radio GPS data, compared to what you
get from an authenticated time standard over NTP.  How reliable is
civilian GPS time in Ukraine these days?

John

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread John Gilmore

> I was also speaking specifically about installing GPS antennas in
> viable places, not using a facility-provided GPS or NTP service.

Am I confused?  Getting the time over a multi-gigabit Internet from a
national time standard agency such as NIST (or your local country's
equivalent) should produce far better accuracy and stability than
relying on locally received GPS signals.  GPS uses very weak radio
signals which are regularly spoofed by all sorts of bad actors:

  https://www.gps.gov/spectrum/jamming/

for all sorts of reasons (like misleading drone navigation):

  https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident

Depending on satnav systems creates a large single point of failure for
worldwide civilian infrastructure.

Jamming GPS with subtly fake time data near big data centers seems like
an easy move that would cause all sorts of distributed algorithms to
start failing in unusual ways.  And in a more serious wartime attack,
many or most GPS satellites themselves would be destroyed or disabled.
Yet digital radio modulations like FT8 or DMR rely on tight time
synchronization among different transmitters.  So do many modern
cellphone modulations -- not to mention distributed database sync
algorithms.  Depending on any of these for emergency communications when
their time comes from GPS, is a recipe for having no communications
during wars or cyber-wars in which GPS satellites are attacked or
jammed.  See a longer explanation here:

  https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/

I suspect that even today, if you rely on civilian GPS time near the US
White House, Pentagon, or other military targets like bases, you will
discover "anomalies" in the local radio GPS data, compared to what you
get from an authenticated time standard over NTP.  How reliable is
civilian GPS time in Ukraine these days?

John

Re: NTP Sync Issue Across Tata (Europe)

Go for it. I’m sure NTS’ complexity clocks lots of hours for expensive 
consultants :)

Me, I’m sticking with GPS.)

-mel via cell

On Aug 8, 2023, at 11:34 AM, Rubens Kuhl  wrote:

So little deployment that has 3500 occurrences according to 
shodan.io.  With such few choices, It should be hard to find 
suitable options.

Rubens

Em ter., 8 de ago. de 2023 13:02, Mel Beckman 
mailto:m...@beckman.org>> escreveu:
I’m familiar with NTS, which is pointedly not NTP.  That’s like saying “TCP 
port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely 
deployed it will be an option. As the RFC was only approved in 2020, that will 
probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention 
the complexity or NTS for hardware implementation in network elements, a 
primary application (https://tinyurl.com/ntsishard).

 -mel

> On Aug 8, 2023, at 8:26 AM, Rubens Kuhl 
> mailto:rube...@gmail.com>> wrote:
>
> On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman 
> mailto:m...@beckman.org>> wrote:
>>
>> Until the Internet NTP network can be made secure, no.
>
> Internet NTP can be made secure, it's called NTS.
> https://developers.cloudflare.com/time-services/nts/ describes it with
> links to the RFC, and describes one of the many NTP servers that is
> available with NTS, time.cloudflare.com. I 
> already mentioned 5 others,
> and there are many more than those 6.
>
>
> Rubens

Re: NTP Sync Issue Across Tata (Europe)

Well, an iLEC CO is hardly a data center. Plus you have to be a CLEC to get in.

I’ve built out CLEC spaces in many COs. They invariably connect via fiber to a 
CLEC core data center, where they invariably run their own GPS clocks. Plus 
they run rubidium clocks in the racks at the CO, so they can freewheel for days.

-mel via cell

On Aug 8, 2023, at 11:21 AM, Mike Hammett  wrote:


Frontier COs is the easiest example.No antennas and no time service. Yes, they 
provide BITS, but BITS isn't time.

I was also speaking specifically about installing GPS antennas in viable 
places, not using a facility-provided GPS or NTP service.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:36:46 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

I’d be interested in an example of a Colo that does NOT provide GPS-based NTP 
even if they don’t let tenants install their own. I’ve never, ever seen one.

 -mel

On Aug 8, 2023, at 8:20 AM, Mike Hammett  wrote:


My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities.


Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
________
From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:05:55 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:


"We use these exclusively in data centers"

How well does GPS work inside the datacenter?



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Rubens Kuhl

So little deployment that has 3500 occurrences according to shodan.io.
With such few choices, It should be hard to find suitable options.

Rubens




Em ter., 8 de ago. de 2023 13:02, Mel Beckman  escreveu:

> I’m familiar with NTS, which is pointedly not NTP.  That’s like saying
> “TCP port 80 can be made secure,: just use a VPN!” Perhaps when NTS is
> widely deployed it will be an option. As the RFC was only approved in 2020,
> that will probably take a decade. Or more. (I’m talking about you, IPv6 :)
> Not to mention the complexity or NTS for hardware implementation in network
> elements, a primary application (https://tinyurl.com/ntsishard).
>
>  -mel
>
> > On Aug 8, 2023, at 8:26 AM, Rubens Kuhl  wrote:
> >
> > On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman  wrote:
> >>
> >> Until the Internet NTP network can be made secure, no.
> >
> > Internet NTP can be made secure, it's called NTS.
> > https://developers.cloudflare.com/time-services/nts/ describes it with
> > links to the RFC, and describes one of the many NTP servers that is
> > available with NTS, time.cloudflare.com. I already mentioned 5 others,
> > and there are many more than those 6.
> >
> >
> > Rubens
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Mike Hammett

Frontier COs is the easiest example.No antennas and no time service. Yes, they 
provide BITS, but BITS isn't time. 

I was also speaking specifically about installing GPS antennas in viable 
places, not using a facility-provided GPS or NTP service. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mike Hammett"  
Cc: nanog@nanog.org, "Mark Tinka"  
Sent: Tuesday, August 8, 2023 10:36:46 AM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

I’d be interested in an example of a Colo that does NOT provide GPS-based NTP 
even if they don’t let tenants install their own. I’ve never, ever seen one. 

-mel 

On Aug 8, 2023, at 8:20 AM, Mike Hammett  wrote: 

My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities. 

Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mike Hammett"  
Cc: nanog@nanog.org, "Mark Tinka"  
Sent: Tuesday, August 8, 2023 10:05:55 AM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing. 

-mel 

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote: 

"We use these exclusively in data centers" 

How well does GPS work inside the datacenter? 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mark Tinka"  
Cc: nanog@nanog.org 
Sent: Saturday, August 5, 2023 2:26:37 PM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

Mark, 

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack 
surface. 

-mel via cell 

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote: 
> 
> 
> 
>> On 8/5/23 20:17, Chris Adams wrote: 
>> 
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just 
>> a vendored entry into the pool (more for load tracking and management). 
> 
> Yes, Andreas clarified in unicast. Will do. Thanks. 
> 
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

I’m familiar with NTS, which is pointedly not NTP.  That’s like saying “TCP 
port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely 
deployed it will be an option. As the RFC was only approved in 2020, that will 
probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention 
the complexity or NTS for hardware implementation in network elements, a 
primary application (https://tinyurl.com/ntsishard).

 -mel 

> On Aug 8, 2023, at 8:26 AM, Rubens Kuhl  wrote:
> 
> On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman  wrote:
>> 
>> Until the Internet NTP network can be made secure, no.
> 
> Internet NTP can be made secure, it's called NTS.
> https://developers.cloudflare.com/time-services/nts/ describes it with
> links to the RFC, and describes one of the many NTP servers that is
> available with NTS, time.cloudflare.com. I already mentioned 5 others,
> and there are many more than those 6.
> 
> 
> Rubens

Re: NTP Sync Issue Across Tata (Europe)

I’d be interested in an example of a Colo that does NOT provide GPS-based NTP 
even if they don’t let tenants install their own. I’ve never, ever seen one.

 -mel

On Aug 8, 2023, at 8:20 AM, Mike Hammett  wrote:

My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities.

Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky.

-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:05:55 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:

"We use these exclusively in data centers"

How well does GPS work inside the datacenter?

-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mark Tinka" 
Cc: nanog@nanog.org
Sent: Saturday, August 5, 2023 2:26:37 PM
Subject: Re: NTP Sync Issue Across Tata (Europe)

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
>
> 
>
>> On 8/5/23 20:17, Chris Adams wrote:
>>
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
>
> Yes, Andreas clarified in unicast. Will do. Thanks.
>
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Rubens Kuhl

On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman  wrote:
>
> Until the Internet NTP network can be made secure, no.

Internet NTP can be made secure, it's called NTS.
https://developers.cloudflare.com/time-services/nts/ describes it with
links to the RFC, and describes one of the many NTP servers that is
available with NTS, time.cloudflare.com. I already mentioned 5 others,
and there are many more than those 6.

Rubens

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Mike Hammett

My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities. 

Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mike Hammett"  
Cc: nanog@nanog.org, "Mark Tinka"  
Sent: Tuesday, August 8, 2023 10:05:55 AM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing. 

-mel 

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote: 

"We use these exclusively in data centers" 

How well does GPS work inside the datacenter? 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mark Tinka"  
Cc: nanog@nanog.org 
Sent: Saturday, August 5, 2023 2:26:37 PM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

Mark, 

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack 
surface. 

-mel via cell 

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote: 
> 
> 
> 
>> On 8/5/23 20:17, Chris Adams wrote: 
>> 
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just 
>> a vendored entry into the pool (more for load tracking and management). 
> 
> Yes, Andreas clarified in unicast. Will do. Thanks. 
> 
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I’d like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical — but conceivable — meteor storm). But that would be a fall-back. I 
would not mix the systems.

 -mel

> On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
> wrote:
> 
> Mel Beckman wrote:-
> 
>> It's a problem that has received a lot of attention in both NTP and
>> aviation navigation circles. What is hard to defend against is total signal
>> suppression via high powered jamming. But that you can do with a
>> geographically diverse GPS NTP network.
> 
> Whilst looking forward to being corrected, GPS (even across multiple
> locations) seems to be a SINGLE source of time.  You seem (have I
> misunderstood?) to be a proponent of using GPS exclusively as the external
> clock source.
> 
> Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
> together with carefully selected Internet-based NTP servers?
> 
> I recall an incident over here in Jersey (the one they named New Jersey
> after!) where our primary telco had a substantial time shift on one of
> their two GPS synced servers.  This managed to adjust the clock on enough
> of their routers that the certificate-based OSPF authentication considered
> the certificates invalid, and caused a failure of almost their whole
> network.
> 
> This is, of course, not to say that GPS is not a very good clock source,
> but rather to wonder whether more diversity would be preferable than using
> it as a single source.
> 
> --
> Best wishes,
> Matthew
> 
> --
>> From: Mel Beckman 
>> To: "Forrest Christian (List Account)" 
>> Cc: Nanog 
>> Date: Mon, 7 Aug 2023 14:03:30 +
>> Subject: Re: NTP Sync Issue Across Tata (Europe)
> 
>> Forrest,
>> 
>> GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but 
>> commercial industrial NTP servers have specific anti-spoofing mitigations. 
>> There are also antenna diversity strategies that vendors support to ensure 
>> the signal being relied upon is coming from the right direction. It's a 
>> problem that has received a lot of attention in both NTP and aviation 
>> navigation circles. What is hard to defend against is total signal 
>> suppression via high powered jamming. But that you can do with a 
>> geographically diverse GPS NTP network.
>> 
>> -mel
>> 
>> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) 
>>  wrote:
>> 
>> ?
>> The problem with relying exclusively on GPS to do time distribution is the 
>> ease with which one can spoof the GPS signals.
>> 
>> With a budget of around $1K, not including a laptop, anyone with decent 
>> technical skills could convince a typical GPS receiver it was at any 
>> position and was at any time in the world.   All it takes is a decent 
>> directional antenna, some SDR hardware, and depending on the location and 
>> directivity of your antenna maybe a smallish amplifier.   There is much 
>> discussion right now in the PNT (Position, Navigation and Timing) community 
>> as to how best to secure the GNSS network, but right now one should consider 
>> the data from GPS to be no more trustworthy than some random NTP server on 
>> the internet.
>> 
>> In order to build a resilient NTP server infrastructure you need multiple 
>> sources of time distributed by multiple methods - typically both via 
>> satellite (GPS) and by terrestrial (NTP) methods.   NTP does a pretty good 
>> job of sorting out multiple time servers and discarding sources that are 
>> lying.  But to do this you need multiple time sources.  A common 
>> recommendation is to run a couple/few NTP servers which only get time from a 
>> GPS receiver and only serve time to a second tier of servers that pull from 
>> both those in-house GPS-timed-NTP servers and other trusted NTP servers.   
>> I'd recommend selecting the time servers to gain geographic diversity, i.e. 
>> poll NIST servers in Maryland and Colorado, and possibly both.
>> 
>> Note that NIST will exchange (via mail) a set of keys with you to talk 
>> encrypted NTP with you.   See 
>> https://www.nist.gov/pml/time-and-frequency-division

Re: NTP Sync Issue Across Tata (Europe)

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:

"We use these exclusively in data centers"

How well does GPS work inside the datacenter?

-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mark Tinka" 
Cc: nanog@nanog.org
Sent: Saturday, August 5, 2023 2:26:37 PM
Subject: Re: NTP Sync Issue Across Tata (Europe)

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
>
> 
>
>> On 8/5/23 20:17, Chris Adams wrote:
>>
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
>
> Yes, Andreas clarified in unicast. Will do. Thanks.
>
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Mike Hammett

"We use these exclusively in data centers" 

How well does GPS work inside the datacenter? 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mark Tinka"  
Cc: nanog@nanog.org 
Sent: Saturday, August 5, 2023 2:26:37 PM 
Subject: Re: NTP Sync Issue Across Tata (Europe) 

Mark, 

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack 
surface. 

-mel via cell 

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote: 
> 
> 
> 
>> On 8/5/23 20:17, Chris Adams wrote: 
>> 
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just 
>> a vendored entry into the pool (more for load tracking and management). 
> 
> Yes, Andreas clarified in unicast. Will do. Thanks. 
> 
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Matthew Richardson via NANOG

Mel Beckman wrote:-

>Its a problem that has received a lot of attention in both NTP and
>aviation navigation circles. What is hard to defend against is total signal
>suppression via high powered jamming. But that you can do with a
>geographically diverse GPS NTP network.

Whilst looking forward to being corrected, GPS (even across multiple
locations) seems to be a SINGLE source of time.  You seem (have I
misunderstood?) to be a proponent of using GPS exclusively as the external
clock source.

Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
together with carefully selected Internet-based NTP servers?

I recall an incident over here in Jersey (the one they named New Jersey
after!) where our primary telco had a substantial time shift on one of
their two GPS synced servers.  This managed to adjust the clock on enough
of their routers that the certificate-based OSPF authentication considered
the certificates invalid, and caused a failure of almost their whole
network.

This is, of course, not to say that GPS is not a very good clock source,
but rather to wonder whether more diversity would be preferable than using
it as a single source.

--
Best wishes,
Matthew

 --
>From: Mel Beckman 
>To: "Forrest Christian (List Account)" 
>Cc: Nanog 
>Date: Mon, 7 Aug 2023 14:03:30 +
>Subject: Re: NTP Sync Issue Across Tata (Europe)

>Forrest,
>
>GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but 
>commercial industrial NTP servers have specific anti-spoofing mitigations. 
>There are also antenna diversity strategies that vendors support to ensure the 
>signal being relied upon is coming from the right direction. Its a problem 
>that has received a lot of attention in both NTP and aviation navigation 
>circles. What is hard to defend against is total signal suppression via high 
>powered jamming. But that you can do with a geographically diverse GPS NTP 
>network.
>
> -mel
>
>On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) 
> wrote:
>
>?
>The problem with relying exclusively on GPS to do time distribution is the 
>ease with which one can spoof the GPS signals.
>
>With a budget of around $1K, not including a laptop, anyone with decent 
>technical skills could convince a typical GPS receiver it was at any position 
>and was at any time in the world.   All it takes is a decent directional 
>antenna, some SDR hardware, and depending on the location and directivity of 
>your antenna maybe a smallish amplifier.   There is much discussion right now 
>in the PNT (Position, Navigation and Timing) community as to how best to 
>secure the GNSS network, but right now one should consider the data from GPS 
>to be no more trustworthy than some random NTP server on the internet.
>
>In order to build a resilient NTP server infrastructure you need multiple 
>sources of time distributed by multiple methods - typically both via satellite 
>(GPS) and by terrestrial (NTP) methods.   NTP does a pretty good job of 
>sorting out multiple time servers and discarding sources that are lying.  But 
>to do this you need multiple time sources.  A common recommendation is to run 
>a couple/few NTP servers which only get time from a GPS receiver and only 
>serve time to a second tier of servers that pull from both those in-house 
>GPS-timed-NTP servers and other trusted NTP servers.   I'd recommend selecting 
>the time servers to gain geographic diversity, i.e. poll NIST servers in 
>Maryland and Colorado, and possibly both.
>
>Note that NIST will exchange (via mail) a set of keys with you to talk 
>encrypted NTP with you.   See 
>https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
> .
>
>
>
>On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman 
>mailto:m...@beckman.org>> wrote:
>GPS Selective Availability did not disrupt the timing chain of GPS, only the 
>ephemeris (position information).  But a government-disrupted timebase 
>scenario has never occurred, while hackers are a documented threat.
>
>DNS has DNSSec, which while not deployed as broadly as we might like, at least 
>lets us know which servers we can trust.
>
>Your own atomic clocks still have to be synced to a common standard to be 
>useful. To what are they syncd? GPS, Ill wager.
>
>I sense hand-waving :)
>
>-mel via cell
>
>On Aug 6, 2023, at 7:04 PM, Rubens Kuhl 
>mailto:rube...@gmail.com>> wrote:
>
>?
>
>
>On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman 
>mailto:m...@beckman.org>> wrote:
>Or one can read recent research papers that thoroughly document the incredible 
>fragility of the existing NTP hierarchy and soberly consider their 
>recommendations for remediation:
>
>The paper suggests the compromi

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Masataka Ohta


Forrest Christian (List Account) wrote:


Depends on how synchronized you need to be.


Sure. But, we should be assuming NTP is mostly enough.


A rubidium oscillator or Chip Scale Atomic Clock is in the price range you
quote.  However, these can drift enough that you should occasionally
synchronize with a reference time source.  This is to ensure continued
millisecond accuracy.  Of course it all depends on how much drift you'll
tolerate, and if you're OK with being within a second, then a rubidium
might be ok.


For millisecond accuracy, Rb clocks do not need any synchronization
for centuries.

Rb clocks on GPS are a lot more frequently synchronized, because
a lot more accuracy is required for positioning (10ns of timing
error means 3m of positioning error).

Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Masataka Ohta

Mel Beckman wrote:

> To be useful, any atomic clocks you operate must be synchronized
> to a Stratum Zero time source, such as GPS.

Only initially.

Precise time is crucial to a variety of economic activities around
the world. Communication systems, electrical power grids, and
financial networks all rely on precision timing for synchronization
and operational efficiency. The free availability of GPS time has
enabled cost savings for companies that depend on precise time and
has led to significant advances in capability.

FYI, time difference between two points is not noticeable, that
is, does not affect correctness of any distributed algorithm,
if the difference is below the communication delay between
the points, which means rough synchronization by NTP is
good enough.

That is an information theoretic version of relativity of simultaneity:

https://en.wikipedia.org/wiki/Relativity_of_simultaneity

For information theoretic simultaneity, you can consider, instead
of light cone, information cone.

Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

2023-08-08 Thread Forrest Christian (List Account)

Depends on how synchronized you need to be.

In the context of running airgapped:

 A rubidium oscillator or Chip Scale Atomic Clock is in the price range you
quote.  However, these can drift enough that you should occasionally
synchronize with a reference time source.  This is to ensure continued
millisecond accuracy.  Of course it all depends on how much drift you'll
tolerate, and if you're OK with being within a second, then a rubidium
might be ok.

Caesium oscillators which have much lower drift are in the $30K-50K range.
These would require significantly less frequent synchronization, but are
definitely not a few thousand dollars.

Note that these are both just oscillators and they need additional support
hardware to be able to be queried by NTP.  Or stated differently,  they
still need a NTP server.  Yes, there are products out there which integrate
everything in one box at an additional cost.

On Mon, Aug 7, 2023, 11:02 PM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Forrest Christian (List Account) wrote:
>
> > In the middle tends to be a more moderate solution which involves a mix
> of
> > time transmission methods from a variety of geographically and/or network
> > diverse sources.  Taking time from the public trusted ntp servers and
> > adding lower cost GPS receivers at diverse points in your network seems
> > like a good compromise in the middle.  That way,  only coordinated
> attacks
> > will be successful.
>
> Instead, just rely on atomic clocks operated by you. They are not
> so expensive (several thousand dollars) and should be accurate
> enough without adjustment for hundreds of years. There can be no
> coordinated attacks. They may be remotely accessed through
> secured NTP.
>
> Masataka Ohta
>

Re: NTP Sync Issue Across Tata (Europe)

Masataka,

To be useful, any atomic clocks you operate must be synchronized to a Stratum 
Zero time source, such as GPS. Such clocks are useful when you need exceptional 
accuracy, such as for Building Integrated Timing Service (BITS), but unless 
they’re synchronized you can’t coordinate time-sensitive activities such as 
digital certificate validation with anyone else on the Internet.

From 
https://www.gps.gov/applications/timing/


Each GPS satellite contains multiple atomic clocks that contribute very precise 
time data to the GPS signals. GPS receivers decode these signals, effectively 
synchronizing each receiver to the atomic clocks. This enables users to 
determine the time to within 100 billionths of a second, without the cost of 
owning and operating atomic clocks.

Precise time is crucial to a variety of economic activities around the world. 
Communication systems, electrical power grids, and financial networks all rely 
on precision timing for synchronization and operational efficiency. The free 
availability of GPS time has enabled cost savings for companies that depend on 
precise time and has led to significant advances in capability.

-mel via cell

On Aug 7, 2023, at 10:04 PM, Masataka Ohta  
wrote:

Forrest Christian (List Account) wrote:

In the middle tends to be a more moderate solution which involves a mix of
time transmission methods from a variety of geographically and/or network
diverse sources.  Taking time from the public trusted ntp servers and
adding lower cost GPS receivers at diverse points in your network seems
like a good compromise in the middle.  That way,  only coordinated attacks
will be successful.

Instead, just rely on atomic clocks operated by you. They are not
so expensive (several thousand dollars) and should be accurate
enough without adjustment for hundreds of years. There can be no
coordinated attacks. They may be remotely accessed through
secured NTP.

   Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Masataka Ohta


Forrest Christian (List Account) wrote:


In the middle tends to be a more moderate solution which involves a mix of
time transmission methods from a variety of geographically and/or network
diverse sources.  Taking time from the public trusted ntp servers and
adding lower cost GPS receivers at diverse points in your network seems
like a good compromise in the middle.  That way,  only coordinated attacks
will be successful.


Instead, just rely on atomic clocks operated by you. They are not
so expensive (several thousand dollars) and should be accurate
enough without adjustment for hundreds of years. There can be no
coordinated attacks. They may be remotely accessed through
secured NTP.

Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Forrest Christian (List Account)

Those particular boxes are not cheap.   (Yes I know the units you talk
about).  Note that some of them rely on terrestrial communication of
ephermis data to validate the GPS data to further make the time more
robust.

I was hopefully trying to dispel the seemingly common thread in this
discussion that somehow a $400 GPS derived ntp clock is more robust than
using the public NTP infrastructure.  Without understanding that GPS
receivers are also subject to similar attacks as NTP, one comes to the
wrong conclusion about how to build a robust timing infrastructure.   GPS
is subject to DoS and spoofing attacks just like NTP, and as such, it's
important to mitigate those as well if you care about time stability.

In the end, it's all about risk vs cost to mitigate risk.If you're
running a tiny network, deriving your time from the public NTP servers is
fine.

 At the other end of the scale is using exotic hardened solutions which
derive their own clock source and can hold over time for days or weeks in
the event of a failure or attack.

In the middle tends to be a more moderate solution which involves a mix of
time transmission methods from a variety of geographically and/or network
diverse sources.  Taking time from the public trusted ntp servers and
adding lower cost GPS receivers at diverse points in your network seems
like a good compromise in the middle.  That way,  only coordinated attacks
will be successful.


On Mon, Aug 7, 2023, 8:03 AM Mel Beckman  wrote:

> Forrest,
>
> GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but
> commercial industrial NTP servers have specific anti-spoofing mitigations.
> There are also antenna diversity strategies that vendors support to ensure
> the signal being relied upon is coming from the right direction. It’s a
> problem that has received a lot of attention in both NTP and aviation
> navigation circles. What is hard to defend against is total signal
> suppression via high powered jamming. But that you can do with a
> geographically diverse GPS NTP network.
>
>  -mel
>
> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <
> li...@packetflux.com> wrote:
>
> 
> The problem with relying exclusively on GPS to do time distribution is the
> ease with which one can spoof the GPS signals.
>
> With a budget of around $1K, not including a laptop, anyone with decent
> technical skills could convince a typical GPS receiver it was at any
> position and was at any time in the world.   All it takes is a decent
> directional antenna, some SDR hardware, and depending on the location and
> directivity of your antenna maybe a smallish amplifier.   There is much
> discussion right now in the PNT (Position, Navigation and Timing) community
> as to how best to secure the GNSS network, but right now one should
> consider the data from GPS to be no more trustworthy than some random NTP
> server on the internet.
>
> In order to build a resilient NTP server infrastructure you need multiple
> sources of time distributed by multiple methods - typically both via
> satellite (GPS) and by terrestrial (NTP) methods.   NTP does a pretty good
> job of sorting out multiple time servers and discarding sources that are
> lying.  But to do this you need multiple time sources.  A common
> recommendation is to run a couple/few NTP servers which only get time from
> a GPS receiver and only serve time to a second tier of servers that pull
> from both those in-house GPS-timed-NTP servers and other trusted NTP
> servers.   I'd recommend selecting the time servers to gain geographic
> diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly
> both.
>
> Note that NIST will exchange (via mail) a set of keys with you to talk
> encrypted NTP with you.   See
> https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
> .
>
>
>
> On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman  wrote:
>
>> GPS Selective Availability did not disrupt the timing chain of GPS, only
>> the ephemeris (position information).  But a government-disrupted timebase
>> scenario has never occurred, while hackers are a documented threat.
>>
>> DNS has DNSSec, which while not deployed as broadly as we might like, at
>> least lets us know which servers we can trust.
>>
>> Your own atomic clocks still have to be synced to a common standard to be
>> useful. To what are they sync’d? GPS, I’ll wager.
>>
>> I sense hand-waving :)
>>
>> -mel via cell
>>
>> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl  wrote:
>>
>> 
>>
>>
>> On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:
>>
>>> Or one can read recent research papers that thoroughly document the
>>> incredible fragility of the existing NTP hierarchy and soberly consider
>>> their recommendations for remediation:
>>>
>>
>> The paper suggests the compromise of critical infrastructure. So, besides
>> not using NTP, why not stop using DNS ? Just populate a hosts file with all
>> you need.
>>
>> BTW, the stratum-0 source you

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Mel Beckman

Forrest,

GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but
commercial industrial NTP servers have specific anti-spoofing mitigations.
There are also antenna diversity strategies that vendors support to ensure the
signal being relied upon is coming from the right direction. It’s a problem
that has received a lot of attention in both NTP and aviation navigation
circles. What is hard to defend against is total signal suppression via high
powered jamming. But that you can do with a geographically diverse GPS NTP
network.

-mel

On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account)
wrote:

The problem with relying exclusively on GPS to do time distribution is the ease
with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent
technical skills could convince a typical GPS receiver it was at any position
and was at any time in the world. All it takes is a decent directional
antenna, some SDR hardware, and depending on the location and directivity of
your antenna maybe a smallish amplifier. There is much discussion right now
in the PNT (Position, Navigation and Timing) community as to how best to secure
the GNSS network, but right now one should consider the data from GPS to be no
more trustworthy than some random NTP server on the internet.

In order to build a resilient NTP server infrastructure you need multiple
sources of time distributed by multiple methods - typically both via satellite
(GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting
out multiple time servers and discarding sources that are lying. But to do
this you need multiple time sources. A common recommendation is to run a
couple/few NTP servers which only get time from a GPS receiver and only serve
time to a second tier of servers that pull from both those in-house
GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting
the time servers to gain geographic diversity, i.e. poll NIST servers in
Maryland and Colorado, and possibly both.

Note that NIST will exchange (via mail) a set of keys with you to talk
encrypted NTP with you. See
https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
.

On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman
mailto:m...@beckman.org>> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the
ephemeris (position information). But a government-disrupted timebase scenario
has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least
lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be
useful. To what are they sync’d? GPS, I’ll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl
mailto:rube...@gmail.com>> wrote:

On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman
mailto:m...@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible
fragility of the existing NTP hierarchy and soberly consider their
recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not
using NTP, why not stop using DNS ? Just populate a hosts file with all you
need.

BTW, the stratum-0 source you suggested is known to have been manipulated in
the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to
bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can
keep using GPS as well. If GPS goes bananas on timing, that source will just be
disregarded (one of the features of the NTP architecture that has been pointed
out over and over in this thread and you keep ignoring it).

Rubens

--
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Rubens Kuhl

The paper suggests the compromise of critical infrastructure. So, besides
not using NTP, why not stop using DNS ? Just populate a hosts file with all
you need.

BTW, the stratum-0 source you suggested is known to have been manipulated
in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
can keep using GPS as well. If GPS goes bananas on timing, that source will
just be disregarded (one of the features of the NTP architecture that has
been pointed out over and over in this thread and you keep ignoring it).

Rubens



Rubens



On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:

> Or one can read recent research papers that thoroughly document the
> incredible fragility of the existing NTP hierarchy and soberly consider
> their recommendations for remediation:
>
> [image: preview.png]
>
> ndss2021_1A-2_24302_paper
> 
> PDF Document · 1.7 MB
> 
>
> 
>
> Or simply use non-Internet NTP servers based on a Stratum-0 GPS source for
> mission-critical network timing.
>
> Until then, we may all wake up one morning and discover massive data
> breaches traced to an unfounded reliance on insecure public NTP servers.
> Then the game truly will be over, but not in our favor.
>
>  -mel
>
> On Aug 6, 2023, at 2:35 PM, Rubens Kuhl  wrote:
>
> Or one can select NTS-capable NTP servers, like those 5:
> a.st1.ntp.br
> b.st1.ntp.br
> c.st1.ntp.br
> d.st1.ntp.br
> gps.ntp.br
>
> Or any other NTP server that has NTS deployed. Game-over for NTP
> impersonation.
>
>
> Rubens
>
> On Sun, Aug 6, 2023 at 4:41 PM Mel Beckman  wrote:
>
>
> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
>
>
>
> -mel
>
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams 
> wrote:
>
>
> 
>
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
>
> Royce
>
>
> On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:
>
>
> William,
>
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These
> flaws make it trivial to spoof NTP packets, and many firewalls have no
> specific protection against this. in one attack the malefactor simply fires
> a continuous stream of NTP packets with invalid time at your firewall. When
> your NTP client queries the spoofed server, the malicious packet is the one
> you likely receive.
>
>
> That’s just one attack vector. There are several others, and all have
> complex remediation. Why should people bother being exposed to the risk at
> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
> already described. Having suffered through such attacks more than once, I
> can say from personal experience that you don’t want to risk it.
>
>
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Dorn Hetzel via NANOG

Diversity from GPS might also be obtained by setting one receiver for GPS
and another for Galileo.  I think I'd skip GLONASS for now :)


On Mon, Aug 7, 2023, 06:09 Rubens Kuhl  wrote:

> > > The paper suggests the compromise of critical infrastructure. So,
> besides not using NTP, why not stop using DNS ? Just populate a hosts file
> with all you need.
> >
> > Well DNS can be cryptographically secured.  There really isn’t any good
> reasons to not sign your zones today.  The majority of responses from
> authoritative servers are validated today so if you sign the responses will
> be checked.  Unfortunately most to those validations still result in
> insecure instead of secure because people are not signing their zones.
>
> So does NTP, with NTS.
>
> https://datatracker.ietf.org/doc/html/rfc8915
>
>
> Rubens
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Rubens Kuhl

> > The paper suggests the compromise of critical infrastructure. So, besides 
> > not using NTP, why not stop using DNS ? Just populate a hosts file with all 
> > you need.
>
> Well DNS can be cryptographically secured.  There really isn’t any good 
> reasons to not sign your zones today.  The majority of responses from 
> authoritative servers are validated today so if you sign the responses will 
> be checked.  Unfortunately most to those validations still result in insecure 
> instead of secure because people are not signing their zones.

So does NTP, with NTS.

https://datatracker.ietf.org/doc/html/rfc8915


Rubens

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Rubens Kuhl

On Sun, Aug 6, 2023 at 11:36 PM Mel Beckman  wrote:
>
> GPS Selective Availability did not disrupt the timing chain of GPS, only the 
> ephemeris (position information).  But a government-disrupted timebase 
> scenario has never occurred, while hackers are a documented threat.
>
> DNS has DNSSec, which while not deployed as broadly as we might like, at 
> least lets us know which servers we can trust.

NTP has NTS, which all 5 servers I mentioned have, and many more also have.

> Your own atomic clocks still have to be synced to a common standard to be 
> useful. To what are they sync’d? GPS, I’ll wager.

Nope, they are synced to the reference atomic clock of National
Observatory, pretty similar to USNO but local.

Rubens

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Giovane C. M. Moura via NANOG






So the Anycast address our devices use internally to find the closest
 NTP server is geo-mapped to MU. 


So indeed, the pool will only send you a single NTP server in this case.
GeoDNS essentially map  you to mu.pool.ntp.org.

You can verify what NTP servers you can expect from the Pool by querying 
it directly (and thus bypassing GeoDNS mappings)


$ dig mu.pool.ntp.org


mu.pool.ntp.org.62  IN  A   197.224.66.40


However, the physical server is 
geo-mapped to the specific countries in Europe, e.g., GB, NL, FR, DE,


What really matters from GeoDNS is the IP address of your client -- the 
one that goes in the NTP query. So if you are using your anycast address 
to query, it does not matter what are the unicast addresses of your servers.



Unless the geo data ntp.org are using is inconsistent, I'd imagine
the servers should be mapped to a European pool, since the physical
address from which the server queries the pool is geo-mapped locally,
for this specific reason.


They also use the latest Maxmind mappings, and I confirmed it 
experimentally. ( I think it's fully automated their update method)


/giovane

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Mark Tinka





On 8/7/23 11:04, Giovane C. M. Moura via NANOG wrote:

TL;DR: I'd guess your NTP Server IP address is geolocated to 
Mauritius. The Mauritius zone[0] on the pool has only one server, so 
you'll only see this one. To fix it, use europe.pool.ntp.org (_do not_ 
use pool.ntp.org).


So the Anycast address our devices use internally to find the closest 
NTP server is geo-mapped to MU. However, the physical server is 
geo-mapped to the specific countries in Europe, e.g., GB, NL, FR, DE, e.t.c.


Unless the geo data ntp.org are using is inconsistent, I'd imagine the 
servers should be mapped to a European pool, since the physical address 
from which the server queries the pool is geo-mapped locally, for this 
specific reason.


Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Giovane C. M. Moura via NANOG


Hi Mark,



I have NTP servers in Europe that are choosing Tata (6453) to get to
 0.freebsd.pool.ntp.org which lives on 197.224.66.40:


NTP is not sync'ing to that address, and sessions stay in an Init
state.
TL;DR: I'd guess your NTP Server IP address is geolocated to Mauritius. 
The Mauritius zone[0] on the pool has only one server, so you'll only 
see this one. To fix it, use europe.pool.ntp.org (_do not_ use 
pool.ntp.org).



Longer answer:

NTP pool folks use GeoDNS[1], which is their DNS server to map clients 
to servers.


The `0.freebsd.pool.ntp.org` name is just an alias for them -- what they 
really do is this:


 * Get geolocation_data(client_IP_address): 
 * check country subzone in NTP pool (e.g, nl.pool.ntp.org [2]):
   * If there are >=1  servers in the zone, return (up to) 4 or them
   * If there is one, then return just one (this is a _known issue_)
   * if there is none, then fall back to the continent zone (Europe[3])

I've seen the same issue before with Guernsey clients (only one server). 
We have contact the pool operators and they are working now on a new 
GeoDNS version to prevent this from happening [4]


More details in [5].

In short, change your ntp configuration; the issue you have is that 
despite having 4k servers on the Pool, this strict GeoDNS mapping 
prevents you from accessing the other servers just bc of your IP 
address. The reasoning is to prevent asymmetric routing [4], but they 
are working on a fix to prevent these scenarios.



/giovane

[0] https://www.ntppool.org/zone/mu
[1] https://github.com/abh/geodns
[2] https://www.ntppool.org/zone/nl
[3] https://www.ntppool.org/zone/europe
[4] https://community.ntppool.org/t/minor-new-features-on-the-website/2947/8
[5] 
https://www.sidnlabs.nl/downloads/5aPx86UtFmvKs6WE3LHwbU/c6acce6a012fe07256bab8caefff54af/Diving_into_the_NTP_Pool.pdf

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Forrest Christian (List Account)

I  forgot to finish my thought in the third paragraph before hitting send.

What I was going to express was that one should choose not only close,
trusted, NTP servers, but also perhaps ones from different government
agencies, or different sources.   Sourcing time from multiple sources not
likely to be affected by the same outage and/or attack is good robust
design.

On Mon, Aug 7, 2023 at 2:39 AM Forrest Christian (List Account) <
li...@packetflux.com> wrote:

> The problem with relying exclusively on GPS to do time distribution is the
> ease with which one can spoof the GPS signals.
>
> With a budget of around $1K, not including a laptop, anyone with decent
> technical skills could convince a typical GPS receiver it was at any
> position and was at any time in the world.   All it takes is a decent
> directional antenna, some SDR hardware, and depending on the location and
> directivity of your antenna maybe a smallish amplifier.   There is much
> discussion right now in the PNT (Position, Navigation and Timing) community
> as to how best to secure the GNSS network, but right now one should
> consider the data from GPS to be no more trustworthy than some random NTP
> server on the internet.
>
> In order to build a resilient NTP server infrastructure you need multiple
> sources of time distributed by multiple methods - typically both via
> satellite (GPS) and by terrestrial (NTP) methods.   NTP does a pretty good
> job of sorting out multiple time servers and discarding sources that are
> lying.  But to do this you need multiple time sources.  A common
> recommendation is to run a couple/few NTP servers which only get time from
> a GPS receiver and only serve time to a second tier of servers that pull
> from both those in-house GPS-timed-NTP servers and other trusted NTP
> servers.   I'd recommend selecting the time servers to gain geographic
> diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly
> both.
>
> Note that NIST will exchange (via mail) a set of keys with you to talk
> encrypted NTP with you.   See
> https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
> .
>
>
>
> On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman  wrote:
>
>> GPS Selective Availability did not disrupt the timing chain of GPS, only
>> the ephemeris (position information).  But a government-disrupted timebase
>> scenario has never occurred, while hackers are a documented threat.
>>
>> DNS has DNSSec, which while not deployed as broadly as we might like, at
>> least lets us know which servers we can trust.
>>
>> Your own atomic clocks still have to be synced to a common standard to be
>> useful. To what are they sync’d? GPS, I’ll wager.
>>
>> I sense hand-waving :)
>>
>> -mel via cell
>>
>> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl  wrote:
>>
>> 
>>
>>
>> On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:
>>
>>> Or one can read recent research papers that thoroughly document the
>>> incredible fragility of the existing NTP hierarchy and soberly consider
>>> their recommendations for remediation:
>>>
>>
>> The paper suggests the compromise of critical infrastructure. So, besides
>> not using NTP, why not stop using DNS ? Just populate a hosts file with all
>> you need.
>>
>> BTW, the stratum-0 source you suggested is known to have been manipulated
>> in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
>> need to bet on that specific state actor not returning to old habits.
>>
>> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
>> can keep using GPS as well. If GPS goes bananas on timing, that source will
>> just be disregarded (one of the features of the NTP architecture that has
>> been pointed out over and over in this thread and you keep ignoring it).
>>
>> Rubens
>>
>>
>
> --
> - Forrest
>


-- 
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Mark Tinka





On 8/5/23 21:26, Mel Beckman wrote:


Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.


To be honest, we have decent resiliency across the network to weather 
issues such as these. While going the GPS route is not a bad idea, I 
have more pressing problems to fix right now.


All that said, the issue seems to have corrected itself re: the Tata 
path (even if it's still the same path).


Appreciate all the feedback and support. Thanks.

Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-07 Thread Forrest Christian (List Account)

The problem with relying exclusively on GPS to do time distribution is the
ease with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent
technical skills could convince a typical GPS receiver it was at any
position and was at any time in the world.   All it takes is a decent
directional antenna, some SDR hardware, and depending on the location and
directivity of your antenna maybe a smallish amplifier.   There is much
discussion right now in the PNT (Position, Navigation and Timing) community
as to how best to secure the GNSS network, but right now one should
consider the data from GPS to be no more trustworthy than some random NTP
server on the internet.

In order to build a resilient NTP server infrastructure you need multiple
sources of time distributed by multiple methods - typically both via
satellite (GPS) and by terrestrial (NTP) methods.   NTP does a pretty good
job of sorting out multiple time servers and discarding sources that are
lying.  But to do this you need multiple time sources.  A common
recommendation is to run a couple/few NTP servers which only get time from
a GPS receiver and only serve time to a second tier of servers that pull
from both those in-house GPS-timed-NTP servers and other trusted NTP
servers.   I'd recommend selecting the time servers to gain geographic
diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly
both.

Note that NIST will exchange (via mail) a set of keys with you to talk
encrypted NTP with you.   See
https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
.

On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman  wrote:

> GPS Selective Availability did not disrupt the timing chain of GPS, only
> the ephemeris (position information).  But a government-disrupted timebase
> scenario has never occurred, while hackers are a documented threat.
>
> DNS has DNSSec, which while not deployed as broadly as we might like, at
> least lets us know which servers we can trust.
>
> Your own atomic clocks still have to be synced to a common standard to be
> useful. To what are they sync’d? GPS, I’ll wager.
>
> I sense hand-waving :)
>
> -mel via cell
>
> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl  wrote:
>
> 
>
>
> On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:
>
>> Or one can read recent research papers that thoroughly document the
>> incredible fragility of the existing NTP hierarchy and soberly consider
>> their recommendations for remediation:
>>
>
> The paper suggests the compromise of critical infrastructure. So, besides
> not using NTP, why not stop using DNS ? Just populate a hosts file with all
> you need.
>
> BTW, the stratum-0 source you suggested is known to have been manipulated
> in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
> need to bet on that specific state actor not returning to old habits.
>
> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
> can keep using GPS as well. If GPS goes bananas on timing, that source will
> just be disregarded (one of the features of the NTP architecture that has
> been pointed out over and over in this thread and you keep ignoring it).
>
> Rubens
>
>

-- 
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

GPS Selective Availability did not disrupt the timing chain of GPS, only the 
ephemeris (position information).  But a government-disrupted timebase scenario 
has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least 
lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be 
useful. To what are they sync’d? GPS, I’ll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl  wrote:




On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible 
fragility of the existing NTP hierarchy and soberly consider their 
recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not 
using NTP, why not stop using DNS ? Just populate a hosts file with all you 
need.

BTW, the stratum-0 source you suggested is known to have been manipulated in 
the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to 
bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can 
keep using GPS as well. If GPS goes bananas on timing, that source will just be 
disregarded (one of the features of the NTP architecture that has been pointed 
out over and over in this thread and you keep ignoring it).

Rubens

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Mark Andrews




> On 7 Aug 2023, at 12:02, Rubens Kuhl  wrote:
> 
> 
> 
> On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:
> Or one can read recent research papers that thoroughly document the 
> incredible fragility of the existing NTP hierarchy and soberly consider their 
> recommendations for remediation:
> 
> The paper suggests the compromise of critical infrastructure. So, besides not 
> using NTP, why not stop using DNS ? Just populate a hosts file with all you 
> need. 

Well DNS can be cryptographically secured.  There really isn’t any good reasons 
to not sign your zones today.  The majority of responses from authoritative 
servers are validated today so if you sign the responses will be checked.  
Unfortunately most to those validations still result in insecure instead of 
secure because people are not signing their zones.

> BTW, the stratum-0 source you suggested is known to have been manipulated in 
> the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to 
> bet on that specific state actor not returning to old habits. 
> 
> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can 
> keep using GPS as well. If GPS goes bananas on timing, that source will just 
> be disregarded (one of the features of the NTP architecture that has been 
> pointed out over and over in this thread and you keep ignoring it). 
> 
> Rubens 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Rubens Kuhl

On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman  wrote:

> Or one can read recent research papers that thoroughly document the
> incredible fragility of the existing NTP hierarchy and soberly consider
> their recommendations for remediation:
>

The paper suggests the compromise of critical infrastructure. So, besides
not using NTP, why not stop using DNS ? Just populate a hosts file with all
you need.

BTW, the stratum-0 source you suggested is known to have been manipulated
in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
can keep using GPS as well. If GPS goes bananas on timing, that source will
just be disregarded (one of the features of the NTP architecture that has
been pointed out over and over in this thread and you keep ignoring it).

Rubens

Re: NTP Sync Issue Across Tata (Europe)

Bill,

You’re mistaking targeted NTP attacks with global ones. Yes, to attack your 
specific NTP client, the attacker has to know which NTP servers you’re using. 
But to simply succeed at random attacks, the attacker need only spoof popular 
servers. This is how time-shifting attacks work. Once an attack succeeds with a 
random victim, the hacker goes to work compromising time-sensitive security 
services.

And don’t forget that anyone can join pool.ntp.org. Including hackers. There is 
no credentialed vetting process.

 -mel

> On Aug 6, 2023, at 4:30 PM, William Herrin  wrote:
> 
> On Sun, Aug 6, 2023 at 1:19 PM Royce Williams  .com> wrote:
>> Wouldn't a robust implementation of peering - say, seven peers, with the NTP 
>> algorithm handily selecting a subset to peer with for each cycle - require 
>> an attacker to know and overwhelm not just one, but a majority of the peer 
>> IPs?
> 
> Hi Royce,
> 
> More or less, yes. There are some corner cases where that may not be
> true, but in terms of designing a useful attack, the attacker has to
> be able to figure out which NTP servers you're talking to, flood you
> with enough packets forged from all of them that the forged packet
> arrives ahead of the real one, generally no more than 10s of
> milliseconds behind the sent packet that only happens once every 15
> minutes or so. And then it has to move you off real time gradually
> enough for the NTP daemon not to decide the peer has become a false
> ticker.
> 
> It's like DNS cache poisoning but a few orders of magnitude harder.
> 
> 
>> This is *not* to say that I'm advocating against maintaining local stratum 
>> 0s as part of a proper operator-grade NTP mesh. (My definition of "robust 
>> implementation" would include both local stratum 0 and a healthy serving of 
>> Internet stratum 1s). I'm just suggesting "don't peer with public servers" 
>> seems draconian given the deliberate design choices of the protocol.
>> 
>> For this audience, this would recommend a tiered system where multiple 
>> mixed-stratum internal stratrum 0+1s would peer with each other, and 
>> maintain internal-facing consensus for all other downstream internal devices 
>> - such that the vast majority of internal peers would indeed not be talking 
>> to the public Internet (but the "parent" peers would, and have the mesh and 
>> mix that I describe).
> 
> 
> Well, it's not really one-size-fits-all.
> 
> Some systems consist of a well defined network containing routers and
> servers with clear boundaries between self and Internet. Your approach
> would work well there.
> 
> Some systems consist of equipment scattered at various data centers,
> interconnected by the Internet itself. Implementing a GPS time
> receiver at every one could get very expensive very fast. Standard
> security rules apply: don't spend more protecting yourself than the
> risk-cost. Risk is threat times vulnerability. Given the complexity of
> the attack, you have to consider whether you're enough of a target
> (threat) for anyone to bother.
> 
> Some systems are heavily virtualized, scattered over many vendors and
> locations. You could -try- to track down a "secured" NTP source from
> the vendor at each location, but that way lies configuration insanity.
> 
> Some systems are air-gapped from the Internet. You're going to get
> time from GPS or the cellular phone network. GPS devices like the one
> Mel pointed out are probably cheaper and more accurate.
> 
> Regards,
> Bill Herrin
> 
> 
> -- 
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Fwd: NTP Sync Issue Across Tata (Europe)

Or one can read recent research papers that thoroughly document the incredible 
fragility of the existing NTP hierarchy and soberly consider their 
recommendations for remediation:

https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf

Or simply use non-Internet NTP servers based on a Stratum-0 GPS source for 
mission-critical network timing.

Until then, we may all wake up one morning and discover massive data breaches 
traced to an unfounded reliance on insecure public NTP servers. Then the game 
truly will be over, but not in our favor.

 -mel

On Aug 6, 2023, at 2:35 PM, Rubens Kuhl  wrote:

Or one can select NTS-capable NTP servers, like those 5:
a.st1.ntp.br
b.st1.ntp.br
c.st1.ntp.br
d.st1.ntp.br
gps.ntp.br

Or any other NTP server that has NTS deployed. Game-over for NTP impersonation.


Rubens

On Sun, Aug 6, 2023 at 4:41 PM Mel Beckman  wrote:

In a nutshell, no. Refer to my prior cites for detailed explanations. For a 
list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#


-mel

On Aug 6, 2023, at 12:03 PM, Royce Williams  wrote:


Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a 
reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread William Herrin

On Sun, Aug 6, 2023 at 1:19 PM Royce Williams  wrote:
> Wouldn't a robust implementation of peering - say, seven peers, with the NTP 
> algorithm handily selecting a subset to peer with for each cycle - require an 
> attacker to know and overwhelm not just one, but a majority of the peer IPs?

Hi Royce,

More or less, yes. There are some corner cases where that may not be
true, but in terms of designing a useful attack, the attacker has to
be able to figure out which NTP servers you're talking to, flood you
with enough packets forged from all of them that the forged packet
arrives ahead of the real one, generally no more than 10s of
milliseconds behind the sent packet that only happens once every 15
minutes or so. And then it has to move you off real time gradually
enough for the NTP daemon not to decide the peer has become a false
ticker.

It's like DNS cache poisoning but a few orders of magnitude harder.

> This is *not* to say that I'm advocating against maintaining local stratum 0s 
> as part of a proper operator-grade NTP mesh. (My definition of "robust 
> implementation" would include both local stratum 0 and a healthy serving of 
> Internet stratum 1s). I'm just suggesting "don't peer with public servers" 
> seems draconian given the deliberate design choices of the protocol.
>
> For this audience, this would recommend a tiered system where multiple 
> mixed-stratum internal stratrum 0+1s would peer with each other, and maintain 
> internal-facing consensus for all other downstream internal devices - such 
> that the vast majority of internal peers would indeed not be talking to the 
> public Internet (but the "parent" peers would, and have the mesh and mix that 
> I describe).

Well, it's not really one-size-fits-all.

Some systems consist of a well defined network containing routers and
servers with clear boundaries between self and Internet. Your approach
would work well there.

Some systems consist of equipment scattered at various data centers,
interconnected by the Internet itself. Implementing a GPS time
receiver at every one could get very expensive very fast. Standard
security rules apply: don't spend more protecting yourself than the
risk-cost. Risk is threat times vulnerability. Given the complexity of
the attack, you have to consider whether you're enough of a target
(threat) for anyone to bother.

Some systems are heavily virtualized, scattered over many vendors and
locations. You could -try- to track down a "secured" NTP source from
the vendor at each location, but that way lies configuration insanity.

Some systems are air-gapped from the Internet. You're going to get
time from GPS or the cellular phone network. GPS devices like the one
Mel pointed out are probably cheaper and more accurate.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Rubens Kuhl

Or one can select NTS-capable NTP servers, like those 5:
a.st1.ntp.br
b.st1.ntp.br
c.st1.ntp.br
d.st1.ntp.br
gps.ntp.br

Or any other NTP server that has NTS deployed. Game-over for NTP impersonation.


Rubens

On Sun, Aug 6, 2023 at 4:41 PM Mel Beckman  wrote:
>
> In a nutshell, no. Refer to my prior cites for detailed explanations. For a 
> list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
>
>
>  -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams  wrote:
>
> 
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a 
> reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:
>>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
>> flaws make it trivial to spoof NTP packets, and many firewalls have no 
>> specific protection against this. in one attack the malefactor simply fires 
>> a continuous stream of NTP packets with invalid time at your firewall. When 
>> your NTP client queries the spoofed server, the malicious packet is the one 
>> you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have 
>> complex remediation. Why should people bother being exposed to the risk at 
>> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve 
>> already described. Having suffered through such attacks more than once, I 
>> can say from personal experience that you don’t want to risk it.
>>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Neil Hanlon

This entirely discounts the fact that bcp-38 and bcp-84 which, more or
less, eliminate this "problem space" entirely.

I find it hard to believe ntp reflection is actually a problem in the year
2023, assuming you're not running a ridiculously old ntp client and have
taken really simple steps to protect your network.

On Sun, Aug 6, 2023, 15:42 Mel Beckman  wrote:

> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
> 
>
>
>  -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams 
> wrote:
>
> 
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:
>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough.
>> These flaws make it trivial to spoof NTP packets, and many firewalls have
>> no specific protection against this. in one attack the malefactor simply
>> fires a continuous stream of NTP packets with invalid time at your
>> firewall. When your NTP client queries the spoofed server, the malicious
>> packet is the one you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have
>> complex remediation. Why should people bother being exposed to the risk at
>> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
>> already described. Having suffered through such attacks more than once, I
>> can say from personal experience that you don’t want to risk it.
>>
>>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Royce Williams

Respectfully, that Wikipedia article (which is mostly about legit but
unauthorized clients overwhelming a given peer) and your cites don't seem
to cover what I was responding to - the "don't peer with public NTP because
someone can flood your firewall and spoof the responses" problem. I just
want to make sure that I'm understanding the distinction betwen this class
and other classes of attack.

Wouldn't a robust implementation of peering - say, seven peers, with the
NTP algorithm handily selecting a subset to peer with for each cycle -
require an attacker to know and overwhelm not just one, but a majority of
the peer IPs?

This is *not* to say that I'm advocating against maintaining local stratum
0s as part of a proper operator-grade NTP mesh. (My definition of "robust
implementation" would include both local stratum 0 and a healthy serving of
Internet stratum 1s). I'm just suggesting "don't peer with public servers"
seems draconian given the deliberate design choices of the protocol.

For this audience, this would recommend a tiered system where multiple
mixed-stratum internal stratrum 0+1s would peer with each other, and
maintain internal-facing consensus for all other downstream internal
devices - such that the vast majority of internal peers would indeed not be
talking to the public Internet (but the "parent" peers would, and have the
mesh and mix that I describe).

And I'm well aware of who I'm saying this to ... so I expect to find out
where I'm wrong, as I keep digging. :D

-- 
Royce

On Sun, Aug 6, 2023 at 11:40 AM Mel Beckman  wrote:

> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
> 
>
>
>  -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams 
> wrote:
>
> 
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:
>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough.
>> These flaws make it trivial to spoof NTP packets, and many firewalls have
>> no specific protection against this. in one attack the malefactor simply
>> fires a continuous stream of NTP packets with invalid time at your
>> firewall. When your NTP client queries the spoofed server, the malicious
>> packet is the one you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have
>> complex remediation. Why should people bother being exposed to the risk at
>> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
>> already described. Having suffered through such attacks more than once, I
>> can say from personal experience that you don’t want to risk it.
>>
>>

Re: NTP Sync Issue Across Tata (Europe)

In a nutshell, no. Refer to my prior cites for detailed explanations. For a 
list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#


 -mel

On Aug 6, 2023, at 12:03 PM, Royce Williams  wrote:


Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a 
reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread James R Cutler

A carefully selected set of stratum 0 sources for a set of stratum 1 servers is 
the heart of good NTP source design. With at least four “local” stratum 1 
servers, Dr. Mills algorithm is excellent at distinguishing truechimers from 
falsetickers and providing a reliable source of monotonic time. DOS is a 
separate problem.

My NTP network deployment experience for a major auto manufacturer, among 
others, is in agreement with William Herin. A GPS NTP source is a valid Stratum 
0 source, but relying on a single instance for local time is not exceedingly 
better than querying time.apple.com  or a similar 
source.
-
James R. Cutler 
> 
> William,
> 
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
> flaws make it trivial to spoof NTP packets, and many firewalls have no 
> specific protection against this. in one attack the malefactor simply fires a 
> continuous stream of NTP packets with invalid time at your firewall. When 
> your NTP client queries the spoofed server, the malicious packet is the one 
> you likely receive.
> 
> That’s just one attack vector. There are several others, and all have complex 
> remediation. Why should people bother being exposed to the risk at all? 
> Simply avoid Internet-routed NTP. there are many solutions, as I’ve already 
> described. Having suffered through such attacks more than once, I can say 
> from personal experience that you don’t want to risk it.
> 
> -mel 
> 
>> On Aug 6, 2023, at 10:53 AM, William Herrin  wrote:
>> 
>> On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman  wrote:
>>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring 
>>> is worthless if you suffer, for example, an NTP DDoS attack.
>> 
>> Hi Mel,
>> 
>> From what I can tell, a fairly simple firewall policy of allow UDP 123
>> from known NTP clients and established connections (I sent them a UDP
>> packet recently) stops every one of those attacks (that's actually an
>> NTP attack and not something else like a DNS attack) except for
>> upstream address hijack that happens to coincide with your system
>> boot. And it still depends on the attacker executing an additional
>> sophisticated attack to do more than cause you a denial of service.
>> 
>> The links you sent are very interesting, at least in an academic
>> sense, but they don't cause me to be unduly concerned about employing
>> NTP.
>> 
>> 
>>> if you can eliminate such security problems for $400, I say it’s cheap at 
>>> twice the price.
>> 
>> Except you can't. Redundancy is required for any critical service. At
>> the $400 price point, your approach has multiple
>> single-points-of-failure. The device itself of course. Your ability to
>> receive continuous non-jammed GPS signals at the location where you're
>> able to place an antenna. And in your plan you'll need one of these in
>> every discontiguous network where you have equipment since you're not
>> doing NTP over the Internet.
>> 
>> Not to mention the operations cost. Keeping track of a six inch brick
>> with a wall wart and an antenna installed at a remote site is... not
>> entirely abnormal but it's a one-off that consumes manpower.
>> 
>> And then you're only vulnerable to the litany of Internet attacks
>> which don't involve NTP. Yay!
>> 
>> Don't get me wrong: the Time Machines TM1000A you recommended looks
>> like a cool little device well worth checking into. As a supplement to
>> Internet NTP, not a replacement.
>> 
>> Regards,
>> Bill Herrin
>> 
>> 
>> --
>> William Herrin
>> b...@herrin.us
>> https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Royce Williams

Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering
a reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:

> William,
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These
> flaws make it trivial to spoof NTP packets, and many firewalls have no
> specific protection against this. in one attack the malefactor simply fires
> a continuous stream of NTP packets with invalid time at your firewall. When
> your NTP client queries the spoofed server, the malicious packet is the one
> you likely receive.
>
> That’s just one attack vector. There are several others, and all have
> complex remediation. Why should people bother being exposed to the risk at
> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
> already described. Having suffered through such attacks more than once, I
> can say from personal experience that you don’t want to risk it.
>
>

Re: NTP Sync Issue Across Tata (Europe)

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

 -mel 

> On Aug 6, 2023, at 10:53 AM, William Herrin  wrote:
> 
> On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman  wrote:
>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring 
>> is worthless if you suffer, for example, an NTP DDoS attack.
> 
> Hi Mel,
> 
> From what I can tell, a fairly simple firewall policy of allow UDP 123
> from known NTP clients and established connections (I sent them a UDP
> packet recently) stops every one of those attacks (that's actually an
> NTP attack and not something else like a DNS attack) except for
> upstream address hijack that happens to coincide with your system
> boot. And it still depends on the attacker executing an additional
> sophisticated attack to do more than cause you a denial of service.
> 
> The links you sent are very interesting, at least in an academic
> sense, but they don't cause me to be unduly concerned about employing
> NTP.
> 
> 
>> if you can eliminate such security problems for $400, I say it’s cheap at 
>> twice the price.
> 
> Except you can't. Redundancy is required for any critical service. At
> the $400 price point, your approach has multiple
> single-points-of-failure. The device itself of course. Your ability to
> receive continuous non-jammed GPS signals at the location where you're
> able to place an antenna. And in your plan you'll need one of these in
> every discontiguous network where you have equipment since you're not
> doing NTP over the Internet.
> 
> Not to mention the operations cost. Keeping track of a six inch brick
> with a wall wart and an antenna installed at a remote site is... not
> entirely abnormal but it's a one-off that consumes manpower.
> 
> And then you're only vulnerable to the litany of Internet attacks
> which don't involve NTP. Yay!
> 
> Don't get me wrong: the Time Machines TM1000A you recommended looks
> like a cool little device well worth checking into. As a supplement to
> Internet NTP, not a replacement.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread William Herrin

On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman  wrote:
> That still leaves you open to NTP attacks. The USNO accuracy and monitoring 
> is worthless if you suffer, for example, an NTP DDoS attack.

Hi Mel,

>From what I can tell, a fairly simple firewall policy of allow UDP 123
from known NTP clients and established connections (I sent them a UDP
packet recently) stops every one of those attacks (that's actually an
NTP attack and not something else like a DNS attack) except for
upstream address hijack that happens to coincide with your system
boot. And it still depends on the attacker executing an additional
sophisticated attack to do more than cause you a denial of service.

The links you sent are very interesting, at least in an academic
sense, but they don't cause me to be unduly concerned about employing
NTP.

> if you can eliminate such security problems for $400, I say it’s cheap at 
> twice the price.

Except you can't. Redundancy is required for any critical service. At
the $400 price point, your approach has multiple
single-points-of-failure. The device itself of course. Your ability to
receive continuous non-jammed GPS signals at the location where you're
able to place an antenna. And in your plan you'll need one of these in
every discontiguous network where you have equipment since you're not
doing NTP over the Internet.

Not to mention the operations cost. Keeping track of a six inch brick
with a wall wart and an antenna installed at a remote site is... not
entirely abnormal but it's a one-off that consumes manpower.

And then you're only vulnerable to the litany of Internet attacks
which don't involve NTP. Yay!

Don't get me wrong: the Time Machines TM1000A you recommended looks
like a cool little device well worth checking into. As a supplement to
Internet NTP, not a replacement.

Regards,
Bill Herrin

--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

Niels,

You’re the first person to mention neutral collocation facilities as a 
requirement. The OP only talked about servers generally. Obviously, building 
your own GPS-based NTP network requires you have visibility to the sky. 
However, that need not be rooftop access. We routinely locate GPS antennae for 
time synch at CLEC colos in an available window, behind the glass. Nobody has 
yet charged us anything more than a one-time cabling fee for that.

But most carrier-neutral colos already have their own GPS-derived, non-Internet 
time source to which you can subscribe. There is thus no reason to build your 
own airgapped network, as one is already available from the facility. If you 
don’t need PTP 50-μs-precise timing for SONET and whatnot, ordinary NTP is very 
cost effective. For example:


About Equinix Precision 
Time
docs.equinix.com
[favicon-16x16.png]

Most colos also offer their own Stratum-1 public NTP servers for free. For 
example, Hurricane Electric (https://www.he.net/adm/ntp.html). Being you’re 
already on-net in the colo, that would still give you NTP that doesn’t transit 
the Internet. Still way better than pool.ntp.org!


 -mel

On Aug 6, 2023, at 4:53 AM, Niels Bakker  wrote:

* m...@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:
if you can eliminate such security problems for $400, I say it’s cheap at twice 
the price.

You must be unfamiliar with the prices neutral colocation facilities charge for 
roof access.


   -- Niels.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-06 Thread Niels Bakker


* m...@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:
if you can eliminate such security problems for $400, I say it’s 
cheap at twice the price.


You must be unfamiliar with the prices neutral colocation facilities 
charge for roof access.



-- Niels.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Mel Beckman

Bill,

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is
worthless if you suffer, for example, an NTP DDoS attack.

[ddos-lc.png]
NTP amplification DDoS
attack
cloudflare.com

There are also replay and Man in the middle attacks (MITM) which can corrupt
local NTP servers’ time basis. Worse, security flaws in NTP make others
security protocols, such as SSL, vulnerable.

https://www.sidn.nl/en/news-and-blogs/security-flaws-in-network-time-protocol-make-other-security-protocols-vulnerable

if you can eliminate such security problems for $400, I say it’s cheap at twice
the price.

-mel

On Aug 5, 2023, at 6:18 PM, William Herrin wrote:

On Sat, Aug 5, 2023 at 12:26 PM Mel Beckman wrote:
You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin

--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread William Herrin

On Sat, Aug 5, 2023 at 12:26 PM Mel Beckman  wrote:
> You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin



-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Rubens Kuhl

If the path has simmetric one way latencies you don't have to pick a lower
latency faulty one. Perhaps creating a selection at startup and then using
that collection ?

Rubens

Em sáb., 5 de ago. de 2023 12:42, Mark Tinka  escreveu:

> Hi all.
>
> I have NTP servers in Europe that are choosing Tata (6453) to get to
> 0.freebsd.pool.ntp.org which lives on 197.224.66.40:
>
> traceroute -I 0.freebsd.pool.ntp.org
> traceroute to 0.freebsd.pool.ntp.org (197.224.66.40), 64 hops max, 48
> byte packets
>  1  ae-2-24.er-01-ams.nl.seacomnet.com (105.26.64.13)  0.300 ms  0.301
> ms  0.215 ms
>  2  ce-0-0-11.cr-01-mrs.fr.seacomnet.com (105.16.8.201)  22.163 ms
> 22.370 ms  22.084 ms
>  3  ce-0-0-3.br-01-mrs.fr.seacomnet.com (105.16.32.254)  20.230 ms
> 20.243 ms  20.139 ms
>  4  ix-hge-0-0-0-28.ecore3.emrs2-marseille.as6453.net (80.231.165.52)
> 21.875 ms  21.679 ms  21.762 ms
>  5  * if-be-3-2.ecore2.emrs2-marseille.as6453.net (195.219.175.0)  42.751
> ms *
>  6  if-ae-25-2.tcore1.ldn-london.as6453.net (195.219.175.5)  43.509 ms
> 43.280 ms  43.353 ms
>  7  195.219.83.158 (195.219.83.158)  203.310 ms  203.452 ms  203.209 ms
>  8  196.20.225.84 (196.20.225.84)  208.289 ms  208.637 ms  208.374 ms
>  9  197.226.230.13 (197.226.230.13)  209.657 ms  209.658 ms  209.830 ms
> 10  197.224.66.40 (197.224.66.40)  208.638 ms  208.632 ms  208.712 ms
>
> NTP is not sync'ing to that address, and sessions stay in an Init state.
>
> Other NTP servers I have in Africa are reaching the same address via local
> Anycast hosters, and sync'ing just fine.
>
> Anyone else seeing this particular issue across Tata in Europe?
>
> Mark.
>

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Mel Beckman

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 8/5/23 20:17, Chris Adams wrote:
>> 
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
> 
> Yes, Andreas clarified in unicast. Will do. Thanks.
> 
> Mark.

Re: NTP Sync Issue Across Tata (Europe)





On 8/5/23 20:17, Chris Adams wrote:


It's the NTP pool people you need to talk to - the .freebsd. bit is just
a vendored entry into the pool (more for load tracking and management).


Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Chris Adams

Once upon a time, Mark Tinka  said:
> On 8/5/23 19:51, Andreas Ott wrote:
> >See for yourself how his pool server scores at
> >https://www.ntppool.org/scores/197.224.66.40
> >
> >I am not sure why it would be inserted into DNS answers for a
> >worldwide pool like 0.freebsd as it clearly does have connectivity
> >issues from some of the pool project's own sensors.
> 
> Many thanks, Andreas.
> 
> I'll take this up with the FreeBSD folk.

It's the NTP pool people you need to talk to - the .freebsd. bit is just
a vendored entry into the pool (more for load tracking and management).

-- 
Chris Adams

Re: NTP Sync Issue Across Tata (Europe)





On 8/5/23 19:51, Andreas Ott wrote:

See for yourself how his pool server scores at 
https://www.ntppool.org/scores/197.224.66.40


I am not sure why it would be inserted into DNS answers for a 
worldwide pool like 0.freebsd as it clearly does have connectivity 
issues from some of the pool project's own sensors.


Many thanks, Andreas.

I'll take this up with the FreeBSD folk.

Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Andreas Ott

See for yourself how his pool server scores at
https://www.ntppool.org/scores/197.224.66.40

I am not sure why it would be inserted into DNS answers for a worldwide
pool like 0.freebsd as it clearly does have connectivity issues from some
of the pool project's own sensors.
-andreas

On Sat, Aug 5, 2023 at 9:37 AM Mark Tinka  wrote:

>
> I was speaking about Europe.
>
> There is no problem in Africa.
>
>

Re: NTP Sync Issue Across Tata (Europe)




On 8/5/23 18:30, Matthew McGehrin wrote:


Hi Mark.

Wouldn't a local server be more optimal?

IE:

   server 0.nl.pool.ntp.org
   server 1.nl.pool.ntp.org
   server 2.nl.pool.ntp.org
   server 3.nl.pool.ntp.org


I have a bunch of servers all over Europe I'd prefer not to touch if 
this is just a transient issue.




Or for Africa
server 0.za.pool.ntp.org
   server 1.za.pool.ntp.org
   server 2.za.pool.ntp.org
   server 3.za.pool.ntp.org


I was speaking about Europe.

There is no problem in Africa.

Mark.

Re: NTP Sync Issue Across Tata (Europe)

2023-08-05 Thread Matthew McGehrin


Hi Mark.

Wouldn't a local server be more optimal?

IE:

   server 0.nl.pool.ntp.org
   server 1.nl.pool.ntp.org
   server 2.nl.pool.ntp.org
   server 3.nl.pool.ntp.org

Or for Africa
   server 0.za.pool.ntp.org
   server 1.za.pool.ntp.org
   server 2.za.pool.ntp.org
   server 3.za.pool.ntp.org

Matthew

On 8/5/2023 10:41 AM, Mark Tinka wrote:

Hi all.

I have NTP servers in Europe that are choosing Tata (6453) to get to 
0.freebsd.pool.ntp.org which lives on 197.224.66.40:


traceroute -I 0.freebsd.pool.ntp.org
traceroute to 0.freebsd.pool.ntp.org (197.224.66.40), 64 hops max, 48 
byte packets



Mark.

NTP Sync Issue Across Tata (Europe)