Re: [chrony-users] Measuring clock drift results

[Steven Sommars]

Suggest checking for competing time synchronization software.  E.g., run #
tcpdump -i any -n -w /tmp/port123.pcap port 123
Look for NTP requests to unexpected systems.

I had a system running ntpd + systemd with sporadic spikes.  Turns out
timesyncd would also adjust the system clock.

.


On Thu, Jul 13, 2023 at 3:32 PM Thangalin  wrote:

> Hi all,
>
> Thank you for suggesting we use /sys/class/pps/pps1/assert to measure the
> clock. Attached are graphs from running both chronyd and ntpd overnight.
> The images are also online:
>
> https://ibb.co/album/5YxH2m
>
> Chrony clearly improves the offsets to sub-20 microseconds much of the
> time. We're seeing some spikes in the timing, though, that exceed ntpd.*
> The chrony configuration file is the same as in the other thread (
> https://www.mail-archive.com/chrony-users@chrony.tuxfamily.org/msg03281.html).
> We can't use the GPIO polling driver until I have a clear yes/no answer on
> the licensing issue.
>
> Other processes may be affected by changing the CPU, so we haven't
> switched to a constant frequency nor disabled the power saving states (e.g.
> idle=poll).
>
> Are there any other ways we could help chronyd stay below 20 microseconds
> (e.g., optimized build)?
>
> Many thanks!
>
> *ntpd had a single 1500+ us outlier, which we removed.
>

Re: [chrony-users] Is my GPS receiver really less accurate than NTP servers?

[Steven Sommars]

Network delay asymmetry can also be introduced by local equipment.  The
scatterplot below shows the round-trip delay between an NTP client and
a nearby NTP server.
on my fiber internet service 1) at the local NTP client and 2) just above
my home router.


If the local client and the remote NTP server both have accurate time it
can be interesting to compare the request delay (client -> server) to the
response delay (server -> client).
[Asymmetric delays are common]

This paper  may
be of interest.
[image: image.png]

On Sat, Apr 16, 2022 at 2:23 AM Rob Janssen  wrote:

> When you have a systemic offset of external services vs your local clocks,
> it is often caused by asymmetric network delay.
> E.g. when using VDSL or Cable, it may well be that your uplink and
> downlink rates are different, and also the error correction parameters are
> different.
> In that case all your external servers get offset by some value, and it is
> the GPS time that is right, not those 20 external references.
>
> I have seen that coming and going on my line, over the years.  The telecom
> operators sometimes decide it is better to have reliability than low
> roundtrip time and enable "long interleaving" and similar parameters, often
> only in one of the directions.  But then later they find themselves in a
> contest for low roundtrip times in the gaming world, and they change that
> again.
> For quite some time I had an offset of a few ms in my VDSL line and even
> tweaked the config for it.  But at the moment it is quite symmetric.
>
> But indeed, also check the pulse width of the PPS signal to see if it
> happens to be the same as your offset.  If so, change the PPS edge.
>
> Rob
>
> On 4/16/22 06:24, Dominick C. Pastore wrote:
> >
> > The one thing that does seem a little suspicious is that the offsets for
> all the servers fall pretty uniformly between about 0 and +5 milliseconds.
> That seems a little too coincidental, like maybe 5ms is the granularity of
> the timer used to timestamp the packets or something. But, admittedly, I
> have no other reason to believe the measurements aren't correct, and that
> sounds like an unlikely explanation. I suppose it's possible the PPS from
> the GPS receiver has a constant 2.5ms delay from somewhere, but I'd be a
> little surprised (it's not a timing GPS, but the datasheet still quotes a
> PPS accuracy within 60ns).
>
>
> --
> To unsubscribe email chrony-users-requ...@chrony.tuxfamily.org
> with "unsubscribe" in the subject.
> For help email chrony-users-requ...@chrony.tuxfamily.org
> with "help" in the subject.
> Trouble?  Email listmas...@chrony.tuxfamily.org.
>
>

Re: [chrony-users] Chrony gets out of sync regularly

[Steven Sommars]

> >>> Why do you need the SHM 0? You can use SHM 1 or SOCK.Some cookbooks
(e.g.,
> https://support.ntp.org/bin/view/Support/ConfiguringSHMRefclocks ) use
both  SHM 0 and SHM 1


That might had been necessary with very old versions of gpsd (IIRC it
originally required something to bring the system clock close to the
PPS before it locked), but with more recent gpsd versions the
configuration should be one of the following:

chronyd/ntpd can take PPS directly & determine the name of the second,
or gpsd can intervene and do some of the work.  The options are confusing
and poorly documented.
gpsd altered the picture without warning; fortunately the number of
affected users seems to be small .

I did a git bisect last year: the problem appeared between release-3.21-26
and release-3.21-30.
This is the unit I've been using: https://v3.airspy.us/product/upu-rpigps/
  Ublox MAX-M8Q

Since the gpsd team doesn't see an issue I stopped looking.  If the results
might be useful I could dig deeper (e.g., turn off GPGSV or other
sentences.).


Steve

Re: [chrony-users] Chrony gets out of sync regularly

[Steven Sommars]

Config:  Host sync'd to nearby stratum 1s.  Jitter ~35 usec. gpsd 3.23
gpspipe, gpspipe -R and ntpshmmon running simultaneously for several hours

 gpspipe -T '%s' --usec -r | sed 's|,.*||'
  Min  Max
 (sec)(sec)
.132419 .168017 GPGBS
.132332 .167903 GPGGA
.132393 .167985 GPGSA
.389284 .549959 GPGSV
.132364 .167938 GPRMC
.132263 .167829 GPZDA

gpspipe shows that the data is sent in binary mode.

I'll send some other details off-list.

>>> Why do you need the SHM 0? You can use SHM 1 or SOCK.
Some cookbooks (e.g.,
https://support.ntp.org/bin/view/Support/ConfiguringSHMRefclocks ) use both
SHM 0 and SHM 1


On Wed, Mar 2, 2022 at 3:34 AM Miroslav Lichvar  wrote:

> On Tue, Mar 01, 2022 at 03:09:27PM -0600, Steven Sommars wrote:
> > I understand the jumps to be gpsd-related.I see identical behavior on
> > one of my RPi4+ublox GPS units.
> > NMEA sentence timing remains consistent. As an optimization some gpsd
> > versions move the GPS into ublox binary mode for the serial data stream.
> >  The ublox serial binary mode timing is not consistent.  The gpsd folks
> > apparently believe that the arrival time of serial data is untrustworthy
> > and should not be used.
>
> Can you try the following command and compare the output when the
> offset jumps?
>
> # gpspipe -T '%s' --usec -r | sed 's|,.*||'
>
> For example, with gpsd 3.23.1 and a ublox unit I have here I get:
>
> 1646208470.067774: $GPZDA
> 1646208470.067792: $GPGGA
> 1646208470.067806: $GPRMC
> 1646208470.067811: $GPGSA
> 1646208470.067814: $GPGBS
> 1646208470.082703: $GPGSV
> 1646208470.082721: $GPGSV
> 1646208470.082726: $GPGSV
>
> The timing of the messages in my case is stable to about 20
> milliseconds. In the "super-raw" (gpspipe -R) output I can see it's
> using the binary protocol.
>
> --
> Miroslav Lichvar
>
>
> --
> To unsubscribe email chrony-users-requ...@chrony.tuxfamily.org
> with "unsubscribe" in the subject.
> For help email chrony-users-requ...@chrony.tuxfamily.org
> with "help" in the subject.
> Trouble?  Email listmas...@chrony.tuxfamily.org.
>
>

Re: [chrony-users] Chrony gets out of sync regularly

[Steven Sommars]

Gabriele provided ntpshmmon output which I've graphed below.  The host was
roughly synchronized, then chrony was disabled for the duration of the
test.  NTP1 (pps) drifted slightly (0.7 msec) during the test, but that was
too small to be visible on the scatterplot.
The notable feature is that NTP0 times (green) regularly jump by 180 msec.

Note: I'm not a gpsd expert, some of this may be inaccurate.
I understand the jumps to be gpsd-related.I see identical behavior on
one of my RPi4+ublox GPS units.
NMEA sentence timing remains consistent. As an optimization some gpsd
versions move the GPS into ublox binary mode for the serial data stream.
 The ublox serial binary mode timing is not consistent.  The gpsd folks
apparently believe that the arrival time of serial data is untrustworthy
and should not be used.

I don't understand gpsd internals; version 3.20 didn't show this behavior.

Steve Sommars

[image: image.png]

On Mon, Feb 28, 2022 at 8:35 AM Gabriele Coppi 
wrote:

>
> On Mon, Feb 28, 2022 at 1:12 PM Miroslav Lichvar 
> wrote:
>
>> On Mon, Feb 28, 2022 at 10:37:09AM +0100, Gabriele Coppi wrote:
>> > On Mon, Feb 28, 2022 at 8:46 AM Miroslav Lichvar 
>> > wrote:
>> > > On Fri, Feb 25, 2022 at 12:10:40PM +0100, Gabriele Coppi wrote:
>> > > > refclock SHM 0 precision 1e-1 delay 0.5 refid GPS
>> > > > refclock SHM 1 precision 1e-7 refid PPS
>> > >
>> > > You should remove the first source and just rely on the PPS-based
>> > > SHM 1, or alternatively use SHM 0 + PPS /dev/pps0. SHM 1 from gpsd is
>> > > sufficient.
>> > >
>> >
>> > I am using the SHM1 because the /dev/pps0 it is always reported as a
>> > falseticker. I checked with ppstest and I got out the signal, so I don't
>> > know why
>>
>> As a falseticker against SHM 0, or some NTP server? If the latter, it
>> might be using the wrong edge of the pulse.
>>
>
> Now it seems working also using
>
> refclock PPS /dev/pps0 precision 1e-7 refid PPS
>
> instead of the SHM1 PPS signal. However, I am still getting falseticker
> after ~15min from both PPS and SHM0 sources. I will try to use only the PPS
> now based on what you wrote below
>
>
>>
>> > > If you see both sources marked as falsetickers (x symbol) in the
>> > > chronyc sources output, that means they don't agree with each other.
>> > > If the offset of the SHM 0 is around 0.25s, small changes in the delay
>> > > would cause the behaviour you see.
>> > >
>> >
>> > Yes, the offset of SHM0 is always between 250 and 500ms.
>>
>> That should be compensated with the offset option. But better it is to
>> not use the SHM 0 at all. As you see, its offset is too variable.
>>
>> --
>> Miroslav Lichvar
>>
>>
>> --
>> To unsubscribe email chrony-users-requ...@chrony.tuxfamily.org
>> with "unsubscribe" in the subject.
>> For help email chrony-users-requ...@chrony.tuxfamily.org
>> with "help" in the subject.
>> Trouble?  Email listmas...@chrony.tuxfamily.org.
>>
>>

Re: [chrony-users] Rate limit question

[Steven Sommars]

When was this discussed on the NTP WG list?


I read the man page section several times:

burst:  This option sets the maximum *number* of responses that can be sent
in a burst,

This doesn't mention rate.


On Wed, Sep 22, 2021 at 10:46 AM Miroslav Lichvar 
wrote:

> On Wed, Sep 22, 2021 at 08:09:47AM -0500, Steven Sommars wrote:
> > Running chrony 4.1-pre1.   chrony.conf has a ratelimit directive
> >ratelimit interval 4 burst 4
> > When a large burst (thousands of requests) from one IP arrives, I expect
> at
> > most 4 packets to be sent in one interval (2^4=16 seconds).  Instead I
> see
> > 1 response per 4 requests on average.
> >
> > Where did I go wrong?
>
> That is expected behavior. See the leak option of the ratelimit
> directive in the chrony.conf man page.
>
> chronyd cannot just stop responding to an address if it appears to be
> sending too many requests. We don't know if it is not an attacker
> sending requests with a spoofed source address trying to deny the
> service to a legitimate client. IIRC we have discussed this on the NTP
> WG list.
>
> The purpose of rate limiting is to save some traffic on broken or
> misconfigured clients. If you don't respond at all, you save 50% of
> the incoming+outgoing traffic. If you respond to 1/4 of requests, you
> save only 37.5%, but still allow the client to synchronize in case it
> is under an attack.
>
> --
> Miroslav Lichvar
>
>
> --
> To unsubscribe email chrony-users-requ...@chrony.tuxfamily.org
> with "unsubscribe" in the subject.
> For help email chrony-users-requ...@chrony.tuxfamily.org
> with "help" in the subject.
> Trouble?  Email listmas...@chrony.tuxfamily.org.
>
>

[chrony-users] Rate limit question

[Steven Sommars]

Running chrony 4.1-pre1.   chrony.conf has a ratelimit directive
   ratelimit interval 4 burst 4
When a large burst (thousands of requests) from one IP arrives, I expect at
most 4 packets to be sent in one interval (2^4=16 seconds).  Instead I see
1 response per 4 requests on average.

Where did I go wrong?

Re: [chrony-users] Large ppm clock slew rate

[Steven Sommars]

Thanks for the tick rate adjustment pointer.   ntpd doesn't adjust the tick
value, as far as I know.
The ntp.org NTP distribution does include the tickadj utility, which pokes
/dev/kmem to adjust the tick value.

The context is the International Space Stations (see thread at
http://lists.febo.com/pipermail/time-nuts_lists.febo.com/2021-January/102525.html
)
The externally observed frequency offset gradually increases from -500 to
-1500 ppm with a repeating pattern.   This is bizarre, see the offset graph
<http://lists.febo.com/pipermail/time-nuts_lists.febo.com/attachments/20210107/0dbbb065/attachment.png>
.
If the cause can be determined, it may be correctable with ntpd or perhaps
switching to chrony.

This particular host is running ntpd 4.2.8, so this isn't the right email
list for detailed ntpd discussions.


On Tue, Jan 19, 2021 at 12:06 PM Bill Unruh  wrote:

> If your clock is out that badly then there is something very wrong with
> your
> onboard clock or in the calibration of the clock initially
>
> The linux kernel has two time adjustments, a fine on ( which has the 500PPM
> limit) and a course one( which has the 1/10 sec/sec limit) Chrony uses
> both.
> Ntpd just uses the fine one.
>
> chrony measures the offset of the clock wrt a clock source and runs a
> linear
> fit on the last N offset readings. It adjusts the clock rate to gradually
> bring the rate to 1s/s with respect to standard (ewxternal ntp source(s).
> The linear fit is tested to see if a linear fit is a reasonable fit to the
> offsets. If not, it decreases N, the number of offsets used, until the fit
> is
> reasonable (Not too many differences between the offsets and the linear
> fit)  with a single sign in a row).
> At worst it uses just three offsets to fit to.  It grows N ( by one at a
> time)
> until the linear fit goes bad, or until a maximum value of N is reched (I
> think it is 64).
>
> The documentation for chrony describes a lot of this. Then there is the
> source
> code itself. Because of the linear fit, it can determine the residual
> offset
> and the rate much more quickly and accurately than does ntp, with chrony
> able
> to get sometimes more than a factor of 10 improvement over ntpd.
> ntpd just uses a simple feedback loop.
> (Note that when chrony alters the rate or the offset of the clock, it
> adjusts
> the whole history of offsets it uses to fit by that change in offset and
> rate,
> to keep the past data relevant to the current running of the clock.
> )
>
>
> William G. Unruh __| Canadian Institute for| Tel: +1(604)822-3273
> Physics _|___ Advanced Research _| Fax: +1(604)822-5324
> UBC, Vancouver,BC _|_ Program in Cosmology |____ un...@physics.ubc.ca
> Canada V6T 1Z1 | and Gravity __|_ www.theory.physics.ubc.ca/
>
> On Mon, 18 Jan 2021, Steven Sommars wrote:
>
> >
> > I have a Linux installation which may require clock slew rate > 500 ppm,
> which exceeds normal
> > adjustimex limits.  Chrony lists a 10 ppm maximum slew rate.  How is
> that done?
> >
> > Is there a document that describes chrony's clock discipline algorithm?
> >
> >

[chrony-users] Large ppm clock slew rate

[Steven Sommars]

I have a Linux installation which may require clock slew rate > 500 ppm,
which exceeds normal adjustimex limits.  Chrony lists a 10 ppm maximum
slew rate.  How is that done?

Is there a document that describes chrony's clock discipline algorithm?