Public bug reported:
chrony: NTP sources left permanently offline after transient network
loss (networkd-dispatcher hook runs before the interface is usable; no
configured.d symlink)
Package: chrony 4.8-2ubuntu1
Release: Ubuntu 26.04
Arch: arm64 (Raspberry Pi 3B, no RTC)
Network backend: systemd-networkd via netplan, wired eth0
networkd-dispatcher: installed and active
SUMMARY
=======
On a systemd-networkd host, chrony's NTP sources can be left permanently
offline after a transient network outage. The networkd-dispatcher hook
runs "chronyc onoffline" while the interface has carrier but is not yet
fully configured - in this case with no DHCP-assigned address or
gateway. With no route present, chronyd correctly marks the affected
sources offline. Once configuration completes and a route exists, no
further dispatcher event runs the hook, so the sources are never brought
back online.
chronyd then free-runs indefinitely, logging "Can't synchronise: no
selectable sources", and sets the kernel unsync flag so timedatectl
reports "System clock synchronized: no".
In my case sources stayed offline for 16.5 hours, until I ran "chronyc
online" by hand. The clock itself remained accurate throughout (~145 us
of NTP time, free-running on the learned frequency), so the fault is
silent unless something is gated on NTPSynchronized. Several systemd
units on this host gate an ExecStartPre on exactly that, and all of them
failed for the duration.
STEPS TO REPRODUCE
==================
On a networkd host running chrony, cause carrier loss followed by a slow
DHCP re-acquisition - e.g. power-cycle the upstream router. In my case a
router firmware update caused ~36 s of carrier flapping, and DHCP did
not complete for a further 2m15s.
OBSERVED
========
systemd-networkd (times UTC; LAN addresses redacted):
06:19:10 eth0: Lost carrier
06:19:10 eth0: DHCP lease lost
06:19:13 eth0: Gained carrier
06:19:19 eth0: Lost carrier
06:19:42 eth0: Gained carrier
06:19:44 eth0: Lost carrier
06:19:46 eth0: Gained carrier
06:22:01 eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired
from 192.168.x.1
chronyd:
06:21:36 Source 185.125.190.123 offline
06:21:36 Source 91.189.91.112 offline
06:21:36 Source 2620:2d:4000:1::1123 offline
06:21:36 Source 91.189.91.113 offline
06:21:36 Can't synchronise: no selectable sources (5 unreachable sources)
06:21:36 Source 2620:2d:4000:1::2123 offline
Note the ordering: chronyd marked all five sources offline at 06:21:36,
twenty-five seconds BEFORE the interface acquired an address and gateway
at 06:22:01. Nothing ran the hook again afterwards.
State 16.5 hours later, before any manual intervention:
$ chronyc tracking
Reference ID : 45AE48D5 (ntp-nts-2.ps5.canonical.com)
Ref time (UTC) : Fri Jul 24 06:15:09 2026
System time : 0.000144547 seconds slow of NTP time
Leap status : Normal
$ timedatectl show -p NTPSynchronized --value
no
"chronyc -n sources" at that point showed all five sources as '^?'
(unusable), Reach 0, LastRx 16h. "sudo chronyc online" restored them
immediately, and NTPSynchronized returned to "yes" within one poll.
DELIBERATE REPRODUCTION
=======================
I reproduced this by physically disconnecting and reconnecting the
Ethernet cable, with networkd-dispatcher at increased verbosity (so
invoked hooks are named) and a logging script installed in every state
directory:
#!/bin/sh
logger -t dispatch-debug "STATE=$STATE IFACE=$IFACE
Admin=$AdministrativeState Oper=$OperationalState"
Full sequence (trimmed to the relevant units; addresses redacted):
04:06:46 kernel: eth0: Link is Down
04:06:46 systemd-networkd: eth0: Lost carrier
04:06:46 systemd-networkd: eth0: DHCP lease lost
04:06:46 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for
interface eth0
04:06:46 dispatch-debug: STATE=no-carrier Admin=configured
Oper=no-carrier
04:07:41 kernel: eth0: Link is Up - 100Mbps/Full
04:07:41 systemd-networkd: eth0: Gained carrier
04:07:41 networkd-dispatcher: Invoking '.../degraded.d/00-debug' for
interface eth0
04:07:41 dispatch-debug: STATE=degraded Admin=configured
Oper=degraded
04:07:41 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for
interface eth0
04:07:41 dispatch-debug: STATE=configuring Admin=configuring
Oper=degraded
04:07:42 networkd-dispatcher: Invoking '.../routable.d/00-debug' for
interface eth0
04:07:42 dispatch-debug: STATE=routable Admin=configuring
Oper=routable
04:07:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for
interface eth0
04:07:43 networkd-dispatcher: Invoking '.../configured.d/00-debug' for
interface eth0
04:07:43 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:07:45 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway
192.168.x.1 acquired
04:07:45 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for
interface eth0
04:07:45 dispatch-debug: STATE=configuring Admin=configuring
Oper=routable
04:07:45 networkd-dispatcher: Invoking '.../configured.d/00-debug' for
interface eth0
04:07:45 dispatch-debug: STATE=configured Admin=configured
Oper=routable
This establishes four things:
1. On carrier loss the dispatched state is "no-carrier", not "off".
chrony's off.d symlink is never invoked by a link flap. The only chrony
symlink that executes in this entire sequence is routable.d/chrony-
onoffline.
2. "routable" is dispatched with AdministrativeState=configuring, at
04:07:42 - three seconds before DHCPv4 supplies an address and gateway
at 04:07:45. Note that OperationalState reads "routable" from 04:07:42
onward, including at both "configured" dispatches; what changes across
the sequence is AdministrativeState (configuring -> configured ->
configuring -> configured). The claim is therefore specific: the
dispatch chrony's hook is attached to fires while networkd's
ADMINISTRATIVE state is still "configuring". The interface is
operationally routable at that point via IPv6 - as the IPv4-only offline
set confirms - but IPv4 is not yet configured.
3. chrony's hook runs at exactly that moment, and nothing runs it again.
Because the only hook is on "routable", and "routable" precedes IPv4
configuration, a link flap does not merely risk leaving IPv4 sources
offline - it does so deterministically.
4. "configured" is dispatched at 04:07:45, immediately after the DHCPv4
lease. A configured.d symlink would therefore be invoked with a working
default route present.
Point 2 answers the question raised upstream in 2023 ("Doesn't the
routable state follow configured?" - msg03269): the "routable" dispatch
precedes the post-DHCP "configured" dispatch.
Note there are no chronyd log lines in this window: the IPv4 sources
were already offline from an identical earlier test, so "chronyc
onoffline" produced no state change and logged nothing. The verification
capture further below shows the same routable.d/chrony-onoffline
invocation immediately followed by "Source ... offline" for all three
IPv4 sources, with the two IPv6 sources untouched - onoffline correctly
reading the routing table as IPv6-routable and IPv4-not-yet.
The defect is not in onoffline's logic. It is that the only dispatch
wired to it fires before IPv4 configuration completes, and nothing re-
runs it afterwards.
Caveat on ordering: the sequence is not monotonic - "configuring" is
dispatched again at 04:07:45 after a "configured" at 04:07:43. On a
flapping link states can re-enter. In this capture "configured" is
dispatched last and after DHCP, so the proposed symlink would have
restored the sources; but it does not by itself guarantee the hook
always runs last.
ANALYSIS
========
/usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline guards its
action with "[ $# -ge 2 ]". networkd-dispatcher passes interface state
through environment variables (IFACE, STATE) and not positional
arguments, so that guard is always false and the script unconditionally
runs "chronyc onoffline". That command determines online/offline from
the routing table, so the outcome depends entirely on WHEN it runs
relative to address and route configuration.
debian/links, from the 4.8-2ubuntu1 source package, installs the script
into two networkd-dispatcher state directories only:
# Update sources in response to systemd-networkd events (LP: #1718227).
# This is reusing the NetworkManager dispatch script which has no hard
# dependency to NetworkManager (not using any of its arguments)
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/routable.d/chrony-onoffline
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/off.d/chrony-onoffline
(Identical in current Debian packaging. "dpkg -L chrony" on the running
system confirms these are the only two symlinks shipped.)
The comment makes clear the argument-less fall-through is deliberate and
understood. The gap is only that "configured" is not in the list, so
nothing runs the hook at the point where networkd reports the link fully
configured - i.e. after DHCP has supplied an address and gateway.
VERIFICATION OF THE PROPOSED FIX
================================
I added the missing symlink locally:
/usr/lib/networkd-dispatcher/configured.d/50-chrony-onoffline-local
-> /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
and repeated the cable disconnect/reconnect:
04:25:59 systemd-networkd: eth0: Lost carrier
04:25:59 systemd-networkd: eth0: DHCP lease lost
04:25:59 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for
interface eth0
04:25:59 dispatch-debug: STATE=no-carrier Admin=configured
Oper=no-carrier
04:26:39 systemd-networkd: eth0: Gained carrier
04:26:39 dispatch-debug: STATE=degraded Admin=configured
Oper=degraded
04:26:40 dispatch-debug: STATE=configuring Admin=configuring
Oper=degraded
04:26:42 dispatch-debug: STATE=routable Admin=configuring
Oper=routable
04:26:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for
interface eth0
04:26:42 chronyd: Source 185.125.190.123 offline
04:26:42 chronyd: Source 91.189.91.112 offline
04:26:42 chronyd: Source 91.189.91.113 offline
04:26:42 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:26:42 networkd-dispatcher: Invoking
'.../configured.d/50-chrony-onoffline-local' for interface eth0
04:26:44 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway
192.168.x.1 acquired
04:26:44 dispatch-debug: STATE=configuring Admin=configuring
Oper=routable
04:26:44 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:26:44 networkd-dispatcher: Invoking
'.../configured.d/50-chrony-onoffline-local' for interface eth0
04:26:44 chronyd: Source 185.125.190.123 online
04:26:44 chronyd: Source 91.189.91.112 online
04:26:44 chronyd: Source 91.189.91.113 online
(The local symlink is named 50-chrony-onoffline-local only to avoid
colliding with any future package-shipped file; it is the same script.)
The defect still occurs: routable.d/chrony-onoffline runs at 04:26:42
and parks the three IPv4 sources. The added hook then runs twice - once
at 04:26:42, still before DHCP, where it correctly makes no change; and
again at 04:26:44 immediately after the DHCPv4 lease, where it restores
all three sources.
Exposure drops from indefinite to two seconds, unattended. The harmless
pre-DHCP invocation also demonstrates that running onoffline on an
additional state introduces no churn - it is self-determining and
idempotent.
This run again confirms that carrier loss dispatches "no-carrier", not
"off": chrony's off.d symlink is not invoked at any point.
Separately, "systemctl restart networkd-dispatcher" silently repairs the
condition via its startup triggers (observed on this host at 04:18:12,
where the restart's replayed "routable" dispatch brought the sources
back online with a route present). That is likely one reason the defect
is rarely reported - any routine restart of the dispatcher, or of
chrony, clears it.
PRIOR REPORTS, AND WHY THIS IS NOT A FEATURE REQUEST
====================================================
This defect has been independently reported before. LP #2043967 (Paul
Gear, November 2023) describes the same mechanism with a different
trigger: after suspend/resume the dispatcher ran "chronyc onoffline"
before IPv6 interfaces had completed Duplicate Address Detection,
leaving those sources ignored "until chronyc onoffline is run again
after the interfaces are usable". That report was marked Incomplete
pending the Ubuntu release, chrony version and reproduction steps, and
expired in January 2024 without them. All three are supplied above.
Taken together, the two reports generalise the defect: DAD and DHCP are
simply two different reasons an interface can reach "routable" before it
is actually usable. The problem is specific to neither - it is that
"routable" does not imply usable, and it is the only state with a chrony
hook attached.
It is also worth addressing the framing of the earlier report, which was
characterised as a feature request. The shipped integration exists
specifically to keep chrony's sources in step with network availability.
On this host it reliably takes sources offline on a link flap and
provides no path to bring them back, producing a 16.5 hour loss of time
synchronisation that required manual intervention to clear. The one-line
addition below restores the integration's stated purpose rather than
extending it.
SUGGESTED FIX
=============
Add a configured.d symlink alongside the existing two, so the hook also
runs once networkd reports the link fully configured:
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/configured.d/chrony-onoffline
Since "chronyc onoffline" is self-determining and idempotent, running it
on an additional state is harmless - demonstrated above, where the pre-
DHCP invocation correctly made no change.
A more targeted fix - guarding the script on $STATE /
$AdministrativeState - was proposed upstream in 2023 (see references).
It has not been applied: examples/chrony.nm-dispatcher.onoffline carries
only the positional-argument guard both in the 4.8 source tarball and in
upstream git master, and is byte-identical to the script shipped on this
system.
A caveat on the symlink as a general fix: the 2023 thread did not stall
for want of evidence - Jongkyu Seok supplied four event-sequence logs
(msg03289). Those logs showed that the order in which
AdministrativeState and OperationalState change VARIES BETWEEN
ENVIRONMENTS, and that linking into configured.d could misfire on a host
where AdministrativeState becomes "configured" while OperationalState is
not yet routable. That is a legitimate objection to the bare symlink as
a universal solution, and no maintainer response to those logs appears
in the archive.
The evidence in this report should therefore be read as a demonstrated
fix FOR THIS CONFIGURATION and an additional data point on ordering, not
as a claim that the symlink is universally correct. Note that adding it
ALONGSIDE the existing routable.d link rather than replacing it means
both orderings receive a dispatch - though the resulting state is still
determined by whichever fires last. If the variability Seok documented
makes a placement-only fix untenable, the guarded-script approach looks
like the more robust route, and that is a decision for upstream.
REFERENCES
==========
- chrony-users, Jongkyu Seok, June 2023 - same diagnosis, proposes the
configured.d script: https://www.mail-archive.com/chrony-
[email protected]/msg03268.html
- Miroslav Lichvar's reply asking for event-sequence examples:
https://www.mail-archive.com/chrony-
[email protected]/msg03269.html
- Jongkyu Seok's reply supplying four event-sequence logs, and noting
that the ordering of AdministrativeState and OperationalState varies
between environments - the thread ends here with no maintainer response
in the archive: https://www.mail-archive.com/chrony-
[email protected]/msg03289.html
- Upstream examples/chrony.nm-dispatcher.onoffline in git master still
has only the "[ $# -ge 2 ]" guard.
- Debian #1093763 - a different defect in the same integration ("chronyc
onoffline" returning non-zero aborting if-up.d, killing the DHCP
client), fixed in chrony 4.8-4 by checking chronyd.sock rather than the
PID file and ensuring hook scripts always exit successfully. Note 4.8-4
postdates the 4.8-2ubuntu1 in Ubuntu 26.04.
- LP #2043967 - same defect, different trigger (IPv6 sources not
restored after suspend/resume, onoffline running before DAD completed).
Filed Nov 2023, marked Incomplete pending release/version/reproduction,
expired Jan 2024.
- LP #2155244 - related but distinct: 20-chrony-dhcp has no networkd-
dispatcher equivalent at all.
- LP #1718227 - origin of the current networkd-dispatcher symlinks,
cited in debian/links.
- LP #2132159 - recent: chrony now declares "Recommends: networkd-
dispatcher" directly, since systemd no longer does. Note this means
networkd-dispatcher is only a Recommends, so on installs without it the
hook never runs in either direction.
** Affects: chrony (Ubuntu)
Importance: Undecided
Status: New
** Affects: chrony (Debian)
Importance: Unknown
Status: Unknown
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161782
Title:
chrony: NTP sources left permanently offline after link flap;
networkd-dispatcher hook runs before the interface is usable
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2161782/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
