** Description changed: + Reported upstream here: https://www.mail-archive.com/chrony- + [email protected]/msg03942.html + chrony: NTP sources left permanently offline after transient network loss (networkd-dispatcher hook runs before the interface is usable; no configured.d symlink) Package: chrony 4.8-2ubuntu1 Release: Ubuntu 26.04 Arch: arm64 (Raspberry Pi 3B, no RTC) Network backend: systemd-networkd via netplan, wired eth0 networkd-dispatcher: installed and active - SUMMARY ======= On a systemd-networkd host, chrony's NTP sources can be left permanently offline after a transient network outage. The networkd-dispatcher hook runs "chronyc onoffline" while the interface has carrier but is not yet fully configured - in this case with no DHCP-assigned address or gateway. With no route present, chronyd correctly marks the affected sources offline. Once configuration completes and a route exists, no further dispatcher event runs the hook, so the sources are never brought back online. chronyd then free-runs indefinitely, logging "Can't synchronise: no selectable sources", and sets the kernel unsync flag so timedatectl reports "System clock synchronized: no". In my case sources stayed offline for 16.5 hours, until I ran "chronyc online" by hand. The clock itself remained accurate throughout (~145 us of NTP time, free-running on the learned frequency), so the fault is silent unless something is gated on NTPSynchronized. Several systemd units on this host gate an ExecStartPre on exactly that, and all of them failed for the duration. - STEPS TO REPRODUCE ================== On a networkd host running chrony, cause carrier loss followed by a slow DHCP re-acquisition - e.g. power-cycle the upstream router. In my case a router firmware update caused ~36 s of carrier flapping, and DHCP did not complete for a further 2m15s. - OBSERVED ======== systemd-networkd (times UTC; LAN addresses redacted): - 06:19:10 eth0: Lost carrier - 06:19:10 eth0: DHCP lease lost - 06:19:13 eth0: Gained carrier - 06:19:19 eth0: Lost carrier - 06:19:42 eth0: Gained carrier - 06:19:44 eth0: Lost carrier - 06:19:46 eth0: Gained carrier - 06:22:01 eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired from 192.168.x.1 + 06:19:10 eth0: Lost carrier + 06:19:10 eth0: DHCP lease lost + 06:19:13 eth0: Gained carrier + 06:19:19 eth0: Lost carrier + 06:19:42 eth0: Gained carrier + 06:19:44 eth0: Lost carrier + 06:19:46 eth0: Gained carrier + 06:22:01 eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired from 192.168.x.1 chronyd: - 06:21:36 Source 185.125.190.123 offline - 06:21:36 Source 91.189.91.112 offline - 06:21:36 Source 2620:2d:4000:1::1123 offline - 06:21:36 Source 91.189.91.113 offline - 06:21:36 Can't synchronise: no selectable sources (5 unreachable sources) - 06:21:36 Source 2620:2d:4000:1::2123 offline + 06:21:36 Source 185.125.190.123 offline + 06:21:36 Source 91.189.91.112 offline + 06:21:36 Source 2620:2d:4000:1::1123 offline + 06:21:36 Source 91.189.91.113 offline + 06:21:36 Can't synchronise: no selectable sources (5 unreachable sources) + 06:21:36 Source 2620:2d:4000:1::2123 offline Note the ordering: chronyd marked all five sources offline at 06:21:36, twenty-five seconds BEFORE the interface acquired an address and gateway at 06:22:01. Nothing ran the hook again afterwards. State 16.5 hours later, before any manual intervention: - $ chronyc tracking - Reference ID : 45AE48D5 (ntp-nts-2.ps5.canonical.com) - Ref time (UTC) : Fri Jul 24 06:15:09 2026 - System time : 0.000144547 seconds slow of NTP time - Leap status : Normal - - $ timedatectl show -p NTPSynchronized --value - no + $ chronyc tracking + Reference ID : 45AE48D5 (ntp-nts-2.ps5.canonical.com) + Ref time (UTC) : Fri Jul 24 06:15:09 2026 + System time : 0.000144547 seconds slow of NTP time + Leap status : Normal + + $ timedatectl show -p NTPSynchronized --value + no "chronyc -n sources" at that point showed all five sources as '^?' (unusable), Reach 0, LastRx 16h. "sudo chronyc online" restored them immediately, and NTPSynchronized returned to "yes" within one poll. - DELIBERATE REPRODUCTION ======================= I reproduced this by physically disconnecting and reconnecting the Ethernet cable, with networkd-dispatcher at increased verbosity (so invoked hooks are named) and a logging script installed in every state directory: - #!/bin/sh - logger -t dispatch-debug "STATE=$STATE IFACE=$IFACE Admin=$AdministrativeState Oper=$OperationalState" + #!/bin/sh + logger -t dispatch-debug "STATE=$STATE IFACE=$IFACE Admin=$AdministrativeState Oper=$OperationalState" Full sequence (trimmed to the relevant units; addresses redacted): - 04:06:46 kernel: eth0: Link is Down - 04:06:46 systemd-networkd: eth0: Lost carrier - 04:06:46 systemd-networkd: eth0: DHCP lease lost - 04:06:46 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for interface eth0 - 04:06:46 dispatch-debug: STATE=no-carrier Admin=configured Oper=no-carrier - - 04:07:41 kernel: eth0: Link is Up - 100Mbps/Full - 04:07:41 systemd-networkd: eth0: Gained carrier - 04:07:41 networkd-dispatcher: Invoking '.../degraded.d/00-debug' for interface eth0 - 04:07:41 dispatch-debug: STATE=degraded Admin=configured Oper=degraded - 04:07:41 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for interface eth0 - 04:07:41 dispatch-debug: STATE=configuring Admin=configuring Oper=degraded - 04:07:42 networkd-dispatcher: Invoking '.../routable.d/00-debug' for interface eth0 - 04:07:42 dispatch-debug: STATE=routable Admin=configuring Oper=routable - 04:07:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for interface eth0 - 04:07:43 networkd-dispatcher: Invoking '.../configured.d/00-debug' for interface eth0 - 04:07:43 dispatch-debug: STATE=configured Admin=configured Oper=routable - 04:07:45 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired - 04:07:45 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for interface eth0 - 04:07:45 dispatch-debug: STATE=configuring Admin=configuring Oper=routable - 04:07:45 networkd-dispatcher: Invoking '.../configured.d/00-debug' for interface eth0 - 04:07:45 dispatch-debug: STATE=configured Admin=configured Oper=routable + 04:06:46 kernel: eth0: Link is Down + 04:06:46 systemd-networkd: eth0: Lost carrier + 04:06:46 systemd-networkd: eth0: DHCP lease lost + 04:06:46 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for interface eth0 + 04:06:46 dispatch-debug: STATE=no-carrier Admin=configured Oper=no-carrier + + 04:07:41 kernel: eth0: Link is Up - 100Mbps/Full + 04:07:41 systemd-networkd: eth0: Gained carrier + 04:07:41 networkd-dispatcher: Invoking '.../degraded.d/00-debug' for interface eth0 + 04:07:41 dispatch-debug: STATE=degraded Admin=configured Oper=degraded + 04:07:41 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for interface eth0 + 04:07:41 dispatch-debug: STATE=configuring Admin=configuring Oper=degraded + 04:07:42 networkd-dispatcher: Invoking '.../routable.d/00-debug' for interface eth0 + 04:07:42 dispatch-debug: STATE=routable Admin=configuring Oper=routable + 04:07:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for interface eth0 + 04:07:43 networkd-dispatcher: Invoking '.../configured.d/00-debug' for interface eth0 + 04:07:43 dispatch-debug: STATE=configured Admin=configured Oper=routable + 04:07:45 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired + 04:07:45 networkd-dispatcher: Invoking '.../configuring.d/00-debug' for interface eth0 + 04:07:45 dispatch-debug: STATE=configuring Admin=configuring Oper=routable + 04:07:45 networkd-dispatcher: Invoking '.../configured.d/00-debug' for interface eth0 + 04:07:45 dispatch-debug: STATE=configured Admin=configured Oper=routable This establishes four things: 1. On carrier loss the dispatched state is "no-carrier", not "off". chrony's off.d symlink is never invoked by a link flap. The only chrony symlink that executes in this entire sequence is routable.d/chrony- onoffline. 2. "routable" is dispatched with AdministrativeState=configuring, at 04:07:42 - three seconds before DHCPv4 supplies an address and gateway at 04:07:45. Note that OperationalState reads "routable" from 04:07:42 onward, including at both "configured" dispatches; what changes across the sequence is AdministrativeState (configuring -> configured -> configuring -> configured). The claim is therefore specific: the dispatch chrony's hook is attached to fires while networkd's ADMINISTRATIVE state is still "configuring". The interface is operationally routable at that point via IPv6 - as the IPv4-only offline set confirms - but IPv4 is not yet configured. 3. chrony's hook runs at exactly that moment, and nothing runs it again. Because the only hook is on "routable", and "routable" precedes IPv4 configuration, a link flap does not merely risk leaving IPv4 sources offline - it does so deterministically. 4. "configured" is dispatched at 04:07:45, immediately after the DHCPv4 lease. A configured.d symlink would therefore be invoked with a working default route present. Point 2 answers the question raised upstream in 2023 ("Doesn't the routable state follow configured?" - msg03269): the "routable" dispatch precedes the post-DHCP "configured" dispatch. Note there are no chronyd log lines in this window: the IPv4 sources were already offline from an identical earlier test, so "chronyc onoffline" produced no state change and logged nothing. The verification capture further below shows the same routable.d/chrony-onoffline invocation immediately followed by "Source ... offline" for all three IPv4 sources, with the two IPv6 sources untouched - onoffline correctly reading the routing table as IPv6-routable and IPv4-not-yet. The defect is not in onoffline's logic. It is that the only dispatch wired to it fires before IPv4 configuration completes, and nothing re- runs it afterwards. Caveat on ordering: the sequence is not monotonic - "configuring" is dispatched again at 04:07:45 after a "configured" at 04:07:43. On a flapping link states can re-enter. In this capture "configured" is dispatched last and after DHCP, so the proposed symlink would have restored the sources; but it does not by itself guarantee the hook always runs last. - ANALYSIS ======== /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline guards its action with "[ $# -ge 2 ]". networkd-dispatcher passes interface state through environment variables (IFACE, STATE) and not positional arguments, so that guard is always false and the script unconditionally runs "chronyc onoffline". That command determines online/offline from the routing table, so the outcome depends entirely on WHEN it runs relative to address and route configuration. debian/links, from the 4.8-2ubuntu1 source package, installs the script into two networkd-dispatcher state directories only: - # Update sources in response to systemd-networkd events (LP: #1718227). - # This is reusing the NetworkManager dispatch script which has no hard - # dependency to NetworkManager (not using any of its arguments) - usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline usr/lib/networkd-dispatcher/routable.d/chrony-onoffline - usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline usr/lib/networkd-dispatcher/off.d/chrony-onoffline + # Update sources in response to systemd-networkd events (LP: #1718227). + # This is reusing the NetworkManager dispatch script which has no hard + # dependency to NetworkManager (not using any of its arguments) + usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline usr/lib/networkd-dispatcher/routable.d/chrony-onoffline + usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline usr/lib/networkd-dispatcher/off.d/chrony-onoffline (Identical in current Debian packaging. "dpkg -L chrony" on the running system confirms these are the only two symlinks shipped.) The comment makes clear the argument-less fall-through is deliberate and understood. The gap is only that "configured" is not in the list, so nothing runs the hook at the point where networkd reports the link fully configured - i.e. after DHCP has supplied an address and gateway. - VERIFICATION OF THE PROPOSED FIX ================================ I added the missing symlink locally: - /usr/lib/networkd-dispatcher/configured.d/50-chrony-onoffline-local - -> /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline + /usr/lib/networkd-dispatcher/configured.d/50-chrony-onoffline-local + -> /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline and repeated the cable disconnect/reconnect: - 04:25:59 systemd-networkd: eth0: Lost carrier - 04:25:59 systemd-networkd: eth0: DHCP lease lost - 04:25:59 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for interface eth0 - 04:25:59 dispatch-debug: STATE=no-carrier Admin=configured Oper=no-carrier - - 04:26:39 systemd-networkd: eth0: Gained carrier - 04:26:39 dispatch-debug: STATE=degraded Admin=configured Oper=degraded - 04:26:40 dispatch-debug: STATE=configuring Admin=configuring Oper=degraded - 04:26:42 dispatch-debug: STATE=routable Admin=configuring Oper=routable - 04:26:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for interface eth0 - 04:26:42 chronyd: Source 185.125.190.123 offline - 04:26:42 chronyd: Source 91.189.91.112 offline - 04:26:42 chronyd: Source 91.189.91.113 offline - 04:26:42 dispatch-debug: STATE=configured Admin=configured Oper=routable - 04:26:42 networkd-dispatcher: Invoking '.../configured.d/50-chrony-onoffline-local' for interface eth0 - 04:26:44 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired - 04:26:44 dispatch-debug: STATE=configuring Admin=configuring Oper=routable - 04:26:44 dispatch-debug: STATE=configured Admin=configured Oper=routable - 04:26:44 networkd-dispatcher: Invoking '.../configured.d/50-chrony-onoffline-local' for interface eth0 - 04:26:44 chronyd: Source 185.125.190.123 online - 04:26:44 chronyd: Source 91.189.91.112 online - 04:26:44 chronyd: Source 91.189.91.113 online + 04:25:59 systemd-networkd: eth0: Lost carrier + 04:25:59 systemd-networkd: eth0: DHCP lease lost + 04:25:59 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug' for interface eth0 + 04:25:59 dispatch-debug: STATE=no-carrier Admin=configured Oper=no-carrier + + 04:26:39 systemd-networkd: eth0: Gained carrier + 04:26:39 dispatch-debug: STATE=degraded Admin=configured Oper=degraded + 04:26:40 dispatch-debug: STATE=configuring Admin=configuring Oper=degraded + 04:26:42 dispatch-debug: STATE=routable Admin=configuring Oper=routable + 04:26:42 networkd-dispatcher: Invoking '.../routable.d/chrony-onoffline' for interface eth0 + 04:26:42 chronyd: Source 185.125.190.123 offline + 04:26:42 chronyd: Source 91.189.91.112 offline + 04:26:42 chronyd: Source 91.189.91.113 offline + 04:26:42 dispatch-debug: STATE=configured Admin=configured Oper=routable + 04:26:42 networkd-dispatcher: Invoking '.../configured.d/50-chrony-onoffline-local' for interface eth0 + 04:26:44 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1 acquired + 04:26:44 dispatch-debug: STATE=configuring Admin=configuring Oper=routable + 04:26:44 dispatch-debug: STATE=configured Admin=configured Oper=routable + 04:26:44 networkd-dispatcher: Invoking '.../configured.d/50-chrony-onoffline-local' for interface eth0 + 04:26:44 chronyd: Source 185.125.190.123 online + 04:26:44 chronyd: Source 91.189.91.112 online + 04:26:44 chronyd: Source 91.189.91.113 online (The local symlink is named 50-chrony-onoffline-local only to avoid colliding with any future package-shipped file; it is the same script.) The defect still occurs: routable.d/chrony-onoffline runs at 04:26:42 and parks the three IPv4 sources. The added hook then runs twice - once at 04:26:42, still before DHCP, where it correctly makes no change; and again at 04:26:44 immediately after the DHCPv4 lease, where it restores all three sources. Exposure drops from indefinite to two seconds, unattended. The harmless pre-DHCP invocation also demonstrates that running onoffline on an additional state introduces no churn - it is self-determining and idempotent. This run again confirms that carrier loss dispatches "no-carrier", not "off": chrony's off.d symlink is not invoked at any point. Separately, "systemctl restart networkd-dispatcher" silently repairs the condition via its startup triggers (observed on this host at 04:18:12, where the restart's replayed "routable" dispatch brought the sources back online with a route present). That is likely one reason the defect is rarely reported - any routine restart of the dispatcher, or of chrony, clears it. - PRIOR REPORTS, AND WHY THIS IS NOT A FEATURE REQUEST ==================================================== This defect has been independently reported before. LP #2043967 (Paul Gear, November 2023) describes the same mechanism with a different trigger: after suspend/resume the dispatcher ran "chronyc onoffline" before IPv6 interfaces had completed Duplicate Address Detection, leaving those sources ignored "until chronyc onoffline is run again after the interfaces are usable". That report was marked Incomplete pending the Ubuntu release, chrony version and reproduction steps, and expired in January 2024 without them. All three are supplied above. Taken together, the two reports generalise the defect: DAD and DHCP are simply two different reasons an interface can reach "routable" before it is actually usable. The problem is specific to neither - it is that "routable" does not imply usable, and it is the only state with a chrony hook attached. It is also worth addressing the framing of the earlier report, which was characterised as a feature request. The shipped integration exists specifically to keep chrony's sources in step with network availability. On this host it reliably takes sources offline on a link flap and provides no path to bring them back, producing a 16.5 hour loss of time synchronisation that required manual intervention to clear. The one-line addition below restores the integration's stated purpose rather than extending it. - SUGGESTED FIX ============= Add a configured.d symlink alongside the existing two, so the hook also runs once networkd reports the link fully configured: - usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline + usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline usr/lib/networkd-dispatcher/configured.d/chrony-onoffline Since "chronyc onoffline" is self-determining and idempotent, running it on an additional state is harmless - demonstrated above, where the pre- DHCP invocation correctly made no change. A more targeted fix - guarding the script on $STATE / $AdministrativeState - was proposed upstream in 2023 (see references). It has not been applied: examples/chrony.nm-dispatcher.onoffline carries only the positional-argument guard both in the 4.8 source tarball and in upstream git master, and is byte-identical to the script shipped on this system. A caveat on the symlink as a general fix: the 2023 thread did not stall for want of evidence - Jongkyu Seok supplied four event-sequence logs (msg03289). Those logs showed that the order in which AdministrativeState and OperationalState change VARIES BETWEEN ENVIRONMENTS, and that linking into configured.d could misfire on a host where AdministrativeState becomes "configured" while OperationalState is not yet routable. That is a legitimate objection to the bare symlink as a universal solution, and no maintainer response to those logs appears in the archive. The evidence in this report should therefore be read as a demonstrated fix FOR THIS CONFIGURATION and an additional data point on ordering, not as a claim that the symlink is universally correct. Note that adding it ALONGSIDE the existing routable.d link rather than replacing it means both orderings receive a dispatch - though the resulting state is still determined by whichever fires last. If the variability Seok documented makes a placement-only fix untenable, the guarded-script approach looks like the more robust route, and that is a decision for upstream. - REFERENCES ========== - chrony-users, Jongkyu Seok, June 2023 - same diagnosis, proposes the configured.d script: https://www.mail-archive.com/chrony- [email protected]/msg03268.html - Miroslav Lichvar's reply asking for event-sequence examples: https://www.mail-archive.com/chrony- [email protected]/msg03269.html - Jongkyu Seok's reply supplying four event-sequence logs, and noting that the ordering of AdministrativeState and OperationalState varies between environments - the thread ends here with no maintainer response in the archive: https://www.mail-archive.com/chrony- [email protected]/msg03289.html - Upstream examples/chrony.nm-dispatcher.onoffline in git master still has only the "[ $# -ge 2 ]" guard. - Debian #1093763 - a different defect in the same integration ("chronyc onoffline" returning non-zero aborting if-up.d, killing the DHCP client), fixed in chrony 4.8-4 by checking chronyd.sock rather than the PID file and ensuring hook scripts always exit successfully. Note 4.8-4 postdates the 4.8-2ubuntu1 in Ubuntu 26.04. - LP #2043967 - same defect, different trigger (IPv6 sources not restored after suspend/resume, onoffline running before DAD completed). Filed Nov 2023, marked Incomplete pending release/version/reproduction, expired Jan 2024. - LP #2155244 - related but distinct: 20-chrony-dhcp has no networkd- dispatcher equivalent at all. - LP #1718227 - origin of the current networkd-dispatcher symlinks, cited in debian/links. - LP #2132159 - recent: chrony now declares "Recommends: networkd- dispatcher" directly, since systemd no longer does. Note this means networkd-dispatcher is only a Recommends, so on installs without it the hook never runs in either direction.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2161782 Title: chrony: NTP sources left permanently offline after link flap; networkd-dispatcher hook runs before the interface is usable To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/chrony/+bug/2161782/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
