If they have only two devices, that's not enough for redundancy. You need a minimum of three. With only two upstream sources, and NTP client cannot accurately determine which is the correct time - whereas with 3, you'll know that 'one' is wrong. That being said if all the Stratum 1 devices rolled back their clocks simultaneously, then all downstream clients would have started to assume that as correct time; once their polling intervals came around (~17 min).
Usually the GPS hardware has the GPS "base week" written in as part of the firmware, to calculate the real time when they report their time back. It's likely that if the devices were all manufactured at the same time, that they would have all died relatively simultaneously. ________________________________ From: AusNOG <[email protected]> on behalf of Mark Smith <[email protected]> Sent: Tuesday, 14 July 2026 15:55 Cc: Ausnog; Greg Price Subject: Re: [AusNOG] Telstra outage? Going from the news reports they only had two. Two services ir devices for redundancy is usually enough. That's what I thought about NTP, until I read the following BCP. The ideal minimum number of NTP time sources for a client is *5*, due to the way time sources are compared to try to identify the best and most accurate one to follow, and then to have redundancy, which comes with having an odd number of NTP sources. RFC8633, "Network Time Protocol Best Current Practices" https://datatracker.ietf.org/doc/html/rfc8633 Also, don't use VMs as NTP time servers. The underlying virtualisation layer can cause the VM's virtual timer interrupt to drift about. You want a hardware timer interrupt to be driving the OS's clock. If Telstra had had 5 Stratacom NTP servers, even if they were all old and had the 20 years ago time fault, it's possible that due to different power on times, the first one that failed and turned time back 20 years would have been ignored by the others and clients because it became a bad time source, and they may have been lucky enough to have time to mitigate the remaining ones failing before it had an impact. Regards, Mark. On Tue, 14 July 2026, 15:30 Luke Thompson, <[email protected]> wrote: "Telstra had a nation-wide network outage last week that affected emergency services. The outage has been pinned on an obsolete Symmetricom SyncServer S300 node, which manages time on the network but resets its 10-bit week counter to zero every 1024 weeks (just under 20 years) in a common and well understood GPS rollover bug that caused the device to reset to 2006. The SyncServer S300 was discontinued in 2016." Cheers, Luke Thompson, CTO The Network Crew P/L E: [email protected] https://tnc.works On 9 July 2026 7:21:52 am Greg Price <[email protected]<mailto:[email protected]>> wrote: The stuff I've read and heard is that it was a software update across multiple systems that provide the reference sources from GNSS that went wrong causing the the reference date/time to jump back decades, and the wheels fall off. This is from the patchwork of sources, but I'd like to hear it definitively from engineers. On 9 Jul 2026, at 7:12 am, Phillip Grasso <[email protected]<mailto:[email protected]>> wrote: Do you think it was related to software patching breaking time services? (NTP?) On Wed, 8 July 2026, 2:48 pm DaZZa, <[email protected]<mailto:[email protected]>> wrote: To paraphrase an oft-repeated meme It's not NTP There's no way it's NTP It was NTP Bets on this just being a speculator to take the heat off until they find out what really went wrong? D On Wed, 8 Jul 2026 at 14:34, Luke Thompson <[email protected]> wrote: > > Telstra's CFO has pointed to time / NTP: > > https://www.theguardian.com/australia-news/2026/jul/08/telstra-outage-down-network-mobile-outages-today > > Seems just over 4 hours of impact, and so far they're unsure what caused it. > > Cheers, > > Luke Thompson, CTO > The Network Crew P/L > > E: [email protected] > https://tnc.works<https://tnc.works/> > > On 8 July 2026 8:43:01 am Luke Thompson <[email protected]> wrote: > >> & likewise, service restored at the moment. >> >> Cheers, >> >> Luke Thompson, CTO >> The Network Crew P/L >> >> [email protected] >> https://tnc.works<https://tnc.works/> >> >> >> On 8/7/2026 8:40 am, [email protected]<mailto:[email protected]> wrote: >>> >>> Good morning >>> >>> Back here. Regional Vic Boost and T prepaid mobile data. >>> >>> ----- Original message ----- >>> From: [email protected]<mailto:[email protected]> >>> To: [email protected]<mailto:[email protected]> >>> Subject: [AusNOG] Telstra outage? >>> Date: Wednesday, 8 July 2026 7:44 AM >>> >>> G'day. >>> >>> How is T where you are? >>> >>> Regional Vic Boost network no calls other than emergency can be made. >>> >>> Cheers >>> Laurie >>> >> _______________________________________________ >> AusNOG mailing list >> [email protected]<mailto:[email protected]> >> https://lists.ausnog.net/mailman/listinfo/ausnog > > > _______________________________________________ > AusNOG mailing list > [email protected]<mailto:[email protected]> > https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list [email protected]<mailto:[email protected]> https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list [email protected]<mailto:[email protected]> https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list [email protected]<mailto:[email protected]> https://lists.ausnog.net/mailman/listinfo/ausnog
_______________________________________________ AusNOG mailing list [email protected] https://lists.ausnog.net/mailman/listinfo/ausnog
