> On Jun 30, 2015, at 3:44 PM, Rob Seaman <[email protected]> wrote: > > Hi Warner, > >> I’m getting hate mail from a former job. Seems like 7 years ago I put some >> stupid code into the tree. It was there for only a year or two, but today it >> took out a few really old systems that were still running this code… >> >> How’s your day going? > > Bummer. You don’t indicate if this is leap second related. Perhaps 7 years > is rounded from 6.5 years for the 2008 leap second? Then one wonders why > this wasn’t an issue in 2012. Or why it would hit in advance of the leap. > > If not related to the leap (or even if it is), I’ll buy you one of those > single malts somebody was talking about.
We were getting crazy leap second notifications from our reference clocks. As a temporary measure to ensure that it didn’t mess up the system, we added a filter so that leap seconds that were too far in the future would get filtered out without thinking through all the consequences. Today was the first day "too far in the future” that a leap second happened on. The ntp reference clock we have on these systems fed in the new leap second (since ntp only advertises the leap second on leap day), and we hit the sanity Assert. This code was only in there for about a year or so, and only a few products were shipped on it. The underlying bug had been found and fixed after a few weeks, but this code lingered until the next rewrite of leap second handling. Most of these events happened after I left the company. I’m still in touch with an engineer that works there who relayed the full, sordid tale of woe to me, along with a heap of scorn for such a stupid failure mode. I’m not proud to admit it, but there you are. A weird leap second bug due to a careless oversight because software is complex and it isn’t always easy to see unintended consequences of things until it is too late. Warner
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ LEAPSECS mailing list [email protected] https://pairlist6.pair.net/mailman/listinfo/leapsecs
