Re: REMINDER: LEAP SECOND
* Harlan Stenn st...@ntp.org Matthew Huff writes: A backward step is a known issue and something that people are more comfortable dealing with as it can happen on any machine with a noisy clock crystal. A clock crystal has to be REALLY bad for ntpd to need to step the clock. Having 61 seconds in a minute or 86401 seconds in a day is a different story. Yeah, leap years suck too. And those jumps around daylight savings time. Hi Harlan, Leap years and DST ladjustments have never caused us any major issues. It seems these code paths are well tested and work fine. The leap second in 2012 however ... total and utter carnage. Application servers, databases, etc. falling over like dominoes. All hands on deck in the middle of the night to clean up. It took days before we stopped finding broken stuff. Maybe all the bugs from 2012 have been fixed. Maybe they haven't. Maybe new ones have been introduced. I'm not terribly optimistic. One example I'm aware of: Cisco Nexus 5010/5020 switches need software that was released as late as 29th of April this year in order to be immune to the crashburn leap second bug CSCub38654. The official «Cisco Suggested release based on software quality, stability and longevity» is older. Go figure. In any case, we're certainly not going to risk it. So our plan is to disconnect our local stratum-2s from their upstreams on June 29th so they (and more crucially, their downstream clients) remain oblivious to the leap second. Come July 1st, we'll reconnect them. The clients' clocks will be 1s (plus any drift) off at that point, but as we're running ntpd with the -x option, that shouldn't cause backwards stepping. Running with slightly incorrect clocks for a few days is a small price to pay to avoid a repeat of 2012's mayhem. Tore
Re: REMINDER: LEAP SECOND
On 24/06/2015 04:33, Harlan Stenn wrote: A clock crystal has to be REALLY bad for ntpd to need to step the clock. or really virtual. Nick
Call for Presentations - DNS-OARC Fall Workshop, Oct 2015
Apologies if you are on multiple lists and see multiple copies of this email. - The next DNS-OARC Fall Workshop will take place in Montreal on Oct 3rd and 4th, the weekend before NANOG65. DNS-OARC is requesting proposals for presentations, with a preference for DNSSEC work. We are looking for the whole spectrum, from measurement work using passive and active techniques, to applications and potential new uses of DNSSEC. This workshop intends to build from previous strong DNS-OARC workshops, where operational content and research is welcome. Presentations from DNS operators are particularly welcome, as well as from DNS researchers. All DNS-related subjects are accepted, introduction to new tools, visualizations, data analysis, DNSSEC and novel uses of the DNS. If you are an DNS-OARC member, and have a sensitive topic you would like to present for members-only, we will accommodate those talks too. Time will be available for lighting talks to fit short presentations (5 to 10 minutes). Workshop Milestones 17 June 2015, Call for Presentations posted 17 June 2015, Open for submissions 30 July 2015, Deadline for submission 20 August 2015, Final Program published 1 October 2015, Final deadline for slideset submission Details for abstract submission will be published here: https://indico.dns-oarc.net/event/24/call-for-abstracts/ The workshop will be organized on different tracks, depending on the topics and the timing of each presentation. If you are interested in a lightning talk, let us know at the time of submission. You can contact the Programme Committee: https://www.dns-oarc.net/oarc/programme via submissi...@dns-oarc.net if you have questions or concerns. Sebastian Castro, for the DNS-OARC Programme Committee DNS-OARC is also seeking sponsorship for this workshop, please contact spon...@dns-oarc.net if your organization is interested in becoming a sponsor. (Please note that DNS-OARC is run on a non-profit basis, and is not in a position to reimburse expenses or time for speakers at its meetings.)
RES: RES: REMINDER: LEAP SECOND
Red Hat recomend update tzdate, this rpm is just for timezones right ? If i won't update it we will have some problem ? -Mensagem original- De: Mark Milhollan [mailto:m...@pixelgate.net] Enviada em: quarta-feira, 24 de junho de 2015 09:19 Para: Leonardo Oliveira Ortiz Assunto: Re: RES: REMINDER: LEAP SECOND On Tue, 23 Jun 2015, Leonardo Oliveira Ortiz wrote: Guys, if we don't have NTP enable on our Linux we still have problem with leap second ?? Without NTP there is no leap second -- you won't even notice it. /mark
Re: REMINDER: LEAP SECOND
Philip Homburg pch-na...@u-1.phicoh.com wrote: For UTC the analog approach would be to keep time in TAI internally and convert to UTC when required. This is much less of a solution than you might hope, because most APIs, protocols, and data formats require UT. (Usually not UTC but a representation isomorphic to traditional UT which ignores leap seconds.) Tony. -- f.anthony.n.finch d...@dotat.at http://dotat.at/ Trafalgar: Variable 3 or 4, but northwesterly 4 or 5 in southeast. Slight, occasionally moderate. Mainly fair. Mainly good.
Re: REMINDER: LEAP SECOND
Yes, the clock has to be bad. Been there, done that, especially early Sun x86 servers. Leap years and DST are both things people and developers are aware of outside of technology, leap seconds, not so much. On Jun 23, 2015, at 11:33 PM, Harlan Stenn st...@ntp.org wrote: Matthew Huff writes: A backward step is a known issue and something that people are more comfortable dealing with as it can happen on any machine with a noisy clock crystal. A clock crystal has to be REALLY bad for ntpd to need to step the clock. Having 61 seconds in a minute or 86401 seconds in a day is a different story. Yeah, leap years suck too. And those jumps around daylight savings time. H
Re: REMINDER: LEAP SECOND
In your letter dated Wed, 24 Jun 2015 08:33:14 +0200 you wrote: Leap years and DST ladjustments have never caused us any major issues. It seems these code paths are well tested and work fine. I seem to remember that they were not tested that well on a certain brand of mobile devices a few years back... In any case, we can abstract from time zones and DST by using UTC internally and then converting to local time in the UI. For UTC the analog approach would be to keep time in TAI internally and convert to UTC when required. There is however a big problem with that. UTC as a time scale is not predictable. There is no way of computing the number of seconds between 2000-01-01 00:00 and 2100-01-01 00:00 because that value is undefined. The net results is that representing, say, 2020-01-01 00:00 as a TAI timestamp is impossible until about 6 months before that date. One way forward for people who for some reason feel attached to representing the rotation of the earth in civil time is to have a scheme for leap second just like leap years. For example, insert a leap second every 18 months. And then revise that scheme once a century to cope unexpect changes in the earth's rotation. (Or just get rid of them all together and move to a different time zone every 4000 years).
Re: REMINDER: LEAP SECOND
Once upon a time, Gary E. Miller g...@rellim.com said: Depends on what your Stratum-1 is syncronized to. Some GPS time sources pass on the leap indicator to NTP. For example, the SiRF-3 GPS, connected by way of gpsd, to ntpd will pass on the leap second. Yep, my ancient old SVeeSix has been showing the leap second for several months; the notification it was added to the GPS signal a while back, either at the start of the year or the start of the quarter (in theory, leap seconds can be added/removed quarterly). -- Chris Adams c...@cmadams.net
Re: REMINDER: LEAP SECOND
In your letter dated Wed, 24 Jun 2015 14:05:34 +0100 you wrote: Philip Homburg pch-na...@u-1.phicoh.com wrote: For UTC the analog approach would be to keep time in TAI internally and convert to UTC when required. This is much less of a solution than you might hope, because most APIs, protocols, and data formats require UT. (Usually not UTC but a representation isomorphic to traditional UT which ignores leap seconds.) Supporting legacy formats can be annoying. In some cases it would be no problem. For example NTP. If there is a defined way to convert between TAI and UTC then converting TAI to NTP timestamps is easy except during an actual leap second. Which is not really a problem. Unix systems would probably need a few new system calls to accept time in TAI. File formats like tar are unlikely to matter much: find a consistent way of encoding time around the leap second and most likely nobody will care. In any case, it would be nice if future formats and systems could have a sensible time keeping system.
Re: REMINDER: LEAP SECOND
On 06/24/2015 12:44 PM, Matthew Huff wrote: It looks like the safest thing for us to do is to keep our NTP servers running and deal with any crashes/issues. That's better than having to deal with FINRA. For what it's worth, Red Hat pushed updates to NTP and to TZDATA. You might want to check the documentation to see if the updates include sane handling of the leap second. (I run CentOS 7.1 and Fedora 20, which is where I saw the updates during my morning maintenance.)
Re: REMINDER: LEAP SECOND
On Mon, Jun 22, 2015 at 7:17 AM, shawn wilson ag4ve...@gmail.com wrote: So, what we should do is make clocks move. 9 slower half of the year (and then speed back up) so that we're really in line with earth's rotational time. I mean we've got the computers to do it (I think most RTC only go down to thousandths so it'll still need a little skewing but I'm sure we'll manage). Ps - if anyone actually does this, I'm going postal. http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html comes dangerously close to your modest proposal. Damian
Re: REMINDER: LEAP SECOND
On Wed, Jun 24, 2015 at 08:33:14AM +0200, Tore Anderson wrote: Leap years and DST ladjustments have never caused us any major issues. It seems these code paths are well tested and work fine. I've seen quite a few people that for whatever reason insist on running systems in local time zones struggle with the DST reverse step. It's not nearly as much of a non-issue as you claim. The leap second in 2012 however ... total and utter carnage. Application servers, databases, etc. falling over like dominoes. All hands on deck in the middle of the night to clean up. It took days before we stopped finding broken stuff. Total and utter carnage is a bit of a stretch. Linux hosts that ran applications dependant on nanosleeps needed reboots. Note that this wasn't an issue in 2009, because the poorly tested change in question hadn't yet been made to the Linux kernel. (Even in 2012, my personal hosts, running a different operating system sailed through it just fine.) At any time, you might have a bad operational day for any number of reasons. Sure, that one was annoying, but to my knowledge nobody died, and a lot of hosts that probably needed one anyway got a reboot. Certainly, lately, I've seen a lot of Linux hosts rebooted more than once for security patching. #opslife? Cheers, --msa
Re: REMINDER: LEAP SECOND
Once upon a time, Majdi S. Abbas m...@latt.net said: Total and utter carnage is a bit of a stretch. Linux hosts that ran applications dependant on nanosleeps needed reboots. Note that this wasn't an issue in 2009, because the poorly tested change in question hadn't yet been made to the Linux kernel. In 2009, there was a different problem. If the system was under sufficient kernel-related load (such as disk I/O), the kernel's attempt to print an informational message that a leap second had been added caused a kernel deadlock, immediately killing the system. I don't remember any widespread Linux-related leap second issues before that though. -- Chris Adams c...@cmadams.net
Re: REMINDER: LEAP SECOND
* Majdi S. Abbas On Wed, Jun 24, 2015 at 08:33:14AM +0200, Tore Anderson wrote: Leap years and DST ladjustments have never caused us any major issues. It seems these code paths are well tested and work fine. I've seen quite a few people that for whatever reason insist on running systems in local time zones struggle with the DST reverse step. It's not nearly as much of a non-issue as you claim. Read again, and note the word us. I am describing my and my employer's experience with past DST changes and leap years, and those have indeed been completely uneventful. YMMV. The leap second in 2012 however ... total and utter carnage. Application servers, databases, etc. falling over like dominoes. All hands on deck in the middle of the night to clean up. It took days before we stopped finding broken stuff. Total and utter carnage is a bit of a stretch. As above, I am speaking only about how the 2012 leap second went down in our infrastructure. I stand by how I described the event. Again, YMMV. If you plan on let your infrastructure deal with the upcoming leap second head-on, I wish you the best of luck. Hopefully all the bugs from 2012 have been fixed. I, however, certainly have no intention of being the one to find out otherwise. Tore
RE: REMINDER: LEAP SECOND
That won't work. Being internally sync'ed isn't good enough for FINRA. All the machines must be synced to an external accurate source at least once per trading day. Our plan is to disable our two stratum 1 servers, and our 3 stratum 2 servers before the leap second turnover, but to be 100% safe we would need to do that 24 hours before, but that would be a violation of FINRA regulations. It looks like the safest thing for us to do is to keep our NTP servers running and deal with any crashes/issues. That's better than having to deal with FINRA. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669 -Original Message- From: Tore Anderson [mailto:t...@fud.no] Sent: Wednesday, June 24, 2015 3:26 PM To: Matthew Huff Cc: nanog2 Subject: Re: REMINDER: LEAP SECOND * Matthew Huff I saw that, but it says the bits are set before 23:59 on the day of insertion, but I was hoping that I could shut it down later than 23:59:59 of the previous day (8pm EST). The reason is FINRA regulations. We have to have the time synced once per trading day before the open according to the regulations. Again AIUI, and I'm no NTP expert so I hope someone corrects me if I'm wrong: If you don't configure the leapfile ntpd option, the Leap Indicator flag will flow down to your servers from the stratum-1s servers you're synchronising from (directly or indirectly). So what I think you could do, is to on the 29th remove all your upstream servers from your NTP server's config, and set fudge 127.127.1.0 stratum 3 or something like that so that clients will still want to sync to it. At that point, your NTP server's clock chip will be the reference clock, which might be drift-prone. To work around that, you could at 8pm on the 30th stop ntpd, manually sync the system clock with ntpdate, and start ntpd again. That should keep your NTP server's clock reasonably synchronised, that provides your clients with (a Leap Indicator-free) NTP service. I make no guarantees that the above will work the way I think it will, though... Try it at your own risk. Tore
Re: REMINDER: LEAP SECOND
* Matthew Huff Does anyone know what the latest that we can run our NTP servers and not distribute the LEAP_SECOND flag to the NTP clients? From http://support.ntp.org/bin/view/Support/NTPRelatedDefinitions: Leap Indicator This is a two-bit code warning of an impending leap second to be inserted in the NTP timescale. The bits are set before 23:59 on the day of insertion and reset after 00:00 on the following day. This causes the number of seconds (rollover interval) in the day of insertion to be increased or decreased by one. So the answer to your question is, AIUI, 2015-06-29 23:59:59. Tore
Re: REMINDER: LEAP SECOND
Does anyone know what the latest that we can run our NTP servers and not distribute the LEAP_SECOND flag to the NTP clients? On Jun 24, 2015, at 2:33 PM, Tore Anderson t...@fud.no wrote: * Majdi S. Abbas On Wed, Jun 24, 2015 at 08:33:14AM +0200, Tore Anderson wrote: Leap years and DST ladjustments have never caused us any major issues. It seems these code paths are well tested and work fine. I've seen quite a few people that for whatever reason insist on running systems in local time zones struggle with the DST reverse step. It's not nearly as much of a non-issue as you claim. Read again, and note the word us. I am describing my and my employer's experience with past DST changes and leap years, and those have indeed been completely uneventful. YMMV. The leap second in 2012 however ... total and utter carnage. Application servers, databases, etc. falling over like dominoes. All hands on deck in the middle of the night to clean up. It took days before we stopped finding broken stuff. Total and utter carnage is a bit of a stretch. As above, I am speaking only about how the 2012 leap second went down in our infrastructure. I stand by how I described the event. Again, YMMV. If you plan on let your infrastructure deal with the upcoming leap second head-on, I wish you the best of luck. Hopefully all the bugs from 2012 have been fixed. I, however, certainly have no intention of being the one to find out otherwise. Tore
RE: REMINDER: LEAP SECOND
I saw that, but it says the bits are set before 23:59 on the day of insertion, but I was hoping that I could shut it down later than 23:59:59 of the previous day (8pm EST). The reason is FINRA regulations. We have to have the time synced once per trading day before the open according to the regulations. We could manually run ntpdate on 100+ servers including 50+ windows servers, but that's not a great solution. Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669 -Original Message- From: Tore Anderson [mailto:t...@fud.no] Sent: Wednesday, June 24, 2015 3:07 PM To: Matthew Huff Cc: nanog2 Subject: Re: REMINDER: LEAP SECOND * Matthew Huff Does anyone know what the latest that we can run our NTP servers and not distribute the LEAP_SECOND flag to the NTP clients? From http://support.ntp.org/bin/view/Support/NTPRelatedDefinitions: Leap Indicator This is a two-bit code warning of an impending leap second to be inserted in the NTP timescale. The bits are set before 23:59 on the day of insertion and reset after 00:00 on the following day. This causes the number of seconds (rollover interval) in the day of insertion to be increased or decreased by one. So the answer to your question is, AIUI, 2015-06-29 23:59:59. Tore
Re: REMINDER: LEAP SECOND
* Matthew Huff I saw that, but it says the bits are set before 23:59 on the day of insertion, but I was hoping that I could shut it down later than 23:59:59 of the previous day (8pm EST). The reason is FINRA regulations. We have to have the time synced once per trading day before the open according to the regulations. Again AIUI, and I'm no NTP expert so I hope someone corrects me if I'm wrong: If you don't configure the leapfile ntpd option, the Leap Indicator flag will flow down to your servers from the stratum-1s servers you're synchronising from (directly or indirectly). So what I think you could do, is to on the 29th remove all your upstream servers from your NTP server's config, and set fudge 127.127.1.0 stratum 3 or something like that so that clients will still want to sync to it. At that point, your NTP server's clock chip will be the reference clock, which might be drift-prone. To work around that, you could at 8pm on the 30th stop ntpd, manually sync the system clock with ntpdate, and start ntpd again. That should keep your NTP server's clock reasonably synchronised, that provides your clients with (a Leap Indicator-free) NTP service. I make no guarantees that the above will work the way I think it will, though... Try it at your own risk. Tore
Re: REMINDER: LEAP SECOND
* Matthew Huff That won't work. Being internally sync'ed isn't good enough for FINRA. All the machines must be synced to an external accurate source at least once per trading day. That was why I proposed to ntpdate on your (upstream-free since the 29th) NTP server(s) sometime on the 30th. That would synchronise its local clock with an external accurate source, without learning the Leap Indicator. Our plan is to disable our two stratum 1 servers, and our 3 stratum 2 servers before the leap second turnover, but to be 100% safe we would need to do that 24 hours before, but that would be a violation of FINRA regulations. If you run your own straum-1 servers, can't you just opt not to configure leapfile? Assuming your own organisation is the only user of those servers, that is (certainly don't do that if it's a public server). After the leap second has passed, you can proceed to correct things. Your clients will then be 1s ahead of correct time, and will need to step/slew their clocks to get in sync. But maybe that's OK as far as FINRA's concerned... It looks like the safest thing for us to do is to keep our NTP servers running and deal with any crashes/issues. That's better than having to deal with FINRA. Maybe. I have no experience with FINRA. :-) Tore
Re: REMINDER: LEAP SECOND
Yo Tore! On Wed, 24 Jun 2015 21:57:28 +0200 Tore Anderson t...@fud.no wrote: If you run your own straum-1 servers, can't you just opt not to configure leapfile? Depends on what your Stratum-1 is syncronized to. Some GPS time sources pass on the leap indicator to NTP. For example, the SiRF-3 GPS, connected by way of gpsd, to ntpd will pass on the leap second. At least in theory. :-) RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1(541)382-8588 pgpZSFURiKdAg.pgp Description: OpenPGP digital signature
