Re: NTP Issues Today

2012-11-21 Thread Robert E. Seastrom
Blake Dunlap iki...@gmail.com writes: That's what happens when you just follow vendor recommendations blindly. If you do follow that on vm's (which can actually be a good practice), make sure they pull from your own time infrastructure, and not just the world at large, and that those servers

Re: NTP Issues Today

2012-11-21 Thread Ryan Malayter
On Nov 19, 2012, at 6:12 PM, Scott Weeks sur...@mauigateway.com wrote: wbai...@satelliteintelligencegroup.com Or you could just concede the fact that the navy is playing with time travel again. -- To finish this thread off for

Re: NTP Issues Today

2012-11-21 Thread Ryan Malayter
On Nov 19, 2012, at 6:12 PM, Scott Weeks sur...@mauigateway.com wrote: Lesson learned: Use more than one NTP source. The lesson is: use MORE THAN TWO diverse NTP sources. A man with two watches has no idea what the time it actually is.

Re: NTP Issues Today

2012-11-21 Thread Neil Harris
On 21/11/12 12:34, Ryan Malayter wrote: On Nov 19, 2012, at 6:12 PM, Scott Weeks sur...@mauigateway.com wrote: Lesson learned: Use more than one NTP source. The lesson is: use MORE THAN TWO diverse NTP sources. A man with two watches has no idea what the time it actually is. Per David

Re: NTP Issues Today

2012-11-21 Thread Sid Rao
Guys: We were synchronized against multiple sources. Unfortunately the Navy NTP source contaminated multiple downstream sources. Unless you can trace all your sources, if these sources all have a root source you will break. Sid Rao | CTI Group | +1 (317) 262-4677 On Nov 21, 2012, at 8:01

RE: NTP Issues Today

2012-11-21 Thread Chuck Church
-Original Message- From: Jimmy Hess [mailto:mysi...@gmail.com] Sent: Tuesday, November 20, 2012 7:50 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today This _should_ have caused NTP to execute a panic shutdown, instead of setting the clock back 30 million seconds

Re: NTP Issues Today

2012-11-21 Thread Greg Ihnen
It sounds like the Navy and who ever else they partner with (NIST?) need some egress filtering on their NTP servers to catch and prevent events like this.

Re: NTP Issues Today

2012-11-21 Thread Jay Ashworth
- Original Message - From: Sid Rao s...@ctigroup.com We were synchronized against multiple sources. Unfortunately the Navy NTP source contaminated multiple downstream sources. Unless you can trace all your sources, if these sources all have a root source you will break. ...

Re: NTP Issues Today

2012-11-21 Thread Majdi S. Abbas
On Wed, Nov 21, 2012 at 10:41:01AM -0500, Jay Ashworth wrote: ... against multiple [Stratum 1] sources... Baby, if you've ever wondered... whether it matters whether your sources are strat 1 or not, now you know -- since there's no real way to get provenance on down-strat time sources that

Re: NTP Issues Today

2012-11-21 Thread Ask Bjørn Hansen
On Nov 20, 2012, at 13:00, Darius Jahandarie djahanda...@gmail.com wrote: Hi everyone, I run the NTP Pool system - http://www.pool.ntp.org/ - so I have some opinions on some of this. :-) But beyond that, I'm honestly rather curious what server selections are a good idea. A first thought

Re: NTP Issues Today

2012-11-20 Thread Sid Rao
, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today In message cameggd4cdqwhxqe_jbvpnr-pkke9lxqa+kzj97anhfonjwz...@mail.gmail.com , Van Wolfe writes: Hello, Did anyone else experience issues with NTP today? We had our server times update to the year 2000

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
In a message written on Mon, Nov 19, 2012 at 04:21:55PM -0700, Van Wolfe wrote: Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. I'm surprised the various time geeks aren't all posting their logs, so

Re: NTP Issues Today

2012-11-20 Thread Steve Meuse
On Tue, Nov 20, 2012 at 11:38 AM, Leo Bicknell bickn...@ufp.org wrote: If your machines switched dates yesterday it probably means you're NTP infrastructure is insufficiently peered and diversified. If you take anything away from this thread, this is it -Steve

Re: NTP Issues Today

2012-11-20 Thread Seth Mattinen
On 11/19/12 6:08 PM, Wallace Keith wrote: Just got paged with a pbx alarm that had 1970 as the year. By the time I logged in , it was showing 2012. Using GPS for time and date. I use GPS for my NTP server and didn't notice anything, but it's PPS disciplined after initial sync so it

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
After some private replies, I'm going to reply to my own post with some information here. It appears many people don't understand how the NTP protocol works. I suspect many people have configured a primary and a backup NTP server on many of their devices. It turns out this is the _WORST_

Re: NTP Issues Today

2012-11-20 Thread Jay Ashworth
- Original Message - From: Leo Bicknell bickn...@ufp.org To protect against two falseticking servers (tick and tock, as we saw on the 19th) you need _FIVE_ servers minimum configured if they are both in the list. More importantly, if you want to protect against a source (GPS, CDMA,

Re: NTP Issues Today

2012-11-20 Thread Jared Mauch
On Nov 20, 2012, at 2:28 PM, Jay Ashworth j...@baylink.com wrote: - Original Message - From: Leo Bicknell bickn...@ufp.org To protect against two falseticking servers (tick and tock, as we saw on the 19th) you need _FIVE_ servers minimum configured if they are both in the list.

Re: NTP Issues Today

2012-11-20 Thread Leo Bicknell
In a message written on Tue, Nov 20, 2012 at 02:28:19PM -0500, Jay Ashworth wrote: I'm curious, Leo, what your internal setup looks like. Do you have an internal pair of masters, all slaved to those externals and one another, with your machines homed to them? Full mesh? Or something else?

Re: NTP Issues Today

2012-11-20 Thread George Herbert
On Nov 20, 2012, at 11:39 AM, Jared Mauch ja...@puck.nether.net wrote: . I've also been looking at an item like this: http://www.netburnerstore.com/ProductDetails.asp?ProductCode=PK70EX-NTP which is about $300 + misc parts. Should be well worth it to avoid a 'major outage' that

Re: NTP Issues Today

2012-11-20 Thread Darius Jahandarie
On Tue, Nov 20, 2012 at 3:15 PM, Leo Bicknell bickn...@ufp.org wrote: For small players, less than 4 sites, typically just use the NTP pool servers, configuring 4 per box minimum. If you want the same protection I just outlined in the paragraph before, make 4 of your servers talk to the

Re: NTP Issues Today

2012-11-20 Thread Mike Lyon
I usually use time.nist.gov. On Tue, Nov 20, 2012 at 1:00 PM, Darius Jahandarie djahanda...@gmail.comwrote: On Tue, Nov 20, 2012 at 3:15 PM, Leo Bicknell bickn...@ufp.org wrote: For small players, less than 4 sites, typically just use the NTP pool servers, configuring 4 per box minimum. If

Re: NTP Issues Today

2012-11-20 Thread Jared Mauch
On Nov 20, 2012, at 4:00 PM, Darius Jahandarie djahanda...@gmail.com wrote: Choosing the first four servers is usually pretty straightforward: *.CC.pool.ntp.org But beyond that, I'm honestly rather curious what server selections are a good idea. A first thought would be an adjacent

Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread Jay Ashworth
- Original Message - From: Darius Jahandarie djahanda...@gmail.com Choosing the first four servers is usually pretty straightforward: *.CC.pool.ntp.org But beyond that, I'm honestly rather curious what server selections are a good idea. A first thought would be an adjacent country,

Re: Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread George Herbert
On Tue, Nov 20, 2012 at 1:53 PM, Jay Ashworth j...@baylink.com wrote: For myself, I usually pick the first three in us.pool.ntp.org, tick and tock, time.nist.gov, and a couple of regionally appropriate large universities. As this week indicated, perhaps tick and tock are not sufficiently

Re: Picking outside NTP servers (Re: NTP Issues Today)

2012-11-20 Thread Majdi S. Abbas
On Tue, Nov 20, 2012 at 04:53:39PM -0500, Jay Ashworth wrote: For myself, I usually pick the first three in us.pool.ntp.org, tick and tock, time.nist.gov, and a couple of regionally appropriate large universities. I'd advise going through the RR for a while, and pick servers close to

Re: NTP Issues Today

2012-11-20 Thread Jimmy Hess
On 11/19/12, Van Wolfe vanwo...@gmail.com wrote: Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. Are you sure that you are actually using NTP to set your clock? For you to sync with 2000, you should

Re: NTP Issues Today

2012-11-20 Thread Darius Jahandarie
On Tue, Nov 20, 2012 at 7:49 PM, Jimmy Hess mysi...@gmail.com wrote: Are you sure that you are actually using NTP to set your clock? For you to sync with 2000, you should have had multiple confused peers from multiple time sources; possibly a false radio signal NTP by default has a

Re: NTP Issues Today

2012-11-20 Thread Damian Menscher
On Tue, Nov 20, 2012 at 4:49 PM, Jimmy Hess mysi...@gmail.com wrote: On 11/19/12, Van Wolfe vanwo...@gmail.com wrote: Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. Are you sure that you are

Re: NTP Issues Today

2012-11-20 Thread Alvaro Pereira
Looks like something bad has happened: Behind the Random NTP Bizarreness of Incorrect Year Being Set https://isc.sans.edu/diary.html?nstoryid=14548 --- A few people have written in within the past 18 hours about their NTP server/clients getting set to the year 2000. The cause of this behavior is

Re: NTP Issues Today

2012-11-20 Thread Blake Dunlap
That's what happens when you just follow vendor recommendations blindly. If you do follow that on vm's (which can actually be a good practice), make sure they pull from your own time infrastructure, and not just the world at large, and that those servers behave in a sane fashion with regard to

Re: NTP Issues Today

2012-11-20 Thread George Herbert
As a reminder - time infrastructure is not recommended for virtualization. Make them physicals. On Tue, Nov 20, 2012 at 5:03 PM, Blake Dunlap iki...@gmail.com wrote: That's what happens when you just follow vendor recommendations blindly. If you do follow that on vm's (which can actually be a

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
--- vanwo...@gmail.com wrote: From: Van Wolfe vanwo...@gmail.com Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. - You need to provide more information. For

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
On 11/19/12 6:32 PM, Scott Weeks sur...@mauigateway.com wrote: --- vanwo...@gmail.com wrote: From: Van Wolfe vanwo...@gmail.com Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012.

Re: NTP Issues Today

2012-11-19 Thread Scott Weeks
--- wbai...@satelliteintelligencegroup.com wrote: From: Warren Bailey wbai...@satelliteintelligencegroup.com Or you could just concede the fact that the navy is playing with time travel again. -- To finish this thread off for the

Re: NTP Issues Today

2012-11-19 Thread Warren Bailey
Or you could just concede the fact that the navy is playing with time travel again. From my Galaxy Note II, please excuse any mistakes. Original message From: Scott Weeks sur...@mauigateway.com Date: 11/19/2012 3:52 PM (GMT-08:00) To: nanog@nanog.org Subject: Re: NTP

Re: NTP Issues Today

2012-11-19 Thread Mark Andrews
In message cameggd4cdqwhxqe_jbvpnr-pkke9lxqa+kzj97anhfonjwz...@mail.gmail.com , Van Wolfe writes: Hello, Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. Thanks, Van NTP should be immune from

RE: NTP Issues Today

2012-11-19 Thread Wallace Keith
: NTP Issues Today In message cameggd4cdqwhxqe_jbvpnr-pkke9lxqa+kzj97anhfonjwz...@mail.gmail.com , Van Wolfe writes: Hello, Did anyone else experience issues with NTP today? We had our server times update to the year 2000 at around 3:30 MT, then revert back to 2012. Thanks, Van NTP should

Re: NTP Issues Today

2012-11-19 Thread George Herbert
logged in , it was showing 2012. Using GPS for time and date. -Original Message- From: Mark Andrews [mailto:ma...@isc.org] Sent: Monday, November 19, 2012 8:42 PM To: Van Wolfe Cc: nanog@nanog.org Subject: Re: NTP Issues Today In message cameggd4cdqwhxqe_jbvpnr-pkke9lxqa