Re: NIST NTP servers
Harlan Stenn writes: > Sharon Goldberg writes: > > Well, if you really want to learn about the NTP servers a target is using > > you can always just sent them a regular NTP timing query (mode 3) and just > > read off the IP address in the reference ID field of the response (mode 4). > > Unless the server is an IPv6 server. This trick only works for IPv4. > > And we have a fix for all of this that will be out soon. Also, the attacker will need to know the correct origin timestamp for the brief window where that attack will work, and even if this happens either the client or the server will see syslog entries alerting to the abuse (if folks are running new enough versions of ntpd). H
Re: NIST NTP servers
Sharon Goldberg writes: > Well, if you really want to learn about the NTP servers a target is using > you can always just sent them a regular NTP timing query (mode 3) and just > read off the IP address in the reference ID field of the response (mode 4). Unless the server is an IPv6 server. This trick only works for IPv4. And we have a fix for all of this that will be out soon. -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: NIST NTP servers
On Wed, 11 May 2016 17:23:31 -0700, Eric Kuhnke said: > average of $150/mo x 500 = $75,000 Id worry more about the fact that somebody is willing to spend $75K/mo to attack me than the fact that it might be possible to wiggle my time base a bit. At that point, you *really* have to worry about other, more productive attacks, such as social engineering (including spear phishing and bribery), or obtaining a 0-day against something in your network. Remember that the FBI has basically admitted that the most effective means of uncloaking a Tor user has been browser 0-days, even though decloaking by pwning a majority of the servers is possible (Seriously - if *you* had $75K/mo to attack somebody, what would *you* spend it on?) pgpaZojwN0Y_I.pgp Description: PGP signature
Re: NIST NTP servers
Wed, May 11, 2016 at 05:20:28PM -0700, Scott Weeks wrote: > --- m...@beckman.org wrote: >> From: Mel Beckman >> >> Accurate time to the millisecond is pretty much >> essential for any network troubleshooting. Say >> you want to diagnose a SIP problem. You collect >> transaction logs from both phones, the VoIP >> gateway, and the PBX. Now you try to merge them >> to derive the sequence of events. You NEED >> millisecond accuracy. >> > > If all logs are sent to a unix server that does > syslogd the log entries would go into the file > in order no matter what timestamp is on them. Modulo latencies and jitter from different machines to the log server. Millisecond precision can be harmed by this, easily. Especially by jitter and order of millisecond here isn't something non-existent in a long-distant networks. -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
Re: NIST NTP servers
maybe try with an odroid? On May 11, 2016 8:45 PM, "Jon Meek" wrote: > A note on using a Raspberry Pi as a NTP server. In my limited home lab > testing the RPi server had enough instability that Internet time sources > were always preferred by my workstation after ntpd had been running for a > while. Presumably this was due to the RPi's clock frequency drifting. At > some point I will look at it again. > > If you do want to build your own Stratum 1 server you might want to glance > at: > > https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf > > and the references there. > > I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but > the test device was clearly not up to the job. At some point I hope to > revisit this and do some more testing like I did for that poster. I'll add > in a CDMA server and a dedicated WWVB receiver. > > Jon >
Re: NIST NTP servers
A note on using a Raspberry Pi as a NTP server. In my limited home lab testing the RPi server had enough instability that Internet time sources were always preferred by my workstation after ntpd had been running for a while. Presumably this was due to the RPi's clock frequency drifting. At some point I will look at it again. If you do want to build your own Stratum 1 server you might want to glance at: https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf and the references there. I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but the test device was clearly not up to the job. At some point I hope to revisit this and do some more testing like I did for that poster. I'll add in a CDMA server and a dedicated WWVB receiver. Jon
Re: NIST NTP servers
With the caveat that if some of the servers are inside your own private network then learning who the servers are might be less useful. But this could be an issue for targets who use servers that are exclusively on the public internet. On Wed, May 11, 2016 at 3:15 PM, Sharon Goldberg wrote: > Well, if you really want to learn about the NTP servers a target is using > you can always just sent them a regular NTP timing query (mode 3) and just > read off the IP address in the reference ID field of the response (mode 4). > > > Reference ID reveals the target that the client is sync'd to. > > If you do this over and over as the client changes the servers it sync's > to, you learn all the servers. > > Or if you are really keen you can use our "kiss-of-death" attack to learn > all the servers a client is willing to take time from. See sections V.B-V.C > of our paper. > > https://eprint.iacr.org/2015/1020.pdf > > Sharon > > > > On Wed, May 11, 2016 at 3:07 PM, Florian Weimer wrote: > >> * Chris Adams: >> >> > First, out of the box, if you use the public pool servers (default >> > config), you'll typically get 4 random (more or less) servers from the >> > pool. There are a bunch, so Joe Random Hacker isn't going to have a >> > high chance of guessing the servers your system is using. >> >> A determined attacker will just run servers in the official pool. >> >> > > > -- > Sharon Goldberg > Computer Science, Boston University > http://www.cs.bu.edu/~goldbe > -- Sharon Goldberg Computer Science, Boston University http://www.cs.bu.edu/~goldbe
Re: NIST NTP servers
Well, if you really want to learn about the NTP servers a target is using you can always just sent them a regular NTP timing query (mode 3) and just read off the IP address in the reference ID field of the response (mode 4). Reference ID reveals the target that the client is sync'd to. If you do this over and over as the client changes the servers it sync's to, you learn all the servers. Or if you are really keen you can use our "kiss-of-death" attack to learn all the servers a client is willing to take time from. See sections V.B-V.C of our paper. https://eprint.iacr.org/2015/1020.pdf Sharon On Wed, May 11, 2016 at 3:07 PM, Florian Weimer wrote: > * Chris Adams: > > > First, out of the box, if you use the public pool servers (default > > config), you'll typically get 4 random (more or less) servers from the > > pool. There are a bunch, so Joe Random Hacker isn't going to have a > > high chance of guessing the servers your system is using. > > A determined attacker will just run servers in the official pool. > > -- Sharon Goldberg Computer Science, Boston University http://www.cs.bu.edu/~goldbe
Re: NIST NTP servers
> On May 11, 2016, at 5:42 PM, Scott Weeks wrote: > > Wouldn't the buffers empty in a FIFO manner? They will empty in whatever order the implementation decides to write them. But what's more important is the order in which the incoming packets are presented to the syslogd process. If you're listening on TCP connections, the receive order is very much determined by the strategy the syslogd implementation uses to read from FDs with available data. I.e. elevator scan, lowest/highest first, circular queue, ... In a threaded implementation, your reader workers, buffer writers, etc., are all at the mercy of the threading implementation; it's difficult to control thread dispatch ordering at that level of granularity. --lyndon
Re: NIST NTP servers
Yo Scott! On Wed, 11 May 2016 17:42:34 -0700 "Scott Weeks" wrote: > > If all logs are sent to a unix server that does > > syslogd the log entries would go into the file > > in order no matter what timestamp is on them. > > syslogd can have quite large buffers. > --- > > > Wouldn't the buffers empty in a FIFO manner? Nope, each local syslog waits until its local buffer is full, or timed out, then sends to the main syslog server. The default flush_timeout() for syslog-ng is 10 seconds. That is a long time when debugging SIP. http://www.syslog.org/syslog-ng/v2/ RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 pgpBtkPYjkTeV.pgp Description: OpenPGP digital signature
Re: NIST NTP servers
--- g...@rellim.com wrote: From: "Gary E. Miller" Yo Scott! On Wed, 11 May 2016 17:20:28 -0700 "Scott Weeks" wrote: > If all logs are sent to a unix server that does > syslogd the log entries would go into the file > in order no matter what timestamp is on them. syslogd can have quite large buffers. --- Wouldn't the buffers empty in a FIFO manner? scott
Re: NIST NTP servers
Yo Scott! On Wed, 11 May 2016 17:20:28 -0700 "Scott Weeks" wrote: > If all logs are sent to a unix server that does > syslogd the log entries would go into the file > in order no matter what timestamp is on them. syslogd can have quite large buffers. RGDS GARY --- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 g...@rellim.com Tel:+1 541 382 8588 pgpByl5us85nK.pgp Description: OpenPGP digital signature
Re: NIST NTP servers
Compared to the scale of the budget of small research projects run by national intelligence agency sized organizations, you wouldn't have to be very well funded to run a sizeable proportion of all tor exit nodes with some degree of plausible deniability... 500 credit cards 500 unique bililng names/addresses and sets of contact info spread 500 1U servers around the world in as many geographically unique locations as you can find, with every dedicated hosting/colo company... average of $150/mo x 500 = $75,000 On Wed, May 11, 2016 at 5:08 PM, wrote: > On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said: > > * Chris Adams: > > > > > First, out of the box, if you use the public pool servers (default > > > config), you'll typically get 4 random (more or less) servers from the > > > pool. There are a bunch, so Joe Random Hacker isn't going to have a > > > high chance of guessing the servers your system is using. > > > > A determined attacker will just run servers in the official pool. > > Such attacks have allegedly been attempted against Tor by certain > very well funded adversaries. > > Thus my statement that if you're seeing that scale attack on your time > sources, the fact that your time source is being attacked is the *least* > of your problems... >
Re: NIST NTP servers
--- m...@beckman.org wrote: From: Mel Beckman Accurate time to the millisecond is pretty much essential for any network troubleshooting. Say you want to diagnose a SIP problem. You collect transaction logs from both phones, the VoIP gateway, and the PBX. Now you try to merge them to derive the sequence of events. You NEED millisecond accuracy. If all logs are sent to a unix server that does syslogd the log entries would go into the file in order no matter what timestamp is on them. scott
Re: NIST NTP servers
On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said: > * Chris Adams: > > > First, out of the box, if you use the public pool servers (default > > config), you'll typically get 4 random (more or less) servers from the > > pool. There are a bunch, so Joe Random Hacker isn't going to have a > > high chance of guessing the servers your system is using. > > A determined attacker will just run servers in the official pool. Such attacks have allegedly been attempted against Tor by certain very well funded adversaries. Thus my statement that if you're seeing that scale attack on your time sources, the fact that your time source is being attacked is the *least* of your problems... pgplgKIyYlw3x.pgp Description: PGP signature
Re: NIST NTP servers
Cellular carriers also use GPS timing for many reasons that are not readily apparent at the layer 3 router/IP/BGP network level. One big need is RF related, back-to-back sector antenna frequency re-use with GPS synced timing on the remote radio heads, such as an ABAB configuration on a tower or rooftop site. The same with some much less costly near consumer grade WISP radio platforms and PTP radio systems nowadays. In such a configuration the GPS timing signal from the local GPS receiver (small cone shaped or puck antennas at the site) is actually the primary, and whatever NTP-based GPS signal the network node might have access to is secondary. On Wed, May 11, 2016 at 12:10 PM, Mel Beckman wrote: > No, many cell carriers run their own completely independent timing > networks. I support some head-ends where they have rubidium clocks and a > T1-delivered time source. They do reference GPS, and many cell sites have > GPS as a backup clock (you can see their conical antennas on the very top > of the tower). But most cellular providers are very protective of their > time sources. They’re also typically clocking SONET networks too, which > requires BITS. > > -mel > > > JAshworth said: > > CDMA and GSM are false diversity: both network types nodes *get their > time* > > from GPS, so far as I know. > > > > On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote: > > > > On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said: > > > >> CDMA and GSM are false diversity: both network types nodes *get their > time* > >> from GPS, so far as I know. > > > > I'll make the fairly reasonable assumption that most readers of this > list have > > networks that span multiple buildings. > > > > If somebody is managing to figure out that you have a GPS in Building > 37, and a > > GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks > at > > other locations and getting close enough to all of them at the same time > to > > conduct a successful spoofing attack, all just to move your time source a > > few seconds off > > > > ... then the fact that GPS is spoofable is probably *NOT* your biggest > > security problem. > > > >
Re: TeamNANOG youtube video seeding
On Tue, 10 May 2016, james machado wrote: First I am thrilled to see older Nanog meetings making it to youtube. Having said that can the people putting up the files put the Nanog meeting number in the title of the videos to make it easier to search and determine relevance? +1 from me. I also sent a message to the TeamNANOG youtube account with the same request. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: NIST NTP servers
No, many cell carriers run their own completely independent timing networks. I support some head-ends where they have rubidium clocks and a T1-delivered time source. They do reference GPS, and many cell sites have GPS as a backup clock (you can see their conical antennas on the very top of the tower). But most cellular providers are very protective of their time sources. They’re also typically clocking SONET networks too, which requires BITS. -mel JAshworth said: > CDMA and GSM are false diversity: both network types nodes *get their time* > from GPS, so far as I know. > On May 11, 2016, at 10:54 AM, valdis.kletni...@vt.edu wrote: > > On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said: > >> CDMA and GSM are false diversity: both network types nodes *get their time* >> from GPS, so far as I know. > > I'll make the fairly reasonable assumption that most readers of this list have > networks that span multiple buildings. > > If somebody is managing to figure out that you have a GPS in Building 37, and > a > GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at > other locations and getting close enough to all of them at the same time to > conduct a successful spoofing attack, all just to move your time source a > few seconds off > > ... then the fact that GPS is spoofable is probably *NOT* your biggest > security problem. >
Re: NIST NTP servers
* Chris Adams: > First, out of the box, if you use the public pool servers (default > config), you'll typically get 4 random (more or less) servers from the > pool. There are a bunch, so Joe Random Hacker isn't going to have a > high chance of guessing the servers your system is using. A determined attacker will just run servers in the official pool.
Re: CALEA
On Tue, 10 May 2016 17:00:54 -0400, Brian Mengel wrote: AFAIK being able to do a lawful intercept on a specific, named, individual's service has been a requirement for providers since 2007. It's been required for longer than that. The telco I worked for over a decade ago didn't build the infrastructure until the FCC said they were going to stop funding upgrades. That really got 'em movin'. (suddenly "data services" people -- i.e. ME -- weren't redheaded stepchildren.) have never heard of a provider, big or small, being called out for being unable to provide this service when requested. Where existing infrastructure is not already in place (read: T1/BRI/etc.), the telco can take up to 60 days to get that setup. I know more than one telco that used that grace period to actually setup CALEA in the first place. did not perform intercepts routinely. The historic published figures (i've not looked in years) suggest CALEA requests are statistically rare. The NC based telco I worked for had never received an order in the then ~40yr life of the company. The mediation server needed to "mediate" between your customer aggregation box and the LEA is not inexpensive. And also is not the telco's problem. Mediation is done by the LEA or 3rd party under contract to any number of agencies. For example, a telco tap order would mirror the control and voice traffic of a POTS line (T1/PRI channel, etc.) into a BRI or specific T1 channel. (dialup was later added, but wasn't required in my era, so we didn't support it.) We used to test that by tapping a tech's phone. Not having any mediation software, all I could do is "yeap, it's sending data" and listen to the voice channels on a t-berd. --Ricky
Re: NIST NTP servers
On 05/11/2016 07:46 AM, Baldur Norddahl wrote: But would you not need to actually spend three times $300 to get a good redundant solution? While we are there, why not go all the way and get a rubidium standard with GPS sync? Anyone know of a (relatively) cheap solution with NTP output? Ebay has several Symmetricom, Microsemi, Datum, Spectracom, and even Agilent solutions for prices from a few hundred US$ to a couple of thousand US$. Even something like the Agilent Z3801, Z3805, or Z3816 can be found for a few hundred US$. New, these things are in the $10,000+ territory. About the same range as mid-range ethernet gear. I like our SSU2000, personally.
Re: NIST NTP servers
On 05/11/2016 12:05 AM, Joe Klein wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? ... I remember it like it was only four years ago oh, wait We have multiple sync sources ourselves, with a Symmetricom (formerly Datum) SSU2000 setup with a cesium PRS, a rubidium secondary, and an ovenized quartz for tertiary oscillators. SSU2000 architecture is separate control and data planes, with time-sync on a different interface from the LAN-facing NTP NIC. And the control plane is firewalled off from the main LAN. An Agilent (now Symmetricom) Z3816 is secondary. PC and SBC (RasPi, etc) oscillators are just not accurate enough for Stratum 1 standards; at best stratum 3 or 4, even when directly GPS-disciplined (stratum is NOT just a synonym for 'level' as a particular stratum really has stability, precision, and accuracy requirements). WWV plus GPS; GPS as you may or may not be aware, is spoofable and is not as accurate as one might want. Neither is WWV. Good reference for time-nuts is, well, the 'time-n...@febo.com' mailing list. (We're a radio astronomy observatory; accurate time and frequency standards are a must here, especially as position accuracy of radio telescopes approach tens of arcseconds).
Re: NIST NTP servers
On Wed, 11 May 2016 15:36:34 -, "Jay R. Ashworth" said: > CDMA and GSM are false diversity: both network types nodes *get their time* > from GPS, so far as I know. I'll make the fairly reasonable assumption that most readers of this list have networks that span multiple buildings. If somebody is managing to figure out that you have a GPS in Building 37, and a GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at other locations and getting close enough to all of them at the same time to conduct a successful spoofing attack, all just to move your time source a few seconds off ... then the fact that GPS is spoofable is probably *NOT* your biggest security problem. pgpYTudayH1Sq.pgp Description: PGP signature
Re: NIST NTP servers
On Wed, May 11, 2016 at 03:24:43PM +, Jay R. Ashworth wrote: > We're all aware this project is underway, right? > > https://www.ntpsec.org/ Despite the name, I'm not aware of any significant protocol changes. It's just a recent fork of the reference implementation minus the refclocks, which isn't particularly helpful if you /don't/ trust network time sources. Long term, be looking at NTS: https://datatracker.ietf.org/doc/draft-ietf-ntp-network-time-security/ In the meanwhile, I'd recommend something along the following lines: - Several nearby upstream servers configured per time server, per site (As diversely as possible.) - Diverse reference clocks (I run everything from WWV to GPS here.) providing authenticated time to your servers. - That all your time servers in all sites be configured in an authenticated full mesh of symmetric peers, allowing the other sites to provide time to a site that has lost its upstream servers or for whatever reason does not trust them at the moment. And of course, ensure any hosts whose clocks you care about are talking to at least a few of these, and preferably several. I know the common case configuration is either default/ntp-pool, or "we have two time servers in this site and everything just chimes from them," but neither is that great of a configuration. --msa
Re: NIST NTP servers
- Original Message - > From: "Mel Beckman" > Read deeper into the thread and you'll find where I sourced inexpensive > RF-based > NTP servers using CDMA, GSM, and even WWV. All radically different > technologies > that are unlikely to have common failure modes. But yes, buying different > brands can't hurt either. CDMA and GSM are false diversity: both network types nodes *get their time* from GPS, so far as I know. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Re: NIST NTP servers
- Original Message - > From: "Jared Mauch" >> Yes, and properly monitor your ntpd instances. > > And upgrade them. > > Some software distributors don’t ship modern software. if you > are using a distribution packaged ntpd it’s likely old and > difficult to determine its lineage due to how it’s packaged. > > If you’re using Redhat based systems consider using chrony > instead, even the new beta fedora 24 uses 4.2.6 derived code > vs 4.2.8 We're all aware this project is underway, right? https://www.ntpsec.org/ Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Re: NIST NTP servers
On 5/10/16 21:05, Joe Klein wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. I believe this will become a stronger need over time, and apply to more than NTP: http://arstechnica.com/security/2016/02/using-ipv6-with-linux-youve-likely-been-visited-by-shodan-and-other-scanners/ 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams wrote: Once upon a time, Mel Beckman said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using. Second, he'd have to guess at least three to "win". Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread. Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams
RE: NIST NTP servers
-Original Message- >From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Leo Bicknell >Sent: Wednesday, May 11, 2016 9:31 AM >To: nanog@nanog.org >Subject: Re: NIST NTP servers >Personally, my network gets NTP from 14 stratum 1 sources right now. >You, and the hacker, do not know which ones. You have to guess at least >8 to get me to move to your "hacked" time. Good luck. >Redundancy is the solution, not a new single point of failure. GPS can be >part of the redundancy, not a sole solution. This seems like the most reasonable advise. If this truly becomes a concern, I would think IPS vendors could implement signatures to look for bad time. Lots of ways to do this - look for a difference between the IPS realtime and NTP status versus the incoming packets. - look for duplicate NTP responses, or responses that weren't requested - duplicate responses, but with differing TTLs, which might hint at one being spoofed. Chuck
Re: NIST NTP servers
GPS + a cesium or rubidium frequency standard is all you need. Too expensive? Then time isn't important to your organization.
Re: NIST NTP servers
In a message written on Wed, May 11, 2016 at 09:00:54AM -0500, Josh Reynolds wrote: > I hope your receivers aren't all from a single source. I have 4 each ACTS, GPS, and CDMA in my list, agumented with a pair of PTP. Amazingly right now all but 3 are within 2 microsconds of each other, and those three outliers are 10 and 35 microseconds off. That's pretty impressive! I didn't have to buy any of them, because various trustable entities run those infrastructures. Some of the trustable entites are the same ones that send the time up to the GPS satellites. :) -- Leo Bicknell - bickn...@ufp.org PGP keys at http://www.ufp.org/~bicknell/ pgpKDSyi44ype.pgp Description: PGP signature
Re: CALEA
In a message written on Tue, May 10, 2016 at 03:00:59PM -0500, Josh Reynolds wrote: > This is a large list that includes many Tier 1 network operators, > government agencies, and Fortune 500 network operators. > > The silence should be telling. NANOG has a strong self-selection for people who run core routing devices and do things like BGP and peering negotiations with other providers. By contrast, CALEA requirements are generally all met by features deployed at the customer-edge. These groups are often a separate silo from the backbone folks at the largest providers. This is likely the wrong list for asking such questions, and the few who do answer is likely to be smaller providers where people wear multiple hats. -- Leo Bicknell - bickn...@ufp.org PGP keys at http://www.ufp.org/~bicknell/ pgpWM43j2G20q.pgp Description: PGP signature
Re: NIST NTP servers
Josh, Read deeper into the thread and you'll find where I sourced inexpensive RF-based NTP servers using CDMA, GSM, and even WWV. All radically different technologies that are unlikely to have common failure modes. But yes, buying different brands can't hurt either. -mel beckman > On May 11, 2016, at 7:15 AM, Josh Reynolds wrote: > > I hope your receivers aren't all from a single source. > > I was in Iraq when this ( > http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/ > ) happened, which meant I had no GPS guided indirect fire assets for 2 > weeks. > >> On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell wrote: >> In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman >> wrote: >>> All because of misplaced trust in a tiny UDP packet that can worm its way >>> into your network from anywhere on the Internet. >>> >>> I say you’re crazy if you don’t run a GPS-based NTP server, especially >>> given that they cost as little as $300 for very solid gear. Heck, get two >>> or three! >> >> You're replacing one single point of failure with another. >> >> Personally, my network gets NTP from 14 stratum 1 sources right now. >> You, and the hacker, do not know which ones. You have to guess at least >> 8 to get me to move to your "hacked" time. Good luck. >> >> Redundancy is the solution, not a new single point of failure. GPS >> can be part of the redundancy, not a sole solution. >> >> -- >> Leo Bicknell - bickn...@ufp.org >> PGP keys at http://www.ufp.org/~bicknell/
Re: NIST NTP servers
Andreas, Most data centers will require a remotely positioned NTP server, which is actually easier and cheaper than a remotely located active GPS antenna. I have placed the $300 commercial NTP servers in an environmental box on the roof, powering t by PoE, without problems. You don't need a redundant network either, nor redundant power. Just plunk down two of these gizmos, or as I suggested elsewhere, deploy one or more CDMA, GSM, or WWV-based clocks, for as much local infrastructure and signal source diversity as you like (I sourced some of these units earlier in the thread, all well less than $1K each. You pay more for diversity, but you get more too. In response to the several DIYers on this thread: Anyone who thinks they're building a raspberry pi-based GPS NTP server for just $150 is kidding themselves. They forgot to include their labor, which is worth far more than the $300 for a commercial unit. The same goes for people who complain about even the minimal $300 price, forgetting that a commercial product has to pay for marketing, support, and make a profit. External NTP has two kinds of vulnerabilities: the ones we know, and the ones we don't. The ones we know are serious enough the pat the ones we don't should be considered with respect. Maybe diversity in Internet sources is a cure, maybe it isn't. But diverse RF sources is demonstrably more secure than the Internet. My point stands: secure external RF NTP sources are so plentiful that relying on Internet NTP is just plain crazy. -mel beckman > On May 11, 2016, at 7:12 AM, Andreas Ott wrote: > > Hi, > >> Boss: That sounds expensive. How much are we talking? >> IT guy: $300 > > Beware! > > Over the past year we made engineering samples to deploy to datacenters. > The goal was to use GPS and PPS to discipline ntpd appliances and serve > as stratum 1 to other NTP distribution servers without the $5k price tag > of commercial NTP 1RU gear. We also deliberately not pursued the path of > running antenna coax from the roof to a receiver, as that is not an > option in all the datacenters where we need to deploy. > > These appliances were built for lesss than $150 as > > (a) Raspberry-Pi with GPS receiver board > > (b) Garmin 18(x) LVC with DB-9 to an "older whitebox server" > > In my experience, most locations inside datacenters where you have good > power and network connectivity have not "good enough" GPS signal reception > due to servers emitting lots of RF noise in the range 1-2 GHz on the > L-band. A brand new suite in the datacenter had OK GPS quality, but > once we added 20+ racks with 40 servers each, GPS reception was pretty > much gone. This hardware was an active antenna with less than 6 feet of > cabling routed to the top of the network cabling rack. Most smartphones > can run an app to show you the GPS signal on the phone, just walk around > your datacenter and compare the signal. > > The only workable solution was to move the GPS clock to a location > where it had good GPS signal but neither redundant network nor conditioned > power (aka. "on my desk near a south facing window"). It also works pretty > well "in my garage". > > In places where GPS reception is good, you can achieve 10E-06 seconds > accuracy over time even with cheap hardware. If you chose to run the DB-9 > NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor > you can still get better than 10E-03 seconds accuracy. > > > -andreas > -- > Andreas Ott (Time-Nut) K6OTT +1.408.431.8727 andr...@naund.org
Re: CALEA
AFAIK being able to do a lawful intercept on a specific, named, individual's service has been a requirement for providers since 2007. I have never heard of a provider, big or small, being called out for being unable to provide this service when requested. I would be surprised if a national broadband ISP with millions of subs did not have this ability and did not perform intercepts routinely. I would be surprised if a small town providing it's own Internet access or small WISP serving a few hundred customers went through the trouble and expense of being able to provide this service. The mediation server needed to "mediate" between your customer aggregation box and the LEA is not inexpensive. I believe there was talk about "trusted third parties" providing mediation-as-a-service but I do not know if any such entities exist. The logistics of running a mediation server in the cloud and being able to signal from the cloud to the aggregation box to begin a mediation and ensuring that the data exported from the ISP to the cloud to the LEA remained private would seem to be significant but not insurmountable. On Tue, May 10, 2016 at 4:11 PM, Josh Reynolds wrote: > The first rule of prism is... > > > *silence* > > > :) > > On Tue, May 10, 2016 at 3:04 PM, Christopher Morrow > wrote: > > > > > > On Tue, May 10, 2016 at 4:00 PM, Josh Reynolds > wrote: > >> > >> This is a large list that includes many Tier 1 network operators, > >> government agencies, and Fortune 500 network operators > > > > > > no one gets calea requests because prism gets all requests? > > >
Re: NIST NTP servers
I hope your receivers aren't all from a single source. I was in Iraq when this ( http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-on-gps/ ) happened, which meant I had no GPS guided indirect fire assets for 2 weeks. On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell wrote: > In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman > wrote: >> All because of misplaced trust in a tiny UDP packet that can worm its way >> into your network from anywhere on the Internet. >> >> I say you’re crazy if you don’t run a GPS-based NTP server, especially given >> that they cost as little as $300 for very solid gear. Heck, get two or three! > > You're replacing one single point of failure with another. > > Personally, my network gets NTP from 14 stratum 1 sources right now. > You, and the hacker, do not know which ones. You have to guess at least > 8 to get me to move to your "hacked" time. Good luck. > > Redundancy is the solution, not a new single point of failure. GPS > can be part of the redundancy, not a sole solution. > > -- > Leo Bicknell - bickn...@ufp.org > PGP keys at http://www.ufp.org/~bicknell/
Re: NIST NTP servers
Hi, > Boss: That sounds expensive. How much are we talking? > IT guy: $300 Beware! Over the past year we made engineering samples to deploy to datacenters. The goal was to use GPS and PPS to discipline ntpd appliances and serve as stratum 1 to other NTP distribution servers without the $5k price tag of commercial NTP 1RU gear. We also deliberately not pursued the path of running antenna coax from the roof to a receiver, as that is not an option in all the datacenters where we need to deploy. These appliances were built for lesss than $150 as (a) Raspberry-Pi with GPS receiver board (b) Garmin 18(x) LVC with DB-9 to an "older whitebox server" In my experience, most locations inside datacenters where you have good power and network connectivity have not "good enough" GPS signal reception due to servers emitting lots of RF noise in the range 1-2 GHz on the L-band. A brand new suite in the datacenter had OK GPS quality, but once we added 20+ racks with 40 servers each, GPS reception was pretty much gone. This hardware was an active antenna with less than 6 feet of cabling routed to the top of the network cabling rack. Most smartphones can run an app to show you the GPS signal on the phone, just walk around your datacenter and compare the signal. The only workable solution was to move the GPS clock to a location where it had good GPS signal but neither redundant network nor conditioned power (aka. "on my desk near a south facing window"). It also works pretty well "in my garage". In places where GPS reception is good, you can achieve 10E-06 seconds accuracy over time even with cheap hardware. If you chose to run the DB-9 NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor you can still get better than 10E-03 seconds accuracy. -andreas -- Andreas Ott (Time-Nut) K6OTT +1.408.431.8727 andr...@naund.org
third party single pole administrator regimes
This is outside plant related. Ignore if all you do is configure routers (not that there is anything wrong with that.) I would be interested in any network provider's experiences with third party single pole administrator regulatory regimes, such as Connecticut's. Please respond privately and I will summarize if there is any interest. thanks -- Fletcher Kittredge GWI www.gwi.net
RE: NIST NTP servers
On 5/10/2016 at 10:30 AM, "Chuck Church" wrote: > >It doesn't really. Granted there are a lot of CVEs coming out for >NTP the >last year or so. But I just don't think there are that many >attacks on it. >It's just not worth the effort. Changing time on devices is more >an >annoyance than anything, and doesn't necessarily get you into a >device. >Sure you can hide your tracks a little by altering time in logs >and altering >it back, but that's more of an in-depth nation-state kind of >attack, not >going to be a script kiddie kind of thing. Just follow the best >practices >for verifying packet sources and NTP security itself, and you >should be ok. > >Chuck I would argue that the fact the NTP can, and has been, be used in DDoS amplification attacks is a serious concern for using protocol going forward. allan
Re: NIST NTP servers
_Everything_ has vulnerabilities and using _any_ external source opens your network and infrastructure to disruptions. NTP has been used for DDoS amplification attacks recently, but so has DNS and other well known/heavily used protocols. With the right protections, syncing with an external NTP source is perfectly acceptable and safe. Further, it’s generally a good idea to ‘peer’ (not just sync) your NTP servers with a few external sources. This removes the dependence on a single source and helps ensure that your time source agrees with the rest of the world. Peering requires interaction with the owners of the remote site, which establishes a basic level of trust that they’ll provide an accurate and stable service. I’ve attached a diagram (sanitized) of what our NTP service will look like after an upcoming refresh. All external sources are trusted and will be peered. All time devices peer with four other sources to ensure there is always a live source to sync/peer with. A DNS record with round-robin is used for local clients to connect to the local Stratum 2 devices. The Stratum 1 GPS will not be directly accessible by users. /Ryan [cid:5676FF89-CBC8-42F7-84CE-69F431C23E48@int.ancker.net] Ryan Harden Research and Advanced Networking Architect University of Chicago - ASN160 P: 773.834.5441 On May 10, 2016, at 5:48 AM, Steven Miano mailto:mian...@gmail.com>> wrote: NTP has vulnerabilities, so using an external source opens your networks and infrastructure to disruptions. Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict incoming traffic and not rely on volunteers or external entities (which may undergo maintenance or budget issues). My preference is more so something akin to the GLN180PEX (I am not affiliated or paid to endorse this product). It allows you to use commodity hardware (like a decommissioned 1U or several preferably) and creation of ones own reliable internal time source(s). Introducing black boxes into a production (revenue generation or expected services by paying customers) environment is undesirable. From there setting up NTPd, Chronyd, and PTPd is up to you. Relying on satellites may seem like just another external reliance, but the next life is proposing a design life of 12 years. On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas mailto:m...@latt.net>> wrote: On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote: NTP has vulnerabilities that make it generally unsuitable for provider networks. I strongly recommend getting a GPS-based time server. These are as cheap as $300. Here is one I use quite a bit: So how does this stop from distributing time to their customers via NTP? GPS doesn't save the protocol, in particular where the S1 clocks involved are embedded devices with rather coarse clocks and timestamping. --msa -- Miano, Steven M. http://stevenmiano.com
VirginMedia AS5089
Can somebody contact me offline from VirginMedia UK regarding a BGP advertisement for 192.34.50.0/24 originating from AS1849 please Thanks Simon
Re: NIST NTP servers
In a message written on Tue, May 10, 2016 at 08:23:04PM +, Mel Beckman wrote: > All because of misplaced trust in a tiny UDP packet that can worm its way > into your network from anywhere on the Internet. > > I say you’re crazy if you don’t run a GPS-based NTP server, especially given > that they cost as little as $300 for very solid gear. Heck, get two or three! You're replacing one single point of failure with another. Personally, my network gets NTP from 14 stratum 1 sources right now. You, and the hacker, do not know which ones. You have to guess at least 8 to get me to move to your "hacked" time. Good luck. Redundancy is the solution, not a new single point of failure. GPS can be part of the redundancy, not a sole solution. -- Leo Bicknell - bickn...@ufp.org PGP keys at http://www.ufp.org/~bicknell/ pgpZ8nfasXwtV.pgp Description: PGP signature
Re: NIST NTP servers
Tue, May 10, 2016 at 04:59:02PM +0200, Stephane Bortzmeyer wrote: > On Tue, May 10, 2016 at 10:52:28AM -0400, > valdis.kletni...@vt.edu wrote > a message of 37 lines which said: > > > Note that they *do* have motivation to keep it working, simply > > because so much of their *own* gear (from gear for individual > > soldiers all the way to strategic bombers and aircraft carriers) > > wants a working GPS signal... > > Yes, but they may switch it off for civilian use (by going encrypted, > for instance) at any time, if it is better for *their* operations. Civilian signals are already degraded in terms of accuracy (without assisted GPS) and military ones are encrypted (but see [1]). The primary reason for encryption is, by the way, to address spoofing issues for the mil people themselves, but it is also good not to arm the potential enemy with the precise coordinates or to be able to do spoofing for them. And since many civilian, but nonetheless, vital orgs are using civilian parts, it could be hard to shut it off without affecting some parts of the infrastructure. Not that nobody wants that, but it will give an unneeded resonance and could lead to the terrible mess. [1] http://www.gps.gov/technical/codeless/ -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
Re: NIST NTP servers
But would you not need to actually spend three times $300 to get a good redundant solution? While we are there, why not go all the way and get a rubidium standard with GPS sync? Anyone know of a (relatively) cheap solution with NTP output? https://en.wikipedia.org/wiki/Rubidium_standard Regards, Baldur On 2016-05-11 09:24, Mel Beckman wrote: Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke mailto:eric.kuh...@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein mailto:jskl...@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman mailto:m...@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams mailto:c...@cmadams.net>> wrote: Once upon a time, Mel Beckman mailto:m...@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our n
Re: NIST NTP servers
Building a S1 system with RaspberryPis would not fly in most of the corporate/enterprise environments I've worked in (random 'appliances', non-uniformity, and lack of support are all glaring issues). Get a PCIe card with a BNC connector and dual power supplies for life in a data center. For home/hobby use a Garmin 18x LVC and any spare compute is a great project: http://www.catb.org/gpsd/gpsd-time-service-howto.html On Wed, May 11, 2016 at 6:47 AM, Dovid Bender wrote: > What about something like this? > http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html > Has anyone used a Pi to create their own server? > > > On Wed, May 11, 2016 at 3:24 AM, Mel Beckman wrote: > > > Regarding Roland’s reference to time and position spoofing via a hacked > > GPS signal, the hacker has to get physical line of sight to the victim’s > > antenna in order to succeed with this attack. That’s likely within a few > > blocks, if not within a few feet. And a rooftop antenna might require a > > drone attack. And how does the drone get guidance without a reliable GPS > > signal? :) > > > > Eric, I agree that sometimes a site can’t get a GPS signal, but in my > > experience designing data centers, that’s still pretty rare. Many NTP > > systems use an active GPS antenna that can be hundreds of feet away. But > > you can always put the $300 NTP server in an outdoor enclosure and power > it > > with PoE. > > > > There’s always CDMA, GSM, and even WWV for a less-accurate plan B time > > source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t > investigated > > yet: > > > > http://www.beaglesoft.com/celsynhowworks.htm > > > > And their $400 WWV-based Stratum 1 time server: > > > > http://www.beaglesoft.com/radsynreceiver.htm > > > > So if you want non-Internet clock diversity, you can have clock > diversity. > > You just have to pay for it. > > > > -mel > > > > On May 10, 2016, at 9:18 PM, Eric Kuhnke > eric.kuh...@gmail.com>> wrote: > > > > For quite some time, in debian the default configuration for the > ntpd.conf > > that ships with the package for the ntpd is to poll from four different, > > semi-randomly assigned DNS pool based sources. I believe the same is true > > for redhat/centos. > > > > In the event that one out of four sources is wildly wrong the ntpd will > > ignore it. > > > > If people have routers/networking equipment inside their network that > only > > supports retrieving ntp from one IP address (or hostname) and have > manually > > configured it to request time from a single external source, not their > own > > internal ntpd that is <10ms away, bad things could definitely happen. > > > > It is worthwhile to have both polling from external sources via IP as > well > > as GPS sync. Many locations in a network have no hope of getting a GPS > > signal or putting an antenna with a clear view to the sky, but may be on > a > > network segment that is <4ms away from many other nodes where you can > > colocate a 1U box and GPS antenna. > > > > On Tue, May 10, 2016 at 9:05 PM, Joe Klein > jskl...@gmail.com>> wrote: > > > > Is this group aware of the incident with tock.usno.navy.mil & > > tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost > 12 > > years for the period of one hour, then return? > > > > The reasons were not fully explained, but the impact was global. Routers, > > switches, power grids, phone systems, certificates, encryption, Kerberos, > > logging and any tightly coupled transaction systems were impacted. > > > > So I began doing 'security research' on the topic (don't confuse me with > > joe hacker), and discovered both interesting and terrifying issues, > which I > > will not disclose on an open forum. > > > > Needless to say, my suggestions are: > > 1. Configure a trusted time source and good time stratum architecture for > > your organization. > > 2. When identifying your source of time, the majority of the technologies > > can be DDOS'ed, spoofed or MITM, so consider using redundant sources and > > authentication. > > 3. For distribution of time information inside your organization, ensure > > your critical systems (Encryption, PKI, transactions, etc) are using your > > redundant sources and authentication. > > 4. Operating systems, programming languages, libraries, and applications > > are sensitive to time changes and can fail in unexpected ways. Test them > > before it's too late. > > 5. Disallow internal system to seek NTP from other sources beyond your > edge > > routers. > > 6. All core time systems should be monitored by your security team or > SOC. > > > > One question, is this a topic anyone would find interested at a future > > NANOG? Something like "Hacking and Defending time?". > > > > > > Joe Klein > > "Inveniam viam aut faciam" > > > > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 > > > > On Tue, May 10, 2016 at 9:59 PM, Mel Beckman > m...@beckman.org>> wrote: > > > > I don't pretend to know all the ways a hacker can find out what nap > > ser
Re: NIST NTP servers
What about something like this? http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html Has anyone used a Pi to create their own server? On Wed, May 11, 2016 at 3:24 AM, Mel Beckman wrote: > Regarding Roland’s reference to time and position spoofing via a hacked > GPS signal, the hacker has to get physical line of sight to the victim’s > antenna in order to succeed with this attack. That’s likely within a few > blocks, if not within a few feet. And a rooftop antenna might require a > drone attack. And how does the drone get guidance without a reliable GPS > signal? :) > > Eric, I agree that sometimes a site can’t get a GPS signal, but in my > experience designing data centers, that’s still pretty rare. Many NTP > systems use an active GPS antenna that can be hundreds of feet away. But > you can always put the $300 NTP server in an outdoor enclosure and power it > with PoE. > > There’s always CDMA, GSM, and even WWV for a less-accurate plan B time > source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated > yet: > > http://www.beaglesoft.com/celsynhowworks.htm > > And their $400 WWV-based Stratum 1 time server: > > http://www.beaglesoft.com/radsynreceiver.htm > > So if you want non-Internet clock diversity, you can have clock diversity. > You just have to pay for it. > > -mel > > On May 10, 2016, at 9:18 PM, Eric Kuhnke eric.kuh...@gmail.com>> wrote: > > For quite some time, in debian the default configuration for the ntpd.conf > that ships with the package for the ntpd is to poll from four different, > semi-randomly assigned DNS pool based sources. I believe the same is true > for redhat/centos. > > In the event that one out of four sources is wildly wrong the ntpd will > ignore it. > > If people have routers/networking equipment inside their network that only > supports retrieving ntp from one IP address (or hostname) and have manually > configured it to request time from a single external source, not their own > internal ntpd that is <10ms away, bad things could definitely happen. > > It is worthwhile to have both polling from external sources via IP as well > as GPS sync. Many locations in a network have no hope of getting a GPS > signal or putting an antenna with a clear view to the sky, but may be on a > network segment that is <4ms away from many other nodes where you can > colocate a 1U box and GPS antenna. > > On Tue, May 10, 2016 at 9:05 PM, Joe Klein jskl...@gmail.com>> wrote: > > Is this group aware of the incident with tock.usno.navy.mil & > tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 > years for the period of one hour, then return? > > The reasons were not fully explained, but the impact was global. Routers, > switches, power grids, phone systems, certificates, encryption, Kerberos, > logging and any tightly coupled transaction systems were impacted. > > So I began doing 'security research' on the topic (don't confuse me with > joe hacker), and discovered both interesting and terrifying issues, which I > will not disclose on an open forum. > > Needless to say, my suggestions are: > 1. Configure a trusted time source and good time stratum architecture for > your organization. > 2. When identifying your source of time, the majority of the technologies > can be DDOS'ed, spoofed or MITM, so consider using redundant sources and > authentication. > 3. For distribution of time information inside your organization, ensure > your critical systems (Encryption, PKI, transactions, etc) are using your > redundant sources and authentication. > 4. Operating systems, programming languages, libraries, and applications > are sensitive to time changes and can fail in unexpected ways. Test them > before it's too late. > 5. Disallow internal system to seek NTP from other sources beyond your edge > routers. > 6. All core time systems should be monitored by your security team or SOC. > > One question, is this a topic anyone would find interested at a future > NANOG? Something like "Hacking and Defending time?". > > > Joe Klein > "Inveniam viam aut faciam" > > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 > > On Tue, May 10, 2016 at 9:59 PM, Mel Beckman m...@beckman.org>> wrote: > > I don't pretend to know all the ways a hacker can find out what nap > servers a company uses, but I can envision a virus that could do that > once > behind a firewall. Every ntp response lists the current reference ntp > server in the next higher stratum. There are many ways that process could > harvest all ntp servers over time, and then pass the public IP back to a > mother ship controller. It could be going on right now. > > My point is, when the fix is so cheap, why put up with this risk at all? > > -mel beckman > > On May 10, 2016, at 5:18 PM, Chris Adams c...@cmadams.net>> wrote: > > Once upon a time, Mel Beckman mailto:m...@beckman.org>> > said: > Boss: So how did a hacker get in and crash our accounting server, > break > our VPNs, and kill our network perform
Re: NIST NTP servers
Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke mailto:eric.kuh...@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein mailto:jskl...@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman mailto:m...@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams mailto:c...@cmadams.net>> wrote: Once upon a time, Mel Beckman mailto:m...@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe R
Re: NIST NTP servers
Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke mailto:eric.kuh...@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein mailto:jskl...@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman mailto:m...@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams mailto:c...@cmadams.net>> wrote: Once upon a time, Mel Beckman mailto:m...@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe R
Re: NIST NTP servers
Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke mailto:eric.kuh...@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein mailto:jskl...@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman mailto:m...@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams mailto:c...@cmadams.net>> wrote: Once upon a time, Mel Beckman mailto:m...@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe R