Re: [ntp:questions] Thoughts on KOD
Harlan Stenn st...@ntp.org wrote: Jason Rabel writes: There are two obvious ways to go for an embedded client. One way would be to use the sntp code as the base. The other would be to either use the current NTP codebase and use the configure options to disable all the refclocks and anything else you didn't want, or wait until we're done with the post-4.2.8 rewrite. For post-4.2.8, we're looking at having a client core with any refclock code being handled a separate process. I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS)... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. That hasn't been my experience. But if those platforms need switches people just need to send in the patches so we can have configure.ac handle them. I'm happy to help out. I don't think it is true at all. I have compiled several existing programs on a Raspberry Pi and I have not encountered any particular problems that one would not see when compiling a program that has only ever been tried on 64-bit Intel on a 32-bit Intel system. As the ARM is usually little-endian, even endianness issues don't arise. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
John Hasler writes: I wrote: Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? Brian Inglis writes: AIUI an updated v4 sntp client will be released to replace ntpdate. How about specifying the sntp core as an RFC for an embedded NTP client? That would be even better. Designers could then claim they have to use it to be standards compliant. Network Time Foundation is also working on some Certification and Compliance programs. If we can get these up and running (it doesn't happen for free) that should also help a lot. -- Harlan Stenn st...@ntp.org http://networktimefoundation.org - be a member! ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Harlan Stenn st...@ntp.org wrote: Rob writes: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. However, it has so many broken and inconsistent bits that some clients believe that they have received a corrupted packet and decide to re-try the request to see if that results in a better response. What are these many broken and inconsistent bits of which you speak? Strange stratum, both leap bits set, etc. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Mon, Jul 07, 2014 at 07:04:01PM +0200, Jan Ceuleers wrote: I'm not sure why sending the requester's timestamp back to him is better than an immutable timestamp. The effect of the former is slow drift, the effect of the latter is (I suspect) no lock at all due to the lack of passage of time. So I think that the latter is more likely to catch the admin's eye. If there is an admin. I think most clients check at least one of the stratum/leap fields and don't use the time stamps from a KOD response to actually update their clock. If the KOD response was modified to set the leap and stratum bits as synchronized, the client would drift slowly away, but ntpd would need to stick to it and never send the client correct time. I agree that purposely serving bad time might be the best way how to get an attention of the user and get the NTP implementation fixed if it can be identified reliably and no innocent clients behind that IP adress are harmed. The identification could be improved, for example by monitoring the distribution of the client's polling interval as simple clients use a fixed interval, but I'm not sure if it's possible to make it so reliable that ntpd could be allowed to send a reponse with purposely bad time. -- Miroslav Lichvar ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Rob writes: Harlan Stenn st...@ntp.org wrote: Rob writes: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. However, it has so many broken and inconsistent bits that some clients believe that they have received a corrupted packet and decide to re-try the request to see if that results in a better response. What are these many broken and inconsistent bits of which you speak? Strange stratum, both leap bits set, etc. The definition of a KoD packet is that the Stratum field is 0, which *by definition* is unspecified/invalid. Neither broken nor inconsistent. If the system sending the KoD sets both leap bits, again, *by definition* that means the sending system is saying my clock is invalid - don't believe it. Neither broken nor inconsistent. If somebody chooses to implement an NTP client that blows past these core definitions in the NTP RFCs that's their fault. We're talking about definitions that are unchanged since NTPv1. I'm not buying your assertion, but I could be missing something. Also, you did say etc and there might be some other cases there that would change my mind. But I doubt it, as S0 is a showstopper, as is LI=3. H ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
There are two obvious ways to go for an embedded client. One way would be to use the sntp code as the base. The other would be to either use the current NTP codebase and use the configure options to disable all the refclocks and anything else you didn't want, or wait until we're done with the post-4.2.8 rewrite. For post-4.2.8, we're looking at having a client core with any refclock code being handled a separate process. I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS)... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/08/2014 12:11 PM, Jason Rabel wrote: There are two obvious ways to go for an embedded client. One way would be to use the sntp code as the base. The other would be to either use the current NTP codebase and use the configure options to disable all the refclocks and anything else you didn't want, or wait until we're done with the post-4.2.8 rewrite. For post-4.2.8, we're looking at having a client core with any refclock code being handled a separate process. I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS)... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. You need to gift wrap it considerable, such that the proliferation of bad-hacked NTPish code gets replaced. Putting a price-tag on it mean that it will prohibit the shift of code, which in itself is a cost. Hobby-hackers already do many first breaks, so why not make sure that their contributions make it into the code such that support for a large range of embedded platforms exists either directly in NTPD or easily accessible port. Another thought is to have people review the NTP/SNTPish code that is out there to see how their complience are, what KOD they would react to and how much effort it would to fix the basics. Then again, the basic problem is that people doesn't upgrade their FW as they should. Listing of implementations, their target environments and versions to use and versions to avoid should maybe be of assistance. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Harlan Stenn st...@ntp.org wrote: Rob writes: Harlan Stenn st...@ntp.org wrote: Rob writes: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. However, it has so many broken and inconsistent bits that some clients believe that they have received a corrupted packet and decide to re-try the request to see if that results in a better response. What are these many broken and inconsistent bits of which you speak? Strange stratum, both leap bits set, etc. The definition of a KoD packet is that the Stratum field is 0, which *by definition* is unspecified/invalid. Neither broken nor inconsistent. If the system sending the KoD sets both leap bits, again, *by definition* that means the sending system is saying my clock is invalid - don't believe it. Neither broken nor inconsistent. What I mean is that those fields are not what the simple client expects to get, and those stupid programmers believe that they can get a correct reply by re-trying the request. If somebody chooses to implement an NTP client that blows past these core definitions in the NTP RFCs that's their fault. We're talking about definitions that are unchanged since NTPv1. Of course when programmers would follow definitions and specifications the whole problem would not exist. But in practice, programmers tweak their code just long enough to get it working during some testing, and they are not prepared to handle errors or replies they never see during testing. I'm not buying your assertion, but I could be missing something. Also, you did say etc and there might be some other cases there that would change my mind. But I doubt it, as S0 is a showstopper, as is LI=3. I have no idea what software was running at the other end and what failure modes it has. I only saw the reactions to obvious things I tried (like dropping requests and sending KOD replies). Things were not looking good. And when I googled for it, I quickly discovered I was not the only one making these observations. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jason Rabel writes: I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS)... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. Debian already builds ntpd on most hardware including ARM and MIPS. Their patches are available, but I should think that they are already sending them upstream. Contact pkg-ntp-maintain...@lists.alioth.debian.org . -- John Hasler jhas...@newsguy.com Dancing Horse Hill Elmwood, WI USA ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jason Rabel wrote: I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS) ... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. Which hardware platforms? The chips and motherboards differ greatly. With which flavors of which OSes? -- E-Mail Sent to this address blackl...@anitech-systems.com will be added to the BlackLists. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jason Rabel writes: There are two obvious ways to go for an embedded client. One way would be to use the sntp code as the base. The other would be to either use the current NTP codebase and use the configure options to disable all the refclocks and anything else you didn't want, or wait until we're done with the post-4.2.8 rewrite. For post-4.2.8, we're looking at having a client core with any refclock code being handled a separate process. I do not know if this is the case with NTP, but quite often it takes considerable hacking of sources to get code to compile on non-x86 embedded hardware (i.e. ARM MIPS)... It would probably help boost usage if someone was assuring NTP sources compile on those platforms without the need for modification. That hasn't been my experience. But if those platforms need switches people just need to send in the patches so we can have configure.ac handle them. I'm happy to help out. H ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 06/07/2014 07:42, Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. Seconded. -- Cheers, David Web: http://www.satsignal.eu ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
detha de...@foad.co.za wrote: There are some differences between open SMTP relays and networks not implementing BCP38. TCP versus UDP, and to quote Paul Vixie from https://queue.acm.org/detail.cfm?id=2578510 ] ... but the big reason why SAV isn't the default is: SAV benefits only ] other people's customers, not an operator's own customers. ] ] There is no way to audit a network from outside to determine if it ] practices SAV. Any kind of compliance testing for SAV has to be done by ] a device that's inside the network whose compliance is in question. That ] means the same network operator who has no incentive in the first place ] to deploy SAV at all is the only party who can tell whether SAV is deployed. That, and mis-configured NTP servers, is why we still have reflection attacks. With SMTP relays the situation was not that different. An open SMTP relay did not do much damage to the owner, it caused a lot of spam to be sent that irritated others. Only by bending the damage to the owner of the server, by blacklisting it everywhere so the regular users could not send mail anymore, the server owners could be convinced to do something. Of course that is sad, but that is reality. With BCP38 (SAV) a similar thing could be done: block certain traffic from neighbors that have been shown not to implement BCP38. Of course it is more difficult because it can not be a simple IP based blocklist, it has to be AS based and has to be effected by peers, who usually have contracts and would not do this kind of thing easily. Without that, NTP server operators are really helpless against attacks, both of their servers and backscatter attacks against innocent victims. But of course it is outside the scope of NTP to do this, it just happens that NTP is a recent victim of this wide misconfiguration problem. The biggest problem with NTP is the amplification factor. With a 1:1 or even 1:1.5 amplification factor, the attacker won't bother, and move to the next target - SNMP is a good candidate. With a 1:12 or better ratio, the attacker is happy. That is no longer true. I have seen attacks with much smaller amplification factors, e.g. using TCP. SYN packets with spoofed sender address and both source and destination set to wellknown ports like 80, 443. This amplifies only a little, but still it is done. I think the source address spoofing problem should be taken care of before it gets completely out of hand. The NTP attacks were only an example. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Mon, 07 Jul 2014 07:50:16 +, Rob wrote: detha de...@foad.co.za wrote: The biggest problem with NTP is the amplification factor. With a 1:1 or even 1:1.5 amplification factor, the attacker won't bother, and move to the next target - SNMP is a good candidate. With a 1:12 or better ratio, the attacker is happy. That is no longer true. I have seen attacks with much smaller amplification factors, e.g. using TCP. SYN packets with spoofed sender address and both source and destination set to wellknown ports like 80, 443. This amplifies only a little, but still it is done. Different attack profile. With an amplification factor of 1 to 2 the purpose of a reflection attack is to hide the attack source (often a few hosts with large pipes), at high PPS rates. With high amplification factors the purpose is to generate a large amount of data using only a small pipe. I think the source address spoofing problem should be taken care of before it gets completely out of hand. The NTP attacks were only an example. Amplification attacks started in earnest with DNS a few years ago, and when major DNS providers (and most implementations) implemented RRL it shifted to NTP. My guess is that it will stay with NTP until either the number of amplifying servers is low enough to be difficult to find, or until a few of the big players get tired of it, and start blocking NTP completely, much like ISPs block TCP/25 on residential networks. BCP38/rpf/SAV will not be implemented soon (if at all) in a lot of networks. Wether one likes it or not, the only practical solution for now is some form of RRL or blacklisting. Both involve keeping state about client requests, either in ntpd or at the IDS/firewall level. So far, ntpd seems to be the easiest place to implement it. -d ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 7/6/2014 2:42 AM, Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. You haven't read the code. Any client that ignores the KOD flag will find (if they ever looked) that their clock will be drifting away further and further from the proper time. When KOD is set the value of the received and sent timestamps are the same as the initial client sent timestamp. It doesn't use the system time for the returned packet. Calculate what this does to the resulting clock. Please also note that there is more than one type of KOD packet. See RFC 5905 Section 7.4. See also Figure 13. You need to clearly distinguish the different ones when talking about them. Most of this discussion seems to be about action a. As discussed above this is an extremely useful feature because any client ignoring the KOD flag and using the packet any way will get pushed way of the actual time that they would normally expect regardless of the client software used. Danny ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
I have to agree on some points with these two below. From my experience also, using KOD usually results in more packet pounding from bad clients (from what I can only assume is poor coding of custom clients). The realization is that many clients don't run the standard NTP distribution, but rather some old hacked down / self-coded minimal NTP / SNTP version to run on embedded hardware. Likewise many of the routers don't even use NTPD code with a constantly running daemon, but rather more along the lines of ntpdate code that cron triggers at regular intervals. Sending a date in the past could trigger the client to treat it as invalid, and it simply will make a request again again, expecting some more legitimate value. Also that could be seen as (unintentionally) malicious. So I do not think that is route we would want to go. I think the best thing to do would be to keep it simple and not reply to a bad client... It might make a few follow-up queries, but eventually it would (hopefully) give up. Adding more code to KOD and increasing its complexity does not seem like a wise choice to me, especially when you would need to consider backwards compatibility, and in the case of bad clients, no compatibility. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. Seconded. I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Another way would be to use a bogus fixed timestamp that is in the past (i.e. one that suggests that there is no passage of time on the server). My recommendation is based on the assumption, yet to be verified in practice, that this server behaviour won't result in worse client behaviour than would be the case if the server just served the client's request as normal. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Has that *always* been the case? Or has the code be changed over time? Remember, not everyone is running the latest development (or even stable) version of NTP... KOD already sets a timestamp that is the requesters timestamp. See my previous response. It's better than your idea since it is gradual. Danny ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 7/6/2014 12:29 PM, Magnus Danielson wrote: Detha, On 07/06/2014 03:23 PM, detha wrote: On Sun, 06 Jul 2014 12:28:09 +, Magnus Danielson wrote: For cases like that, KOD won't help at all. All the state table/KOD/filter rules mitigation approaches I have seen so far are limited to one server. Maybe time to take a look at a DNSBL-type approach for abusive clients; that way, once a client is labeled 'abusive', it will stop working with any of the pool servers that use the blacklist. Policies (for how long to auto-blacklist, how to prevent DoS by blacklisting the competition, how to 'promise to behave and express-delist' etc.) to be defined. Maybe. For the moment I think it is sufficient if we provide a mechanism by which offenders gets reported to *some* system. We *could* also provide a method by which white/black-list can be dynamically set from an external source, so enough hooks exists, but I do not think that NTPD should be burned with the rest of that system. Once NTPD can report it feels offended by a source, and beyond KODing it also report to some external mechanism that could potentially block it by any external means, NTPD does not have to do much more. My point being with this line of thinking is that KOD in itself makes assumptions on the offending source actually respects it, and while KOD rules probably can be improved, it does not provide a very effective means of protection with sources not respecting KOD, and thus we also needs to think i broader terms. In my mind, the defenses is according to these lines: 0) NTPD tolerates a source, packet approval checks 1) NTPD does not tolerate a source, fires of KOD, source is expected to shut up 2) NTPD admin does not tolerate a source, blacklist it, NTPD will drop the traffic 3) NTPD admin does not tolerate a source, filters in in box firewall, box firewall drops the traffic 4) NTPD admin does not tolerate a source, filters in network firewall, network firewall drops the traffic Notice how step 2-4 moves the traffic load further away from NTPD process, interface and eventually subnetwork. What I proposed would allow for automation of these steps. It is reasonable that escalation should be done when a source does not respect KOD and keeps transmitting requests. It is also resonable that blocking times out, such that blocking is removed after some reasonable time, as offenders can be on dynamic addresses, and usually works over limited time when intentional. How to automate step 2-4 is however not a core concern for NTPD, but feeding the data out of NTPD in a way that is handy for such a mechanism is. Separate block-log file as I proposed is probably better than only a syslog file, as it removes the need to parse syslog for matching blocks, but rather can focus on changes in a dedicated file. The experience with blocking has actually being negative and we have seen traffic actually INCREASE after it is blocked because the client, not having received a response, tries more often. This has been observed in the wild. Danny ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 7/6/2014 7:22 AM, Jan Ceuleers wrote: On 07/06/2014 11:23 AM, Rob wrote: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. Sorry, I was not clear. By a 'regular' response I mean one that has a non-zero stratum value. I had actually forgotten that a stratum value of zero indicates that the server is not synchronised (as it is a collision with LI=3, which also means that). So I guess I'm dropping my first suggestion. The second-one stands: pick a non-zero stratum value and report an immutable time stamp. Note that the stratum field occupies 8 bits in the packet format, but currently only values between 0 and 15 are defined (where we have seen that a value of 0 is not uniformly understood by real-life clients). So the choice of stratum value should be in the range 1-15. I have no particular preference for the immutable time stamp value to pick. Could be zero, could be some other meaningful value (such as 0xeee4baadeee4baad - twice Eek! Bad!). KOD already sets a timestamp that is the requesters timestamp. See my previous response. It's better than your idea since it is gradual. Danny ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Mon, Jul 07, 2014 at 08:35:25AM +0100, David Taylor wrote: Seconded. Why remove KOD when it has to be expressly enabled (via restrict kod and limited)? I'd rather see a two tier system, where you can enable the use of KOD beyond the initial rate limit, and a second limit beyond which requests are simply ignored. But I don't understand why anyone would remove functionality that the server administrator has to expressly configure to enable. --msa ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Danny, On 07/07/2014 04:00 PM, Danny Mayer wrote: On 7/6/2014 2:42 AM, Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. You haven't read the code. Any client that ignores the KOD flag will find (if they ever looked) that their clock will be drifting away further and further from the proper time. When KOD is set the value of the received and sent timestamps are the same as the initial client sent timestamp. It doesn't use the system time for the returned packet. Calculate what this does to the resulting clock. Please also note that there is more than one type of KOD packet. See RFC 5905 Section 7.4. See also Figure 13. You need to clearly distinguish the different ones when talking about them. Most of this discussion seems to be about action a. As discussed above this is an extremely useful feature because any client ignoring the KOD flag and using the packet any way will get pushed way of the actual time that they would normally expect regardless of the client software used. Which would make sense if the client has multiple sources and is a relatively decent NTP client. Issues we have seen is outside of the NTP client realm. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/07/2014 04:10 PM, Danny Mayer wrote: The experience with blocking has actually being negative and we have seen traffic actually INCREASE after it is blocked because the client, not having received a response, tries more often. This has been observed in the wild. This might be true for proper NTP clients, but I wonder if this is true for faked NTP requests from DDOSers. KOD fills no purpose for DDOSers, so massive attacks is best handled by dropping that traffic, and possibly push the dropping away from the node and subnet running the server. For more modest overload scenarios as miss-configured or otherwise error-ed NTP clients, I believe that what you describe is correct. Let's not confuse these different scenarios, as they most probably have different solutions. My point was that DDOS amplification/relaying should be considered, as we need that solved, while KOD refinements is maybe nice but addresses another problem. I don't think you will be able to handle the DDOS issues without doing blocking, and you want that blocking to move away from your server in order to reduce the impact of the service. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/07/2014 04:50 PM, Majdi S. Abbas wrote: On Mon, Jul 07, 2014 at 08:35:25AM +0100, David Taylor wrote: Seconded. Why remove KOD when it has to be expressly enabled (via restrict kod and limited)? I'd rather see a two tier system, where you can enable the use of KOD beyond the initial rate limit, and a second limit beyond which requests are simply ignored. But I don't understand why anyone would remove functionality that the server administrator has to expressly configure to enable. I think KOD is fine for it's intended purpose, but it does not solve this other problem we are having. Thus, a two-tire solution is what I advocate for. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 7/7/2014 10:08 AM, Jason Rabel wrote: Has that *always* been the case? Or has the code be changed over time? I'm not sure how far back this goes. I can try and find out. I do remember the discussion I had with Dave Mills on this but that was just a couple of years ago. I will need to find the code that does this and dig into the archives for this one. Remember, not everyone is running the latest development (or even stable) version of NTP... Yep. But this is not a recent development. KOD already sets a timestamp that is the requesters timestamp. See my previous response. It's better than your idea since it is gradual. Danny Danny ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 2014-07-07, Majdi S. Abbas m...@latt.net wrote: On Mon, Jul 07, 2014 at 08:35:25AM +0100, David Taylor wrote: Seconded. Why remove KOD when it has to be expressly enabled (via restrict kod and limited)? I'd rather see a two tier system, where you can enable the use of KOD beyond the initial rate limit, and a second limit beyond which requests are simply ignored. But I don't understand why anyone would remove functionality that the server administrator has to expressly configure to enable. Because it gives the administrator ( who probably does know the ins and outs of the program) a false sense that he is doing something useful by enabling it. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Danny Mayer ma...@ntp.org wrote: On 7/6/2014 2:42 AM, Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. You haven't read the code. Any client that ignores the KOD flag will find (if they ever looked) that their clock will be drifting away further and further from the proper time. When KOD is set the value of the received and sent timestamps are the same as the initial client sent timestamp. It doesn't use the system time for the returned packet. Calculate what this does to the resulting clock. Please also note that there is more than one type of KOD packet. See RFC 5905 Section 7.4. See also Figure 13. You need to clearly distinguish the different ones when talking about them. Most of this discussion seems to be about action a. As discussed above this is an extremely useful feature because any client ignoring the KOD flag and using the packet any way will get pushed way of the actual time that they would normally expect regardless of the client software used. All that does not matter much because the client will usually not notice it and will continue to send the requests you don't like. When KOD would have been part of the standard from the beginning, one could expect that most clients would handle it and exit or warn the admin. However, it was added later, and clients do exist that treat KOD as a mishap that can be corrected by re-trying the request. Immediately, not after 60 seconds or so. So when you reply KOD to every request, you end up getting as many requests as the ping-pong time to the client allows. Of course, that has already been remedied by not replying with KOD every time, but the basic issue still remains. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Danny writes: You haven't read the code. Any client that ignores the KOD flag will find (if they ever looked) that their clock will be drifting away further and further from the proper time. When KOD is set the value of the received and sent timestamps are the same as the initial client sent timestamp. It doesn't use the system time for the returned packet. Calculate what this does to the resulting clock. Yes but the consumer running the router of microwave oven or whatever won't notice so you still won't get rid of them. -- John Hasler jhas...@newsguy.com Dancing Horse Hill Elmwood, WI USA ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jason writes: The realization is that many clients don't run the standard NTP distribution, but rather some old hacked down / self-coded minimal NTP / SNTP version to run on embedded hardware. Likewise many of the routers don't even use NTPD code with a constantly running daemon, but rather more along the lines of ntpdate code that cron triggers at regular intervals. Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? -- John Hasler jhas...@newsguy.com Dancing Horse Hill Elmwood, WI USA ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/07/2014 04:04 PM, Danny Mayer wrote: I have no particular preference for the immutable time stamp value to pick. Could be zero, could be some other meaningful value (such as 0xeee4baadeee4baad - twice Eek! Bad!). KOD already sets a timestamp that is the requesters timestamp. See my previous response. It's better than your idea since it is gradual. Danny, (What's with the mine is better than yours thing?) I'm not sure why sending the requester's timestamp back to him is better than an immutable timestamp. The effect of the former is slow drift, the effect of the latter is (I suspect) no lock at all due to the lack of passage of time. So I think that the latter is more likely to catch the admin's eye. If there is an admin. Jan ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Magnus Danielson mag...@rubidium.dyndns.org wrote: On 07/07/2014 04:10 PM, Danny Mayer wrote: The experience with blocking has actually being negative and we have seen traffic actually INCREASE after it is blocked because the client, not having received a response, tries more often. This has been observed in the wild. This might be true for proper NTP clients, but I wonder if this is true for faked NTP requests from DDOSers. KOD fills no purpose for DDOSers, Those results were already obtained BEFORE the DDOS problem appeared. Early experience in the NTP Pool was that some people run really broken NTP clients (not ntpd but some quickly written Windows program or router firmware code) that do bad things like: - send their requests on the top of the hour or minute - send requests at a high rate (e.g. once every 10-20 seconds) - when a request does not result in a quick reply or the reply is not what the caller expects, quickly retry the request - have no error counting whatsoever (e.g. when you don't reply, the same client is still requesting 3 weeks later at the same interval) All those problems were not solved by sending KOD and some of them even not by sending no reply at all. In fact, some of those broken clients re-try after 1-2 seconds when you don't reply, and when you do reply KOD they immediately re-try. When you reply with correct time they come back after 15 seconds. You pick your alternative. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? Actually, Busybox does have a ntp daemon... Where the code comes from I do not know. I've tried running it on a couple residential-grade routers and to be honest it runs like crap. Running it as a server (in theory to re-distribute time to your lan) is even worse and basically useless and a waste of resources. I can't really say if it is the hardware or software that is the problem because I never bothered to try and diagnose it any deeper. It's been a while since I've looked over any ntp-like code in some of the open source router projects. Most are more concerned with other features than getting the router's clock to nanosecond precision. Like I said before, most are just some hack of ntpdate to get the time and run as a cron job every few hours. I think if the NTP people wanted to help mitigate what most of the headaches issues are out on the net, they would work with the big networking companies to ensure their code is compliant with what is acceptable communications error handling. One small mistake in their code has serious repercussions when they churn out these devices by the tens of thousands (or more) before catching their error... ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Rob wrote:
All those problems were not solved by sending KOD
and some of them even not by sending no reply at all.
In fact, some of those broken clients re-try after 1-2 seconds
when you don't reply,
and when you do reply KOD they immediately re-try.
When you reply with correct time they come back after 15 seconds.
You pick your alternative.
Which product / abused NTP Server / Network / ...
immediately re-tried if they got a KOD?
{I don't recall that one.}
--
E-Mail Sent to this address blackl...@anitech-systems.com
will be added to the BlackLists.
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Hi-- On Jul 7, 2014, at 2:03 PM, E-Mail Sent to this address will be added to the BlackLists Null@BlackList.Anitech-Systems.invalid wrote: Which product / abused NTP Server / Network / ... immediately re-tried if they got a KOD? I've seen that sort of misbehavior from a Lucent / Avaya PBX. (AFAICT for a remote system, anyway.) Regards, -- -Chuck PS: Your anti-spam stuff is annoying. Can't you at least set the Reply-to: header...? ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Rob writes: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. However, it has so many broken and inconsistent bits that some clients believe that they have received a corrupted packet and decide to re-try the request to see if that results in a better response. What are these many broken and inconsistent bits of which you speak? -- Harlan Stenn st...@ntp.org http://networktimefoundation.org - be a member! ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 2014-07-07 10:00, John Hasler wrote: Jason writes: The realization is that many clients don't run the standard NTP distribution, but rather some old hacked down / self-coded minimal NTP / SNTP version to run on embedded hardware. Likewise many of the routers don't even use NTPD code with a constantly running daemon, but rather more along the lines of ntpdate code that cron triggers at regular intervals. Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? AIUI an updated v4 sntp client will be released to replace ntpdate. How about specifying the sntp core as an RFC for an embedded NTP client? -- Take care. Thanks, Brian Inglis ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jochen Bern wrote: The straightforward approach to doing so would be to send out not plain go DIAFs, but messages along the lines of I'm willing to service your further requests *if* you label them with this random ID (and behave). More modern ntpd with the Command Line Option -I, and / or the MiscOpt nic / interface configuration directive, could probably get the MAC address of the interface it using, (or some other uniqueish id) hash it, ... ntpd\ntp_io.clink interface into list of known interfaces ep_univ_iid; /* iface ID from MAC address */ scan_univ_iid; /* see RFC 4291 */ ep_privacy;/* random local iface ID */ scan_privacy; /* see RFC 4941 */ I don't know where you'd stick the data, perhaps in an extension field. Similar to the ipv4 ref clock id lack of resolution issue, when IPv6 is the source? -- E-Mail Sent to this address blackl...@anitech-systems.com will be added to the BlackLists. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jason Rabel wrote:
I think the best thing to do would be to keep it simple
and not reply to a bad client...
It might make a few follow-up queries,
but eventually it would (hopefully) give up.
Broken ones might not {didn't}.
Most of the big abuse issues I read about,
no reply was one of the worse things to do,
it just made the clients hammer faster.
http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse
NETGEAR and the University of Wisconsin–Madison
--
E-Mail Sent to this address blackl...@anitech-systems.com
will be added to the BlackLists.
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
John Hasler wrote: Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? http://wiki.openwrt.org/doc/howto/ntp.client -- E-Mail Sent to this address blackl...@anitech-systems.com will be added to the BlackLists. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
I wrote: Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? Brian Inglis writes: AIUI an updated v4 sntp client will be released to replace ntpdate. How about specifying the sntp core as an RFC for an embedded NTP client? That would be even better. Designers could then claim they have to use it to be standards compliant. -- John Hasler jhas...@newsguy.com Dancing Horse Hill Elmwood, WI USA ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Brian Inglis writes: On 2014-07-07 10:00, John Hasler wrote: Jason writes: The realization is that many clients don't run the standard NTP distribution, but rather some old hacked down / self-coded minimal NTP / SNTP version to run on embedded hardware. Likewise many of the routers don't even use NTPD code with a constantly running daemon, but rather more along the lines of ntpdate code that cron triggers at regular intervals. Would it be useful to offer an official minimal implementation intended for embedded systems so that these people won't feel the need to code their own? Maybe add minimal NTP support to Busybox? AIUI an updated v4 sntp client will be released to replace ntpdate. How about specifying the sntp core as an RFC for an embedded NTP client? There are two obvious ways to go for an embedded client. One way would be to use the sntp code as the base. The other would be to either use the current NTP codebase and use the configure options to disable all the refclocks and anything else you didn't want, or wait until we're done with the post-4.2.8 rewrite. For post-4.2.8, we're looking at having a client core with any refclock code being handled a separate process. H ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/06/2014 08:42 AM, Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. Seconded. I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Another way would be to use a bogus fixed timestamp that is in the past (i.e. one that suggests that there is no passage of time on the server). My recommendation is based on the assumption, yet to be verified in practice, that this server behaviour won't result in worse client behaviour than would be the case if the server just served the client's request as normal. Jan ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. However, it has so many broken and inconsistent bits that some clients believe that they have received a corrupted packet and decide to re-try the request to see if that results in a better response. (of course a programmer that would even try that, will not be clever enough to put a retry limit or an increasing delay in the code. so those badly written clients just start to hammer on the server) Another way would be to use a bogus fixed timestamp that is in the past (i.e. one that suggests that there is no passage of time on the server). Probably that would be better, but of course KOD has already been defined and changing its definition yet again would be risky as well. My recommendation is based on the assumption, yet to be verified in practice, that this server behaviour won't result in worse client behaviour than would be the case if the server just served the client's request as normal. This has to be tested very well. When I was still in the IPv4 NTP pool I had some very bad experiences with KOD. And I was not the only one. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. I'm afraid this is exactly right: KOD is a way to keep honest guys honest, i.e. it only helps against programmers/users why actually try (hard) to do the right thing. Currently it will cause a badly configured ntpd installation (burst + minpoll 4 + maxpoll 4) to possibly stop using any server which sends back KOD, but only if it also uses the pool directive to actively search out the best servers. I don't want to think about users actively trying to generate as much traffic as possible. :-( Terje -- - Terje.Mathisen at tmsw.no almost all programming can be viewed as an exercise in caching ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Magnus Danielson wrote: Harlan, On 07/05/2014 11:40 PM, Harlan Stenn wrote: Folks, I was chatting with PHK about: http://support.ntp.org/bin/view/Dev/NtpProtocolResponseMatrix http://bugs.ntp.org/show_bug.cgi?id=2367 and how we probably want to extend KOD coverage to more than just the limited case. I was assuming folks would want finer-grained control over this behavior, and thought about being able to choose any of kod-limited, kod-noserve, and kod-query. PHK suggested that we consider going the other way - KOD would mean Send KODs whenever appropriate. We want KOD generation to be at worst equally expensive as a normal reply, particularly for a high-performance multi-threaded/wire-speed server. This is actually _very_ hard to do, since a default reply to a client request is by far the most common pattern, something that people have already implemented in FPGAs. 1 Gbit/s corresponds to more than a million request/reply pairs per second, this is just within the realm of the possible for a multi-core/multi-thread server where every thread can handle these requests autonomously, passing more complicated stuff on to a regular (full ntpd) backend thread. Even simple KOD processing might well slow this code down too much to keep up. I wonder what the costs/benefits will be when weighing the extra complexity of multiple choices against when the defaults change and we get new behavior that we can't tune, that costs us in X and Y. This gets a bit more complicated when taking into consideration: - we'll get more traffic from a NAT gateway - - do we need to be able to configure a threshhold for this case? - we should pay attention to how a client, whom we find to be abusive, reacts to: - - getting no response - - getting a KOD response and adapt accordingly. Seems like an obviously good idea, but see below: Any forced extra processing becomes another way to DOS the ntp server. Discussion appreciated. There is also the aspect when KOD does not bite. We have seen that. Like other forms of defenses, inserting drop rules into firewall rules for the offending node is an alternative to consider. KOD only bites for nodes which follows the protocol, but somehow is offending in their configuration. More offensive configuration or packet generation will render KOD relatively useless. Thus, there might be a limit on how much effort should be going into perfecting KOD-generation when maybe raising the bar even further is needed. Then, we should also consider how KOD and drop-rule triggering can be used to trigger denial of service, and how to potentially protect against them. This was my immediate worry: Forcing KOD to maintain even more state per session would make it easier to overload the NTP server. Terje -- - Terje.Mathisen at tmsw.no almost all programming can be viewed as an exercise in caching ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/06/2014 11:23 AM, Rob wrote: Jan Ceuleers jan.ceule...@computer.org wrote: I recommend providing motivation for the undesired clients to stop using the server, by the server sending a regular response indicating that it is not synchronised or replying in some other way that has no timekeeping value to the offending client. Well, that is what KOD actually is. Sorry, I was not clear. By a 'regular' response I mean one that has a non-zero stratum value. I had actually forgotten that a stratum value of zero indicates that the server is not synchronised (as it is a collision with LI=3, which also means that). So I guess I'm dropping my first suggestion. The second-one stands: pick a non-zero stratum value and report an immutable time stamp. Note that the stratum field occupies 8 bits in the packet format, but currently only values between 0 and 15 are defined (where we have seen that a value of 0 is not uniformly understood by real-life clients). So the choice of stratum value should be in the range 1-15. I have no particular preference for the immutable time stamp value to pick. Could be zero, could be some other meaningful value (such as 0xeee4baadeee4baad - twice Eek! Bad!). Jan ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 07/06/2014 12:38 PM, Terje Mathisen wrote: Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. I'm afraid this is exactly right: KOD is a way to keep honest guys honest, i.e. it only helps against programmers/users why actually try (hard) to do the right thing. Currently it will cause a badly configured ntpd installation (burst + minpoll 4 + maxpoll 4) to possibly stop using any server which sends back KOD, but only if it also uses the pool directive to actively search out the best servers. Maybe it's time to figure out how to auto-tune configurations as a better alternative than people keep following aged advice. In the meanwhile, make sure that good concrete advice with a section of don't do this anymore is on ntp.org. I don't want to think about users actively trying to generate as much traffic as possible. :-( Unfortunately we need to. The use of NTP features as accelerator in DDOS attack happen this spring. We had to turn of nice features, which in itself becomes a form of DOS. If we rather had ways to protect a server (remember that clients also act as servers) so that proper use does not cause loss of service, but aggressive use cause block-out. Soft-state remembering signaling peers for some time and then forget them to keep statistics of packets per time-period, and if the signaling peer acts reasonably well it is stays, overtransmitting packets will cause black-listing. KOD is the least, but inserting drop rules into the local host should follow, and possibly push the block rule into the network to clear off the machine and part of the network with the offending traffic. For cases like that, KOD won't help at all. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Sun, 06 Jul 2014 12:28:09 +, Magnus Danielson wrote: On 07/06/2014 12:38 PM, Terje Mathisen wrote: Rob wrote: Harlan Stenn st...@ntp.org wrote: Discussion appreciated. I think it is best to remove KOD from ntpd. It does not serve a useful purpose, because precisely the kind of clients that you want to say goodbye to, do not support it. In real life it has either no effect at all, or it even has a negative effect because the client does not understand it and re-tries the request sooner than it would when no reply was sent at all. I'm afraid this is exactly right: KOD is a way to keep honest guys honest, i.e. it only helps against programmers/users why actually try (hard) to do the right thing. Currently it will cause a badly configured ntpd installation (burst + minpoll 4 + maxpoll 4) to possibly stop using any server which sends back KOD, but only if it also uses the pool directive to actively search out the best servers. Maybe it's time to figure out how to auto-tune configurations as a better alternative than people keep following aged advice. In the meanwhile, make sure that good concrete advice with a section of don't do this anymore is on ntp.org. A proper default configuration (i.e. no fiddling) already auto-tunes sufficient for 90% of the cases. I don't want to think about users actively trying to generate as much traffic as possible. :-( Unfortunately we need to. The use of NTP features as accelerator in DDOS attack happen this spring. We had to turn of nice features, which in itself becomes a form of DOS. If we rather had ways to protect a server (remember that clients also act as servers) so that proper use does not cause loss of service, but aggressive use cause block-out. Soft-state remembering signaling peers for some time and then forget them to keep statistics of packets per time-period, and if the signaling peer acts reasonably well it is stays, overtransmitting packets will cause black-listing. KOD is the least, but inserting drop rules into the local host should follow, and possibly push the block rule into the network to clear off the machine and part of the network with the offending traffic. For cases like that, KOD won't help at all. All the state table/KOD/filter rules mitigation approaches I have seen so far are limited to one server. Maybe time to take a look at a DNSBL-type approach for abusive clients; that way, once a client is labeled 'abusive', it will stop working with any of the pool servers that use the blacklist. Policies (for how long to auto-blacklist, how to prevent DoS by blacklisting the competition, how to 'promise to behave and express-delist' etc.) to be defined. -d ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Detha, On 07/06/2014 03:23 PM, detha wrote: On Sun, 06 Jul 2014 12:28:09 +, Magnus Danielson wrote: For cases like that, KOD won't help at all. All the state table/KOD/filter rules mitigation approaches I have seen so far are limited to one server. Maybe time to take a look at a DNSBL-type approach for abusive clients; that way, once a client is labeled 'abusive', it will stop working with any of the pool servers that use the blacklist. Policies (for how long to auto-blacklist, how to prevent DoS by blacklisting the competition, how to 'promise to behave and express-delist' etc.) to be defined. Maybe. For the moment I think it is sufficient if we provide a mechanism by which offenders gets reported to *some* system. We *could* also provide a method by which white/black-list can be dynamically set from an external source, so enough hooks exists, but I do not think that NTPD should be burned with the rest of that system. Once NTPD can report it feels offended by a source, and beyond KODing it also report to some external mechanism that could potentially block it by any external means, NTPD does not have to do much more. My point being with this line of thinking is that KOD in itself makes assumptions on the offending source actually respects it, and while KOD rules probably can be improved, it does not provide a very effective means of protection with sources not respecting KOD, and thus we also needs to think i broader terms. In my mind, the defenses is according to these lines: 0) NTPD tolerates a source, packet approval checks 1) NTPD does not tolerate a source, fires of KOD, source is expected to shut up 2) NTPD admin does not tolerate a source, blacklist it, NTPD will drop the traffic 3) NTPD admin does not tolerate a source, filters in in box firewall, box firewall drops the traffic 4) NTPD admin does not tolerate a source, filters in network firewall, network firewall drops the traffic Notice how step 2-4 moves the traffic load further away from NTPD process, interface and eventually subnetwork. What I proposed would allow for automation of these steps. It is reasonable that escalation should be done when a source does not respect KOD and keeps transmitting requests. It is also resonable that blocking times out, such that blocking is removed after some reasonable time, as offenders can be on dynamic addresses, and usually works over limited time when intentional. How to automate step 2-4 is however not a core concern for NTPD, but feeding the data out of NTPD in a way that is handy for such a mechanism is. Separate block-log file as I proposed is probably better than only a syslog file, as it removes the need to parse syslog for matching blocks, but rather can focus on changes in a dedicated file. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On 2014-07-05 15:40, Harlan Stenn wrote: Folks, I was chatting with PHK about: http://support.ntp.org/bin/view/Dev/NtpProtocolResponseMatrix http://bugs.ntp.org/show_bug.cgi?id=2367 and how we probably want to extend KOD coverage to more than just the limited case. I was assuming folks would want finer-grained control over this behavior, and thought about being able to choose any of kod-limited, kod-noserve, and kod-query. PHK suggested that we consider going the other way - KOD would mean Send KODs whenever appropriate. I wonder what the costs/benefits will be when weighing the extra complexity of multiple choices against when the defaults change and we get new behavior that we can't tune, that costs us in X and Y. This gets a bit more complicated when taking into consideration: - we'll get more traffic from a NAT gateway - - do we need to be able to configure a threshhold for this case? - we should pay attention to how a client, whom we find to be abusive, reacts to: - - getting no response - - getting a KOD response and adapt accordingly. Discussion appreciated. Add exponential backoff to KOD responses to each source address such that every time you get another packet within the threshold, you increase the timeout during which you ignore incoming packets, before you again send a KOD response: maybe use limit*count or leak^count for the repeat offenders. Avoid logging DoS possibilities by logging only when more than maxburst packets have been received, and increase that count each time logging occurs: log only each time a power of maxburst is exceeded: 8, 64, 512, ... Have the MRUlist manage itself such that it recycles entries only when they are more than minpoll seconds old, or the list has reached its size limit. Avoid memory and time issues by preallocating the desired MRUlist size at startup, and if that fails, retry with half the number of entries until it succeeds. Log a count of entries recycled sooner than minpool every hour (add to systats?) and log the MRUlist size allocated at startup. Ignore the NAT gateway issue: the NAT gateways will certainly ignore NTP just as they ignore BCP 38: no benefit until their clients complain and it costs them! ;^ -- Take care. Thanks, Brian Inglis ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Sun, 06 Jul 2014 16:29:27 +, Magnus Danielson wrote: Detha, On 07/06/2014 03:23 PM, detha wrote: On Sun, 06 Jul 2014 12:28:09 +, Magnus Danielson wrote: For cases like that, KOD won't help at all. All the state table/KOD/filter rules mitigation approaches I have seen so far are limited to one server. Maybe time to take a look at a DNSBL-type approach for abusive clients; that way, once a client is labeled 'abusive', it will stop working with any of the pool servers that use the blacklist. Policies (for how long to auto-blacklist, how to prevent DoS by blacklisting the competition, how to 'promise to behave and express-delist' etc.) to be defined. Maybe. For the moment I think it is sufficient if we provide a mechanism by which offenders gets reported to *some* system. We *could* also provide a method by which white/black-list can be dynamically set from an external source, so enough hooks exists, but I do not think that NTPD should be burned with the rest of that system. Agreed, not core functionality for ntpd; but it would be nice to have a hook where one can 'vet' an incoming IP address, much like the RBL hooks are implemented in mail agents. Once NTPD can report it feels offended by a source, and beyond KODing it also report to some external mechanism that could potentially block it by any external means, NTPD does not have to do much more. My point being with this line of thinking is that KOD in itself makes assumptions on the offending source actually respects it, and while KOD rules probably can be improved, it does not provide a very effective means of protection with sources not respecting KOD, and thus we also needs to think i broader terms. In my mind, the defenses is according to these lines: 0) NTPD tolerates a source, packet approval checks 1) NTPD does not tolerate a source, fires of KOD, source is expected to shut up 2) NTPD admin does not tolerate a source, blacklist it, NTPD will drop the traffic 3) NTPD admin does not tolerate a source, filters in in box firewall, box firewall drops the traffic 4) NTPD admin does not tolerate a source, filters in network firewall, network firewall drops the traffic Notice how step 2-4 moves the traffic load further away from NTPD process, interface and eventually subnetwork. What I proposed would allow for automation of these steps. It is reasonable that escalation should be done when a source does not respect KOD and keeps transmitting requests. It is also resonable that blocking times out, such that blocking is removed after some reasonable time, as offenders can be on dynamic addresses, and usually works over limited time when intentional. How to automate step 2-4 is however not a core concern for NTPD, but feeding the data out of NTPD in a way that is handy for such a mechanism is. Separate block-log file as I proposed is probably better than only a syslog file, as it removes the need to parse syslog for matching blocks, but rather can focus on changes in a dedicated file. More logfiles that fill up disks. This is one time where KOD packets come in handy: it should be easy to construct a snort/whatever IDS signature for KOD packets, and let the IDS take care of firewalling off the offender as per site policy. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
detha de...@foad.co.za wrote: Maybe. For the moment I think it is sufficient if we provide a mechanism by which offenders gets reported to *some* system. We *could* also provide a method by which white/black-list can be dynamically set from an external source, so enough hooks exists, but I do not think that NTPD should be burned with the rest of that system. Agreed, not core functionality for ntpd; but it would be nice to have a hook where one can 'vet' an incoming IP address, much like the RBL hooks are implemented in mail agents. The problem is that source addresses can be spoofed. There is the BCP38 document that recommends providers to implement countermeasures against that, but it is not widely followed, mostly because there is no financial gain for those that do it. This reminds me of the situation with open SMTP relays. Those once were common practice (and even had a function), and when it became apparent that they had to be closed because of the abuse, there also was a large number of providers that did not want to cooperate. There were forced to do so by an RBL system that started to refuse mail from systems that were not appropriately configured. Probably the only way to get BCP38 implemented is to develop a similar system that just rejects traffic orginating at providers who don't do source address filtering. Without that, NTP server operators are really helpless against attacks, both of their servers and backscatter attacks against innocent victims. But of course it is outside the scope of NTP to do this, it just happens that NTP is a recent victim of this wide misconfiguration problem. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On Sun, 06 Jul 2014 18:51:16 +, Rob wrote: detha de...@foad.co.za wrote: Maybe. For the moment I think it is sufficient if we provide a mechanism by which offenders gets reported to *some* system. We *could* also provide a method by which white/black-list can be dynamically set from an external source, so enough hooks exists, but I do not think that NTPD should be burned with the rest of that system. Agreed, not core functionality for ntpd; but it would be nice to have a hook where one can 'vet' an incoming IP address, much like the RBL hooks are implemented in mail agents. The problem is that source addresses can be spoofed. There is the BCP38 document that recommends providers to implement countermeasures against that, but it is not widely followed, mostly because there is no financial gain for those that do it. Not such a big problem; if the source address is spoofed, chances are 99.99% that the packet is part of some DDoS attack, and blacklisting that IP will frustrate that attack (which is why a 'central' place where abusers are logged would help against NTP reflection attacks: once triggered, a lot of potential reflectors become useless to the attacker). The 0.01% where the purpose of the spoofed IP was to get someone's IP blacklisted will have to be handled somehow, either with timeouts or express whitelisting. This reminds me of the situation with open SMTP relays. Those once were common practice (and even had a function), and when it became apparent that they had to be closed because of the abuse, there also was a large number of providers that did not want to cooperate. There were forced to do so by an RBL system that started to refuse mail from systems that were not appropriately configured. Probably the only way to get BCP38 implemented is to develop a similar system that just rejects traffic orginating at providers who don't do source address filtering. There are some differences between open SMTP relays and networks not implementing BCP38. TCP versus UDP, and to quote Paul Vixie from https://queue.acm.org/detail.cfm?id=2578510 ] ... but the big reason why SAV isn't the default is: SAV benefits only ] other people's customers, not an operator's own customers. ] ] There is no way to audit a network from outside to determine if it ] practices SAV. Any kind of compliance testing for SAV has to be done by ] a device that's inside the network whose compliance is in question. That ] means the same network operator who has no incentive in the first place ] to deploy SAV at all is the only party who can tell whether SAV is deployed. That, and mis-configured NTP servers, is why we still have reflection attacks. Without that, NTP server operators are really helpless against attacks, both of their servers and backscatter attacks against innocent victims. But of course it is outside the scope of NTP to do this, it just happens that NTP is a recent victim of this wide misconfiguration problem. The biggest problem with NTP is the amplification factor. With a 1:1 or even 1:1.5 amplification factor, the attacker won't bother, and move to the next target - SNMP is a good candidate. With a 1:12 or better ratio, the attacker is happy. One way to weed out misconfigured servers would be to take a page from the mail operators book about 10 years ago, probe the source of incoming NTP requests for a misconfigured server (supports monlist on the public side etc.) and blacklist that source. Draconian? Yes. Effective? Maybe. -d ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
On -10.01.-28163 20:59, Harlan Stenn wrote: This gets a bit more complicated when taking into consideration: - we'll get more traffic from a NAT gateway - - do we need to be able to configure a threshhold for this case? Can't say much about KOD as-is, but here's my .02 on the net-behind-NAT scenario: If -- you want to fine-tune limits according to the number of actual clients behind the NAT, *or* -- want to keep providing service to genuine clients behind a NAT gateway while defending against co-located noncooperative bad apples then you have an interest to make the NATed clients identifiable (beyond what OS fingerprinting can do already). The straightforward approach to doing so would be to send out not plain go DIAFs, but messages along the lines of I'm willing to service your further requests *if* you label them with this random ID (and behave). Regards, J. Bern -- *NEU* - NEC IT-Infrastruktur-Produkte im http://www.linworks-shop.de/: Server--Storage--Virtualisierung--Management SW--Passion for Performance Jochen Bern, Systemingenieur --- LINworks GmbH http://www.LINworks.de/ Postfach 100121, 64201 Darmstadt | Robert-Koch-Str. 9, 64331 Weiterstadt PGP (1024D/4096g) FP = D18B 41B1 16C0 11BA 7F8C DCF7 E1D5 FAF4 444E 1C27 Tel. +49 6151 9067-231, Zentr. -0, Fax -299 - Amtsg. Darmstadt HRB 85202 Unternehmenssitz Weiterstadt, Geschäftsführer Metin Dogan, Oliver Michel ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
[ntp:questions] Thoughts on KOD
Folks, I was chatting with PHK about: http://support.ntp.org/bin/view/Dev/NtpProtocolResponseMatrix http://bugs.ntp.org/show_bug.cgi?id=2367 and how we probably want to extend KOD coverage to more than just the limited case. I was assuming folks would want finer-grained control over this behavior, and thought about being able to choose any of kod-limited, kod-noserve, and kod-query. PHK suggested that we consider going the other way - KOD would mean Send KODs whenever appropriate. I wonder what the costs/benefits will be when weighing the extra complexity of multiple choices against when the defaults change and we get new behavior that we can't tune, that costs us in X and Y. This gets a bit more complicated when taking into consideration: - we'll get more traffic from a NAT gateway - - do we need to be able to configure a threshhold for this case? - we should pay attention to how a client, whom we find to be abusive, reacts to: - - getting no response - - getting a KOD response and adapt accordingly. Discussion appreciated. -- Harlan Stenn st...@ntp.org http://networktimefoundation.org - be a member! ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Harlan, On 07/05/2014 11:40 PM, Harlan Stenn wrote: Folks, I was chatting with PHK about: http://support.ntp.org/bin/view/Dev/NtpProtocolResponseMatrix http://bugs.ntp.org/show_bug.cgi?id=2367 and how we probably want to extend KOD coverage to more than just the limited case. I was assuming folks would want finer-grained control over this behavior, and thought about being able to choose any of kod-limited, kod-noserve, and kod-query. PHK suggested that we consider going the other way - KOD would mean Send KODs whenever appropriate. I wonder what the costs/benefits will be when weighing the extra complexity of multiple choices against when the defaults change and we get new behavior that we can't tune, that costs us in X and Y. This gets a bit more complicated when taking into consideration: - we'll get more traffic from a NAT gateway - - do we need to be able to configure a threshhold for this case? - we should pay attention to how a client, whom we find to be abusive, reacts to: - - getting no response - - getting a KOD response and adapt accordingly. Discussion appreciated. There is also the aspect when KOD does not bite. We have seen that. Like other forms of defenses, inserting drop rules into firewall rules for the offending node is an alternative to consider. KOD only bites for nodes which follows the protocol, but somehow is offending in their configuration. More offensive configuration or packet generation will render KOD relatively useless. Thus, there might be a limit on how much effort should be going into perfecting KOD-generation when maybe raising the bar even further is needed. Then, we should also consider how KOD and drop-rule triggering can be used to trigger denial of service, and how to potentially protect against them. Sorry for muddling your water even more. Cheers, Magnus ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Magnus,
Yes, we know that if we decide to track finely-grained behavior we'll
need to watch how {IP,port} responds when getting {no,KOD} responses.
We might just want a syslog entry for KOD, because it's clear that there
can come a time when we don't want to rely on the remote side doing
anything.
Unless there is a better solution. I like the syslog idea because we
can tag it and let other mechanisms decide what to do with that raw
information.
H
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Thoughts on KOD
Harlan,
On 07/06/2014 02:18 AM, Harlan Stenn wrote:
Magnus,
Yes, we know that if we decide to track finely-grained behavior we'll
need to watch how {IP,port} responds when getting {no,KOD} responses.
Just want to gently remind you.
We might just want a syslog entry for KOD, because it's clear that there
can come a time when we don't want to rely on the remote side doing
anything.
Unless there is a better solution. I like the syslog idea because we
can tag it and let other mechanisms decide what to do with that raw
information.
For that purpose it may be good to allow for a separate log for sent KOD
messages, besides properly log to syslog. A script or program can then
monitor it for updates and insert rules, without having to filter the
syslog.
Cheers,
Magnus
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
