[dns-operations] Prevalence of query/response logging?
I know that some DNS operators disable logging of queries/responses due to the overhead of doing so - are most folks on this list with large-scale DNS recursive and/or authoritative DNS infrastructure disabling logging, enabling it, and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? Thanks much! -- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Equo ne credite, Teucri. -- Laocoön ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Prevalence of query/response logging?
On Fri, Jul 04, 2014 at 06:00:48PM +0700, Roland Dobbins rdobb...@arbor.net wrote a message of 23 lines which said: I know that some DNS operators disable logging of queries/responses due to the overhead of doing so Logging in the name server itself is typically very slow, take resources and, more seriously, add a new feature (which means new bugs and new security issues) to a critical software. So, indeed, it should not be done. and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? Following OARC workshops, it seems many operators of authoritative name servers log everything, with capture taps + a NoSQL-bigdata-thing. There are also captures of traffic at recursors, for instance Farsight' SIE, which logs the answers, and have interesting services on the top of it (such as DNSDB). ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Prevalence of query/response logging?
On Jul 4, 2014, at 6:44 PM, Stephane Bortzmeyer bortzme...@nic.fr wrote: Following OARC workshops, it seems many operators of authoritative name servers log everything, with capture taps + a NoSQL-bigdata-thing. Gotcha, makes sense. There are also captures of traffic at recursors, for instance Farsight' SIE, which logs the answers, and have interesting services on the top of it (such as DNSDB). Yes, DNSDB is quite interesting. Thanks much! -- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Equo ne credite, Teucri. -- Laocoön ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Prevalence of query/response logging?
On Fri, Jul 04, 2014 at 06:00:48PM +0700, Roland Dobbins wrote: I know that some DNS operators disable logging of queries/responses due to almost all, I would suggest. the overhead of doing so - are most folks on this list with large-scale DNS recursive and/or authoritative DNS infrastructure disabling logging, enabling it, and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? We've had great results with a format that stores all relevant details. It is called PCAP. Much recommended for serious setups, especially if you can do it out of band so it doesn't impact the servers itself. I know Nominet has a very powerful packet logging setup that they plan to offer commercially. Bert ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Need contacts
On Wed, Jul 02, 2014 at 10:28:31PM +0200, bert hubert bert.hub...@netherlabs.nl wrote a message of 7 lines which said: On Wed, Jul 02, 2014 at 09:36:38PM +0200, Stephane Bortzmeyer wrote: We know how to use dig and whois :-) The No-IP zones are all back to No-IP (from Microsoft) and seem to work. ORG isn't yet and they are generating oodles of servfails still. It took more time for .org but it's now done. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Prevalence of query/response logging?
On 07/04/2014 07:44 AM, Stephane Bortzmeyer wrote: On Fri, Jul 04, 2014 at 06:00:48PM +0700, Roland Dobbins rdobb...@arbor.net wrote a message of 23 lines which said: and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? Following OARC workshops, it seems many operators of authoritative name servers log everything, with capture taps We recently finished cleaning up the data from the DITL2014 collection exercise, captured and shared by many authoritative operators in exactly this way. You can see who contributed and what data is available at: https://www.dns-oarc.net/node/341 Keith ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Forcing BIND to randomly expire records from cache ahead of time
On Thu, Jul 3, 2014 at 6:04 PM, Tim Wicinski tjw.i...@gmail.com wrote: Mark Unbound has this feature, but its' a % of the TTL (oh they may of changed this). You may be also interested in this idea which was floated during IETF, and not rejected, just a small sliver of useful customer base: http://tools.ietf.org/id/draft-wkumari-dnsop-hammer-00.txt ... and, almost exactly one year later, we are *finally* rev'ing that document (I just edited it this afternoon, Suzanne is giving it the once over as we speak, and we are planning on submitting in the next hour (you know, just before the draft cutoff :-) - “I love deadlines. I love the whooshing noise they make as they go by.” — Douglas Adams, The Salmon of Doubt ) The new version mainly described the general concept, and that OpenDNS, Unbound and BIND 10 do something like this. In a somewhat contrived bit of writing, it also manages to keep the Stop! Hammer time! references... W On 7/3/14 5:06 PM, Mark Pettit wrote: Hi, folks. I have an issue with BIND cache timeouts, and I was hoping someone else might have some idea how to fix this. Here's the situation: we have a large number of servers that do a huge number of DNS lookups at the top of every minute. The TTL for the records they're looking up is 3600. What we've noticed is that on a host with a recently-restarted copy of BIND, we see huge spikes in DNS latency every 61 minutes. This makes logical sense, given the behavior of the DNS lookups. What is more interesting is that on hosts that have been running BIND for a very long time (on the order of months), the spikiness is not visible. Our speculation is that over time, due to the interaction between the 3600 TTL and the once every minute lookup behavior, cache misses become randomly distributed throughout the hour, and don't cause the spiky behavior that is observed initially. One of our ideas to resolve this is to randomize the TTLs in the zone files, causing them to expire out of cache at different times, thus forcing more-rapid distribution of cache misses across the hour. However, this would involve some massive edits to our zone files, and isn't really ideal. What *would* be ideal would be if we could tell BIND to randomly expire some small percentage of cached entries ahead of the actual TTL expiration. This would serve the same purpose as assigning random TTLs to the actual records in the zone files. Does BIND have a config option like this? Has anyone else ever encountered this issue, and if so, how did you address it? Thanks for any advice, and I hope everyone has a fantastic Fourth of July weekend. Mark Pettit ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Prevalence of query/response logging?
On Fri, Jul 4, 2014 at 1:00 PM, Roland Dobbins rdobb...@arbor.net wrote: [..] authoritative DNS infrastructure disabling logging, enabling it, and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? At dnswl.org, we use a dedicated logging on a selection of the authoritative servers. The logging through libpcap; we keep two bits of information: the query source IP and the query itself. In order to reduce the data volume, this data gets aggregated with counters (ie, ip + count, query + count), regularly written to files and then sent to a central log collector once in a while for further aggregation. This removes the logging overhead from the handling of the DNS request, although at some CPU cost. We don't care too much if we lose some data, as long as the data loss is approximately consistent. We don't need the logs for forensic analysis, but only to get relative sizes of our users and what they are querying. (And we don't need or want to know who is querying what, that's why the data is taken apart and aggregated independently from the start, consciously destroying the link between the two.) -- Matthias ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
[dns-operations] 'dnstap' (Re: Prevalence of query/response logging?)
Roland Dobbins wrote: I know that some DNS operators disable logging of queries/responses due to the overhead of doing so - are most folks on this list with large-scale DNS recursive and/or authoritative DNS infrastructure disabling logging, enabling it, and/or logging queries/responses out-of-band via packet-capture taps, databases, etc.? we've been using PCAP to do passive dns collection since 2008 or so, and we've determined that it is unsuitable. the overhead of reassembling fragmented IP datagrams is fairly low, but the overhead and complexity of reassembling TCP streams is extreme -- post-processor i know is willing to pick through a TCP stream inside a PCAP file processing each transaction. our first attempt was NCAP, which was better in that it represented DNS messages rather than IP packets. however this approach quickly showed its age and naivety when we tried to express orphan query, no response seen or orphan response, no query seen or this is a query and response pair. our next attempt was NMSG, which was better in that it was based on google protocol buffers, which are extensible without fork lift upgrades to your producers and consumers, and which are extremely performant. our PCAP sensor today generates NMSG output, which is then post-processed into many intermediate real time channels and ultimately informs the DNSDB passive DNS database. however, the reliance on PCAP turns out to be a problem for appliances, and the problem of collecting TCP remains difficult, and there are many events we care about which don't appear on the wire at all (cache expiry vs. cache purge, for example). therefore, 'dnstap', as presented by robert edmonds of farsight during the recent dns-oarc meeting in warsaw. 'dnstap' output can be created by PCAP feeds, but it is optimized for in-nameserver data collection, and it supports many different message types, and is extensible, and very, very performant. 'dnstap' tries hard to offload its CPU and memory costs to other processes, and is unlike syslog() completely asynchronous -- no queries are ever dropped because a 'dnstap' consumer couldn't keep up, and thus the primary purpose of name service is not threatened by the secondary purpose of gather telemetry. see https://dnstap.info/ for more details. so far this is live in knot (including kdig, which can now read and generate 'dnstap' format telemetry streams), and has been implemented on a development branch of unbound. we hope to get it into BIND and NSD in the next few months. we received an indication of no interest from powerdns, but i'm hoping bert will relent later on. dnstap is completely open source, with a BSD-style license (Apache 2.0). it is sponsored by farsight because we need a uniform DNS telemetry format for our business purposes. we are giving it away in order to make the world better, primarily for our own products and customers, but by necessary extension, better for everybody including our competitors. vixie ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] 'dnstap' (Re: Prevalence of query/response logging?)
On Jul 5, 2014, at 5:04 AM, Paul Vixie p...@redbarn.org wrote: dnstap is completely open source, with a BSD-style license (Apache 2.0). it is sponsored by farsight because we need a uniform DNS telemetry format for our business purposes. I read the dnstap preso with great interest when it was posted, and this appears to be the way to go, moving forward - one hopes we can get the option for dnstap telemetry natively exported over IPFIX, as this would make it easier to perform combinatorial analytics with flow telemetry generated via network infrastructure, as well as speed up the implementation of operationally useful collection/analytical systems. Thanks much! -- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Equo ne credite, Teucri. -- Laocoön ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Forcing BIND to randomly expire records from cache ahead of time
On Friday, July 4, 2014, Warren Kumari war...@kumari.net wrote: On Thu, Jul 3, 2014 at 6:04 PM, Tim Wicinski tjw.i...@gmail.com javascript:; wrote: Mark Unbound has this feature, but its' a % of the TTL (oh they may of changed this). You may be also interested in this idea which was floated during IETF, and not rejected, just a small sliver of useful customer base: http://tools.ietf.org/id/draft-wkumari-dnsop-hammer-00.txt ... and, almost exactly one year later, we are *finally* rev'ing that document (I just edited it this afternoon, Suzanne is giving it the once over as we speak, and we are planning on submitting in the next hour (you know, just before the draft cutoff :-) - “I love deadlines. I love the whooshing noise they make as they go by.” — Douglas Adams, The Salmon of Doubt ) The new version mainly described the general concept, and that OpenDNS, Unbound and BIND 10 do something like this. In a somewhat contrived bit of writing, it also manages to keep the Stop! Hammer time! references... ... and we managed to squeeze it in under the deadline. I did (of course) mean BIND 9.10 in the previous mail, and not BIND 10, which is a whole different kettle of fish. That'll teach me to write mail inna rush... W W On 7/3/14 5:06 PM, Mark Pettit wrote: Hi, folks. I have an issue with BIND cache timeouts, and I was hoping someone else might have some idea how to fix this. Here's the situation: we have a large number of servers that do a huge number of DNS lookups at the top of every minute. The TTL for the records they're looking up is 3600. What we've noticed is that on a host with a recently-restarted copy of BIND, we see huge spikes in DNS latency every 61 minutes. This makes logical sense, given the behavior of the DNS lookups. What is more interesting is that on hosts that have been running BIND for a very long time (on the order of months), the spikiness is not visible. Our speculation is that over time, due to the interaction between the 3600 TTL and the once every minute lookup behavior, cache misses become randomly distributed throughout the hour, and don't cause the spiky behavior that is observed initially. One of our ideas to resolve this is to randomize the TTLs in the zone files, causing them to expire out of cache at different times, thus forcing more-rapid distribution of cache misses across the hour. However, this would involve some massive edits to our zone files, and isn't really ideal. What *would* be ideal would be if we could tell BIND to randomly expire some small percentage of cached entries ahead of the actual TTL expiration. This would serve the same purpose as assigning random TTLs to the actual records in the zone files. Does BIND have a config option like this? Has anyone else ever encountered this issue, and if so, how did you address it? Thanks for any advice, and I hope everyone has a fantastic Fourth of July weekend. Mark Pettit ___ dns-operations mailing list dns-operations@lists.dns-oarc.net javascript:; https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs ___ dns-operations mailing list dns-operations@lists.dns-oarc.net javascript:; https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
