Roland Dobbins wrote: > I know that some DNS operators disable logging of queries/responses due to > the overhead of doing so - are most folks on this list with large-scale DNS > recursive and/or authoritative DNS infrastructure disabling logging, enabling > it, and/or logging queries/responses out-of-band via packet-capture taps, > databases, etc.?
we've been using PCAP to do passive dns collection since 2008 or so, and we've determined that it is unsuitable. the overhead of reassembling fragmented IP datagrams is fairly low, but the overhead and complexity of reassembling TCP streams is extreme -- post-processor i know is willing to pick through a TCP stream inside a PCAP file processing each transaction. our first attempt was NCAP, which was better in that it represented DNS messages rather than IP packets. however this approach quickly showed its age and naivety when we tried to express "orphan query, no response seen" or "orphan response, no query seen" or "this is a query and response pair". our next attempt was NMSG, which was better in that it was based on google protocol buffers, which are extensible without fork lift upgrades to your producers and consumers, and which are extremely performant. our PCAP sensor today generates NMSG output, which is then post-processed into many intermediate real time channels and ultimately informs the DNSDB passive DNS database. however, the reliance on PCAP turns out to be a problem for appliances, and the problem of collecting TCP remains difficult, and there are many events we care about which don't appear on the wire at all (cache expiry vs. cache purge, for example). therefore, 'dnstap', as presented by robert edmonds of farsight during the recent dns-oarc meeting in warsaw. 'dnstap' output can be created by PCAP feeds, but it is optimized for in-nameserver data collection, and it supports many different message types, and is extensible, and very, very performant. 'dnstap' tries hard to offload its CPU and memory costs to other processes, and is unlike syslog() completely asynchronous -- no queries are ever dropped because a 'dnstap' consumer couldn't keep up, and thus the primary purpose of "name service" is not threatened by the secondary purpose of "gather telemetry". see https://dnstap.info/ for more details. so far this is live in knot (including kdig, which can now read and generate 'dnstap' format telemetry streams), and has been implemented on a development branch of unbound. we hope to get it into BIND and NSD in the next few months. we received an indication of no interest from powerdns, but i'm hoping bert will relent later on. dnstap is completely open source, with a BSD-style license (Apache 2.0). it is sponsored by farsight because we need a uniform DNS telemetry format for our business purposes. we are giving it away in order to make the world better, primarily for our own products and customers, but by necessary extension, better for everybody including our competitors. vixie _______________________________________________ dns-operations mailing list [email protected] https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
