Re: REST API for recursive queries
Roee Mayerowicz wrote: > I have ~700k (and growing) domain names that should be resolved daily. > I'm trying to make it efficient as possible using the recursive BIND > server (do you know a better option?), the goal is to get 2000 queries > per second with minimum server\s cost. I do bulk lookups on that kind of scale when I am preparing a recursive server to go into production. I use this small (250 line) program as a front end to adns that works the way I like. It can easily manage thousands of queries per second. https://git.uis.cam.ac.uk/x/uis/ipreg/adns-masterfile.git (That URL may stop working within the next few months because we're moving to GitLab and my old git server will be shut down, though I would like to find somewhere to host redirection tombstones...) Tony. -- f.anthony.n.finchhttps://dotat.at/ North Foreland to Selsey Bill: Westerly 5 or 6, decreasing 3 or 4, becoming variable 2 to 4 later. Slight or moderate, becoming slight later. Showers, occasional rain later. Moderate or good, occasionally poor. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: REST API for recursive queries
I have ~700k (and growing) domain names that should be resolved daily. I'm trying to make it efficient as possible using the recursive BIND server (do you know a better option?), the goal is to get 2000 queries per second with minimum server\s cost. I thought using a single packet for multiple queries might be more efficient than multiple UDPs. I'll try reading more about adns to reach more queries at the same TCP connection. Any better ideas? From: bind-users on behalf of Roee Mayerowicz Sent: Tuesday, May 4, 2021 3:41 PM To: bind-users@lists.isc.org Subject: REST API for recursive queries CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hey, Do you know of a way to ask multiple DNS queries in a recursive bind server at the same packet\request? Using DoH might work? How? Is there a plugin which does that? Tnx ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: REST API for recursive queries
Petr Menšík wrote: > Because BIND uses DNS protocol only and not any dbus or former lwres > protocol, you can count only querying -t ANY for single name as > something similar. ANY queries don't necessarily give you all the records :-) In situations where a DNS client wants to do multiple queries at once, it can either send a load of UDP queries then wait for the answers, or if it has a TCP connection open, write all the queries in one go, then read the answers. There's not really much need for fancy features to support multiple questions when you can do hundreds of concurrent queries with one or two sockets. Happy Eyeballs version 2 requires concurrent DNS queries https://tools.ietf.org/html/rfc8305#section-3 I like to use `adns` for bulk concurrent queries http://www.chiark.greenend.org.uk/~ian/adns/ Much newer is getdns which has a more JSON-friendly design. https://getdnsapi.net/ Tony. -- f.anthony.n.finchhttps://dotat.at/ Biscay: West or northwest 5 or 6, becoming variable 2 to 4 later. Moderate or rough, becoming moderate. Rain at first. Good, occasionally moderate. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: REST API for recursive queries
systemd-resolved has private api, which attempts to do multiple DNS queries for one originating query. But it is not accepted to do that using DNS protocol, it uses d-bus calls I think. Because BIND uses DNS protocol only and not any dbus or former lwres protocol, you can count only querying -t ANY for single name as something similar. But DNS protocol is quite light weight. Multiple UDP queries are still fast to serve. Can you explain, why are you looking for single query? It seems to me tool like command "host example.com", which runs 3 queries on the name for you, might work. It does 3 queries, but from just single call. Would that work for you? On 5/4/21 2:41 PM, Roee Mayerowicz wrote: > Hey, > Do you know of a way to ask multiple DNS queries in a recursive bind server > at the same packet\request? Using DoH might work? How? Is there a plugin > which does that? > > Tnx > -- Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemen...@redhat.com PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB OpenPGP_signature Description: OpenPGP digital signature ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: REST API for recursive queries
You don't say /why/ you want to do this. This forwarder only does a single
request per TCP connection and also supports TLS:
https://github.com/m3047/tcp_only_forwarder/blob/master/forwarder.py
If you want to run DoT, I'm pretty sure that's on the BIND roadmap. The
BIND distro has provided instructions for setting up Nginx as an SSL
terminator in front of BIND in contrib/dnspriv/.
If you're trying to authenticate DNS queries/responses, you can also look
at using TSIG.
On Tue, 4 May 2021, Roee Mayerowicz wrote:
Do you know of a way to ask multiple DNS queries in a recursive bind
server at the same packet\request? Using DoH might work? How? Is there a
plugin which does that?
There is no way to send multiple requests in a single UDP datagram, but
you can send multiple requests in a TCP connection. There is only ever
supposed to be exactly one RR in the QUERY section.
--
Fred Morris
--
#!/usr/bin/python3
# Copyright (c) 2021 by Fred Morris Tacoma WA
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Multiple requests in a single TCP stream.
There is no way to send multiple queries in a single UDP datagram.
Tweak the following to your needs:
* 10.0.0.220 => your server address
* sophia.m3047. => a query name
* flame.m3047. => another query name
Mind the trailing dot at the end of the FQDNs.
"""
import socket
import dns.message
SERVER = ('10.0.0.220', 53)
BIG_ENDIAN = { 'byteorder':'big', 'signed':False }
def main():
sock = socket.create_connection(SERVER)
req = dns.message.make_query('sophia.m3047.','A')
wire_req = req.to_wire()
sock.send(len(wire_req).to_bytes(2, **BIG_ENDIAN) + wire_req)
resp_length = sock.recv(2)
wire_resp = sock.recv(int.from_bytes(resp_length, **BIG_ENDIAN))
resp = dns.message.from_wire(wire_resp)
print(resp)
req = dns.message.make_query('flame.m3047.','A')
wire_req = req.to_wire()
sock.send(len(wire_req).to_bytes(2, **BIG_ENDIAN) + wire_req)
resp_length = sock.recv(2)
wire_resp = sock.recv(int.from_bytes(resp_length, **BIG_ENDIAN))
resp = dns.message.from_wire(wire_resp)
print(resp)
sock.close()
return
if __name__ == '__main__':
main()
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: REST API for recursive queries
On Tue, May 4, 2021 at 8:42 AM Roee Mayerowicz wrote: > Do you know of a way to ask multiple DNS queries in a recursive bind server > at the same packet\request? > Using DoH might work? How? Is there a plugin which does that? The short answer is no, but it might not be answering the question you're really trying to ask. In strict terms of what would constitute "the same request", though, no. While you could conceive of a legally-formed DNS packet that had multiple questions in the Question section, a server has no way to acceptably indicate the proper response for all questions. In some cases, it might be obvious -- say, asking for the address of a.example.com and b.example.com, and them both having addresses -- but things quickly get out of hand when you look at the problems of indicating the many other ways that DNS can answer, like NXDOMAIN, NODATA, or delegation. With various forms of DNS TCP connections -- vanilla DNS, DNS over TLS (DoT), DNS over HTTPS (DoH) -- you can put multiple DNS request messages over the same connection. But that's not quite the same as "at the same packet\request". It also can depend on the end points; you might want to shove 1000 requests down a TCP connection, but server policy might limit the number it will actually process before terminating the link. And plugins are specific to a particular software package. Plugin to what? BIND and other major DNS resolvers and authoritative servers support TCP technologies natively. The clients that talk to them are numerous, with varying degrees of support for both TCP initiation and multi-request streaming. -- tale ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
REST API for recursive queries
Hey, Do you know of a way to ask multiple DNS queries in a recursive bind server at the same packet\request? Using DoH might work? How? Is there a plugin which does that? Tnx ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
