[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Fixed my dnsmasq systemd-resolve race with dnsmasq config. /etc/dnsmasq.d/myconfig #PES 20180808 dnsmasq and systemd-resolve conflict. # dont use /etc/resolv.conf, go direct to systemd-resolve, only bind to lo no-resolv bind-interfaces interface=lo server=127.0.0.53 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
The same problem arises, only when in dnsmasq I register local addresses on the virtual local server address=/.dev/192.168.56.20 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
You're right, it turns out dnsmasq was only installed because I had installed polipo, which depends on the dnsmasq package. Purging polipo removes dnsmasq. I have been running polipo (with config files unmodified from default) for several years, including since fresh installing Ubuntu-Gnome 16.10. Only since the upgrade to 17.04 two weeks ago has this problem emerged. Presumably it will hit every user who installs polipo or dnsmasq in 17.04. $ cat /etc/hosts # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN # 127.0.0.53 is the systemd-resolved stub resolver. # run "systemd-resolve --status" to see details about the actual nameservers. nameserver 127.0.0.1 $ $ dpkg -l dnsmasq\* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==---= ii dnsmasq2.76-5 all Small caching DNS proxy and DHCP/ ii dnsmasq-base 2.76-5 amd64Small caching DNS proxy and DHCP/ $ $ cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifupdown,keyfile [ifupdown] managed=false -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Where does the dnsmasq package come from that is listening on 127.0.0.1? dnsmasq should normally not be listening on that address if it was spawned by NetworkManager. Do you have the 'dnsmasq' package installed instead of dnsmasq-base? dnsmasq-base can be used in a variety of scenarios as a local nameserver and/or for lxc/libvirt and other uses. dnsmasq ships a service file which by default might conflict with resolved. If you need dnsmasq to run a nameserver on that system, you would have to make sure the configuration of the whole system is such that both would not conflict. This can be achieved in two different ways: - Setting DNS= in resolved if you want to use resolved as an authoritative nameserver to provide dnsmasq with the information it needs (but not circle back to dnsmasq). - Setting 'no-resolv' and possibly other settings for the dnsmasq instance, such that it doesn't go ask systemd-resolved for nameservers, and that it does not update resolv.conf if it should not. ** Changed in: dnsmasq (Ubuntu) Assignee: (unassigned) => Mathieu Trudel-Lapierre (cyphermox) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
What are the contents of /etc/resolv.conf when this happens? What is the output of 'dpkg -l dnsmasq\*'? What are the contents of /etc/NetworkManager/NetworkManager.conf? Your bug report shows that this system was installed with 16.10 media and then upgraded to 17.04. How long ago did you upgrade? Have you rebooted since upgrading? ** Changed in: dnsmasq (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
I asked on IRC: 18:15 rbasak: AFAIK dnsmasq is dropped out of the default DNS stack again in 17.04, so that shouldn't be happening? 18:15 certainly, resolved+networkd+resolvconf+dnsmasq might have bugs like this 18:15 but that's not the config we're supposed to be shipping So I think dnsmasq isn't supposed to be the default on 17.04. Since this is a non-default configuration, I'll set Importance to Low. ** Changed in: dnsmasq (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
I just found that I can reproduce this loop state by reconnecting to my Wi-Fi access point. The storm starts right after this line, 5 seconds after the link comes up: systemd-resolved[1188]: Server 127.0.0.1 does not support DNSSEC, downgrading to non-DNSSEC mode. The (much shorter this time) syslog extract is attached. The only reference I can find to 127.0.0.53 in the settings (other than comment lines) is in /run/resolvconf/interface/systemd-resolved: nameserver 127.0.0.53 So I also don't know how dnsmasq gets that address unless it reads that config file for some reason (no mention of it in source), derives it from an incoming request, or gets it via D-Bus. ** Attachment added: "syslog_dns_storm_1.txt" https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+attachment/4837813/+files/syslog_dns_storm_1.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Looking again. the loop probably involves systemd-resolverd too, dnsmasq forwards to 127.0.0.53 which is systemd-resolverd, and systemd-resolverd then returns it to dnsmasq at 127.0.0.1 Why, oh why is Ubuntu running both? Cheers, Simon. On 14/03/17 11:15, Paul wrote: > I have cpulimit(1) watching dnsmasq now, so it only goes berserk for a > second before being killed, but the attached syslog extract captures the > moments before and during the DNS storm. These particular lookups are > mostly originated by Transmission, but previously the storms have > happened when there were no Transmission processes running, with queries > from Firefox or perhaps some unidentified Gnome weather applet. > > ** Attachment added: "syslog_dns_storm.txt" > > https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+attachment/4837521/+files/syslog_dns_storm.txt > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Ok, so the amplification is arising from dnsmasq looping queries around 127.0.0.1 -> 127.0.0.53 -> 127.0.0.1 -> . It would be really useful to get dnsmasq's idea of what it's upstreams are. We know that 127.0.0.1 is in the list from your previous post, and I guess that dnsmasq has successfully worked out not to use that as it loops back to itself. It's very likely that it didn't work out that 127.0.0.53 also loops back to itself too, but it's not clear how that's getting into the list of upstreams. This is starting to look like an Ubuntu/systemd plumbing problem, rather than a dnsmasq bug. Simon. On 14/03/17 11:15, Paul wrote: > I have cpulimit(1) watching dnsmasq now, so it only goes berserk for a > second before being killed, but the attached syslog extract captures the > moments before and during the DNS storm. These particular lookups are > mostly originated by Transmission, but previously the storms have > happened when there were no Transmission processes running, with queries > from Firefox or perhaps some unidentified Gnome weather applet. > > ** Attachment added: "syslog_dns_storm.txt" > > https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+attachment/4837521/+files/syslog_dns_storm.txt > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
I have cpulimit(1) watching dnsmasq now, so it only goes berserk for a second before being killed, but the attached syslog extract captures the moments before and during the DNS storm. These particular lookups are mostly originated by Transmission, but previously the storms have happened when there were no Transmission processes running, with queries from Firefox or perhaps some unidentified Gnome weather applet. ** Attachment added: "syslog_dns_storm.txt" https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+attachment/4837521/+files/syslog_dns_storm.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Are we clear that this is a dnsmasq problem, and not a systemd-resolved
one? Can you add --log-queries to the dnsmasq configuration and see what
dnsmasq is doing? That should demonstrate if the loop is dnsmasq
forwarding to itself, of if the problem is something else.
Cheers,
Simon.
On 13/03/17 08:46, Paul wrote:
> There aren't any such entries in syslog, presumably because I had
> hardcoded two upstream servers (208.67.222.222 and 208.67.220.220) using
> the GUI Wi-Fi settings dialog in 16.10 and they're not changing. Oddly,
> I can't see that setting in the 17.04 dialog, even though "systemd-
> resolve --status" correctly reports them:
>
> Global
> DNS Servers: 127.0.0.1
> DNSSEC NTA: 10.in-addr.arpa
> 16.172.in-addr.arpa
> 168.192.in-addr.arpa
> 17.172.in-addr.arpa
> 18.172.in-addr.arpa
> 19.172.in-addr.arpa
> 20.172.in-addr.arpa
> 21.172.in-addr.arpa
> 22.172.in-addr.arpa
> 23.172.in-addr.arpa
> 24.172.in-addr.arpa
> 25.172.in-addr.arpa
> 26.172.in-addr.arpa
> 27.172.in-addr.arpa
> 28.172.in-addr.arpa
> 29.172.in-addr.arpa
> 30.172.in-addr.arpa
> 31.172.in-addr.arpa
> corp
> d.f.ip6.arpa
> home
> internal
> intranet
> lan
> local
> private
> test
>
> Link 2 (wlp2s0)
> Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
>LLMNR setting: yes
> MulticastDNS setting: no
> DNSSEC setting: allow-downgrade
> DNSSEC supported: no
> DNS Servers: 208.67.222.222
> 208.67.220.220
> DNS Domain: local
>
> The requests against the upstream server are disturbingly fast, I'm
> surprised I haven't been blacklisted yet.
>
> 19:40:12.000415 IP hostname.50776 > resolver1.opendns.com.domain: 42051+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C...X.5.=nB.C...4.1.168.192.in-addr.arpa...)
> 19:40:12.000920 IP hostname.59219 > resolver1.opendns.com.domain: 14223+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C...S.5.=..74.1.168.192.in-addr.arpa...)
> 19:40:12.001411 IP hostname.51647 > resolver1.opendns.com.domain: 2501+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C.5.=.Z
> 4.1.168.192.in-addr.arpa...)
> 19:40:12.001885 IP hostname.33104 > resolver1.opendns.com.domain: 30929+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C...P.5.=..x4.1.168.192.in-addr.arpa...)
> 19:40:12.002412 IP hostname.47231 > resolver1.opendns.com.domain: 46563+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C.5.=j{.4.1.168.192.in-addr.arpa...)
> 19:40:12.004238 IP hostname.57292 > resolver1.opendns.com.domain: 61082+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C.5.=
> w.4.1.168.192.in-addr.arpa...)
> 19:40:12.008187 IP hostname.49786 > resolver1.opendns.com.domain: 14681+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C...z.5.=.
> 9Y...4.1.168.192.in-addr.arpa...)
> 19:40:12.008926 IP hostname.53171 > resolver1.opendns.com.domain: 24423+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C.5.=.._g...4.1.168.192.in-addr.arpa...)
> 19:40:12.009629 IP hostname.37811 > resolver1.opendns.com.domain: 55231+%
> [1au] PTR? 4.1.168.192.in-addr.arpa. (53)
> E..Q..@.@C.5.=mk.4.1.168.192.in-addr.arpa...)
>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1672099
Title:
DNS loop, >5,000 queries per second for minutes at a time
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
There aren't any such entries in syslog, presumably because I had
hardcoded two upstream servers (208.67.222.222 and 208.67.220.220) using
the GUI Wi-Fi settings dialog in 16.10 and they're not changing. Oddly,
I can't see that setting in the 17.04 dialog, even though "systemd-
resolve --status" correctly reports them:
Global
DNS Servers: 127.0.0.1
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 2 (wlp2s0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: allow-downgrade
DNSSEC supported: no
DNS Servers: 208.67.222.222
208.67.220.220
DNS Domain: local
The requests against the upstream server are disturbingly fast, I'm
surprised I haven't been blacklisted yet.
19:40:12.000415 IP hostname.50776 > resolver1.opendns.com.domain: 42051+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C...X.5.=nB.C...4.1.168.192.in-addr.arpa...)
19:40:12.000920 IP hostname.59219 > resolver1.opendns.com.domain: 14223+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C...S.5.=..74.1.168.192.in-addr.arpa...)
19:40:12.001411 IP hostname.51647 > resolver1.opendns.com.domain: 2501+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C.5.=.Z
4.1.168.192.in-addr.arpa...)
19:40:12.001885 IP hostname.33104 > resolver1.opendns.com.domain: 30929+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C...P.5.=..x4.1.168.192.in-addr.arpa...)
19:40:12.002412 IP hostname.47231 > resolver1.opendns.com.domain: 46563+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C.5.=j{.4.1.168.192.in-addr.arpa...)
19:40:12.004238 IP hostname.57292 > resolver1.opendns.com.domain: 61082+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C.5.=
w.4.1.168.192.in-addr.arpa...)
19:40:12.008187 IP hostname.49786 > resolver1.opendns.com.domain: 14681+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C...z.5.=.
9Y...4.1.168.192.in-addr.arpa...)
19:40:12.008926 IP hostname.53171 > resolver1.opendns.com.domain: 24423+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C.5.=.._g...4.1.168.192.in-addr.arpa...)
19:40:12.009629 IP hostname.37811 > resolver1.opendns.com.domain: 55231+% [1au]
PTR? 4.1.168.192.in-addr.arpa. (53)
E..Q..@.@C.5.=mk.4.1.168.192.in-addr.arpa...)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1672099
Title:
DNS loop, >5,000 queries per second for minutes at a time
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Whenever the set of servers to which dnsmasq is forwarding queries changes, the whole set is logged to syslog. It would be useful to have that information. On 13/03/17 00:01, Paul wrote: > Restarting dnsmasq immediately stops an ongoing DNS storm. > The actual upstream server used can change unpredictably, so that's no surprise. Cheers, Simon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1672099] Re: DNS loop, >5, 000 queries per second for minutes at a time
Restarting dnsmasq immediately stops an ongoing DNS storm. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1672099 Title: DNS loop, >5,000 queries per second for minutes at a time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1672099/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
