[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Patch added: "lp1668771-bionic.debdiff" https://bugs.launchpad.net/systemd/+bug/1668771/+attachment/5278752/+files/lp1668771-bionic.debdiff ** Summary changed: - systemd-resolved negative caching for extended period of time + [SRU] systemd-resolved negative caching for extended period of time -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: [SRU] systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Description changed: - 231-9ubuntu3 + [Impact] - If a DNS lookup returns SERVFAIL, systemd-resolved seems to cache the + * If a DNS lookup returns SERVFAIL, systemd-resolved seems to cache the result for very long (infinity?). I have to restart systemd-resolved to have the negative caching purged. - After SERVFAIL DNS server issue has been resolved, chromium/firefox + * After SERVFAIL DNS server issue has been resolved, chromium/firefox still returns DNS error despite host can correctly resolve the name. + + [Test Case] + + * If a lookup returns SERVFAIL systemd-resolved will cache the result for 30s (See 201d995), + however, there are several use cases on which this condition is not acceptable (See #5552 comments) + and the only workaround would be to disable cache entirely or flush it , which isn't optimal. + + * Configure /etc/systemd/resolved.conf as follows: + + Cache=yes (default) + + * Restart systemd-resolved (systemctl restart systemd-resolved.service) + + * Run a host/getent command against a entry that will return SERVFAIL + and check the journalctl output to see that the reply gets served from + cache. + + root@systemd-disco:/home/ubuntu# host www.no-record.cl + Host www.montemar.cl not found: 2(SERVFAIL) + root@systemd-disco:/home/ubuntu# journalctl -u systemd-resolved -n + -- Logs begin at Fri 2019-07-12 18:09:42 UTC, end at Tue 2019-07-23 15:10:17 UTC. -- + Jul 23 15:10:10 systemd-disco systemd-resolved[1282]: Transaction 6222 for on scope dns on ens3/* now complete with + Jul 23 15:10:10 systemd-disco systemd-resolved[1282]: Sending response packet with id 61042 on interface 1/AF_INET. + Jul 23 15:10:10 systemd-disco systemd-resolved[1282]: Freeing transaction 6222. + Jul 23 15:10:17 systemd-disco systemd-resolved[1282]: Got DNS stub UDP query packet for id 53580 + Jul 23 15:10:17 systemd-disco systemd-resolved[1282]: Looking up RR for www.no-record.cl IN A. + Jul 23 15:10:17 systemd-disco systemd-resolved[1282]: RCODE SERVFAIL cache hit for www.no-record.cl IN A + Jul 23 15:10:17 systemd-disco systemd-resolved[1282]: Transaction 58570 for < www.no-record.cl IN A> on scope dns on ens3/* now complete with scope dns on ens3/. + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Using feature level UDP for transaction 22382. + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Sending query packet with id 22382. + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Processing incoming packet on transaction 22382 (rcode=SERVFAIL). + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Server returned error: SERVFAIL + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Not caching negative entry for: www.metaklass.org IN A, cache mode set to no-negative + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Transaction 22382 for on scope dns on ens3/ now complete with from network (unsigned). + Jul 12 18:48:31 systemd-disco systemd-resolved[2635]: Sending response packet with id 31060 on interface 1/AF_INET. + + The following patch https://github.com/systemd/systemd/pull/13047 + implements the required changes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Merge proposal linked: https://code.launchpad.net/~ddstreet/ubuntu/+source/systemd/+git/systemd/+merge/370455 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Tags added: sts sts-sru-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
The attachment "lp1668771-eoan.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Patch added: "lp1668771-eoan.debdiff" https://bugs.launchpad.net/systemd/+bug/1668771/+attachment/5278115/+files/lp1668771-eoan.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Changed in: systemd (Ubuntu Disco) Importance: Undecided => High ** Changed in: systemd (Ubuntu Bionic) Importance: Undecided => High ** Changed in: systemd (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Also affects: systemd (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: systemd (Ubuntu Disco) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: systemd (Ubuntu Bionic) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: systemd (Ubuntu Xenial) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Changed in: systemd (Ubuntu Xenial) Status: New => In Progress ** Changed in: systemd (Ubuntu Bionic) Status: New => In Progress ** Changed in: systemd (Ubuntu Disco) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Also affects: systemd (Ubuntu Disco) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Eoan) Importance: High Assignee: Jorge Niedbalski (niedbalski) Status: In Progress ** Also affects: systemd (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
The proposal to extend the 'cache' option with 'no-negative' has been merged upstream. I will proceed with the backports to Ubuntu on the affected LTS releases. [0] https://github.com/systemd/systemd/pull/13047 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
I've made a proposal to change the resolved.conf Cache option to a tri- state "no, no-negative, yes" values. [0] If a lookup returns SERVFAIL systemd-resolved will cache the result for 30s (See 201d995), however, there are several use cases on which this condition is not acceptable (See #5552 comments) and the only workaround would be to disable cache entirely or flush it , which isn't optimal. This change adds the 'no-negative' option when set it avoids putting in cache negative answers but still works the same heuristics for positive answers. [0] https://github.com/systemd/systemd/pull/13047 ** Changed in: systemd (Ubuntu) Importance: Undecided => High ** Changed in: systemd (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Changed in: systemd (Ubuntu) Assignee: (unassigned) => Jorge Niedbalski (niedbalski) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
This is especially upsetting in an OpenStack and server environment where external DNS is being used to reach/resolve API endpoints and other systems such as database servers. A small outage to DNS becomes a potentially unbounded outage when the SERVFAIL responses are cached indefinitely, requiring manual intervention on each host in addition to fixing the cause of the SERVFAIL. ** Tags added: canonical-bootstack -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
This affects bionic openstack cloud environments when os-*-hostname is configured for keystone, and the keystone entry is deleted temporarily from upstream dns, or the upstream dns fails providing no record for the lookup of keystone.endpoint.domain.com. We have to then flush all caches across the cloud once DNS issue is resolved, rather than auto-healing at 60 seconds as if we were running nscd with negative-ttl set to 60 seconds. Ultimately, a negative TTL that is settable would be ideal, or the ability to not cache negative hits would also be useful. Only workaround now is to not use caches or to operationally flush caches as needed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
** Changed in: systemd Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1668771] Re: systemd-resolved negative caching for extended period of time
I believe this should be filed upstream instead. ** Bug watch added: github.com/systemd/systemd/issues #5552 https://github.com/systemd/systemd/issues/5552 ** Also affects: systemd via https://github.com/systemd/systemd/issues/5552 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668771 Title: systemd-resolved negative caching for extended period of time To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1668771/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs