[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Changed in: resolvconf (Ubuntu Disco) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Before trying to handle this as a BUG (and I know the behavior points to a bug) please have a look at https://github.com/Azure/WALinuxAgent/issues/1673 ** Bug watch added: github.com/Azure/WALinuxAgent/issues #1673 https://github.com/Azure/WALinuxAgent/issues/1673 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Tags added: id-5cde5f8331588344774efccb -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Also affects: resolvconf (Ubuntu) Importance: Undecided Status: New ** Changed in: resolvconf (Ubuntu Bionic) Status: New => Triaged ** Changed in: resolvconf (Ubuntu Cosmic) Status: New => Triaged ** Changed in: resolvconf (Ubuntu Disco) Status: New => Triaged ** Changed in: resolvconf (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
The conclusion of a very long IRC discussion about how to fix this is that we should change the resolvconf package in the presence of resolved to emit only 127.0.0.53 into /etc/resolv.conf, and redirect all other servers to resolved. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** No longer affects: systemd (Ubuntu Trusty) ** No longer affects: systemd (Ubuntu Xenial) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
The fix (workaround) for this bug in bionic and cosmic was to add 'options edns0' to the /etc/resolv.conf file via the systemd stub- resolv.conf file. However, when the resolvconf package is installed, due to bug 1817903, the 'options edns0' is stripped out of the /etc/resolv.conf file. This means anyone on bionic or cosmic that has the resolvconf package installed will not have 'options edns0' in their /etc/resolv.conf file, and will again experience this bug. In disco, systemd-resolved has DNS TCP pipelining correctly implemented, so this bug will not affect disco, regardless of whether edns0 is specified in /etc/resolv.conf. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Changed in: systemd Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
It seems this breaks dns lookups on some system, see #1817903 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
This bug was fixed in the package systemd - 239-7ubuntu10.7 --- systemd (239-7ubuntu10.7) cosmic; urgency=medium * d/p/resolve-enable-EDNS0-towards-the-127.0.0.53-stub-res.patch getaddrinfo() failures when fallback to dns tcp queries, so enable edns0 in resolv.conf (LP: #1811471) [ Victor Tapia ] * d/p/resolved-Increase-size-of-TCP-stub-replies.patch dns failures with edns0 disabled and truncated response (LP: #1804487) -- Dan Streetman Tue, 29 Jan 2019 14:19:39 -0500 ** Changed in: systemd (Ubuntu Cosmic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
This bug was fixed in the package systemd - 237-3ubuntu10.12 --- systemd (237-3ubuntu10.12) bionic; urgency=medium * d/p/resolve-enable-EDNS0-towards-the-127.0.0.53-stub-res.patch getaddrinfo() failures when fallback to dns tcp queries, so enable edns0 in resolv.conf (LP: #1811471) [ Victor Tapia ] * d/p/resolved-Increase-size-of-TCP-stub-replies.patch dns failures with edns0 disabled and truncated response (LP: #1804487) -- Dan Streetman Tue, 29 Jan 2019 14:26:48 -0500 ** Changed in: systemd (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
All remaining bionic and cosmic autopkgtest regression failures should be ignored. bionic regressions: systemd on all archs have failed for months, ignore linux-gcp-edge fails due to timeout in test while rebuilding; ignore linux has flaky tests - intermittently fails for a long time, ignore linux-oracle fails due to out of disk space while rebuilding; ignore gvfs/s390x has always failed, ignore cosmic regressions: gvfs/s390x has always failed, ignore systemd has failed intermittently for months; ignore hddemux fails due to bug 1814062, ignore linux has flaky tests - intermittently fails for a long time, ignore -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
This bug was fixed in the package systemd - 240-5ubuntu3 --- systemd (240-5ubuntu3) disco; urgency=medium * debian/tests: blacklist upstream test-24-unit-tests on ppc64le. Fails, not a regression as it's a new test case, which was never before executed on ppc64le. File: debian/tests/upstream https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8062b9a2712c390010d2948eaf764a1b52e68715 -- Dimitri John Ledkov Sat, 02 Feb 2019 11:05:12 +0100 ** Changed in: systemd (Ubuntu Disco) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
hddemux failure should be ignored; its autopkgtests are fixed in -proposed with bug 1814062 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
bionic regressions: systemd on all archs have failed for months. tests should be ignored. snapd on all archs have failed intermittently for very long time. tests are flaky and should be ignored. remaining tests being retried: linux-gcp-edge (system problem - oom while testing) linux (flaky tests - intermittently fails for a long time) linux-oracle (system problem - out of disk space while testing) cosmic regressions: hddemux on all archs started failing recently; the version in -proposed appears to be fixed, so the failure of this pkg can be ignored as it's not caused by this sru. remaining tests being retried: apt (flaky test - fails intermittently in the same way for a while) linux (flaky tests - intermittently fails for a long time) snapd/amd64 (flaky test, test watchdog has 1 second timeout, and timed out) systemd (test output hard to read - seems to be timeout, likely overloaded test system) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
autopkgtest regression failure analysis/justifications for bionic: systemd/s390x - failing since last november. gvfs/s390x - failing since 2017. snapd/s390x - flaky test that fails more than 1/2 the time since forever. snapd/ppc64el - same as s390x for cosmic: gvfs/s390x - almost always failed since forever. other bionic and cosmic autopkgtest regressions look like flaky tests, or autopkgtest system failures (e.g. can't reach apt repository). i have retried them all - will analyze again if the retest fails. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Bionic verified ii systemd 237-3ubuntu10.12 amd64 system and service manager $ ping testing.irongiantdesign.com PING testing.irongiantdesign.com (253.0.0.6) 56(84) bytes of data. ** Tags removed: verification-needed verification-needed-bionic ** Tags added: verification-done verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Cosmic Verified ii systemd 239-7ubuntu10.7 amd64 system and service manager ping testing.irongiantdesign.com PING testing.irongiantdesign.com (253.0.0.15) 56(84) bytes of data. ** Tags removed: verification-needed-cosmic ** Tags added: verification-done-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Hello Dan, or anyone else affected, Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.12 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Hello Dan, or anyone else affected, Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.7 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Cosmic) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Changed in: systemd (Ubuntu Disco) Status: In Progress => Fix Committed ** Changed in: systemd (Ubuntu Disco) Assignee: Dan Streetman (ddstreet) => Dimitri John Ledkov (xnox) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Changed in: systemd (Ubuntu Xenial) Status: In Progress => Invalid ** Changed in: systemd (Ubuntu Trusty) Status: In Progress => Invalid ** Changed in: systemd (Ubuntu Xenial) Importance: High => Undecided ** Changed in: systemd (Ubuntu Trusty) Importance: High => Undecided ** Changed in: systemd (Ubuntu Trusty) Assignee: Dan Streetman (ddstreet) => (unassigned) ** Changed in: systemd (Ubuntu Xenial) Assignee: Dan Streetman (ddstreet) => (unassigned) ** Description changed: [Impact] The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers. Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server. By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records. When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response. However, glibc issues DNS queries for both A and records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets). Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection. That would be bad enough, as glibc is (rightly) expecting a response to both its A and queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error. Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the response). [Test Case] This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet. Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding: options use-vc With that option, glibc will fail to lookup 100% of DNS names,
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Description changed: [Impact] The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers. Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server. By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records. When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response. However, glibc issues DNS queries for both A and records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets). Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection. That would be bad enough, as glibc is (rightly) expecting a response to both its A and queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error. Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the response). [Test Case] This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet. Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding: options use-vc With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries. Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Patch removed: "lp1811471-disco.debdiff" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233684/+files/lp1811471-disco.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Patch added: "lp1811471-disco.debdiff" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233684/+files/lp1811471-disco.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
> Also is this then just not a simple cherrypick of: > > https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8 that's one way to work around it, although glibc is not necessarily the only thing that might do pipelined TCP dns lookups to the local stub resolver (though I have no examples of anything else that does). It certainly should fix/workaround this for Ubuntu installs using the default systemd-resolved setup and only having issues with getaddrinfo() failures. I still plan to fix systemd's stub resolver to correctly respond to pipelined TCP dns queries. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Also is this then just not a simple cherrypick of: https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8 ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
I am happy to add "options edns0" in the generated file by resolved. But we also need to file this case upstream, and start implementing pipelined requests handling in resolved too. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
Adding "options edns0" to /etc/resolv.conf ended up resolving bug 1805027 for me. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Changed in: systemd Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1811471] Re: local resolver stub fails to handle multiple TCP dns queries
** Bug watch added: github.com/systemd/systemd/issues #11332 https://github.com/systemd/systemd/issues/11332 ** Also affects: systemd via https://github.com/systemd/systemd/issues/11332 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1811471 Title: local resolver stub fails to handle multiple TCP dns queries To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1811471/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
