Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On Fri, Sep 10, 2021 at 09:50:42PM +0200, Thomas Goirand wrote: > On 9/10/21 11:40 AM, Filippo Giunchedi wrote: > > On Thu, Sep 09, 2021 at 09:32:34AM +0200, Thomas Goirand wrote: > >> Hi, > >> > >> Thanks a lot for working on this, it really is helpful. > >> > >> The pull request you're pointing at contains multiple commits. Would you > >> be able to transform this into a patch against the Eventlet versions > >> 0.26.1 (currently in Stable) and 0.30.2 (in Unstable and Testing)? If > >> you provide it, then I'll be very happy to add the patches to these > >> Debian packages. If I'm asking it's not because I don't want to do it > >> myself, but because you wrote it, you may be better at understanding how > >> to backport the patches. > > > > Certainly, I did port the patch to our internal repo for Bullseye. You can > > find > > the commit below, which modulo the changelog version obviously should work > > as-is. > > > > https://github.com/wikimedia/operations-debs-python-eventlet/commit/a93d2e0cd2cdf3efcd7915cb781355d58e5728ab > > > > I didn't change > > 'Replace-dnspython-_compute_expiration-by-_compute_times.patch' > > for a cleaner diff, although that patch a whole I think can be replaced with > > the PR's diff. What do you think? > > > > best, > > Filippo > > > > Hi, > > I'll try to get this in Bullseye proper. Thanks a lot for your work, > this is definitively very helpful, and may solve troubles with swift's > cname middleware also. You are welcome, and thank you for pushing to get the update in Bullseye > > I'm not sure about > Replace-dnspython-_compute_expiration-by-_compute_times.patch, though > it's probably better, from the Debian Stable perspective, to not touch > the patches that are already there, so it is easier for the Stable > release team to review it. Agreed > I will also need a patch against the version 0.30.2-2 currently in > unstable/bookworms (again: otherwise the Debian Stable release team may > complain about it). Could you provide one? For sure, I have added the patches in this MR. Let me know what you think! https://salsa.debian.org/python-team/packages/python-eventlet/-/merge_requests/2 best, Filippo
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On 9/10/21 11:40 AM, Filippo Giunchedi wrote: > On Thu, Sep 09, 2021 at 09:32:34AM +0200, Thomas Goirand wrote: >> Hi, >> >> Thanks a lot for working on this, it really is helpful. >> >> The pull request you're pointing at contains multiple commits. Would you >> be able to transform this into a patch against the Eventlet versions >> 0.26.1 (currently in Stable) and 0.30.2 (in Unstable and Testing)? If >> you provide it, then I'll be very happy to add the patches to these >> Debian packages. If I'm asking it's not because I don't want to do it >> myself, but because you wrote it, you may be better at understanding how >> to backport the patches. > > Certainly, I did port the patch to our internal repo for Bullseye. You can > find > the commit below, which modulo the changelog version obviously should work > as-is. > > https://github.com/wikimedia/operations-debs-python-eventlet/commit/a93d2e0cd2cdf3efcd7915cb781355d58e5728ab > > I didn't change > 'Replace-dnspython-_compute_expiration-by-_compute_times.patch' > for a cleaner diff, although that patch a whole I think can be replaced with > the PR's diff. What do you think? > > best, > Filippo > Hi, I'll try to get this in Bullseye proper. Thanks a lot for your work, this is definitively very helpful, and may solve troubles with swift's cname middleware also. I'm not sure about Replace-dnspython-_compute_expiration-by-_compute_times.patch, though it's probably better, from the Debian Stable perspective, to not touch the patches that are already there, so it is easier for the Stable release team to review it. I will also need a patch against the version 0.30.2-2 currently in unstable/bookworms (again: otherwise the Debian Stable release team may complain about it). Could you provide one? Cheers, Thomas Goirand (zigo)
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On Thu, Sep 09, 2021 at 09:32:34AM +0200, Thomas Goirand wrote: > Hi, > > Thanks a lot for working on this, it really is helpful. > > The pull request you're pointing at contains multiple commits. Would you > be able to transform this into a patch against the Eventlet versions > 0.26.1 (currently in Stable) and 0.30.2 (in Unstable and Testing)? If > you provide it, then I'll be very happy to add the patches to these > Debian packages. If I'm asking it's not because I don't want to do it > myself, but because you wrote it, you may be better at understanding how > to backport the patches. Certainly, I did port the patch to our internal repo for Bullseye. You can find the commit below, which modulo the changelog version obviously should work as-is. https://github.com/wikimedia/operations-debs-python-eventlet/commit/a93d2e0cd2cdf3efcd7915cb781355d58e5728ab I didn't change 'Replace-dnspython-_compute_expiration-by-_compute_times.patch' for a cleaner diff, although that patch a whole I think can be replaced with the PR's diff. What do you think? best, Filippo
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On 9/7/21 10:05 AM, Filippo Giunchedi wrote: > On Tue, Aug 24, 2021 at 02:32 PM, Filippo Giunchedi wrote: >> I was able to get python3-eventlet to play nice with dnspython2 by >> integrating https://github.com/eventlet/eventlet/pull/722 from upstream. > > Upstream has merged the PR, please consider updating the patch in the > package. Possibily for a point release too? > > best, > Filippo > Hi, Thanks a lot for working on this, it really is helpful. The pull request you're pointing at contains multiple commits. Would you be able to transform this into a patch against the Eventlet versions 0.26.1 (currently in Stable) and 0.30.2 (in Unstable and Testing)? If you provide it, then I'll be very happy to add the patches to these Debian packages. If I'm asking it's not because I don't want to do it myself, but because you wrote it, you may be better at understanding how to backport the patches. Cheers, Thomas Goirand (zigo)
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On Tue, Aug 24, 2021 at 02:32 PM, Filippo Giunchedi wrote: > I was able to get python3-eventlet to play nice with dnspython2 by > integrating https://github.com/eventlet/eventlet/pull/722 from upstream. Upstream has merged the PR, please consider updating the patch in the package. Possibily for a point release too? best, Filippo
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On Tue, Aug 24, 2021 at 09:52 AM, Filippo Giunchedi wrote:
> On Tue, Jun 08, 2021 at 10:03 AM, Filippo Giunchedi wrote:
> > Package: swift-container
> > Version: 2.26.0-10
> > Severity: important
> > File: /usr/bin/swift-container-reconciler
> >
> > Dear Maintainer,
> > I'm experimenting with Swift on Bullseye and came across a problem with
> > container-reconciler (possibly others) when using hostnames in
> > memcache_servers. Namely these errors:
>
> In the "possibly others" category, swift-dispersion-report is also 100%
> broken in Bullseye:
I was able to get python3-eventlet to play nice with dnspython2 by
integrating https://github.com/eventlet/eventlet/pull/722 from upstream.
See debdiff attached for the result against Bullseye's python-eventlet
diff -Nru python-eventlet-0.26.1/debian/changelog python-eventlet-0.26.1/debian/changelog
--- python-eventlet-0.26.1/debian/changelog 2021-05-11 08:03:43.0 +0200
+++ python-eventlet-0.26.1/debian/changelog 2021-08-24 14:04:54.0 +0200
@@ -1,3 +1,10 @@
+python-eventlet (0.26.1-8~wmf1) bullseye; urgency=medium
+
+ * Fix dnspython 2 compat
+ ** See also https://github.com/eventlet/eventlet/pull/722
+
+ -- Filippo Giunchedi Tue, 24 Aug 2021 14:04:54 +0200
+
python-eventlet (0.26.1-7) unstable; urgency=medium
* CVE-2021-21419: Malicious peer may exhaust memory on Eventlet side
diff -Nru python-eventlet-0.26.1/debian/greendns.orig.py python-eventlet-0.26.1/debian/greendns.orig.py
--- python-eventlet-0.26.1/debian/greendns.orig.py 2021-05-11 08:03:43.0 +0200
+++ python-eventlet-0.26.1/debian/greendns.orig.py 2021-08-24 14:04:54.0 +0200
@@ -120,12 +120,13 @@
return is_ipv4_addr(host) or is_ipv6_addr(host)
-def compute_expiration(query, timeout):
-# NOTE(ralonsoh): in dnspython v2.0.0, "_compute_expiration" was replaced
-# by "_compute_times".
-if hasattr(query, '_compute_expiration'):
+# NOTE(ralonsoh): in dnspython v2.0.0, "_compute_expiration" was replaced
+# by "_compute_times".
+if hasattr(dns.query, '_compute_expiration'):
+def compute_expiration(query, timeout):
return query._compute_expiration(timeout)
-else:
+else:
+def compute_expiration(query, timeout):
return query._compute_times(timeout)[1]
@@ -669,8 +670,21 @@
raise dns.exception.Timeout
+# Test if raise_on_truncation is an argument we should handle.
+# It was newly added in dnspython 2.0
+try:
+dns.message.from_wire("", raise_on_truncation=True)
+except dns.message.ShortHeader:
+_handle_raise_on_truncation = True
+except TypeError:
+# Argument error, there is no argument "raise_on_truncation"
+_handle_raise_on_truncation = False
+
+
def udp(q, where, timeout=DNS_QUERY_TIMEOUT, port=53,
-af=None, source=None, source_port=0, ignore_unexpected=False):
+af=None, source=None, source_port=0, ignore_unexpected=False,
+one_rr_per_rrset=False, ignore_trailing=False,
+raise_on_truncation=False, sock=None):
"""coro friendly replacement for dns.query.udp
Return the response obtained after sending a query via UDP.
@@ -695,7 +709,21 @@
@type source_port: int
@param ignore_unexpected: If True, ignore responses from unexpected
sources. The default is False.
-@type ignore_unexpected: bool"""
+@type ignore_unexpected: bool
+@param one_rr_per_rrset: If True, put each RR into its own
+RRset.
+@type one_rr_per_rrset: bool
+@param ignore_trailing: If True, ignore trailing
+junk at end of the received message.
+@type ignore_trailing: bool
+@param raise_on_truncation: If True, raise an exception if
+the TC bit is set.
+@type raise_on_truncation: bool
+@param sock: the socket to use for the
+query. If None, the default, a socket is created. Note that
+if a socket is provided, it must be a nonblocking datagram socket,
+and the source and source_port are ignored.
+@type sock: socket.socket | None"""
wire = q.to_wire()
if af is None:
@@ -717,7 +745,10 @@
if source is not None:
source = (source, source_port, 0, 0)
-s = socket.socket(af, socket.SOCK_DGRAM)
+if sock:
+s = sock
+else:
+s = socket.socket(af, socket.SOCK_DGRAM)
s.settimeout(timeout)
try:
expiration = compute_expiration(dns.query, timeout)
@@ -765,14 +796,23 @@
finally:
s.close()
-r = dns.message.from_wire(wire, keyring=q.keyring, request_mac=q.mac)
+if _handle_raise_on_truncation:
+r = dns.message.from_wire(wire, keyring=q.keyring, request_mac=q.mac,
+ one_rr_per_rrset=one_rr_per_rrset,
+ ignore_trailing=ignore_trailing,
+ raise_on_truncation=raise_on_truncation)
+else:
+r = dns.message.from_wire(wire, keyring=q.keyring, request_mac=q.mac,
+
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
On Tue, Jun 08, 2021 at 10:03 AM, Filippo Giunchedi wrote: > Package: swift-container > Version: 2.26.0-10 > Severity: important > File: /usr/bin/swift-container-reconciler > > Dear Maintainer, > I'm experimenting with Swift on Bullseye and came across a problem with > container-reconciler (possibly others) when using hostnames in > memcache_servers. Namely these errors: In the "possibly others" category, swift-dispersion-report is also 100% broken in Bullseye: $ swift-dispersion-report --dump-json swift-dispersion-report --dump-json -d Traceback (most recent call last): File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 435, in resolve return _proxy.query(name, rdtype, raise_on_no_answer=raises, File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 391, in query return end() File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 370, in end raise result[1] File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 351, in step a = fun(*args, **kwargs) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration) dns.exception.Timeout: The DNS operation timed out after 5.1069724559783936 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1310, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1380, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1301, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1089, in _send_output self.send(msg) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1018, in send self.connect() File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1481, in connect super().connect() File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 989, in connect self.sock = self._create_connection( File "/usr/lib/python3/dist-packages/eventlet/green/socket.py", line 44, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 528, in getaddrinfo qname, addrs = _getaddrinfo_lookup(host, family, flags) File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 501, in _getaddrinfo_lookup raise err File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 490, in _getaddrinfo_lookup answer = resolve(host, qfamily, False, use_network=use_network) File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 443, in resolve raise EAI_EAGAIN_ERROR File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 490, in _getaddrinfo_lookup answer = resolve(host, qfamily, False, use_network=use_network) File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 443, in resolve raise EAI_EAGAIN_ERROR File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1310, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1380, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1301, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1089, in _send_output self.send(msg) File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1018, in send self.connect() File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 1481, in connect super().connect() File "/usr/lib/python3/dist-packages/eventlet/green/http/client.py", line 989, in connect self.sock = self._create_connection( File "/usr/lib/python3/dist-packages/eventlet/green/socket.py", line 44, in create_connection for res in getaddrinfo(host, port, 0,
Bug#989600: /usr/bin/swift-container-reconciler: reconciler's memcache connections fail when using hostnames
Package: swift-container Version: 2.26.0-10 Severity: important File: /usr/bin/swift-container-reconciler Dear Maintainer, I'm experimenting with Swift on Bullseye and came across a problem with container-reconciler (possibly others) when using hostnames in memcache_servers. Namely these errors: Jun 08 09:54:08 ms-be-01 swift-container-reconciler[70736]: Timeout getting a connection to memcached: HOST1:11211: MemcachePoolTimeout (1.0s) (txn: txf2bfe46649374ed6b1a47-0060bf3e3f) Jun 08 09:54:09 ms-be-01 swift-container-reconciler[70736]: Timeout getting a connection to memcached: HOST2:11211: MemcachePoolTimeout (1.0s) (txn: txf2bfe46649374ed6b1a47-0060bf3e3f) and I have HOST1 HOST2 in container-reconciler.conf: memcache_servers = HOST1:11211,HOST2:11211 Manually testing the connection works as expected, and after some debugging it looks like using ip addresses in the configuration works, unlike using hostnames. In this case hostname resolution happens via DNS, which makes me think this is related to #971530. The bug is possibly affecting other parts of swift + memcache, though I haven't been able to find other examples in my testing so far. best, Filippo -- System Information: Debian Release: 11.0 APT prefers testing-security APT policy: (500, 'testing-security'), (500, 'testing-debug'), (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 5.10.0-6-cloud-amd64 (SMP w/1 CPU thread) Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages swift-container depends on: ii init-system-helpers 1.60 ii lsb-base 11.1.0 ii openstack-pkg-tools 117 ii python3 3.9.2-3 ii python3-pastescript 2.0.2-4 ii python3-swift 2.26.0-10 ii rsync 3.2.3-4 ii swift 2.26.0-10 ii uwsgi-plugin-python3 2.0.19.1-6 Versions of packages swift-container recommends: pn swift-drive-audit swift-container suggests no packages. -- Configuration Files: /etc/swift/container-reconciler.conf [Errno 13] Permission denied: '/etc/swift/container-reconciler.conf' /etc/swift/container-server.conf [Errno 13] Permission denied: '/etc/swift/container-server.conf' /etc/swift/internal-client.conf [Errno 13] Permission denied: '/etc/swift/internal-client.conf' /etc/swift/swift-container-server-uwsgi.ini [Errno 13] Permission denied: '/etc/swift/swift-container-server-uwsgi.ini' -- no debconf information
