[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.20.04.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) focal; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-requests.patch Upstream cherry-pick. Thanks: Adam Collard. (LP: #1908452) -- Paride Legovini Fri, 11 Jun 2021 16:51:14 +0200 ** Changed in: simplestreams (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
= focal verification =
# Start by showing we can still reproduce the problem w/o the -proposed
packages:
ubuntu@dannf-test:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string
--string maas.io --algo bm -j DROP
ubuntu@dannf-test:~$ cat > repro.py
#!/usr/bin/env python3
from simplestreams.contentsource import RequestsUrlReader
url = "https://images.maas.io/ephemeral-v3/stable/streams/v1/index.sjson;
r = RequestsUrlReader(url)
ubuntu@dannf-test:~$ python3 ./repro.py & sleep 60
[1] 1687
ubuntu@dannf-test:~$ sudo strace -p 1687
strace: Process 1687 attached
read(3, ^Cstrace: Process 1687 detached
ubuntu@dannf-test:~$ fg
python3 ./repro.py
^CTraceback (most recent call last):
File "./repro.py", line 6, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers)
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1914, in
do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt
# Now upgrade and demonstrate the problem is fixed
ubuntu@dannf-test:~$ sudo apt install python3-simplestreams simplestreams
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
python3-simplestreams simplestreams
2 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Need to get 37.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64
python3-simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.04.1 [31.8 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64
simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.04.1 [6016 B]
Fetched 37.8 kB in 0s (96.3 kB/s)
(Reading database ... 58591 files and directories currently installed.)
Preparing to unpack .../python3-simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.04.
1_all.deb ...
Unpacking python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) over (0.1.
0-30-g3cc8988a-0ubuntu1) ...
Preparing to unpack .../simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.04.1_all.de
b ...
Unpacking simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) over (0.1.0-30-g3c
c8988a-0ubuntu1) ...
Setting up python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) ...
Setting up simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) ...
ubuntu@dannf-test:~$ python3 ./repro.py & sleep 60
[1] 3121
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in
do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1622, in
_raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.20.10.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) groovy; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-requests.patch Upstream cherry-pick. Thanks: Adam Collard. (LP: #1908452) -- Paride Legovini Fri, 11 Jun 2021 16:51:14 +0200 ** Changed in: simplestreams (Ubuntu Groovy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.21.04.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) hirsute; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-requests.patch Upstream cherry-pick. Thanks: Adam Collard. (LP: #1908452) -- Paride Legovini Fri, 11 Jun 2021 16:51:14 +0200 ** Changed in: simplestreams (Ubuntu Hirsute) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Restoring verification-needed-focal tag; bdmurray correctly points out my focal verification was actually done on groovy :( ** Tags removed: verification-done verification-done-focal ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
= focal verification =
# Start by showing we can still reproduce the problem w/o the -proposed
packages:
ubuntu@avoton07:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string
--string maas.io --algo bm -j DROP
ubuntu@avoton07:~$ cat > repro.py
#!/usr/bin/env python3
from simplestreams.contentsource import RequestsUrlReader
url = "https://images.maas.io/ephemeral-v3/stable/streams/v1/index.sjson;
r = RequestsUrlReader(url)
ubuntu@avoton07:~$ python3 ./repro.py & sleep 60
[1] 3225
ubuntu@avoton07:~$ sudo strace -p 3225
strace: Process 3225 attached
read(3, ^Cstrace: Process 3225 detached
ubuntu@avoton07:~$ fg
python3 ./repro.py
^CTraceback (most recent call last):
File "./repro.py", line 6, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers)
File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 530, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 670, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 361, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 377, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 488,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1933, in
do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt
# Now upgrade and demonstrate the problem is fixed
ubuntu@avoton07:~$ sudo apt install simplestreams python3-simplestreams
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
python3-simplestreams simplestreams
2 upgraded, 0 newly installed, 0 to remove and 67 not upgraded.
Need to get 37.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64
python3-simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.10.1 [31.8 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64
simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.10.1 [6012 B]
Fetched 37.8 kB in 0s (91.6 kB/s)
(Reading database ... 77499 files and directories currently installed.)
Preparing to unpack
.../python3-simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.10.1_all.deb ...
Unpacking python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Preparing to unpack
.../simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.10.1_all.deb ...
Unpacking simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Setting up python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) ...
Setting up simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) ...
ubuntu@avoton07:~$ python3 ./repro.py & sleep 60
[1] 3406
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 488,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1934, in
do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1646, in
_raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 361, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 377, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Hello Patricia, or anyone else affected, Accepted simplestreams into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.20.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
= groovy verification =
# Start by showing we can still reproduce the problem w/o the -proposed
packages:
ubuntu@avoton07:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string
--string maas.io --algo bm -j DROP
ubuntu@avoton07:~$ cat > repro.py
#!/usr/bin/env python3
from simplestreams.contentsource import RequestsUrlReader
url = "https://images.maas.io/ephemeral-v3/stable/streams/v1/index.sjson;
r = RequestsUrlReader(url)
ubuntu@avoton07:~$ python3 ./repro.py & sleep 60
[1] 3225
# 60 seconds have passed, still hung:
ubuntu@avoton07:~$ sudo strace -p 3225
strace: Process 3225 attached
read(3, ^Cstrace: Process 3225 detached
ubuntu@avoton07:~$ fg
python3 ./repro.py
^CTraceback (most recent call last):
File "./repro.py", line 6, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers)
File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 530, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 670, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 361, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 377, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 488,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1933, in
do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt
# Now upgrade and demonstrate the problem is fixed
ubuntu@avoton07:~$ sudo apt install simplestreams python3-simplestreams
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
python3-simplestreams simplestreams
2 upgraded, 0 newly installed, 0 to remove and 67 not upgraded.
Need to get 37.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64
python3-simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.10.1 [31.8 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64
simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.20.10.1 [6012 B]
Fetched 37.8 kB in 0s (91.6 kB/s)
(Reading database ... 77499 files and directories currently installed.)
Preparing to unpack
.../python3-simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.10.1_all.deb ...
Unpacking python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Preparing to unpack
.../simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.20.10.1_all.deb ...
Unpacking simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Setting up python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) ...
Setting up simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) ...
ubuntu@avoton07:~$ python3 ./repro.py & sleep 60
[1] 3406
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 488,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1934, in
do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1646, in
_raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 361, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 377, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
= hirsute verification =
# Start by showing we can still reproduce the problem w/o the -proposed
packages:
ubuntu@avoton02:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string
--string maas.io --algo bm -j DROP
ubuntu@avoton02:~$ python3 ./repro.py & sleep 60
[1] 3386
# 60 seconds have passed, still hung:
ubuntu@avoton02:~$ sudo strace -p 3386
strace: Process 3386 attached
read(3, ^Cstrace: Process 3386 detached
ubuntu@avoton02:~$ fg
python3 ./repro.py
^CTraceback (most recent call last):
File "/home/ubuntu/./repro.py", line 6, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers)
File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012,
in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 428, in
ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 472, in
_ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
KeyboardInterrupt
# Now upgrade and demonstrate the problem is fixed
ubuntu@avoton02:~$ sudo apt install python3-simplestreams simplestreams -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
python3-simplestreams simplestreams
2 upgraded, 0 newly installed, 0 to remove and 68 not upgraded.
Need to get 31.8 kB/37.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu hirsute-proposed/main amd64
python3-simplestreams all 0.1.0-30-g3cc8988a-0ubuntu1.21.04.1 [31.8 kB]
Fetched 31.8 kB in 0s (119 kB/s)
(Reading database ... 79414 files and directories currently installed.)
Preparing to unpack
.../python3-simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.21.04.1_all.deb ...
Unpacking python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Preparing to unpack
.../simplestreams_0.1.0-30-g3cc8988a-0ubuntu1.21.04.1_all.deb ...
Unpacking simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) over
(0.1.0-30-g3cc8988a-0ubuntu1) ...
Setting up python3-simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) ...
Setting up simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...
Running kernel seems to be up-to-date.
The processor microcode seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
ubuntu@avoton02:~$ python3 ./repro.py & sleep 60
[1] 3605
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012,
in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 428, in
ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 472, in
_ssl_wrap_socket_impl
return
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Hello Patricia, or anyone else affected, Accepted simplestreams into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.20.10.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-groovy. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Tags added: verification-needed-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Hello Patricia, or anyone else affected, Accepted simplestreams into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.21.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Tags added: verification-needed verification-needed-hirsute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams (Ubuntu Focal) Status: In Progress => Fix Committed ** Changed in: simplestreams (Ubuntu Groovy) Status: In Progress => Fix Committed ** Changed in: simplestreams (Ubuntu Hirsute) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404261 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Also affects: simplestreams (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: simplestreams (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: simplestreams (Ubuntu Groovy) Assignee: (unassigned) => Paride Legovini (paride) ** Changed in: simplestreams (Ubuntu Hirsute) Assignee: (unassigned) => Paride Legovini (paride) ** Changed in: simplestreams (Ubuntu Groovy) Status: New => In Progress ** Changed in: simplestreams (Ubuntu Hirsute) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404259 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams (Ubuntu Focal) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404202 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Description changed: + [Impact] + + The bug is about simplestreams possibly getting stuck waiting forever + for an an HTTP response that never comes, e.g. because of networking + issues. This can potentially affect any package depending on + simplestreams, but specifically it was reported affecting MAAS, where it + causes server deployments to timeout. + + [Test Plan] + + Ideally this should be tested by building a MAAS snap with the + simplestreams package including the fix, verifying that is works as + expected. + + [Regression Potential] + + Very little. Scenarios where it takes more than 10s for a remote server + to provide simplestreams with the data it requested are unlikely, but + can't be fully excluded. + + [Original Description] + = How to determine you are seeing this problem = Does your MAAS server seem to get "hung up", where deployments suddenly start failing w/ lots of connection timeouts to the MAAS server? Get a list of pids of your regiond processes: $ ps -ef | grep regiond Run strace on each one to see if one is stuck in a connect() or recv() call: $ sudo strace -p $pid recv(... (normally you should see a lot of epoll_ctl() calls go by if not hung) If one is hung, use lsof to see what it is connected to: sudo lsof -i -a -p $pid If you see an open connection to your images server, then this maybe your problem. sudo kill -9 of the hung pid will cause it to respawn and recover. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams (Ubuntu Focal) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
smoser sponsored the upload: $ dput ubuntu ../out/simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.changes Checking signature on .changes gpg: ../out/simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.changes: Valid signature from 1E4410A4024BC6F0 Checking signature on .dsc gpg: ../out/simplestreams_0.1.0-46-gb1f40d99-0ubuntu1.dsc: Valid signature from 1E4410A4024BC6F0 Uploading to ubuntu (via ftp to upload.ubuntu.com): Uploading simplestreams_0.1.0-46-gb1f40d99-0ubuntu1.dsc: done. Uploading simplestreams_0.1.0-46-gb1f40d99.orig.tar.gz: done. Uploading simplestreams_0.1.0-46-gb1f40d99-0ubuntu1.debian.tar.xz: done. Uploading simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.buildinfo: done. Uploading simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.changes: done. Successfully uploaded packages. Unfortunately the fix for this bug (f37d2ed2) doesn't mention this bug with the right syntax, so this bug won't get auto-updated by Launchpad. Will update it manually. ** Changed in: simplestreams Status: Fix Committed => Fix Released ** Changed in: simplestreams (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Also affects: simplestreams (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: simplestreams (Ubuntu Focal) Status: New => Triaged ** Changed in: simplestreams (Ubuntu) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams (Ubuntu) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
We've hit this again on our production MAAS server, see below. We're in a position where we regularly need to check if our regiond processes are hung and restart them. I'd therefore like to ask for a bump in priority here. Any chance for a fix soon on the 2.9 branch? Does MAAS need simplestreams to cut a release before this fix can be picked up? ubuntu@maas:~$ ps -ef | grep region root 633597 633491 0 14:07 ?00:02:28 python3 /snap/maas/11778/sbin/regiond root 633814 633597 0 14:07 ?00:33:34 python3 /snap/maas/11778/sbin/regiond root 633816 633597 0 14:07 ?00:34:25 python3 /snap/maas/11778/sbin/regiond root 633817 633597 0 14:07 ?00:35:24 python3 /snap/maas/11778/sbin/regiond root 633818 633597 0 14:07 ?00:34:17 python3 /snap/maas/11778/sbin/regiond ubuntu648515 648489 0 18:19 pts/000:00:00 grep --color=auto region ubuntu@maas:~$ sudo strace -p 633597 strace: Process 633597 attached epoll_wait(3, [], 10, 94) = 0 epoll_wait(3, [], 10, 0)= 0 epoll_wait(3, [], 10, 99) = 0 epoll_wait(3, [], 10, 399) = 0 epoll_wait(3, [], 10, 0)= 0 epoll_wait(3, ^Cstrace: Process 633597 detached ubuntu@maas:~$ sudo strace -p 633814 strace: Process 633814 attached recvfrom(30, ^Cstrace: Process 633814 detached ubuntu@maas:~$ sudo lsof -i -a -p 633814 COMMANDPID USER FD TYPEDEVICE SIZE/OFF NODE NAME python3 633814 root 10u IPv6 696471999 0t0 TCP maas:5240->10.229.192.54:54706 (ESTABLISHED) python3 633814 root 14u IPv6 680163730 0t0 TCP *:5251 (LISTEN) python3 633814 root 15u IPv4 696151424 0t0 TCP localhost:51510->localhost:postgresql (ESTABLISHED) python3 633814 root 16u IPv6 683429160 0t0 TCP maas:5240->1.general.morphis.uk.vpn:38664 (CLOSE_WAIT) python3 633814 root 17u IPv4 680183835 0t0 TCP localhost:59674->localhost:postgresql (ESTABLISHED) python3 633814 root 18u IPv4 696173835 0t0 TCP localhost:51520->localhost:postgresql (ESTABLISHED) python3 633814 root 19u IPv6 680176399 0t0 TCP maas:5251->maas:45382 (CLOSE_WAIT) python3 633814 root 20u IPv4 696078469 0t0 TCP localhost:51486->localhost:postgresql (ESTABLISHED) python3 633814 root 21u IPv6 680176401 0t0 TCP *:5240 (LISTEN) python3 633814 root 22u IPv4 696206297 0t0 TCP localhost:51932->localhost:postgresql (ESTABLISHED) python3 633814 root 23u IPv6 680199940 0t0 TCP maas:5240->2.general.dannf.us.vpn:55866 (CLOSE_WAIT) python3 633814 root 26u IPv6 680154669 0t0 TCP maas:5240->2.general.dannf.us.vpn:55848 (ESTABLISHED) python3 633814 root 27u IPv4 696430193 0t0 TCP localhost:51948->localhost:postgresql (ESTABLISHED) python3 633814 root 28u IPv6 680199946 0t0 TCP maas:5240->2.general.dannf.us.vpn:55868 (CLOSE_WAIT) python3 633814 root 29u IPv4 696373443 0t0 TCP localhost:51938->localhost:postgresql (ESTABLISHED) python3 633814 root 30u IPv4 696457905 0t0 TCP maas:49652->images-maas-io.sawo.canonical.com:http (ESTABLISHED) python3 633814 root 31u IPv4 696394600 0t0 TCP localhost:51944->localhost:postgresql (ESTABLISHED) python3 633814 root 32u IPv4 696155256 0t0 TCP localhost:51502->localhost:postgresql (ESTABLISHED) ubuntu@maas:~$ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Description changed: = How to determine you are seeing this problem = Does your MAAS server seem to get "hung up", where deployments suddenly start failing w/ lots of connection timeouts to the MAAS server? Get a list of pids of your regiond processes: $ ps -ef | grep regiond Run strace on each one to see if one is stuck in a connect() or recv() call: $ sudo strace -p $pid recv(... (normally you should see a lot of epoll_ctl() calls go by if not hung) If one is hung, use lsof to see what it is connected to: sudo lsof -i -a -p $pid If you see an open connection to your images server, then this maybe - your problem. + your problem. sudo kill -9 of the hung pid will cause it to respawn and + recover. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Description changed: - We are having an issue with our production MAAS - The web UI is available normally, we can start to deploy, but the result is a failure - systems get stuck during `Loading ephemeral` step: + = How to determine you are seeing this problem = + Does your MAAS server seem to get "hung up", where deployments suddenly start failing w/ lots of connection timeouts to the MAAS server? - ``` - Tue, 15 Dec. 2020 23:08:57Node - Powered off 'akis'. - Tue, 15 Dec. 2020 23:05:25Marking node failed - Node operation 'Deploying' timed out after 30 minutes. - Tue, 15 Dec. 2020 22:35:31Loading ephemeral - Tue, 15 Dec. 2020 22:34:35Performing PXE boot - Tue, 15 Dec. 2020 22:31:35Powering node on - Tue, 15 Dec. 2020 22:31:35Node - Started deploying 'akis'. - Tue, 15 Dec. 2020 22:31:35Deploying - Tue, 15 Dec. 2020 22:31:09Node - Acquired 'akis'. - ``` + Get a list of pids of your regiond processes: + $ ps -ef | grep regiond - It's the 3rd time we are seeing this behavior, which is fixed after a - restart. + Run strace on each one to see if one is stuck in a connect() or recv() call: + $ sudo strace -p $pid + recv(... - MAAS version: 2.8.2 (8577-g.a3e674063) + (normally you should see a lot of epoll_ctl() calls go by if not hung) + + If one is hung, use lsof to see what it is connected to: + sudo lsof -i -a -p $pid + + If you see an open connection to your images server, then this maybe + your problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
fyi, we saw this recur even after switching to an HTTP images mirror. We were seeing deployments fail, and I found a regiond process was again hung in a system call reading a socket from images.maas.io:80. This is our production MAAS so I didn't spend a lot of time debugging it - I kill -9'd the process and MAAS respawned/recovered. But the highlight here is that this does not appear to be specific to SSL connections. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Changed in: simplestreams Status: New => In Progress ** Changed in: simplestreams Assignee: (unassigned) => Adam Collard (adam-collard) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Merge proposal linked: https://code.launchpad.net/~adam-collard/simplestreams/+git/simplestreams/+merge/397354 ** Changed in: maas Status: New => In Progress ** Changed in: maas Importance: Undecided => High ** Changed in: maas Assignee: (unassigned) => Lee Trager (ltrager) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
** Also affects: simplestreams Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
@dannf - Thanks for the great deep dive! It seems the issue is in python3-simplestreams. According to [1] "Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely" The timeout needs to be set in RequestsUrlReader which is fairly deep in SimpleStreams code, MAAS does not directly interact with it at all. simplestreams uses Urllib2UrlReader as an alternative if requests isn't available and no timeout is set there either. RequestsUrlReader/Urllib2UrlReader is called by UrlMirrorReader which MAAS does call. We need to figure out a good default timeout value for both RequestsUrlReader and Urllib2UrlReader. We also need to determine whether this should be configurable by the caller or not, MAAS will most likely use the default value. [1] https://requests.readthedocs.io/en/master/user/quickstart/#timeouts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Here's a demo of adding a timeout. I'm not sure if this has to be done
in the simplestreams library or if it can somehow be set by a caller:
$ diff -u /usr/lib/python3/dist-packages/simplestreams/contentsource.py.orig
/usr/lib/python3/dist-packages/simplestreams/contentsource.py
--- /usr/lib/python3/dist-packages/simplestreams/contentsource.py.orig
2021-01-29 21:39:55.695029811 +
+++ /usr/lib/python3/dist-packages/simplestreams/contentsource.py
2021-01-29 21:40:57.327987600 +
@@ -378,7 +378,7 @@
if headers == {}:
headers = None
-self.req = requests.get(url, stream=True, auth=auth, headers=headers)
+self.req = requests.get(url, stream=True, auth=auth, headers=headers,
timeout=10)
self.r_iter = None
if buflen is None:
buflen = READ_BUFFER_SIZE
--
$ ./repro.py
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in
do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1622, in
_raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 488,
in wrap_socket
raise timeout("select timed out")
socket.timeout: select timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in
urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in
increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in
_make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 330, in
_raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='images.maas.io',
port=443): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./repro.py", line 8, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers,
timeout=10)
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='images.maas.io',
port=443): Read timed out. (read timeout=10)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1908452
Title:
MAAS stops working and deployment fails after `Loading ephemeral` step
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
A reproducer would be nice you say? Well, you can point your MAAS server to
this URL for MAAS images:
https://images.maas.io/ephemeral-v3/daily/
Then you can simulate whatever-the-heck is going on in our lab
environment by adding an iptables rule that will drop packets containing
the server cert from the image server:
-
$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io
--algo bm -j DROP
-
Then initiate an image sync. You'll find that roughly 25% of API
connections hang indefinitely:
-
ubuntu@maas:~$ wget --tries=1 http://localhost:5240/MAAS/rpc -O /dev/null
--2021-01-29 20:59:42-- http://localhost:5240/MAAS/rpc
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5240... connected.
HTTP request sent, awaiting response...
-
What's that? Would be nice if it were a standalone reproducer? Well, you
can do the same thing with the same iptables rule and a simple script
that makes the same simplestreams API call as regiond does:
-
$ cat repro.py
#!/usr/bin/env python3
from simplestreams.contentsource import RequestsUrlReader
url = "https://images.maas.io/ephemeral-v3/stable/streams/v1/index.sjson;
r = RequestsUrlReader(url)
-
This will hang seemingly forever. strace can be used to demonstrate it
is also hung in a read call, as I showed regiond was in Comment #11:
-
$ sudo strace -p 562333
strace: Process 562333 attached
read(20, ^Cstrace: Process 562333 detached
-
And if you ^c it, it will show same backtrace I extracted from regiond
in comment #17:
-
$ ./repro.py
^CTraceback (most recent call last):
File "./repro.py", line 6, in
r = RequestsUrlReader(url)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line
381, in __init__
self.req = requests.get(url, stream=True, auth=auth, headers=headers)
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in
urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in
_make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in
_validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in
connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in
ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485,
in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1914, in
do_handshake
result = _lib.SSL_do_handshake(self._ssl)
KeyboardInterrupt
-
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1908452
Title:
MAAS stops working and deployment fails after `Loading ephemeral` step
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step
Here's a similar report of pythonssl hanging in the handshake: https://bugs.python.org/issue34438 Suggestion there from upstream is to add a timeout. simplestreams seems to be using a requests object to make the connection. From simplestreams/contentsource.py: 381self.req = requests.get(url, stream=True, auth=auth, headers=headers) And while I'm not sure this is the canonical docs for this version, I did come across a statement at https://requests.readthedocs.io/en/master/user/quickstart/ that says: = Note = timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do not time out. So it seems like adding a timeout might be the way to go. I'm not sure at what layer though - should simplestreams hardcode one itself? Should it allow one to be passed in? ** Bug watch added: Python Roundup #34438 http://bugs.python.org/issue34438 ** Also affects: simplestreams (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
