A reproducer would be nice you say? Well, you can point your MAAS server to this URL for MAAS images: https://images.maas.io/ephemeral-v3/daily/
Then you can simulate whatever-the-heck is going on in our lab environment by adding an iptables rule that will drop packets containing the server cert from the image server: ------------- $ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io --algo bm -j DROP ------------- Then initiate an image sync. You'll find that roughly 25% of API connections hang indefinitely: ------------- ubuntu@maas:~$ wget --tries=1 http://localhost:5240/MAAS/rpc -O /dev/null --2021-01-29 20:59:42-- http://localhost:5240/MAAS/rpc Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:5240... connected. HTTP request sent, awaiting response... ------------- What's that? Would be nice if it were a standalone reproducer? Well, you can do the same thing with the same iptables rule and a simple script that makes the same simplestreams API call as regiond does: ------------------------- $ cat repro.py #!/usr/bin/env python3 from simplestreams.contentsource import RequestsUrlReader url = "https://images.maas.io/ephemeral-v3/stable/streams/v1/index.sjson"; r = RequestsUrlReader(url) ------------------------- This will hang seemingly forever. strace can be used to demonstrate it is also hung in a read call, as I showed regiond was in Comment #11: ------------------------- $ sudo strace -p 562333 strace: Process 562333 attached read(20, ^Cstrace: Process 562333 detached <detached ...> ------------------------- And if you ^c it, it will show same backtrace I extracted from regiond in comment #17: ------------------------- $ ./repro.py ^CTraceback (most recent call last): File "./repro.py", line 6, in <module> r = RequestsUrlReader(url) File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line 381, in __init__ self.req = requests.get(url, stream=True, auth=auth, headers=headers) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect self.sock = ssl_wrap_socket( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket cnx.do_handshake() File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1914, in do_handshake result = _lib.SSL_do_handshake(self._ssl) KeyboardInterrupt ------------------------- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` step To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1908452/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
