[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-29 Thread Launchpad Bug Tracker
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.20.04.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.04.1) focal; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-requ

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-29 Thread dann frazier
= focal verification = # Start by showing we can still reproduce the problem w/o the -proposed packages: ubuntu@dannf-test:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io --algo bm -j DROP ubuntu@dannf-test:~$ cat > repro.py #!/usr/bin/env python3 from simplestreams.

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-29 Thread Launchpad Bug Tracker
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.20.10.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.20.10.1) groovy; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-req

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-29 Thread Launchpad Bug Tracker
This bug was fixed in the package simplestreams - 0.1.0-30-g3cc8988a- 0ubuntu1.21.04.1 --- simplestreams (0.1.0-30-g3cc8988a-0ubuntu1.21.04.1) hirsute; urgency=medium * Add 10s timeout to out-going requests to avoid blocking. - Via: d/patches/0001-Add-10s-timeout-to-out-going-re

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-29 Thread dann frazier
Restoring verification-needed-focal tag; bdmurray correctly points out my focal verification was actually done on groovy :( ** Tags removed: verification-done verification-done-focal ** Tags added: verification-needed verification-needed-focal -- You received this bug notification because you ar

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-22 Thread dann frazier
= focal verification = # Start by showing we can still reproduce the problem w/o the -proposed packages: ubuntu@avoton07:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io --algo bm -j DROP ubuntu@avoton07:~$ cat > repro.py #!/usr/bin/env python3 from simplestreams.cont

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-22 Thread Brian Murray
Hello Patricia, or anyone else affected, Accepted simplestreams into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.20.04.1 in a few hours, and then in the -proposed repository. Please help us by testi

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-22 Thread dann frazier
= groovy verification = # Start by showing we can still reproduce the problem w/o the -proposed packages: ubuntu@avoton07:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io --algo bm -j DROP ubuntu@avoton07:~$ cat > repro.py #!/usr/bin/env python3 from simplestreams.con

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-22 Thread dann frazier
= hirsute verification = # Start by showing we can still reproduce the problem w/o the -proposed packages: ubuntu@avoton02:~$ sudo iptables -A INPUT -p tcp -s 91.189.88.136 -m string --string maas.io --algo bm -j DROP ubuntu@avoton02:~$ python3 ./repro.py & sleep 60 [1] 3386 # 60 seconds have pas

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-18 Thread Timo Aaltonen
Hello Patricia, or anyone else affected, Accepted simplestreams into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.20.10.1 in a few hours, and then in the -proposed repository. Please help us by test

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-18 Thread Timo Aaltonen
Hello Patricia, or anyone else affected, Accepted simplestreams into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0-30-g3cc8988a- 0ubuntu1.21.04.1 in a few hours, and then in the -proposed repository. Please help us by tes

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-17 Thread Paride Legovini
** Changed in: simplestreams (Ubuntu Focal) Status: In Progress => Fix Committed ** Changed in: simplestreams (Ubuntu Groovy) Status: In Progress => Fix Committed ** Changed in: simplestreams (Ubuntu Hirsute) Status: In Progress => Fix Committed -- You received this bug not

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-16 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404261 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-16 Thread Paride Legovini
** Also affects: simplestreams (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: simplestreams (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: simplestreams (Ubuntu Groovy) Assignee: (unassigned) => Paride Legovini (paride) ** Changed i

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-16 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404259 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-15 Thread Paride Legovini
** Changed in: simplestreams (Ubuntu Focal) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-15 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~paride/ubuntu/+source/simplestreams/+git/simplestreams/+merge/404202 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-15 Thread Paride Legovini
** Description changed: + [Impact] + + The bug is about simplestreams possibly getting stuck waiting forever + for an an HTTP response that never comes, e.g. because of networking + issues. This can potentially affect any package depending on + simplestreams, but specifically it was reported affe

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-06-10 Thread Paride Legovini
** Changed in: simplestreams (Ubuntu Focal) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fai

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-05-18 Thread Paride Legovini
smoser sponsored the upload: $ dput ubuntu ../out/simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.changes Checking signature on .changes gpg: ../out/simplestreams_0.1.0-46-gb1f40d99-0ubuntu1_source.changes: Valid signature from 1E4410A4024BC6F0 Checking signature on .dsc gpg: ../out/simplestream

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-05-18 Thread Paride Legovini
** Also affects: simplestreams (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: simplestreams (Ubuntu Focal) Status: New => Triaged ** Changed in: simplestreams (Ubuntu) Status: Triaged => In Progress -- You received this bug notification because you are a

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-05-18 Thread Paride Legovini
** Changed in: simplestreams (Ubuntu) Assignee: (unassigned) => Paride Legovini (paride) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails aft

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-04-01 Thread dann frazier
We've hit this again on our production MAAS server, see below. We're in a position where we regularly need to check if our regiond processes are hung and restart them. I'd therefore like to ask for a bump in priority here. Any chance for a fix soon on the 2.9 branch? Does MAAS need simplestreams to

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-03-19 Thread dann frazier
** Description changed: = How to determine you are seeing this problem = Does your MAAS server seem to get "hung up", where deployments suddenly start failing w/ lots of connection timeouts to the MAAS server? Get a list of pids of your regiond processes: $ ps -ef | grep regiond R

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-03-16 Thread dann frazier
** Description changed: - We are having an issue with our production MAAS - The web UI is available normally, we can start to deploy, but the result is a failure - systems get stuck during `Loading ephemeral` step: + = How to determine you are seeing this problem = + Does your MAAS server seem t

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-03-16 Thread dann frazier
fyi, we saw this recur even after switching to an HTTP images mirror. We were seeing deployments fail, and I found a regiond process was again hung in a system call reading a socket from images.maas.io:80. This is our production MAAS so I didn't spend a lot of time debugging it - I kill -9'd the pr

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-04 Thread Paride Legovini
** Changed in: simplestreams (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral` ste

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-03 Thread Adam Collard
** Changed in: simplestreams Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemeral

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-02 Thread Adam Collard
** Changed in: simplestreams Status: New => In Progress ** Changed in: simplestreams Assignee: (unassigned) => Adam Collard (adam-collard) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/190

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-02 Thread Adam Collard
** Merge proposal linked: https://code.launchpad.net/~adam-collard/simplestreams/+git/simplestreams/+merge/397354 ** Changed in: maas Status: New => In Progress ** Changed in: maas Importance: Undecided => High ** Changed in: maas Assignee: (unassigned) => Lee Trager (ltrager)

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-02 Thread Adam Collard
** Also affects: simplestreams Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1908452 Title: MAAS stops working and deployment fails after `Loading ephemer

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-02-01 Thread Lee Trager
@dannf - Thanks for the great deep dive! It seems the issue is in python3-simplestreams. According to [1] "Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely" The timeout needs to be set in RequestsUrlReader wh

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-01-29 Thread dann frazier
Here's a demo of adding a timeout. I'm not sure if this has to be done in the simplestreams library or if it can somehow be set by a caller: $ diff -u /usr/lib/python3/dist-packages/simplestreams/contentsource.py.orig /usr/lib/python3/dist-packages/simplestreams/contentsource.py --- /usr/lib/pyth

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-01-29 Thread dann frazier
A reproducer would be nice you say? Well, you can point your MAAS server to this URL for MAAS images: https://images.maas.io/ephemeral-v3/daily/ Then you can simulate whatever-the-heck is going on in our lab environment by adding an iptables rule that will drop packets containing the server cer

[Bug 1908452] Re: MAAS stops working and deployment fails after `Loading ephemeral` step

2021-01-26 Thread dann frazier
Here's a similar report of pythonssl hanging in the handshake: https://bugs.python.org/issue34438 Suggestion there from upstream is to add a timeout. simplestreams seems to be using a requests object to make the connection. From simplestreams/contentsource.py: 381self.req = requests.ge