[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Thank you for resolving this well before Let's Encrypt's brownouts! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
This bug was fixed in the package python-certbot - 0.27.0-1~ubuntu18.04.2 --- python-certbot (0.27.0-1~ubuntu18.04.2) bionic; urgency=medium * Interpret the Let's Encrypt ACMEv1 endpoint in renewal configs as Let's Encrypt's ACMEv2 endpoint instead, as the former is deprecated and will stop being functional soon (LP: #1893274). -- Robie Basak Tue, 27 Oct 2020 16:00:51 + ** Changed in: python-certbot (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
This bug was fixed in the package python-certbot - 0.40.0-1ubuntu0.1 --- python-certbot (0.40.0-1ubuntu0.1) focal; urgency=medium * Interpret the Let's Encrypt ACMEv1 endpoint in renewal configs as Let's Encrypt's ACMEv2 endpoint instead, as the former is deprecated and will stop being functional soon (LP: #1893274). * Remove failing test since TLSSNI01 objects are no longer exported by python3-acme (LP: #1876933). -- Robie Basak Mon, 26 Oct 2020 14:42:45 + ** Changed in: python-certbot (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Prepared the test env as described above on Bionic and Focal. With the current -release package on renewal I got: Focal: ubuntu@bos01-amd64-certbot-focal:~$ sudo certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/cb-test-focal.dd-dns.de.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Plugins selected: Authenticator apache, Installer apache Attempting to renew cert (cb-test-focal.dd-dns.de) from /etc/letsencrypt/renewal/cb-test-focal.dd-dns.de.conf produced an unexpected error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/47ae0d179cac064a0853a666b64b9017 does not exist. Skipping. All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-focal.dd-dns.de/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-focal.dd-dns.de/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 renew failure(s), 0 parse failure(s) Bionic ubuntu@bos01-amd64-certbot-bionic:~$ sudo certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/cb-test-bionic.dd-dns.de.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Plugins selected: Authenticator apache, Installer apache Attempting to renew cert (cb-test-bionic.dd-dns.de) from /etc/letsencrypt/renewal/cb-test-bionic.dd-dns.de.conf produced an unexpected error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/5ece9e900665a4ad152750c4869a6214 does not exist. Skipping. All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-bionic.dd-dns.de/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-bionic.dd-dns.de/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 renew failure(s), 0 parse failure(s) Upgrading to proposed ... Focal: ubuntu@bos01-amd64-certbot-focal:~$ sudo apt install certbot python3-certbot Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: python3-certbot-nginx python-certbot-doc The following packages will be upgraded: certbot python3-certbot 2 upgraded, 0 newly installed, 0 to remove and 8 not upgraded. Need to get 241 kB of archives. After this operation, 2048 B disk space will be freed. Get:1 http://us.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 certbot all 0.40.0-1ubuntu0.1 [17.9 kB] Get:2 http://us.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 python3-certbot all 0.40.0-1ubuntu0.1 [223 kB] Fetched 241 kB in 0s (9692 kB/s) (Reading database ... 96443 files and directories currently installed.) Preparing to unpack .../certbot_0.40.0-1ubuntu0.1_all.deb ... Unpacking certbot (0.40.0-1ubuntu0.1) over (0.40.0-1) ... Preparing to unpack .../python3-certbot_0.40.0-1ubuntu0.1_all.deb ... Unpacking python3-certbot (0.40.0-1ubuntu0.1) over (0.40.0-1) ... Setting up python3-certbot (0.40.0-1ubuntu0.1) ... Setting up certbot (0.40.0-1ubuntu0.1) ... Processing triggers for man-db (2.9.1-1) ... Bionic: ubuntu@bos01-amd64-certbot-bionic:~$ sudo apt install certbot python3-certbot Reading package lists... Done Building dependency tree Reading state information... Done The following package was automatically installed and is no longer required: grub-pc-bin Use 'sudo apt autoremove' to remove it. Suggested packages: python3-certbot-nginx python-certbot-doc The following packages will be upgraded: certbot python3-certbot 2 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. Need to get 219 kB of archives. After this operation, 3072 B of additional disk space will be used. Get:1 http://us.archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 certbot all 0.27.0-1~ubuntu18.04.2 [18.1 kB] Get:2 http://us.archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 python3-certbot all 0.27.0-1~ubuntu18.04.2 [201 kB] Fetched 219 kB in 0s (14.8 MB/s) (Reading database ... 93724 files and directories currently installed.) Preparing to unpack .../certbot_0.27.0-1~ubuntu18.04.2_all.deb ... Unpacking certbot (0.27.0-1~ubuntu18.04.2) over (0.27.0-1~ubuntu18.04.1) ... Preparing to unpack .../python3-certbot_0.27.0-1~ubuntu18.04.2_all.deb ... Unpacking python3-certbot (0.27.0-1~ubuntu18.04.2) over (0.27.0-1~ubuntu18.04.1) ... Setting up
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Hello Erica, or anyone else affected, Accepted python-certbot into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python- certbot/0.40.0-1ubuntu0.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: python-certbot (Ubuntu Focal) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-focal ** Changed in: python-certbot (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Ok we have: - the packaging changes ready - reviewed the packaging changes - created a test build - come up with test instructions for verification - exercised the test on the test build I think this is ready to enter SRU [1] processing. I'm uploading to Focal/Bionic-unapproved [1]: https://wiki.ubuntu.com/StableReleaseUpdates -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Thank you all, I'll take over as Robie is unavailable atm. The test suggestions of Brad were awesome, I think I have created something SRU compatible out of that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
We need a Bionic and a Focal system for that reachable from the Internet with a DNS name. I have created those test systems temporarily on canonistack (an internal openstack service) These notes are mostly for myself to re-do it once things land in proposed, but also illustrate for everyone how this can be done. # get Focal $ openstack server create --key-name paelzer_canonistack-bos01 --flavor cpu2-ram2-disk10 --image 3eee21dc-0862-4181-b2cc-7ece13651edf bos01-amd64-certbot-focal $ openstack floating ip create net_external $ openstack server add floating ip bos01-amd64-certbot-focal # get Bionic $ openstack server create --key-name paelzer_canonistack-bos01 --flavor cpu2-ram2-disk10 --image dffafdf1-8a98-4b7a-9809-6f9d406da38f bos01-amd64-certbot-bionic $ openstack floating ip create net_external $ openstack server add floating ip bos01-amd64-certbot-bionic # open for 80/443 $ openstack security group rule create --dst-port 80 --protocol tcp default $ openstack security group rule create --dst-port 443 --protocol tcp default # I had no free DNS service at hand, but domains left # So I created two DNS names for those two public IPs that I had # Not sharing this here thou to keep my api key safe :-) # I ended up with cb-test-focal.dd-dns.de / cb-test-focal.dd-dns.de which I # can re-use for this and then will drop. # get certbot (I'll use apache for testing) $ sudo apt install python3-certbot python3-certbot-apache # Get a ACMEv2 cert (current default) $ sudo certbot --apache # go along the questions and use the DNS you have set up # modify the server endpoint to v1 manually # thanks Brad for the suggestion $ sudo sed -i -e 's/acme-v02/acme-v01/g' /etc/letsencrypt/renewal/* # renew Cert (will try to use the patched v1 sever) $ sudo certbot renew --force-renewal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Testing the test instructions: Without the fix this indeed fails Note: You can run it with --verbose to see the issue in more detail if needed. Plugins selected: Authenticator apache, Installer apache Attempting to renew cert (cb-test-bionic.dd-dns.de) from /etc/letsencrypt/renewal/cb-test-bionic.dd-dns.de.conf produced an unexpected error: Account at /etc/letsencrypt/accounts/acme-v01.api. letsencrypt.org/directory/ does not exist. Skipping. All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-bionic.dd-dns.de/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/cb-test-bionic.dd-dns.de/fullchain.pem (failure) Install the fix (from -proposed eventually). Upgrade works fine. Running renewal again with the fix works: $ sudo certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/cb-test-focal.dd-dns.de.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Using server https://acme-v02.api.letsencrypt.org/directory instead of legacy https://acme-v01.api.letsencrypt.org/directory Plugins selected: Authenticator apache, Installer apache Renewing an existing certificate - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - new certificate deployed with reload of apache server; fullchain is /etc/letsencrypt/live/cb-test-focal.dd-dns.de/fullchain.pem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all renewals succeeded. The following certs have been renewed: /etc/letsencrypt/live/cb-test-focal.dd-dns.de/fullchain.pem (success) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - We can see the expected avme-v2 redirection message \o/ And renewal worked. $ sudo certbot certificates Before Expiry Date: 2021-02-03 13:11:37+00:00 (VALID: 89 days) Now Expiry Date: 2021-02-03 13:24:49+00:00 (VALID: 89 days) ** Description changed: [Impact] Certbot users who first used < 0.26.0 have their configurations locked into using ACMEv1. This is a deprecated protocol. Let's Encrypt brownouts for ACMEv1 are scheduled to begin at the beginning of 2021, and Let's Encrypt will stop serving ACMEv1 in June 2021. Based on Let's Encrypt's metrics, 23,847 users were counted as being locked into ACMEv1 in this way. These users will start receiving certification renewal failures unless they are patched. Users affected are users who first used Certbot on Xenial or first used Certbot on the release pocket version of Certbot in Bionic. Users who first used Certbot >= 0.26.0 are not affected. This includes users who used Certbot on Bionic after 0.27.0-1~ubuntu18.04.1 (published 2019-10-29) and users who first used Certbot on Focal or above. [Test Case] - TBC + Get Focal/Bionic systems that you can get a public IP and DNSname on + comment #17 shows how to do so with Canonistack, but any other method is fine as well + + 1. get certbot (I'll use apache for testing) + $ sudo apt install apache2 python3-certbot python3-certbot-apache + 2. Get a ACMEv2 cert (current default) + $ sudo certbot --apache + # go along the questions and use the DNS you have set up + 3. modify the server endpoint to v1 manually + $ sudo sed -i -e 's/acme-v02/acme-v01/g' /etc/letsencrypt/renewal/* + 4. renew Cert (will try to use the patched v1 sever) + $ sudo certbot renew --force-renewal + + This will fail without the fix and "simulate" what will happen to old installs (which had a v1 config) after the upgrade. + Comment #18 has sample output of good/bad case. [Regression Potential] Since the endpoint is being changed, users who are controlling reachable endpoints (such as with egress firewalls or proxies) may not be able to reach the new endpoint until they have adjusted their configurations. However as the old endpoint will stop functioning soon, deliberately making this change appears to be the least worst option. Renewal configuration parsing of the server URL is being modified. Users with unusual configurations such as those that have different server URLs defined may find themselves on untested paths. Users trying to debug a problem configuration will find it surprising that a configuration that specifies the LE ACMEv1 endpoint goes to the LE ACMEv2 endpoint instead. However, again this seems to be the least worst option. [Further Details] Let’s Encrypt is in the process of shutting down ACMEv1. The full shutdown process will be completed in June 2021 with temporary brown- outs starting at the beginning of the year; more specific details are available at
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
** Changed in: python-certbot (Ubuntu Bionic) Assignee: Robie Basak (racb) => Christian Ehrhardt (paelzer) ** Changed in: python-certbot (Ubuntu Focal) Assignee: Robie Basak (racb) => Christian Ehrhardt (paelzer) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Testing this is pretty tricky because it's specific to the setup of Let's Encrypt's production ACMEv1 endpoint which no longer lets people create accounts or obtain certificates for new domains and we hardcoded the server URL. The way I'd test it is: 1. Obtain a real, trusted certificate from Let's Encrypt's production ACMEv2 endpoint. This is the default server for Certbot. 2. Run a command like `sudo sed -i 's/acme-v02/acme-v01/g' /etc/letsencrypt/renewal/*` to change the saved `server` value from ACMEv2 to ACMEv1. 3. Run `sudo certbot renew --force-renewal`. Pre-patch, Certbot should crash because it tries to use ACMEv1 (and an account doesn't exist), but post-patch it should switch to ACMEv2 and successfully renew the certificate. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
I think that's the same setup I mentioned above, where account reuse only works in one direction, and therefore it fails because the v2 account isn't present on the v1 server. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Unfortunately, pebble doesn't support ACMEv1. This is the same problem I've run into on the Debian side; I've backported the patches and prepared updated packages, but I'm hesitant to send them for upload to stable without some mechanism to test them. LE's testing server no longer responds to ACMEv1 renewals, correct? If I got a new certificate using ACMEv2, manually added the v1 URLs into the renewal config, and tried to do a renewal, would that error? On Fri, Oct 30, 2020 at 7:20 PM Erica Portnoy <1893...@bugs.launchpad.net> wrote: > > We use pebble for testing: https://github.com/letsencrypt/pebble/ > > I'm actually not personally familiar with it though, as I haven't had > the need to set one up and modify it myself, but I understand it's > fairly straightforward to get running. > > -- > You received this bug notification because you are subscribed to python- > certbot in Ubuntu. > https://bugs.launchpad.net/bugs/1893274 > > Title: > Certbot will stop working for 23,847 users with upcoming Let's Encrypt > deprecation > > Status in python-certbot package in Ubuntu: > Fix Released > Status in python-certbot source package in Bionic: > In Progress > Status in python-certbot source package in Focal: > In Progress > > Bug description: > [Impact] > > Certbot users who first used < 0.26.0 have their configurations locked > into using ACMEv1. This is a deprecated protocol. Let's Encrypt > brownouts for ACMEv1 are scheduled to begin at the beginning of 2021, > and Let's Encrypt will stop serving ACMEv1 in June 2021. > > Based on Let's Encrypt's metrics, 23,847 users were counted as being > locked into ACMEv1 in this way. These users will start receiving > certification renewal failures unless they are patched. > > Users affected are users who first used Certbot on Xenial or first > used Certbot on the release pocket version of Certbot in Bionic. > > Users who first used Certbot >= 0.26.0 are not affected. This includes > users who used Certbot on Bionic after 0.27.0-1~ubuntu18.04.1 > (published 2019-10-29) and users who first used Certbot on Focal or > above. > > [Test Case] > > TBC > > [Regression Potential] > > Since the endpoint is being changed, users who are controlling > reachable endpoints (such as with egress firewalls or proxies) may not > be able to reach the new endpoint until they have adjusted their > configurations. However as the old endpoint will stop functioning > soon, deliberately making this change appears to be the least worst > option. > > Renewal configuration parsing of the server URL is being modified. > Users with unusual configurations such as those that have different > server URLs defined may find themselves on untested paths. > > Users trying to debug a problem configuration will find it surprising > that a configuration that specifies the LE ACMEv1 endpoint goes to the > LE ACMEv2 endpoint instead. However, again this seems to be the least > worst option. > > [Further Details] > > Let’s Encrypt is in the process of shutting down ACMEv1. The full > shutdown process will be completed in June 2021 with temporary brown- > outs starting at the beginning of the year; more specific details are > available at https://community.letsencrypt.org/t/end-of-life-plan-for- > acmev1/88430. > > When ACMEv1 is shut down, many older versions of Certbot will be > unable to get new certificates. ACMEv2 support was first made default > in 0.26.0 for new certificates, but it wasn’t until 1.6.0 that > certificates which had originally been issued using ACMEv1 were > transitioned to ACMEv2. The original update was supposed to move > people off of ACMEv1, but due to some old configuration management > code, we missed a small group of early Certbot users. > > Based on recent counts, there are a total of 23,847 distinct non-EOL > Ubuntu users still using ACMEv1 who use the version of Certbot > packaged in their system’s package manager (the versions available in > 16.04 universe, 16.04 universe updates, 18.04 universe, 18.04 universe > updates, and 20.04). These users will no longer receive certs in June, > but would be automatically upgraded to ACMEv2 if the package for their > system were updated. > > The commit that switches ACMEv1 users to ACMEv2 is here: > https://github.com/certbot/certbot/commit/340a4280eacc3eac8915996d89ff0c0a0cd023f9 > One option to address the upcoming shutdown is to backport the commit into > older versions of Certbot. > > Another option to address the shutdown, which is preferable from our > perspective, would be to update Certbot to 1.6.0+. First, there’s the > inherent risk in backporting an individual change, especially onto > much older code. Released versions are tested extensively both on our > systems and by our users, so we’re much more sure of their stability > than a backported patch. Additionally, Certbot continues to improve > over
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
We use pebble for testing: https://github.com/letsencrypt/pebble/ I'm actually not personally familiar with it though, as I haven't had the need to set one up and modify it myself, but I understand it's fairly straightforward to get running. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
I'd be surprised if that worked? Boulder has account reuse, but that's for reusing an account created on v1 account with v2, not the converse. My initial test of just putting in the old server URL doesn't seem to be working, anyway. (Note that you have to add a server URL to the top section of the renewal config, since this fix is specifically for the older configs where we saved it there automatically; we don't save it there anymore. The `server` line under `[renewalparams]` is something different.) We certainly relied on testing with an actual, old account when we first implemented this. Alex might have set up some sort of manual local Boulder config for testing, but I'm not sure exactly what he did; he mentions it here: https://github.com/certbot/certbot/issues/7979#issuecomment-635010077. To test with that we'd also probably have to go in and manually change the URLs being switched, since the URL being replaced is hard-coded. ** Bug watch added: github.com/certbot/certbot/issues #7979 https://github.com/certbot/certbot/issues/7979 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Hi Erica, I'm just after some way to verify whether the bug exists or not. Then by verifying that the bug exists without this update, we can test the update by verifying that the bug goes away. We'll do this once the proposed package is built and ready to publish. It doesn't have to be automated - having some manual steps ready to perform would also be fine. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
> It's a little tricky to test for this, because you either need to have an old account lying around (as new ACMEv1 accounts can't be created anymore), or modify a boulder instance and then run that. Ah. I see the issue. If it's not too difficult then instructions on setting up and modifying a boulder instance would be useful. If that seems unreasonably awkward, then we can be pragmatic and just do the best we can. Perhaps we could manually modify a configuration to the old endpoint and check that it (locally) redirects the configuration to the new endpoint, for example. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Ah right, it was https://github.com/mnordhoff who had one; evidence of it working is in the PR discussion here: https://github.com/certbot/certbot/pull/8053#issuecomment-639248307 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Robie, are you looking for a live/integration test? There's a unit test in the linked commit. It's a little tricky to test for this, because you either need to have an old account lying around (as new ACMEv1 accounts can't be created anymore), or modify a boulder instance and then run that. I think we found someone with an old account to test it at the time, but I'd have to look into it to see if we still have one. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
> Brad, I'd appreciate your review wrt. your comment in https://bugs.launchpad.net/ubuntu/+source/python-certbot- nginx/+bug/1875471/comments/8 please. AIUI, I'm not breaking python- certbot-apache itself, just its tests, right? In other words, with my proposed fix I'm just moving a test failure from python-certbot to python-certbot-apache? I think it's even better than that. The change you made to Certbot's tests should have no effect on any other package. The tests for python- certbot-apache should still be broken as described at https://bugs.launchpad.net/ubuntu/+source/python-certbot- apache/+bug/1876934, but this change shouldn't add any new failures there. The concern about this change that I tried to express at https://bugs.launchpad.net/ubuntu/+source/python-certbot- nginx/+bug/1875471/comments/8 is we're deleting unit tests for the class certbot.plugins.common.TLSSNI01 while keeping it around and used in python-certbot-apache including for users at runtime. The test file you modified was previously completely broken though and you fixed it allowing the other tests there to run so in general I think it's an improvement. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Test PPA builds available at ppa:racb/experimental (https://launchpad.net/~racb/+archive/ubuntu/experimental) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
I've prepared backports of the configuration parsing adjustment upstream commit that interprets the ACMEv1 LE endpoint as the ACMEv2 LE endpoint instead. For Focal, I also removed/altered the test that expects challenges.TLSSNI01 to exist, again cherry-picked from upstream, to fix the FTBFS (bug 1876933). This might impact python-certbot-apache's test though (https://bugs.launchpad.net/ubuntu/+source/python-certbot- nginx/+bug/1875471/comments/8). Erica, please could you help with a test case that reproduces the problem and can verify the fix? We could for example install python- certbot from Bionic as released (0.23.0-1), but how would we then test where the renewal attempt endpoint goes before and after applying this fix? Brad, I'd appreciate your review wrt. your comment in https://bugs.launchpad.net/ubuntu/+source/python-certbot- nginx/+bug/1875471/comments/8 please. AIUI, I'm not breaking python- certbot-apache itself, just its tests, right? In other words, with my proposed fix I'm just moving a test failure from python-certbot to python-certbot-apache? If so I can look into fixing the python-certbot- apache tests too, though that'll only really help if we need to update that package in Focal in future, rather than impacting users today. ** Changed in: python-certbot (Ubuntu Bionic) Status: Triaged => In Progress ** Changed in: python-certbot (Ubuntu Focal) Status: Triaged => In Progress ** Changed in: python-certbot (Ubuntu Bionic) Assignee: (unassigned) => Robie Basak (racb) ** Changed in: python-certbot (Ubuntu Focal) Assignee: (unassigned) => Robie Basak (racb) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
** Merge proposal linked: https://code.launchpad.net/~racb/ubuntu/+source/python-certbot/+git/python-certbot/+merge/393039 ** Merge proposal linked: https://code.launchpad.net/~racb/ubuntu/+source/python-certbot/+git/python-certbot/+merge/393040 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Groovy released with 1.7.0-1, so Groovy and Hirsute are fixed. ** Changed in: python-certbot (Ubuntu) Status: Triaged => Fix Released ** Description changed: + [Impact] + + Certbot users who first used < 0.26.0 have their configurations locked + into using ACMEv1. This is a deprecated protocol. Let's Encrypt + brownouts for ACMEv1 are scheduled to begin at the beginning of 2021, + and Let's Encrypt will stop serving ACMEv1 in June 2021. + + Based on Let's Encrypt's metrics, 23,847 users were counted as being + locked into ACMEv1 in this way. These users will start receiving + certification renewal failures unless they are patched. + + Users affected are users who first used Certbot on Xenial or first used + Certbot on the release pocket version of Certbot in Bionic. + + Users who first used Certbot >= 0.26.0 are not affected. This includes + users who used Certbot on Bionic after 0.27.0-1~ubuntu18.04.1 (published + 2019-10-29) and users who first used Certbot on Focal or above. + + [Test Case] + + TBC + + [Regression Potential] + + Since the endpoint is being changed, users who are controlling reachable + endpoints (such as with egress firewalls or proxies) may not be able to + reach the new endpoint until they have adjusted their configurations. + However as the old endpoint will stop functioning soon, deliberately + making this change appears to be the least worst option. + + Renewal configuration parsing of the server URL is being modified. Users + with unusual configurations such as those that have different server + URLs defined may find themselves on untested paths. + + Users trying to debug a problem configuration will find it surprising + that a configuration that specifies the LE ACMEv1 endpoint goes to the + LE ACMEv2 endpoint instead. However, again this seems to be the least + worst option. + + [Further Details] + Let’s Encrypt is in the process of shutting down ACMEv1. The full shutdown process will be completed in June 2021 with temporary brown- outs starting at the beginning of the year; more specific details are available at https://community.letsencrypt.org/t/end-of-life-plan-for- acmev1/88430. When ACMEv1 is shut down, many older versions of Certbot will be unable to get new certificates. ACMEv2 support was first made default in 0.26.0 for new certificates, but it wasn’t until 1.6.0 that certificates which had originally been issued using ACMEv1 were transitioned to ACMEv2. The original update was supposed to move people off of ACMEv1, but due to some old configuration management code, we missed a small group of early Certbot users. Based on recent counts, there are a total of 23,847 distinct non-EOL Ubuntu users still using ACMEv1 who use the version of Certbot packaged in their system’s package manager (the versions available in 16.04 universe, 16.04 universe updates, 18.04 universe, 18.04 universe updates, and 20.04). These users will no longer receive certs in June, but would be automatically upgraded to ACMEv2 if the package for their system were updated. The commit that switches ACMEv1 users to ACMEv2 is here: https://github.com/certbot/certbot/commit/340a4280eacc3eac8915996d89ff0c0a0cd023f9 One option to address the upcoming shutdown is to backport the commit into older versions of Certbot. Another option to address the shutdown, which is preferable from our perspective, would be to update Certbot to 1.6.0+. First, there’s the inherent risk in backporting an individual change, especially onto much older code. Released versions are tested extensively both on our systems and by our users, so we’re much more sure of their stability than a backported patch. Additionally, Certbot continues to improve over time, closing up bugs, supporting more edge cases, improving usability, and offering more robust and modern security practices. Since we made backwards incompatible changes in 0.40.0 and 1.0.0, to update Certbot to a newer version, our other components will have to be updated as well. Certbot relies on our other libraries `acme` and `josepy`, and we have a series of plugins which will need to be updated as well, including the `certbot-nginx` and `certbot-apache` plugins, as well as our `certbot-dns-*` plugins. Certbot 1.0.0 in particular contained significant API changes, and if any of our packages are updated to 1.0.0 or newer, it will probably be easiest to update all of them. josepy may be fine depending on the version of certbot, as certbot 1.0.0 relies on `josepy>=1.1.0`, which is already available packaged on all relevant systems. But Certbot 1.0.0 also requires `acme>=0.40.0`, which is only one release behind 1.0.0, so it would probably be easier to update it to a matching version. Basically, I would recommend choosing a certbot version, then updating `acme`, `certbot-nginx`, `certbot-apache`, and `certbot-dns-*` to that version. None of our 3rd party
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
** Changed in: python-certbot (Ubuntu) Status: New => Triaged ** Also affects: python-certbot (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: python-certbot (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: python-certbot (Ubuntu Bionic) Status: New => Triaged ** Changed in: python-certbot (Ubuntu Focal) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
> Could you confirm which Ubuntu releases require this please? Is it all of 16.04, 18.04 and 20.04? Is the version in Ubuntu Groovy (1.7.0-1 currently, not yet released) affected? Yes, it is all of those, in both universe and universe updates (for 16.04 and 18.04, unless 20.04 now has universe updates that I've missed). It's anything less than 1.6.0. Version 1.7.0 in Groovy already has the patch and should not need to be updated. > Going forwards, I suggest that the policy we adopt in making a decision on whether to update distribution certbot packaging in Ubuntu should be to prefer cherry-picks if they are reasonably simple to achieve, but permit major version updates when cherry-picks aren't practical to solve an "Internet deprecation". Honestly, I think this is hard to predict ahead of time. The ACME protocol, having been officially standardized, is certainly more stable now. Now that Certbot has passed 1.0.0, our API is more stable as well; if we plan to change it, we'll bump the major version number. If we were updating packages that were all past 1.0 to some 1.x, I'd certainly be more inclined to just update the whole package. In this situation, that is obviously not the case, and so just applying a single patch makes sense. But I could see that going the other way in a different situation. Thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1893274] Re: Certbot will stop working for 23, 847 users with upcoming Let's Encrypt deprecation
Hi Erica, Thank you for looking after the certbot packages in Ubuntu and being proactive about managing service deprecations as always. My feeling on this one is to cherry-pick the commit you identified only, if that is all that is required? Could you confirm which Ubuntu releases require this please? Is it all of 16.04, 18.04 and 20.04? Is the version in Ubuntu Groovy (1.7.0-1 currently, not yet released) affected? I appreciate your preference to update our existing releases to 1.6.0+, especially considering your confidence in your own newer releases as opposed to cherry-picking that you have not tested. However I think the other side of the trade-off is in the complexity that you describe in the number of dependencies that would need to be updated or introduced. While we might have confidence that certbot 1.6.0+ when correctly presented with the necessary dependencies will work correctly, there's a risk that packaging will get it wrong and certbot don't correctly get that - for example in specific upgrade paths. This already happened to us in our first attempt to update certbot like this (that we realized and thus didn't release, but that did delay us). Also, distribution package consumers prefer stability in the "doesn't change behaviour" sense. Given that historically we find it difficult to find volunteers to work on this, the triviality of this particular fix, and my points in the previous paragraph, I think we should focus on the cherry-pick. If you could confirm the affected release list please, I (or someone else on my team) can drive the SRU process for this update based on a cherry-pick. Going forwards, I suggest that the policy we adopt in making a decision on whether to update distribution certbot packaging in Ubuntu should be to prefer cherry-picks if they are reasonably simple to achieve, but permit major version updates when cherry-picks aren't practical to solve an "Internet deprecation". Users could then expect distribution certbot packaging to avoid changing behaviour when possible, but still change behaviour where that is required to keep it working. Users who specifically want to upgrade to newer certbot behaviour but remain on an old distribution release now have the option of using the snap. What changed my opinion from before, when we set up the certbot exception, is that the complexity of the necessary changes to certbot needed to keep it working as Let's Encrypt and ACME have changed seem to me to have reduced considerably over time. I think this is a sign of maturity of the project. I think users expect the churn in stable distribution release packages to reduce accordingly. However I appreciate that we might yet need major changes in the future and so I don't rule out using the standing exception again should that become necessary. What do you think? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1893274 Title: Certbot will stop working for 23,847 users with upcoming Let's Encrypt deprecation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-certbot/+bug/1893274/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
