[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
the problem only recurred after a reboot and it was promptly fixed by installing the package from -proposed. verification for noble done ** Tags removed: verification-needed verification-needed-noble ** Tags added: verification-done verification-done-noble -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Took me 20min to trigger the retry for 5 tests. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I retried the failed autopkgtests, as they failed due to a network problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I am currently travelling but otherwise happy to help with testing -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I was expecting the new version to be 3.7.4-1ubuntu5.1 instead of 3.7.4-1ubuntu5.24.04.1, but the former one works. ** Changed in: apt-cacher-ng (Ubuntu Noble) Status: Confirmed => Fix Committed ** Tags added: verification-needed verification-needed-noble -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Actually, I reproduced the problem in a noble armhf container after a few seconds of usage. ** Changed in: apt-cacher-ng (Ubuntu Noble) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Hi Rolf Leggewie (r0lf), you seem to be running a patched version of apt-cacher-ng with the fix. Would you be able to, if this SRU is accepted: a) restore to the "buggy" version, confirm the 100% CPU usage; b) update to apt-cacher-ng which will be in noble-proposed, and confirm the problem no longer happens, after a "reasonable amount of time"? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Coming back to this in another patch pilot shift, I'm sorely tempted to
mark this "incomplete" until someone can come up with a reliable way to
actually replicate the issue. However ... at the same time, the fix is
trivial and quite clearly correcting something wrong (long != 64-bits on
armhf, or any 32-bit arch I'm aware of for that matter).
Maybe it's sufficient this fixes things for the original reporter (or
Armbian users more generally)? I'm going to sponsor this, and add a
caveat to the test plan that this can't be directly replicated on
Ubuntu, so testing should be done there to avoid regression more
generally, but testing of the fix will need to be done on affected
platforms.
** Description changed:
[IMPACT]
The version of acng in noble results in 100% load on a single core on 32
bit platforms like armhf. The reason is explained at the Debian BTS and
further below at "ORIGINAL REPORT".
[TEST CASE]
+ On affected platforms (Armbian?):
+
Install noble on a 32-bit platform. armhf lends itself nicely as being
widely available in the form of older Raspberries and clones. Install
the unpatched version of apt-cacher-ng. Start the apt-cacher-ng service
and wait for a few minutes and you should see one core being blocked at
100%. Then compile the fix as provided in the debdiff and install. The
problem should go away.
+
+ On Ubuntu (which doesn't appear affected, but to guard against
+ regression):
+
+ * To be repeated for both 32-bit and 64-bit archs, e.g. armhf and arm64
+ * Install noble on boot media (e.g. SD card)
+ * Boot and enable proposed (https://wiki.ubuntu.com/Testing/EnableProposed)
+ * sudo apt install -t noble-proposed apt-cacher-ng
+ * sudo systemctl start apt-cacher-ng
+ * echo 'Acquire::http { Proxy "http://localhost:3142";; }' >
/etc/apt/apt.conf.d/00proxy
+ * sudo apt update
+ * sudo apt upgrade
+ * Ensure update and upgrade proceed normally and no cores are pegged afterward
[REGRESSION POTENTIAL]
the patch was taken from upstream and backported to noble, but I can't
comment further on potential regressions
[ORIGINAL REPORT]
Helge Deller reported to the Debian BTS a problem affecting certain
platforms including armhf that lead to acng consuming 100% on one core.
This was fixed already in the latest upstream release 3.7.5 which has
already landed in o-series and p-series. I kindly ask for an SRU to at
least noble. Jammy is most likely not affected.
From the original bug report:
"A few weeks back Debian moved to using a 64-bit time_t type
even on 32-bit platforms. Since then I noticed that apt-cacher-ng
will at some point starting to spin with using 100% CPU load, and
stop doing any further apt-cacher-ng functionality like downloads.
Today I found time to debug the issue, and the attached patch fixes
it for me on the hppa platform. I'm sure other 32-bit arches like
armhf and armel suffer from the same problem.
The problem is, that during cleanup phase apt-cacher-ng will wait like this:
- wait_for(g, std::min(long(delta), long(84600)), 1);
Above you can see that delta and 84600 will be converted to "long" type.
A "long" type on 32-bit platforms is limited to 32-bit. On 64-bit
platforms this is 64-bit, which is why 64-bit arches are not affected.
So, after we now use 64-bit time_t values on 32-bit, the 64-bit values
gets truncated to 32-bit and will wait too long if "delta" is a huge negative
value.
This is how to fix it:
+ wait_for(g, std::min((time_t)(delta), (time_t)(84600)),
1);
[UBUNTU+1]
This affects ports.ubuntu.com for example for armbian noble and is
already fixed in o-series and p-series.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2100887
Title:
SRU: time_t overflow on certain architectures (100% CPU)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I think to trigger the issue, you need to arrive to the rounding error in
delta, which happens if every cleanup task returned END_OF_TIME and we subtract
now timestamp from it:
---
#include
#include
#define MAX_VAL(x) (std::numeric_limits< x >::max())
static const time_t END_OF_TIME(MAX_VAL(time_t)-2);
int main() {
time_t now = time(0);
time_t diff = END_OF_TIME - now;
std::cout << (diff) << " " << long(diff) << std::endl;
}
---
$ ./a.out
9223372035111660800 -174311500
I am not sure how to reproduce it reliably with the service itself,
maybe it is possible to provide a unit test with gtest/gmock or an
integration test.
Also it seems that it should be sufficient to define time_t DAY = 84600;
and use it in
wait_for(g, std::min(delta, DAY), 1);
without other changes, as the variables are already time_t and the
invalid conversion happens inside this line.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2100887
Title:
SRU: time_t overflow on certain architectures (100% CPU)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I'm attempting to reproduce this, just to verify this SRU will actually
fix something but so far I'm failing. Attempts so far:
* i386 is out since there hasn't been an apt-cacher-ng i386 build since
bionic, so the only possible architecture this could affect is armhf
* Testing with armhf on noble is tricky as we stopped producing a raspi
armhf kernel so there is no noble armhf image for raspi, and upgrading
from jammy doesn't work
* Tested with a Pi 5 8GB running noble (arm64), with a noble (armhf) LXD
container as follows:
$ lxc launch ubuntu:n/armhf acngtest
$ lxc shell acngtest
# apt update && apt install apt-cacher-ng
Waited several minutes, but no excessive usage shown in top. Just in
case actual usage of the proxy matters, attempted the following:
# echo 'Acquire::http { Proxy "http://localhost:3142";; }' >
/etc/apt/apt.conf.d/00proxy
# apt update
# apt -y full-upgrade
This pulled in several things, but still no spike in CPU usage after
waiting several minutes. I then left it running all night and still
nothing. Am I missing something to replicate this issue?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2100887
Title:
SRU: time_t overflow on certain architectures (100% CPU)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Saw this reported on the sru matrix channel; I suspect this wants an ubuntu-sponsors subscription so it appears on the sponsoring report. I'll add that and try and get to this in my current patch pilot shift. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
ftr, this is in oracular and up ** Changed in: apt-cacher-ng (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I hope this qualifies as "Bugs which do not fit under above categories, but (1) have an obviously safe patch" ** Description changed: Helge Deller reported to the Debian BTS a problem affecting certain platforms including armhf that lead to acng consuming 100% on one core. This was fixed already in the latest upstream release 3.7.5 which has already landed in o-series and p-series. I kindly ask for an SRU to at least noble. Jammy is most likely not affected. From the original bug report: "A few weeks back Debian moved to using a 64-bit time_t type even on 32-bit platforms. Since then I noticed that apt-cacher-ng will at some point starting to spin with using 100% CPU load, and stop doing any further apt-cacher-ng functionality like downloads. Today I found time to debug the issue, and the attached patch fixes it for me on the hppa platform. I'm sure other 32-bit arches like armhf and armel suffer from the same problem. The problem is, that during cleanup phase apt-cacher-ng will wait like this: - wait_for(g, std::min(long(delta), long(84600)), 1); Above you can see that delta and 84600 will be converted to "long" type. A "long" type on 32-bit platforms is limited to 32-bit. On 64-bit platforms this is 64-bit, which is why 64-bit arches are not affected. So, after we now use 64-bit time_t values on 32-bit, the 64-bit values gets truncated to 32-bit and will wait too long if "delta" is a huge negative value. This is how to fix it: + wait_for(g, std::min((time_t)(delta), (time_t)(84600)), 1); Can you please apply the patch to the next upload of apt-cacher-ng ?" This affects ports.ubuntu.com for example for armbian noble and is already fixed in o-series and p-series. + + + SRU TEMPLATE + + + [IMPACT] + + The version of acng in noble results in 100% load on a single core on 32 + bit platforms like armhf. The reason is explained above. + + [TEST CASE] + + Install noble on a 32-bit platform. armhf lends itself nicely as being + widely available in the form of older Raspberries and clones. Install + the unpatched version of apt-cacher-ng. Wait for a few minutes and you + should see one core being blocked at 100%. Then compile the fix as + provided in the debdiff and install. The problem should go away. + + [REGRESSION POTENTIAL] + + the patch was taken from upstream and backported to noble, but I can't + comment further on potential regressions ** Description changed: + [IMPACT] + + The version of acng in noble results in 100% load on a single core on 32 + bit platforms like armhf. The reason is explained at the Debian BTS and + further below at "ORIGINAL REPORT". + + [TEST CASE] + + Install noble on a 32-bit platform. armhf lends itself nicely as being + widely available in the form of older Raspberries and clones. Install + the unpatched version of apt-cacher-ng. Start the apt-cacher-ng service + and wait for a few minutes and you should see one core being blocked at + 100%. Then compile the fix as provided in the debdiff and install. The + problem should go away. + + [REGRESSION POTENTIAL] + + the patch was taken from upstream and backported to noble, but I can't + comment further on potential regressions + + [ORIGINAL REPORT] + Helge Deller reported to the Debian BTS a problem affecting certain platforms including armhf that lead to acng consuming 100% on one core. This was fixed already in the latest upstream release 3.7.5 which has already landed in o-series and p-series. I kindly ask for an SRU to at least noble. Jammy is most likely not affected. From the original bug report: "A few weeks back Debian moved to using a 64-bit time_t type even on 32-bit platforms. Since then I noticed that apt-cacher-ng will at some point starting to spin with using 100% CPU load, and stop doing any further apt-cacher-ng functionality like downloads. Today I found time to debug the issue, and the attached patch fixes it for me on the hppa platform. I'm sure other 32-bit arches like armhf and armel suffer from the same problem. The problem is, that during cleanup phase apt-cacher-ng will wait like this: - wait_for(g, std::min(long(delta), long(84600)), 1); Above you can see that delta and 84600 will be converted to "long" type. A "long" type on 32-bit platforms is limited to 32-bit. On 64-bit platforms this is 64-bit, which is why 64-bit arches are not affected. So, after we now use 64-bit time_t values on 32-bit, the 64-bit values gets truncated to 32-bit and will wait too long if "delta" is a huge negative value. This is how to fix it: + wait_for(g, std::min((time_t)(delta), (time_t)(84600)), 1); Can you please apply the patch to the next upload of apt-cacher-ng ?" + [UBUNTU+1] + This affects ports.ubuntu.com for example for armbian noble and is already fixed in o-series and p-series. -
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
Has been working fine for me for the last 10 days or so, no more 100% CPU -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
** Also affects: apt-cacher-ng (Ubuntu Noble) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
I've done a number of SRUs in the past and would have thought this bug would certainly qualify. Not sure if the SRU criteria have changed in the last few years but when I read https://documentation.ubuntu.com/sru/en/latest/reference/requirements/#reference- what-is-acceptable-to-sru the bar seems to be much higher than what I used to remember. While 100% CPU is certainly very, very unfortunate, I am not sure it qualifies as "severe regression". Asking for guidance before I prepare the SRU template. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
** Patch added: "debdiff for noble" https://bugs.launchpad.net/debian/+source/apt-cacher-ng/+bug/2100887/+attachment/5862440/+files/LP2100887.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
** Changed in: apt-cacher-ng (Debian) Status: Unknown => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
upstream fix: https://salsa.debian.org/blade/apt-cacher- ng/-/commit/99d7ab63fc812344105ecea2c6ad4a74aee518bd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2100887] Re: SRU: time_t overflow on certain architectures (100% CPU)
** Description changed: Helge Deller reported to the Debian BTS a problem affecting certain platforms including armhf that lead to acng consuming 100% on one core. - This was fixed already in the latest upstream release which has already - landed in o-series and p-series. I kindly ask for an SRU to at least - noble. Jammy is most likely not affected. + This was fixed already in the latest upstream release 3.7.5 which has + already landed in o-series and p-series. I kindly ask for an SRU to at + least noble. Jammy is most likely not affected. From the original bug report: "A few weeks back Debian moved to using a 64-bit time_t type even on 32-bit platforms. Since then I noticed that apt-cacher-ng will at some point starting to spin with using 100% CPU load, and stop doing any further apt-cacher-ng functionality like downloads. Today I found time to debug the issue, and the attached patch fixes it for me on the hppa platform. I'm sure other 32-bit arches like armhf and armel suffer from the same problem. The problem is, that during cleanup phase apt-cacher-ng will wait like this: - wait_for(g, std::min(long(delta), long(84600)), 1); Above you can see that delta and 84600 will be converted to "long" type. A "long" type on 32-bit platforms is limited to 32-bit. On 64-bit platforms this is 64-bit, which is why 64-bit arches are not affected. So, after we now use 64-bit time_t values on 32-bit, the 64-bit values gets truncated to 32-bit and will wait too long if "delta" is a huge negative value. This is how to fix it: + wait_for(g, std::min((time_t)(delta), (time_t)(84600)), 1); Can you please apply the patch to the next upload of apt-cacher-ng ?" This affects ports.ubuntu.com for example for armbian noble and is already fixed in o-series and p-series. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100887 Title: SRU: time_t overflow on certain architectures (100% CPU) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/2100887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
