** Description changed:
+ [impact]
+
+ autopkgtests are failing/flaky and prevent other packages from migrating
+ to -updates
+
+ [test case]
+
+ check systemd autopkgtest history
+
+ [regression potential]
+
+ in regard to the changed test cases, any regression would likely result
+ in either an incorrectly passed test, or an incorrectly failed test.
+
+ [scope]
+
+ this is needed for x, b, and f.
+
+ tests in g appear to be mostly stable, but I've opened MR (linked from
+ this bug) to update the tests there as well.
+
+ i don't plan to update x, as it's reaching ESM in ~6 months, and
+ backporting the test fixes is more work than just a simple code copy,
+ since there are additional differences/changes needed in the older
+ version of systemd (and python3). the failing/flaky tests in x have been
+ like that forever, and people have just retried them; we can keep
+ retrying them until x moves into ESM next year.
+
+ [original description]
+
Hi,
we had such cases in the past like bug 1817721 for bionic and maybe bug
1892130 is about the same as well. There were more but I didn't want to search
for all of them - what I checked is that there are no open ones clearly
pointing out the recent further drop in already flaky subtests.
In particular the tests "tests-in-lxd" and "systemd-fsckd" were known to
be flaky before, but got even worse.
Here stats of the last 40 runs, it might be a coincidences that this is
after 246-2ubuntu1 landed. Could as well be any other change
groovy
- amd64
+ amd64
tests-in-lxd (F 42% S 0% B 10% => P 45%/)
FFFFBFFFFFFFB....FF.B.....F.....F...FBF
build-login (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
unit-config (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
networkd-testpy (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
boot-and-services (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
boot-smoke (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
logind (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
storage (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
upstream (F 35% S 0% B 10% => P 52%/)
..FFB.FFF.FFB....FF.B.....F.F..F....FBF
udev (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
systemd-fsckd (F 37% S 0% B 10% => P 50%/)
FFFFBFFFFFFFB.FF...FB.....F..........B.
root-unittests (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
- ppc64el
+ ppc64el
tests-in-lxd (F 25% S 0% B 0% => P 75%/)
FFFF....FF............FFF.....F.........
systemd-fsckd (F 35% S 0% B 0% => P 65%/)
FFFFFFF...FF........F....FF.F..F........
root-unittests (F 2% S 0% B 0% => P 97%/)
..............................F.........
- s390x
+ s390x
tests-in-lxd (F 52% S 0% B 0% => P 47%/)
FFFFFFF.FFFFFFF.FF.........FFFF...F.....
timedated (F 2% S 0% B 0% => P 97%/)
...........F............................
upstream (F 17% S 0% B 0% => P 82%/)
.....F......F.F.............FFF...F.....
systemd-fsckd (F 32% S 0% B 0% => P 67%/)
FFFFFFF..FF..F.................FF..F....
root-unittests (F 10% S 0% B 0% => P 90%/)
............................FFF...F.....
- arm64
+ arm64
tests-in-lxd (F 40% S 0% B 2% => P 57%/)
FFFFF.B...FFF.FF..F..F.........FFF.F....
logind (F 2% S 0% B 2% => P 95%/)
......B...................F.............
upstream (F 22% S 0% B 2% => P 75%/)
...F.FB.....F.F.............F..FFF.F....
root-unittests (F 12% S 0% B 2% => P 85%/)
......B.F...........F.F........F...F....
(I'm sure LP will make this unreadable, but is is nice in monospace)
Whatever the root cause is - the success rate of these has reduced so
much that the (even formerly questionable) practice of retry-until-
success won't work anymore.
-
- I have run the two tests in a local VM and systemd-fsckd works there while
tests-in-lxd seems to trip over the old flaky fellow being "boot-and-services".
+ I have run the two tests in a local VM and systemd-fsckd works there
+ while tests-in-lxd seems to trip over the old flaky fellow being "boot-
+ and-services".
We had the discussion in the past, but I think I need to again bring up
the suggestion to skip "tests-in-lxd" and "systemd-fsckd" until they are
on reasonable success rates.
** Changed in: systemd (Ubuntu Bionic)
Importance: Undecided => Medium
** Changed in: systemd (Ubuntu Bionic)
Status: New => In Progress
** Changed in: systemd (Ubuntu Bionic)
Assignee: (unassigned) => Dan Streetman (ddstreet)
** Changed in: systemd (Ubuntu Focal)
Importance: Undecided => Medium
** Changed in: systemd (Ubuntu Focal)
Status: Confirmed => In Progress
** Changed in: systemd (Ubuntu Focal)
Assignee: (unassigned) => Dan Streetman (ddstreet)
** Description changed:
[impact]
autopkgtests are failing/flaky and prevent other packages from migrating
to -updates
[test case]
- check systemd autopkgtest history
+ check autopkgtest history
[regression potential]
in regard to the changed test cases, any regression would likely result
in either an incorrectly passed test, or an incorrectly failed test.
[scope]
- this is needed for x, b, and f.
+ for systemd, this is needed for x, b, and f.
tests in g appear to be mostly stable, but I've opened MR (linked from
this bug) to update the tests there as well.
i don't plan to update x, as it's reaching ESM in ~6 months, and
backporting the test fixes is more work than just a simple code copy,
since there are additional differences/changes needed in the older
version of systemd (and python3). the failing/flaky tests in x have been
like that forever, and people have just retried them; we can keep
retrying them until x moves into ESM next year.
[original description]
Hi,
we had such cases in the past like bug 1817721 for bionic and maybe bug
1892130 is about the same as well. There were more but I didn't want to search
for all of them - what I checked is that there are no open ones clearly
pointing out the recent further drop in already flaky subtests.
In particular the tests "tests-in-lxd" and "systemd-fsckd" were known to
be flaky before, but got even worse.
Here stats of the last 40 runs, it might be a coincidences that this is
after 246-2ubuntu1 landed. Could as well be any other change
groovy
amd64
tests-in-lxd (F 42% S 0% B 10% => P 45%/)
FFFFBFFFFFFFB....FF.B.....F.....F...FBF
build-login (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
unit-config (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
networkd-testpy (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
boot-and-services (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
boot-smoke (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
logind (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
storage (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
upstream (F 35% S 0% B 10% => P 52%/)
..FFB.FFF.FFB....FF.B.....F.F..F....FBF
udev (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
systemd-fsckd (F 37% S 0% B 10% => P 50%/)
FFFFBFFFFFFFB.FF...FB.....F..........B.
root-unittests (F 0% S 0% B 10% => P 87%/)
....B.......B.......B................B.
ppc64el
tests-in-lxd (F 25% S 0% B 0% => P 75%/)
FFFF....FF............FFF.....F.........
systemd-fsckd (F 35% S 0% B 0% => P 65%/)
FFFFFFF...FF........F....FF.F..F........
root-unittests (F 2% S 0% B 0% => P 97%/)
..............................F.........
s390x
tests-in-lxd (F 52% S 0% B 0% => P 47%/)
FFFFFFF.FFFFFFF.FF.........FFFF...F.....
timedated (F 2% S 0% B 0% => P 97%/)
...........F............................
upstream (F 17% S 0% B 0% => P 82%/)
.....F......F.F.............FFF...F.....
systemd-fsckd (F 32% S 0% B 0% => P 67%/)
FFFFFFF..FF..F.................FF..F....
root-unittests (F 10% S 0% B 0% => P 90%/)
............................FFF...F.....
arm64
tests-in-lxd (F 40% S 0% B 2% => P 57%/)
FFFFF.B...FFF.FF..F..F.........FFF.F....
logind (F 2% S 0% B 2% => P 95%/)
......B...................F.............
upstream (F 22% S 0% B 2% => P 75%/)
...F.FB.....F.F.............F..FFF.F....
root-unittests (F 12% S 0% B 2% => P 85%/)
......B.F...........F.F........F...F....
(I'm sure LP will make this unreadable, but is is nice in monospace)
Whatever the root cause is - the success rate of these has reduced so
much that the (even formerly questionable) practice of retry-until-
success won't work anymore.
I have run the two tests in a local VM and systemd-fsckd works there
while tests-in-lxd seems to trip over the old flaky fellow being "boot-
and-services".
We had the discussion in the past, but I think I need to again bring up
the suggestion to skip "tests-in-lxd" and "systemd-fsckd" until they are
on reasonable success rates.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1892358
Title:
autopkgtest success rate dropped inhibiting proposed migration
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/build-essential/+bug/1892358/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
