[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Launchpad Bug Tracker]

This bug was fixed in the package autopkgtest - 5.0

---
autopkgtest (5.0) unstable; urgency=medium

  [ Martin Pitt ]
  * Drop obsolete adt-* CLI
autopkgtest 4.0 with its "autopkgtest" program has been around for a
year, Debian's and Ubuntu's CI moved to that, and the Debian 9.0 release
is behind us. Drop "adt-run" and the corresponding adt-virt-* and
adt-build-* command line interfaces now.
Add Breaks: for debci for versions that still relied on adt-*.
  * lxc, qemu, ssh: Shorten path of shared directory.
This helps to reduce socket path lengths in tests.
(See Debian #860554, LP: #1680577)
  * Drop obsolete ubuntu-touch-session setup script.
This hasn't been maintained/tested for ages, and with the demise of
Ubuntu Touch this is now completely obsolete. (Closes: #835836)
  * Avoid bare "except:" clauses
  * tests/autopkgtest_args: pyflakes/pycodestyle fixes
  * tests: Don't run LXC tests with --ephemeral.
It's not the default and ephemeral containers tend to fail to start too
often.
  * tests/testpkg: Update Standards-Version and debhelper level.
Priority "extra" is deprecated, so move to "optional".
  * README.package-tests.rst: Clarify role of control file, add simple
examples (Closes: #870654)
  * README.package-tests.rst: Update Testsuite: documentation.
All supported Debian releases now have a new enough dpkg, and Ubuntu
14.04 will not get new tests any more. (Closes: #876008)
  * Run apt-ftparchive for local debs on testbed instead of host.
This is slightly more correct as it will produce the format that the
testbed's apt expects (although that doesn't matter much). It also
allows autopkgtest to be run on non-Debian hosts.
As apt-utils is not essential, install it if necessary.
  * README.package-tests.rst: Clarify Testsuite: values
`autopkgtest` is not any more the only defined value for Testsuite:, as
the paragraph below already explains.
  * setup-commands/setup-testbed: Look at /etc/os-release for determining
distribution name. Stop hardcoding "debian|ubuntu|kali" and get this
information from os-release's ID field. (Closes: #868678)
  * debian/rules: Drop obsolete pysupport override.
Not necessary any more since dh compat level 9.
  * Bump Standards-Version to 4.1.0. (No changes necessary.)

  [ Stéphane Graber ]
  * debian/tests/lxd: Update to support LXD network/storage API

  [ Steve Langasek ]
  * Fix release name parsing with arch-qualified sources.list
(Closes: #870393)

  [ Antonio Terceiro ]
  * autopkgtest-build-lxc: Correctly clean cache for Debian containers

 -- Martin Pitt   Sun, 17 Sep 2017 16:50:21 +0200

** Changed in: autopkgtest (Ubuntu)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Iain Lane]

Nope, thank you - that's all done now AFAICS. Thanks for the reviews! :)

** Changed in: auto-package-testing
   Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Ubuntu Foundations Team Bug Bot]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

Thanks! I pushed your patch to the autopkgtest git. Not sure if there is
still an infra change to do (for the "Auto Package Testing" task).

** Also affects: autopkgtest (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: autopkgtest (Ubuntu)
   Status: New => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

This is harder to work around/catch, as in the new case the test does
*not* time out, it just kills sshd (or something in the kernel that
breaks ssh/networking). In general these are cases that we do want to
treat as "tmpfail" and auto-restart, I don't want to treat an auxverb
failure as failure in general.

Perhaps we need to introduce some kind of retry counter, but this would
need to span at least half a day -- three tmpfails on the same worker in
a row are usually a sign of a broken cloud or a broken testbed image,
not a test failure. So perhaps some logic to check if other tests
tmpfail on the same worker/cloud, and if not then call that test a
failure.

This would all require state keeping, which we don't currently do (the
only state is the AMQP queue contents).

** Changed in: autopkgtest (Ubuntu)
   Status: In Progress => Triaged

** Changed in: autopkgtest (Ubuntu)
   Importance: High => Medium

** Package changed: autopkgtest (Ubuntu) => auto-package-testing

** Changed in: auto-package-testing
Milestone: ubuntu-16.10 => None

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

On the infra it now looks like this:


ERROR - 02:45:43 DEBUG| [stdout]
ERROR - 03:19:00 ERROR| [stderr] Run parser regression tests ... ok
ERROR - 03:19:01 ERROR| [stderr] test_regression_testsuite 
(__main__.ApparmorTestsuites)
ERROR - 03:19:05 DEBUG| [stdout]   preparing apparmor_2.10.95-4ubuntu5.dsc... 
done
ERROR - 03:19:05 DEBUG| [stdout]
ERROR - Killed
ERROR - 
ERROR - Due to the above error(s), this script is unable to continue and is 
terminating.
ERROR - 
Connection to 10.42.46.237 closed by remote host.
autopkgtest-virt-ssh [05:07:27]: ERROR: Removing temporary files on testbed 
timed out
autopkgtest [05:07:36]: ERROR: testbed failure: testbed auxverb failed with 
exit code 255

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

It still seems to happen after that patch.

** Changed in: autopkgtest (Ubuntu)
   Status: Fix Committed => In Progress

** Changed in: autopkgtest (Ubuntu)
 Assignee: Martin Pitt (pitti) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

Fix and test cases:
https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=4a0587a05

Rolled out to production.

** Changed in: autopkgtest (Ubuntu)
   Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

* Create a container, install ssh and your ssh key:

  lxc launch autopkgtest/ubuntu/yakkety/amd64 y1
  lxc exec y1 -- apt install -y openssh-server
  lxc file push -p --uid=1000 --gid=1000 ~/.ssh/id_rsa.pub 
y1/home/ubuntu/.ssh/authorized_keys

 * Get the IP from "lxc list" and verify that you can ssh in without a
password.

 * Create this test:

   mkdir -p /tmp/t/debian/tests/
   echo 'Test-Command: pkill -e -STOP sshd; sleep 5' > 
/tmp/t/debian/tests/control

 * Run it:

   lxc exec y1 -- pkill -e -CONT sshd
   autopkgtest --timeout-test=1 /tmp/t -- ssh -l ubuntu -H $IP

   ($IP is the container's IP)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1630578/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs