Package: release.debian.org Severity: normal User: release.debian....@packages.debian.org Usertags: unblock
Hello release team, Please consider autopkgtest 4.4 [1] for Jessie. It contains mostly bug fixes and one small new feature which doesn't affect Debian. IMHO it's low risk as (1) it's not a package that users usually have installed, it's a developer CI/QA tool, and (2) it has quite an extensive testsuite (including an autopkgtest that autopkgtests itself ☺). I attach the complete debdiff and do some changelog risk annotation here. | autopkgtest (4.4) unstable; urgency=medium | | [ Martin Pitt ] | * doc/README.package-tests.rst: Document network access (Closes: #851556) Documentation only, no risk. | * qemu: Robustify ssh port locks. | Stop assuming that /run/lock is user-writable (it is not in non-Debian | systems). Instead create the lock file in /tmp and use 'x' to avoid | /tmp file races. Does affect Debian as it changes existing behaviour, but I've tested that quite extensively. Note that we don't use the QEMU runner in production on ci.d.n. | * tests/autopkgtest: Fix crashes when running on non-apt system No runtime impact, self-tests only. | * Fix candidate version detection for packages containing regexp operators | '+' and '.' are valid characters in a Debian package name. Escape them | in the call to apt-cache policy so that we get what we want to know. | (Closes: #855954) This is an important fix that I'd like to get into stretch to fix testing packages with particular names when using apt pinning. | [ Iain Lane ] | * Fix build_source to work if "Package-List" is the last line in the apt | output (Closes: #851899) Corner case, but low risk fix and covered by several tests. | * autopkgtest-virt-lxd: Check uptime for reboot waiting. (LP: #1654025) lxd is not in Debian, thus very little risk. This change has been in Ubuntu's production CI for several months now. | * Add a debug-fail hook and implement it for autopkgtest-virt-ssh. | At the minute, this is mainly so that the nova script can have its | failure information (`nova console-log') propagated up to the output, so | that in the case of kernel panics or other random failures we get useful | output that the driver of autopkgtest (e.g. autopkgtest-cloud) can look | at. (LP: #1630578) Somewhat intrusive, but again this change has been in Ubuntu's production CI for several months now. We don't (yet) use this Openstack cloud instance testing mode in Debian's CI. | * autopkgtest-build-lxd: Allow overriding the target release by setting | RELEASE=. This will cause the container to be dist-upgraded to the new | release. Useful for the very early stages of a release when the LXD | images on images.linuxcontainers.org don't exist yet. lxd is not in Debian, thus practically no risk. | [ Barry Warsaw ] | * Pass Dpkg::Options::=--force-confnew to apt-get install. | This avoids dpkg from prompting for conffile installation when you always | want the new conffile in the testbed anyway. (Closes: #852475) This is another important bug fix which screws up some tests completely. Thanks, Pitti unblock autopkgtest/4.4 [1] https://tracker.debian.org/news/844681
diff --git a/debian/changelog b/debian/changelog
index 03bc2c0..7435939 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,39 @@
+autopkgtest (4.4) unstable; urgency=medium
+
+ [ Martin Pitt ]
+ * doc/README.package-tests.rst: Document network access (Closes: #851556)
+ * qemu: Robustify ssh port locks.
+ Stop assuming that /run/lock is user-writable (it is not in non-Debian
+ systems). Instead create the lock file in /tmp and use 'x' to avoid
+ /tmp file races.
+ * tests/autopkgtest: Fix crashes when running on non-apt system
+ * Fix candidate version detection for packages containing regexp operators
+ '+' and '.' are valid characters in a Debian package name. Escape them
+ in the call to apt-cache policy so that we get what we want to know.
+ (Closes: #855954)
+
+ [ Iain Lane ]
+ * Fix build_source to work if "Package-List" is the last line in the apt
+ output (Closes: #851899)
+ * autopkgtest-virt-lxd: Check uptime for reboot waiting. (LP: #1654025)
+ * Add a debug-fail hook and implement it for autopkgtest-virt-ssh.
+ At the minute, this is mainly so that the nova script can have its
+ failure information (`nova console-log') propagated up to the output, so
+ that in the case of kernel panics or other random failures we get useful
+ output that the driver of autopkgtest (e.g. autopkgtest-cloud) can look
+ at. (LP: #1630578)
+ * autopkgtest-build-lxd: Allow overriding the target release by setting
+ RELEASE=. This will cause the container to be dist-upgraded to the new
+ release. Useful for the very early stages of a release when the LXD
+ images on images.linuxcontainers.org don't exist yet.
+
+ [ Barry Warsaw ]
+ * Pass Dpkg::Options::=--force-confnew to apt-get install.
+ This avoids dpkg from prompting for conffile installation when you always
+ want the new conffile in the testbed anyway. (Closes: #852475)
+
+ -- Martin Pitt <mp...@debian.org> Sun, 30 Apr 2017 19:09:57 +0200
+
autopkgtest (4.3) unstable; urgency=medium
[ SZALAY Attila ]
diff --git a/doc/README.package-tests.rst b/doc/README.package-tests.rst
index 4775a07..5c9e1f3 100644
--- a/doc/README.package-tests.rst
+++ b/doc/README.package-tests.rst
@@ -314,4 +314,25 @@ Example:
reboot
fi
+Network access
+--------------
+autopkgtest needs access to the network at least for downloading test
+dependencies and possibly dist-upgrading testbeds. In environments with
+restricted internet access you need to set up an apt proxy and configure
+the testbed to use it. (Note that the standard tools like
+autopkgtest-build-lxc or mk-sbuild automatically use the apt proxy from
+the host system.)
+
+In general, tests are also allowed to access the internet. As this
+usually makes tests less reliable, this should be kept to a minimum; but
+for many packages their main purpose is to interact with remote web
+services and thus their testing should actually cover those too, to
+ensure that the distribution package keeps working with their
+corresponding web service.
+
+Debian's production CI infrastructure allows unrestricted network
+access, in Ubuntu's infrastructure access to sites other than
+`*.ubuntu.com` and `*.launchpad.net` happens via a proxy (limited to
+DNS and http/https).
+
.. vim: ft=rst tw=72
diff --git a/lib/VirtSubproc.py b/lib/VirtSubproc.py
index d76a861..688e67e 100644
--- a/lib/VirtSubproc.py
+++ b/lib/VirtSubproc.py
@@ -700,6 +700,14 @@ def prepare():
sethandlers(handler)
+def cmd_auxverb_debug_fail(c, ce):
+ cmdnumargs(c, ce)
+ try:
+ adtlog.info(caller.hook_debug_fail())
+ except AttributeError:
+ pass
+
+
def mainloop():
global in_mainloop
in_mainloop = True
diff --git a/lib/adt_testbed.py b/lib/adt_testbed.py
index d759322..7f74b1b 100644
--- a/lib/adt_testbed.py
+++ b/lib/adt_testbed.py
@@ -460,6 +460,7 @@ class Testbed:
adtlog.debug('testbed command exited with code %i' % proc.returncode)
if proc.returncode in (254, 255):
+ self.command('auxverb_debug_fail')
self.bomb('testbed auxverb failed with exit code %i' %
proc.returncode)
return (proc.returncode, out, err)
@@ -522,6 +523,7 @@ Description: satisfy autopkgtest test dependencies
'--assume-yes --fix-broken '
'-o APT::Status-Fd=3 '
'-o APT::Install-Recommends=%s '
+ '-o Dpkg::Options::=--force-confnew '
'-o Debug::pkgProblemResolver=true
3>&2 2>&1' %
(' '.join(self.eatmydata_prefix),
recommends)],
kind='install',
stderr=subprocess.PIPE)
diff --git a/runner/autopkgtest b/runner/autopkgtest
index daabdba..2f43446 100755
--- a/runner/autopkgtest
+++ b/runner/autopkgtest
@@ -374,10 +374,10 @@ def build_source(kind, arg, built_binaries):
# very old source packages don't have Package-List: yet, fall back to
Binary:
# (Binary: is generally not sufficient as it gets truncated for long
lists)
create_command = ('pkgs=$(apt-cache showsrc --only-source %(src)s || [
$? != 100 ] || apt-cache showsrc %(src)s); '
- 'pkgs=$(echo "$pkgs" | awk "/^Package: / { if (\$2
!= \\"%(src)s\\") { skippar=1; next; } else { skippar=0}} { if (skippar) next;
} /^Binary:/ { sub(/^Binary:/, \\"\\"); gsub(/,/, \\"\\");
split(\$0,oldpkgs)}; /^Package-List:/ { inlist=1; have_pl=1; delete thissrc; if
(\$2) thissrc[\$2] = 1; next } (/^ / && inlist == 1) { thissrc[\$1] = 1; next }
{ if (!inlist) next; inlist=0; if (intersect) {for (p in pkgs) { if (!(p in
thissrc)) delete pkgs[p]; else remaining=1}; if (!remaining) {for (p in
thissrc) {pkgs[p] = 1}} } else { for (p in thissrc) { pkgs[p] = 1}; intersect=1
} } END {if (have_pl) { for (p in pkgs) print p } else {for (p in oldpkgs)
print oldpkgs[p]} }");'
+ 'pkgs=$(echo "$pkgs\n" | awk "/^Package: / { if (\$2
!= \\"%(src)s\\") { skippar=1; next; } else { skippar=0}} { if (skippar) next;
} /^Binary:/ { sub(/^Binary:/, \\"\\"); gsub(/,/, \\"\\");
split(\$0,oldpkgs)}; /^Package-List:/ { inlist=1; have_pl=1; delete thissrc; if
(\$2) thissrc[\$2] = 1; next } (/^ / && inlist == 1) { thissrc[\$1] = 1; next }
{ if (!inlist) next; inlist=0; if (intersect) {for (p in pkgs) { if (!(p in
thissrc)) delete pkgs[p]; else remaining=1}; if (!remaining) {for (p in
thissrc) {pkgs[p] = 1}} } else { for (p in thissrc) { pkgs[p] = 1}; intersect=1
} } END {if (have_pl) { for (p in pkgs) print p } else {for (p in oldpkgs)
print oldpkgs[p]} }");'
' [ -n "$pkgs" ] || exit 1; '
' for pkg in $pkgs; do'
- ' pkg_candidate=$(apt-cache policy "^$pkg\$"|sed -n
"/Candidate:/ { s/^.* //; /none/d; p}") || continue;'
+ ' pkg_candidate=$(apt-cache policy "^$(echo $pkg |
sed -r "s/([.+])/\\\\\1/g")\$"|sed -n "/Candidate:/ { s/^.* //; /none/d; p}")
|| continue;'
' [ -n "$pkg_candidate" ] || continue; '
' show=$(apt-cache show $pkg=$pkg_candidate | grep
"^Source:" || true);'
' [ "$pkg" = "%(src)s" ] || echo "$show" | grep -q
"^Source: %(src)s\\b" || continue; '
diff --git a/tests/autopkgtest b/tests/autopkgtest
index 27c8361..618d3cc 100755
--- a/tests/autopkgtest
+++ b/tests/autopkgtest
@@ -43,13 +43,19 @@ import testarchive
# in some corner cases apt-get download might not be available in a build
# environment, so check if this actually works
-have_apt = subprocess.call(['apt-get', 'download', 'gir1.2-json-1.0'],
- stdout=subprocess.PIPE, stderr=subprocess.PIPE,
- cwd=os.environ.get('TMPDIR', '/tmp')) == 0
+try:
+ have_apt = subprocess.call(['apt-get', 'download', 'gir1.2-json-1.0'],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ cwd=os.environ.get('TMPDIR', '/tmp')) == 0
+except OSError:
+ have_apt = False
-have_apt_src = subprocess.call(['apt-cache', 'showsrc', 'coreutils'],
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT) == 0
+try:
+ have_apt_src = subprocess.call(['apt-cache', 'showsrc', 'coreutils'],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT) == 0
+except OSError:
+ have_apt_src = False
have_ubuntu_device_flash = subprocess.call(['which', 'ubuntu-device-flash'],
stdout=subprocess.PIPE,
@@ -920,10 +926,9 @@ bad FAIL non-zero exit status 1
@unittest.skipIf(os.getuid() == 0, 'needs to run as user')
@unittest.skipIf(os.path.exists('/usr/lib/python3/dist-packages/wand/'),
'needs python3-wand uninstalled')
- @unittest.skipUnless(have_apt, 'needs apt-get download working')
- @unittest.skipUnless(subprocess.call(['apt-cache', 'show', 'python3-wand'],
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT) == 0,
+ @unittest.skipUnless(have_apt and subprocess.call(['apt-cache', 'show',
'python3-wand'],
+ stdout=subprocess.PIPE,
+
stderr=subprocess.STDOUT) == 0,
'needs python3-wand package')
def test_tmp_install_imagemagick(self):
'''temp dir unpack of imagemagick dependencies'''
@@ -1704,7 +1709,7 @@ Restrictions: needs-root
# modify fake apt-cache to show single-line P-L
with open(os.path.join(self.chroot, 'usr', 'bin', 'apt-cache'), 'w')
as f:
- f.write('if [ "$1" = showsrc ]; then printf "Package-List: $3 deb
utils optional arch=any\\nFormat: 1.0\\n"; fi\n')
+ f.write('if [ "$1" = showsrc ]; then printf "Package-List: $3 deb
utils optional arch=any\\n"; fi\n')
p = self.build_src('Tests: pass\nDepends:\nRestrictions: needs-root\n',
{'pass': '#!/bin/sh\necho I am fine\n'})
diff --git a/tools/autopkgtest-build-lxd b/tools/autopkgtest-build-lxd
index ebb8472..8607975 100755
--- a/tools/autopkgtest-build-lxd
+++ b/tools/autopkgtest-build-lxd
@@ -83,15 +83,16 @@ setup() {
ARCH=$(lxc exec "$CONTAINER" -- dpkg --print-architecture </dev/null)
DISTRO=$(lxc exec "$CONTAINER" -- sh -ec 'lsb_release -si 2>/dev/null || .
/etc/os-release; echo "${NAME% *}"' </dev/null)
- RELEASE=$(lxc exec "$CONTAINER" -- sh -ec 'lsb_release -sc 2>/dev/null ||
awk "/^deb/ {sub(/\\[.*\\]/, \"\", \$0); print \$3; quit}"
/etc/apt/sources.list' </dev/null)
- echo "Container finished booting. Distribution $DISTRO, release $RELEASE,
architecture $ARCH"
+ CRELEASE=$(lxc exec "$CONTAINER" -- sh -ec 'lsb_release -sc 2>/dev/null ||
awk "/^deb/ {sub(/\\[.*\\]/, \"\", \$0); print \$3; quit}"
/etc/apt/sources.list' </dev/null)
+ echo "Container finished booting. Distribution $DISTRO, release $CRELEASE,
architecture $ARCH"
+ RELEASE=${RELEASE:-${CRELEASE}}
# find setup-testbed script
for script in $(dirname $(dirname "$0"))/setup-commands/setup-testbed \
/usr/share/autopkgtest/setup-commands/setup-testbed; do
if [ -r "$script" ]; then
echo "Running setup script $script..."
- lxc exec "$CONTAINER" -- env MIRROR=${MIRROR:-} sh < "$script"
+ lxc exec "$CONTAINER" -- env MIRROR=${MIRROR:-} RELEASE=${RELEASE}
sh < "$script"
break
fi
done
diff --git a/tools/autopkgtest-build-lxd.1 b/tools/autopkgtest-build-lxd.1
index 72d0f73..399d618 100644
--- a/tools/autopkgtest-build-lxd.1
+++ b/tools/autopkgtest-build-lxd.1
@@ -22,6 +22,12 @@ You can specify an apt proxy to use in the container in the
environment variable. If you have an apt proxy configured on the host, the
container will automatically use this, otherwise there is no default.
+For dist-upgrading \fIimage\fR to a newer release you can specify the
+destination release in the
+.B $RELEASE
+environment variable. This is useful when a new distribution series has just
+been opened and there are no standard images for that available yet.
+
.SH EXAMPLE
Build a local autopkgtest container image for Debian sid i386, based on the
diff --git a/virt/autopkgtest-virt-lxd b/virt/autopkgtest-virt-lxd
index c453724..4c7319b 100755
--- a/virt/autopkgtest-virt-lxd
+++ b/virt/autopkgtest-virt-lxd
@@ -184,12 +184,46 @@ def hook_revert():
hook_open()
+def get_uptime():
+ try:
+ (rc, out, _) = VirtSubproc.execute_timeout(
+ None, 10, ['lxc', 'exec', container_name, '--', 'cat',
'/proc/uptime'],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ if rc != 0:
+ return
+
+ return float(out.split()[0])
+ except IndexError:
+ return
+
+
def hook_wait_reboot():
adtlog.debug('hook_wait_reboot: waiting for container to shut down...')
# "lxc exec" exits with 0 when the container stops, so just wait longer
# than our timeout
- VirtSubproc.execute_timeout(None, 300, ['lxc', 'exec', container_name,
- 'sleep', '3600'])
+ initial_uptime = get_uptime()
+
+ adtlog.debug('hook_wait_reboot: container up for %s, waiting for reboot' %
initial_uptime)
+
+ for retry in range(20):
+ time.sleep(5)
+
+ current_uptime = get_uptime()
+
+ # container is probably in the very late stages of shutting down, just
+ # keep trying, if this persists we'll bomb out later on
+ if not current_uptime:
+ continue
+
+ if current_uptime < initial_uptime:
+ adtlog.debug('hook_wait_reboot: container now up for %s - has
rebooted (initial uptime %s)' % (current_uptime, initial_uptime))
+ break
+ else:
+ adtlog.debug('hook_wait_reboot: container now up for %s - has not
rebooted (initial uptime %s)' % (current_uptime, initial_uptime))
+ else:
+ VirtSubproc.bomb('timed out waiting for container %s to restart' %
container_name)
+
adtlog.debug('hook_wait_reboot: container restarted, waiting for boot to
finish')
wait_booted()
diff --git a/virt/autopkgtest-virt-qemu b/virt/autopkgtest-virt-qemu
index 37d078d..eb3b162 100755
--- a/virt/autopkgtest-virt-qemu
+++ b/virt/autopkgtest-virt-qemu
@@ -49,7 +49,6 @@ args = None
workdir = None
p_qemu = None
ssh_port = None
-ssh_port_lock = None
normal_user = None
qemu_cmd_default = None
@@ -451,26 +450,25 @@ def get_cpuflag():
def find_free_port(start):
'''Find an unused port in the range [start, start+50)'''
- global ssh_port_lock
-
for p in range(start, start + 50):
adtlog.debug('find_free_port: trying %i' % p)
try:
+ lockfile = '/tmp/autopkgtest-virt-qemu.port.%i' % p
+ f = None
try:
- ssh_port_lock = open('/run/lock/autopkgtest-virt-qemu.port.%i'
% p, 'w')
- fcntl.flock(ssh_port_lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ f = open(lockfile, 'x')
+ os.unlink(lockfile)
+ fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
except (IOError, OSError):
adtlog.debug('find_free_port: %i is locked' % p)
- if ssh_port_lock:
- ssh_port_lock.close()
- ssh_port_lock = None
continue
+ finally:
+ if f:
+ f.close()
s = socket.create_connection(('127.0.0.1', p))
# if that works, the port is taken
s.close()
- ssh_port_lock.close()
- ssh_port_lock = None
continue
except socket.error as e:
if e.errno == errno.ECONNREFUSED:
diff --git a/virt/autopkgtest-virt-ssh b/virt/autopkgtest-virt-ssh
index a065186..9ad5a20 100755
--- a/virt/autopkgtest-virt-ssh
+++ b/virt/autopkgtest-virt-ssh
@@ -163,7 +163,7 @@ def parse_args():
capabilities += args.capability
-def execute_setup_script(command, fail_ok=False):
+def execute_setup_script(command, fail_ok=False, print_stderr=True):
'''Run the --setup-script, if given.
Arguments passed after -- to the main program are passed verbatim to the
@@ -173,10 +173,13 @@ def execute_setup_script(command, fail_ok=False):
:param command: Command to execute. The command must match a function in
the ssh script
- :param fail_ok: If True, failures will not cause bombing, and this function
- returns the exit code instead.
+ :param fail_ok: If True, failures will not cause bombing
+ :return: A tuple (return code, stdout, stderr). stdout and stderr may be
+ None, for example if the script fails and fail_ok is True.
'''
global sshconfig, args
+ out = None
+ err = None
if args.setup_script:
fpath = args.setup_script
@@ -195,13 +198,20 @@ def execute_setup_script(command, fail_ok=False):
adtlog.debug('Executing setup script: %s' % ' '.join(cmd))
(status, out, err) = VirtSubproc.execute_timeout(
- None, 1800, cmd, stdout=subprocess.PIPE)
+ None,
+ 1800,
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ if print_stderr:
+ # Keep outputting the error on stderr as well as capturing it
+ sys.stderr.write(err)
if status != 0:
err = 'setup script failed with code %i: %s' % (status,
' '.join(cmd))
if fail_ok:
adtlog.debug(err)
- return status
+ return (status, out, err)
else:
execute_setup_script('debug-failure', fail_ok=True)
VirtSubproc.bomb(err)
@@ -218,7 +228,7 @@ def execute_setup_script(command, fail_ok=False):
if a is not None:
sshconfig[param] = a
- return 0
+ return (0, out, err)
def host_setup(command):
@@ -399,6 +409,15 @@ def can_sudo(ssh_cmd):
return (None, None)
+def hook_debug_fail():
+ # Don't print stderr; if we're being called for the hook, we assume the
+ # caller is going to do that.
+ (status, out, err) = execute_setup_script('debug-failure',
+ fail_ok=True,
+ print_stderr=False)
+ return err
+
+
def hook_open():
host_setup('open')
@@ -440,7 +459,7 @@ def hook_wait_reboot():
global sshcmd
if args.setup_script:
- rc = execute_setup_script('wait-reboot', fail_ok=True)
+ (rc, _, _) = execute_setup_script('wait-reboot', fail_ok=True)
else:
# if we don't have a setup script, use the fallback below
rc = 1
signature.asc
Description: PGP signature
