Hi Martin, So I've spent some time again on this and have separated this into three patches:
0001-adt-virt-qemu-Implement-support-for-nested-base-imag.patch
This adds the image specified to adt-virt-qemu as an additional read
only virtio block device (with serial BASEIMAGE) to the VM (in
contrast to my first patch, this is the actual data, not a qcow2
image).
It sets the ADT_BASEIMAGE environment variable to that device.
It also does two things with udev in setup-testbed:
1) it creates an udev rule to create a symlink /dev/baseimage to
the new image (ADT_BASEIMAGE remains the absolute path for now,
though)
2) it modifies 60-persistent-storage.rules to skip blkid processing
of UUID etc. for the base image, since the base image is likely
the same as the main image, so UUIDs would match. Otherwise the
/dev/disk/by-uuid symlinks would point to the base image drive.
The same as with my first patch: if multiple images are specified on
the adt-virt-qemu command line, it is impossible to determine what
an approrpriate base image is, hence my patch doesn't set the base
image in that case, though you may specify one explicitly via the
command line.
0002-adt-virt-qemu-Use-host-CPU-type-by-default.patch
Passes -cpu host to QEMU by default _unless_ --qemu-command= is
specified. (If qemu-command is specified, it might be the case that
someone wants to run a fully emulated different architecture, and
then -cpu host will fail spectacularly.)
This can be overridden by the --qemu-cpu option for adt-virt-qemu,
which then replaces the value of QEMU's -cpu parameter. If
--qemu-cpu=qemu-default is specified, the -cpu option is dropped
completely.
0003-Add-needs-baseimage-restriction.patch
I split this out from the first patch, because this is what you
didn't like the first time around. I still think a new restriction is
a good idea, in order to be able to distinguish between skipped tests
and tests that were actually successful.
(Of course, without a base image, one could do vmdebootstrap within
the outer VM, but that would be a huge waste of resources, without
a base image the only sensible thing to do is to skip the tests. And
I'd really prefer .)
All patches also update the documentation. Note that since autopkgtest
specification is now part of Debian policy, if you agree to my third
patch, an update to the policy package would also be required, which
should probably only follow after this has been in use for a while, so
that we could still change it if we determine some problems. I'd be
willing to take care of that part.
I've tested this with Debian sid and Ubuntu Xenial images (created via
vmdebootstrap and adt-buildvm-ubuntu-cloud, as per the manpage) and it
works on both.
CAVEAT though:
The first patch that adds the base image as an additional read-only
drive has a slight problem if old images are used: since UUIDs are
the same for filesystems on the main virtio drive and the base image
"drive" (because udev processing of the base image "drive" is not
disabled), partitions on both drives will have the same UUID, and
apparently (at least on sid) udev selects the second drive for the
symlinks, so /dev/disk/by-uuid/* will point to /dev/vdb*.
This is not a problem for the default images (even those that were
created from older versions), because they only contain one partition
and the initiramfs is not affected, but if someone has multiple
partitions in their test image, this will cause problems.
I don't think this is a huge deal, beacuse you have to regenerate VM
base images for ADT regularly anyway, but this is something to keep
in mind (and maybe requires a NEWS entry?).
I don't have a good solution for that other than to disable the base
image logic unless explicitly enabled - but I'd really like to see
this on by default.
I'd really like to see this moving forward.
Thoughts?
Regards,
Christian
From 5ee53e4ce023521b43a67708a9c533a9c3a5b02a Mon Sep 17 00:00:00 2001
From: Christian Seiler <christ...@iwakd.de>
Date: Fri, 26 Feb 2016 16:14:29 +0100
Subject: [PATCH 1/3] adt-virt-qemu: Implement support for nested base images.
This adds initial support for nested base images to be passed into the
test environment, so that nested VMs may be used in tests.
By default, a copy of the main image is passed to the VM, but it may
be overwritten by the --nested-baseimage command line parameter. Note
that this parameter becomes mandatory for supporting this if more than
one image is speicifed on the regular command line.
setup-testbed is modified to modify 60-persistent-storage.rules: the
base image that is passed into the system should not be scanned for
file system UUIDs etc., because they are likely to be duplicates of the
UUIDs of the image that was booted from.
Closes: #800845
---
debian/changelog | 5 +++++
setup-commands/setup-testbed | 16 ++++++++++++++++
virt-subproc/adt-virt-qemu | 19 +++++++++++++++++--
virt-subproc/adt-virt-qemu.1 | 14 ++++++++++++++
4 files changed, 52 insertions(+), 2 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 4936ea8..b9c6a1a 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,11 +1,16 @@
autopkgtest (3.19.4) UNRELEASED; urgency=medium
+ [ Martin Pitt ]
* setup-commands/setup-testbed: Ensure that removing cruft does not remove
cloud-init. (LP: #1539126)
* setup-commands/setup-testbed: Purge lxd and lxc.
* adt-virt-lxc: Don't fail on deprecation warnings of lxc-clone and
lxc-start-ephemeral. (LP: #1549995)
+ [ Christian Seiler ]
+ * adt-virt-qemu: Implement support for nested base images.
+ (Closes: #800845)
+
-- Martin Pitt <mp...@debian.org> Tue, 23 Feb 2016 18:21:51 +0100
autopkgtest (3.19.3) unstable; urgency=medium
diff --git a/setup-commands/setup-testbed b/setup-commands/setup-testbed
index b7f586f..71a2087 100755
--- a/setup-commands/setup-testbed
+++ b/setup-commands/setup-testbed
@@ -142,6 +142,22 @@ if [ -z "${ADT_IS_SETUP_COMMAND:-}" ]; then
fi
fi
+# set up base image udev rules
+if [ -z "${ADT_IS_SETUP_COMMAND:-}" ]; then
+ # Modify persistent storage rules to make sure that partitions on
+ # the base image are not scanned for UUID etc., because typically
+ # an overlay over the base image is used for the main disk, and
+ # we don't want to have to deal with conflicting UUIDs and such.
+ awk '/^[^ ].*IMPORT\{builtin\}=\"blkid\"/ {
+ print "KERNEL==\"vd*\", ENV{ID_SERIAL}==\"BASEIMAGE\", ENV{UDISKS_IGNORE}=\"1\", GOTO=\"persistent_storage_end\"";
+ }
+ { print; }' < "$root/lib/udev/rules.d/60-persistent-storage.rules" \
+ > "$root/etc/udev/rules.d/60-persistent-storage.rules"
+ # Create /dev/baseimage symlink for the base image.
+ printf '%s\n' 'KERNEL=="vd*[!0-9]", ENV{ID_SERIAL}=="BASEIMAGE", SYMLINK+="baseimage"' \
+ > "$root/etc/udev/rules.d/61-baseimage.rules"
+fi
+
# go-faster apt/dpkg
echo "Acquire::Languages \"none\";" > "$root"/etc/apt/apt.conf.d/90nolanguages
echo 'force-unsafe-io' > "$root"/etc/dpkg/dpkg.cfg.d/autopkgtest
diff --git a/virt-subproc/adt-virt-qemu b/virt-subproc/adt-virt-qemu
index 69d68f9..819f96e 100755
--- a/virt-subproc/adt-virt-qemu
+++ b/virt-subproc/adt-virt-qemu
@@ -80,6 +80,8 @@ def parse_args():
help='Enable debugging output')
parser.add_argument('--qemu-options',
help='Pass through arguments to QEMU command.')
+ parser.add_argument('--nested-baseimage',
+ help='VM base image for use inside the VM (nested VMs)')
parser.add_argument('image', nargs='+',
help='disk image to add to the VM (in order)')
@@ -87,6 +89,8 @@ def parse_args():
if args.debug:
adtlog.verbosity = 2
+ if args.nested_baseimage is None and len(args.image) == 1:
+ args.nested_baseimage = args.image[0]
def prepare_overlay():
@@ -274,6 +278,13 @@ EOF
def make_auxverb(shared_dir):
'''Create auxverb script'''
+ envvars = ''
+ if args.nested_baseimage is not None:
+ # don't use /dev/baseimage for now because we can't
+ # be sure that images created already include the
+ # necessary udev rule
+ envvars += 'export ADT_BASEIMAGE=/dev/vd%c ;' % chr(ord('a') + len(args.image))
+
auxverb = os.path.join(workdir, 'runcmd')
with open(auxverb, 'w') as f:
f.write('''#!%(py)s
@@ -340,7 +351,8 @@ t_stderr.start()
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect('%(tty)s')
cmd = '/bin/eofcat %%(d)s/stdin_eof %%(d)s/exit.tmp < %%(d)s/stdin | ' \\
- '(%%(c)s >> %%(d)s/stdout 2>> %%(d)s/stderr; echo $? > %%(d)s/exit.tmp);' \\
+ '(%(envvars)s ' \\
+ '%%(c)s >> %%(d)s/stdout 2>> %%(d)s/stderr; echo $? > %%(d)s/exit.tmp);' \\
'mv %%(d)s/exit.tmp %%(d)s/exit\\n' %% \\
{'d': job_guest, 'c': ' '.join(map(pipes.quote, sys.argv[1:]))}
s.send(cmd.encode())
@@ -370,7 +382,7 @@ t_stdout.join()
t_stderr.join()
# code 255 means that the auxverb itself failed, so translate
sys.exit(rc == 255 and 253 or rc)
-''' % {'py': sys.executable, 'tty': os.path.join(workdir, 'ttyS1'), 'dir': shared_dir})
+''' % {'py': sys.executable, 'tty': os.path.join(workdir, 'ttyS1'), 'dir': shared_dir, 'envvars': envvars})
os.chmod(auxverb, 0o755)
@@ -476,6 +488,9 @@ def hook_open():
for i, image in enumerate(args.image[1:]):
argv.append('-drive')
argv.append('file=%s,if=virtio,index=%i,readonly' % (image, i + 1))
+ if args.nested_baseimage is not None:
+ argv.append('-drive')
+ argv.append('file=%s,if=virtio,index=%i,readonly,serial=BASEIMAGE' % (args.nested_baseimage, len(args.image)))
if os.path.exists('/dev/kvm'):
argv.append('-enable-kvm')
diff --git a/virt-subproc/adt-virt-qemu.1 b/virt-subproc/adt-virt-qemu.1
index 81bb11f..9c29602 100644
--- a/virt-subproc/adt-virt-qemu.1
+++ b/virt-subproc/adt-virt-qemu.1
@@ -97,6 +97,20 @@ Enable debugging output.
.BI "--qemu-options=" arguments
Pass through arguments to QEMU command; e. g. --qemu-options='-readconfig qemu.cfg'
+.TP
+.BI "--nested-baseimage=" image
+Base image that is passed to the VM in a read-only mode. It may then be
+used by the tests to start a nested VM. It is typically not required to
+specify this option, because if there is only one image specified to
+.BR adt-virt-qemu ,
+this option defaults to that image. Only if there is more than one
+image listed on the command line will this option become necessary.
+
+If a base image is available within the virtual machine, the
+.I ADT_BASEIMAGE
+environment variable will be set to point to that image within the
+virtual machine (the variable will not be set otherwise).
+
.SH CONFIGURATION FILES
If you use lots of options or images, you can put parts of, or the whole
command line into a text file, with one line per option. E. g. you can create a
--
2.1.4
From 718663dae3056f4b602ffd711c3742060343aa85 Mon Sep 17 00:00:00 2001
From: Christian Seiler <christ...@iwakd.de>
Date: Fri, 26 Feb 2016 16:14:47 +0100
Subject: [PATCH 2/3] adt-virt-qemu: Use host CPU type by default
Tell QEMU to use the host's CPU type by default so that nested KVM
support is possible. Add an option to allow the user to override
this setting.
---
debian/changelog | 1 +
virt-subproc/adt-virt-qemu | 19 ++++++++++++++++++-
virt-subproc/adt-virt-qemu.1 | 32 ++++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/debian/changelog b/debian/changelog
index b9c6a1a..82a9a07 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -10,6 +10,7 @@ autopkgtest (3.19.4) UNRELEASED; urgency=medium
[ Christian Seiler ]
* adt-virt-qemu: Implement support for nested base images.
(Closes: #800845)
+ * adt-virt-qemu: Use host CPU type by default
-- Martin Pitt <mp...@debian.org> Tue, 23 Feb 2016 18:21:51 +0100
diff --git a/virt-subproc/adt-virt-qemu b/virt-subproc/adt-virt-qemu
index 819f96e..1f4f192 100755
--- a/virt-subproc/adt-virt-qemu
+++ b/virt-subproc/adt-virt-qemu
@@ -52,10 +52,11 @@ p_qemu = None
ssh_port = None
ssh_port_lock = None
normal_user = None
+qemu_cmd_default = None
def parse_args():
- global args
+ global args, qemu_cmd_default
parser = ArgumentParser()
@@ -78,6 +79,8 @@ def parse_args():
help='Show boot messages from serial console')
parser.add_argument('-d', '--debug', action='store_true',
help='Enable debugging output')
+ parser.add_argument('--qemu-cpu',
+ help='The QEMU CPU type to use (default: host).')
parser.add_argument('--qemu-options',
help='Pass through arguments to QEMU command.')
parser.add_argument('--nested-baseimage',
@@ -495,6 +498,20 @@ def hook_open():
if os.path.exists('/dev/kvm'):
argv.append('-enable-kvm')
+ if args.qemu_cpu is None:
+ if args.qemu_command == qemu_cmd_default:
+ cpu_type = 'host'
+ else:
+ # We are using a different qemu command, which means that
+ # -cpu host might not work (because it's fully emulated,
+ # for example), so don't set the option at all.
+ cpu_type = None
+ else:
+ cpu_type = args.qemu_cpu
+ if cpu_type is not None and cpu_type != 'qemu-default':
+ argv.append('-cpu')
+ argv.append(cpu_type)
+
# pass through option to qemu
if args.qemu_options:
argv.extend(args.qemu_options.split())
diff --git a/virt-subproc/adt-virt-qemu.1 b/virt-subproc/adt-virt-qemu.1
index 9c29602..c3a3473 100644
--- a/virt-subproc/adt-virt-qemu.1
+++ b/virt-subproc/adt-virt-qemu.1
@@ -94,6 +94,38 @@ Show boot messages from serial console.
Enable debugging output.
.TP
+.BI "--qemu-cpu=" type
+The QEMU CPU type to use. This defaults to
+.I host
+if the
+.B --qemu-command
+option has not been specified, which passes through all of the host CPU's
+features into the VM and allows the use of nested KVM if enabled on the host.
+For emulating a more generic CPU on x86 systems that still has KVM support, one
+may use the
+.BI --qemu-cpu= kvm64
+or
+.BI --qemu-cpu= kvm32
+CPU variaants.
+
+Using
+.BI --qemu-cpu= qemu-default
+disables this option and lets QEMU select the default CPU type for the given
+architecture.
+
+Nested KVM currently only works on x86 systems. To enable it, one needs to
+additionally set some module parameters on the host, by creating a file
+.I /etc/modprobe.d/nested_kvm.conf
+with the contents
+
+.EX
+options kvm_intel nested=1
+options kvm_amd nested=1
+.EE
+
+and rebooting and/or reloading the KVM modules.
+
+.TP
.BI "--qemu-options=" arguments
Pass through arguments to QEMU command; e. g. --qemu-options='-readconfig qemu.cfg'
--
2.1.4
From 135988ff18e58f93db204bbba42a5d544435397d Mon Sep 17 00:00:00 2001
From: Christian Seiler <christ...@iwakd.de>
Date: Fri, 26 Feb 2016 16:14:49 +0100
Subject: [PATCH 3/3] Add needs-baseimage restriction
Add the needs-baseimage restriction for package tests that need to be
able to start a VM inside the test environment.
---
debian/changelog | 1 +
doc/README.package-tests.rst | 20 ++++++++++++++++++++
lib/testdesc.py | 9 ++++++++-
virt-subproc/adt-virt-qemu | 4 +++-
4 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/debian/changelog b/debian/changelog
index 82a9a07..b5b2e94 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -11,6 +11,7 @@ autopkgtest (3.19.4) UNRELEASED; urgency=medium
* adt-virt-qemu: Implement support for nested base images.
(Closes: #800845)
* adt-virt-qemu: Use host CPU type by default
+ * Add needs-baseimage restriction
-- Martin Pitt <mp...@debian.org> Tue, 23 Feb 2016 18:21:51 +0100
diff --git a/doc/README.package-tests.rst b/doc/README.package-tests.rst
index a7874e5..78248bf 100644
--- a/doc/README.package-tests.rst
+++ b/doc/README.package-tests.rst
@@ -207,6 +207,26 @@ needs-recommends
Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies.
+needs-baseimage
+ The test needs to have a read-only base image available so it may
+ create an overlay and start a qemu/KVM virtual machine inside the
+ test environment.
+
+ This is useful for testing network client packages that require
+ kernel support (NFS, CIFS, iSCSI, NBD, etc.): the external testing
+ environment sets up a minimalistic server environment and then
+ starts a virtual machine that tests the client.
+
+ While currently only adt-virt-qemu supports this, this restriction
+ is independent of the isolation level. If the setup of the server
+ also needs a specific isolation level, that should be specified
+ additionally.
+
+ The environment variable ADT_BASEIMAGE will be set to the absolute
+ path of the base image. If the test environment supports it, this
+ variable will be available irrespective of whether this restriction
+ was added to the test or not.
+
Defined features
----------------
diff --git a/lib/testdesc.py b/lib/testdesc.py
index c5ecf83..e50c651 100644
--- a/lib/testdesc.py
+++ b/lib/testdesc.py
@@ -42,7 +42,8 @@ import adtlog
known_restrictions = ['rw-build-tree', 'breaks-testbed', 'needs-root',
'build-needed', 'allow-stderr', 'isolation-container',
- 'isolation-machine', 'needs-recommends']
+ 'isolation-machine', 'needs-recommends',
+ 'needs-baseimage']
class Unsupported(Exception):
@@ -161,6 +162,12 @@ class Test:
'Test needs root on testbed which is not '
'available')
+ if 'needs-baseimage' in self.restrictions and \
+ 'provides-baseimage' not in caps:
+ raise Unsupported(self.name,
+ 'Test needs baseimage inside testbed which '
+ 'is not available')
+
#
# Parsing for Debian source packages
#
diff --git a/virt-subproc/adt-virt-qemu b/virt-subproc/adt-virt-qemu
index 1f4f192..7dbd8ee 100755
--- a/virt-subproc/adt-virt-qemu
+++ b/virt-subproc/adt-virt-qemu
@@ -583,13 +583,15 @@ def hook_forked_inchild():
def hook_capabilities():
- global normal_user
+ global normal_user, args
caps = ['revert', 'revert-full-system', 'root-on-testbed',
'isolation-machine', 'reboot']
# disabled, see hook_downtmp()
# caps.append('downtmp-host=%s' % os.path.join(workdir, 'shared', 'tmp'))
if normal_user:
caps.append('suggested-normal-user=' + normal_user)
+ if args.nested_baseimage is not None:
+ caps.append('provides-baseimage')
return caps
--
2.1.4
signature.asc
Description: OpenPGP digital signature
