On Tue, Jul 20, 2021 at 7:49 PM Bruce Ashfield <[email protected]> wrote:
>
> On Thu, Jul 8, 2021 at 12:43 AM Christopher Clark
> <[email protected]> wrote:
> >
> > From: Christopher Clark <[email protected]>
> >
> > The Xen hypervisor built for Arm 64-bit targets can be launched with
> > runqemu by providing a Device Tree binary and configuration for Qemu,
> > which enables interactive testing of Xen images.
> >
> > Add qemuboot-xen-dtb.bbclass to add a new bitbake task for generating
> > the dtb file by using lopper on a device tree produced by Qemu.
> >
> > Add qemuboot-xen-defaults.bbclass to supply working default parameters
> > for the qemuarm64 machine and general support for qemuboot for Xen.
> >
> > Add the new qemuboot-xen classes to the xen-image-minimal image, along
> > with variable settings needed for qemuboot to run the image for
> > qemuarm64.
> >
> > The memory setting for Qemu has been increased to 512M after testing
> > with successful boots. Default fstype for qemux86-64 was found to need a
> > stronger override in testing. Provide an override for the serial console
> > setting for the qemuarm64 machine when the Xen distro feature is
> > enabled for the console in qemuboot, following the example of similar
> > machine settings in meta-arm-bsp.
> >
> > Development aided by this script by Stewart Hildebrand of DornerWorks:
> > https://gist.github.com/stewdk/110f43e0cc1d905fc6ed4c7e10d8d35e
> >
> > Signed-off-by: Christopher Clark <[email protected]>
> > ---
> > classes/qemuboot-xen-defaults.bbclass | 29 +++
> > classes/qemuboot-xen-dtb.bbclass | 209 +++++++++++++++++++
> > conf/distro/include/meta-virt-xen.inc | 3 +
> > recipes-extended/images/xen-image-minimal.bb | 6 +-
> > 4 files changed, 245 insertions(+), 2 deletions(-)
> > create mode 100644 classes/qemuboot-xen-defaults.bbclass
> > create mode 100644 classes/qemuboot-xen-dtb.bbclass
> >
> > diff --git a/classes/qemuboot-xen-defaults.bbclass
> > b/classes/qemuboot-xen-defaults.bbclass
> > new file mode 100644
> > index 0000000..82cad49
> > --- /dev/null
> > +++ b/classes/qemuboot-xen-defaults.bbclass
> > @@ -0,0 +1,29 @@
> > +# Set defaults for booting Xen images with qemuboot
> > +
> > +# Xen and Dom0 command line options
> > +QB_XEN_CMDLINE_EXTRA ??= "dom0_mem=192M"
> > +QB_XEN_DOM0_BOOTARGS ??= \
> > + "console=hvc0 earlyprintk clk_ignore_unused root=/dev/vda"
> > +
> > +# Launch with one initial domain, dom0, with one boot module, the kernel
> > +DOM0_KERNEL="${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
> > +DOM0_KERNEL_LOAD_ADDR="0x45000000"
> > +QB_XEN_DOMAIN_MODULES =
> > "${DOM0_KERNEL}:${DOM0_KERNEL_LOAD_ADDR}:multiboot,kernel"
> > +
> > +# Qemuboot for 64-bit Arm uses the QB_DEFAULT_KERNEL method to load Xen
> > +# and the device loader option for the dom0 kernel:
> > +QB_OPT_APPEND_append_aarch64 = " \
> > + -device loader,file=${DOM0_KERNEL},addr=${DOM0_KERNEL_LOAD_ADDR} \
> > + "
> > +QB_DEFAULT_KERNEL_aarch64 = "xen-${MACHINE}"
> > +
> > +# 64-bit Arm: gic version 3
> > +QB_MACHINE_aarch64 = "-machine virt,gic-version=3 -machine
> > virtualization=true"
> > +
> > +# Use overrides for stronger assignment here as default memory set by the
> > +# machines is not high enough
> > +QB_MEM_qemuarm64 = "-m 512"
>
> What about making that memory value a variable (with 512 as the
> default) ? That allows us to change it without needing to always
> modify the bbclass.
>
> Almost all of my K8S runqemu launches need 512 or 1G of memory, so
> this is definitely not uncommon!
ack - will do that for v2
>
> > +
> > +# 64-bit Arm: qemuboot with a device tree binary
> > +QB_DTB_aarch64 = "${IMAGE_NAME}.qemuboot.dtb"
> > +QB_DTB_LINK_aarch64 = "${IMAGE_LINK_NAME}.qemuboot.dtb"
> > diff --git a/classes/qemuboot-xen-dtb.bbclass
> > b/classes/qemuboot-xen-dtb.bbclass
> > new file mode 100644
> > index 0000000..08f9b02
> > --- /dev/null
> > +++ b/classes/qemuboot-xen-dtb.bbclass
> > @@ -0,0 +1,209 @@
> > +# Enable booting Xen with qemuboot / runqemu: generate device tree
> > +#
> > +# Copyright (c) 2021 Star Lab Corp. All rights reserved.
> > +#
> > +# Author: Christopher Clark <[email protected]>
> > +
> > +# Interface variables:
> > +#
> > +# QB_DTB : defined in qemuboot.bbclass.
> > +# If set, this class will generate the specified device tree file.
> > +#
> > +# QB_XEN_CMDLINE_EXTRA :
> > +# A string to be appended to the default Xen hypervisor boot command line,
> > +# for supplying Xen boot options.
> > +# The device tree that this bbclass generates will contain Xen command
> > +# line options to connect the Xen console to the Qemu serial port.
> > +#
> > +# QB_XEN_DOMAIN_MODULES :
> > +# A space-separated list of colon-separated entries:
> > +# "<file for the module>:<load memory address>:<module compatibility
> > string>"
> > +#
> > +# QB_XEN_DOM0_BOOTARGS :
> > +# A string for specifying Dom0 boot options for the Xen section of the
> > device
> > +# tree.
> > +#
> > +# QB_XEN_DTB_TASK_DEPENDS :
> > +# The task dependencies for the DTB generation. A default is provided.
> > +#
> > +# See also: Other QB_ variables as defined by the qemuboot.bbclass.
> > +
> > +write_lops_xen_section() {
> > + cat <<EOF >"$1"
> > +/dts-v1/;
> > +/ {
> > + compatible = "system-device-tree-v1";
> > + lops {
> > + /* Connect the Xen console to Qemu dtuart */
> > + lop_1 {
> > + compatible = "system-device-tree-v1,lop,code-v1";
> > + code = "
> > + chosen = node.tree['/chosen']
> > + stdout_path = str(chosen['stdout-path'].value[0])
> > + chosen['xen,xen-bootargs'] = \
> > + 'console=dtuart dtuart=%s' % stdout_path
> > + return True
> > + ";
> > + };
> > + lop_2 {
> > + compatible = "system-device-tree-v1,lop,modify";
> > + modify = "/chosen:xen,dom0-bootargs:${QB_XEN_DOM0_BOOTARGS}";
> > + };
> > + lop_3 {
> > + compatible = "system-device-tree-v1,lop,modify";
> > + modify = "/chosen:#address-cells:<1>";
> > + };
> > + lop_4 {
> > + compatible = "system-device-tree-v1,lop,modify";
> > + modify = "/chosen:#size-cells:<1>";
> > + };
> > + };
> > +};
> > +EOF
> > +}
> > +
> > +write_lop_add_to_xen_cmdline() {
> > + EXTRA_XEN_BOOTARGS="$2"
> > + cat <<EOF >"$1"
> > +/dts-v1/;
> > +/ {
> > + compatible = "system-device-tree-v1";
> > + lops {
> > + lop_1 {
> > + compatible = "system-device-tree-v1,lop,code-v1";
> > + options = "extra_xen_bootargs:${EXTRA_XEN_BOOTARGS}";
> > + code = "
> > + chosen = node.tree['/chosen']
> > + xen_bootargs = str(chosen['xen,xen-bootargs'].value)
> > + chosen['xen,xen-bootargs'] = '%s %s' % \
> > + (xen_bootargs, extra_xen_bootargs)
> > + return True
> > + ";
> > + };
> > + };
> > +};
> > +EOF
> > +}
> > +
> > +write_lop_add_chosen_module() {
> > + ADDR="$2"
> > + SIZE="$3"
> > + MODULE_TYPE="$4"
> > + cat <<EOF >"$1"
> > +/dts-v1/;
> > +/ {
> > + compatible = "system-device-tree-v1";
> > + lops {
> > + lop_1 {
> > + compatible = "system-device-tree-v1,lop,add";
> > + node_src = "module@${ADDR}";
> > + node_dest = "/chosen/module@${ADDR}";
> > +
> > + module@${ADDR} {
> > + compatible = "multiboot,module", "${MODULE_TYPE}";
> > + reg = <${ADDR} ${SIZE}>;
> > + };
> > + };
> > + };
> > +};
> > +EOF
> > +}
> > +
> > +generate_xen_qemuboot_dtb() {
> > + # First: invoke qemu to generate an initial device tree.
> > + # Parameters supplied here inspired by inspection of:
> > + # runqemu "${IMAGE_BASENAME}" nographic slirp \
> > + # qemuparams='-dtb "" -machine dumpdtb=${B}/qemu-dumped.dtb'
> > + ${QB_SYSTEM_NAME} \
> > + -device qemu-xhci \
> > + -device usb-tablet \
> > + -device usb-kbd \
> > + -machine virt,gic-version=3 \
> > + -machine virtualization=true \
> > + ${QB_CPU} \
> > + ${QB_SMP} \
> > + ${QB_MEM} \
> > + -nographic \
> > + -serial mon:stdio \
> > + -machine "dumpdtb=${B}/qemu-dumped.dtb"
> > +
> > + # Lopper generates temporary files in cwd, so run it within ${B}
> > + cd "${B}"
> > + write_lops_xen_section "${B}/lop-insert-xen-section.dts"
> > +
> > + write_lop_add_to_xen_cmdline "${B}/lop-xen-cmdline.dts" \
> > + "${QB_XEN_CMDLINE_EXTRA}"
> > +
> > + if [ -z "${QB_XEN_DOMAIN_MODULES}" ]; then
> > + bbwarn "No domain modules: please set QB_XEN_DOMAIN_MODULES"
> > + fi
> > +
> > + for DOMAIN_MODULE in ${QB_XEN_DOMAIN_MODULES}
> > + do
> > + MODULE_FILE="$(echo ${DOMAIN_MODULE} | cut -f1 -d:)"
> > + ADDR="$(echo ${DOMAIN_MODULE} | cut -f2 -d:)"
> > + MODULE_TYPE="$(echo ${DOMAIN_MODULE} | cut -f3 -d:)"
> > + RESOLVED_FILE="$(readlink -f ${MODULE_FILE})"
> > + SIZE=$(printf '0x%x\n' $(stat -c '%s' "${RESOLVED_FILE}"))
> > + [ "x${SIZE}" != "x0x0" ] || bbfatal No module: "${MODULE_FILE}"
> > + write_lop_add_chosen_module "${B}/lop-add-module-${ADDR}.dts" \
> > + "${ADDR}" "${SIZE}" "${MODULE_TYPE}"
> > + LOP_MODULE_ARGS="${LOP_MODULE_ARGS} -i
> > ${B}/lop-add-module-${ADDR}.dts"
> > + done
> > +
> > + QEMUBOOT_DTB="${IMGDEPLOYDIR}/${QB_DTB}"
> > + QEMUBOOT_DTB_LINK="${IMGDEPLOYDIR}/${QB_DTB_LINK}"
> > +
> > + lopper.py --werror --verbose \
> > + -i "${B}/lop-insert-xen-section.dts" \
> > + -i "${B}/lop-xen-cmdline.dts" \
> > + ${LOP_MODULE_ARGS} \
> > + -f -o "${QEMUBOOT_DTB}" \
> > + "${B}/qemu-dumped.dtb"
> > +
> > + # To assist debugging:
> > + dtc -I dtb -O dts -o "${B}/output.dts" "${QEMUBOOT_DTB}"
> > +
> > + if [ "${QEMUBOOT_DTB_LINK}" != "${QEMUBOOT_DTB}" ] ; then
> > + if [ -e "${QEMUBOOT_DTB_LINK}" ] ; then
> > + rm "${QEMUBOOT_DTB_LINK}"
> > + fi
> > + ln -s "${QB_DTB}" "${QEMUBOOT_DTB_LINK}"
> > + fi
> > +}
> > +
> > +do_write_xen_qemuboot_dtb() {
> > + # Not all architectures qemuboot with a device tree binary, so check
> > + # to see if one is needed. This allows this bbclass file to be used
> > + # in the same image recipe for multiple architectures.
> > + if [ -n "${QB_DTB}" ] ; then
> > + generate_xen_qemuboot_dtb
> > + fi
> > +}
> > +
> > +addtask do_write_xen_qemuboot_dtb after do_write_qemuboot_conf before
> > do_image
> > +# Task dependency:
> > +# An expected common case is that the kernel for at least one of the
> > initial
> > +# domains (eg. dom0) is deployed from the virtual/kernel recipe, so
> > +# add virtual/kernel:do_deploy as a task dependency here since the kernel
> > size
> > +# needs to be known for generating the device tree.
> > +# Dependencies are only introduced if a device tree will be generated.
> > +QB_XEN_DTB_TASK_DEPENDS ?= " \
> > + ${@[ ' \
> > + qemu-helper-native:do_populate_sysroot \
> > + lopper-native:do_populate_sysroot \
> > + dtc-native:do_populate_sysroot \
> > + virtual/kernel:do_deploy \
> > + ', ''][d.getVar('QB_DTB') == '' or d.getVar('QB_DTB') is None]} \
> > + "
> > +do_write_xen_qemuboot_dtb[depends] = "${QB_XEN_DTB_TASK_DEPENDS}"
> > +
> > +def qemuboot_dtb_vars(d):
> > + build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE',
> > + 'KERNEL_IMAGETYPE', 'IMAGE_NAME', 'IMAGE_LINK_NAME',
> > + 'STAGING_DIR_NATIVE', 'STAGING_BINDIR_NATIVE',
> > + 'STAGING_DIR_HOST', 'SERIAL_CONSOLES']
> > + return build_vars + [k for k in d.keys() if k.startswith('QB_')]
> > +
> > +do_write_qemuboot_dtb[vardeps] += "${@' '.join(qemuboot_dtb_vars(d))}"
> > +do_write_qemuboot_dtb[vardepsexclude] += "TOPDIR"
> > diff --git a/conf/distro/include/meta-virt-xen.inc
> > b/conf/distro/include/meta-virt-xen.inc
> > index 02a7810..2becba1 100644
> > --- a/conf/distro/include/meta-virt-xen.inc
> > +++ b/conf/distro/include/meta-virt-xen.inc
> > @@ -10,3 +10,6 @@ XEN_RPI4_64_CONFIG_PATH ??= \
> >
> > 'dynamic-layers/raspberrypi/conf/distro/include/xen-raspberrypi4-64.inc'
> > include ${@bb.utils.contains('MACHINE', 'raspberrypi4-64', \
> > '${XEN_RPI4_64_CONFIG_PATH}', '', d)}
> > +
> > +# Set serial for working qemuboot console
> > +SERIAL_CONSOLES_qemuarm64 ?= "115200;ttyAMA0"
> > diff --git a/recipes-extended/images/xen-image-minimal.bb
> > b/recipes-extended/images/xen-image-minimal.bb
> > index 6733801..2175e63 100644
> > --- a/recipes-extended/images/xen-image-minimal.bb
> > +++ b/recipes-extended/images/xen-image-minimal.bb
> > @@ -31,7 +31,7 @@ XEN_PCIBACK_MODULE_x86-64 = "kernel-module-xen-pciback"
> >
> > LICENSE = "MIT"
> >
> > -inherit core-image
> > +inherit core-image qemuboot-xen-defaults qemuboot-xen-dtb
> >
> > do_check_xen_state() {
> > if [ "${@bb.utils.contains('DISTRO_FEATURES', 'xen', ' yes', 'no',
> > d)}" = "no" ]; then
> > @@ -74,10 +74,12 @@ build_syslinux_cfg () {
> >
> > # Enable runqemu. eg: runqemu xen-image-minimal nographic slirp
> > WKS_FILE_x86-64 = "directdisk-xen.wks"
> > -QB_MEM ?= "-m 400"
> > +QB_MEM ?= "-m 512"
>
> Since you are bumping the default to 512 here. Do we even need that
> qemuarm64 specific override in the bbclass ?
ack, I don't think we do.
I think the following works OK: put this into the qemuboot-xen-defaults.bbclass:
+# Increase the default qemu memory allocation to allow for the hypervisor.
+# Use a weak assignment to allow for change of default and override elsewhere.
+QB_MEM_VALUE ??= "512"
+QB_MEM = "-m ${QB_MEM_VALUE}"
and then setting QB_MEM at all can be dropped from the
xen-image-minimal recipe (since the default is now fine).
If an image does need to update the memory setting, it can set QB_MEM_VALUE
which has a weak default so is easily overridden. It's also
MACHINE-agnostic, which is preferable.
Does that sound OK?
Christopher
>
> Bruce
>
> > QB_DEFAULT_KERNEL ?= "none"
> > QB_DEFAULT_FSTYPE ?= "wic"
> > +QB_DEFAULT_FSTYPE_qemux86-64 = "wic"
> > QB_FSINFO ?= "wic:kernel-in-fs"
> > +QB_SERIAL_OPT = "-serial mon:stdio"
> > # qemux86-64 machine does not include 'wic' in IMAGE_FSTYPES, which is
> > needed
> > # to boot this image, so add it here:
> > IMAGE_FSTYPES_qemux86-64 += "wic"
> > --
> > 2.25.1
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6678):
https://lists.yoctoproject.org/g/meta-virtualization/message/6678
Mute This Topic: https://lists.yoctoproject.org/mt/84061644/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
