Re: [RFC PATCH 06/11] tests/avocado: Add FreeBSD distro boot tests for ppc
On Tue, Oct 10, 2023 at 8:23 PM Nicholas Piggin wrote: > On Wed Oct 11, 2023 at 7:55 AM AEST, Warner Losh wrote: > > On Tue, Oct 10, 2023 at 1:53 AM Nicholas Piggin > wrote: > > > > > FreeBSD project provides qcow2 images that work well for testing QEMU. > > > Add pseries tests for HPT and Radix, KVM and TCG. > > > > > > Other architectures could be added so this does not get a ppc_ prefix > > > but is instead named similarly to boot_linux. > > > > > > Cc: Warner Losh > > > Signed-off-by: Nicholas Piggin > > > > > > CC'ing Warner to check if it's okay for us to use these images and > > > any comments or suggestions. avocado tests have many Linux boots so > we'd > > > do much better to expand test coverage by adding some other systems. > > > > > > > I like this I'm a little worried at the exact hash encoded in it, but > > since there's a checksum > > to match, it's OK I guess. It will give this code a shelf-life of months, > > IIRC our retention policy. > > The oldest 15.0 CURRENT image on there is May 1st, so ~6 months? That's > not too bad. There are some release qcow2 images as well which sound > like they're maintained longer-term: > > https://download.freebsd.org/releases/VM-IMAGES/ > > No builds for powerpc, but those might be preferable for other targets. > > Another option for powerpc might be to use a release .iso. It's much > nicer to have a qcow image already installed though. I'll tinker with > it a bit more, but may stick with the snapshot for now. > I'll try to track that down. It may just be an oversight since powerpc64le is new. Warner > Thanks, > Nick >
Re: [RFC PATCH 06/11] tests/avocado: Add FreeBSD distro boot tests for ppc
On Tue, Oct 10, 2023 at 6:36 PM Nicholas Piggin wrote:
> On Wed Oct 11, 2023 at 7:55 AM AEST, Warner Losh wrote:
> > On Tue, Oct 10, 2023 at 1:53 AM Nicholas Piggin
> wrote:
> >
> > > FreeBSD project provides qcow2 images that work well for testing QEMU.
> > > Add pseries tests for HPT and Radix, KVM and TCG.
> > >
> > > Other architectures could be added so this does not get a ppc_ prefix
> > > but is instead named similarly to boot_linux.
> > >
> > > Cc: Warner Losh
> > > Signed-off-by: Nicholas Piggin
> > >
> > > CC'ing Warner to check if it's okay for us to use these images and
> > > any comments or suggestions. avocado tests have many Linux boots so
> we'd
> > > do much better to expand test coverage by adding some other systems.
> > >
> >
> > I like this I'm a little worried at the exact hash encoded in it, but
> > since there's a checksum
> > to match, it's OK I guess. It will give this code a shelf-life of months,
> > IIRC our retention policy.
>
> Oh I didn't realise, I saw some 2021 dates in the directory listing but
> looks
> like they're not for the artifacts themselves.
>
> I don't suppose you know if there are any long-term artifacts kept
> around, or someone who I could ask?
>
> The downside of using short term images is that it can be harder to
> reproduce reports from others, bisect, run manual testing, etc. I think
> these would still be useful, so long as they get updated regularly.
>
Yes. We're in kind of a weird zone. powerpc64le is a new architecture, so
hasn't had artifacts for long. 14.0 is in progress, but not done yet, so
there's
no 'long term stable' version to use yet.
I don't know what our current retention policy is, hence my caution.
> >
> > Other than that, I think this is good. Not familiar enough with Avocado
> to
> > understand
> > skipping for gitlab CI, but given the extreme crunch on minutes, I think
> > that's OK.
>
> Yeah I'm not sure what the situation there is, I didn't want to add new
> tests of any significant weight yet. We could always flip it on later if
> people want it.
>
That makes sense.
> >
> > Other than one nit below which is fine if it is intentionally left behind
> > (or removed):
> >
> > Reviewed-by: Warner Losh
> >
> > Please don't hesitate to reach out to me if this is failing. I'll act as
> a
> > backstop to get
> > it to the right people.
>
> Thanks Warner.
>
You bet. I'll give a heads up once we have 14.0 out so we can switch to
a more stable artifact.
Warner
> >
> > Warner
> >
> >
> [snip]
>
> > > +def run_pseries_test(self, force_HPT=False):
> > > +# We need zstd for all the tuxrun tests
> > > +# See
> https://github.com/avocado-framework/avocado/issues/5609
> > > +zstd = find_command('zstd', False)
> > > +if zstd is False:
> > > +self.cancel('Could not find "zstd", which is required to '
> > > +'decompress rootfs')
> > > +self.zstd = zstd
> > > +
> > > +drive_url = ('
> > >
> https://artifact.ci.freebsd.org/snapshot/15.0-CURRENT/a2440348eed75bb7682579af0905b652747fd016/powerpc/powerpc64le/disk.qcow2.zst
> > > ')
> > > +drive_hash = '8ab11a05ccab3d44215fd4667a70454ed10a203f'
> > > +drive_path_zstd = self.fetch_asset(drive_url,
> > > asset_hash=drive_hash)
> > > +drive_path = os.path.join(self.workdir, 'disk.qcow2')
> > > +# archive.zstd_uncompress(drive_path_zstd, drive_path)
> > >
> >
> > Why is this commented out? It looks like a leftover maybe?
> >
>
> Ah yes, avocado recently got zstd_uncompress but it seems not
> available for QEMU yet so we have to do it by hand. I'll remove.
>
> Thanks,
> Nick
>
Re: [RFC PATCH 06/11] tests/avocado: Add FreeBSD distro boot tests for ppc
On Wed Oct 11, 2023 at 7:55 AM AEST, Warner Losh wrote: > On Tue, Oct 10, 2023 at 1:53 AM Nicholas Piggin wrote: > > > FreeBSD project provides qcow2 images that work well for testing QEMU. > > Add pseries tests for HPT and Radix, KVM and TCG. > > > > Other architectures could be added so this does not get a ppc_ prefix > > but is instead named similarly to boot_linux. > > > > Cc: Warner Losh > > Signed-off-by: Nicholas Piggin > > > > CC'ing Warner to check if it's okay for us to use these images and > > any comments or suggestions. avocado tests have many Linux boots so we'd > > do much better to expand test coverage by adding some other systems. > > > > I like this I'm a little worried at the exact hash encoded in it, but > since there's a checksum > to match, it's OK I guess. It will give this code a shelf-life of months, > IIRC our retention policy. The oldest 15.0 CURRENT image on there is May 1st, so ~6 months? That's not too bad. There are some release qcow2 images as well which sound like they're maintained longer-term: https://download.freebsd.org/releases/VM-IMAGES/ No builds for powerpc, but those might be preferable for other targets. Another option for powerpc might be to use a release .iso. It's much nicer to have a qcow image already installed though. I'll tinker with it a bit more, but may stick with the snapshot for now. Thanks, Nick
Re: [RFC PATCH 06/11] tests/avocado: Add FreeBSD distro boot tests for ppc
On Wed Oct 11, 2023 at 7:55 AM AEST, Warner Losh wrote:
> On Tue, Oct 10, 2023 at 1:53 AM Nicholas Piggin wrote:
>
> > FreeBSD project provides qcow2 images that work well for testing QEMU.
> > Add pseries tests for HPT and Radix, KVM and TCG.
> >
> > Other architectures could be added so this does not get a ppc_ prefix
> > but is instead named similarly to boot_linux.
> >
> > Cc: Warner Losh
> > Signed-off-by: Nicholas Piggin
> >
> > CC'ing Warner to check if it's okay for us to use these images and
> > any comments or suggestions. avocado tests have many Linux boots so we'd
> > do much better to expand test coverage by adding some other systems.
> >
>
> I like this I'm a little worried at the exact hash encoded in it, but
> since there's a checksum
> to match, it's OK I guess. It will give this code a shelf-life of months,
> IIRC our retention policy.
Oh I didn't realise, I saw some 2021 dates in the directory listing but looks
like they're not for the artifacts themselves.
I don't suppose you know if there are any long-term artifacts kept
around, or someone who I could ask?
The downside of using short term images is that it can be harder to
reproduce reports from others, bisect, run manual testing, etc. I think
these would still be useful, so long as they get updated regularly.
>
> Other than that, I think this is good. Not familiar enough with Avocado to
> understand
> skipping for gitlab CI, but given the extreme crunch on minutes, I think
> that's OK.
Yeah I'm not sure what the situation there is, I didn't want to add new
tests of any significant weight yet. We could always flip it on later if
people want it.
>
> Other than one nit below which is fine if it is intentionally left behind
> (or removed):
>
> Reviewed-by: Warner Losh
>
> Please don't hesitate to reach out to me if this is failing. I'll act as a
> backstop to get
> it to the right people.
Thanks Warner.
>
> Warner
>
>
[snip]
> > +def run_pseries_test(self, force_HPT=False):
> > +# We need zstd for all the tuxrun tests
> > +# See https://github.com/avocado-framework/avocado/issues/5609
> > +zstd = find_command('zstd', False)
> > +if zstd is False:
> > +self.cancel('Could not find "zstd", which is required to '
> > +'decompress rootfs')
> > +self.zstd = zstd
> > +
> > +drive_url = ('
> > https://artifact.ci.freebsd.org/snapshot/15.0-CURRENT/a2440348eed75bb7682579af0905b652747fd016/powerpc/powerpc64le/disk.qcow2.zst
> > ')
> > +drive_hash = '8ab11a05ccab3d44215fd4667a70454ed10a203f'
> > +drive_path_zstd = self.fetch_asset(drive_url,
> > asset_hash=drive_hash)
> > +drive_path = os.path.join(self.workdir, 'disk.qcow2')
> > +# archive.zstd_uncompress(drive_path_zstd, drive_path)
> >
>
> Why is this commented out? It looks like a leftover maybe?
>
Ah yes, avocado recently got zstd_uncompress but it seems not
available for QEMU yet so we have to do it by hand. I'll remove.
Thanks,
Nick
Re: [RFC PATCH 06/11] tests/avocado: Add FreeBSD distro boot tests for ppc
On Tue, Oct 10, 2023 at 1:53 AM Nicholas Piggin wrote:
> FreeBSD project provides qcow2 images that work well for testing QEMU.
> Add pseries tests for HPT and Radix, KVM and TCG.
>
> Other architectures could be added so this does not get a ppc_ prefix
> but is instead named similarly to boot_linux.
>
> Cc: Warner Losh
> Signed-off-by: Nicholas Piggin
>
> CC'ing Warner to check if it's okay for us to use these images and
> any comments or suggestions. avocado tests have many Linux boots so we'd
> do much better to expand test coverage by adding some other systems.
>
I like this I'm a little worried at the exact hash encoded in it, but
since there's a checksum
to match, it's OK I guess. It will give this code a shelf-life of months,
IIRC our retention policy.
Other than that, I think this is good. Not familiar enough with Avocado to
understand
skipping for gitlab CI, but given the extreme crunch on minutes, I think
that's OK.
Other than one nit below which is fine if it is intentionally left behind
(or removed):
Reviewed-by: Warner Losh
Please don't hesitate to reach out to me if this is failing. I'll act as a
backstop to get
it to the right people.
Warner
> ---
> tests/avocado/boot_freebsd.py | 109 ++
> 1 file changed, 109 insertions(+)
> create mode 100644 tests/avocado/boot_freebsd.py
>
> diff --git a/tests/avocado/boot_freebsd.py b/tests/avocado/boot_freebsd.py
> new file mode 100644
> index 00..9a499a28ad
> --- /dev/null
> +++ b/tests/avocado/boot_freebsd.py
> @@ -0,0 +1,109 @@
> +# Functional tests that boot FreeBSD in various configurations
> +#
> +# Copyright (c) 2023 IBM Corporation
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado import skipUnless
> +from avocado import skipIf
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import wait_for_console_pattern
> +from avocado_qemu import exec_command
> +from avocado.utils import archive
> +from avocado.utils import process
> +from avocado.utils.path import find_command
> +
> +class BootFreeBSDPPC64(QemuSystemTest):
> +"""
> +:avocado: tags=arch:ppc64
> +"""
> +
> +timeout = 360
> +
> +def run_pseries_test(self, force_HPT=False):
> +# We need zstd for all the tuxrun tests
> +# See https://github.com/avocado-framework/avocado/issues/5609
> +zstd = find_command('zstd', False)
> +if zstd is False:
> +self.cancel('Could not find "zstd", which is required to '
> +'decompress rootfs')
> +self.zstd = zstd
> +
> +drive_url = ('
> https://artifact.ci.freebsd.org/snapshot/15.0-CURRENT/a2440348eed75bb7682579af0905b652747fd016/powerpc/powerpc64le/disk.qcow2.zst
> ')
> +drive_hash = '8ab11a05ccab3d44215fd4667a70454ed10a203f'
> +drive_path_zstd = self.fetch_asset(drive_url,
> asset_hash=drive_hash)
> +drive_path = os.path.join(self.workdir, 'disk.qcow2')
> +# archive.zstd_uncompress(drive_path_zstd, drive_path)
>
Why is this commented out? It looks like a leftover maybe?
> +
> +cmd = f"{self.zstd} -d {drive_path_zstd} -o {drive_path}"
> +process.run(cmd)
> +
> +drive = f"file={drive_path},format=qcow2,if=virtio"
> +
> +self.vm.set_console()
> +if force_HPT:
> +self.vm.add_args('-m', '4g')
> +else:
> +self.vm.add_args('-m', '1g')
> +self.vm.add_args('-smp', '4')
> +self.vm.add_args('-drive', drive)
> +self.vm.add_args('-net', 'nic,model=virtio')
> +self.vm.launch()
> +
> +wait_for_console_pattern(self, 'Hit [Enter] to boot immediately,
> or any other key for command prompt.')
> +if force_HPT:
> +exec_command(self, 'x')
> +wait_for_console_pattern(self, 'OK')
> +exec_command(self, 'set radix_mmu=0')
> +exec_command(self, 'boot')
> +wait_for_console_pattern(self, 'cas: selected hash MMU',
> 'panic:')
> +else:
> +exec_command(self, '')
> +wait_for_console_pattern(self, 'cas: selected radix MMU')
> +
> +wait_for_console_pattern(self, 'FreeBSD 15.0-CURRENT #0 a244034:
> Mon Sep 25 02:05:22 UTC 2023', 'panic:')
> +wait_for_console_pattern(self, 'FreeBSD/SMP: Multiprocessor
> System Detected: 4 CPUs')
> +wait_for_console_pattern(self, 'FreeBSD/powerpc (Amnesiac)
> (ttyu0)', 'panic:')
> +
> +@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> +def test_pseries_tcg(self):
> +"""
> +:avocado: tags=arch:ppc64
> +:avocado: tags=machine:pseries
> +:avocado: tags=accel:tcg
> +"""
> +self.require_accelerator("tcg")
> +self.run_pseries_test()
> +
> +@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> +def test_pseries_hpt_tcg(self):
> +
