Re: [PATCH] MAINTAINERS: Remove me as a reviewer for the build and test/avocado

[Willian Rampazzo]

On Tue, Nov 23, 2021 at 7:37 AM Philippe Mathieu-Daudé
 wrote:
>
> On 11/22/21 20:11, Willian Rampazzo wrote:
> > Remove me as a reviewer for the Build and test automation and the
> > Integration Testing with the Avocado Framework and add Beraldo
> > Leal.
>
> Thank you Willian for your QEMU contributions and reviews,
> they helped the community in having a better idea of what
> integration testing is and is not, and how to use the
> Avocado framework, having a better idea what Avocado can
> not do for QEMU needs. Also thank you for improving the
> documentation :)

My pleasure!

Thanks for taking the time to help me understand the basics of QEMU.

>
> Beraldo, thank you for replacing Willian as an upstream
> Avocado <-> QEMU projects bridge :)
>
> Reviewed-by: Philippe Mathieu-Daudé 
>
> > Signed-off-by: Willian Rampazzo 
> > ---
> >  MAINTAINERS | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index d3879aa3c1..8f5156bfa7 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -3469,7 +3469,7 @@ M: Alex Bennée 
> >  M: Philippe Mathieu-Daudé 
> >  M: Thomas Huth 
> >  R: Wainer dos Santos Moschetta 
> > -R: Willian Rampazzo 
> > +R: Beraldo Leal 
> >  S: Maintained
> >  F: .github/lockdown.yml
> >  F: .gitlab-ci.yml
> > @@ -3507,7 +3507,7 @@ W: https://trello.com/b/6Qi1pxVn/avocado-qemu
> >  R: Cleber Rosa 
> >  R: Philippe Mathieu-Daudé 
> >  R: Wainer dos Santos Moschetta 
> > -R: Willian Rampazzo 
> > +R: Beraldo Leal 
> >  S: Odd Fixes
> >  F: tests/avocado/
> >
> >
>

Re: [NOTFORMERGE PATCH 5/5] tests/avocado: Test NetBSD 9.2 on the Jazz Magnum machine

[Willian Rampazzo]

the
paths.

The documentation about the data folder is here:
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html?highlight=data#accessing-test-data-files

> +nvram_xz_path = self.fetch_asset('file://' + nvram_xz_path,
> + asset_hash=nvram_xz_hash,
> + algorithm='sha1')
> +mac = '00:00:00:02:03:04'
> +
> +with lzma.open(nvram_xz_path, 'rb') as f_in:
> +with open(nvram_path, 'wb') as f_out:
> +shutil.copyfileobj(f_in, f_out)
> +f_out.seek(nvram_size)
> +f_out.write(b'\0')
> +
> +self.vm.set_console()
> +self.vm.add_args('-bios', ntprom_path,
> + '-drive', 
> 'if=scsi,unit=2,media=cdrom,format=raw,file='
> +   + drive_path,
> + '-global', 'ds1225y.filename=' + nvram_path,
> + '-nic', 'user,model=dp83932,mac=' + mac)
> +self.vm.launch()
> +
> +console_pattern = 'ARC Multiboot Version 174 (SGI Version 2.6)'
> +wait_for_console_pattern(self, console_pattern)
> +
> +wait_for_console_pattern(self, 'Use the arrow keys to select.')
> +
> +# Press cursor control 'Down' to select the "Run a program" menu
> +exec_command(self, '\x1b[B')
> +
> +program = 'scsi(0)cdrom(2)fdisk(0)boot scsi(0)cdrom(2)fdisk(0)netbsd'
> +exec_command(self, program)
> +wait_for_console_pattern(self, 'NetBSD/arc Bootstrap, Revision 1.1')
> +
> +# Terminal type? [vt100]
> +console_pattern = 'erase ^H, werase ^W, kill ^U, intr ^C, status ^T'
> +wait_for_console_pattern(self, console_pattern)
> +
> +# (I)nstall, (S)hell or (H)alt
> +exec_command_and_wait_for_pattern(self, '', 'Erase is backspace.')
> +exec_command(self, 'S')
> +interrupt_interactive_console_until_pattern(self, '#')
> +
> +exec_command_and_wait_for_pattern(self, 'ifconfig', 'address: ' + 
> mac)
> +interrupt_interactive_console_until_pattern(self, '#')
> +
> +exec_command(self, 'ifconfig sn0 10.0.2.3/24')
> +interrupt_interactive_console_until_pattern(self, '#')
> +
> +exec_command_and_wait_for_pattern(self, 'ping -c 3 10.0.2.2',
> +'3 packets transmitted, 3 packets received, 0.0% packet 
> loss')
> +
> +exec_command_and_wait_for_pattern(self, 'shutdown -r now',
> +  'rebooting...')

Nice!

Except for the nvram.bin.xz,

Reviewed-by: Willian Rampazzo 

[PATCH] MAINTAINERS: Remove me as a reviewer for the build and test/avocado

[Willian Rampazzo]

Remove me as a reviewer for the Build and test automation and the
Integration Testing with the Avocado Framework and add Beraldo
Leal.

Signed-off-by: Willian Rampazzo 
---
 MAINTAINERS | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index d3879aa3c1..8f5156bfa7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3469,7 +3469,7 @@ M: Alex Bennée 
 M: Philippe Mathieu-Daudé 
 M: Thomas Huth 
 R: Wainer dos Santos Moschetta 
-R: Willian Rampazzo 
+R: Beraldo Leal 
 S: Maintained
 F: .github/lockdown.yml
 F: .gitlab-ci.yml
@@ -3507,7 +3507,7 @@ W: https://trello.com/b/6Qi1pxVn/avocado-qemu
 R: Cleber Rosa 
 R: Philippe Mathieu-Daudé 
 R: Wainer dos Santos Moschetta 
-R: Willian Rampazzo 
+R: Beraldo Leal 
 S: Odd Fixes
 F: tests/avocado/
 
-- 
2.33.1

Re: [PATCH 5/7] python/machine: handle "fast" QEMU terminations

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:49 PM John Snow  wrote:
>
> In the case that the QEMU process actually launches -- but then dies so
> quickly that we can't establish a QMP connection to it -- QEMUMachine
> currently calls _post_shutdown() assuming that it never launched the VM
> process.
>
> This isn't true, though: it "merely" may have failed to establish a QMP
> connection and the process is in the middle of its own exit path.
>
> If we don't wait for the subprocess, the caller may get a bogus `None`
> return for .exitcode(). This behavior was observed from
> device-crash-test; after the switch to Async QMP, the timings were
> changed such that it was now seemingly possible to witness the failure
> of "vm.launch()" *prior* to the exitcode becoming available.
>
> The semantic of the `_launched` property is changed in this
> patch. Instead of representing the condition "launch() executed
> successfully", it will now represent "has forked a child process
> successfully". This way, wait() when called in the exit path won't
> become a no-op.
> Signed-off-by: John Snow 

Nitpick: double signed-off-by

>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 19 ---
>  1 file changed, 12 insertions(+), 7 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 7/7] python/aqmp: fix send_fd_scm for python 3.6.x

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:53 PM John Snow  wrote:
>
> 3.6 doesn't play keepaway with the socket object, so we don't need to go
> fishing for it on this version. In fact, so long as 'sendmsg' is still
> available, it's probably preferable to just use that method and only go
> fishing for forbidden details when we absolutely have to.
>
> Reported-by: Thomas Huth 
> Signed-off-by: John Snow 
> ---
>  python/qemu/aqmp/qmp_client.py | 9 ++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH] tests/avocado: ppc: Add smoke tests for MPC7400 and MPC7450 families

[Willian Rampazzo]

One more comment, sorry.

On Fri, Nov 19, 2021 at 10:54 AM Fabiano Rosas  wrote:
>
> These tests ensure that our emulation for these cpus is not completely
> broken and we can at least run OpenBIOS on them.
>
> $ make check-avocado AVOCADO_TESTS=../tests/avocado/ppc_74xx.py
>
> Signed-off-by: Fabiano Rosas 
>
> ---
> Note that the 7450s are currently broken, depending on the fix for
> https://gitlab.com/qemu-project/qemu/-/issues/86 that is being
> discussed in the thread:
>
> https://lists.nongnu.org/archive/html/qemu-ppc/2021-11/msg00289.html
> ---
>  tests/avocado/ppc_74xx.py | 134 ++
>  1 file changed, 134 insertions(+)
>  create mode 100644 tests/avocado/ppc_74xx.py
>
> diff --git a/tests/avocado/ppc_74xx.py b/tests/avocado/ppc_74xx.py
> new file mode 100644
> index 00..72b99d39c0
> --- /dev/null
> +++ b/tests/avocado/ppc_74xx.py
> @@ -0,0 +1,134 @@
> +# Smoke tests for 74xx cpus (aka G4).
> +#
> +# Copyright (c) 2021, IBM Corp.
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +from avocado import skip
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import wait_for_console_pattern
> +
> +class ppc74xxCpu(QemuSystemTest):
> +
> +timeout = 5
> +
> +def test_ppc_7400(self):
> +"""
> +:avocado: tags=arch:ppc

All your tests are using the same "arch" tag, so you can define it in
the class and avoid duplication. This is more aesthetic, so it is up
to you.

Re: [PATCH] tests/avocado: ppc: Add smoke tests for MPC7400 and MPC7450 families

[Willian Rampazzo]

On Fri, Nov 19, 2021 at 10:54 AM Fabiano Rosas  wrote:
>
> These tests ensure that our emulation for these cpus is not completely
> broken and we can at least run OpenBIOS on them.
>
> $ make check-avocado AVOCADO_TESTS=../tests/avocado/ppc_74xx.py
>
> Signed-off-by: Fabiano Rosas 
>
> ---
> Note that the 7450s are currently broken, depending on the fix for
> https://gitlab.com/qemu-project/qemu/-/issues/86 that is being
> discussed in the thread:
>
> https://lists.nongnu.org/archive/html/qemu-ppc/2021-11/msg00289.html
> ---
>  tests/avocado/ppc_74xx.py | 134 ++
>  1 file changed, 134 insertions(+)
>  create mode 100644 tests/avocado/ppc_74xx.py
>
> diff --git a/tests/avocado/ppc_74xx.py b/tests/avocado/ppc_74xx.py
> new file mode 100644
> index 00..72b99d39c0
> --- /dev/null
> +++ b/tests/avocado/ppc_74xx.py
> @@ -0,0 +1,134 @@
> +# Smoke tests for 74xx cpus (aka G4).
> +#
> +# Copyright (c) 2021, IBM Corp.
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +from avocado import skip

You are not using this import in the file. No need to import it here.

Removing the unused import, the code looks good to me, so:

Reviewed-by: Willian Rampazzo 

Re: [PATCH 4/7] python/machine: move more variable initializations to _pre_launch

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:51 PM John Snow  wrote:
>
> No need to clear them only to set them later.
>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 16 
>  1 file changed, 8 insertions(+), 8 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 3/7] python/machine: add instance disambiguator to default nickname

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:49 PM John Snow  wrote:
>
> If you create two instances of QEMUMachine(), they'll both create the
> same nickname by default -- which is not that helpful.
>
> Luckily, they'll both create unique temporary directories ... but due to
> user configuration, they may share logging and sockfile directories,
> meaning two instances can collide. The Python logging will also be quite
> confusing, with no differentiation between the two instances.
>
> Add an instance disambiguator (The memory address of the instance) to
> the default nickname to foolproof this in all cases.
>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 2/7] python/machine: remove _remove_monitor_sockfile property

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:49 PM John Snow  wrote:
>
> It doesn't matter if it was the user or the class itself that specified
> where the sockfile should be created; the fact is that if we are using a
> sockfile here, we created it and we can clean it up.
>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 5 +
>  1 file changed, 1 insertion(+), 4 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 1/7] python/machine: add @sock_dir property

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 5:50 PM John Snow  wrote:
>
> Analogous to temp_dir and log_dir, add a sock_dir property that defaults
> to @temp_dir -- instead of base_temp_dir -- when the user hasn't
> overridden the sock dir value in the initializer.
>
> This gives us a much more unique directory to put sockfiles in by default.
>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 17 +
>  1 file changed, 13 insertions(+), 4 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [qemu-web PATCH] remove deployment phase from CI

[Willian Rampazzo]

On Thu, Nov 18, 2021 at 4:50 AM Paolo Bonzini  wrote:
>
> qemu.org is now served via a reverse proxy from qemu-project.gitlab.io; it 
> does
> not need anymore the rsync step to the QEMU project's shell server.
> Remove it from the CI.
>
> Signed-off-by: Paolo Bonzini 
> ---
>  .gitlab-ci.yml | 24 
>  1 file changed, 24 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes

[Willian Rampazzo]

On Tue, Nov 16, 2021 at 1:33 PM Thomas Huth  wrote:
>
> The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
> be scheduled, so while the build test itself finishes within 60 minutes,
> the total run time of the jobs can be longer due to this waiting time.
> Thus let's increase the timeout on the gitlab side a little bit, so
> that these jobs are not marked as failing just because of the delay.
>
> Signed-off-by: Thomas Huth 
> ---
>  .gitlab-ci.d/cirrus.yml | 1 +
>  1 file changed, 1 insertion(+)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 6/6] gitlab-ci: Split custom-runners.yml in one file per runner

[Willian Rampazzo]

On Mon, Nov 15, 2021 at 11:29 AM Alex Bennée  wrote:
>
> From: Philippe Mathieu-Daudé 
>
> To ease maintenance, add the custom-runners/ directory and
> split custom-runners.yml in 3 files, all included by the
> current custom-runners.yml:
>  - ubuntu-18.04-s390x.yml
>  - ubuntu-20.04-aarch64.yml
>  - centos-stream-8-x86_64.yml
>
> Signed-off-by: Philippe Mathieu-Daudé 
> Signed-off-by: Alex Bennée 
> Message-Id: <2025095608.2436223-1-phi...@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml   | 268 +-
>  .../custom-runners/centos-stream-8-x86_64.yml |  28 ++
>  .../custom-runners/ubuntu-18.04-s390x.yml | 118 
>  .../custom-runners/ubuntu-20.04-aarch64.yml   | 118 
>  4 files changed, 268 insertions(+), 264 deletions(-)
>  create mode 100644 .gitlab-ci.d/custom-runners/centos-stream-8-x86_64.yml
>  create mode 100644 .gitlab-ci.d/custom-runners/ubuntu-18.04-s390x.yml
>  create mode 100644 .gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 3/6] tests/vm: don't build using TCG by default

[Willian Rampazzo]

On Mon, Nov 15, 2021 at 11:29 AM Alex Bennée  wrote:
>
> While it is useful to run these images using TCG their performance
> will not be anything like the native guests. Don't do it by default.
>
> Fixes: https://gitlab.com/qemu-project/qemu/-/issues/393
> Signed-off-by: Alex Bennée 
> ---
>  tests/vm/Makefile.include | 17 +
>  1 file changed, 13 insertions(+), 4 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 2/6] tests/vm: sort the special variable list

[Willian Rampazzo]

On Mon, Nov 15, 2021 at 11:29 AM Alex Bennée  wrote:
>
> Making the list alphabetical makes it easier to find the config option
> you are looking for.
>
> Signed-off-by: Alex Bennée 
> ---
>  tests/vm/Makefile.include | 12 ++--
>  1 file changed, 6 insertions(+), 6 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH] gitlab: skip cirrus jobs on master and stable branches

[Willian Rampazzo]

On Tue, Nov 16, 2021 at 8:28 AM Daniel P. Berrangé  wrote:
>
> On the primary QEMU repository we want the CI jobs to run on the staging
> branch as a gating CI test.
>
> Cirrus CI has very limited job concurrency, so if there are too many
> jobs triggered they'll queue up and hit the GitLab CI job timeout before
> they complete on Cirrus.
>
> If we let Cirrus jobs run again on the master branch immediately after
> merging from staging, that just increases the chances jobs will get
> queued and subsequently timeout.
>
> The same applies for merges to the stable branches.
>
> User forks meanwhile should be allowed to run Cirrus CI jobs freely.
>
> Signed-off-by: Daniel P. Berrangé 
> ---
>  .gitlab-ci.d/cirrus.yml | 3 +++
>  1 file changed, 3 insertions(+)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v2 1/1] Jobs based on custom runners: add CentOS Stream 8

[Willian Rampazzo]

On Thu, Nov 11, 2021 at 1:06 PM Cleber Rosa  wrote:
>
> This introduces three different parts of a job designed to run
> on a custom runner managed by Red Hat.  The goals include:
>
>   a) propose a model for other organizations that want to onboard
>  their own runners, with their specific platforms, build
>  configuration and tests.
>
>   b) bring awareness to the differences between upstream QEMU and the
>  version available under CentOS Stream, which is "A preview of
>  upcoming Red Hat Enterprise Linux minor and major releases".
>
>   c) because of b), it should be easier to identify and reduce the gap
>  between Red Hat's downstream and upstream QEMU.
>
> The components of this custom job are:
>
>   I) OS build environment setup code:
>
>  - additions to the existing "build-environment.yml" playbook
>that can be used to set up CentOS/EL 8 systems.
>
>  - a CentOS Stream 8 specific "build-environment.yml" playbook
>that adds to the generic one.
>
>  II) QEMU build configuration: a script that will produce binaries with
>  features as similar as possible to the ones built and packaged on
>  CentOS stream 8.
>
> III) Scripts that define the minimum amount of testing that the
>  binaries built with the given configuration (point II) under the
>  given OS build environment (point I) should be subjected to.
>
>  IV) Job definition: GitLab CI jobs that will dispatch the build/test
>  jobs (see points #II and #III) to the machine specifically
>  configured according to #I.
>
> Signed-off-by: Cleber Rosa 
> ---
>  .gitlab-ci.d/custom-runners.yml   |  29 +++
>  docs/devel/ci-jobs.rst.inc|   7 +
>  .../org.centos/stream/8/build-environment.yml |  51 +
>  .../ci/org.centos/stream/8/x86_64/configure   | 208 ++
>  .../org.centos/stream/8/x86_64/test-avocado   |  70 ++
>  scripts/ci/org.centos/stream/README   |  17 ++
>  scripts/ci/setup/build-environment.yml|  38 
>  7 files changed, 420 insertions(+)
>  create mode 100644 scripts/ci/org.centos/stream/8/build-environment.yml
>  create mode 100755 scripts/ci/org.centos/stream/8/x86_64/configure
>  create mode 100755 scripts/ci/org.centos/stream/8/x86_64/test-avocado
>  create mode 100644 scripts/ci/org.centos/stream/README
>

Maybe it is too late, but just for the records:

Reviewed-by: Willian Rampazzo 
Tested-by: Willian Rampazzo 

CI job on a custom VM runner:
https://gitlab.com/willianrampazzo/qemu/-/jobs/1778451942

Re: [PATCH] tests/avocado: Remove p7zip binary availability check

[Willian Rampazzo]

On Sat, Nov 6, 2021 at 3:14 PM Philippe Mathieu-Daudé  wrote:
>
> The single use of the 7z binary has been removed in commit a30e114f3
> ("tests/acceptance: remove Armbian 19.11.3 test for orangepi-pc"),
> we don't need to check for this binary availability anymore.
>
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
> Based-on: <20211105155354.154864-1-willi...@redhat.com>
> ---
>  tests/avocado/boot_linux_console.py | 7 ---
>  1 file changed, 7 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v2 1/2] tests/acceptance: introduce new check-avocado tartget

[Willian Rampazzo]

On Mon, Nov 8, 2021 at 6:49 AM Philippe Mathieu-Daudé  wrote:
>
> On 11/8/21 10:24, Daniel P. Berrangé wrote:
> > On Mon, Nov 08, 2021 at 08:59:51AM +0100, Thomas Huth wrote:
> >> On 05/11/2021 16.53, Willian Rampazzo wrote:
> >>> This introduces a new `make` target, `check-avocado`, and adds a
> >>> deprecation message about the `check-acceptance` target. This is
> >>> a preparation for renaming the `tests/acceptance` folder to
> >>>   `tests/avocado`.
> >>>
> >>> The plan is to remove the call to the `check-avocado` target one
> >>> or two months after the release and leave the warning to force
> >>> people to move to the new `check-avocado` target.
> >>>
> >>> Later, the `check-acceptance` target can be removed. The intent
> >>> is to avoid a direct impact during the current soft freeze.
> >>>
> >>> Suggested-by: Philippe Mathieu-Daudé 
> >>> Signed-off-by: Willian Rampazzo 
> >>> ---
> >>>   docs/about/deprecated.rst | 13 +
> >>>   tests/Makefile.include| 17 -
> >>>   2 files changed, 25 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
> >>> index 56f9ad15ab..7bf8da8325 100644
> >>> --- a/docs/about/deprecated.rst
> >>> +++ b/docs/about/deprecated.rst
> >>> @@ -410,3 +410,16 @@ nanoMIPS ISA
> >>>   The ``nanoMIPS`` ISA has never been upstreamed to any compiler 
> >>> toolchain.
> >>>   As it is hard to generate binaries for it, declare it deprecated.
> >>> +
> >>> +Testing
> >>> +---
> >>> +
> >>> +Renaming of the acceptance folder to avocado
> >>> +
> >>> +
> >>> +The ``tests/acceptance`` folder was never used to store acceptance tests
> >>> +in terms of software engineering. This naming can confuse developers
> >>> +adding tests using the Avocado Framework to this folder. The folder
> >>> +name change to ``tests/avocado`` also changed the ``make`` target from
> >>> +``check-acceptance`` to ``check-avocado``. In this case, the use of the
> >>> +``check-acceptance`` target is deprecated.
> >>
> >> Not sure whether we need  to document this in deprecated.rst, too, since
> >> we're normally only listing the things here that affect the users of the
> >> qemu binaries, not the people who want to recompile and run the tests...
> >> OTOH, I don't mind too much either if we list it here... Anybody else got 
> >> an
> >> opinion on this?
> >
> > Deprecations are only things for user facing changes in the apps.
>
> OK.
>
> > For build system changes we don't bother with any deprecation cycle.
> > Just make the change immediately and document it in the release notes.
>
> Understood.
>
> Willian, do you mind updating the release notes?
> https://wiki.qemu.org/ChangeLog/6.2#Testing_and_CI
>

Sure, I can do that, but I think I need to wait for the patch to be
merged, right?

[PATCH v2 0/2] tests/acceptance: rename tests acceptance to tests avocado

[Willian Rampazzo]

In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

GitLab pipeline with new naming: 
https://gitlab.com/willianrampazzo/qemu/-/pipelines/403056475

Changes from v1:
 - Split changes on Makefile leaving `check-acceptance` available and
   adding a deprecate warning message when it is used.
   (Suggested-by: Philippe Mathieu-Daudé)
 - Remove unrelated changes.

Signed-off-by: Willian Rampazzo 

Willian Rampazzo (2):
  tests/acceptance: introduce new check-avocado tartget
  tests/acceptance: rename tests acceptance to tests avocado

 .gitlab-ci.d/buildtest-template.yml   |  2 +-
 .gitlab-ci.d/buildtest.yml| 56 +--
 MAINTAINERS   | 52 -
 configure |  2 +-
 docs/about/deprecated.rst | 13 +
 docs/devel/build-system.rst   |  2 +-
 docs/devel/ci-definitions.rst.inc |  2 +-
 docs/devel/testing.rst| 49 
 docs/system/arm/orangepi.rst  |  8 +--
 python/qemu/machine/README.rst|  2 +-
 python/qemu/qmp/README.rst|  2 +-
 python/qemu/utils/README.rst  |  2 +-
 tests/Makefile.include| 21 ---
 tests/acceptance/README.rst   | 10 
 tests/avocado/README.rst  | 10 
 .../avocado_qemu/__init__.py  |  2 +-
 tests/{acceptance => avocado}/boot_linux.py   |  0
 .../boot_linux_console.py |  0
 tests/{acceptance => avocado}/boot_xen.py |  0
 tests/{acceptance => avocado}/cpu_queries.py  |  0
 .../empty_cpu_model.py|  0
 tests/{acceptance => avocado}/hotplug_cpu.py  |  0
 tests/{acceptance => avocado}/info_usernet.py |  0
 tests/{acceptance => avocado}/intel_iommu.py  |  0
 tests/{acceptance => avocado}/linux_initrd.py |  2 +-
 .../linux_ssh_mips_malta.py   |  0
 .../machine_arm_canona1100.py |  0
 .../machine_arm_integratorcp.py   |  0
 .../machine_arm_n8x0.py   |  0
 tests/{acceptance => avocado}/machine_avr6.py |  2 +-
 .../machine_m68k_nextcube.py  |  0
 .../machine_microblaze.py |  0
 .../machine_mips_fuloong2e.py |  0
 .../machine_mips_loongson3v.py|  0
 .../machine_mips_malta.py |  0
 .../machine_rx_gdbsim.py  |  0
 .../machine_s390_ccw_virtio.py|  0
 .../machine_sparc64_sun4u.py  |  0
 .../machine_sparc_leon3.py|  0
 tests/{acceptance => avocado}/migration.py|  0
 tests/{acceptance => avocado}/multiprocess.py |  0
 .../pc_cpu_hotplug_props.py   |  0
 tests/{acceptance => avocado}/ppc_405.py  |  0
 tests/{acceptance => avocado}/ppc_bamboo.py   |  0
 .../{acceptance => avocado}/ppc_mpc8544ds.py  |  0
 tests/{acceptance => avocado}/ppc_prep_40p.py |  0
 tests/{acceptance => avocado}/ppc_pseries.py  |  0
 .../ppc_virtex_ml507.py   |  0
 .../{acceptance => avocado}/replay_kernel.py  |  0
 tests/{acceptance => avocado}/replay_linux.py |  0
 .../reverse_debugging.py  |  0
 tests/{acceptance => avocado}/smmu.py |  0
 tests/{acceptance => avocado}/tcg_plugins.py  |  0
 .../tesseract_utils.py|  0
 tests/{acceptance => avocado}/version.py  |  0
 tests/{acceptance => avocado}/virtio-gpu.py   |  0
 .../virtio_check_params.py|  0
 .../{acceptance => avocado}/virtio_version.py |  0
 .../virtiofs_submounts.py |  0
 .../virtiofs_submounts.py.data/cleanup.sh |  0
 .../guest-cleanup.sh  |  0
 .../virtiofs_submounts.py.data/guest.sh   |  0
 .../virtiofs_submounts.py.data/host.sh|  0
 tests/{acceptance => avocado}/vnc.py  |  0
 .../x86_cpu_model_versions.py |  0
 65 files changed, 129 insertions(+), 110 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/avocado/README.rst
 rename tests/{acceptance => avocado}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => avocado}/boot_linux.py (100%)
 rename tests/{acceptance => avocado}/boot_linux_console.py (100%)
 rename tests/{acceptance => avocado}/boot_xen.py (100%)
 rename tests/{acceptance => avocado}/cpu_queries.py (100%)
 rename tests/{acceptance => avocado}/empty_cpu_model.py (100%)
 rename t

Re: [PATCH] docs/about/deprecated: Remove empty 'related binaries' section

[Willian Rampazzo]

On Fri, Nov 5, 2021 at 11:31 AM Philippe Mathieu-Daudé
 wrote:
>
> Commit 497a30dbb06 ("qemu-img: Require -F with -b backing image")
> removed the content of the "Related binaries" section but forgot
> to remove the section title. Since it is now empty, remove it too.
>
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  docs/about/deprecated.rst | 3 ---
>  1 file changed, 3 deletions(-)
>

Reviewed-by: Willian Rampazzo 

[PATCH v2 2/2] tests/acceptance: rename tests acceptance to tests avocado

[Willian Rampazzo]

In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

For the Orange Pi PC documentation:
Reviewed-by: Niek Linnenbank 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Signed-off-by: Willian Rampazzo 
---
 .gitlab-ci.d/buildtest-template.yml   |  2 +-
 .gitlab-ci.d/buildtest.yml| 56 +--
 MAINTAINERS   | 52 -
 configure |  2 +-
 docs/devel/build-system.rst   |  2 +-
 docs/devel/ci-definitions.rst.inc |  2 +-
 docs/devel/testing.rst| 49 
 docs/system/arm/orangepi.rst  |  8 +--
 python/qemu/machine/README.rst|  2 +-
 python/qemu/qmp/README.rst|  2 +-
 python/qemu/utils/README.rst  |  2 +-
 tests/Makefile.include|  4 +-
 tests/acceptance/README.rst   | 10 
 tests/avocado/README.rst  | 10 
 .../avocado_qemu/__init__.py  |  2 +-
 tests/{acceptance => avocado}/boot_linux.py   |  0
 .../boot_linux_console.py |  0
 tests/{acceptance => avocado}/boot_xen.py |  0
 tests/{acceptance => avocado}/cpu_queries.py  |  0
 .../empty_cpu_model.py|  0
 tests/{acceptance => avocado}/hotplug_cpu.py  |  0
 tests/{acceptance => avocado}/info_usernet.py |  0
 tests/{acceptance => avocado}/intel_iommu.py  |  0
 tests/{acceptance => avocado}/linux_initrd.py |  2 +-
 .../linux_ssh_mips_malta.py   |  0
 .../machine_arm_canona1100.py |  0
 .../machine_arm_integratorcp.py   |  0
 .../machine_arm_n8x0.py   |  0
 tests/{acceptance => avocado}/machine_avr6.py |  2 +-
 .../machine_m68k_nextcube.py  |  0
 .../machine_microblaze.py |  0
 .../machine_mips_fuloong2e.py |  0
 .../machine_mips_loongson3v.py|  0
 .../machine_mips_malta.py |  0
 .../machine_rx_gdbsim.py  |  0
 .../machine_s390_ccw_virtio.py|  0
 .../machine_sparc64_sun4u.py  |  0
 .../machine_sparc_leon3.py|  0
 tests/{acceptance => avocado}/migration.py|  0
 tests/{acceptance => avocado}/multiprocess.py |  0
 .../pc_cpu_hotplug_props.py   |  0
 tests/{acceptance => avocado}/ppc_405.py  |  0
 tests/{acceptance => avocado}/ppc_bamboo.py   |  0
 .../{acceptance => avocado}/ppc_mpc8544ds.py  |  0
 tests/{acceptance => avocado}/ppc_prep_40p.py |  0
 tests/{acceptance => avocado}/ppc_pseries.py  |  0
 .../ppc_virtex_ml507.py   |  0
 .../{acceptance => avocado}/replay_kernel.py  |  0
 tests/{acceptance => avocado}/replay_linux.py |  0
 .../reverse_debugging.py  |  0
 tests/{acceptance => avocado}/smmu.py |  0
 tests/{acceptance => avocado}/tcg_plugins.py  |  0
 .../tesseract_utils.py|  0
 tests/{acceptance => avocado}/version.py  |  0
 tests/{acceptance => avocado}/virtio-gpu.py   |  0
 .../virtio_check_params.py|  0
 .../{acceptance => avocado}/virtio_version.py |  0
 .../virtiofs_submounts.py |  0
 .../virtiofs_submounts.py.data/cleanup.sh |  0
 .../guest-cleanup.sh  |  0
 .../virtiofs_submounts.py.data/guest.sh   |  0
 .../virtiofs_submounts.py.data/host.sh|  0
 tests/{acceptance => avocado}/vnc.py  |  0
 .../x86_cpu_model_versions.py |  0
 64 files changed, 104 insertions(+), 105 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/avocado/README.rst
 rename tests/{acceptance => avocado}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => avocado}/boot_linux.py (100%)
 rename tests/{acceptance => avocado}/boot_linux_console.py (100%)
 rename tests/{acceptance => avocado}/boot_xen.py (100%)
 rename tests/{acceptance => avocado}/cpu_queries.py (100%)
 rename tests/{acceptance => avocado}/empty_cpu_model.py (100%)
 rename tests/{acceptance => avocado}/hotplug_cpu.py (100%)
 rename tests/{acceptance => avocado}/info_usernet.py (100%)
 rename tests/{acceptance => avocado}/intel_iommu.py (100%)
 rename tests/{acceptance => avocado}/linux_initrd.py (99%)
 rename tests/{acceptance => avocado}/linux_ssh_mips_malta.py (100%)
 rename tests/{acceptance => avocado}/machine_arm_ca

[PATCH v2 1/2] tests/acceptance: introduce new check-avocado tartget

[Willian Rampazzo]

This introduces a new `make` target, `check-avocado`, and adds a
deprecation message about the `check-acceptance` target. This is
a preparation for renaming the `tests/acceptance` folder to
 `tests/avocado`.

The plan is to remove the call to the `check-avocado` target one
or two months after the release and leave the warning to force
people to move to the new `check-avocado` target.

Later, the `check-acceptance` target can be removed. The intent
is to avoid a direct impact during the current soft freeze.

Suggested-by: Philippe Mathieu-Daudé 
Signed-off-by: Willian Rampazzo 
---
 docs/about/deprecated.rst | 13 +
 tests/Makefile.include| 17 -
 2 files changed, 25 insertions(+), 5 deletions(-)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 56f9ad15ab..7bf8da8325 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -410,3 +410,16 @@ nanoMIPS ISA
 
 The ``nanoMIPS`` ISA has never been upstreamed to any compiler toolchain.
 As it is hard to generate binaries for it, declare it deprecated.
+
+Testing
+---
+
+Renaming of the acceptance folder to avocado
+
+
+The ``tests/acceptance`` folder was never used to store acceptance tests
+in terms of software engineering. This naming can confuse developers
+adding tests using the Avocado Framework to this folder. The folder
+name change to ``tests/avocado`` also changed the ``make`` target from
+``check-acceptance`` to ``check-avocado``. In this case, the use of the
+``check-acceptance`` target is deprecated.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8434a33fe6..8e8ee58493 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
@echo " $(MAKE) check-tcgRun TCG tests"
@echo " $(MAKE) check-softfloat  Run FPU emulation tests"
 endif
-   @echo " $(MAKE) check-acceptance Run acceptance (functional) tests 
for currently configured targets"
+   @echo " $(MAKE) check-avocadoRun avocado (integration) tests 
for currently configured targets"
@echo
@echo " $(MAKE) check-report.tap Generates an aggregated TAP test 
report"
@echo " $(MAKE) check-venv   Creates a Python venv for tests"
@@ -24,7 +24,7 @@ endif
@echo
@echo "The following are useful for CI builds"
@echo " $(MAKE) check-build  Build most test binaris"
-   @echo " $(MAKE) get-vm-imagesDownloads all images used by 
acceptance tests, according to configured targets (~350 MB each, 1.5 GB max)"
+   @echo " $(MAKE) get-vm-imagesDownloads all images used by 
avocado tests, according to configured targets (~350 MB each, 1.5 GB max)"
@echo
@echo
@echo "The variable SPEED can be set to control the gtester speed 
setting."
@@ -83,7 +83,7 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 
 # Python venv for running tests
 
-.PHONY: check-venv check-acceptance
+.PHONY: check-venv check-avocado check-acceptance 
check-acceptance-deprecated-warning
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
@@ -127,12 +127,12 @@ get-vm-image-fedora-31-%: check-venv
$(call quiet-command, \
  $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \
  --distro=fedora --distro-version=31 --arch=$*, \
-   "AVOCADO", "Downloading acceptance tests VM image for $*")
+   "AVOCADO", "Downloading avocado tests VM image for $*")
 
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
$(FEDORA_31_DOWNLOAD))
 
-check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
+check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
$(call quiet-command, \
 $(TESTS_VENV_DIR)/bin/python -m avocado \
 --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
@@ -142,6 +142,13 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
get-vm-images
 $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
 "AVOCADO", "tests/acceptance")
 
+check-acceptance-deprecated-warning:
+   @echo
+   @echo "Note '$(MAKE) check-acceptance' is deprecated, use '$(MAKE) 
check-avocado' instead."
+   @echo
+
+check-acceptance: check-acceptance-deprecated-warning | check-avocado
+
 # Consolidated targets
 
 .PHONY: check-block check check-clean get-vm-images
-- 
2.33.1

Re: [PATCH 1/1] tests/acceptance: rename tests acceptance to tests avocado

[Willian Rampazzo]

On Thu, Nov 4, 2021 at 9:50 AM Philippe Mathieu-Daudé  wrote:
>
> On 11/3/21 22:14, Willian Rampazzo wrote:
> > In the discussion about renaming the `tests/acceptance` [1], the
> > conclusion was that the folders inside `tests` are related to the
> > framework running the tests and not directly related to the type of
> > the tests.
> >
> > This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
> > CI related files and the documentation.
> >
> > [1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html
> >
> > Signed-off-by: Willian Rampazzo 
> > ---
> >  64 files changed, 113 insertions(+), 118 deletions(-)
>
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
>
> > -.PHONY: check-venv check-acceptance
> > +.PHONY: check-venv check-avocado
>
> > -check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> > +check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >   $(call quiet-command, \
> >  $(TESTS_VENV_DIR)/bin/python -m avocado \
> >  --show=$(AVOCADO_SHOW) run 
> > --job-results-dir=$(TESTS_RESULTS_DIR) \
>
> We should keep check-acceptance, eventually printing a deprecation
> message. Smth like:
>
>   check-acceptance: check-avocado a
>   @echo "Note 'make $@' is deprecated, use 'make $<' instead" >&2
>

I agree.

> > --- a/tests/acceptance/ppc_prep_40p.py
> > +++ b/tests/avocado/ppc_prep_40p.py
> > @@ -13,6 +13,10 @@
> >
> >
> >  class IbmPrep40pMachine(Test):
> > +"""
> > +:avocado: tags=arch:ppc
> > +:avocado: tags=machine:40p
> > +"""
> >
> >  timeout = 60
> >
> > @@ -24,8 +28,6 @@ class IbmPrep40pMachine(Test):
> >  @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted 
> > code')
> >  def test_factory_firmware_and_netbsd(self):
> >  """
> > -:avocado: tags=arch:ppc
> > -:avocado: tags=machine:40p
> >  :avocado: tags=os:netbsd
> >  :avocado: tags=slowness:high
> >  """
> > @@ -48,10 +50,6 @@ def test_factory_firmware_and_netbsd(self):
> >  wait_for_console_pattern(self, 'Model: IBM PPS Model 6015')
> >
> >  def test_openbios_192m(self):
> > -"""
> > -:avocado: tags=arch:ppc
> > -:avocado: tags=machine:40p
> > -"""
> >  self.vm.set_console()
> >  self.vm.add_args('-m', '192') # test fw_cfg
> >
> > @@ -62,8 +60,6 @@ def test_openbios_192m(self):
> >
> >  def test_openbios_and_netbsd(self):
> >  """
> > -:avocado: tags=arch:ppc
> > -:avocado: tags=machine:40p
> >  :avocado: tags=os:netbsd
> >  """
> >  drive_url = ('https://archive.netbsd.org/pub/NetBSD-archive/'
>
> Unrelated change, otherwise:

Ouch, looks like I took one of my test changes. I'll wait for some
more comments and remove this change on a v2.

>
> Reviewed-by: Philippe Mathieu-Daudé 
> Tested-by: Philippe Mathieu-Daudé 
>

Thanks!

[PATCH 1/1] tests/acceptance: rename tests acceptance to tests avocado

[Willian Rampazzo]

In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

Signed-off-by: Willian Rampazzo 
---
 .gitlab-ci.d/buildtest-template.yml   |  2 +-
 .gitlab-ci.d/buildtest.yml| 56 +--
 MAINTAINERS   | 52 -
 configure |  2 +-
 docs/devel/build-system.rst   |  2 +-
 docs/devel/ci-definitions.rst.inc |  2 +-
 docs/devel/testing.rst| 49 
 docs/system/arm/orangepi.rst  |  8 +--
 python/qemu/machine/README.rst|  2 +-
 python/qemu/qmp/README.rst|  2 +-
 python/qemu/utils/README.rst  |  2 +-
 tests/Makefile.include| 14 ++---
 tests/acceptance/README.rst   | 10 
 tests/avocado/README.rst  | 10 
 .../avocado_qemu/__init__.py  |  2 +-
 tests/{acceptance => avocado}/boot_linux.py   |  0
 .../boot_linux_console.py |  0
 tests/{acceptance => avocado}/boot_xen.py |  0
 tests/{acceptance => avocado}/cpu_queries.py  |  0
 .../empty_cpu_model.py|  0
 tests/{acceptance => avocado}/hotplug_cpu.py  |  0
 tests/{acceptance => avocado}/info_usernet.py |  0
 tests/{acceptance => avocado}/intel_iommu.py  |  0
 tests/{acceptance => avocado}/linux_initrd.py |  2 +-
 .../linux_ssh_mips_malta.py   |  0
 .../machine_arm_canona1100.py |  0
 .../machine_arm_integratorcp.py   |  0
 .../machine_arm_n8x0.py   |  0
 tests/{acceptance => avocado}/machine_avr6.py |  2 +-
 .../machine_m68k_nextcube.py  |  0
 .../machine_microblaze.py |  0
 .../machine_mips_fuloong2e.py |  0
 .../machine_mips_loongson3v.py|  0
 .../machine_mips_malta.py |  0
 .../machine_rx_gdbsim.py  |  0
 .../machine_s390_ccw_virtio.py|  0
 .../machine_sparc64_sun4u.py  |  0
 .../machine_sparc_leon3.py|  0
 tests/{acceptance => avocado}/migration.py|  0
 tests/{acceptance => avocado}/multiprocess.py |  0
 .../pc_cpu_hotplug_props.py   |  0
 tests/{acceptance => avocado}/ppc_405.py  |  0
 tests/{acceptance => avocado}/ppc_bamboo.py   |  0
 .../{acceptance => avocado}/ppc_mpc8544ds.py  |  0
 tests/{acceptance => avocado}/ppc_prep_40p.py | 12 ++--
 tests/{acceptance => avocado}/ppc_pseries.py  |  0
 .../ppc_virtex_ml507.py   |  0
 .../{acceptance => avocado}/replay_kernel.py  |  0
 tests/{acceptance => avocado}/replay_linux.py |  0
 .../reverse_debugging.py  |  0
 tests/{acceptance => avocado}/smmu.py |  0
 tests/{acceptance => avocado}/tcg_plugins.py  |  0
 .../tesseract_utils.py|  0
 tests/{acceptance => avocado}/version.py  |  0
 tests/{acceptance => avocado}/virtio-gpu.py   |  0
 .../virtio_check_params.py|  0
 .../{acceptance => avocado}/virtio_version.py |  0
 .../virtiofs_submounts.py |  0
 .../virtiofs_submounts.py.data/cleanup.sh |  0
 .../guest-cleanup.sh  |  0
 .../virtiofs_submounts.py.data/guest.sh   |  0
 .../virtiofs_submounts.py.data/host.sh|  0
 tests/{acceptance => avocado}/vnc.py  |  0
 .../x86_cpu_model_versions.py |  0
 64 files changed, 113 insertions(+), 118 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/avocado/README.rst
 rename tests/{acceptance => avocado}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => avocado}/boot_linux.py (100%)
 rename tests/{acceptance => avocado}/boot_linux_console.py (100%)
 rename tests/{acceptance => avocado}/boot_xen.py (100%)
 rename tests/{acceptance => avocado}/cpu_queries.py (100%)
 rename tests/{acceptance => avocado}/empty_cpu_model.py (100%)
 rename tests/{acceptance => avocado}/hotplug_cpu.py (100%)
 rename tests/{acceptance => avocado}/info_usernet.py (100%)
 rename tests/{acceptance => avocado}/intel_iommu.py (100%)
 rename tests/{acceptance => avocado}/linux_initrd.py (99%)
 rename tests/{acceptance => avocado}/linux_ssh_mips_malta.py (100%)
 rename tests/{acceptance => avocado}/machine_arm_canona1100.py (100%)
 rename tests/{acceptance => avocado}/machine_arm_integratorcp.py (100%)
 rename tests/{acceptance =>

[PATCH 0/1] tests/acceptance: rename tests acceptance to tests avocado

[Willian Rampazzo]

In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

GitLab pipeline with new naming: 
https://gitlab.com/willianrampazzo/qemu/-/pipelines/401431592
Signed-off-by: Willian Rampazzo 

Willian Rampazzo (1):
  tests/acceptance: rename tests acceptance to tests avocado

 .gitlab-ci.d/buildtest-template.yml   |  2 +-
 .gitlab-ci.d/buildtest.yml| 56 +--
 MAINTAINERS   | 52 -
 configure |  2 +-
 docs/devel/build-system.rst   |  2 +-
 docs/devel/ci-definitions.rst.inc |  2 +-
 docs/devel/testing.rst| 49 
 docs/system/arm/orangepi.rst  |  8 +--
 python/qemu/machine/README.rst|  2 +-
 python/qemu/qmp/README.rst|  2 +-
 python/qemu/utils/README.rst  |  2 +-
 tests/Makefile.include| 14 ++---
 tests/acceptance/README.rst   | 10 
 tests/avocado/README.rst  | 10 
 .../avocado_qemu/__init__.py  |  2 +-
 tests/{acceptance => avocado}/boot_linux.py   |  0
 .../boot_linux_console.py |  0
 tests/{acceptance => avocado}/boot_xen.py |  0
 tests/{acceptance => avocado}/cpu_queries.py  |  0
 .../empty_cpu_model.py|  0
 tests/{acceptance => avocado}/hotplug_cpu.py  |  0
 tests/{acceptance => avocado}/info_usernet.py |  0
 tests/{acceptance => avocado}/intel_iommu.py  |  0
 tests/{acceptance => avocado}/linux_initrd.py |  2 +-
 .../linux_ssh_mips_malta.py   |  0
 .../machine_arm_canona1100.py |  0
 .../machine_arm_integratorcp.py   |  0
 .../machine_arm_n8x0.py   |  0
 tests/{acceptance => avocado}/machine_avr6.py |  2 +-
 .../machine_m68k_nextcube.py  |  0
 .../machine_microblaze.py |  0
 .../machine_mips_fuloong2e.py |  0
 .../machine_mips_loongson3v.py|  0
 .../machine_mips_malta.py |  0
 .../machine_rx_gdbsim.py  |  0
 .../machine_s390_ccw_virtio.py|  0
 .../machine_sparc64_sun4u.py  |  0
 .../machine_sparc_leon3.py|  0
 tests/{acceptance => avocado}/migration.py|  0
 tests/{acceptance => avocado}/multiprocess.py |  0
 .../pc_cpu_hotplug_props.py   |  0
 tests/{acceptance => avocado}/ppc_405.py  |  0
 tests/{acceptance => avocado}/ppc_bamboo.py   |  0
 .../{acceptance => avocado}/ppc_mpc8544ds.py  |  0
 tests/{acceptance => avocado}/ppc_prep_40p.py | 12 ++--
 tests/{acceptance => avocado}/ppc_pseries.py  |  0
 .../ppc_virtex_ml507.py   |  0
 .../{acceptance => avocado}/replay_kernel.py  |  0
 tests/{acceptance => avocado}/replay_linux.py |  0
 .../reverse_debugging.py  |  0
 tests/{acceptance => avocado}/smmu.py |  0
 tests/{acceptance => avocado}/tcg_plugins.py  |  0
 .../tesseract_utils.py|  0
 tests/{acceptance => avocado}/version.py  |  0
 tests/{acceptance => avocado}/virtio-gpu.py   |  0
 .../virtio_check_params.py|  0
 .../{acceptance => avocado}/virtio_version.py |  0
 .../virtiofs_submounts.py |  0
 .../virtiofs_submounts.py.data/cleanup.sh |  0
 .../guest-cleanup.sh  |  0
 .../virtiofs_submounts.py.data/guest.sh   |  0
 .../virtiofs_submounts.py.data/host.sh|  0
 tests/{acceptance => avocado}/vnc.py  |  0
 .../x86_cpu_model_versions.py |  0
 64 files changed, 113 insertions(+), 118 deletions(-)
 delete mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/avocado/README.rst
 rename tests/{acceptance => avocado}/avocado_qemu/__init__.py (99%)
 rename tests/{acceptance => avocado}/boot_linux.py (100%)
 rename tests/{acceptance => avocado}/boot_linux_console.py (100%)
 rename tests/{acceptance => avocado}/boot_xen.py (100%)
 rename tests/{acceptance => avocado}/cpu_queries.py (100%)
 rename tests/{acceptance => avocado}/empty_cpu_model.py (100%)
 rename tests/{acceptance => avocado}/hotplug_cpu.py (100%)
 rename tests/{acceptance => avocado}/info_usernet.py (100%)
 rename tests/{acceptance => avocado}/intel_iommu.py (100%)
 rename tests/{acceptance => avocado}/linux_initrd.py (99%)
 rename tests/{acceptance => avocado}/linux_ssh_mips_malta.py (100%)
 rena

Re: [qemu-web PATCH] remove deployment phase from CI

[Willian Rampazzo]

On Wed, Nov 3, 2021 at 5:16 AM Paolo Bonzini  wrote:
>
> qemu.org is now served via a reverse proxy from qemu-project.gitlab.io; it 
> does
> not need anymore the rsync step to the QEMU project\s shell server.
> Remove it from the CI.
>
> Signed-off-by: Paolo Bonzini 
> ---
>  .gitlab-ci.yml | 24 
>  1 file changed, 24 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 5/6] docker: update 'python' dockerfile to use Fedora registry

[Willian Rampazzo]

On Wed, Nov 3, 2021 at 11:50 AM John Snow  wrote:
>
> Following the lead of 102cd5c294dc, switch from using Docker hub to the
> Fedora registry itself.
>
> Signed-off-by: John Snow 
> ---
>  tests/docker/dockerfiles/python.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 4/6] docker: update Fedora-based cross-compiler containers to Fedora 34

[Willian Rampazzo]

On Wed, Nov 3, 2021 at 11:50 AM John Snow  wrote:
>
> Similarly to the last patch, Fedora 33 will be EOL by the time QEMU 6.2
> releases, so bump the version for all of the other Fedora containers,
> too.
>
> Signed-off-by: John Snow 
> ---
>  tests/docker/dockerfiles/fedora-cris-cross.docker  | 2 +-
>  tests/docker/dockerfiles/fedora-win32-cross.docker | 2 +-
>  tests/docker/dockerfiles/fedora-win64-cross.docker | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 3/6] docker: update fedora container to Fedora 34

[Willian Rampazzo]

On Wed, Nov 3, 2021 at 11:50 AM John Snow  wrote:
>
> Fedora 33 will be EOL 2021-11-23, before QEMU 6.2 will release. Upgrade
> the version we're testing as our minimum now.
>
> Signed-off-by: John Snow 
> ---
>  tests/docker/dockerfiles/fedora.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v4 2/6] tests/acceptance: Make pick_default_qemu_bin() more generic

[Willian Rampazzo]

On Mon, Sep 27, 2021 at 1:31 PM Philippe Mathieu-Daudé  wrote:
>
> Make pick_default_qemu_bin() generic to find qemu-system or
> qemu-user binaries.
>
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 10 +-
>  1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> b/tests/acceptance/avocado_qemu/__init__.py
> index 8fcbed74849..2b9b5dd27fe 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -52,7 +52,7 @@ def is_readable_executable_file(path):
>  return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
>
>
> -def pick_default_qemu_bin(arch=None):
> +def pick_default_qemu_bin(bin_prefix='qemu-system-', arch=None):
>  """
>  Picks the path of a QEMU binary, starting either in the current working
>  directory or in the source tree root directory.
> @@ -71,7 +71,7 @@ def pick_default_qemu_bin(arch=None):
>  # qemu binary path does not match arch for powerpc, handle it
>  if 'ppc64le' in arch:
>  arch = 'ppc64'
> -qemu_bin_relative_path = "./qemu-system-%s" % arch
> +qemu_bin_relative_path = os.path.join(".", bin_prefix + arch)
>  if is_readable_executable_file(qemu_bin_relative_path):
>  return qemu_bin_relative_path
>
> @@ -185,14 +185,14 @@ def _get_unique_tag_val(self, tag_name):
>  return vals.pop()
>  return None
>
> -def setUp(self):
> +def setUp(self, bin_prefix):
>  self.arch = self.params.get('arch',
>  default=self._get_unique_tag_val('arch'))
>
>  self.cpu = self.params.get('cpu',
> default=self._get_unique_tag_val('cpu'))
>
> -default_qemu_bin = pick_default_qemu_bin(arch=self.arch)
> +default_qemu_bin = pick_default_qemu_bin(bin_prefix, arch=self.arch)
>  self.qemu_bin = self.params.get('qemu_bin',
>  default=default_qemu_bin)
>  if self.qemu_bin is None:
> @@ -220,7 +220,7 @@ class Test(QemuBaseTest):
>  def setUp(self):
>  self._vms = {}
>
> -super(Test, self).setUp()
> +super(Test, self).setUp('qemu-system-')

If you need to change something else in this patch, consider using PEP3135:

super().setUp('qemu-system-')

Anyway,

Reviewed-by: Willian Rampazzo 

>
>  self.machine = self.params.get('machine',
> 
> default=self._get_unique_tag_val('machine'))
> --
> 2.31.1
>

Re: [PATCH v4 4/6] tests/acceptance: Share useful helpers from virtiofs_submounts test

[Willian Rampazzo]

On Mon, Sep 27, 2021 at 1:31 PM Philippe Mathieu-Daudé  wrote:
>
> Move the useful has_cmd()/has_cmds() helpers from the virtiofs
> test to the avocado_qemu public class.
>
> Reviewed-by: Wainer dos Santos Moschetta 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 57 ++
>  tests/acceptance/virtiofs_submounts.py| 59 +--
>  2 files changed, 59 insertions(+), 57 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v4 6/6] tests/acceptance: Rename avocado_qemu.Test -> QemuSystemTest

[Willian Rampazzo]

 """
> +class QemuSystemTest(QemuBaseTest):
> +"""Facilitates system emulation tests."""
>
>  def setUp(self):
>  self._vms = {}
>
> -super(Test, self).setUp('qemu-system-')
> +super(QemuSystemTest, self).setUp('qemu-system-')

If you take my suggestion in one of the previous patches, you don't
need this change here.

>
>  self.machine = self.params.get('machine',
> 
> default=self._get_unique_tag_val('machine'))
> @@ -515,11 +512,11 @@ def default_kernel_params(self):
>  return self._info.get('kernel_params', None)
>
>
> -class LinuxTest(Test, LinuxSSHMixIn):
> +class LinuxTest(QemuSystemTest, LinuxSSHMixIn):
>  """Facilitates having a cloud-image Linux based available.
>
>  For tests that indend to interact with guests, this is a better choice

If you touch this patch again, please, s/indend/intend/

So far, looks good to me

Reviewed-by: Willian Rampazzo 

Re: [PATCH v4 3/6] tests/acceptance: Introduce QemuUserTest base class

[Willian Rampazzo]

On Mon, Sep 27, 2021 at 1:31 PM Philippe Mathieu-Daudé  wrote:
>
> Similarly to the 'System' Test base class with methods for testing
> system emulation, the QemuUserTest class contains methods useful to
> test user-mode emulation.
>
> Reviewed-by: Wainer dos Santos Moschetta 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 18 ++
>  1 file changed, 18 insertions(+)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> b/tests/acceptance/avocado_qemu/__init__.py
> index 2b9b5dd27fe..2f59e171e49 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -20,6 +20,7 @@
>  from avocado.utils import cloudinit
>  from avocado.utils import datadrainer
>  from avocado.utils import network
> +from avocado.utils import process
>  from avocado.utils import ssh
>  from avocado.utils import vmimage
>  from avocado.utils.path import find_command
> @@ -300,6 +301,23 @@ def tearDown(self):
>  self._sd = None
>
>
> +class QemuUserTest(QemuBaseTest):
> +"""Facilitates user-mode emulation tests."""
> +
> +def setUp(self):
> +self._ldpath = []
> +super(QemuUserTest, self).setUp('qemu-')

Consider using PEP3135,

super().setUp('qemu-')

> +
> +def add_ldpath(self, ldpath):
> +self._ldpath += [os.path.abspath(ldpath)]

The use of the append method is more common here:

self._ldpath.append(os.path.abspath(ldpath))

> +
> +def run(self, bin_path, args=[]):
> +qemu_args = " ".join(["-L %s" % ldpath for ldpath in self._ldpath])
> +bin_args = " ".join(args)
> +return process.run("%s %s %s %s" % (self.qemu_bin, qemu_args,
> +bin_path, bin_args))
> +
> +
>  class LinuxSSHMixIn:
>  """Contains utility methods for interacting with a guest via SSH."""

Besides some Python related suggestions, the code looks good.

Reviewed-by: Willian Rampazzo 

>
> --
> 2.31.1
>

Re: [PATCH v4 1/6] tests/acceptance: Extract QemuBaseTest from Test

[Willian Rampazzo]

 asset_hash=None, algorithm=None,
> -locations=None, expire=None,
> -find_only=False, cancel_on_missing=True):
> -return super(Test, self).fetch_asset(name,
> -asset_hash=asset_hash,
> -        algorithm=algorithm,
> -locations=locations,
> -expire=expire,
> -find_only=find_only,
> -cancel_on_missing=cancel_on_missing)
> -
>
>  class LinuxSSHMixIn:
>  """Contains utility methods for interacting with a guest via SSH."""
> --
> 2.31.1
>

Except for one (or two, if you consider the first) small comment,
looks good to me, so

Reviewed-by: Willian Rampazzo 

Re: [PATCH v4 5/6] tests/acceptance: Add bFLT loader linux-user test

[Willian Rampazzo]

Hi, Phill,

On Mon, Sep 27, 2021 at 1:31 PM Philippe Mathieu-Daudé  wrote:
>
> Add a very quick test that runs a busybox binary in bFLT format:
>
>   $ AVOCADO_ALLOW_UNTRUSTED_CODE=1 \
> avocado --show=app run -t linux_user tests/acceptance/load_bflt.py
>   JOB ID : db94d5960ce564c50904d666a7e259148c27e88f
>   JOB LOG: ~/avocado/job-results/job-2019-06-25T10.52-db94d59/job.log
>(1/1) tests/acceptance/load_bflt.py:LoadBFLT.test_stm32: PASS (0.15 s)
>   RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>   JOB TIME   : 0.54 s
>
> Reviewed-by: Willian Rampazzo 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  tests/acceptance/load_bflt.py | 54 +++
>  1 file changed, 54 insertions(+)
>  create mode 100644 tests/acceptance/load_bflt.py
>
> diff --git a/tests/acceptance/load_bflt.py b/tests/acceptance/load_bflt.py
> new file mode 100644
> index 000..f071a979d8e
> --- /dev/null
> +++ b/tests/acceptance/load_bflt.py
> @@ -0,0 +1,54 @@
> +# Test the bFLT loader format
> +#
> +# Copyright (C) 2019 Philippe Mathieu-Daudé 
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import os
> +import bz2
> +import subprocess
> +
> +from avocado import skipUnless
> +from avocado_qemu import QemuUserTest
> +from avocado_qemu import has_cmd
> +
> +
> +class LoadBFLT(QemuUserTest):
> +
> +def extract_cpio(self, cpio_path):
> +"""
> +Extracts a cpio archive into the test workdir
> +
> +:param cpio_path: path to the cpio archive
> +"""
> +cwd = os.getcwd()
> +os.chdir(self.workdir)
> +with bz2.open(cpio_path, 'rb') as archive_cpio:
> +subprocess.run(['cpio', '-i'], input=archive_cpio.read(),
> +   stderr=subprocess.DEVNULL)
> +os.chdir(cwd)
> +
> +@skipUnless(*has_cmd('cpio'))
> +@skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> +def test_stm32(self):
> +"""
> +:avocado: tags=arch:arm
> +:avocado: tags=linux_user
> +:avocado: tags=quick
> +"""
> +# See https://elinux.org/STM32#User_Space
> +rootfs_url = ('https://elinux.org/images/5/51/'
> +  'Stm32_mini_rootfs.cpio.bz2')
> +rootfs_hash = '9f065e6ba40cce7411ba757f924f30fcc57951e6'
> +rootfs_path_bz2 = self.fetch_asset(rootfs_url, 
> asset_hash=rootfs_hash)
> +busybox_path = self.workdir + "/bin/busybox"

If there are other changes to this patch, also, change this to use the
`os` library:

busybox_path = os.path.join(self.workdir, "/bin/busybox")

Sorry for not catching this before.

> +
> +self.extract_cpio(rootfs_path_bz2)
> +
> +res = self.run(busybox_path)
> +ver = 'BusyBox v1.24.0.git (2015-02-03 22:17:13 CET) multi-call 
> binary.'
> +self.assertIn(ver, res.stdout_text)
> +
> +res = self.run(busybox_path, ['uname', '-a'])
> +unm = 'armv7l GNU/Linux'
> +self.assertIn(unm, res.stdout_text)
> --
> 2.31.1
>

Re: [PATCH 1/1] gitlab-ci: Only push docker images to registry from /master branch

[Willian Rampazzo]

On Tue, Oct 26, 2021 at 11:55 AM Philippe Mathieu-Daudé
 wrote:
>
> Users expect images pulled from registry.gitlab.com/qemu-project/qemu/
> to be stable. QEMU repository workflow pushes merge candidates to
> the /staging branch, and on success the same commit is pushed as
> /master. If /staging fails, we do not want to push the built images
> to the registry. Therefore limit the 'docker push' command to the
> /master branch on the mainstream CI. The fork behavior is unchanged.
>
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  .gitlab-ci.d/container-template.yml | 11 ++-
>  .gitlab-ci.d/edk2.yml   | 11 ++-
>  .gitlab-ci.d/opensbi.yml| 11 ++-
>  3 files changed, 30 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 09/28] tests/docker: split PARTIAL into PARTIAL and VIRTUAL images

[Willian Rampazzo]

On Tue, Oct 26, 2021 at 7:22 AM Alex Bennée  wrote:
>
> This is mostly to ensure we don't include the toolchain and bootstrap
> builds in DOCKER_IMAGES which is useful when verifying all images
> still build.
>
> Signed-off-by: Alex Bennée 
> ---
>  tests/docker/Makefile.include | 8 +---
>  1 file changed, 5 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 05/28] tests/docker: Add debian-nios2-cross image

[Willian Rampazzo]

On Tue, Oct 26, 2021 at 7:22 AM Alex Bennée  wrote:
>
> From: Richard Henderson 
>
> Build the entire cross tool chain from source.
> For this reason, default to caching.
>
> Signed-off-by: Richard Henderson 
> [AJB: honour NOUSER in cached fetch and build, update MAINTAINERS]
> Signed-off-by: Alex Bennée 
> Message-Id: <20211014224435.2539547-6-richard.hender...@linaro.org>
> ---
>  MAINTAINERS   |  1 +
>  tests/docker/Makefile.include | 25 ++
>  .../debian-nios2-cross.d/build-toolchain.sh   | 87 +++
>  .../dockerfiles/debian-toolchain.docker   | 36 
>  4 files changed, 149 insertions(+)
>  create mode 100755 
> tests/docker/dockerfiles/debian-nios2-cross.d/build-toolchain.sh
>  create mode 100644 tests/docker/dockerfiles/debian-toolchain.docker
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 06/28] tests/docker: Add debian-microblaze-cross image

[Willian Rampazzo]

On Tue, Oct 26, 2021 at 7:22 AM Alex Bennée  wrote:
>
> From: Richard Henderson 
>
> Build the entire cross tool chain from source.
> For this reason, default to caching.
>
> Signed-off-by: Richard Henderson 
> [AJB: Update MAINTAINERS]
> Signed-off-by: Alex Bennée 
> Message-Id: <20211014224435.2539547-7-richard.hender...@linaro.org>
> ---
>  MAINTAINERS   |  1 +
>  tests/docker/Makefile.include |  6 ++
>  .../build-toolchain.sh| 88 +++
>  3 files changed, 95 insertions(+)
>  create mode 100755 
> tests/docker/dockerfiles/debian-microblaze-cross.d/build-toolchain.sh
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH v1 04/28] gitlab-ci: Remove special casing for hexagon testing

[Willian Rampazzo]

On Tue, Oct 26, 2021 at 7:22 AM Alex Bennée  wrote:
>
> From: Richard Henderson 
>
> Now that the hexagon container is using a pre-built
> toolchain, we do not need to make the build optional.
>
> Signed-off-by: Richard Henderson 
> Signed-off-by: Alex Bennée 
> Message-Id: <20211014224435.2539547-5-richard.hender...@linaro.org>
> ---
>  .gitlab-ci.d/buildtest.yml   |  4 
>  .gitlab-ci.d/container-cross.yml | 27 +++
>  2 files changed, 3 insertions(+), 28 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Willian Rampazzo]

On Wed, Oct 6, 2021 at 4:17 PM Willian Rampazzo  wrote:
>
> On Fri, Sep 24, 2021 at 4:01 PM Cleber Rosa  wrote:
> >
> > The arch and machine tags apply to all tests, so let's define them
> > only once.
> >
> > Signed-off-by: Cleber Rosa 
> > ---
> >  tests/acceptance/ppc_prep_40p.py | 12 
> >  1 file changed, 4 insertions(+), 8 deletions(-)
> >
>
> Reviewed-by: Willian Rampazzo 

Also:
Tested-by: Willian Rampazzo 

Re: [PATCH 16/16] tests/acceptance/ppc_prep_40p.py: unify tags

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 4:01 PM Cleber Rosa  wrote:
>
> The arch and machine tags apply to all tests, so let's define them
> only once.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/ppc_prep_40p.py | 12 
>  1 file changed, 4 insertions(+), 8 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 13/16] tests/acceptance/boot_xen.py: use class attribute

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:59 PM Cleber Rosa  wrote:
>
> Rather than defining a single use variable, let's just use the class
> attribute directly.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/boot_xen.py | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 09/16] tests/acceptance/boot_xen.py: merge base classes

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:58 PM Cleber Rosa  wrote:
>
> While it's a good practice to have reusable base classes, in this
> specific case there's no other user of the BootXenBase class.
>
> By unifying the class used in this test, we can improve readability
> and have the opportunity to add some future improvements in a clearer
> fashion.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/boot_xen.py | 5 +
>  1 file changed, 1 insertion(+), 4 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 10/16] tests/acceptance/boot_xen.py: unify tags

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:58 PM Cleber Rosa  wrote:
>
> Because all tests share the same tags, it's possible to have all of
> them at the class level.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/boot_xen.py | 26 +-
>  1 file changed, 5 insertions(+), 21 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 11/16] tests/acceptance/boot_xen.py: fetch kernel during test setUp()

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:59 PM Cleber Rosa  wrote:
>
> The kernel is a common blob used in all tests.  By moving it to the
> setUp() method, the "fetch asset" plugin will recognize the kernel and
> attempt to fetch it and cache it before the tests are started.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/boot_xen.py | 13 ++---
>  1 file changed, 6 insertions(+), 7 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 03/16] Acceptance Tests: add mechanism for listing tests

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa  wrote:
>
> It is helpful to know the tests that would be executed with a "make
> check-acceptance" without executing them.  Let's introduce a "make
> list-acceptance" rule for that purpose.
>
> Signed-off-by: Cleber Rosa 
> ---
>  tests/Makefile.include | 8 
>  1 file changed, 8 insertions(+)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 2c03256ae8..d1f90572a7 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -17,6 +17,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),)
> @echo " $(MAKE) check-softfloat  Run FPU emulation tests"
>  endif
> @echo " $(MAKE) check-acceptance Run acceptance (functional) 
> tests for currently configured targets"
> +   @echo " $(MAKE) check-acceptance List acceptance (functional) 
> tests for currently configured targets"

s/check-acceptance/list-acceptance/

> @echo
> @echo " $(MAKE) check-report.tap Generates an aggregated TAP test 
> report"
> @echo " $(MAKE) check-venv   Creates a Python venv for tests"
> @@ -135,6 +136,13 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
> get-vm-images
>  $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>  "AVOCADO", "tests/acceptance")
>
> +list-acceptance: check-venv
> +   $(call quiet-command, \
> +$(TESTS_VENV_DIR)/bin/python -m avocado list \
> +--filter-by-tags-include-empty 
> --filter-by-tags-include-empty-key \
> +$(AVOCADO_TAGS) tests/acceptance, \

It would be interesting to adjust the command to use the
$(AVOCADO_CMDLINE_TAGS) and allow users to restrict the tests listed
by its tags.

> +"AVOCADO", "list tests/acceptance")
> +
>  # Consolidated targets
>
>  .PHONY: check-block check check-clean get-vm-images
> --
> 2.31.1
>

Re: [PATCH 01/16] Acceptance Tests: bump Avocado requirement to 91.0

[Willian Rampazzo]

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa  wrote:
>
> Avocado 91.0, brings, among other changes, a switch to a new runner
> implementation, known as "nrunner".  While my personal testing shows
> that it's suitable for running the QEMU tests, there are some
> considerations that should be addressed before the change.
>
> For instance, the "nrunner" implementation will run tests in parallel
> by default.  With more tests running, and resources shared, some tests
> may fail with timeouts on some environments.
>
> So, for now, let's bump the Avocado version to allow for the other
> features to be used.  And the "nrunner" implementation to be used by
> those that choose to do so.
>
> More information can be found at the release notes at:
>https://avocado-framework.readthedocs.io/en/latest/releases/91_0.html
>
> Signed-off-by: Cleber Rosa 
> ---
>  docs/devel/testing.rst | 12 
>  tests/Makefile.include |  2 ++
>  tests/requirements.txt |  2 +-
>  3 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 4a0abbf23d..2818c58ff8 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -740,6 +740,18 @@ may be invoked by running:
>
>tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
>
> +Running tests in parallel
> +-
> +
> +The current ``make check-acceptance`` target continues to use the
> +Avocado runner implementation simply known as "runner".  But, it's now
> +possible to opt-in and choose the "nrunner" implementation, which,
> +among other things, allows for parallel execution of tests:
> +
> +.. code::
> +
> +  make AVOCADO_RUNNER=nrunner check-acceptance

The examples in the documentation are all setting the environment
variable at the end of the `make` command. For consistency, I would
move the definition of the AVOCADO_RUNNER to the end of the command.

> +
>  Manual Installation
>  ---
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 6e16c05f10..ac289a2e41 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -92,6 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # Any number of command separated loggers are accepted.  For more
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=app
> +AVOCADO_RUNNER=runner
>  AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
>
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> @@ -128,6 +129,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
> get-vm-images
> $(call quiet-command, \
>  $(TESTS_VENV_DIR)/bin/python -m avocado \
>  --show=$(AVOCADO_SHOW) run 
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> +--test-runner=$(AVOCADO_RUNNER) \
>  --filter-by-tags-include-empty 
> --filter-by-tags-include-empty-key \
>  $(AVOCADO_TAGS) \
>  $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index a21b59b443..40af24c664 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,5 +1,5 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -avocado-framework==88.1
> +avocado-framework==91.0
>  pycdlib==1.11.0
> --
> 2.31.1
>

Apart from a tiny documentation change, the code is welcome as it
allows using the new Avocado runner.

Reviewed-by: Willian Rampazzo 

Re: [RFC PATCH] .github: move repo lockdown to the v2 configuration

[Willian Rampazzo]

On Mon, Oct 4, 2021 at 12:45 PM Alex Bennée  wrote:
>
> I was getting prompted by GitHub for new permissions but it turns out
> per https://github.com/dessant/repo-lockdown/issues/6:
>
>   Repo Lockdown has been rewritten for GitHub Actions, offering new
>   features and better control over your automation presets. The legacy
>   GitHub App has been deprecated, and the public instance of the app
>   has been shut down.
>
> So this is what I've done.
>
> Signed-off-by: Alex Bennée 
> ---
>  .github/lockdown.yml   | 34 
>  .github/workflows/lockdown.yml | 47 ++
>  2 files changed, 47 insertions(+), 34 deletions(-)
>  delete mode 100644 .github/lockdown.yml
>  create mode 100644 .github/workflows/lockdown.yml
>
> diff --git a/.github/lockdown.yml b/.github/lockdown.yml
> deleted file mode 100644
> index d3546bd2bc..00
> --- a/.github/lockdown.yml
> +++ /dev/null
> @@ -1,34 +0,0 @@
> -# Configuration for Repo Lockdown - https://github.com/dessant/repo-lockdown
> -
> -# Close issues and pull requests
> -close: true
> -
> -# Lock issues and pull requests
> -lock: true
> -
> -issues:
> -  comment: |
> -Thank you for your interest in the QEMU project.
> -
> -This repository is a read-only mirror of the project's repostories hosted
> -at https://gitlab.com/qemu-project/qemu.git.
> -The project does not process issues filed on GitHub.
> -
> -The project issues are tracked on GitLab:
> -https://gitlab.com/qemu-project/qemu/-/issues
> -
> -QEMU welcomes bug report contributions. You can file new ones on:
> -https://gitlab.com/qemu-project/qemu/-/issues/new
> -
> -pulls:
> -  comment: |
> -Thank you for your interest in the QEMU project.
> -
> -This repository is a read-only mirror of the project's repostories hosted
> -on https://gitlab.com/qemu-project/qemu.git.
> -The project does not process merge requests filed on GitHub.
> -
> -QEMU welcomes contributions of code (either fixing bugs or adding new
> -functionality). However, we get a lot of patches, and so we have some
> -guidelines about contributing on the project website:
> -https://www.qemu.org/contribute/
> diff --git a/.github/workflows/lockdown.yml b/.github/workflows/lockdown.yml
> new file mode 100644
> index 00..20e6208487
> --- /dev/null
> +++ b/.github/workflows/lockdown.yml
> @@ -0,0 +1,47 @@
> +# Configuration for Repo Lockdown - https://github.com/dessant/repo-lockdown
> +
> +name: 'Repo Lockdown'
> +
> +on:
> +  issues:
> +types: opened

I see the "issues" tab is already disabled on the QEMU repository at
GitHub. Maybe we don't need an action for that.

> +  pull_request_target:
> +types: opened
> +
> +permissions:
> +  issues: write
> +  pull-requests: write
> +
> +jobs:
> +  action:
> +runs-on: ubuntu-latest
> +steps:
> +  - uses: dessant/repo-lockdown@v2
> +with:
> +  issue-comment: |
> +Thank you for your interest in the QEMU project.
> +
> +This repository is a read-only mirror of the project's 
> repostories hosted
> +at https://gitlab.com/qemu-project/qemu.git.
> +The project does not process issues filed on GitHub.
> +
> +The project issues are tracked on GitLab:
> +https://gitlab.com/qemu-project/qemu/-/issues
> +
> +QEMU welcomes bug report contributions. You can file new ones on:
> +https://gitlab.com/qemu-project/qemu/-/issues/new
> +  lock-issue: true
> +  close-issue: true
> +  pull-comment: |
> +Thank you for your interest in the QEMU project.
> +
> +This repository is a read-only mirror of the project's 
> repostories hosted
> +on https://gitlab.com/qemu-project/qemu.git.
> +The project does not process merge requests filed on GitHub.
> +
> +QEMU welcomes contributions of code (either fixing bugs or 
> adding new
> +        functionality). However, we get a lot of patches, and so we have 
> some
> +guidelines about contributing on the project website:
> +https://www.qemu.org/contribute/
> +  lock-pull: true
> +  close-pull: true
> --
> 2.30.2
>
>

Besides the "issues" comment, it looks good to me.

Reviewed-by: Willian Rampazzo 

Re: [PATCH v3] nbd/server: Add --selinux-label option

[Willian Rampazzo]

On Thu, Sep 30, 2021 at 5:55 AM Vladimir Sementsov-Ogievskiy
 wrote:
>
> 9/30/21 11:47, Richard W.M. Jones wrote:
> > Under SELinux, Unix domain sockets have two labels.  One is on the
> > disk and can be set with commands such as chcon(1).  There is a
> > different label stored in memory (called the process label).  This can
> > only be set by the process creating the socket.  When using SELinux +
> > SVirt and wanting qemu to be able to connect to a qemu-nbd instance,
> > you must set both labels correctly first.
> >
> > For qemu-nbd the options to set the second label are awkward.  You can
> > create the socket in a wrapper program and then exec into qemu-nbd.
> > Or you could try something with LD_PRELOAD.
> >
> > This commit adds the ability to set the label straightforwardly on the
> > command line, via the new --selinux-label flag.  (The name of the flag
> > is the same as the equivalent nbdkit option.)
> >
> > A worked example showing how to use the new option can be found in
> > this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1984938
> >
> > Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1984938
> > Signed-off-by: Richard W.M. Jones 
> > Reviewed-by: Daniel P. Berrangé 
> > Signed-off-by: Eric Blake 
>
> this should be Reviewed-by?

Maybe, because of this:
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg07081.html

I got confused with this v3.

Re: [PATCH 4/6] avocado_qemu: tweak ssh connect method

[Willian Rampazzo]

On Mon, Sep 27, 2021 at 11:12 AM Philippe Mathieu-Daudé
 wrote:
>
> On 9/20/21 22:49, Willian Rampazzo wrote:
> > The current implementation will crash if the connection fails as the
> > `time` module is not imported. This fixes the import problem and tweaks
> > the connection to wait progressively when the connection fails.
> >
> > Signed-off-by: Willian Rampazzo 
> > ---
> >  tests/acceptance/avocado_qemu/__init__.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> > b/tests/acceptance/avocado_qemu/__init__.py
> > index edb9ed7485..c3613f9262 100644
> > --- a/tests/acceptance/avocado_qemu/__init__.py
> > +++ b/tests/acceptance/avocado_qemu/__init__.py
> > @@ -13,6 +13,7 @@
> >  import shutil
> >  import sys
> >  import tempfile
> > +import time
> >  import uuid
> >
> >  import avocado
> > @@ -305,8 +306,7 @@ def ssh_connect(self, username, credential, 
> > credential_is_key=True):
> >  self.ssh_session.connect()
> >  return
> >  except:
> > -time.sleep(4)
>
> 10 * 4 = 40
>
> > -pass
> > +time.sleep(i)
>
> sum([0..10[) = 45
>
> The described tweak. Almost the same, OK.
>

The idea is that, hopefully, the connection will complete in one of
the first tries, so the overall wait time will be less than using a
fixed number. In the worst case, it will wait 5 seconds more than the
original code.

> >  self.fail('ssh connection timeout')
> >
> >  def ssh_command(self, command):
> >
>

Re: [PATCH 05/16] Acceptance Tests: add standard clean up at test tearDown()

[Willian Rampazzo]

Hi, Cleber,

On Fri, Sep 24, 2021 at 3:57 PM Cleber Rosa  wrote:
>
> The avocado.Test class, used as the basis of the avocado_qemu.Test
> class, performs a clean of temporary directories up as part of its own
> tearDown() implementation.
>
> But the avocado_qemu.Test class is currently missing the same clean
> up, as it implemented its own tearDown() method without resorting to
> the upper class behavior.
>
> This brings avocado_qemu.Test behavior in sync with the standard
> avocado.Test behavior and prevents temporary directories from
> cluttering the test results directory (unless instructed to do so with
> Avocado's "--keep-tmp" option).
>
> Reported-by: Peter Maydell 
> Signed-off-by: Cleber Rosa 
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 1 +
>  1 file changed, 1 insertion(+)
>

There was a comment from Wainer in the original patch and as you were
on vacation I submitted a respin last Monday:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05117.html

I also did some refactoring/fixes to the avocado_qemu here:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05212.html

[PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance

[Willian Rampazzo]

Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.

This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.

This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.

e.g.:

AVOCADO_TAGS="foo bar baz" make check-acceptance

Signed-off-by: Willian Rampazzo 
Tested-by: Wainer dos Santos Moschetta 
Reviewed-by: Wainer dos Santos Moschetta 
Reviewed-by: Thomas Huth 
---
 docs/devel/testing.rst | 14 ++
 tests/Makefile.include | 12 +---
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..d1841e35d5 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -732,6 +732,20 @@ available.  On Debian and Ubuntu based systems, depending 
on the
 specific version, they may be on packages named ``python3-venv`` and
 ``python3-pip``.
 
+It is also possible to run tests based on tags using the
+``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
+variable:
+
+.. code::
+
+   make check-acceptance AVOCADO_TAGS=quick
+
+Note that tags separated with commas have an AND behavior, while tags
+separated by spaces have an OR behavior. For more information on Avocado
+tags, see:
+
+ 
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
+
 The scripts installed inside the virtual environment may be used
 without an "activation".  For instance, the Avocado test runner
 may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..f6484e5b31 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+ifndef AVOCADO_TAGS
+   AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \
+$(filter %-softmmu,$(TARGETS)))
+else
+   AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
+endif
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
$(call quiet-command, \
@@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
get-vm-images
$(call quiet-command, \
 $(TESTS_VENV_DIR)/bin/python -m avocado \
 --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
---filter-by-tags-include-empty --filter-by-tags-include-empty-key \
-$(AVOCADO_TAGS) \
+$(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
+   --filter-by-tags-include-empty-key) \
+$(AVOCADO_CMDLINE_TAGS) \
 $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
 "AVOCADO", "tests/acceptance")
 
-- 
2.31.1

[PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance

[Willian Rampazzo]

Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TESTS environment variable.

Suggested-by: Daniel P. Berrangé 
Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 27 +++
 tests/Makefile.include |  5 -
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index c9f6b97f87..64c9744795 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -746,6 +746,33 @@ tags, see:
 
  
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
 
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-acceptance`` command, set the ``AVOCADO_TESTS``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TESTS=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 
$FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
 The scripts installed inside the virtual environment may be used
 without an "activation".  For instance, the Avocado test runner
 may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index f6484e5b31..e69c4fae53 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+ifndef AVOCADO_TESTS
+   AVOCADO_TESTS=tests/acceptance
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -136,7 +139,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
get-vm-images
 $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
--filter-by-tags-include-empty-key) \
 $(AVOCADO_CMDLINE_TAGS) \
-$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
+$(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
 "AVOCADO", "tests/acceptance")
 
 # Consolidated targets
-- 
2.31.1

[PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test

[Willian Rampazzo]

Add instructions to the Acceptance tests section about running a
single test file or a test within the test file.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 28 
 1 file changed, 28 insertions(+)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index d1841e35d5..c9f6b97f87 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -754,6 +754,34 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+Note that if ``make check-acceptance`` was not executed before, it is
+possible to create the Python virtual environment with the dependencies
+needed running:
+
+ .. code::
+
+  make check-venv
+
+It is also possible to run tests from a single file or a single test within
+a test file. To run tests from a single file within the build tree, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
+
+Valid test names are visible in the output from any previous execution
+of Avocado or ``make check-acceptance``, and can also be queried using:
+
+ .. code::
+
+  tests/venv/bin/avocado list tests/acceptance
+
 Manual Installation
 ---
 
-- 
2.31.1

[PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance

[Willian Rampazzo]

Improve the documentation about how to run a single test within a file
or all the tests from a file using the standalone avocado command.

Add a way to run tests based on tags using the `make check-acceptance`
command.

Add a way to run specific tests using the `make check-acceptance` command.

GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/376122402

Changes from V2:
  - Rename the environment variable to AVOCADO_TESTS
  - Adjust the documentation to mention `make check-venv`
  - Adjust the documentation to mention `avocado list` command
  - Add the AVOCADO_TAGS implementation patch to the series.

Changes from V1:
  - Rename TESTFILES to AVOCADO_TEST_FILES on patch 2
  - Add Suggested-by tag on patch 2

Willian Rampazzo (3):
  tests/Makefile: allow control over tags during check-acceptance
  docs/devel/testing: add instruction to run a single acceptance test
  tests/Makefile: add AVOCADO_TESTS option to make check-acceptance

 docs/devel/testing.rst | 69 ++
 tests/Makefile.include | 17 ---
 2 files changed, 82 insertions(+), 4 deletions(-)

-- 
2.31.1

Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance

[Willian Rampazzo]

On Thu, Sep 23, 2021 at 6:42 AM Daniel P. Berrangé  wrote:
>
> On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote:
> > On 9/22/21 21:46, Willian Rampazzo wrote:
> > > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
> > >  wrote:
> > > >
> > > > On 9/22/21 21:03, Willian Rampazzo wrote:
> > > > > Add the possibility of running all the tests from a single file, or
> > > > > multiple files, running a single test within a file or multiple tests
> > > > > within multiple files using `make check-acceptance` and the TESTFILES
> > > > > environment variable.
> > > > >
> > > > > Signed-off-by: Willian Rampazzo 
> > > > > ---
> > > > >docs/devel/testing.rst | 27 +++
> > > > >tests/Makefile.include |  5 -
> > > > >2 files changed, 31 insertions(+), 1 deletion(-)
> > > >
> > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > > > > index 6e16c05f10..82d7ef7a20 100644
> > > > > --- a/tests/Makefile.include
> > > > > +++ b/tests/Makefile.include
> > > > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> > > > >TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> > > > >TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> > > > >TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > > > > +ifndef TESTFILES
> > > > > + TESTFILES=tests/acceptance
> > > > > +endif
> > > > ># Controls the output generated by Avocado when running tests.
> > > > ># Any number of command separated loggers are accepted.  For more
> > > > ># information please refer to "avocado --help".
> > > > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
> > > > > get-vm-images
> > > > >--show=$(AVOCADO_SHOW) run 
> > > > > --job-results-dir=$(TESTS_RESULTS_DIR) \
> > > > >--filter-by-tags-include-empty 
> > > > > --filter-by-tags-include-empty-key \
> > > > >$(AVOCADO_TAGS) \
> > > > > -$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> > > > > +$(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
> > > >
> > > > Since this is Avocado specific, maybe call the variable
> > > > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
> > >
> > > I don't see a problem with changing that to AVOCADO_TESTFILES. I was
> > > trying to make things shorter and easy to remember. If the too-long
> > > variable name is not a problem, I can change that.
> >
> > This is the generic tests/Makefile, so $TESTFILES might be confusing,
> > which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW,
> > AVOCADO_TAGS).
>
> IIUC, this is not actually just test files - it is test files plus the
> test names. So better just  $(AVOCADO_TESTS)
>

Ack. I'll send another version soon.

>
> Regards,
> Daniel
> --
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
>

Re: [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test

[Willian Rampazzo]

On Thu, Sep 23, 2021 at 5:27 AM Daniel P. Berrangé  wrote:
>
> On Wed, Sep 22, 2021 at 04:03:23PM -0300, Willian Rampazzo wrote:
> > Add instructions to the Acceptance tests section about running a
> > single test file or a test within the test file.
> >
> > Signed-off-by: Willian Rampazzo 
> > ---
> >  docs/devel/testing.rst | 14 ++
> >  1 file changed, 14 insertions(+)
> >
> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > index 4a0abbf23d..b03df34f7b 100644
> > --- a/docs/devel/testing.rst
> > +++ b/docs/devel/testing.rst
> > @@ -740,6 +740,20 @@ may be invoked by running:
> >
> >tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
> >
> > +It is also possible to run tests from a single file or a single test
> > +within a test file. To run tests from a single file within the build
> > +tree, use:
> > +
> > + .. code::
> > +
> > +  tests/venv/bin/avocado run tests/acceptance/$TESTFILE
>
> Before running this users may well need to run
>
>make check-venv
>
> we can't assume they have previously done "make check-acceptance"
> as they're possibly just reproducing a failure from gitlab
> CI locally, not running the whole suite.

Indeed!

>
> > +
> > +To run a single test within a test file, use:
> > +
> > + .. code::
> > +
> > +  tests/venv/bin/avocado run 
> > tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
> > +
>
> Valid test names are visible in the output from any previous execution
> of avocado, and can also be queried using
>
>   tests/venv/bin/avocado list tests/acceptance
>

Ack.

> >  Manual Installation
> >  ---
> >
> > --
> > 2.31.1
> >
>
> Regards,
> Daniel
> --
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
>

[PATCH v2 1/2] docs/devel/testing: add instruction to run a single acceptance test

[Willian Rampazzo]

Add instructions to the Acceptance tests section about running a
single test file or a test within the test file.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..b03df34f7b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -740,6 +740,20 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+It is also possible to run tests from a single file or a single test
+within a test file. To run tests from a single file within the build
+tree, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
+
 Manual Installation
 ---
 
-- 
2.31.1

[PATCH v2 2/2] tests/Makefile: add AVOCADO_TEST_FILES option to make check-acceptance

[Willian Rampazzo]

Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TEST_FILES environment variable.

Suggested-by: Daniel P. Berrangé 
Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 27 +++
 tests/Makefile.include |  5 -
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index b03df34f7b..785f82fc65 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -732,6 +732,33 @@ available.  On Debian and Ubuntu based systems, depending 
on the
 specific version, they may be on packages named ``python3-venv`` and
 ``python3-pip``.
 
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-acceptance`` command, set the ``AVOCADO_TEST_FILES``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TEST_FILES=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TEST_FILES='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TEST_FILES=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance AVOCADO_TEST_FILES='$FILEPATH1:$TESTCLASS1.$TESTNAME1 
$FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
 The scripts installed inside the virtual environment may be used
 without an "activation".  For instance, the Avocado test runner
 may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..1f1b4aa089 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+ifndef AVOCADO_TEST_FILES
+   AVOCADO_TEST_FILES=tests/acceptance
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
get-vm-images
 --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
 --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
 $(AVOCADO_TAGS) \
-$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
+$(if $(GITLAB_CI),,--failfast) $(AVOCADO_TEST_FILES), \
 "AVOCADO", "tests/acceptance")
 
 # Consolidated targets
-- 
2.31.1

[PATCH v2 0/2] docs/devel/testing: improvements on make check-acceptance

[Willian Rampazzo]

Improve the documentation about how to run a single test within a file
or all the tests from a file using the standalone avocado command.

Add a way to run specific tests using the `make check-acceptance` command.

GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/375517782

Changes from V1:
  - Rename TESTFILES to AVOCADO_TEST_FILES on patch 2
  - Add Suggested-by tag on patch 2

Willian Rampazzo (2):
  docs/devel/testing: add instruction to run a single acceptance test
  tests/Makefile: add AVOCADO_TEST_FILES option to make check-acceptance

 docs/devel/testing.rst | 41 +
 tests/Makefile.include |  5 -
 2 files changed, 45 insertions(+), 1 deletion(-)

-- 
2.31.1

Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance

[Willian Rampazzo]

On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
 wrote:
>
> On 9/22/21 21:03, Willian Rampazzo wrote:
> > Add the possibility of running all the tests from a single file, or
> > multiple files, running a single test within a file or multiple tests
> > within multiple files using `make check-acceptance` and the TESTFILES
> > environment variable.
> >
> > Signed-off-by: Willian Rampazzo 
> > ---
> >   docs/devel/testing.rst | 27 +++
> >   tests/Makefile.include |  5 -
> >   2 files changed, 31 insertions(+), 1 deletion(-)
>
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 6e16c05f10..82d7ef7a20 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> >   TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> >   TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> >   TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > +ifndef TESTFILES
> > + TESTFILES=tests/acceptance
> > +endif
> >   # Controls the output generated by Avocado when running tests.
> >   # Any number of command separated loggers are accepted.  For more
> >   # information please refer to "avocado --help".
> > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
> > get-vm-images
> >   --show=$(AVOCADO_SHOW) run 
> > --job-results-dir=$(TESTS_RESULTS_DIR) \
> >   --filter-by-tags-include-empty 
> > --filter-by-tags-include-empty-key \
> >   $(AVOCADO_TAGS) \
> > -$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> > +$(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
>
> Since this is Avocado specific, maybe call the variable
> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?

I don't see a problem with changing that to AVOCADO_TESTFILES. I was
trying to make things shorter and easy to remember. If the too-long
variable name is not a problem, I can change that.

>
> >   "AVOCADO", "tests/acceptance")
> >
> >   # Consolidated targets
> >
>

[PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance

[Willian Rampazzo]

Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the TESTFILES
environment variable.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 27 +++
 tests/Makefile.include |  5 -
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index b03df34f7b..f887463f2e 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -732,6 +732,33 @@ available.  On Debian and Ubuntu based systems, depending 
on the
 specific version, they may be on packages named ``python3-venv`` and
 ``python3-pip``.
 
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-acceptance`` command, set the ``TESTFILES``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+  make check-acceptance TESTFILES=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance TESTFILES='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+  make check-acceptance TESTFILES=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+  make check-acceptance TESTFILES='$FILEPATH1:$TESTCLASS1.$TESTNAME1 
$FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
 The scripts installed inside the virtual environment may be used
 without an "activation".  For instance, the Avocado test runner
 may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..82d7ef7a20 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+ifndef TESTFILES
+   TESTFILES=tests/acceptance
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
get-vm-images
 --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
 --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
 $(AVOCADO_TAGS) \
-$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
+$(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
 "AVOCADO", "tests/acceptance")
 
 # Consolidated targets
-- 
2.31.1

[PATCH 0/2] docs/devel/testing: improvements on make check-acceptance

[Willian Rampazzo]

Improve the documentation about how to run a single test within a file
or all the tests from a file using the standalone avocado command.

Add a way to run specific tests using the `make check-acceptance` command.

GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/375486996

Willian Rampazzo (2):
  docs/devel/testing: add instruction to run a single acceptance test
  tests/Makefile: add TESTFILES option to make check-acceptance

 docs/devel/testing.rst | 41 +
 tests/Makefile.include |  5 -
 2 files changed, 45 insertions(+), 1 deletion(-)

-- 
2.31.1

Re: [PATCH v3] tests/acceptance: allow control over tags during check-acceptance

[Willian Rampazzo]

Ping? Any good soul to send a pull request?

On Tue, Jul 13, 2021 at 6:03 PM Willian Rampazzo  wrote:
>
> Although it is possible to run a specific test using the avocado
> command-line, a user may want to use a specific tag while running the
> ``make check-acceptance`` during the development or debugging.
>
> This allows using the AVOCADO_TAGS environment variable where the user
> takes total control of which tests should run based on the tags defined.
>
> This also makes the check-acceptance command flexible to restrict tests
> based on tags while running on CI.
>
> e.g.:
>
> AVOCADO_TAGS="foo bar baz" make check-acceptance
>
> Signed-off-by: Willian Rampazzo 
> Tested-by: Wainer dos Santos Moschetta 
> Reviewed-by: Wainer dos Santos Moschetta 
> ---
>  docs/devel/testing.rst | 14 ++
>  tests/Makefile.include | 12 +---
>  2 files changed, 23 insertions(+), 3 deletions(-)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 4e42392810..a81b44cf18 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -760,6 +760,20 @@ in the current directory, tagged as "quick", run:
>
>avocado run -t quick .
>
> +It is also possible to run tests based on tags using the
> +``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
> +variable:
> +
> +.. code::
> +
> +   AVOCADO_TAGS=quick make check-acceptance
> +
> +Note that tags separated with commas have an AND behavior, while tags
> +separated by spaces have an OR behavior. For more information on Avocado
> +tags, see:
> +
> + 
> https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
> +
>  The ``avocado_qemu.Test`` base test class
>  -
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index e4dcb17329..0ce331f9a9 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # Any number of command separated loggers are accepted.  For more
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=app
> -AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
> +ifndef AVOCADO_TAGS
> +   AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \
> +$(filter 
> %-softmmu,$(TARGETS)))
> +else
> +   AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
> +endif
>
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> $(call quiet-command, \
> @@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) 
> get-vm-images
> $(call quiet-command, \
>  $(TESTS_VENV_DIR)/bin/python -m avocado \
>  --show=$(AVOCADO_SHOW) run 
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> ---filter-by-tags-include-empty 
> --filter-by-tags-include-empty-key \
> -$(AVOCADO_TAGS) \
> +$(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> +   --filter-by-tags-include-empty-key) \
> +$(AVOCADO_CMDLINE_TAGS) \
>  $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>  "AVOCADO", "tests/acceptance")
>
> --
> 2.31.1
>
>

[PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test

[Willian Rampazzo]

Add instructions to the Acceptance tests section about running a
single test file or a test within the test file.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/testing.rst | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..b03df34f7b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -740,6 +740,20 @@ may be invoked by running:
 
   tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
 
+It is also possible to run tests from a single file or a single test
+within a test file. To run tests from a single file within the build
+tree, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+  tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
+
 Manual Installation
 ---
 
-- 
2.31.1

Re: [RFC PATCH] tests/docker: add a debian-native image and make available

[Willian Rampazzo]

On Wed, Sep 22, 2021 at 12:15 PM Alex Bennée  wrote:
>
> This image is intended for building whatever the native versions of
> QEMU are for the host architecture. This will hopefully be an aid for
> 3rd parties who want to be able to build QEMU themselves without
> redoing all the dependencies themselves.
>
> We disable the registry because we currently don't have multi-arch
> support there.
>
> Signed-off-by: Alex Bennée 
> Cc: Anders Roxell 
> ---
>  tests/docker/Makefile.include |  4 ++
>  tests/docker/common.rc| 10 +++-
>  tests/docker/dockerfiles/debian-native.docker | 49 +++
>  3 files changed, 61 insertions(+), 2 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-native.docker
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 3b03763186..2f276cc4b2 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -145,6 +145,10 @@ docker-image-debian-s390x-cross: docker-image-debian10
>  docker-image-debian-sh4-cross: docker-image-debian10
>  docker-image-debian-sparc64-cross: docker-image-debian10
>
> +# The native build should never use the registry
> +docker-image-debian-native: DOCKER_REGISTRY=
> +
> +
>  #
>  # The build rule for hexagon-cross is special in so far for most of
>  # the time we don't want to build it. While dockers caching does avoid
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index c5cc33d366..e6f8cee0d6 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -12,8 +12,14 @@
>  # the top-level directory.
>
>  # This might be set by ENV of a docker container... it is always

if this is overridden when the user sets it, maybe remove the "always".

> -# overriden by TARGET_LIST if the user sets it.
> -DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"}
> +# overriden by TARGET_LIST if the user sets it. We special case

s/overriden/overridden/
Also, is there some word missing after "We"? Or you meant "With"?

> +# "none" to allow for other options like --disable-tcg to restrict the
> +# builds we eventually do.
> +if test "$DEF_TARGET_LIST" = "none"; then
> +DEF_TARGET_LIST=""
> +else
> +DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"}
> +fi
>
>  requires_binary()
>  {
> diff --git a/tests/docker/dockerfiles/debian-native.docker 
> b/tests/docker/dockerfiles/debian-native.docker
> new file mode 100644
> index 00..efd55cb6e0
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-native.docker
> @@ -0,0 +1,49 @@
> +#
> +# Docker Debian Native
> +#
> +# This this intended to build QEMU on native host systems. Debian is

s/This this/This/ or you meant "This is..."

> +# chosen due to the broadest range on supported host systems for QEMU.
> +#
> +# This docker target is based on the docker.io Debian Bullseye base
> +# image rather than QEMU's base because we would otherwise confuse the
> +# build grabbing stuff from the registry built for other
> +# architectures.
> +#
> +FROM docker.io/library/debian:bullseye-slim
> +MAINTAINER Alex Bennée 
> +
> +# Duplicate deb line as deb-src
> +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> 
> /etc/apt/sources.list
> +
> +# Install common build utilities
> +RUN apt update && \
> +DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
> +
> +RUN apt update && \
> +DEBIAN_FRONTEND=noninteractive eatmydata \
> +apt build-dep -yy --arch-only qemu
> +
> +RUN apt update && \
> +DEBIAN_FRONTEND=noninteractive eatmydata \
> +apt install -y --no-install-recommends \
> +cscope \
> +genisoimage \
> +exuberant-ctags \
> +global \
> +libbz2-dev \
> +liblzo2-dev \
> +libgcrypt20-dev \
> +libfdt-dev \
> +librdmacm-dev \
> +libsasl2-dev \
> +libsnappy-dev \
> +libvte-dev \
> +netcat-openbsd \
> +ninja-build \
> +openssh-client \
> +python3-numpy \
> +python3-opencv \
> +python3-venv
> +
> +ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS
> +ENV DEF_TARGET_LIST "none"
> --
> 2.30.2
>

Overall, looks good to me,

Acked-by: Willian Rampazzo 

Re: [PATCH 2/6] avocado_qemu: standardize supper() call following PEP3135

[Willian Rampazzo]

Ouch, s/supper/super/ in the email title.

On Mon, Sep 20, 2021 at 5:52 PM Willian Rampazzo  wrote:
>
> PEP3135 states when calling super(), there is no need to use arguments.
> This changes the calls on avocado_qemu to standardize according to
> PEP3135 and avoid warnings from linters.
>
> Signed-off-by: Willian Rampazzo 
> ---
>  tests/acceptance/avocado_qemu/__init__.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py 
> b/tests/acceptance/avocado_qemu/__init__.py
> index d9e1b32aa1..d2077d63cd 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -282,7 +282,7 @@ def fetch_asset(self, name,
>  asset_hash=None, algorithm=None,
>  locations=None, expire=None,
>  find_only=False, cancel_on_missing=True):
> -return super(Test, self).fetch_asset(name,
> +return super().fetch_asset(name,
>  asset_hash=asset_hash,
>  algorithm=algorithm,
>  locations=locations,
> @@ -470,7 +470,7 @@ def _set_distro(self):
>  self.distro.checksum = distro_checksum
>
>  def setUp(self, ssh_pubkey=None, network_device_type='virtio-net'):
> -super(LinuxTest, self).setUp()
> +super().setUp()
>  self._set_distro()
>  self.vm.add_args('-smp', '2')
>  self.vm.add_args('-m', '1024')
> --
> 2.31.1
>
>

[PATCH 6/6] avocado_qemu: fix inheritance order on LinuxTest class

[Willian Rampazzo]

Class hierarchy on Python is defined from right to left. Although the
current code is not harmful, let's fix it to avoid problems in the future.

Signed-off-by: Willian Rampazzo 
---
 tests/acceptance/avocado_qemu/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index 35318ce2a9..1841053e2c 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -424,7 +424,7 @@ def default_kernel_params(self):
 return self._info.get('kernel_params', None)
 
 
-class LinuxTest(Test, LinuxSSHMixIn):
+class LinuxTest(LinuxSSHMixIn, Test):
 """Facilitates having a cloud-image Linux based available.
 
 For tests that indend to interact with guests, this is a better choice
-- 
2.31.1

[PATCH 5/6] avocado_qemu: explicitly return None to avoid R1710

[Willian Rampazzo]

The linter is complaining the `pick_default_qemu_bin` is not explicitly
returning None. Fix it to explicitly return None and avoid R1710
inconsistent-return-statements.

Signed-off-by: Willian Rampazzo 
---
 tests/acceptance/avocado_qemu/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index c3613f9262..35318ce2a9 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -72,6 +72,7 @@ def pick_default_qemu_bin(arch=None):
   qemu_bin_relative_path)
 if is_readable_executable_file(qemu_bin_from_bld_dir_path):
 return qemu_bin_from_bld_dir_path
+return None
 
 
 def _console_interaction(test, success_message, failure_message,
-- 
2.31.1

[PATCH 4/6] avocado_qemu: tweak ssh connect method

[Willian Rampazzo]

The current implementation will crash if the connection fails as the
`time` module is not imported. This fixes the import problem and tweaks
the connection to wait progressively when the connection fails.

Signed-off-by: Willian Rampazzo 
---
 tests/acceptance/avocado_qemu/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index edb9ed7485..c3613f9262 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -13,6 +13,7 @@
 import shutil
 import sys
 import tempfile
+import time
 import uuid
 
 import avocado
@@ -305,8 +306,7 @@ def ssh_connect(self, username, credential, 
credential_is_key=True):
 self.ssh_session.connect()
 return
 except:
-time.sleep(4)
-pass
+time.sleep(i)
 self.fail('ssh connection timeout')
 
 def ssh_command(self, command):
-- 
2.31.1

[PATCH 3/6] avocado_qemu: fix import module based on isort

[Willian Rampazzo]

Signed-off-by: Willian Rampazzo 
---
 tests/acceptance/avocado_qemu/__init__.py | 18 +-
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index d2077d63cd..edb9ed7485 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -12,19 +12,13 @@
 import os
 import shutil
 import sys
-import uuid
 import tempfile
+import uuid
 
 import avocado
-
-from avocado.utils import cloudinit
-from avocado.utils import datadrainer
-from avocado.utils import network
-from avocado.utils import ssh
-from avocado.utils import vmimage
+from avocado.utils import cloudinit, datadrainer, network, ssh, vmimage
 from avocado.utils.path import find_command
 
-
 #: The QEMU build root directory.  It may also be the source directory
 #: if building from the source dir, but it's safer to use BUILD_DIR for
 #: that purpose.  Be aware that if this code is moved outside of a source
@@ -42,11 +36,9 @@
 sys.path.append(os.path.join(SOURCE_DIR, 'python'))
 
 from qemu.machine import QEMUMachine
-from qemu.utils import (
-get_info_usernet_hostfwd_port,
-kvm_available,
-tcg_available,
-)
+from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
+tcg_available)
+
 
 def is_readable_executable_file(path):
 return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
-- 
2.31.1

[PATCH 1/6] Acceptance Tests: add standard clean up at test tearDown()

[Willian Rampazzo]

From: Cleber Rosa 

The avocado.Test class, used as the basis of the avocado_qemu.Test
class, performs a clean of temporary directories up as part of its own
tearDown() implementation.

But the avocado_qemu.Test class is currently missing the same clean
up, as it implemented its own tearDown() method without resorting to
the upper class behavior.

This brings avocado_qemu.Test behavior in sync with the standard
avocado.Test behavior and prevents temporary directories from
cluttering the test results directory (unless instructed to do so with
Avocado's "--keep-tmp" option).

Reported-by: Peter Maydell 
Signed-off-by: Cleber Rosa 
[willianr: respin to new Python super format]
Signed-off-by: Willian Rampazzo 

---
v2
  - adjust super call to conform with PEP3135
---
 tests/acceptance/avocado_qemu/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index 2c4fef3e14..d9e1b32aa1 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -276,6 +276,7 @@ def tearDown(self):
 for vm in self._vms.values():
 vm.shutdown()
 self._sd = None
+super().tearDown()
 
 def fetch_asset(self, name,
 asset_hash=None, algorithm=None,
-- 
2.31.1

[PATCH 0/6] avocado_qemu: small fixes and tweaks

[Willian Rampazzo]

Standardize the super() calls according to PEP3135; fix import order based on
isort; fix and tweak the ssh `connect` method; explicitly return None on
`pick_default_qemu_bin` function to avoid linters R1710; and fix inheritance
order on `LinuxTest` to avoid future problems.

Note: borrowed 
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05117.html
  for completeness.

Cleber Rosa (1):
  Acceptance Tests: add standard clean up at test tearDown()

Willian Rampazzo (5):
  avocado_qemu: standardize supper() call following PEP3135
  avocado_qemu: fix import module based on isort
  avocado_qemu: tweak ssh connect method
  avocado_qemu: explicitly return None to avoid R1710
  avocado_qemu: fix inheritance order on LinuxTest class

 tests/acceptance/avocado_qemu/__init__.py | 30 +--
 1 file changed, 12 insertions(+), 18 deletions(-)

-- 
2.31.1

[PATCH 2/6] avocado_qemu: standardize supper() call following PEP3135

[Willian Rampazzo]

PEP3135 states when calling super(), there is no need to use arguments.
This changes the calls on avocado_qemu to standardize according to
PEP3135 and avoid warnings from linters.

Signed-off-by: Willian Rampazzo 
---
 tests/acceptance/avocado_qemu/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index d9e1b32aa1..d2077d63cd 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -282,7 +282,7 @@ def fetch_asset(self, name,
 asset_hash=None, algorithm=None,
 locations=None, expire=None,
 find_only=False, cancel_on_missing=True):
-return super(Test, self).fetch_asset(name,
+return super().fetch_asset(name,
 asset_hash=asset_hash,
 algorithm=algorithm,
 locations=locations,
@@ -470,7 +470,7 @@ def _set_distro(self):
 self.distro.checksum = distro_checksum
 
 def setUp(self, ssh_pubkey=None, network_device_type='virtio-net'):
-super(LinuxTest, self).setUp()
+super().setUp()
 self._set_distro()
 self.vm.add_args('-smp', '2')
 self.vm.add_args('-m', '1024')
-- 
2.31.1

Re: [PATCH 0/2] Acceptance Tests: clean up of temporary dirs and MAINTAINERS entry

[Willian Rampazzo]

On Mon, Sep 20, 2021 at 10:59 AM Peter Maydell  wrote:
>
> On Tue, 3 Aug 2021 at 20:35, Cleber Rosa  wrote:
> >
> > This is a reply to an issue[1] reported by Peter, and while at it, an
> > update of the MAINTAINERS entry so other people now the status and
> > where to go for help with regards to the acceptance tests'
> > infrastructure.
> >
> > [1] https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg00349.html
>
> Ping? I was reminded of this today, and it looks like
> the acceptance tests are still wasting gigabytes of
> everybody's disk space with non-cleaned-up tempfiles...

Cleber will be out until next Wed. I took the bug fix patch and
changed it based on Wainer's comment:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg05117.html

>
> -- PMM
>

[PATCH v2] Acceptance Tests: add standard clean up at test tearDown()

[Willian Rampazzo]

From: Cleber Rosa 

The avocado.Test class, used as the basis of the avocado_qemu.Test
class, performs a clean of temporary directories up as part of its own
tearDown() implementation.

But the avocado_qemu.Test class is currently missing the same clean
up, as it implemented its own tearDown() method without resorting to
the upper class behavior.

This brings avocado_qemu.Test behavior in sync with the standard
avocado.Test behavior and prevents temporary directories from
cluttering the test results directory (unless instructed to do so with
Avocado's "--keep-tmp" option).

Reported-by: Peter Maydell 
Signed-off-by: Cleber Rosa 
[willianr: respin to new Python super format]
Signed-off-by: Willian Rampazzo 

---
v2
  - adjust super call to conform with PEP3135
---
 tests/acceptance/avocado_qemu/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/acceptance/avocado_qemu/__init__.py 
b/tests/acceptance/avocado_qemu/__init__.py
index 2c4fef3e14..d9e1b32aa1 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -276,6 +276,7 @@ def tearDown(self):
 for vm in self._vms.values():
 vm.shutdown()
 self._sd = None
+super().tearDown()
 
 def fetch_asset(self, name,
 asset_hash=None, algorithm=None,
-- 
2.31.1

Re: how do I run a single acceptance test?

[Willian Rampazzo]

Hi, Peter,

On Mon, Sep 20, 2021 at 7:56 AM Peter Maydell  wrote:
>
> Hi; I'm trying to bisect a problem that's causing an acceptance
> test failure, but docs/devel/testing.rst doesn't have any info about
> how to run just one acceptance test rather than the whole lot
> via 'make check-acceptance'. Is this possible?
>
I know this was discussed on IRC, but just for documenting on the
mailing list, you can use the following, as an example, to run all the
tests from file boot_linux_console.py:

build/tests/venv/bin/avocado run build/tests/acceptance/boot_linux_console.py

Or, if you want to run a specific test:

build/tests/venv/bin/avocado run
build/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d

I'll send an update to the documentation explaining the above.

Regards,

Willian


> (It's pretty slow to have to run the whole thing, because we still
> have those horrifically slow BootLinuxAarch64 tests in there which
> eat the bulk of the time and are not interesting for this bisection.)
>
> thanks
> -- PMM
>

Re: [PATCH 1/2] python: Update for pylint 2.10

[Willian Rampazzo]

On Thu, Sep 16, 2021 at 3:29 PM John Snow  wrote:
>
> A few new annoyances. Of note is the new warning for an unspecified
> encoding when opening a text file, which actually does indicate a
> potentially real problem; see
> https://www.python.org/dev/peps/pep-0597/#motivation
>
> Use LC_CTYPE to determine an encoding to use for interpreting QEMU's
> terminal output. Note that Python states: "language code and encoding
> may be None if their values cannot be determined" -- use a platform
> default as a backup.
>
> Notes: Passing encoding=None will generate a suppressed warning on
> Python 3.10+ that 'None' should not be passed as the encoding
> argument. This behavior may be deprecated in the future and the default
> switched to be a ubiquitous UTF-8. Opting in to the locale default will
> be done by passing the encoding 'locale', but that isn't available in
> 3.6 through 3.9. Presumably this warning will be unsuppressed some time
> prior to the actual switch and we can re-investigate these issues at
> that time if necessary.
>
> Signed-off-by: John Snow 
> ---
>  python/qemu/machine/machine.py | 7 ++-
>  python/setup.cfg       | 1 +
>  2 files changed, 7 insertions(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 2/2] python: pylint 2.11 support

[Willian Rampazzo]

On Thu, Sep 16, 2021 at 3:24 PM John Snow  wrote:
>
> We're not ready to enforce f-strings everywhere, so just silence this
> new warning.

Oh, boy! We are good, thanks :)

>
> Signed-off-by: John Snow 
> ---
>  python/setup.cfg | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [RFC 1/1] docs/deve/ci-plan: define a high-level plan for the QEMU GitLab CI

[Willian Rampazzo]

On Wed, Sep 15, 2021 at 11:07 AM Daniel P. Berrangé  wrote:
>
> On Wed, Sep 15, 2021 at 10:51:56AM -0300, Willian Rampazzo wrote:
> > On Wed, Sep 15, 2021 at 6:00 AM Daniel P. Berrangé  
> > wrote:
> > >
> > > On Tue, Sep 14, 2021 at 03:48:30PM -0300, Willian Rampazzo wrote:
> > > > This adds a high-level plan for the QEMU GitLab CI based on use cases.
> > > > The idea is to have a base for evolving the QEMU CI. It sets high-level
> > > > characteristics for the QEMU CI use cases, which helps guide its
> > > > development.
> > > >
> > > > Signed-off-by: Willian Rampazzo 
> > > > ---
> > > >  docs/devel/ci-plan.rst | 77 ++
> > > >  docs/devel/ci.rst  |  1 +
> > > >  2 files changed, 78 insertions(+)
> > > >  create mode 100644 docs/devel/ci-plan.rst
> > > >
> > > > diff --git a/docs/devel/ci-plan.rst b/docs/devel/ci-plan.rst
> > > > new file mode 100644
> > > > index 00..5e95b6bcea
> > > > --- /dev/null
> > > > +++ b/docs/devel/ci-plan.rst
> > > > @@ -0,0 +1,77 @@
> > > > +The GitLab CI structure
> > > > +===
> > > > +
> > > > +This section describes the current state of the QEMU GitLab CI and the
> > > > +high-level plan for its future.
> > > > +
> > > > +Current state
> > > > +-
> > > > +
> > > > +The mainstream QEMU project considers the GitLab CI its primary CI 
> > > > system.
> > > > +Currently, it runs 120+ jobs, where ~36 are container build jobs, 69 
> > > > are QEMU
> > > > +build jobs, ~22 are test jobs, 1  is a web page deploy job, and 1 is an
> > > > +external job covering Travis jobs execution.
> > > > +
> > > > +In the current state, every push a user does to its fork runs most of 
> > > > the jobs
> > > > +compared to the jobs running on the main repository. The exceptions 
> > > > are the
> > > > +acceptance tests jobs, which run automatically on the main repository 
> > > > only.
> > > > +Running most of the jobs in the user's fork or the main repository is 
> > > > not
> > > > +viable. The job number tends to increase, becoming impractical to run 
> > > > all of
> > > > +them on every single push.
> > > > +
> > > > +Future of QEMU GitLab CI
> > > > +
> > > > +
> > > > +Following is a proposal to establish a high-level plan and set the
> > > > +characteristics for the QEMU GitLab CI. The idea is to organize the CI 
> > > > by use
> > > > +cases, avoid wasting resources and CI minutes, anticipating the time 
> > > > GitLab
> > > > +starts to enforce minutes limits soon.
> > > > +
> > > > +Use cases
> > > > +^
> > > > +
> > > > +Below is a list of the most common use cases for the QEMU GitLab CI.
> > > > +
> > > > +Gating
> > > > +""""""
> > > > +
> > > > +The gating set of jobs runs on the maintainer's pull requests when the 
> > > > project
> > > > +leader pushes them to the staging branch of the project. The gating CI 
> > > > pipeline
> > > > +has the following characteristics:
> > > > +
> > > > + * Jobs tagged as gating run as part of the gating CI pipeline;
> > > > + * The gating CI pipeline consists of stable jobs;
> > > > + * The execution duration of the gating CI pipeline should, as much as 
> > > > possible,
> > > > +   have an upper bound limit of 2 hours.
> > > > +
> > > > +Developers
> > > > +""""""""""
> > > > +
> > > > +A developer working on a new feature or fixing an issue may want to 
> > > > run/propose
> > > > +a specific set of tests. Those tests may, eventually, benefit other 
> > > > developers.
> > > > +A developer CI pipeline has the following characteristics:
> > > > +
> > > > + * It is easy to run current tests available in the project;
> > > > + * It is easy to add new tests or remove unneeded tests;
> > > > + * It is flexible enough to allow changes in the current jobs.
> > > &g

Re: [PATCH 2/2] gitlab: fix passing of TEST_TARGETS env to cirrus

[Willian Rampazzo]

On Wed, Sep 15, 2021 at 9:55 AM Daniel P. Berrangé  wrote:
>
> A typo meant the substitution would not work, and the placeholder in the
> target file didn't even exist.
>
> The result was that tests were never run on the FreeBSD and macOS jobs,
> only a basic build.
>
> Signed-off-by: Daniel P. Berrangé 
> ---
>  .gitlab-ci.d/cirrus.yml   | 2 +-
>  .gitlab-ci.d/cirrus/build.yml | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 2/2] gitlab: Add cross-riscv64-system, cross-riscv64-user

[Willian Rampazzo]

On Tue, Sep 14, 2021 at 3:58 PM Richard Henderson
 wrote:
>
> Signed-off-by: Richard Henderson 
> ---
>  .gitlab-ci.d/crossbuilds.yml | 14 ++
>  1 file changed, 14 insertions(+)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH] gitlab-ci: Mark manual-only jobs as allow_failure

[Willian Rampazzo]

On Wed, Sep 15, 2021 at 9:34 AM Peter Maydell  wrote:
>
> If a gitlab CI job is marked as manual-only but is not marked
> as allow_failure, then gitlab considers that the pipeline is
> "blocked" until the job has been manually triggered. We need
> to mark these manual-only jobs as also allow_failure: true
> so that gitlab doesn't insist that they have run before it
> will consider the pipeline to be complete.
>
> Fixes: 4c9af1ea1457782cf0adb29
> Signed-off-by: Peter Maydell 
> ---
>  .gitlab-ci.d/custom-runners.yml | 16 
>  1 file changed, 16 insertions(+)
>

Reviewed-by: Willian Rampazzo 

Re: [RFC 1/1] docs/deve/ci-plan: define a high-level plan for the QEMU GitLab CI

[Willian Rampazzo]

On Wed, Sep 15, 2021 at 6:00 AM Daniel P. Berrangé  wrote:
>
> On Tue, Sep 14, 2021 at 03:48:30PM -0300, Willian Rampazzo wrote:
> > This adds a high-level plan for the QEMU GitLab CI based on use cases.
> > The idea is to have a base for evolving the QEMU CI. It sets high-level
> > characteristics for the QEMU CI use cases, which helps guide its
> > development.
> >
> > Signed-off-by: Willian Rampazzo 
> > ---
> >  docs/devel/ci-plan.rst | 77 ++
> >  docs/devel/ci.rst  |  1 +
> >  2 files changed, 78 insertions(+)
> >  create mode 100644 docs/devel/ci-plan.rst
> >
> > diff --git a/docs/devel/ci-plan.rst b/docs/devel/ci-plan.rst
> > new file mode 100644
> > index 00..5e95b6bcea
> > --- /dev/null
> > +++ b/docs/devel/ci-plan.rst
> > @@ -0,0 +1,77 @@
> > +The GitLab CI structure
> > +===
> > +
> > +This section describes the current state of the QEMU GitLab CI and the
> > +high-level plan for its future.
> > +
> > +Current state
> > +-
> > +
> > +The mainstream QEMU project considers the GitLab CI its primary CI system.
> > +Currently, it runs 120+ jobs, where ~36 are container build jobs, 69 are 
> > QEMU
> > +build jobs, ~22 are test jobs, 1  is a web page deploy job, and 1 is an
> > +external job covering Travis jobs execution.
> > +
> > +In the current state, every push a user does to its fork runs most of the 
> > jobs
> > +compared to the jobs running on the main repository. The exceptions are the
> > +acceptance tests jobs, which run automatically on the main repository only.
> > +Running most of the jobs in the user's fork or the main repository is not
> > +viable. The job number tends to increase, becoming impractical to run all 
> > of
> > +them on every single push.
> > +
> > +Future of QEMU GitLab CI
> > +
> > +
> > +Following is a proposal to establish a high-level plan and set the
> > +characteristics for the QEMU GitLab CI. The idea is to organize the CI by 
> > use
> > +cases, avoid wasting resources and CI minutes, anticipating the time GitLab
> > +starts to enforce minutes limits soon.
> > +
> > +Use cases
> > +^
> > +
> > +Below is a list of the most common use cases for the QEMU GitLab CI.
> > +
> > +Gating
> > +""""""
> > +
> > +The gating set of jobs runs on the maintainer's pull requests when the 
> > project
> > +leader pushes them to the staging branch of the project. The gating CI 
> > pipeline
> > +has the following characteristics:
> > +
> > + * Jobs tagged as gating run as part of the gating CI pipeline;
> > + * The gating CI pipeline consists of stable jobs;
> > + * The execution duration of the gating CI pipeline should, as much as 
> > possible,
> > +   have an upper bound limit of 2 hours.
> > +
> > +Developers
> > +""""""""""
> > +
> > +A developer working on a new feature or fixing an issue may want to 
> > run/propose
> > +a specific set of tests. Those tests may, eventually, benefit other 
> > developers.
> > +A developer CI pipeline has the following characteristics:
> > +
> > + * It is easy to run current tests available in the project;
> > + * It is easy to add new tests or remove unneeded tests;
> > + * It is flexible enough to allow changes in the current jobs.
> > +
> > +Maintainers
> > +"""""""""""
> > +
> > +When accepting developers' patches, a maintainer may want to run a specific
> > +test set. A maintainer CI pipeline has the following characteristics:
> > +
> > + * It consists of tests that are valuable for the subsystem;
> > + * It is easy to run a set of specific tests available in the project;
> > + * It is easy to add new tests or remove unneeded tests.
>
>
> Neither of these describe why I use CI as a developer and/or subsys
> maintainer.
>
> My desire with using CI is to (as close as possible) be able to
> execute the exact same  set of tests that will be run by gating CI
> on pull requests.

I totally understand your desire and I think it is valid.

What I'm trying with this proposal is the same strategy we used when
we started planning for the gating CI. The decision was to start
small. Today the CI grew and we don´t have a so called gating CI yet,
but a bunch of jobs that runs on staging branch, some needing
reevaluation whet

Re: [PATCH] gitlab-ci: Make more custom runner jobs manual, and don't allow failure

[Willian Rampazzo]

On Tue, Sep 14, 2021 at 4:18 PM Peter Maydell  wrote:
>
> On Mon, 13 Sept 2021 at 11:19, Peter Maydell  wrote:
> >
> > Currently we define a lot of jobs for our custom runners:
> > for both aarch64 and s390x we have
> >  - all-linux-static
> >  - all
> >  - alldbg
> >  - clang (manual)
> >  - tci
> >  - notcg (manual)
> >
> > This is overkill.  The main reason to run on these hosts is to get
> > coverage for the host architecture; we can leave the handling of
> > differences like debug vs non-debug to the x86 CI jobs.
> >
> > The jobs are also generally running OK; they occasionally fail due to
> > timeouts, which is likely because we're overloading the machine by
> > asking it to run 4 CI jobs at once plus the ad-hoc CI.
> >
> > Remove the 'allow_failure' tag from all these jobs, and switch the
> > s390x-alldbg, aarch64-all, s390x-tci and aarch64-tci jobs to manual.
> > This will let us make the switch for s390x and aarch64 hosts from
> > the ad-hoc CI to gitlab.
> >
> > Signed-off-by: Peter Maydell 
>
> It looks like this change has resulted in pipelines ending
> up in a "blocked" state:
>
> https://gitlab.com/qemu-project/qemu/-/pipelines
>
> I'm not sure why this is -- is it perhaps because there were
> other jobs that depended on the now-manual-only jobs ?
> Can somebody suggest a fix ?

There are a couple of issues listed on GitLab main repository
reporting the same behavior. When you remove the allow_failure: true,
it is set to the default, false. As other stages may depend on that
job and it is now set to not allow failure, the pipeline is marked as
blocked.

Some people reported setting the jobs to allow_failure: true "solved"
the problem.

References:
https://gitlab.com/gitlab-org/gitlab/-/issues/39534
https://gitlab.com/gitlab-org/gitlab/-/issues/31415
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/66602

>
> thanks
> -- PMM
>

[RFC 1/1] docs/deve/ci-plan: define a high-level plan for the QEMU GitLab CI

[Willian Rampazzo]

This adds a high-level plan for the QEMU GitLab CI based on use cases.
The idea is to have a base for evolving the QEMU CI. It sets high-level
characteristics for the QEMU CI use cases, which helps guide its
development.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/ci-plan.rst | 77 ++
 docs/devel/ci.rst  |  1 +
 2 files changed, 78 insertions(+)
 create mode 100644 docs/devel/ci-plan.rst

diff --git a/docs/devel/ci-plan.rst b/docs/devel/ci-plan.rst
new file mode 100644
index 00..5e95b6bcea
--- /dev/null
+++ b/docs/devel/ci-plan.rst
@@ -0,0 +1,77 @@
+The GitLab CI structure
+===
+
+This section describes the current state of the QEMU GitLab CI and the
+high-level plan for its future.
+
+Current state
+-
+
+The mainstream QEMU project considers the GitLab CI its primary CI system.
+Currently, it runs 120+ jobs, where ~36 are container build jobs, 69 are QEMU
+build jobs, ~22 are test jobs, 1  is a web page deploy job, and 1 is an
+external job covering Travis jobs execution.
+
+In the current state, every push a user does to its fork runs most of the jobs
+compared to the jobs running on the main repository. The exceptions are the
+acceptance tests jobs, which run automatically on the main repository only.
+Running most of the jobs in the user's fork or the main repository is not
+viable. The job number tends to increase, becoming impractical to run all of
+them on every single push.
+
+Future of QEMU GitLab CI
+
+
+Following is a proposal to establish a high-level plan and set the
+characteristics for the QEMU GitLab CI. The idea is to organize the CI by use
+cases, avoid wasting resources and CI minutes, anticipating the time GitLab
+starts to enforce minutes limits soon.
+
+Use cases
+^
+
+Below is a list of the most common use cases for the QEMU GitLab CI.
+
+Gating
+""""""
+
+The gating set of jobs runs on the maintainer's pull requests when the project
+leader pushes them to the staging branch of the project. The gating CI pipeline
+has the following characteristics:
+
+ * Jobs tagged as gating run as part of the gating CI pipeline;
+ * The gating CI pipeline consists of stable jobs;
+ * The execution duration of the gating CI pipeline should, as much as 
possible,
+   have an upper bound limit of 2 hours.
+
+Developers
+""""""""""
+
+A developer working on a new feature or fixing an issue may want to run/propose
+a specific set of tests. Those tests may, eventually, benefit other developers.
+A developer CI pipeline has the following characteristics:
+
+ * It is easy to run current tests available in the project;
+ * It is easy to add new tests or remove unneeded tests;
+ * It is flexible enough to allow changes in the current jobs.
+
+Maintainers
+"""""""""""
+
+When accepting developers' patches, a maintainer may want to run a specific
+test set. A maintainer CI pipeline has the following characteristics:
+
+ * It consists of tests that are valuable for the subsystem;
+ * It is easy to run a set of specific tests available in the project;
+ * It is easy to add new tests or remove unneeded tests.
+
+Scheduled / periodic pipelines
+""""""""""""""""""""""""""""""
+
+The scheduled CI pipeline runs periodically on the master/main branch of the
+project. It covers as many jobs as needed or allowed by the execution duration
+of GitLab CI. The main idea of this pipeline is to run jobs that are not part
+of any other use cases due to some limitations, like execution duration, or
+flakiness. This pipeline may be helpful, for example, to collect test/job
+statistics or to define test/job stability. The scheduled CI pipeline should
+not act as a gating CI pipeline.
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 8d95247188..c9a43f997d 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -9,5 +9,6 @@ found at::
https://wiki.qemu.org/Testing/CI
 
 .. include:: ci-definitions.rst
+.. include:: ci-plan.rst
 .. include:: ci-jobs.rst
 .. include:: ci-runners.rst
-- 
2.31.1

[RFC 0/1] docs/deve/ci-plan: define a high-level plan for the QEMU GitLab CI

[Willian Rampazzo]

This adds a high-level plan for the QEMU GitLab CI based on use cases.
The idea is to have a base for evolving the QEMU CI. It sets high-level
characteristics for the QEMU CI use cases, which helps guide its
development.

There is an opportunity to discuss the high-level QEMU CI plan and some of
the implementation details during the KVM Forum.

Willian Rampazzo (1):
  docs/deve/ci-plan: define a high-level plan for the QEMU GitLab CI

 docs/devel/ci-plan.rst | 77 ++
 docs/devel/ci.rst  |  1 +
 2 files changed, 78 insertions(+)
 create mode 100644 docs/devel/ci-plan.rst

-- 
2.31.1

Re: [PATCH] gitlab-ci: Make more custom runner jobs manual, and don't allow failure

[Willian Rampazzo]

On Mon, Sep 13, 2021 at 7:22 AM Peter Maydell  wrote:
>
> Currently we define a lot of jobs for our custom runners:
> for both aarch64 and s390x we have
>  - all-linux-static
>  - all
>  - alldbg
>  - clang (manual)
>  - tci
>  - notcg (manual)
>
> This is overkill.  The main reason to run on these hosts is to get
> coverage for the host architecture; we can leave the handling of
> differences like debug vs non-debug to the x86 CI jobs.
>
> The jobs are also generally running OK; they occasionally fail due to
> timeouts, which is likely because we're overloading the machine by
> asking it to run 4 CI jobs at once plus the ad-hoc CI.
>
> Remove the 'allow_failure' tag from all these jobs, and switch the
> s390x-alldbg, aarch64-all, s390x-tci and aarch64-tci jobs to manual.
> This will let us make the switch for s390x and aarch64 hosts from
> the ad-hoc CI to gitlab.
>
> Signed-off-by: Peter Maydell 
> ---
>  .gitlab-ci.d/custom-runners.yml | 20 --------
>  1 file changed, 8 insertions(+), 12 deletions(-)
>

Reviewed-by: Willian Rampazzo 

[PATCH v2 1/1] docs: add definitions of terms for CI/testing

[Willian Rampazzo]

To understand the current state of QEMU CI/testing and have a base to
discuss the plans for the future, it is important to define some usual
terms. This patch defines the terms for "Automated tests", "Unit
testing", "Functional testing", "System testing", "Flaky tests",
"Gating", and "Continuous Integration".

Signed-off-by: Willian Rampazzo 
Reviewed-by: Philippe Mathieu-Daudé 
---
 docs/devel/ci-definitions.rst | 121 ++
 docs/devel/ci.rst |   1 +
 2 files changed, 122 insertions(+)
 create mode 100644 docs/devel/ci-definitions.rst

diff --git a/docs/devel/ci-definitions.rst b/docs/devel/ci-definitions.rst
new file mode 100644
index 00..32e22ff468
--- /dev/null
+++ b/docs/devel/ci-definitions.rst
@@ -0,0 +1,121 @@
+Definition of terms
+===
+
+This section defines the terms used in this document and correlates them with
+what is currently used on QEMU.
+
+Automated tests
+---
+
+An automated test is written on a test framework using its generic test
+functions/classes. The test framework can run the tests and report their
+success or failure [1]_.
+
+An automated test has essentially three parts:
+
+1. The test initialization of the parameters, where the expected parameters,
+   like inputs and expected results, are set up;
+2. The call to the code that should be tested;
+3. An assertion, comparing the result from the previous call with the expected
+   result set during the initialization of the parameters. If the result
+   matches the expected result, the test has been successful; otherwise, it has
+   failed.
+
+Unit testing
+
+
+A unit test is responsible for exercising individual software components as a
+unit, like interfaces, data structures, and functionality, uncovering errors
+within the boundaries of a component. The verification effort is in the
+smallest software unit and focuses on the internal processing logic and data
+structures. A test case of unit tests should be designed to uncover errors due
+to erroneous computations, incorrect comparisons, or improper control flow 
[2]_.
+
+On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
+
+Functional testing
+--
+
+A functional test focuses on the functional requirement of the software.
+Deriving sets of input conditions, the functional tests should fully exercise
+all the functional requirements for a program. Functional testing is
+complementary to other testing techniques, attempting to find errors like
+incorrect or missing functions, interface errors, behavior errors, and
+initialization and termination errors [3]_.
+
+On QEMU, functional testing is represented by the 'check-qtest' target from
+'make'.
+
+System testing
+--
+
+System tests ensure all application elements mesh properly while the overall
+functionality and performance are achieved [4]_. Some or all system components
+are integrated to create a complete system to be tested as a whole. System
+testing ensures that components are compatible, interact correctly, and
+transfer the right data at the right time across their interfaces. As system
+testing focuses on interactions, use case-based testing is a practical approach
+to system testing [5]_. Note that, in some cases, system testing may require
+interaction with third-party software, like operating system images, databases,
+networks, and so on.
+
+On QEMU, system testing is represented by the 'check-acceptance' target from
+'make'.
+
+Flaky tests
+---
+
+A flaky test is defined as a test that exhibits both a passing and a failing
+result with the same code on different runs. Some usual reasons for an
+intermittent/flaky test are async wait, concurrency, and test order dependency
+[6]_.
+
+Gating
+--
+
+A gate restricts the move of code from one stage to another on a
+test/deployment pipeline. The step move is granted with approval. The approval
+can be a manual intervention or a set of tests succeeding [7]_.
+
+On QEMU, the gating process happens during the pull request. The approval is
+done by the project leader running its own set of tests. The pull request gets
+merged when the tests succeed.
+
+Continuous Integration (CI)
+---
+
+Continuous integration (CI) requires the builds of the entire application and
+the execution of a comprehensive set of automated tests every time there is a
+need to commit any set of changes [8]_. The automated tests can be composed of
+the unit, functional, system, and other tests.
+
+Keynotes about continuous integration (CI) [9]_:
+
+1. System tests may depend on external software (operating system images,
+   firmware, database, network).
+2. It may take a long time to build and test. It may be impractical to build
+   the system being developed several times per day.
+3. If the development platform is different from the target platform, it 

[PATCH v2 0/1] docs: add definitions of terms for CI/testing

[Willian Rampazzo]

To understand the current state of QEMU CI/testing and have a base to
discuss the plans for the future, it is important to define some usual
terms. This patch defines the terms for "Automated tests", "Unit
testing", "Functional testing", "System testing", "Flaky tests",
"Gating", and "Continuous Integration".

Signed-off-by: Willian Rampazzo 

- Changes from v1:
  - Add Philippe R-b
  - Remove reference to Daniel's document split patch.

Willian Rampazzo (1):
  docs: add definitions of terms for CI/testing

 docs/devel/ci-definitions.rst | 121 ++
 docs/devel/ci.rst |   1 +
 2 files changed, 122 insertions(+)
 create mode 100644 docs/devel/ci-definitions.rst

-- 
2.31.1

Re: [PATCH v2 0/2] hw/arm/raspi: Remove deprecated raspi2/raspi3 aliases

[Willian Rampazzo]

Hi, Phil,

On Thu, Aug 26, 2021 at 1:49 PM Philippe Mathieu-Daudé  wrote:
>
> Hi Peter,
>
> On 7/9/21 6:00 PM, Peter Maydell wrote:
> > On Fri, 9 Jul 2021 at 16:33, Peter Maydell  wrote:
> >>
> >> On Thu, 8 Jul 2021 at 15:55, Philippe Mathieu-Daudé  
> >> wrote:
> >>>
> >>> Since v1:
> >>> - renamed tests (Peter)
> >>>
> >>> Philippe Mathieu-Daudé (2):
> >>>   tests: Remove uses of deprecated raspi2/raspi3 machine names
> >>>   hw/arm/raspi: Remove deprecated raspi2/raspi3 aliases
> >>
> >>
> >>
> >> Applied to target-arm.next, thanks.
> >
> > I found that this seems to break 'make check':
> >
> >  (09/52) 
> > tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_initrd:
> > ERROR: Could not perform graceful shutdown (40.38 s)
>
> I can not reproduce. Maybe something got changed in Python/Avocado
> since... I'm going to simply respin (updating 6.1 -> 6.2).

I also could not reproduce. I checked and nothing changed on the
Avocado side. The version is still the same on the QEMU side.

>
> >
> > Dropped from target-arm.next.
> >
> > thanks
> > -- PMM
> >
>

Re: [PATCH v3 1/2] tests: Remove uses of deprecated raspi2/raspi3 machine names

[Willian Rampazzo]

On Fri, Aug 27, 2021 at 3:14 AM Philippe Mathieu-Daudé  wrote:
>
> Commit 155e1c82ed0 deprecated the raspi2/raspi3 machine names.
> Use the recommended new names: raspi2b and raspi3b.
>
> Signed-off-by: Philippe Mathieu-Daudé 
> ---
>  docs/devel/qgraph.rst   | 38 -
>  tests/qtest/libqos/qgraph.h |  6 ++--
>  tests/qtest/libqos/qgraph_internal.h|  2 +-
>  tests/qtest/boot-serial-test.c  |  2 +-
>  tests/qtest/libqos/arm-raspi2-machine.c |  8 +++---
>  tests/unit/test-qgraph.c|  2 +-
>  tests/acceptance/boot_linux_console.py  |  6 ++--
>  7 files changed, 32 insertions(+), 32 deletions(-)
>

Reviewed-by: Willian Rampazzo 

Re: [PATCH 1/2] docs: split the CI docs into two files

[Willian Rampazzo]

On Thu, Aug 12, 2021 at 3:04 PM Daniel P. Berrangé  wrote:
>
> This splits the CI docs into one file talking about job setup and usage
> and another file describing provisioning of custom runners.
>
> Signed-off-by: Daniel P. Berrangé 
> ---
>  docs/devel/ci-jobs.rst|  40 ++
>  docs/devel/ci-runners.rst | 117 
>  docs/devel/ci.rst | 159 +-
>  3 files changed, 159 insertions(+), 157 deletions(-)
>  create mode 100644 docs/devel/ci-jobs.rst
>  create mode 100644 docs/devel/ci-runners.rst
>

Reviewed-by: Willian Rampazzo 

[PATCH 1/2] docs: split the CI docs into two files

[Willian Rampazzo]

From: Daniel P. Berrangé 

This splits the CI docs into one file talking about job setup and usage
and another file describing provisioning of custom runners.

Signed-off-by: Daniel P. Berrangé 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20210812180403.4129067-2-berra...@redhat.com>
---
 docs/devel/ci-jobs.rst|  40 ++
 docs/devel/ci-runners.rst | 117 
 docs/devel/ci.rst | 159 +-
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
new file mode 100644
index 00..9cd9819786
--- /dev/null
+++ b/docs/devel/ci-jobs.rst
@@ -0,0 +1,40 @@
+Custom CI/CD variables
+==
+
+QEMU CI pipelines can be tuned by setting some CI environment variables.
+
+Set variable globally in the user's CI namespace
+
+
+Variables can be set globally in the user's CI namespace setting.
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
+
+Set variable manually when pushing a branch or tag to the user's repository
+---
+
+Variables can be set manually when pushing a branch or tag, using
+git-push command line arguments.
+
+Example setting the QEMU_CI_EXAMPLE_VAR variable:
+
+.. code::
+
+   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
+
+For further information about how to set these variables, please refer to::
+
+  
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
+
+Here is a list of the most used variables:
+
+QEMU_CI_AVOCADO_TESTING
+~~~
+By default, tests using the Avocado framework are not run automatically in
+the pipelines (because multiple artifacts have to be downloaded, and if
+these artifacts are not already cached, downloading them make the jobs
+reach the timeout limit). Set this variable to have the tests using the
+Avocado framework run automatically.
+
diff --git a/docs/devel/ci-runners.rst b/docs/devel/ci-runners.rst
new file mode 100644
index 00..7817001fb2
--- /dev/null
+++ b/docs/devel/ci-runners.rst
@@ -0,0 +1,117 @@
+Jobs on Custom Runners
+==
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Custom runners entail custom machines.  To see a list of the machines
+currently deployed in the QEMU GitLab CI and their maintainers, please
+refer to the QEMU `wiki `__.
+
+Machine Setup Howto
+---
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  This playbook consists on the installation of various
+required packages (and a general package update while at it).  It
+currently covers a number of different Linux distributions, but it can
+be expanded to cover other systems.
+
+The minimum required version of Ansible successfully tested in this
+playbook is 2.8.0 (a version check is embedded within the playbook
+itself).  To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
+
+Please note that most of the tasks in the playbook require superuser
+privileges, such as those from the ``root`` account or those obtained
+by ``sudo``.  If necessary, please refer to ``ansible-playbook``
+options such as ``--become``, 

[PATCH 2/2] docs: add definitions of terms for CI/testing

[Willian Rampazzo]

To understand the current state of QEMU CI/testing and have a base to
discuss the plans for the future, it is important to define some usual
terms. This patch defines the terms for "Automated tests", "Unit
testing", "Functional testing", "System testing", "Flaky tests",
"Gating", and "Continuous Integration".

The first patch was borrowed from
20210812180403.4129067-1-berra...@redhat.com.

Signed-off-by: Willian Rampazzo 
---
 docs/devel/ci-definitions.rst | 121 ++
 docs/devel/ci.rst |   1 +
 2 files changed, 122 insertions(+)
 create mode 100644 docs/devel/ci-definitions.rst

diff --git a/docs/devel/ci-definitions.rst b/docs/devel/ci-definitions.rst
new file mode 100644
index 00..32e22ff468
--- /dev/null
+++ b/docs/devel/ci-definitions.rst
@@ -0,0 +1,121 @@
+Definition of terms
+===
+
+This section defines the terms used in this document and correlates them with
+what is currently used on QEMU.
+
+Automated tests
+---
+
+An automated test is written on a test framework using its generic test
+functions/classes. The test framework can run the tests and report their
+success or failure [1]_.
+
+An automated test has essentially three parts:
+
+1. The test initialization of the parameters, where the expected parameters,
+   like inputs and expected results, are set up;
+2. The call to the code that should be tested;
+3. An assertion, comparing the result from the previous call with the expected
+   result set during the initialization of the parameters. If the result
+   matches the expected result, the test has been successful; otherwise, it has
+   failed.
+
+Unit testing
+
+
+A unit test is responsible for exercising individual software components as a
+unit, like interfaces, data structures, and functionality, uncovering errors
+within the boundaries of a component. The verification effort is in the
+smallest software unit and focuses on the internal processing logic and data
+structures. A test case of unit tests should be designed to uncover errors due
+to erroneous computations, incorrect comparisons, or improper control flow 
[2]_.
+
+On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
+
+Functional testing
+--
+
+A functional test focuses on the functional requirement of the software.
+Deriving sets of input conditions, the functional tests should fully exercise
+all the functional requirements for a program. Functional testing is
+complementary to other testing techniques, attempting to find errors like
+incorrect or missing functions, interface errors, behavior errors, and
+initialization and termination errors [3]_.
+
+On QEMU, functional testing is represented by the 'check-qtest' target from
+'make'.
+
+System testing
+--
+
+System tests ensure all application elements mesh properly while the overall
+functionality and performance are achieved [4]_. Some or all system components
+are integrated to create a complete system to be tested as a whole. System
+testing ensures that components are compatible, interact correctly, and
+transfer the right data at the right time across their interfaces. As system
+testing focuses on interactions, use case-based testing is a practical approach
+to system testing [5]_. Note that, in some cases, system testing may require
+interaction with third-party software, like operating system images, databases,
+networks, and so on.
+
+On QEMU, system testing is represented by the 'check-acceptance' target from
+'make'.
+
+Flaky tests
+---
+
+A flaky test is defined as a test that exhibits both a passing and a failing
+result with the same code on different runs. Some usual reasons for an
+intermittent/flaky test are async wait, concurrency, and test order dependency
+[6]_.
+
+Gating
+--
+
+A gate restricts the move of code from one stage to another on a
+test/deployment pipeline. The step move is granted with approval. The approval
+can be a manual intervention or a set of tests succeeding [7]_.
+
+On QEMU, the gating process happens during the pull request. The approval is
+done by the project leader running its own set of tests. The pull request gets
+merged when the tests succeed.
+
+Continuous Integration (CI)
+---
+
+Continuous integration (CI) requires the builds of the entire application and
+the execution of a comprehensive set of automated tests every time there is a
+need to commit any set of changes [8]_. The automated tests can be composed of
+the unit, functional, system, and other tests.
+
+Keynotes about continuous integration (CI) [9]_:
+
+1. System tests may depend on external software (operating system images,
+   firmware, database, network).
+2. It may take a long time to build and test. It may be impractical to build
+   the system being developed several times per day.
+3. If the development plat

[PATCH 0/2] docs: add definitions of terms for CI/testing

[Willian Rampazzo]

To understand the current state of QEMU CI/testing and have a base to
discuss the plans for the future, it is important to define some usual
terms. This patch defines the terms for "Automated tests", "Unit
testing", "Functional testing", "System testing", "Flaky tests",
"Gating", and "Continuous Integration".

The first patch was borrowed from
20210812180403.4129067-1-berra...@redhat.com.

Signed-off-by: Willian Rampazzo 

Daniel P. Berrangé (1):
  docs: split the CI docs into two files

Willian Rampazzo (1):
  docs: add definitions of terms for CI/testing

 docs/devel/ci-definitions.rst | 121 +
 docs/devel/ci-jobs.rst|  40 +
 docs/devel/ci-runners.rst | 117 +
 docs/devel/ci.rst | 160 +-
 4 files changed, 281 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-definitions.rst
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

-- 
2.31.1

Re: [PATCH] build: remove redundant 'check-build' make target

[Willian Rampazzo]

On Tue, Aug 10, 2021 at 1:44 PM Daniel P. Berrangé  wrote:
>
> The 'check-build' make target was added as a way to build all the unit
> test binaries, since the standard 'all' target would not trigger this.
>
> Since the switch to meson, however, 'all' will now include the 'test'
> binaries. As a result, 'check-build' is a no-op:
>
>$ make all check-build
>..snip lots of output...
>make: Nothing to be done for 'check-build'.
>

Confirmed!

> Signed-off-by: Daniel P. Berrangé 
> ---
>  .gitlab-ci.d/buildtest.yml   | 9 -
>  .gitlab-ci.d/crossbuild-template.yml | 6 +++---
>  tests/Makefile.include   | 3 ---
>  3 files changed, 3 insertions(+), 15 deletions(-)
>

Reviewed-by: Willian Rampazzo