Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin  wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory 
> > > > like
> > > > tests/pytests/ for example, as long as it downloads your binaries 
> > > > separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test 
> > > > runner
> > > > and not the meson test harness, so having a new approach for 
> > > > python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this 
> > > > goes
> > > > into tests/qtest (since it really does not belong there), and please 
> > > > don't
> > > > add more external stuff via git submodules, that's really the wrong 
> > > > approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?
> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.
>
> Let's start with building things from source. Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.

Well we need to make a call as to how to make things work offline, if
that's what we wanted.  I have made changes in v2 to download the
binaries from my github repo. It works great. If download fails, it
skips the test. If those files are available before, it will not
attempt to download them.
https://pastebin.com/nrcdN8WS and
https://pastebin.com/m7Dx3PZk

The Dockerfile and build script I have in the repo will generate these
artifacts. So if you wanted to run the test offline, either build
those from source and place them where the test expects to find them
or build them from source using Dockerfile/build script and place them
there before running the test.

>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Mon, Jul 4, 2022 at 7:02 PM Ani Sinha  wrote:
>
> On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin  wrote:
> >
> > On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > > but I thought you were suggesting we built bits every time the test is 
> > > run?
> >
> > In my opinion 3 scenarios are worth supporting:
> > - people not touching ACPI, including users - simply don't run the tests,
> > comparing tables with expected output should be enough
> > - people making changes indirectly affecting ACPI -
> > use tests to validate that tables are still well formed,
> > using a pre built binary should be enough IMO
> > - people working on ACPI - use tests to analyse the tables,
> > building from source might be necessary for debugging,
> > sources change very rarely
> > - people developing the tests
> > building from source is required
> >
> > So I would suggest basically two github repos, one with binaries one with
> > sources. We'll keep relevant hashes to use in the script.
> > All in all not that different from submodules but I guess
> > people have submodules and that is that.
> >
> > And I personally would probably not tie it to CI whoever owns the
> > repository can worry about the builds, and I think keeping
> > things distributed is important.
> >
> > So
> > - people not touching ACPI - make check should see directory not found
> > and skip the test
> > - people making changes indirectly affecting ACPI -
> > check out binaries and use
> > - people working on ACPI -
> > see that source directory is present, go there
> > and run make. should not rebuild each time right?
> > - people developing the tests
> > building from source is required
>
> Ok I have now committed a Dockerfile that has all the build time
> dependencies and builds bits and generates the tarballs/zip file that
> my test requires:
> https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
>
> We just need to fork the repo and generate automated builds with this
> Dockerfile and put the binaries somewhere. This should also help
> developers requiring to rebuild bits when necessary.
> Oh and btw, I also made bits compliant with the latest gcc 11 compiler
> that comes with Ubuntu 22.04 LTS :-)

Pushed a build script here:
https://github.com/ani-sinha/bits/commit/90b99ef05d55ead4b33b2fb19ad07dfb9682ec92

and the bios bits binaries are in this branch:
https://github.com/ani-sinha/bits/tree/bits-builds

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin  wrote:
>
> On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > but I thought you were suggesting we built bits every time the test is run?
>
> In my opinion 3 scenarios are worth supporting:
> - people not touching ACPI, including users - simply don't run the tests,
> comparing tables with expected output should be enough
> - people making changes indirectly affecting ACPI -
> use tests to validate that tables are still well formed,
> using a pre built binary should be enough IMO
> - people working on ACPI - use tests to analyse the tables,
> building from source might be necessary for debugging,
> sources change very rarely
> - people developing the tests
> building from source is required
>
> So I would suggest basically two github repos, one with binaries one with
> sources. We'll keep relevant hashes to use in the script.
> All in all not that different from submodules but I guess
> people have submodules and that is that.
>
> And I personally would probably not tie it to CI whoever owns the
> repository can worry about the builds, and I think keeping
> things distributed is important.
>
> So
> - people not touching ACPI - make check should see directory not found
> and skip the test
> - people making changes indirectly affecting ACPI -
> check out binaries and use
> - people working on ACPI -
> see that source directory is present, go there
> and run make. should not rebuild each time right?
> - people developing the tests
> building from source is required

Ok I have now committed a Dockerfile that has all the build time
dependencies and builds bits and generates the tarballs/zip file that
my test requires:
https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile

We just need to fork the repo and generate automated builds with this
Dockerfile and put the binaries somewhere. This should also help
developers requiring to rebuild bits when necessary.
Oh and btw, I also made bits compliant with the latest gcc 11 compiler
that comes with Ubuntu 22.04 LTS :-)

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> but I thought you were suggesting we built bits every time the test is run?

In my opinion 3 scenarios are worth supporting:
- people not touching ACPI, including users - simply don't run the tests,
comparing tables with expected output should be enough
- people making changes indirectly affecting ACPI -
use tests to validate that tables are still well formed,
using a pre built binary should be enough IMO
- people working on ACPI - use tests to analyse the tables,
building from source might be necessary for debugging,
sources change very rarely
- people developing the tests
building from source is required

So I would suggest basically two github repos, one with binaries one with
sources. We'll keep relevant hashes to use in the script.
All in all not that different from submodules but I guess
people have submodules and that is that.

And I personally would probably not tie it to CI whoever owns the
repository can worry about the builds, and I think keeping
things distributed is important.

So
- people not touching ACPI - make check should see directory not found
and skip the test
- people making changes indirectly affecting ACPI -
check out binaries and use
- people working on ACPI -
see that source directory is present, go there
and run make. should not rebuild each time right?
- people developing the tests
building from source is required


-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 3:12 PM Michael S. Tsirkin  wrote:
>
> On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin  wrote:
> > >
> > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  
> > > > wrote:
> > > > >
> > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > > I even wouldn't mind if you put your python stuff in a new 
> > > > > > directory like
> > > > > > tests/pytests/ for example, as long as it downloads your binaries 
> > > > > > separately
> > > > > > - as I wrote in another mail, the avocado framework rather looks 
> > > > > > like an
> > > > > > oddball in our test framework nowadays since it uses a separate 
> > > > > > test runner
> > > > > > and not the meson test harness, so having a new approach for 
> > > > > > python-based
> > > > > > tests is maybe even a good idea. I just really want to avoid that 
> > > > > > this goes
> > > > > > into tests/qtest (since it really does not belong there), and 
> > > > > > please don't
> > > > > > add more external stuff via git submodules, that's really the wrong 
> > > > > > approach
> > > > > > for this.
> > > > >
> > > > > I get it, people hate submodules with passion.  I think trying another
> > > > > approach for testing that is neither avocado nor qtest is
> > > > > not too bad. As long as this is not user visible, we can
> > > > > allow ourselves space to experiment.
> > > > >
> > > > > OK so, how about this:
> > > > > - put it in a new directory: tests/roms?
> > > > > - create repo for a fork of biosbits under git.qemu.org
> > > > > - roll our own analog to git submodules: a script
> > > > >   that clones the repo
> > > >
> > > > No need to clone the whole repo. We can simply download the binaries
> > > > that the girlab CI job would generate from the bits sources in that
> > > > repo.
> > > > We need to clone if we are always building bits from source for every
> > > > test. That is not necessary IMHO since much of the bits package would
> > > > remain as is without modification.
> > >
> > > IMHO CI job idea isn't great since isn't versioned at all, is it?
> >
> > bits is versioned yes, in a crude way. every time you make a commit in
> > the top level repo, the version would increment by one.
>
> Is it easy to find out which source was this generated from?

yes because this is how the version number is generated:

expr 2000 + `git rev-list HEAD 2>/dev/null | wc -l`

That means we simply count the number of commits in the top level
directory. If we overwrote history by force pushing, then well yes all
bets are off.

> And is there a promise to keep these around indefinitely?

We do not need to maintain multiple versions of bits or increment the
version number every time we wrote a test. Typically new
commits/changes will be made if we wanted to fix a bug or upgrade
acpica or whatever. Test scripts are python based and can be pushed
into bits while generating a new iso for testing purposes.

>
> > > Also as long as test passes, sure. But if it fails one will
> > > need the sources to investigate.
> >
> > sources might also be needed to write the tests.
> >
> > >
> > > Let's start with building things from source.
> >
> > hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> > build depending on parallelity and build host.
>
> Right. But whoever wants to use these just needs to do it once.

but I thought you were suggesting we built bits every time the test is run?

>
>
> > Add an option
> > > of prebuilt binaries as an optimization once things
> > > stabilize.
> > >
> > >
> > > > > - new target make check-roms,
> > > >
> > > > I think make pytest or some such is better and more generic if other
> > > > such tests in other areas follow suit.
> > >
> > > The name is not critical in my mind, but I think we need to decide
> > > what exactly differentiates it from other tests.
> > >
> > >
> > > >
> > > > if the clone exists locally -
> > > > >   run the test, if not - skip it
> > > >
> > > > if download of the bits binaries fail, skip it.
> > >
> > > You seem to be recreating either git or avocado or both here.
> > >
> > > Personally I want something that works offline.
> > >
> > >
> > >
> > > > > - as for using pre-generates ISOs as an optimization,
> > > > >   I'm not sure how important that is, if yes -
> > > > >   we can add another repo and another make target along the
> > > > >   same lines
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > MST
> > > > >
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin  wrote:
> >
> > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  
> > > wrote:
> > > >
> > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > I even wouldn't mind if you put your python stuff in a new directory 
> > > > > like
> > > > > tests/pytests/ for example, as long as it downloads your binaries 
> > > > > separately
> > > > > - as I wrote in another mail, the avocado framework rather looks like 
> > > > > an
> > > > > oddball in our test framework nowadays since it uses a separate test 
> > > > > runner
> > > > > and not the meson test harness, so having a new approach for 
> > > > > python-based
> > > > > tests is maybe even a good idea. I just really want to avoid that 
> > > > > this goes
> > > > > into tests/qtest (since it really does not belong there), and please 
> > > > > don't
> > > > > add more external stuff via git submodules, that's really the wrong 
> > > > > approach
> > > > > for this.
> > > >
> > > > I get it, people hate submodules with passion.  I think trying another
> > > > approach for testing that is neither avocado nor qtest is
> > > > not too bad. As long as this is not user visible, we can
> > > > allow ourselves space to experiment.
> > > >
> > > > OK so, how about this:
> > > > - put it in a new directory: tests/roms?
> > > > - create repo for a fork of biosbits under git.qemu.org
> > > > - roll our own analog to git submodules: a script
> > > >   that clones the repo
> > >
> > > No need to clone the whole repo. We can simply download the binaries
> > > that the girlab CI job would generate from the bits sources in that
> > > repo.
> > > We need to clone if we are always building bits from source for every
> > > test. That is not necessary IMHO since much of the bits package would
> > > remain as is without modification.
> >
> > IMHO CI job idea isn't great since isn't versioned at all, is it?
> 
> bits is versioned yes, in a crude way. every time you make a commit in
> the top level repo, the version would increment by one.

Is it easy to find out which source was this generated from?
And is there a promise to keep these around indefinitely?

> > Also as long as test passes, sure. But if it fails one will
> > need the sources to investigate.
> 
> sources might also be needed to write the tests.
> 
> >
> > Let's start with building things from source.
> 
> hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> build depending on parallelity and build host.

Right. But whoever wants to use these just needs to do it once.


> Add an option
> > of prebuilt binaries as an optimization once things
> > stabilize.
> >
> >
> > > > - new target make check-roms,
> > >
> > > I think make pytest or some such is better and more generic if other
> > > such tests in other areas follow suit.
> >
> > The name is not critical in my mind, but I think we need to decide
> > what exactly differentiates it from other tests.
> >
> >
> > >
> > > if the clone exists locally -
> > > >   run the test, if not - skip it
> > >
> > > if download of the bits binaries fail, skip it.
> >
> > You seem to be recreating either git or avocado or both here.
> >
> > Personally I want something that works offline.
> >
> >
> >
> > > > - as for using pre-generates ISOs as an optimization,
> > > >   I'm not sure how important that is, if yes -
> > > >   we can add another repo and another make target along the
> > > >   same lines
> > > >
> > > >
> > > >
> > > > --
> > > > MST
> > > >
> >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin  wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory 
> > > > like
> > > > tests/pytests/ for example, as long as it downloads your binaries 
> > > > separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test 
> > > > runner
> > > > and not the meson test harness, so having a new approach for 
> > > > python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this 
> > > > goes
> > > > into tests/qtest (since it really does not belong there), and please 
> > > > don't
> > > > add more external stuff via git submodules, that's really the wrong 
> > > > approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?

bits is versioned yes, in a crude way. every time you make a commit in
the top level repo, the version would increment by one.

> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.

sources might also be needed to write the tests.

>
> Let's start with building things from source.

hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
build depending on parallelity and build host.

Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.
>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  wrote:
> >
> > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > I even wouldn't mind if you put your python stuff in a new directory like
> > > tests/pytests/ for example, as long as it downloads your binaries 
> > > separately
> > > - as I wrote in another mail, the avocado framework rather looks like an
> > > oddball in our test framework nowadays since it uses a separate test 
> > > runner
> > > and not the meson test harness, so having a new approach for python-based
> > > tests is maybe even a good idea. I just really want to avoid that this 
> > > goes
> > > into tests/qtest (since it really does not belong there), and please don't
> > > add more external stuff via git submodules, that's really the wrong 
> > > approach
> > > for this.
> >
> > I get it, people hate submodules with passion.  I think trying another
> > approach for testing that is neither avocado nor qtest is
> > not too bad. As long as this is not user visible, we can
> > allow ourselves space to experiment.
> >
> > OK so, how about this:
> > - put it in a new directory: tests/roms?
> > - create repo for a fork of biosbits under git.qemu.org
> > - roll our own analog to git submodules: a script
> >   that clones the repo
> 
> No need to clone the whole repo. We can simply download the binaries
> that the girlab CI job would generate from the bits sources in that
> repo.
> We need to clone if we are always building bits from source for every
> test. That is not necessary IMHO since much of the bits package would
> remain as is without modification.

IMHO CI job idea isn't great since isn't versioned at all, is it?
Also as long as test passes, sure. But if it fails one will
need the sources to investigate.

Let's start with building things from source. Add an option
of prebuilt binaries as an optimization once things
stabilize.


> > - new target make check-roms,
> 
> I think make pytest or some such is better and more generic if other
> such tests in other areas follow suit.

The name is not critical in my mind, but I think we need to decide
what exactly differentiates it from other tests.


> 
> if the clone exists locally -
> >   run the test, if not - skip it
> 
> if download of the bits binaries fail, skip it.

You seem to be recreating either git or avocado or both here.

Personally I want something that works offline.



> > - as for using pre-generates ISOs as an optimization,
> >   I'm not sure how important that is, if yes -
> >   we can add another repo and another make target along the
> >   same lines
> >
> >
> >
> > --
> > MST
> >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin  wrote:
>
> On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > I even wouldn't mind if you put your python stuff in a new directory like
> > tests/pytests/ for example, as long as it downloads your binaries separately
> > - as I wrote in another mail, the avocado framework rather looks like an
> > oddball in our test framework nowadays since it uses a separate test runner
> > and not the meson test harness, so having a new approach for python-based
> > tests is maybe even a good idea. I just really want to avoid that this goes
> > into tests/qtest (since it really does not belong there), and please don't
> > add more external stuff via git submodules, that's really the wrong approach
> > for this.
>
> I get it, people hate submodules with passion.  I think trying another
> approach for testing that is neither avocado nor qtest is
> not too bad. As long as this is not user visible, we can
> allow ourselves space to experiment.
>
> OK so, how about this:
> - put it in a new directory: tests/roms?
> - create repo for a fork of biosbits under git.qemu.org
> - roll our own analog to git submodules: a script
>   that clones the repo

No need to clone the whole repo. We can simply download the binaries
that the girlab CI job would generate from the bits sources in that
repo.
We need to clone if we are always building bits from source for every
test. That is not necessary IMHO since much of the bits package would
remain as is without modification.

> - new target make check-roms,

I think make pytest or some such is better and more generic if other
such tests in other areas follow suit.

if the clone exists locally -
>   run the test, if not - skip it

if download of the bits binaries fail, skip it.

> - as for using pre-generates ISOs as an optimization,
>   I'm not sure how important that is, if yes -
>   we can add another repo and another make target along the
>   same lines
>
>
>
> --
> MST
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> I even wouldn't mind if you put your python stuff in a new directory like
> tests/pytests/ for example, as long as it downloads your binaries separately
> - as I wrote in another mail, the avocado framework rather looks like an
> oddball in our test framework nowadays since it uses a separate test runner
> and not the meson test harness, so having a new approach for python-based
> tests is maybe even a good idea. I just really want to avoid that this goes
> into tests/qtest (since it really does not belong there), and please don't
> add more external stuff via git submodules, that's really the wrong approach
> for this.

I get it, people hate submodules with passion.  I think trying another
approach for testing that is neither avocado nor qtest is
not too bad. As long as this is not user visible, we can
allow ourselves space to experiment.

OK so, how about this:
- put it in a new directory: tests/roms?
- create repo for a fork of biosbits under git.qemu.org
- roll our own analog to git submodules: a script
  that clones the repo
- new target make check-roms, if the clone exists locally -
  run the test, if not - skip it
- as for using pre-generates ISOs as an optimization,
  I'm not sure how important that is, if yes -
  we can add another repo and another make target along the
  same lines



-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 15.53, Ani Sinha wrote:



On Tue, Jun 28, 2022 at 19:15 Peter Maydell > wrote:


On Tue, 28 Jun 2022 at 14:23, Ani Sinha mailto:a...@anisinha.ca>> wrote:
 > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé
mailto:berra...@redhat.com>> wrote:
 > > This proposed biosbits test also involves a considerable download.
 >
 > I do not think 50 MB is "considerable" . Last time I tried to run
 > avocado tests, my laptop ran out of disk space!

I think 50MB is pretty big. It might be smaller than some other
avocado tests, but it's not exactly the "no binary involved"
that most qtests are.


Well bios-tables-test uses the binary blobs of the acpi tables. Only 
difference is that in this case, we could maintain them within  the qemu 
tree. In this case the blob in slightly larger and comes from a third party. 
That is the difference.


"slightly larger" ... it apparently contains already a complete grub and 
python 2.7 interpreter ... I'd consider that as *much* larger than the ~ 2k 
loc bios-tables-test ;-)



Very few 'make check' tests even run code in the guest.

So bits test is similar here. It runs code in the guest vm.


The qtests that run some TCG code only use some few lines of code. The tests 
that run third party firmware images are the avocado tests.



There are definitely some awkwardnesses with 'check-avocado',
but we should work on fixing those, not use them as a reason
to refuse to put tests into the avocado tests if that's where
they fit best.

I think this test fits best in the qtrst not with the integration test 
framework.


I disagree. Third party binary blob tests are certainly nothing for the 
qtest directory.


Very few path developers will ever run it and wrote new tests for 
it if we have it there. I would be terribly discouraged if that’s where this 
test landed.


I see your point - I'm also hardly ever running whole "make check-avocado" 
since it's too slow/fat/broken. But using this as a reason to stick your 
test into the qtest framework also does not work/scale: Being one of the 
s390x maintainers - What if I now also want my s390x related tests to be 
executed by everybody? The kernel + initrd images from 
tests/avocado/machine_s390_ccw_virtio.py are also not that big, ca. 50 MiB 
each. Should I now move that out of the avocade directory, too, and force 
everybody to consume it via a submodule? Then who's next? QEMU has 21 target 
architectures, if every maintainers adds a 50 MiB test to the tree, we're at 
more than a gigabyte of data already that you'd have to download before you 
were able to run "make check". This simply does not scale.


So the avocado framework is currently still the best place that we have for 
such tests. You just have to get used to consume it via "tags" instead of 
running the whole "make check-avocado" suite. For example, being a s390x 
maintainer, I'm only running:


 make check-venv
 ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/

and that finishes within some few minutes and only downloads some few 
hundreds of megabytes. You could do something similar with acpi. You could 
even wrap it in a "make check-avocado-acpi" target if you don't like to 
execute avocado directly.


I even wouldn't mind if you put your python stuff in a new directory like 
tests/pytests/ for example, as long as it downloads your binaries separately 
- as I wrote in another mail, the avocado framework rather looks like an 
oddball in our test framework nowadays since it uses a separate test runner 
and not the meson test harness, so having a new approach for python-based 
tests is maybe even a good idea. I just really want to avoid that this goes 
into tests/qtest (since it really does not belong there), and please don't 
add more external stuff via git submodules, that's really the wrong approach 
for this.


 Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 06:52:55PM +0530, Ani Sinha wrote:
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

No it doesn't. Any change in qemu can break ACPI - it's enough to
register an MR overlapping the ACPI registers.  So we have a basic test
making sure ACPI is in expected state and have everyone run it.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 01:55:48PM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell  
> > wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> > 
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
> 
> This proposed biosbits test also involves a considerable download.
> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits. So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest. Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
> 
> With regards,
> Daniel

Optional is fine. I would like to make life easy for
developers working on ACPI though, the field is unforgiving
enough as it is.
How about we skip the test if the submodule is not checked out?

> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 14:53, Ani Sinha  wrote:
>
>
>
> On Tue, Jun 28, 2022 at 19:15 Peter Maydell  wrote:
>>
>> On Tue, 28 Jun 2022 at 14:23, Ani Sinha  wrote:
>> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé  
>> > wrote:
>> > > This proposed biosbits test also involves a considerable download.
>> >
>> > I do not think 50 MB is "considerable" . Last time I tried to run
>> > avocado tests, my laptop ran out of disk space!
>>
>> I think 50MB is pretty big. It might be smaller than some other
>> avocado tests, but it's not exactly the "no binary involved"
>> that most qtests are.
>
>
> Well bios-tables-test uses the binary blobs of the acpi tables. Only 
> difference is that in this case, we could maintain them within  the qemu 
> tree. In this case the blob in slightly larger and comes from a third party. 
> That is the difference.

Yes. That is exactly the difference that means it should go in
the avocado tests.

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 19:15 Peter Maydell 
wrote:

> On Tue, 28 Jun 2022 at 14:23, Ani Sinha  wrote:
> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé 
> wrote:
> > > This proposed biosbits test also involves a considerable download.
> >
> > I do not think 50 MB is "considerable" . Last time I tried to run
> > avocado tests, my laptop ran out of disk space!
>
> I think 50MB is pretty big. It might be smaller than some other
> avocado tests, but it's not exactly the "no binary involved"
> that most qtests are.


Well bios-tables-test uses the binary blobs of the acpi tables. Only
difference is that in this case, we could maintain them within  the qemu
tree. In this case the blob in slightly larger and comes from a third
party. That is the difference.



>
> > > The test is said to be irrelevant for anyone except those working
> > > on a fairly narrow set of QEMU firmware related bits.
> >
> > Well ok that is just a bad argument. You can say the same thing for
> > most qtests. In fact, that is why most qtetes can run directly simply
> > by passing QTEST_QEMU_BINARY in the environment. No need to go through
> > make check. Same with the bits test. It can be run directly.
>
> 'make check' is generally the small, fast, no-binary-blobs tests.


See above.


> Very few 'make check' tests even run code in the guest.


So bits test is similar here. It runs code in the guest vm.


>
> > So by the same
> > > rationale we shouldn't impose that burden on everyone working on
> > > QEMU by having it in qtest.
> >
> > So why burden everyone by having bios-tables-test when it only affects
> > acpi/smbios developers?
>
> Because it's small and fast and doesn't have a binary blob to download.
>
> There are definitely some awkwardnesses with 'check-avocado',
> but we should work on fixing those, not use them as a reason
> to refuse to put tests into the avocado tests if that's where
> they fit best.


I think this test fits best in the qtrst not with the integration test
framework. Very few path developers will ever run it and wrote new tests
for it if we have it there. I would be terribly discouraged if that’s where
this test landed.


>
> -- PMM
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 14:23, Ani Sinha  wrote:
> On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé  
> wrote:
> > This proposed biosbits test also involves a considerable download.
>
> I do not think 50 MB is "considerable" . Last time I tried to run
> avocado tests, my laptop ran out of disk space!

I think 50MB is pretty big. It might be smaller than some other
avocado tests, but it's not exactly the "no binary involved"
that most qtests are.

> > The test is said to be irrelevant for anyone except those working
> > on a fairly narrow set of QEMU firmware related bits.
>
> Well ok that is just a bad argument. You can say the same thing for
> most qtests. In fact, that is why most qtetes can run directly simply
> by passing QTEST_QEMU_BINARY in the environment. No need to go through
> make check. Same with the bits test. It can be run directly.

'make check' is generally the small, fast, no-binary-blobs tests.
Very few 'make check' tests even run code in the guest.

> So by the same
> > rationale we shouldn't impose that burden on everyone working on
> > QEMU by having it in qtest.
>
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

Because it's small and fast and doesn't have a binary blob to download.

There are definitely some awkwardnesses with 'check-avocado',
but we should work on fixing those, not use them as a reason
to refuse to put tests into the avocado tests if that's where
they fit best.

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé  wrote:
>
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell  
> > wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> >
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
>
> This proposed biosbits test also involves a considerable download.

I do not think 50 MB is "considerable" . Last time I tried to run
avocado tests, my laptop ran out of disk space!

> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits.

Well ok that is just a bad argument. You can say the same thing for
most qtests. In fact, that is why most qtetes can run directly simply
by passing QTEST_QEMU_BINARY in the environment. No need to go through
make check. Same with the bits test. It can be run directly.

So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest.

So why burden everyone by having bios-tables-test when it only affects
acpi/smbios developers?

Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
>
> With regards,
> Daniel
> --
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell  
> wrote:
> >
> > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:
> > > I think the main difference is not even in how it works, it's
> > > in what it does. Which is check that ACPI tables are sane.
> > > Who cares about that? Well developers do when they change the
> > > tables. Users really don't because for users we have the expected
> > > tables in tree and we check against these.
> >
> > It wants to build and run a big guest binary blob -- that to me is
> > the main difference. Users don't much care about any of our tests,
> 
> perhaps but we do enforce patch submitters to make sure make check
> passes before submitting patches. make check-avocado is not run as
> part of make check, requires considerable disk space to download all
> guest images and hence generally not run by patch submitters. Making
> bits parts of avocado tests almost defeats the purpose of having this
> test at all.

This proposed biosbits test also involves a considerable download.
The test is said to be irrelevant for anyone except those working
on a fairly narrow set of QEMU firmware related bits. So by the same
rationale we shouldn't impose that burden on everyone working on
QEMU by having it in qtest. Making it entirely optional, only
downloaded by avocado on demand, for the people who need to run
the tests is best.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 14.36, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell  wrote:


On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:

I think the main difference is not even in how it works, it's
in what it does. Which is check that ACPI tables are sane.
Who cares about that? Well developers do when they change the
tables. Users really don't because for users we have the expected
tables in tree and we check against these.


It wants to build and run a big guest binary blob -- that to me is
the main difference. Users don't much care about any of our tests,


perhaps but we do enforce patch submitters to make sure make check
passes before submitting patches. make check-avocado is not run as
part of make check, requires considerable disk space to download all
guest images and hence generally not run by patch submitters. Making
bits parts of avocado tests almost defeats the purpose of having this
test at all.
You can define sane subsets of the avocado tests by using tags. For example, 
as I'm one of the s390x maintainers, I'm also only running the tests that 
are tagged with tags=arch:s390x there. You could simply introduce an "acpi" 
tag for your tests, too.


 Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell  wrote:
>
> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:
> > I think the main difference is not even in how it works, it's
> > in what it does. Which is check that ACPI tables are sane.
> > Who cares about that? Well developers do when they change the
> > tables. Users really don't because for users we have the expected
> > tables in tree and we check against these.
>
> It wants to build and run a big guest binary blob -- that to me is
> the main difference. Users don't much care about any of our tests,

perhaps but we do enforce patch submitters to make sure make check
passes before submitting patches. make check-avocado is not run as
part of make check, requires considerable disk space to download all
guest images and hence generally not run by patch submitters. Making
bits parts of avocado tests almost defeats the purpose of having this
test at all.

> whether they're under 'make check' or 'make check-avocado' or the
> iotests framework. The reason to pick one framework or another
> is mostly I think whether the properties of the test are such
> that one framework works better. Avocado is (for better or worse)
> the one we have for dealing with "actually run a guest machine
> with a big lump of guest code in it".
>
> -- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin  wrote:
> I think the main difference is not even in how it works, it's
> in what it does. Which is check that ACPI tables are sane.
> Who cares about that? Well developers do when they change the
> tables. Users really don't because for users we have the expected
> tables in tree and we check against these.

It wants to build and run a big guest binary blob -- that to me is
the main difference. Users don't much care about any of our tests,
whether they're under 'make check' or 'make check-avocado' or the
iotests framework. The reason to pick one framework or another
is mostly I think whether the properties of the test are such
that one framework works better. Avocado is (for better or worse)
the one we have for dealing with "actually run a guest machine
with a big lump of guest code in it".

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin  wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and 
> > > > interacting
> > > > with it, then it does feel like the scope of this testing is more 
> > > > appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to 
> > > > use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I think the main difference is not even in how it works, it's
in what it does. Which is check that ACPI tables are sane.
Who cares about that? Well developers do when they change the
tables. Users really don't because for users we have the expected
tables in tree and we check against these.


-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin  wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and 
> > > > interacting
> > > > with it, then it does feel like the scope of this testing is more 
> > > > appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to 
> > > > use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I wonder why it needs grub BTW. Let's just load all of disk into ram
and be done with it?


> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 4:00 PM Ani Sinha  wrote:
>
> On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé  
> wrote:
> >
> > On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé 
> > > > >  wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > > No problem with that. So that's venv. But do we 
> > > > > > > > > > > > > > > need pip and pulling
> > > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff 
> > > > > > > > > > > > > from
> > > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > > Are the libraries in question not packaged on major 
> > > > > > > > > > > > > distros?
> > > > > > > > > > > >
> > > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > > >
> > > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > > >
> > > > > > > > > > > > I do not think this is packaged by default. It's such a 
> > > > > > > > > > > > basic library
> > > > > > > > > > > > for parsing test output that maybe we can keep this 
> > > > > > > > > > > > somewhere within
> > > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > > >
> > > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > > >
> > > > > > > > > > Unlike BITS, this one is likely going to be maintained for 
> > > > > > > > > > a while and
> > > > > > > > > > will receive new releases through
> > > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > > >
> > > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > > >
> > > > > > > > > John Snow is currently working on the "Pythonification" of 
> > > > > > > > > various QEMU
> > > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > > >
> > > > > > > > >Thomas
> > > > > > > >
> > > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > > all options: check for the module and use it if there, if not
> > > > > > > > use one from system if not there install with pip ..
> > > > > > > > But yea, I'm not sure what's best either.
> > > > > > >
> > > > > > > submodules create a dependency on an internet connection, too. So 
> > > > > > > before you
> > > > > > > add yet another submodule (which have a couple of other 
> > > > > > > disadvantages), I
> > > > > > > think you could also directly use the venv here.
> > > > > >
> > > > > > Definitely not submodules.
> > > > > >
> > > > > > We need to get out of the mindset that submodules are needed for 
> > > > > > every new
> > > > > > dependancy we add. Submodules are only appropriate if the external 
> > > > > > project
> > > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or 
> > > > > > if we
> > > > > > need to bundle in order to prevent a regression for previously 
> > > > > > deployed
> > > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > > supported platforms.
> > > > > >
> > > > > > This does not apply in this case, because the proposed use of tappy 
> > > > > > is
> > > > > > merely for a test case. Meson just needs to check if tappy exists 
> > > > > > and if
> > > > > > it does, then use it, otherwise skip the tests that need it. The 
> > > > > > user can
> > > > > > arrange to install tappy, as they do with the majority of other 
> > > > > > deps.
> > > > > >
> > > > > > If John's venv stuff is relevant, then we don't even need the meson 
> > > > > > checks,
> > > > > > just delegate to the venv setup.
> > > > > >
> > > > > > Regardless, no submodules are needed or desirable.
> > > > >
> > > > > What about keeping biosbits stuff? Source or pre-built.
> > > >
> > > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > > IMHO, especially for grub as a GPL'd 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin  wrote:
> >
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and 
> > > interacting
> > > with it, then it does feel like the scope of this testing is more 
> > > appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> image. It runs off grub, that is, directly from bootloader stage. The
> interraction with the VM is minimal.

Just because it doesn't run a whole Linux kernel, doesn't make it
not a guest OS image. It is merely unsual in that it can do everything
it needs from grub stage, because it is just poking low level BIOS
stuff and doesn't need a full OS like Linux on top. This is still
functional integration testing IMHO and relevant to avocado in QEMU.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:48 PM Daniel P. Berrangé  wrote:
>
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and 
> > > interacting
> > > with it, then it does feel like the scope of this testing is more 
> > > appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
>
> That's essentially what this is doing - its a custom guest OS rather

Daniel, there is no OS :-) Like there is no kernel image, no
persistent real file system (there is a fake one called pyfs), no
packages, nothing. Its very minimalistic image narrowly targeted for
one thing alone - to test the bios.

> than a common distro IIUC
>
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
>
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado. If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.
>
> With regards,
> Daniel
> --
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:18:14AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and 
> > > interacting
> > > with it, then it does feel like the scope of this testing is more 
> > > appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > > 
> > > With regards,
> > > Daniel
> > 
> > I feel avocado is directed towards booting full fledged guest OS.
> 
> That's essentially what this is doing - its a custom guest OS rather
> than a common distro IIUC

yes but then so is bios tables test - we generate the disk on the fly.

> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado.

The idea of downloading large images as part of the test as opposed to
part of setup is a large part of what makes it flaky - due to
running into unpredictable latency and errors on both the internet and
local IO side of things.

> If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.

A larger system has more potential for bugs ;) I'm with Ani here. Using
a system that is overkill will just make debugging more painful than it
needs to be.

> With regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 12.21, Michael S. Tsirkin wrote:

On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:

Now that you mention it, avocado does feel like a more appropriate fit.
IIUC the biosbits project appears to be effectively providing a custom
guest OS ISO image. IOW this testing is quite biased towards being
integration testing which is the target of avocado, while qtest is much
more to the unit testing end of the spectrum. This would avoid all the
discussion and patches around introducing python to qtest

With regards,
Daniel


Yes it's an ISO image but not a full OS.


The avocado framework is not meant for full OS testing only. We have a 
couple of tests there that just run a small firmware image. For example:


- tests/avocado/machine_m68k_nextcube.py : Just runs the original firmware 
of the NexT Cube machine


- tests/avocado/ppc_405.py : Just runs an U-Boot firmware image

Some of the tests even do not download anything at all and just run a QEMU 
instance via python, e.g.:


- tests/avocado/virtio_check_params.py

- tests/avocado/x86_cpu_model_versions.py

So I fail to see why biosbits should be so much different here.

Of course we could argue whether the avocado framework is really such a 
great fit for the QEMU test suite (as I said in another mail, it rather 
feels like an oddball nowadays), but that's a different argument.


 Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé  wrote:
>
> On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé 
> > > >  wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > No problem with that. So that's venv. But do we 
> > > > > > > > > > > > > > need pip and pulling
> > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > >
> > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > >
> > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > >
> > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff 
> > > > > > > > > > > > from
> > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > Are the libraries in question not packaged on major 
> > > > > > > > > > > > distros?
> > > > > > > > > > >
> > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > >
> > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > >
> > > > > > > > > > > I do not think this is packaged by default. It's such a 
> > > > > > > > > > > basic library
> > > > > > > > > > > for parsing test output that maybe we can keep this 
> > > > > > > > > > > somewhere within
> > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > >
> > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > >
> > > > > > > > > Unlike BITS, this one is likely going to be maintained for a 
> > > > > > > > > while and
> > > > > > > > > will receive new releases through
> > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > >
> > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > >
> > > > > > > > John Snow is currently working on the "Pythonification" of 
> > > > > > > > various QEMU
> > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > >
> > > > > > > >Thomas
> > > > > > >
> > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > all options: check for the module and use it if there, if not
> > > > > > > use one from system if not there install with pip ..
> > > > > > > But yea, I'm not sure what's best either.
> > > > > >
> > > > > > submodules create a dependency on an internet connection, too. So 
> > > > > > before you
> > > > > > add yet another submodule (which have a couple of other 
> > > > > > disadvantages), I
> > > > > > think you could also directly use the venv here.
> > > > >
> > > > > Definitely not submodules.
> > > > >
> > > > > We need to get out of the mindset that submodules are needed for 
> > > > > every new
> > > > > dependancy we add. Submodules are only appropriate if the external 
> > > > > project
> > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if 
> > > > > we
> > > > > need to bundle in order to prevent a regression for previously 
> > > > > deployed
> > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > supported platforms.
> > > > >
> > > > > This does not apply in this case, because the proposed use of tappy is
> > > > > merely for a test case. Meson just needs to check if tappy exists and 
> > > > > if
> > > > > it does, then use it, otherwise skip the tests that need it. The user 
> > > > > can
> > > > > arrange to install tappy, as they do with the majority of other deps.
> > > > >
> > > > > If John's venv stuff is relevant, then we don't even need the meson 
> > > > > checks,
> > > > > just delegate to the venv setup.
> > > > >
> > > > > Regardless, no submodules are needed or desirable.
> > > >
> > > > What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin  wrote:
>
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more 
> > appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> >
> > With regards,
> > Daniel
>
> I feel avocado is directed towards booting full fledged guest OS.
> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Yes 100% agree with Michael on this. Biobits is *not* booting any OS
image. It runs off grub, that is, directly from bootloader stage. The
interraction with the VM is minimal.

>
> --
> MST
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more 
> > appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> > 
> > With regards,
> > Daniel
> 
> I feel avocado is directed towards booting full fledged guest OS.

That's essentially what this is doing - its a custom guest OS rather
than a common distro IIUC

> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Well avocado is as reliable as the tests you write for it. The problems
are largely around the images being used in avocado. If the biosbits
testing system is reliable, then avocado will be too, and if they not
reliable, then it will affect qtest too.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:
> Now that you mention it, avocado does feel like a more appropriate fit.
> IIUC the biosbits project appears to be effectively providing a custom
> guest OS ISO image. IOW this testing is quite biased towards being
> integration testing which is the target of avocado, while qtest is much
> more to the unit testing end of the spectrum. This would avoid all the
> discussion and patches around introducing python to qtest
> 
> With regards,
> Daniel

Yes it's an ISO image but not a full OS. So IMHO not really a
integration test, more like  a unit test for ACPI - current
tests we have just compare ACPI to expected file which is
okay but very limited.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  
> > > wrote:
> > > > 
> > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > No problem with that. So that's venv. But do we need 
> > > > > > > > > > > > > pip and pulling
> > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > 
> > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > 
> > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > 
> > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > 
> > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > the net anyway.
> > > > > > > > > > > Are the libraries in question not packaged on major 
> > > > > > > > > > > distros?
> > > > > > > > > > 
> > > > > > > > > > Currently I only need this:
> > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > 
> > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > 
> > > > > > > > > > I do not think this is packaged by default. It's such a 
> > > > > > > > > > basic library
> > > > > > > > > > for parsing test output that maybe we can keep this 
> > > > > > > > > > somewhere within
> > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > 
> > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > 
> > > > > > > > Unlike BITS, this one is likely going to be maintained for a 
> > > > > > > > while and
> > > > > > > > will receive new releases through
> > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > 
> > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > 
> > > > > > > John Snow is currently working on the "Pythonification" of 
> > > > > > > various QEMU
> > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > 
> > > > > > >Thomas
> > > > > > 
> > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > all options: check for the module and use it if there, if not
> > > > > > use one from system if not there install with pip ..
> > > > > > But yea, I'm not sure what's best either.
> > > > > 
> > > > > submodules create a dependency on an internet connection, too. So 
> > > > > before you
> > > > > add yet another submodule (which have a couple of other 
> > > > > disadvantages), I
> > > > > think you could also directly use the venv here.
> > > > 
> > > > Definitely not submodules.
> > > > 
> > > > We need to get out of the mindset that submodules are needed for every 
> > > > new
> > > > dependancy we add. Submodules are only appropriate if the external 
> > > > project
> > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > need to bundle in order to prevent a regression for previously deployed
> > > > QEMU installs where the dependancy is known not to exist on all our
> > > > supported platforms.
> > > > 
> > > > This does not apply in this case, because the proposed use of tappy is
> > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > it does, then use it, otherwise skip the tests that need it. The user 
> > > > can
> > > > arrange to install tappy, as they do with the majority of other deps.
> > > > 
> > > > If John's venv stuff is relevant, then we don't even need the meson 
> > > > checks,
> > > > just delegate to the venv setup.
> > > > 
> > > > Regardless, no submodules are needed or desirable.
> > > 
> > > What about keeping biosbits stuff? Source or pre-built.
> > 
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> > 
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> > 
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 06:12:48AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance.
> 
> Well looks like classical mere aggregation to me ... license issues
> need to be figured out if we are to distribute things but I think
> this is basically what distros do anyway.
> 
> And I doubt we want to support arbitrary versions of grub etc,
> they are very distro specific tools.
> I don't see why we can't have the resulting ISOs in some submodule -
> nothing requires us to distribute it in qemu tarballs.

If we don't distribute it in the tarballs then these tests won't be
usable for testing release builds of QEMU which feels wrong to me.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> If it is actually booting a real guest image (from biosbits) and interacting
> with it, then it does feel like the scope of this testing is more appropriate
> to QEMU's avocado framework than qtest, especially given the desire to use
> python for it all.
> 
> With regards,
> Daniel

I feel avocado is directed towards booting full fledged guest OS.
It makes it much easier to figure out guest issues but it also
prone to false positives and is harder to debug as a result.
Booting a minimal image like this shouldn't require that.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> So bundling a pre-built biosbits in QEMU appears to mean that we're in
> turn going to unexpectedly bundle a bunch of other 3rd party projects
> too, all with dubious license compliance.

Well looks like classical mere aggregation to me ... license issues
need to be figured out if we are to distribute things but I think
this is basically what distros do anyway.

And I doubt we want to support arbitrary versions of grub etc,
they are very distro specific tools.
I don't see why we can't have the resulting ISOs in some submodule -
nothing requires us to distribute it in qemu tarballs.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 02:05:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth  wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  
> > >> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> >  On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >  No problem with that. So that's venv. But do we need pip and 
> >  pulling
> >  packages from the net during testing?
> > >>>
> > >>> We do that too. See requirements.txt in tests/
> > >>> Following two are downloaded:
> > >>> avocado-framework==88.1
> > >>> pycdlib==1.11.0
> > >>>
> > >>> Also see this line in Makefie.include:
> > >>>
> > >>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>
> > >> Right but that's avocado since it pulls lots of stuff from
> > >> the net anyway.
> > >> Are the libraries in question not packaged on major distros?
> > >
> > > Currently I only need this:
> > > https://github.com/python-tap/tappy
> > > which is the basic TAP processing library for python.
> > >
> > > It seems its only installed through pip:
> > > https://tappy.readthedocs.io/en/latest/
> > >
> > > I do not think this is packaged by default. It's such a basic 
> > > library
> > > for parsing test output that maybe we can keep this somewhere 
> > > within
> > > the python src tree? Not sure ...
> > 
> >  It's pretty small for sure. Another submodule?
> > >>>
> > >>> Unlike BITS, this one is likely going to be maintained for a while 
> > >>> and
> > >>> will receive new releases through
> > >>> https://pypi.org/project/tap.py/
> > >>> so forking is OK but someone has to keep this updated.
> > >>>
> > >>> I am open to anything. Whatever feels right is fine to me.
> > >>
> > >> John Snow is currently working on the "Pythonification" of various 
> > >> QEMU
> > >> bits, I think you should loop him into this discussion, too.
> > >>
> > >>Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> >  submodules create a dependency on an internet connection, too. So 
> >  before you
> >  add yet another submodule (which have a couple of other 
> >  disadvantages), I
> >  think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for every 
> > >>> new
> > >>> dependancy we add. Submodules are only appropriate if the external 
> > >>> project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > >>> need to bundle in order to prevent a regression for previously deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy is
> > >>> merely for a test case. Meson just needs to check if tappy exists and if
> > >>> it does, then use it, otherwise skip the tests that need it. The user 
> > >>> can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson 
> > >>> checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 02:19:41PM +0530, Ani Sinha wrote:
> 
> 
> On Tue, Jun 28, 2022 at 14:05 Ani Sinha  wrote:
> 
> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth  wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
> berra...@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> >  On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >  No problem with that. So that's venv. But do we need pip 
> and
> pulling
> >  packages from the net during testing?
> > >>>
> > >>> We do that too. See requirements.txt in tests/
> > >>> Following two are downloaded:
> > >>> avocado-framework==88.1
> > >>> pycdlib==1.11.0
> > >>>
> > >>> Also see this line in Makefie.include:
> > >>>
> > >>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>
> > >> Right but that's avocado since it pulls lots of stuff from
> > >> the net anyway.
> > >> Are the libraries in question not packaged on major distros?
> > >
> > > Currently I only need this:
> > > https://github.com/python-tap/tappy
> > > which is the basic TAP processing library for python.
> > >
> > > It seems its only installed through pip:
> > > https://tappy.readthedocs.io/en/latest/
> > >
> > > I do not think this is packaged by default. It's such a basic
> library
> > > for parsing test output that maybe we can keep this somewhere
> within
> > > the python src tree? Not sure ...
> > 
> >  It's pretty small for sure. Another submodule?
> > >>>
> > >>> Unlike BITS, this one is likely going to be maintained for a
> while and
> > >>> will receive new releases through
> > >>> https://pypi.org/project/tap.py/
> > >>> so forking is OK but someone has to keep this updated.
> > >>>
> > >>> I am open to anything. Whatever feels right is fine to me.
> > >>
> > >> John Snow is currently working on the "Pythonification" of 
> various
> QEMU
> > >> bits, I think you should loop him into this discussion, too.
> > >>
> > >>    Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> >  submodules create a dependency on an internet connection, too. So
> before you
> >  add yet another submodule (which have a couple of other
> disadvantages), I
> >  think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for
> every new
> > >>> dependancy we add. Submodules are only appropriate if the external
> project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or 
> if
> we
> > >>> need to bundle in order to prevent a regression for previously
> deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy
> is
> > >>> merely for a test case. Meson just needs to check if tappy exists 
> and
> if
> > >>> it does, then use it, otherwise skip the tests that need it. The 
> user
> can
> > >>> arrange to install tappy, as they do with the majority of other 
> deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson
> checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 08:48:42AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip 
> > > > > > > > > > and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > > 
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > > 
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > 
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > 
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > 
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > > 
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > 
> > > > > > > I do not think this is packaged by default. It's such a basic 
> > > > > > > library
> > > > > > > for parsing test output that maybe we can keep this somewhere 
> > > > > > > within
> > > > > > > the python src tree? Not sure ...
> > > > > > 
> > > > > > It's pretty small for sure. Another submodule?
> > > > > 
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > > 
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > > 
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > > 
> > > >   Thomas
> > > 
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
> 
> Definitely not submodules.
> 
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
> 
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.

I think that's reasonable, except I think we should
make configure do the checks as that is when users see the output
of what is available.

By the way, it might make sense
to have a script setting up developer dependencies for qemu
for major platforms.

> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.

IIUC venv does not install things itself. pip does.

> Regardless, no submodules are needed or desirable.
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and 
> > > > > > > > > pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic 
> > > > > > library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too.

Yes but this dependency is explicit.

> So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.
> 
>  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth  wrote:
>
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  
> >> wrote:
> >>>
> >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
>  On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>  No problem with that. So that's venv. But do we need pip and 
>  pulling
>  packages from the net during testing?
> >>>
> >>> We do that too. See requirements.txt in tests/
> >>> Following two are downloaded:
> >>> avocado-framework==88.1
> >>> pycdlib==1.11.0
> >>>
> >>> Also see this line in Makefie.include:
> >>>
> >>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>
> >> Right but that's avocado since it pulls lots of stuff from
> >> the net anyway.
> >> Are the libraries in question not packaged on major distros?
> >
> > Currently I only need this:
> > https://github.com/python-tap/tappy
> > which is the basic TAP processing library for python.
> >
> > It seems its only installed through pip:
> > https://tappy.readthedocs.io/en/latest/
> >
> > I do not think this is packaged by default. It's such a basic 
> > library
> > for parsing test output that maybe we can keep this somewhere within
> > the python src tree? Not sure ...
> 
>  It's pretty small for sure. Another submodule?
> >>>
> >>> Unlike BITS, this one is likely going to be maintained for a while and
> >>> will receive new releases through
> >>> https://pypi.org/project/tap.py/
> >>> so forking is OK but someone has to keep this updated.
> >>>
> >>> I am open to anything. Whatever feels right is fine to me.
> >>
> >> John Snow is currently working on the "Pythonification" of various QEMU
> >> bits, I think you should loop him into this discussion, too.
> >>
> >>Thomas
> >
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
>  submodules create a dependency on an internet connection, too. So before 
>  you
>  add yet another submodule (which have a couple of other disadvantages), I
>  think you could also directly use the venv here.
> >>>
> >>> Definitely not submodules.
> >>>
> >>> We need to get out of the mindset that submodules are needed for every new
> >>> dependancy we add. Submodules are only appropriate if the external project
> >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> >>> need to bundle in order to prevent a regression for previously deployed
> >>> QEMU installs where the dependancy is known not to exist on all our
> >>> supported platforms.
> >>>
> >>> This does not apply in this case, because the proposed use of tappy is
> >>> merely for a test case. Meson just needs to check if tappy exists and if
> >>> it does, then use it, otherwise skip the tests that need it. The user can
> >>> arrange to install tappy, as they do with the majority of other deps.
> >>>
> >>> If John's venv stuff is relevant, then we don't even need the meson 
> >>> checks,
> >>> just delegate to the venv setup.
> >>>
> >>> Regardless, no submodules are needed or desirable.
> >>
> >> What about keeping biosbits stuff? Source or pre-built.
> >
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> >
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> >
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance. I don't think this looks like
> > something we should have in qemu.git or qemu tarballs. It will also
> > make it challenging for the distro to take biosbits at all, unless
> > those 3rd party bundles can be eliminated in favour of using existing
> > builds their have packaged for grub, python, libffi, etc.
>
> So if this depends on some third party binary bits, I think this is pretty
> similar to the tests in the avocado directory ... there we download third
> party binaries, too... Wouldn't it make sense to adapt your tests to that
> 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 14:05 Ani Sinha  wrote:

> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth  wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
> berra...@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> >  On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >  No problem with that. So that's venv. But do we need pip
> and pulling
> >  packages from the net during testing?
> > >>>
> > >>> We do that too. See requirements.txt in tests/
> > >>> Following two are downloaded:
> > >>> avocado-framework==88.1
> > >>> pycdlib==1.11.0
> > >>>
> > >>> Also see this line in Makefie.include:
> > >>>
> > >>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>
> > >> Right but that's avocado since it pulls lots of stuff from
> > >> the net anyway.
> > >> Are the libraries in question not packaged on major distros?
> > >
> > > Currently I only need this:
> > > https://github.com/python-tap/tappy
> > > which is the basic TAP processing library for python.
> > >
> > > It seems its only installed through pip:
> > > https://tappy.readthedocs.io/en/latest/
> > >
> > > I do not think this is packaged by default. It's such a basic
> library
> > > for parsing test output that maybe we can keep this somewhere
> within
> > > the python src tree? Not sure ...
> > 
> >  It's pretty small for sure. Another submodule?
> > >>>
> > >>> Unlike BITS, this one is likely going to be maintained for a
> while and
> > >>> will receive new releases through
> > >>> https://pypi.org/project/tap.py/
> > >>> so forking is OK but someone has to keep this updated.
> > >>>
> > >>> I am open to anything. Whatever feels right is fine to me.
> > >>
> > >> John Snow is currently working on the "Pythonification" of
> various QEMU
> > >> bits, I think you should loop him into this discussion, too.
> > >>
> > >>Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> >  submodules create a dependency on an internet connection, too. So
> before you
> >  add yet another submodule (which have a couple of other
> disadvantages), I
> >  think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for
> every new
> > >>> dependancy we add. Submodules are only appropriate if the external
> project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or
> if we
> > >>> need to bundle in order to prevent a regression for previously
> deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy
> is
> > >>> merely for a test case. Meson just needs to check if tappy exists
> and if
> > >>> it does, then use it, otherwise skip the tests that need it. The
> user can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson
> checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 10.23, Daniel P. Berrangé wrote:

On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  wrote:


On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:

On 28/06/2022 09.10, Michael S. Tsirkin wrote:

On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?


We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))


Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?


Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...


It's pretty small for sure. Another submodule?


Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I am open to anything. Whatever feels right is fine to me.


John Snow is currently working on the "Pythonification" of various QEMU
bits, I think you should loop him into this discussion, too.

   Thomas


submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.


submodules create a dependency on an internet connection, too. So before you
add yet another submodule (which have a couple of other disadvantages), I
think you could also directly use the venv here.


Definitely not submodules.

We need to get out of the mindset that submodules are needed for every new
dependancy we add. Submodules are only appropriate if the external project
is designed to be used as a copylib (eg the keycodemapdb tool), or if we
need to bundle in order to prevent a regression for previously deployed
QEMU installs where the dependancy is known not to exist on all our
supported platforms.

This does not apply in this case, because the proposed use of tappy is
merely for a test case. Meson just needs to check if tappy exists and if
it does, then use it, otherwise skip the tests that need it. The user can
arrange to install tappy, as they do with the majority of other deps.

If John's venv stuff is relevant, then we don't even need the meson checks,
just delegate to the venv setup.

Regardless, no submodules are needed or desirable.


What about keeping biosbits stuff? Source or pre-built.


Shipping them as pre-built binaries in QEMU is not a viable option
IMHO, especially for grub as a GPL'd project we need to be extremely
clear about the exact corresponding source and build process for any
binary.

For this kind of thing I would generally expect the distro to provide
packages that we consume. Looking at biosbits I see it is itself
bundling a bunch more 3rd party projects, libffi, grub2, and including
even an ancient version of python as a submodule.

So bundling a pre-built biosbits in QEMU appears to mean that we're in
turn going to unexpectedly bundle a bunch of other 3rd party projects
too, all with dubious license compliance. I don't think this looks like
something we should have in qemu.git or qemu tarballs. It will also
make it challenging for the distro to take biosbits at all, unless
those 3rd party bundles can be eliminated in favour of using existing
builds their have packaged for grub, python, libffi, etc.


So if this depends on some third party binary bits, I think this is pretty 
similar to the tests in the avocado directory ... there we download third 
party binaries, too... Wouldn't it make sense to adapt your tests to that 
framework?


 Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  
> wrote:
> >
> > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > No problem with that. So that's venv. But do we need pip 
> > > > > > > > > > > and pulling
> > > > > > > > > > > packages from the net during testing?
> > > > > > > > > >
> > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > Following two are downloaded:
> > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > >
> > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > >
> > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > >
> > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > the net anyway.
> > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > >
> > > > > > > > Currently I only need this:
> > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > which is the basic TAP processing library for python.
> > > > > > > >
> > > > > > > > It seems its only installed through pip:
> > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > >
> > > > > > > > I do not think this is packaged by default. It's such a basic 
> > > > > > > > library
> > > > > > > > for parsing test output that maybe we can keep this somewhere 
> > > > > > > > within
> > > > > > > > the python src tree? Not sure ...
> > > > > > >
> > > > > > > It's pretty small for sure. Another submodule?
> > > > > >
> > > > > > Unlike BITS, this one is likely going to be maintained for a while 
> > > > > > and
> > > > > > will receive new releases through
> > > > > > https://pypi.org/project/tap.py/
> > > > > > so forking is OK but someone has to keep this updated.
> > > > > >
> > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > >
> > > > > John Snow is currently working on the "Pythonification" of various 
> > > > > QEMU
> > > > > bits, I think you should loop him into this discussion, too.
> > > > >
> > > > >   Thomas
> > > >
> > > > submodule does not mean we fork necessarily. We could have
> > > > all options: check for the module and use it if there, if not
> > > > use one from system if not there install with pip ..
> > > > But yea, I'm not sure what's best either.
> > >
> > > submodules create a dependency on an internet connection, too. So before 
> > > you
> > > add yet another submodule (which have a couple of other disadvantages), I
> > > think you could also directly use the venv here.
> >
> > Definitely not submodules.
> >
> > We need to get out of the mindset that submodules are needed for every new
> > dependancy we add. Submodules are only appropriate if the external project
> > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > need to bundle in order to prevent a regression for previously deployed
> > QEMU installs where the dependancy is known not to exist on all our
> > supported platforms.
> >
> > This does not apply in this case, because the proposed use of tappy is
> > merely for a test case. Meson just needs to check if tappy exists and if
> > it does, then use it, otherwise skip the tests that need it. The user can
> > arrange to install tappy, as they do with the majority of other deps.
> >
> > If John's venv stuff is relevant, then we don't even need the meson checks,
> > just delegate to the venv setup.
> >
> > Regardless, no submodules are needed or desirable.
> 
> What about keeping biosbits stuff? Source or pre-built.

Shipping them as pre-built binaries in QEMU is not a viable option
IMHO, especially for grub as a GPL'd project we need to be extremely
clear about the exact corresponding source and build process for any
binary.

For this kind of thing I would generally expect the distro to provide
packages that we consume. Looking at biosbits I see it is itself
bundling a bunch more 3rd party projects, libffi, grub2, and including
even an ancient version of python as a submodule.

So bundling a pre-built biosbits in QEMU appears to mean that we're in
turn going to unexpectedly bundle a bunch of other 3rd party projects
too, all with dubious license compliance. I don't think this looks like
something we should have in qemu.git or qemu tarballs. It will also
make it challenging for the distro to take biosbits at all, unless
those 3rd party bundles can be eliminated in favour of using existing
builds their have packaged for grub, python, libffi, etc.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  wrote:
> >
> > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  
> > > wrote:
> > > >
> > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > The README file is added describing the directory structure and the 
> > > > > purpose
> > > > > of every file it contains. It also describes how to add new tests, 
> > > > > make changes
> > > > > to existing tests or bits config files or regenerate the bits 
> > > > > software.
> > > > >
> > > > > Signed-off-by: Ani Sinha 
> > > > > ---
> > > > >  tests/qtest/acpi-bits/README | 168 
> > > > > +++
> > > > >  1 file changed, 168 insertions(+)
> > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > >
> > > > > diff --git a/tests/qtest/acpi-bits/README 
> > > > > b/tests/qtest/acpi-bits/README
> > > > > new file mode 100644
> > > > > index 00..97b15f1665
> > > > > --- /dev/null
> > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > @@ -0,0 +1,168 @@
> > > > > +=
> > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > +=
> > > > > +
> > > > > +Biosbits is a software written by Josh Triplett that can be 
> > > > > downloaded by
> > > > > +visiting https://biosbits.org/. The github codebase can be found 
> > > > > here:
> > > > > +https://github.com/biosbits/bits/tree/master. It is a software that 
> > > > > exercizes
> > > > > +the bios components such as acpi and smbios tables directly through 
> > > > > acpica
> > > > > +bios interpreter (a freely available C based library written by 
> > > > > Intel,
> > > > > +downloadable from https://acpica.org/ and is included with biosbits) 
> > > > > without an
> > > > > +operating system getting involved in between.
> > > > > +There are several advantages to directly testing the bios in a real 
> > > > > physical
> > > > > +machine or VM as opposed to indirectly discovering bios issues 
> > > > > through the
> > > > > +operating system. For one thing, the OSes tend to hide bios problems 
> > > > > from the
> > > > > +end user. The other is that we have more control of what we wanted 
> > > > > to test
> > > > > +and how by directly using acpica interpreter on top of the bios on a 
> > > > > running
> > > > > +system. More details on the inspiration for developing biosbits and 
> > > > > its real
> > > > > +life uses can be found in (a) and (b).
> > > > > +This directory contains QEMU qtests written in python that exercizes 
> > > > > the QEMU
> > > > > +bios components using biosbits and reports test failures.
> > > > > +
> > > > > +These tests use python virtual environment. In debian/ubuntu system, 
> > > > > the tests
> > > > > +would require python3.8-venv and python3-pip packages to be 
> > > > > installed.
> > > >
> > > > Why do we mess with venv and pip? Certainly possible but
> > > > what's wrong with using distro provided packages?
> > >
> > > There are two things:
> > > (a) We are already using pip and venv for our avocado based
> > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > tests.
> > > (b) the venv is primarily needed because I wanted to take advantage of
> > > our rich python library that handles QEMU based machines. There are
> > > python qtest libraries as well. These are well tested and used with
> > > integration tests and I wanted to keep the test part of the code
> > > simple by simply reusing them. however, in order to use them, we need
> > > a venv environment within which these qemu python libraries are
> > > installed. Integration tests does the same thing.
> > >
> > > A note about my language of choice - python. I gave a lot of thoughts
> > > on this. We do not do a lot of stuff here. All we do is:
> > > (a) generate bits iso.
> > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within 
> > > the vm.
> > > (c) collect and analyze logs.
> > >
> > > We are not inspecting guest memory or manipulating devices or pci
> > > buses. We do not need the power of C here. We need something that is
> > > simple to write, easy to maintain and understand and can deal with
> > > things like manipulating text files and configs easily. Python seems a
> > > better fit for the role.
> >
> > No problem with that. So that's venv. But do we need pip and pulling
> > packages from the net during testing?
> 
> We do that too. See requirements.txt in tests/
> Following two are downloaded:
> avocado-framework==88.1
> pycdlib==1.11.0
> 
> Also see this line in Makefie.include:
> 
> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?

> >
> > > >
> 

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 09.49, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth  wrote:


On 28/06/2022 09.10, Michael S. Tsirkin wrote:

On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?


We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))


Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?


Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...


It's pretty small for sure. Another submodule?


Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I am open to anything. Whatever feels right is fine to me.


John Snow is currently working on the "Pythonification" of various QEMU
bits, I think you should loop him into this discussion, too.

   Thomas


submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.


submodules create a dependency on an internet connection, too. So before you
add yet another submodule (which have a couple of other disadvantages), I
think you could also directly use the venv here.


Not sure what you mean by directly using the venv.


I meant installing the dependency via pip in a venv, as you are currently 
doing it via the acpi-bits-test-venv.sh script.


 Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé  wrote:
>
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip 
> > > > > > > > > > and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > >
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > >
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > >
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > >
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > >
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > >
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > >
> > > > > > > I do not think this is packaged by default. It's such a basic 
> > > > > > > library
> > > > > > > for parsing test output that maybe we can keep this somewhere 
> > > > > > > within
> > > > > > > the python src tree? Not sure ...
> > > > > >
> > > > > > It's pretty small for sure. Another submodule?
> > > > >
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > >
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > >
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > >
> > > >   Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> >
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
>
> Definitely not submodules.
>
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
>
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.
>
> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.
>
> Regardless, no submodules are needed or desirable.

What about keeping biosbits stuff? Source or pre-built.


>
> With regards,
> Daniel
> --
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth  wrote:
>
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>  No problem with that. So that's venv. But do we need pip and pulling
>  packages from the net during testing?
> >>>
> >>> We do that too. See requirements.txt in tests/
> >>> Following two are downloaded:
> >>> avocado-framework==88.1
> >>> pycdlib==1.11.0
> >>>
> >>> Also see this line in Makefie.include:
> >>>
> >>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>
> >> Right but that's avocado since it pulls lots of stuff from
> >> the net anyway.
> >> Are the libraries in question not packaged on major distros?
> >
> > Currently I only need this:
> > https://github.com/python-tap/tappy
> > which is the basic TAP processing library for python.
> >
> > It seems its only installed through pip:
> > https://tappy.readthedocs.io/en/latest/
> >
> > I do not think this is packaged by default. It's such a basic library
> > for parsing test output that maybe we can keep this somewhere within
> > the python src tree? Not sure ...
> 
>  It's pretty small for sure. Another submodule?
> >>>
> >>> Unlike BITS, this one is likely going to be maintained for a while and
> >>> will receive new releases through
> >>> https://pypi.org/project/tap.py/
> >>> so forking is OK but someone has to keep this updated.
> >>>
> >>> I am open to anything. Whatever feels right is fine to me.
> >>
> >> John Snow is currently working on the "Pythonification" of various QEMU
> >> bits, I think you should loop him into this discussion, too.
> >>
> >>   Thomas
> >
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
>
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Not sure what you mean by directly using the venv.

>
>   Thomas
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P . Berrangé]

On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and 
> > > > > > > > > pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic 
> > > > > > library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Definitely not submodules.

We need to get out of the mindset that submodules are needed for every new
dependancy we add. Submodules are only appropriate if the external project
is designed to be used as a copylib (eg the keycodemapdb tool), or if we
need to bundle in order to prevent a regression for previously deployed
QEMU installs where the dependancy is known not to exist on all our
supported platforms.

This does not apply in this case, because the proposed use of tappy is
merely for a test case. Meson just needs to check if tappy exists and if
it does, then use it, otherwise skip the tests that need it. The user can
arrange to install tappy, as they do with the majority of other deps.

If John's venv stuff is relevant, then we don't even need the meson checks,
just delegate to the venv setup.

Regardless, no submodules are needed or desirable.

With regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > No problem with that. So that's venv. But do we need pip and 
> > > > > > > pulling
> > > > > > > packages from the net during testing?
> > > > > > 
> > > > > > We do that too. See requirements.txt in tests/
> > > > > > Following two are downloaded:
> > > > > > avocado-framework==88.1
> > > > > > pycdlib==1.11.0
> > > > > > 
> > > > > > Also see this line in Makefie.include:
> > > > > > 
> > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > 
> > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > the net anyway.
> > > > > Are the libraries in question not packaged on major distros?
> > > > 
> > > > Currently I only need this:
> > > > https://github.com/python-tap/tappy
> > > > which is the basic TAP processing library for python.
> > > > 
> > > > It seems its only installed through pip:
> > > > https://tappy.readthedocs.io/en/latest/
> > > > 
> > > > I do not think this is packaged by default. It's such a basic library
> > > > for parsing test output that maybe we can keep this somewhere within
> > > > the python src tree? Not sure ...
> > > 
> > > It's pretty small for sure. Another submodule?
> > 
> > Unlike BITS, this one is likely going to be maintained for a while and
> > will receive new releases through
> > https://pypi.org/project/tap.py/
> > so forking is OK but someone has to keep this updated.
> > 
> > I am open to anything. Whatever feels right is fine to me.
> 
> John Snow is currently working on the "Pythonification" of various QEMU
> bits, I think you should loop him into this discussion, too.
> 
>  Thomas

submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 09.10, Michael S. Tsirkin wrote:

On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?


We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))


Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?


Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...


It's pretty small for sure. Another submodule?


Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I am open to anything. Whatever feels right is fine to me.


John Snow is currently working on the "Pythonification" of various QEMU
bits, I think you should loop him into this discussion, too.

  Thomas


submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.


submodules create a dependency on an internet connection, too. So before you 
add yet another submodule (which have a couple of other disadvantages), I 
think you could also directly use the venv here.


 Thomas

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

+danPb who pointed me to the TAP stuff. Maybe he has some good ideas on this.

On Tue, Jun 28, 2022 at 12:27 PM Ani Sinha  wrote:
>
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin  wrote:
> >
> > On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin  
> > > wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  
> > > > > wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin 
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > > The README file is added describing the directory structure 
> > > > > > > > > and the purpose
> > > > > > > > > of every file it contains. It also describes how to add new 
> > > > > > > > > tests, make changes
> > > > > > > > > to existing tests or bits config files or regenerate the bits 
> > > > > > > > > software.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Ani Sinha 
> > > > > > > > > ---
> > > > > > > > >  tests/qtest/acpi-bits/README | 168 
> > > > > > > > > +++
> > > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > > >
> > > > > > > > > diff --git a/tests/qtest/acpi-bits/README 
> > > > > > > > > b/tests/qtest/acpi-bits/README
> > > > > > > > > new file mode 100644
> > > > > > > > > index 00..97b15f1665
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > > +=
> > > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > > +=
> > > > > > > > > +
> > > > > > > > > +Biosbits is a software written by Josh Triplett that can be 
> > > > > > > > > downloaded by
> > > > > > > > > +visiting https://biosbits.org/. The github codebase can be 
> > > > > > > > > found here:
> > > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a 
> > > > > > > > > software that exercizes
> > > > > > > > > +the bios components such as acpi and smbios tables directly 
> > > > > > > > > through acpica
> > > > > > > > > +bios interpreter (a freely available C based library written 
> > > > > > > > > by Intel,
> > > > > > > > > +downloadable from https://acpica.org/ and is included with 
> > > > > > > > > biosbits) without an
> > > > > > > > > +operating system getting involved in between.
> > > > > > > > > +There are several advantages to directly testing the bios in 
> > > > > > > > > a real physical
> > > > > > > > > +machine or VM as opposed to indirectly discovering bios 
> > > > > > > > > issues through the
> > > > > > > > > +operating system. For one thing, the OSes tend to hide bios 
> > > > > > > > > problems from the
> > > > > > > > > +end user. The other is that we have more control of what we 
> > > > > > > > > wanted to test
> > > > > > > > > +and how by directly using acpica interpreter on top of the 
> > > > > > > > > bios on a running
> > > > > > > > > +system. More details on the inspiration for developing 
> > > > > > > > > biosbits and its real
> > > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > > +This directory contains QEMU qtests written in python that 
> > > > > > > > > exercizes the QEMU
> > > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > > +
> > > > > > > > > +These tests use python virtual environment. In debian/ubuntu 
> > > > > > > > > system, the tests
> > > > > > > > > +would require python3.8-venv and python3-pip packages to be 
> > > > > > > > > installed.
> > > > > > > >
> > > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > > what's wrong with using distro provided packages?
> > > > > > >
> > > > > > > There are two things:
> > > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include 
> > > > > > > under
> > > > > > > tests.
> > > > > > > (b) the venv is primarily needed because I wanted to take 
> > > > > > > advantage of
> > > > > > > our rich python library that handles QEMU based machines. There 
> > > > > > > are
> > > > > > > python qtest libraries as well. These are well tested and used 
> > > > > > > with
> > > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > > simple by simply reusing them. however, in order to use them, we 
> > > > > > > need
> > > > > > > a venv environment within which these qemu python libraries are
> > > > > > > installed. Integration tests does the same thing.
> > > > > > >
> > > > > > > A note about my language 

venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 08.57, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin  wrote:


On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin  wrote:


On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  wrote:


On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:

On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  wrote:


On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:

The README file is added describing the directory structure and the purpose
of every file it contains. It also describes how to add new tests, make changes
to existing tests or bits config files or regenerate the bits software.

Signed-off-by: Ani Sinha 
---
  tests/qtest/acpi-bits/README | 168 +++
  1 file changed, 168 insertions(+)
  create mode 100644 tests/qtest/acpi-bits/README

diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
new file mode 100644
index 00..97b15f1665
--- /dev/null
+++ b/tests/qtest/acpi-bits/README
@@ -0,0 +1,168 @@
+=
+ACPI/SMBIOS QTESTS USING BIOSBITS
+=
+
+Biosbits is a software written by Josh Triplett that can be downloaded by
+visiting https://biosbits.org/. The github codebase can be found here:
+https://github.com/biosbits/bits/tree/master. It is a software that exercizes
+the bios components such as acpi and smbios tables directly through acpica
+bios interpreter (a freely available C based library written by Intel,
+downloadable from https://acpica.org/ and is included with biosbits) without an
+operating system getting involved in between.
+There are several advantages to directly testing the bios in a real physical
+machine or VM as opposed to indirectly discovering bios issues through the
+operating system. For one thing, the OSes tend to hide bios problems from the
+end user. The other is that we have more control of what we wanted to test
+and how by directly using acpica interpreter on top of the bios on a running
+system. More details on the inspiration for developing biosbits and its real
+life uses can be found in (a) and (b).
+This directory contains QEMU qtests written in python that exercizes the QEMU
+bios components using biosbits and reports test failures.
+
+These tests use python virtual environment. In debian/ubuntu system, the tests
+would require python3.8-venv and python3-pip packages to be installed.


Why do we mess with venv and pip? Certainly possible but
what's wrong with using distro provided packages?


There are two things:
(a) We are already using pip and venv for our avocado based
integration tests. Look for TESTS_VENV_DIR in Makefile.include under
tests.
(b) the venv is primarily needed because I wanted to take advantage of
our rich python library that handles QEMU based machines. There are
python qtest libraries as well. These are well tested and used with
integration tests and I wanted to keep the test part of the code
simple by simply reusing them. however, in order to use them, we need
a venv environment within which these qemu python libraries are
installed. Integration tests does the same thing.

A note about my language of choice - python. I gave a lot of thoughts
on this. We do not do a lot of stuff here. All we do is:
(a) generate bits iso.
(b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
(c) collect and analyze logs.

We are not inspecting guest memory or manipulating devices or pci
buses. We do not need the power of C here. We need something that is
simple to write, easy to maintain and understand and can deal with
things like manipulating text files and configs easily. Python seems a
better fit for the role.


No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?


We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))


Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?


Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...


It's pretty small for sure. Another submodule?


Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin  wrote:
>
> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin  wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  
> > > > wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin 
> > > > > >  wrote:
> > > > > > >
> > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > The README file is added describing the directory structure and 
> > > > > > > > the purpose
> > > > > > > > of every file it contains. It also describes how to add new 
> > > > > > > > tests, make changes
> > > > > > > > to existing tests or bits config files or regenerate the bits 
> > > > > > > > software.
> > > > > > > >
> > > > > > > > Signed-off-by: Ani Sinha 
> > > > > > > > ---
> > > > > > > >  tests/qtest/acpi-bits/README | 168 
> > > > > > > > +++
> > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > >
> > > > > > > > diff --git a/tests/qtest/acpi-bits/README 
> > > > > > > > b/tests/qtest/acpi-bits/README
> > > > > > > > new file mode 100644
> > > > > > > > index 00..97b15f1665
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > +=
> > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > +=
> > > > > > > > +
> > > > > > > > +Biosbits is a software written by Josh Triplett that can be 
> > > > > > > > downloaded by
> > > > > > > > +visiting https://biosbits.org/. The github codebase can be 
> > > > > > > > found here:
> > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software 
> > > > > > > > that exercizes
> > > > > > > > +the bios components such as acpi and smbios tables directly 
> > > > > > > > through acpica
> > > > > > > > +bios interpreter (a freely available C based library written 
> > > > > > > > by Intel,
> > > > > > > > +downloadable from https://acpica.org/ and is included with 
> > > > > > > > biosbits) without an
> > > > > > > > +operating system getting involved in between.
> > > > > > > > +There are several advantages to directly testing the bios in a 
> > > > > > > > real physical
> > > > > > > > +machine or VM as opposed to indirectly discovering bios issues 
> > > > > > > > through the
> > > > > > > > +operating system. For one thing, the OSes tend to hide bios 
> > > > > > > > problems from the
> > > > > > > > +end user. The other is that we have more control of what we 
> > > > > > > > wanted to test
> > > > > > > > +and how by directly using acpica interpreter on top of the 
> > > > > > > > bios on a running
> > > > > > > > +system. More details on the inspiration for developing 
> > > > > > > > biosbits and its real
> > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > +This directory contains QEMU qtests written in python that 
> > > > > > > > exercizes the QEMU
> > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > +
> > > > > > > > +These tests use python virtual environment. In debian/ubuntu 
> > > > > > > > system, the tests
> > > > > > > > +would require python3.8-venv and python3-pip packages to be 
> > > > > > > > installed.
> > > > > > >
> > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > what's wrong with using distro provided packages?
> > > > > >
> > > > > > There are two things:
> > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > > tests.
> > > > > > (b) the venv is primarily needed because I wanted to take advantage 
> > > > > > of
> > > > > > our rich python library that handles QEMU based machines. There are
> > > > > > python qtest libraries as well. These are well tested and used with
> > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > simple by simply reusing them. however, in order to use them, we 
> > > > > > need
> > > > > > a venv environment within which these qemu python libraries are
> > > > > > installed. Integration tests does the same thing.
> > > > > >
> > > > > > A note about my language of choice - python. I gave a lot of 
> > > > > > thoughts
> > > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > > (a) generate bits iso.
> > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests 
> > > > > > within the vm.
> > > > > > (c) collect and analyze logs.
> > > > > >
> > > > > > We are not inspecting guest memory 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin  wrote:
>
> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  wrote:
> > >
> > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  
> > > > wrote:
> > > > >
> > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > The README file is added describing the directory structure and the 
> > > > > > purpose
> > > > > > of every file it contains. It also describes how to add new tests, 
> > > > > > make changes
> > > > > > to existing tests or bits config files or regenerate the bits 
> > > > > > software.
> > > > > >
> > > > > > Signed-off-by: Ani Sinha 
> > > > > > ---
> > > > > >  tests/qtest/acpi-bits/README | 168 
> > > > > > +++
> > > > > >  1 file changed, 168 insertions(+)
> > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > >
> > > > > > diff --git a/tests/qtest/acpi-bits/README 
> > > > > > b/tests/qtest/acpi-bits/README
> > > > > > new file mode 100644
> > > > > > index 00..97b15f1665
> > > > > > --- /dev/null
> > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > @@ -0,0 +1,168 @@
> > > > > > +=
> > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > +=
> > > > > > +
> > > > > > +Biosbits is a software written by Josh Triplett that can be 
> > > > > > downloaded by
> > > > > > +visiting https://biosbits.org/. The github codebase can be found 
> > > > > > here:
> > > > > > +https://github.com/biosbits/bits/tree/master. It is a software 
> > > > > > that exercizes
> > > > > > +the bios components such as acpi and smbios tables directly 
> > > > > > through acpica
> > > > > > +bios interpreter (a freely available C based library written by 
> > > > > > Intel,
> > > > > > +downloadable from https://acpica.org/ and is included with 
> > > > > > biosbits) without an
> > > > > > +operating system getting involved in between.
> > > > > > +There are several advantages to directly testing the bios in a 
> > > > > > real physical
> > > > > > +machine or VM as opposed to indirectly discovering bios issues 
> > > > > > through the
> > > > > > +operating system. For one thing, the OSes tend to hide bios 
> > > > > > problems from the
> > > > > > +end user. The other is that we have more control of what we wanted 
> > > > > > to test
> > > > > > +and how by directly using acpica interpreter on top of the bios on 
> > > > > > a running
> > > > > > +system. More details on the inspiration for developing biosbits 
> > > > > > and its real
> > > > > > +life uses can be found in (a) and (b).
> > > > > > +This directory contains QEMU qtests written in python that 
> > > > > > exercizes the QEMU
> > > > > > +bios components using biosbits and reports test failures.
> > > > > > +
> > > > > > +These tests use python virtual environment. In debian/ubuntu 
> > > > > > system, the tests
> > > > > > +would require python3.8-venv and python3-pip packages to be 
> > > > > > installed.
> > > > >
> > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > what's wrong with using distro provided packages?
> > > >
> > > > There are two things:
> > > > (a) We are already using pip and venv for our avocado based
> > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > tests.
> > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > our rich python library that handles QEMU based machines. There are
> > > > python qtest libraries as well. These are well tested and used with
> > > > integration tests and I wanted to keep the test part of the code
> > > > simple by simply reusing them. however, in order to use them, we need
> > > > a venv environment within which these qemu python libraries are
> > > > installed. Integration tests does the same thing.
> > > >
> > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > on this. We do not do a lot of stuff here. All we do is:
> > > > (a) generate bits iso.
> > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests 
> > > > within the vm.
> > > > (c) collect and analyze logs.
> > > >
> > > > We are not inspecting guest memory or manipulating devices or pci
> > > > buses. We do not need the power of C here. We need something that is
> > > > simple to write, easy to maintain and understand and can deal with
> > > > things like manipulating text files and configs easily. Python seems a
> > > > better fit for the role.
> > >
> > > No problem with that. So that's venv. But do we need pip and pulling
> > > packages from the net during testing?
> >
> > We do that too. See requirements.txt in tests/
> > Following two are downloaded:
> > avocado-framework==88.1
> > 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin  wrote:
> >
> > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  
> > > wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  
> > > > > wrote:
> > > > > >
> > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > The README file is added describing the directory structure and 
> > > > > > > the purpose
> > > > > > > of every file it contains. It also describes how to add new 
> > > > > > > tests, make changes
> > > > > > > to existing tests or bits config files or regenerate the bits 
> > > > > > > software.
> > > > > > >
> > > > > > > Signed-off-by: Ani Sinha 
> > > > > > > ---
> > > > > > >  tests/qtest/acpi-bits/README | 168 
> > > > > > > +++
> > > > > > >  1 file changed, 168 insertions(+)
> > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > >
> > > > > > > diff --git a/tests/qtest/acpi-bits/README 
> > > > > > > b/tests/qtest/acpi-bits/README
> > > > > > > new file mode 100644
> > > > > > > index 00..97b15f1665
> > > > > > > --- /dev/null
> > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > @@ -0,0 +1,168 @@
> > > > > > > +=
> > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > +=
> > > > > > > +
> > > > > > > +Biosbits is a software written by Josh Triplett that can be 
> > > > > > > downloaded by
> > > > > > > +visiting https://biosbits.org/. The github codebase can be found 
> > > > > > > here:
> > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software 
> > > > > > > that exercizes
> > > > > > > +the bios components such as acpi and smbios tables directly 
> > > > > > > through acpica
> > > > > > > +bios interpreter (a freely available C based library written by 
> > > > > > > Intel,
> > > > > > > +downloadable from https://acpica.org/ and is included with 
> > > > > > > biosbits) without an
> > > > > > > +operating system getting involved in between.
> > > > > > > +There are several advantages to directly testing the bios in a 
> > > > > > > real physical
> > > > > > > +machine or VM as opposed to indirectly discovering bios issues 
> > > > > > > through the
> > > > > > > +operating system. For one thing, the OSes tend to hide bios 
> > > > > > > problems from the
> > > > > > > +end user. The other is that we have more control of what we 
> > > > > > > wanted to test
> > > > > > > +and how by directly using acpica interpreter on top of the bios 
> > > > > > > on a running
> > > > > > > +system. More details on the inspiration for developing biosbits 
> > > > > > > and its real
> > > > > > > +life uses can be found in (a) and (b).
> > > > > > > +This directory contains QEMU qtests written in python that 
> > > > > > > exercizes the QEMU
> > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > +
> > > > > > > +These tests use python virtual environment. In debian/ubuntu 
> > > > > > > system, the tests
> > > > > > > +would require python3.8-venv and python3-pip packages to be 
> > > > > > > installed.
> > > > > >
> > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > what's wrong with using distro provided packages?
> > > > >
> > > > > There are two things:
> > > > > (a) We are already using pip and venv for our avocado based
> > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > tests.
> > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > our rich python library that handles QEMU based machines. There are
> > > > > python qtest libraries as well. These are well tested and used with
> > > > > integration tests and I wanted to keep the test part of the code
> > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > a venv environment within which these qemu python libraries are
> > > > > installed. Integration tests does the same thing.
> > > > >
> > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > (a) generate bits iso.
> > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests 
> > > > > within the vm.
> > > > > (c) collect and analyze logs.
> > > > >
> > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > buses. We do not need the power of C here. We need something that is
> > > > > simple to write, easy to maintain and understand and can deal with
> > > > > things like manipulating text files and configs easily. Python seems a
> > > > > better fit 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  wrote:
> >
> > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > The README file is added describing the directory structure and the 
> > > purpose
> > > of every file it contains. It also describes how to add new tests, make 
> > > changes
> > > to existing tests or bits config files or regenerate the bits software.
> > >
> > > Signed-off-by: Ani Sinha 
> > > ---
> > >  tests/qtest/acpi-bits/README | 168 +++
> > >  1 file changed, 168 insertions(+)
> > >  create mode 100644 tests/qtest/acpi-bits/README
> > >
> > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > new file mode 100644
> > > index 00..97b15f1665
> > > --- /dev/null
> > > +++ b/tests/qtest/acpi-bits/README
> > > @@ -0,0 +1,168 @@
> > > +=
> > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > +=
> > > +
> > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > +https://github.com/biosbits/bits/tree/master. It is a software that 
> > > exercizes
> > > +the bios components such as acpi and smbios tables directly through 
> > > acpica
> > > +bios interpreter (a freely available C based library written by Intel,
> > > +downloadable from https://acpica.org/ and is included with biosbits) 
> > > without an
> > > +operating system getting involved in between.
> > > +There are several advantages to directly testing the bios in a real 
> > > physical
> > > +machine or VM as opposed to indirectly discovering bios issues through 
> > > the
> > > +operating system. For one thing, the OSes tend to hide bios problems 
> > > from the
> > > +end user. The other is that we have more control of what we wanted to 
> > > test
> > > +and how by directly using acpica interpreter on top of the bios on a 
> > > running
> > > +system. More details on the inspiration for developing biosbits and its 
> > > real
> > > +life uses can be found in (a) and (b).
> > > +This directory contains QEMU qtests written in python that exercizes the 
> > > QEMU
> > > +bios components using biosbits and reports test failures.
> > > +
> > > +These tests use python virtual environment. In debian/ubuntu system, the 
> > > tests
> > > +would require python3.8-venv and python3-pip packages to be installed.
> >
> > Why do we mess with venv and pip? Certainly possible but
> > what's wrong with using distro provided packages?
> 
> There are two things:
> (a) We are already using pip and venv for our avocado based
> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> tests.
> (b) the venv is primarily needed because I wanted to take advantage of
> our rich python library that handles QEMU based machines. There are
> python qtest libraries as well. These are well tested and used with
> integration tests and I wanted to keep the test part of the code
> simple by simply reusing them. however, in order to use them, we need
> a venv environment within which these qemu python libraries are
> installed. Integration tests does the same thing.
> 
> A note about my language of choice - python. I gave a lot of thoughts
> on this. We do not do a lot of stuff here. All we do is:
> (a) generate bits iso.
> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the 
> vm.
> (c) collect and analyze logs.
> 
> We are not inspecting guest memory or manipulating devices or pci
> buses. We do not need the power of C here. We need something that is
> simple to write, easy to maintain and understand and can deal with
> things like manipulating text files and configs easily. Python seems a
> better fit for the role.

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?

> >
> > > +
> > > +A brief description of the contents of this directory follows:
> > > +
> > > +├── acpi-bits-test.py
> > > +├── acpi-bits-test-venv.sh
> > > +├── bits-config
> > > +│   ├── bits-cfg.txt
> > > +│   └── meson.build
> > > +├── bits-tests
> > > +│   ├── meson.build
> > > +│   ├── smbios.py
> > > +│   ├── smilatency.py
> > > +│   ├── testacpi.py
> > > +│   └── testcpuid.py
> > > +├── meson.build
> > > +├── prebuilt
> > > +│   ├── bits-2095-grub.tar.gz
> > > +│   ├── bits-2095.zip
> > > +│   └── meson.build
> > > +├── README
> > > +└── requirements.txt
> > > +
> > > +acpi-bits:
> > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the 
> > > virtual
> > > +   environment necessary for the actual python test script to run. 
> > > Amongst
> > > +   other things, it makes sure that QEMU python library is available 
> > > within
> > > +   that venv so that QEMU machines can be forked. QEMU python 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin  wrote:
>
> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  wrote:
> > >
> > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > The README file is added describing the directory structure and the 
> > > > purpose
> > > > of every file it contains. It also describes how to add new tests, make 
> > > > changes
> > > > to existing tests or bits config files or regenerate the bits software.
> > > >
> > > > Signed-off-by: Ani Sinha 
> > > > ---
> > > >  tests/qtest/acpi-bits/README | 168 +++
> > > >  1 file changed, 168 insertions(+)
> > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > >
> > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > new file mode 100644
> > > > index 00..97b15f1665
> > > > --- /dev/null
> > > > +++ b/tests/qtest/acpi-bits/README
> > > > @@ -0,0 +1,168 @@
> > > > +=
> > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > +=
> > > > +
> > > > +Biosbits is a software written by Josh Triplett that can be downloaded 
> > > > by
> > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > +https://github.com/biosbits/bits/tree/master. It is a software that 
> > > > exercizes
> > > > +the bios components such as acpi and smbios tables directly through 
> > > > acpica
> > > > +bios interpreter (a freely available C based library written by Intel,
> > > > +downloadable from https://acpica.org/ and is included with biosbits) 
> > > > without an
> > > > +operating system getting involved in between.
> > > > +There are several advantages to directly testing the bios in a real 
> > > > physical
> > > > +machine or VM as opposed to indirectly discovering bios issues through 
> > > > the
> > > > +operating system. For one thing, the OSes tend to hide bios problems 
> > > > from the
> > > > +end user. The other is that we have more control of what we wanted to 
> > > > test
> > > > +and how by directly using acpica interpreter on top of the bios on a 
> > > > running
> > > > +system. More details on the inspiration for developing biosbits and 
> > > > its real
> > > > +life uses can be found in (a) and (b).
> > > > +This directory contains QEMU qtests written in python that exercizes 
> > > > the QEMU
> > > > +bios components using biosbits and reports test failures.
> > > > +
> > > > +These tests use python virtual environment. In debian/ubuntu system, 
> > > > the tests
> > > > +would require python3.8-venv and python3-pip packages to be installed.
> > >
> > > Why do we mess with venv and pip? Certainly possible but
> > > what's wrong with using distro provided packages?
> >
> > There are two things:
> > (a) We are already using pip and venv for our avocado based
> > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > tests.
> > (b) the venv is primarily needed because I wanted to take advantage of
> > our rich python library that handles QEMU based machines. There are
> > python qtest libraries as well. These are well tested and used with
> > integration tests and I wanted to keep the test part of the code
> > simple by simply reusing them. however, in order to use them, we need
> > a venv environment within which these qemu python libraries are
> > installed. Integration tests does the same thing.
> >
> > A note about my language of choice - python. I gave a lot of thoughts
> > on this. We do not do a lot of stuff here. All we do is:
> > (a) generate bits iso.
> > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within 
> > the vm.
> > (c) collect and analyze logs.
> >
> > We are not inspecting guest memory or manipulating devices or pci
> > buses. We do not need the power of C here. We need something that is
> > simple to write, easy to maintain and understand and can deal with
> > things like manipulating text files and configs easily. Python seems a
> > better fit for the role.
>
> No problem with that. So that's venv. But do we need pip and pulling
> packages from the net during testing?

We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

>
> > >
> > > > +
> > > > +A brief description of the contents of this directory follows:
> > > > +
> > > > +├── acpi-bits-test.py
> > > > +├── acpi-bits-test-venv.sh
> > > > +├── bits-config
> > > > +│   ├── bits-cfg.txt
> > > > +│   └── meson.build
> > > > +├── bits-tests
> > > > +│   ├── meson.build
> > > > +│   ├── smbios.py
> > > > +│   ├── smilatency.py
> > > > +│   ├── testacpi.py
> > > > +│   └── testcpuid.py
> > > > +├── meson.build
> > > > +├── prebuilt
> > > > +│   

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin  wrote:
>
> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > The README file is added describing the directory structure and the purpose
> > of every file it contains. It also describes how to add new tests, make 
> > changes
> > to existing tests or bits config files or regenerate the bits software.
> >
> > Signed-off-by: Ani Sinha 
> > ---
> >  tests/qtest/acpi-bits/README | 168 +++
> >  1 file changed, 168 insertions(+)
> >  create mode 100644 tests/qtest/acpi-bits/README
> >
> > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > new file mode 100644
> > index 00..97b15f1665
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/README
> > @@ -0,0 +1,168 @@
> > +=
> > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > +=
> > +
> > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > +visiting https://biosbits.org/. The github codebase can be found here:
> > +https://github.com/biosbits/bits/tree/master. It is a software that 
> > exercizes
> > +the bios components such as acpi and smbios tables directly through acpica
> > +bios interpreter (a freely available C based library written by Intel,
> > +downloadable from https://acpica.org/ and is included with biosbits) 
> > without an
> > +operating system getting involved in between.
> > +There are several advantages to directly testing the bios in a real 
> > physical
> > +machine or VM as opposed to indirectly discovering bios issues through the
> > +operating system. For one thing, the OSes tend to hide bios problems from 
> > the
> > +end user. The other is that we have more control of what we wanted to test
> > +and how by directly using acpica interpreter on top of the bios on a 
> > running
> > +system. More details on the inspiration for developing biosbits and its 
> > real
> > +life uses can be found in (a) and (b).
> > +This directory contains QEMU qtests written in python that exercizes the 
> > QEMU
> > +bios components using biosbits and reports test failures.
> > +
> > +These tests use python virtual environment. In debian/ubuntu system, the 
> > tests
> > +would require python3.8-venv and python3-pip packages to be installed.
>
> Why do we mess with venv and pip? Certainly possible but
> what's wrong with using distro provided packages?

There are two things:
(a) We are already using pip and venv for our avocado based
integration tests. Look for TESTS_VENV_DIR in Makefile.include under
tests.
(b) the venv is primarily needed because I wanted to take advantage of
our rich python library that handles QEMU based machines. There are
python qtest libraries as well. These are well tested and used with
integration tests and I wanted to keep the test part of the code
simple by simply reusing them. however, in order to use them, we need
a venv environment within which these qemu python libraries are
installed. Integration tests does the same thing.

A note about my language of choice - python. I gave a lot of thoughts
on this. We do not do a lot of stuff here. All we do is:
(a) generate bits iso.
(b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
(c) collect and analyze logs.

We are not inspecting guest memory or manipulating devices or pci
buses. We do not need the power of C here. We need something that is
simple to write, easy to maintain and understand and can deal with
things like manipulating text files and configs easily. Python seems a
better fit for the role.

>
> > +
> > +A brief description of the contents of this directory follows:
> > +
> > +├── acpi-bits-test.py
> > +├── acpi-bits-test-venv.sh
> > +├── bits-config
> > +│   ├── bits-cfg.txt
> > +│   └── meson.build
> > +├── bits-tests
> > +│   ├── meson.build
> > +│   ├── smbios.py
> > +│   ├── smilatency.py
> > +│   ├── testacpi.py
> > +│   └── testcpuid.py
> > +├── meson.build
> > +├── prebuilt
> > +│   ├── bits-2095-grub.tar.gz
> > +│   ├── bits-2095.zip
> > +│   └── meson.build
> > +├── README
> > +└── requirements.txt
> > +
> > +acpi-bits:
> > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > +   environment necessary for the actual python test script to run. Amongst
> > +   other things, it makes sure that QEMU python library is available within
> > +   that venv so that QEMU machines can be forked. QEMU python library can 
> > be
> > +   found within python/ directory in QEMU source.
> > +   After setting up the virtual environment, it runs the python test script
> > +   from within that environment.
> > +   If you want to enable verbose mode only for bits test and run make 
> > check,
> > +   one trick is to add V=1 before the call to execute the python script in
> > +   this file.
> > + - acpi-bits-test.py: This is the main python 

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> The README file is added describing the directory structure and the purpose
> of every file it contains. It also describes how to add new tests, make 
> changes
> to existing tests or bits config files or regenerate the bits software.
> 
> Signed-off-by: Ani Sinha 
> ---
>  tests/qtest/acpi-bits/README | 168 +++
>  1 file changed, 168 insertions(+)
>  create mode 100644 tests/qtest/acpi-bits/README
> 
> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> new file mode 100644
> index 00..97b15f1665
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/README
> @@ -0,0 +1,168 @@
> +=
> +ACPI/SMBIOS QTESTS USING BIOSBITS
> +=
> +
> +Biosbits is a software written by Josh Triplett that can be downloaded by
> +visiting https://biosbits.org/. The github codebase can be found here:
> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> +the bios components such as acpi and smbios tables directly through acpica
> +bios interpreter (a freely available C based library written by Intel,
> +downloadable from https://acpica.org/ and is included with biosbits) without 
> an
> +operating system getting involved in between.
> +There are several advantages to directly testing the bios in a real physical
> +machine or VM as opposed to indirectly discovering bios issues through the
> +operating system. For one thing, the OSes tend to hide bios problems from the
> +end user. The other is that we have more control of what we wanted to test
> +and how by directly using acpica interpreter on top of the bios on a running
> +system. More details on the inspiration for developing biosbits and its real
> +life uses can be found in (a) and (b).
> +This directory contains QEMU qtests written in python that exercizes the QEMU
> +bios components using biosbits and reports test failures.
> +
> +These tests use python virtual environment. In debian/ubuntu system, the 
> tests
> +would require python3.8-venv and python3-pip packages to be installed.

Why do we mess with venv and pip? Certainly possible but
what's wrong with using distro provided packages?

> +
> +A brief description of the contents of this directory follows:
> +
> +├── acpi-bits-test.py
> +├── acpi-bits-test-venv.sh
> +├── bits-config
> +│   ├── bits-cfg.txt
> +│   └── meson.build
> +├── bits-tests
> +│   ├── meson.build
> +│   ├── smbios.py
> +│   ├── smilatency.py
> +│   ├── testacpi.py
> +│   └── testcpuid.py
> +├── meson.build
> +├── prebuilt
> +│   ├── bits-2095-grub.tar.gz
> +│   ├── bits-2095.zip
> +│   └── meson.build
> +├── README
> +└── requirements.txt
> +
> +acpi-bits:
> + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> +   environment necessary for the actual python test script to run. Amongst
> +   other things, it makes sure that QEMU python library is available within
> +   that venv so that QEMU machines can be forked. QEMU python library can be
> +   found within python/ directory in QEMU source.
> +   After setting up the virtual environment, it runs the python test script
> +   from within that environment.
> +   If you want to enable verbose mode only for bits test and run make check,
> +   one trick is to add V=1 before the call to execute the python script in
> +   this file.
> + - acpi-bits-test.py: This is the main python test script that generates a
> +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and 
> reports
> +   test failures. This is the script one would be interested in if they 
> wanted
> +   to add or change some component of the log parsing, add a new commandline 
> to
> +   how QEMU is spawned etc. Test writers typically would not need to modify
> +   this script unless they wanted to enhance or change the log parsing for
> +   their tests.
> + - requirements.txt: This text file contains the dependent python libraries
> +   needed for the tests to run. If a new dependent library is needed, it 
> would
> +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> +   it when setting up the python virtual environment using pip.
> + - README: This text file.
> +
> +acpi-bits/bits-config:
> +   This location contains biosbits config files that determine how the 
> software
> +   runs the tests.
> + - bits-config.txt: this is the biosbits config file that determines what 
> tests
> +   or actions are performed by bits. The description of the config options 
> are
> +   provided in the file itself.
> +
> +acpi-bits/prebuilt:
> +   This location contains prebuilt biosbits binaries that are used to 
> generate
> +   the bits iso file for testing.
> + - bits-2095.zip: The contents from this zip archive are the main contents of
> +the iso file that are used for testing. This