On Thu, 2023-12-14 at 16:32 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
> On 13/12/2023 20:55, Richard Purdie wrote:
> > On Wed, 2023-12-13 at 20:30 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
> > > This takes the work rburton did on image screenshot testing and
> > > expands it.
> > >
> > > Right now this only works for qemux86-64. Some standardization
> > > of screensize/resolution needs to happen with runqemu params for other
> > > machines. There is an issue in qemux86-64 (and possibly others) where
> > > the screenswitch icon is only half present. This causes the test to
> > > fail.
> > >
> > > This test takes a screendump of a qemu image (for now, just
> > > core-image-sato for qemux86-64), and compares it to an image we
> > > have on record. Some normalisation of the different qemu configs
> > > need to happen to be able to support all machines. Example, the
> > > qemuarm64 screen size is much larger than the qemux86-64.
> > >
> > > The image we have on record contains a blanked out clock. We do
> > > the same blanking out process for the screenshot, so the images should
> > > have zero differences. If they do, we fail.
> > >
> > > In order to enable this test, you will need meta-openembedded/meta-oe in
> > > your bblayers.conf and the following in local.conf:
> > >
> > > IMAGE_CLASSES += "testimage"
> > > TEST_SUITES = "login"
> > > IMAGE_INSTALL:append = " python3-qemu-qmp "
> > > TESTIMAGEDEPENDS:append:qemuall = "
> > > imagemagick-native:do_populate_sysroot "
> > >
> > > Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <[email protected]>
> > > Co-authored-by: Ross Burton <[email protected]>
> > > Co-authored-by: Eilís 'pidge' Ní Fhlannagáin <[email protected]>
> > > ---
> > > .../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes
> > > meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++
> > > 2 files changed, 34 insertions(+)
> > > create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
> > > create mode 100644 meta/lib/oeqa/runtime/cases/login.py
> > >
> > > diff --git a/meta/lib/oeqa/runtime/cases/login.py
> > > b/meta/lib/oeqa/runtime/cases/login.py
> > > new file mode 100644
> > > index 00000000000..9e351fdeb29
> > > --- /dev/null
> > > +++ b/meta/lib/oeqa/runtime/cases/login.py
> > > @@ -0,0 +1,34 @@
> > > +#
> > > +# Copyright OpenEmbedded Contributors
> > > +#
> > > +# SPDX-License-Identifier: MIT
> > > +#
> > > +
> > > +import subprocess
> > > +from oeqa.runtime.case import OERuntimeTestCase
> > > +import tempfile
> > > +from oeqa.runtime.decorator.package import OEHasPackage
> > > +
> > > +class LoginTest(OERuntimeTestCase):
> > > +
> > > + @OEHasPackage(['python3-qemu-qmp'])
> >
> > This only works if we have python3-qemu-qmp in the target? I thought
> > the monitor was external?
> >
> > > + def test_screenshot(self):
> > > + if self.td.get('MACHINE') != "qemux86-64":
> > > + self.fail
> >
> > So we're going to throw failures on all other machines? Wouldn't a skip
> > be more appropriate? Can we provide images for other machines?
>
> I can get images for other machines, but it's going to take a bit to run
> everything. I've debug code to grab those images. The other option is to
> ensure the other qemu machines have something like QB_GRAPHICS =
> "-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions
One step at a time I guess but I would like this to ultimately work
with our commonly used machines.
> >
> > Have a look at things like:
> > @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific
> > currently")
> >
> > > +
> > > + if bb.utils.which(os.getenv('PATH'), "convert") is not None and
> > > bb.utils.which(os.getenv('PATH'), "compare") is not None:
> >
> > This works for nativesdk-imagemagick but it would need convert+compare
> > in HOSTTOOLS to work for a host binary. I'm torn on the best way to
> > find that otherwise.
>
> Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
>
> >
> > > + with
> > > tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login",
> > > suffix=".png") as t:
> > > + ret = self.target.runner.run_monitor("screendump",
> > > args={"filename": t.name, "format":"png"})
> > > + # Use the meta-oe version of convert, along with it's
> > > suffix
> > > + cmd = "convert.im7 {0} -fill white -draw 'rectangle
> > > 600,10 640,22' {1}".format(t.name, t.name)
> > > + proc = subprocess.Popen(cmd, shell=True,
> > > stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> > > + output, error = proc.communicate()
> >
> > You probably want subprocess.check_result()
> >
>
> check_output, but yes, that works here.... but...
I meant check_call() since you don't use the output.
> > > +
> > > + # Use the meta-oe version of compare, along with it's
> > > suffix
> > > + cmd = "compare.im7 -metric MSE {0}
> > > {1}/meta/files/image-tests/core-image-sato-{2}.png
> > > /dev/null".format(t.name, self.td.get('COREBASE'), self.td.get('MACHINE'))
> > > + proc = subprocess.Popen(cmd, shell=True,
> > > stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> > > + output, error = proc.communicate()
> >
> > and check_output()
>
> ...doesn't here. In this case, we should probably either use run() or
> Popen() because imagemagick returns non-failing output on stderr (which
> is what we need) as well stdout (which we don't really want). Using
> Popen allows us to do this a bit cleaner as opposed to check_output.
You can use stderr=subprocess.STDOUT with check_output?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192397):
https://lists.openembedded.org/g/openembedded-core/message/192397
Mute This Topic: https://lists.openembedded.org/mt/103157570/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
