On Thu, 2023-12-14 at 16:46 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
> This takes the work rburton did on image screenshot testing and
> expands it.
>
> Right now this only works for qemux86-64. Some standardization
> of screensize/resolution needs to happen with runqemu params for other
> machines.
I suspect that we don't want to make everything the same so we may need
to allow the code to adapt to the sizes we have, I think there are only
a small number of different ones. The annoying part may be just working
out the clock mask sizes for the different ones since we can easily
have different machine specific references.
> There is an issue in qemux86-64 (and possibly others) where
> the screenswitch/home icon is only half present. This causes the test to
> fail. I'm unsure if this is a QMP/qemu issue but I've certainly seen
> the half-rendered icon, so it's not Imagemagick.
These are the kinds of issues we want to start catching and why this is
important!
> This test takes a screendump of a qemu image (for now, just
> core-image-sato for qemux86-64), and compares it to an image we
> have on record. Some normalisation of the different qemu configs
> need to happen to be able to support all machines. Example, the
> qemuarm64 screen size is much larger than the qemux86-64.
>
> The image we have on record contains a blanked out clock. We do
> the same blanking out process for the screenshot, so the images should
> have zero differences. If they do, we fail.
>
> In order to enable this test, you will need meta-openembedded/meta-oe in
> your bblayers.conf and the following in local.conf:
>
> IMAGE_CLASSES += "testimage"
> TEST_SUITES = "login"
> TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
or HOSTTOOLS += "convert compare"
?
> diff --git a/meta/lib/oeqa/runtime/cases/login.py
> b/meta/lib/oeqa/runtime/cases/login.py
I just noticed the name. Does "login" make sense for a desktop
screenshot? I can kind of see the logic but I'd not be looking in that
file for this :/.
> new file mode 100644
> index 00000000000..30074f71b66
> --- /dev/null
> +++ b/meta/lib/oeqa/runtime/cases/login.py
> @@ -0,0 +1,42 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +import subprocess
> +from oeqa.runtime.case import OERuntimeTestCase
> +import tempfile
> +from oeqa.runtime.decorator.package import OEHasPackage
> +import shutil
> +
> +class LoginTest(OERuntimeTestCase):
> +
> + def test_screenshot(self):
> + # Leaving this limited to qemux86-64, until I can get the same
> resolution
> + # on all the qemu machines.
> + if self.td.get('MACHINE') != "qemux86-64":
> + self.skipTest("")
You mean
self.skipTest("Test only compatible with qemux86-64")
right? ;-)
> +
> + if bb.utils.which(os.getenv('PATH'), "convert.im7") is not None and
> bb.utils.which(os.getenv('PATH'), "compare.im7") is not None:
> + with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login",
> suffix=".png") as t:
> + ret = self.target.runner.run_monitor("screendump",
> args={"filename": t.name, "format":"png"})
> + # Use the meta-oe version of convert, along with it's suffix
> + cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10
> 640,22' {1}".format(t.name, t.name)
> + subprocess.check_output(cmd.split(), shell=True,
> stderr=subprocess.PIPE)
> +
> + # Uncomment for debugging/capturing other machine images.
> + # shutil.copy(t.name,
> os.getenv('HOME')+"/"+self.td.get('MACHINE')+".png")
> +
> + # Use the meta-oe version of compare, along with it's suffix
>
> + cmd = "compare.im7 -metric MSE {0}
> {1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name,
> self.td.get('COREBASE'), self.td.get('MACHINE'))
> + # We use Popen here because compare outputs the values we
> care about on stderr.
> + proc = subprocess.Popen(cmd, shell=True,
> stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> + output, error = proc.communicate()
> +
> + # In theory this should be 0, however I've seen some odd
> things with the Home icon/screen switcher.
> + # We should as some point, log the diff image for debugging,
> but for now, just report that it's different.
> + diff=float(error.decode('utf-8').replace("(",
> "").replace(")","").split()[1])
> + self.assertEqual(0, diff, "Screenshot diff is %s." %
> (str(diff)))
> + else:
> + self.skipTest("Could not find imagemagick tools.")
Upon failure do we want to copy to some randomly named file in /tmp/ so
we can at least have a chance to see what went wrong (printing the name
in the test failure message)? I think currently the image would be lost
making debugging hard?
The selftests use OEQA_DEBUGGING_SAVED_OUTPUT but I'm not sure that is
available in the runtime test environment at present unfortunately.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192410):
https://lists.openembedded.org/g/openembedded-core/message/192410
Mute This Topic: https://lists.openembedded.org/mt/103173636/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
