On 14/12/2023 17:29, Richard Purdie wrote:
On Thu, 2023-12-14 at 16:32 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
On 13/12/2023 20:55, Richard Purdie wrote:
On Wed, 2023-12-13 at 20:30 +0000, Eilís 'pidge' Ní Fhlannagáin wrote:
This takes the work rburton did on image screenshot testing and
expands it.
Right now this only works for qemux86-64. Some standardization
of screensize/resolution needs to happen with runqemu params for other
machines. There is an issue in qemux86-64 (and possibly others) where
the screenswitch icon is only half present. This causes the test to
fail.
This test takes a screendump of a qemu image (for now, just
core-image-sato for qemux86-64), and compares it to an image we
have on record. Some normalisation of the different qemu configs
need to happen to be able to support all machines. Example, the
qemuarm64 screen size is much larger than the qemux86-64.
The image we have on record contains a blanked out clock. We do
the same blanking out process for the screenshot, so the images should
have zero differences. If they do, we fail.
In order to enable this test, you will need meta-openembedded/meta-oe in
your bblayers.conf and the following in local.conf:
IMAGE_CLASSES += "testimage"
TEST_SUITES = "login"
IMAGE_INSTALL:append = " python3-qemu-qmp "
TESTIMAGEDEPENDS:append:qemuall = " imagemagick-native:do_populate_sysroot "
Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pi...@baylibre.com>
Co-authored-by: Ross Burton <ross.bur...@arm.com>
Co-authored-by: Eilís 'pidge' Ní Fhlannagáin <pi...@baylibre.com>
---
.../core-image-sato-qemux86-64.png | Bin 0 -> 46986 bytes
meta/lib/oeqa/runtime/cases/login.py | 34 ++++++++++++++++++
2 files changed, 34 insertions(+)
create mode 100644 meta/files/image-tests/core-image-sato-qemux86-64.png
create mode 100644 meta/lib/oeqa/runtime/cases/login.py
diff --git a/meta/lib/oeqa/runtime/cases/login.py
b/meta/lib/oeqa/runtime/cases/login.py
new file mode 100644
index 00000000000..9e351fdeb29
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/login.py
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import subprocess
+from oeqa.runtime.case import OERuntimeTestCase
+import tempfile
+from oeqa.runtime.decorator.package import OEHasPackage
+
+class LoginTest(OERuntimeTestCase):
+
+ @OEHasPackage(['python3-qemu-qmp'])
This only works if we have python3-qemu-qmp in the target? I thought
the monitor was external?
+ def test_screenshot(self):
+ if self.td.get('MACHINE') != "qemux86-64":
+ self.fail
So we're going to throw failures on all other machines? Wouldn't a skip
be more appropriate? Can we provide images for other machines?
I can get images for other machines, but it's going to take a bit to run
everything. I've debug code to grab those images. The other option is to
ensure the other qemu machines have something like QB_GRAPHICS =
"-device virtio-gpu-pci, xres=800,yres=600" to normalise the resolutions
One step at a time I guess but I would like this to ultimately work
with our commonly used machines.
Agreed. I've some patches to the qemu configs that should get us what we
want, but I need to test them.
Have a look at things like:
@skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently")
+
+ if bb.utils.which(os.getenv('PATH'), "convert") is not None and
bb.utils.which(os.getenv('PATH'), "compare") is not None:
This works for nativesdk-imagemagick but it would need convert+compare
in HOSTTOOLS to work for a host binary. I'm torn on the best way to
find that otherwise.
Fixed to rely on the imagemagicks provided by TESTIMAGEDEPENDS:append.
+ with tempfile.NamedTemporaryFile(prefix="oeqa-screenshot-login",
suffix=".png") as t:
+ ret = self.target.runner.run_monitor("screendump", args={"filename": t.name,
"format":"png"})
+ # Use the meta-oe version of convert, along with it's suffix
+ cmd = "convert.im7 {0} -fill white -draw 'rectangle 600,10 640,22'
{1}".format(t.name, t.name)
+ proc = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ output, error = proc.communicate()
You probably want subprocess.check_result()
check_output, but yes, that works here.... but...
I meant check_call() since you don't use the output.
Ah, I have it as check_output. I can submit a patch to move it to that
function with my qemu patches, if what we have is ok.
+
+ # Use the meta-oe version of compare, along with it's suffix
+ cmd = "compare.im7 -metric MSE {0}
{1}/meta/files/image-tests/core-image-sato-{2}.png /dev/null".format(t.name,
self.td.get('COREBASE'), self.td.get('MACHINE'))
+ proc = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ output, error = proc.communicate()
and check_output()
...doesn't here. In this case, we should probably either use run() or
Popen() because imagemagick returns non-failing output on stderr (which
is what we need) as well stdout (which we don't really want). Using
Popen allows us to do this a bit cleaner as opposed to check_output.
You can use stderr=subprocess.STDOUT with check_output?
Yes, but doing so mixes stdout and stderr from my recollection. With
compare (and a few other imagemagick tools) stdout is used for image
output and stderr is (as in this case), the numerical diff of the image,
which I wanted to avoid.
In this case, we don't happen to see the output because we're directing
to /dev/null, but something like:
compare -metric MSE /home/pidge/oeqa-screenshot-loginzm8bokga.png
../meta/files/image-tests/core-image-sato-qemux86-64.png png:-
makes a stdout/stderr mess if we use check_output. I eventually want to
play with piping the output to other things, to get some more info
(identify is an option), for error tracking and such.
-e
Cheers,
Richard
--
Eilís 'pidge' Ní Fhlannagáin
BayLibre - At the Heart of Embedded Linux
www.baylibre.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192400):
https://lists.openembedded.org/g/openembedded-core/message/192400
Mute This Topic: https://lists.openembedded.org/mt/103157570/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
