On Mon, 1 Jun 2026, Richard Purdie via lists.openembedded.org wrote:
On Mon, 2026-06-01 at 07:06 +0000, Daniel Turull wrote:
-----Original Message-----
From: [email protected]
<[email protected]> On Behalf Of
Richard Purdie via lists.openembedded.org
Sent: Sunday, 31 May 2026 10:50
To: [email protected]; Mark Hatle
<[email protected]>; openembedded-
[email protected]
Subject: Re: [Openembedded-architecture] Stable version upgrades on
OE
stable branches
Also remember that most ptests are functional tests for specific
things. What OE needs is integration tests, which are quite
different.
I'd therefore argue that ptests aren't as helpful as people think
as
functional tests can pass but the overall integration can be
totally
broken. If I had to prioritise, I'd prefer more integration tests
rather than ptests, we have very few of them and they are much more
important IMO. ptests do have a place but I think people miss this
difference.
Richard, could you good examples of integration tests that can be use
as a reference to improve the situation? By integration test, do you
mean the oe-core selftests?
I can point our team to start looking at them
Let me first give a couple of illustrations.
The biggest win we ever had was the image boot test. If you can confirm
the image boots, that is a huge improvement and the act of booting
tests a large and important part of the system.
Another key example was the on target compiler being able to compile
and run a "Hello World" binary. That confirms the compiler, linker,
headers and so on all are correct and working together. You can see the
gcc ones here:
https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/runtime/cases/gcc.py
ptest framework was intended for a slightly different use case than
local qemu Device-Under-Test, where you have good control of the
underlying network between test executor, and DUT.
When you have a physical board farm, local or decentralized, you would
want to remove the network(as much as you can) from the DUT dependencies.
Historical data at the time, showed that it was a large culprit in
test flakeyness due to it changing test timing and intermittent network
timeouts due to then beeing dependent on other test suites or by other
teams sharing the lab network. By moving the test executor to the target
machine, you avoid some of intermittent network issues.
That said, a combination of approaches might be a good thing where it is
best suitable. ptests are certainly not a candidate for boot tests f.ex.
But please consider the board farm scenario when choosing.
They're very simple but effective. I appreciate many won't ship such a
thing in their production images, but for OE-Core, it is a key data
point that keeps our toolchains functional.
The key question then becomes, what other simple things can we run/do
in the target image that would show the system is functional and all
working together correctly?
One idea is to check binaries can be executed successfully (e.g. even
just show their help output). That means they're at least valid
binaries with their libraries present.
Another thought would be to import python modules and check there
aren't import errors.
There are probably many other things we could do with a bit of thought,
the aim being, to see the pieces on the target are functional and all
work together. We'd ideally focus on things with larger dependency
chains, where the test tests the dependencies as well as the item under
test.
Does that help describe the kinds of things I'm thinking of?
There is obviously overlap with some areas of ptests and we don't want
to duplicate that but I believe there is a lot of scope to improve in
this area.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#2373):
https://lists.openembedded.org/g/openembedded-architecture/message/2373
Mute This Topic: https://lists.openembedded.org/mt/119437109/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
