* The next tier could be an automated test suite that runs in a
simulator.
This would be accessible to the greatest numbers of developers and
contributors, because there's nothing to buy.
* The highest tier could be testing on a reference hardware board.
I think we need to assure that these two "tiers" are compatible and use
the same test infrastructure. The should have these things in common:
1. Same top-level test infrastructure,
2. Common assessment of test results from the target console output.
Going from the first to the second is essentially replacing the
software-to-software interfaces:
3. Adding a reference board
4. Adding test harness hardware that is controllable by software to
load code, reset the board, collect serial console data.
5. Other hardware plumbing as need for the test.
For example, network testing on the Linux simulator would require use of
the Linux TAP device and some physical network, and test servers/clients
to close the loop on the network tests. The hardware-in-loop case would
connect directly to the physical network, otherwise it would be the same
test.
PX4 has a rack of supported hardware and runs flight qualification tests
on hardware periodically. The scope of testing an OS is not easily
constrained to a series of test like that, but the basic PX4 setup would
be a good general model.
To keep the scope under control, I would avoid low-level testing of all
OS interfaces. Instead I would focus on higher level usage. networking
is a good example. ELF module support is another. USB devices, USB
host, etc.
There is an OS interface test in apps/testing/ostest. Currently, it
does not do an exhaustive test of interfaces. It only verifies basic
functionality of interfaces and should be extended to cover all
interfaces, all interface features, and handling of bad values. The
simulator is perfectly suited for that kind of testing. I would reserve
hardware-based testing for things that are higher level integrations of
capabilities and that are sensitive to race conditions. Hardware-based
testing would be better for stress and duration testing.
