Jan Kiszka wrote:
Hi all,
Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.
Jim Cromie wrote:
...
FWIW, I noted that xeno-test is not running these:
- switchbench
- switchtest
- irqbench
Im not sure they belong in xeno-test though, since they dont
appear to produce output that shows good vs bad performance,
only an informal 'sanity' check.
Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.
elementary stability says make it 1st test. before longer running
latency tests. TBD..
Including irqbench is more tricky as real hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).
Regarding the output of the various benchmarks I would like to cite
myself here:
https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html
[And as one of the major xeno-test contributors, you may feel included
by the term test team. ;)]
LOL at 2nd-to-last paragraph.
wrt data collection, any updates on LTT or relayfs ?
iirc LTT was split to create relayfs and LTT++, but the latter is WIP.
With them, data-collection becomes comparatively limitless.
also, Niklaus will be happy to hear I feel ownership (ie guilt) about a
xeno-test bug
where workloads get orphaned by middle 'workload-manager' shell not catching
a terminating condition and cleaning up. Im not thrilled about bashing my
way thru this jobctl problem, but I'll knuckle down someday (soon?),
reduce it
to an context-free bash script/apparatus for us to kick tires on
busybox, etc..
Then fold into xeno-test and submit
And technically, dont regression tests have to yield
a PASS / FAIL decision ? ;-)
Simple regular output is a good idea whenever the result is simple to
express. A fatally crashing switch test due to broken support on arch
XYZ will make it hard to issue FAIL... :)
True, but that tells us something, doesnt it ?
presume a regression test that prints this --
1..4
ok 1 - Creating test program
ok 2 - Test program runs, no error
not ok 3 - infinite loop # TODO halting problem unsolved
not ok 4 - infinite loop 2 # TODO halting problem unsolved
we can know:
- prog expects to complete 4 tests, and does so. (no segfault)
- fails 2 of them - and which ones.
http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap
has a nice code sample, which, at its core, is:
#include tap.h
plan_tests(4);
ok(0, Creating test prog);
ok(some_function(), Test program runs, no error\n);
...
I havent looked, but it looks purely header / macros / static inlines.
Theres a full TAP model, but we can use just the basics.
http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod#Got_spare_tuits%3F
One aspect we might reject is the rule about other print output starting
with # .
such other output is allowed by test harness, which complains otherwize.
Speaking more broadly, there are 3 possible kinds of test-progs
- regression tests
PASS / FAIL - either by exit(rc),
or by printf( %s\n, rc ? not-ok : ok)
see perl's regression test suite ( 100k separate tests )
usually test details, are not tutorial
Have you checked what is already under sim/skins/*/testsuite? I must
confess I don't know if it is easily compilable for non-simulated
execution as well. The best thing would be a test framework that builds
both for the simulator and for real usage on the target.
sheepishly never have tried the sim, beyond 1,2x, punted on some
dependency issues..
Obviously thats no longer sufficient :-}
- performance tests
progs stress xenomai, print performance data
should help see performance problems, expose bugs
xeno-test aims to collect performance data
performance data must be expressive
(switchtest is perhaps insufficient here)
See my note above. I think some approach with a generic data collection
suite + various data generators would be really fantastic! Just takes
some brain(s) to design it and some hands to hack it...
Taking the vision apart (for inspection), we have :
- xeno-test - shell based, semi-primitive,
captures logs while running:
machine-factors probes/reports, and performance tests
workload management (semi-broken)
semi-functional data delivery service (environmental challenges)
email delivery wont work for me, others w/o local mail set up.
- Niklaus' ruby-on-rails ideas
(his xeno-test++ code to list was tantalizing, but Ill admit, havent
looked since :-(
- big issue - server side availability
- klive.org
python based client server system
uses twisted library - very network apps centric.
implies server-side sophistication
maybe even extensible, perhaps such that we could piggyback on their
