Jan Kiszka wrote:
Hi all,
Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.
Jim Cromie wrote:
...
FWIW, I noted that xeno-test is not running these:
- switchbench
- switchtest
- irqbench
Im not sure they belong in xeno-test though, since they dont
appear to produce output that shows good vs bad performance,
only an informal 'sanity' check.
Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.
elementary stability says make it 1st test. before longer running
latency tests. TBD..
Including irqbench is more tricky as "real" hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).
Regarding the output of the various benchmarks I would like to cite
myself here:
https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html
[And as one of the major xeno-test contributors, you may feel included
by the term "test team". ;)]
LOL at 2nd-to-last paragraph.
wrt data collection, any updates on LTT or relayfs ?
iirc LTT was split to create relayfs and LTT++, but the latter is WIP.
With them, data-collection becomes comparatively limitless.
also, Niklaus will be happy to hear I feel ownership (ie guilt) about a
xeno-test bug
where workloads get orphaned by middle 'workload-manager' shell not catching
a terminating condition and cleaning up. Im not thrilled about bashing my
way thru this jobctl problem, but I'll knuckle down someday (soon?),
reduce it
to an context-free bash script/apparatus for us to kick tires on
busybox, etc..
Then fold into xeno-test and submit
And technically, dont regression tests have to yield
a PASS / FAIL decision ? ;-)
Simple regular output is a good idea whenever the result is simple to
express. A fatally crashing switch test due to broken support on arch
XYZ will make it hard to issue "FAIL"... :)
True, but that tells us something, doesnt it ?
presume a regression test that prints this --
1..4
ok 1 - Creating test program
ok 2 - Test program runs, no error
not ok 3 - infinite loop # TODO halting problem unsolved
not ok 4 - infinite loop 2 # TODO halting problem unsolved
we can know:
- prog expects to complete 4 tests, and does so. (no segfault)
- fails 2 of them - and which ones.
http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap
has a nice code sample, which, at its core, is:
#include "tap.h"
plan_tests(4);
ok(0, "Creating test prog");
ok(some_function(), "Test program runs, no error\n");
...
I havent looked, but it looks purely header / macros / static inlines.
Theres a full TAP model, but we can use just the basics.
http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod#Got_spare_tuits%3F
One aspect we might reject is the rule about other print output starting
with "# ".
such other output is allowed by test harness, which complains otherwize.
Speaking more broadly, there are 3 possible kinds of test-progs
- regression tests
PASS / FAIL - either by exit(rc),
or by printf( "%s\n", rc ? "not-ok" : "ok")
see perl's regression test suite ( 100k separate tests )
usually test details, are not tutorial
Have you checked what is already under sim/skins/*/testsuite? I must
confess I don't know if it is easily compilable for non-simulated
execution as well. The best thing would be a test framework that builds
both for the simulator and for "real" usage on the target.
<sheepishly> never have tried the sim, beyond 1,2x, punted on some
dependency issues..
Obviously thats no longer sufficient :-}
- performance tests
progs stress xenomai, print performance data
should help see performance problems, expose bugs
xeno-test aims to collect performance data
performance data must be expressive
(switchtest is perhaps insufficient here)
See my note above. I think some approach with a generic data collection
suite + various data generators would be really fantastic! Just takes
some brain(s) to design it and some hands to hack it...
Taking the vision apart (for inspection), we have :
- xeno-test - shell based, semi-primitive,
captures logs while running:
machine-factors probes/reports, and performance tests
workload management (semi-broken)
semi-functional data delivery service (environmental challenges)
email delivery wont work for me, others w/o local mail set up.
- Niklaus' ruby-on-rails ideas
(his xeno-test++ code to list was tantalizing, but Ill admit, havent
looked since :-(
- big issue - server side availability
- klive.org
python based client & server system
uses twisted library - very network apps centric.
implies server-side sophistication
maybe even extensible, perhaps such that we could piggyback on their
service (pipe dream ?)
- analysing the data (phase 2)
big brains - think here !
gnuplot has capabilties, but ideosyncracys too.
also is flat-file centric, and *needs* massaging scripts.
tedious, lots of moving parts.
'R' is an analytical system
big, powerful, complex, -ENOTIME
punt on this:
hacker exploration project(s)
if we can just deliver good data regularly from a bunch of
machines, phase 1 is done.
- examples / tutorials
ex: satch.c simple, clear progs (low feature clutter, etc)
Id like to see all demo/**/ progs in single dir
forex satch-native, satch-vxworks, etc ..
makes for easier browsing
simple makefile
builds out-of-tree
handles kernel-modules and user-progs
(Ive seen some clean ones, cant find now. Mine are crufty:-(
'patterns' of usage
IWBGreat if we had common usage patterns isolated,
named, and described
Basically full ack! Wolfgang and I had this discussion recently in a
private thread on how to deal with demos for the new CAN stack. Xenomai
really needs more of this, and it needs it in a structured, regularly
organised way. Some of my thoughts on a demo repository:
o Should be located in the same SVN repos, maybe under /demo, in order
to keep it in sync with ongoing API enhancements/modifications.
But the distribution should be separate from the release (just like
the simulator).
im just happy to pull it from SVN.
a separate cd demos ; make is fine.
o Must be well commented, i.e. should not only show how things can be
done, but also why (or why not). That matches very well with your
patterns.
o Should include both simple beginners examples as well as a few more
complex demos.
o Full ack to your Makefile requirement: stand-alone kernel (Hannes has
a nice generic one for both 2.4 and 2.6) and user-space makefiles. No
autotool stuff here. At the same time, we need a script to build and
maybe even run them automatically to detect breakages.
This will certainly be a longer process and will take some helping
hands, but we first of all need to agree on the basic structure and
requirements, and then give it an (even small) start.
Towards this last item, Ive done 2 things:
- poached code from Hannes Mayer :-)
http://www.captain.at/xenomai.php
task-timers.c does periodic timer 3 ways:
sleeper, waiter, alarm.
- scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
cleaned up, 1/2 compile now
maybe theres examples-tutorials-patterns fodder in here.
attached tarball has these in 2 top-level dirs.
Id like to see if theres a place for them long-term, and clean
them up so theyre correct and helpful.
They can serve as a source, but may need to be aligned to a fairly
strict quality level we should apply on all examples. You know, the
clearer the code is, the better the user can adopt it, the less
questions pop up on xenomai-help - and the more time we have to hack
nice new features instead. =8)
So, how to proceed?
<aside> wiki++ </>
1 file at a time - I suppose..
I'll start by poaching Hannes' Makefile, bundling it into an examples/
dir with
my 3-way version of his timer programs. Id like see the target files
appear as 0 len files,
( presuming this necessary for svn diff to see code I drop in )
Then we can discuss the more nuanced improvements /* adding tutorial,
guidance */
After that, its just step and repeat, tossing crap along the way.
Of course, if anyone wants to pick up bits of the tarball, feel welcome :-D.
I stopped fixing compile errors where they 1st became unobvious.
Jan
-jimc
_______________________________________________
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core
