Jan Kiszka wrote:
Hi all,

Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.

Jim Cromie wrote:
FWIW, I noted that xeno-test is not running these:
- switchbench
- switchtest
- irqbench

Im not sure they belong in xeno-test though, since they dont
appear to produce output that shows good vs bad performance,
only an informal 'sanity' check.

Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.

elementary stability says make it 1st test. before longer running latency tests. TBD..

Including irqbench is more tricky as "real" hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).

Regarding the output of the various benchmarks I would like to cite
myself here:


[And as one of the major xeno-test contributors, you may feel included
by the term "test team". ;)]

LOL at 2nd-to-last paragraph.

wrt data collection, any updates on LTT or relayfs ?
iirc LTT was split to create relayfs and LTT++, but the latter is WIP.
With them, data-collection becomes comparatively limitless.

also, Niklaus will be happy to hear I feel ownership (ie guilt) about a xeno-test bug
where workloads get orphaned by middle 'workload-manager' shell not catching
a terminating condition and cleaning up.  Im not thrilled about bashing my
way thru this jobctl problem, but I'll knuckle down someday (soon?), reduce it to an context-free bash script/apparatus for us to kick tires on busybox, etc..
Then fold into xeno-test and submit

And technically, dont regression tests have to yield
a PASS / FAIL decision ?  ;-)

Simple regular output is a good idea whenever the result is simple to
express. A fatally crashing switch test due to broken support on arch
XYZ will make it hard to issue "FAIL"... :)

True, but that tells us something, doesnt it ?

presume a regression test that prints this --

   ok 1 - Creating test program
   ok 2 - Test program runs, no error
   not ok 3 - infinite loop # TODO halting problem unsolved
   not ok 4 - infinite loop 2 # TODO halting problem unsolved

we can know:
- prog expects to complete 4 tests, and does so. (no segfault)
- fails 2 of them - and which ones.

has a nice code sample, which, at its core, is:
#include "tap.h"
ok(0, "Creating test prog");
ok(some_function(), "Test program runs, no error\n");

I havent looked, but it looks purely header / macros / static inlines.
Theres a full TAP model, but we can use just the basics.


One aspect we might reject is the rule about other print output starting with "# ".
such other output is allowed by test harness, which complains otherwize.

Speaking more broadly, there are 3 possible kinds of test-progs

- regression tests
   PASS / FAIL - either by exit(rc),
      or by printf( "%s\n", rc ? "not-ok" : "ok")
      see perl's regression test suite ( 100k separate tests )
      usually test details, are not tutorial

Have you checked what is already under sim/skins/*/testsuite? I must
confess I don't know if it is easily compilable for non-simulated
execution as well. The best thing would be a test framework that builds
both for the simulator and for "real" usage on the target.

<sheepishly> never have tried the sim, beyond 1,2x, punted on some dependency issues..
Obviously thats no longer sufficient :-}
- performance tests
   progs stress xenomai, print performance data
   should help see performance problems, expose bugs
   xeno-test aims to collect performance data
   performance data must be expressive
      (switchtest is perhaps insufficient here)

See my note above. I think some approach with a generic data collection
suite + various data generators would be really fantastic! Just takes
some brain(s) to design it and some hands to hack it...

Taking the vision apart (for inspection), we have :

- xeno-test - shell based, semi-primitive,
    captures logs while running:
    machine-factors probes/reports, and performance tests
   workload management (semi-broken)
   semi-functional data delivery service (environmental challenges)
      email delivery wont work for me, others w/o local mail set up.

- Niklaus' ruby-on-rails ideas (his xeno-test++ code to list was tantalizing, but Ill admit, havent looked since :-(

- big issue - server side availability

- klive.org
   python based client & server system
   uses twisted library - very network apps centric.
   implies server-side sophistication
maybe even extensible, perhaps such that we could piggyback on their service (pipe dream ?)

- analysing the data (phase 2)
   big brains - think here !
   gnuplot has capabilties, but ideosyncracys too.
also is flat-file centric, and *needs* massaging scripts. tedious, lots of moving parts.
   'R' is an analytical system
      big, powerful, complex, -ENOTIME
   punt on this:
      hacker exploration project(s)
if we can just deliver good data regularly from a bunch of machines, phase 1 is done.

- examples / tutorials
   ex: satch.c    simple, clear progs (low feature clutter, etc)
   Id like to see all demo/**/ progs in single dir
      forex satch-native, satch-vxworks, etc ..
      makes for easier browsing
   simple makefile
      builds out-of-tree
      handles kernel-modules and user-progs
      (Ive seen some clean ones, cant find now.  Mine are crufty:-(
   'patterns' of usage
      IWBGreat if we had common usage patterns isolated,
      named, and described

Basically full ack! Wolfgang and I had this discussion recently in a
private thread on how to deal with demos for the new CAN stack. Xenomai
really needs more of this, and it needs it in a structured, regularly
organised way. Some of my thoughts on a demo repository:

 o Should be located in the same SVN repos, maybe under /demo, in order
   to keep it in sync with ongoing API enhancements/modifications.
   But the distribution should be separate from the release (just like
   the simulator).

im just happy to pull it from SVN.
a separate cd demos ; make is fine.

 o Must be well commented, i.e. should not only show how things can be
   done, but also why (or why not). That matches very well with your

 o Should include both simple beginners examples as well as a few more
   complex demos.
o Full ack to your Makefile requirement: stand-alone kernel (Hannes has
   a nice generic one for both 2.4 and 2.6) and user-space makefiles. No
   autotool stuff here. At the same time, we need a script to build and
   maybe even run them automatically to detect breakages.

This will certainly be a longer process and will take some helping
hands, but we first of all need to agree on the basic structure and
requirements, and then give it an (even small) start.

Towards this last item, Ive done 2 things:

- poached code from Hannes Mayer :-)
   task-timers.c does periodic timer 3 ways:
      sleeper, waiter, alarm.

- scrounged old rtai/fusion code (ls -l says Jul 05 ;-)
   cleaned up, 1/2 compile now
   maybe theres examples-tutorials-patterns fodder in here.

attached tarball has these in 2 top-level dirs.
Id like to see if theres a place for them long-term, and clean
them up so theyre correct and helpful.

They can serve as a source, but may need to be aligned to a fairly
strict quality level we should apply on all examples. You know, the
clearer the code is, the better the user can adopt it, the less
questions pop up on xenomai-help - and the more time we have to hack
nice new features instead. =8)

So, how to proceed?
<aside> wiki++ </>

1 file at a time - I suppose..

I'll start by poaching Hannes' Makefile, bundling it into an examples/ dir with my 3-way version of his timer programs. Id like see the target files appear as 0 len files,
( presuming this necessary for svn diff to see code I drop in )

Then we can discuss the more nuanced improvements /* adding tutorial, guidance */

After that, its just step and repeat, tossing crap along the way.

Of course, if anyone wants to pick up bits of the tarball, feel welcome :-D.
I stopped fixing compile errors where they 1st became unobvious.



Xenomai-core mailing list

Reply via email to