Re: [Xenomai-core] Re: Test, benchmark
Jan Kiszka wrote: list=a,b,c,d for option in `echo $list|sed -e's/,/\n/g'`; do case $option in a) do_this ;; b) do_that ;; ... esac done What about: save_ifs=$IFS IFS=, for option in $list; do case $option in ... esac done IFS=$save_ifs -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] Re: Test, benchmark
Jan Kiszka wrote:
Hi all,
Jim raised these issues nicely to a generic level. I would like to pick
it up and add some thoughts.
Jim Cromie wrote:
...
FWIW, I noted that xeno-test is not running these:
- switchbench
- switchtest
- irqbench
Im not sure they belong in xeno-test though, since they dont
appear to produce output that shows good vs bad performance,
only an informal 'sanity' check.
Including switchtest depends on if xeno-test should also do some
elementary stability tests. This can be derived from performance tests
as well, but Gilles' switchtest does it for the various switching
constellations more systematically.
Including irqbench is more tricky as real hardware and a second box
are always involved here (so far it only works over null-modem, need to
be extended to some GPIOs or parallel port).
just responding to small part now..
this patch adds switchtest, switchbench (and drops switch) and irqbench.
each test-prog has a corresponding $XENOT_progname
with which you can inject new test arguments individually.
Most of these can be undef'd, except for XENOT_IRQBENCH,
which needs to be set in order for test to run (since the test requires
additional resources, as you noted above)
WRT switchtest, the -T option is useful, and makes its inclusion
possible :-)
xeno-test adds -T 120, which you can override as follows
XENOT_SWITCHTEST='-T 300'
# other useful ones
XENOT_CYCLIC='-v' # make it verbose
Fri Aug 18 07:06:20 MDT 2006
running: ./run -- -n -T 120 # switchtest
*
*
* Type ^C to stop this application.
*
*
[ 1574.162754] Xenomai: starting RTDM services.
cpu 0: 2079 context switches.
cpu 0: 4212 context switches.
cpu 0: 6336 context switches.
cpu 0: 8442 context switches.
...
cpu 0: 246981 context switches.
cpu 0: 249096 context switches.
cpu 0: 250263 context switches.
[ 1698.479703] Xenomai: stopping RTDM services.
wrt the data emitted, what can we learn from the numbers ?
They look to be increasing linearly, with some noise/perturbations.
We could do some statistics, but whats useful ?
Histogramming, averaging the delta-context-switches ?
Also, I see from the help-text that it does many kinds of context switches.
Does it make sense to run each kind for a bunch of samples,
so that we can see # and variation for each kind of switch ?
Other things (for other emails)
1 - one more stats/histogram suggests that it should be in a library
or at least a separate object-file.Any thoughts / prefs / advice ?
Perhaps a 'do it the way I did in X'
2 - I believe Ive tracked down xeno-test's problem cleaning up workloads.
mkload is missing an 'exec', so the collected pid is that of an intermediate
shell, which is either killed, or goes away by itself, leaving the actual
workload reparented to init. Ive got the spawn-cleanup mechanics working
in a separate script that works -
a - it respawns tasks that have finished
forex dd if=/dev/hda1 of=/dev/null
will complete, since hda1 is a finite device ( unlike /dev/zero )
b - it kills the tasks it started before it exits.
c - but needs more testing..
Index: scripts/xeno-test.in
===
--- scripts/xeno-test.in(revision 1453)
+++ scripts/xeno-test.in(working copy)
@@ -1,7 +1,9 @@
#! /bin/sh
-# Adapted to be run also under the BusyBox. If you want to test it under the
BusyBox use
-# busybox sh xeno-test
-# A BusyBox = 1.1.3 with a make defconfig should provide all needed applets.
+
+# Adapted to be run also under the BusyBox.
+# If you want to test it this way, do: sh xeno-test
+# BusyBox = 1.1.3 with a make defconfig should provide all needed applets.
+
myusage() {
cat 1 EOF
xeno-test [options]
@@ -190,16 +192,30 @@
boxstatus
(
cd `dirname $0`/../testsuite/latency
- loudly ./run -- $opts -t0
- loudly ./run -- $opts -t1
- loudly ./run -- $opts -t2
+ loudly ./run -- $XENOT_LATENCY $opts -t0 '# latency'
+ loudly ./run -- $XENOT_LATENCY $opts -t1 '# latency'
+ loudly ./run -- $XENOT_LATENCY $opts -t2 '# latency'
)
-( cd `dirname $0`/../testsuite/switch
- loudly ./run -- '# switch'
+( cd `dirname $0`/../testsuite/switchtest
+ loudly ./run -- -n -T 120 $XENOT_SWITCHTEST '# switchtest'
)
+( cd `dirname $0`/../testsuite/switchbench
+ loudly ./run -- -p 10 -n -l 1000 $XENOT_SWITCHBENCH '# switchbench'
+)
( cd `dirname $0`/../testsuite/cyclic
- loudly ./run -- -p 10 -n -l 1000 '# cyclictest'
+ loudly ./run -- -p 10 -n -l 1000 $XENOT_CYCLIC '# cyclictest'
)
+
+if [ $XENOT_IRQBENCH != ] ; then
+ (
+ cd `dirname $0`/../testsuite/irqbench
+ loudly ./run -- -P 10 $XENOT_IRQBENCH -t0 '# irqbench user'
+ loudly ./run -- -P 10 $XENOT_IRQBENCH -t1 '# irqbench kernel'
+ loudly ./run -- -P 10 $XENOT_IRQBENCH -t2 '# irqbench irq-handler'
+ loudly ./run -- -P 10
Re: [Xenomai-core] Re: Test, benchmark
Gilles Chanteperdrix wrote: A question: I see that you always use the -n option, do you have problems running the test without this option ? When launched with the -n option switchtest does not test cpu context switches. s/cpu/FPU/ -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] Re: Test, benchmark
Jim Cromie wrote: ... just responding to small part now.. this patch adds switchtest, switchbench (and drops switch) and irqbench. each test-prog has a corresponding $XENOT_progname with which you can inject new test arguments individually. Most of these can be undef'd, except for XENOT_IRQBENCH, which needs to be set in order for test to run (since the test requires additional resources, as you noted above) I doesn't necessarily require additional parameters (default is the first serial port on PCs). Why not adding a switch to xeno_test instead? Something like -a additional-tests (e.g. -a irqbench,whateverbench). However, please don't forget to document this extension (man page?). Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Re: Test, benchmark
Gilles Chanteperdrix wrote: Jim Cromie wrote: [ 1574.162754] Xenomai: starting RTDM services. cpu 0: 2079 context switches. cpu 0: 4212 context switches. cpu 0: 6336 context switches. cpu 0: 8442 context switches. ... cpu 0: 246981 context switches. cpu 0: 249096 context switches. cpu 0: 250263 context switches. [ 1698.479703] Xenomai: stopping RTDM services. wrt the data emitted, what can we learn from the numbers ? They look to be increasing linearly, with some noise/perturbations. We could do some statistics, but whats useful ? Histogramming, averaging the delta-context-switches ? Also, I see from the help-text that it does many kinds of context switches. Does it make sense to run each kind for a bunch of samples, so that we can see # and variation for each kind of switch ? No, because one of the threads in the chain of context switches is sleeping, otherwise the program would completely block your box. So, the figures are largely irrelevant, the only important thing about them is that they are increasing, it proves that the test is really switching contexts. But the test fails if the value stop increasing, so no, the output is useless. Note that if you add the -q option, the program will be silent and only print the final count of context switches. A question: I see that you always use the -n option, do you have problems running the test without this option ? When launched with the -n option switchtest does not test cpu context switches. I think I added it at some point when I wasnt getting output. It works without the -n too, which should be added via XENOT_SWITCHTEST, not stuffed in by default. You could just edit it out of patch, if otherwize satisfied... fwiw, Im not averse to expanding to XENOTEST_OPTS_foo, if you think thats better (its probly more self-explanatory when found in an .rc file) (its obviously trivial to redo the patch, lemme know) BTW, running without -n, and with -T 120 (same as before) I get more total context switches: [ 1075.064980] Xenomai: starting RTDM services. Testing FPU check routines... r0: 1 != 2 r1: 1 != 2 r2: 1 != 2 r3: 1 != 2 r4: 1 != 2 r5: 1 != 2 r6: 1 != 2 r7: 1 != 2 FPU check routines OK. cpu 0: 5727 context switches. cpu 0: 11477 context switches. cpu 0: 17227 context switches. cpu 0: 23000 context switches. ... cpu 0: 673164 context switches. cpu 0: 678914 context switches. cpu 0: 684664 context switches. cpu 0: 688620 context switches. [ 708.976879] Xenomai: stopping RTDM services. Offhand, that seems counterintuitive that -n gives lower growth-rate of total switches, since its equivalent to a shorter list of 'threadspec's Also, 2 possible output change requests: a - print per-sample measures, not accumulating ones. this is more consistent with latency, which prints the latencies seen over the 1-sec sample period This also feeds better into histogram, w/o adding 'delta' logic to histogrammer b - label output lines more in spirit of latency RTT| 00:00:01 (in-kernel periodic task, 100 us period, priority 99) RTH|-lat min|-lat avg|-lat max|-overrun|lat best|---lat worst RTD| 4.733| 13.145| 18.518| 0| 4.733| 18.518 RTD| 4.868| 13.169| 24.735| 0| 4.733| 24.735 RTD| 5.014| 13.104| 36.270| 0| 4.733| 36.270 RTD| 4.905| 13.111| 36.394| 0| 4.733| 36.394 I have some misgivings about asking for this : - current output needs massaging, esp b4 feeding to gnuplot Someday, I'll sit down and write a script to reformat the data the way gnuplot wants it, then we'll have some idea whether current form is sub-optimal. - Ideally, we'd be using relayfs to collect data. More library-fodder ?? Lastly, we once had testsuite/README, I suppose it was dropped as being fatally-out-of-date. Presuming we want a new fresh one, could you add an empty file to svn, so that we can patch against it ? thanks jimc ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Re: Test, benchmark
Jim Cromie wrote: ... Also, 2 possible output change requests: a - print per-sample measures, not accumulating ones. this is more consistent with latency, which prints the latencies seen over the 1-sec sample period This also feeds better into histogram, w/o adding 'delta' logic to histogrammer b - label output lines more in spirit of latency My idea on this is to get that part (the formatting and dumping) out of the individual test, into a shared benchmark library. That would consolidate things, simplify the individual tests (irqbench is lacking it therefore), and help to modify things later when requirements for the data processing change/enhance. Would you like to work on this? Of course, you can count on useless to weird comments by us when you'll post any API design or patch. ;) Jan PS: But I'm not sure if that library would apply well to switchtest... signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] Re: Test, benchmark
Jan Kiszka wrote: Jim Cromie wrote: ... just responding to small part now.. this patch adds switchtest, switchbench (and drops switch) and irqbench. each test-prog has a corresponding $XENOT_progname with which you can inject new test arguments individually. Most of these can be undef'd, except for XENOT_IRQBENCH, which needs to be set in order for test to run (since the test requires additional resources, as you noted above) I doesn't necessarily require additional parameters (default is the first serial port on PCs). Why not adding a switch to xeno_test instead? Something like -a additional-tests (e.g. -a irqbench,whateverbench). However, please don't forget to document this extension (man page?). Jan several reasons for my preference: - xeno-test is already cluttered with options, many of which propagate down to tests - as more tests are added, more option-clashing is inevitable. this makes pass-downs more complicated. - prog-specific options isolate us from option clashes / churn - assuming -a prog, need to handle multiples on line (I havent done that with shell getopts) I have to wonder if all shells have uniform getopt behavior - we want to work on bash, sh, ash, dash ... - I get to duck the manpage update ;-) ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Re: Test, benchmark
Jim Cromie wrote: I think I added it at some point when I wasnt getting output. It works without the -n too, which should be added via XENOT_SWITCHTEST, not stuffed in by default. You could just edit it out of patch, if otherwize satisfied... Patch applied without -n, thanks. -- Gilles Chanteperdrix. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] Re: Test, benchmark, demo frameworks
Jim Cromie wrote: Jan Kiszka wrote: ... So, how to proceed? aside wiki++ / 1 file at a time - I suppose.. I'll start by poaching Hannes' Makefile, bundling it into an examples/ dir with my 3-way version of his timer programs. Id like see the target files appear as 0 len files, ( presuming this necessary for svn diff to see code I drop in ) Then we can discuss the more nuanced improvements /* adding tutorial, guidance */ After that, its just step and repeat, tossing crap along the way. Ok, to ease pushing things into folders, I would suggest creating this structure in the root dir of trunk: examples/native/... examples/native/kernel/... examples/posix/... examples/... examples/drivers/can/... examples/drivers/serial/... All stuff self-contained, i.e. one should simply cd to the folder type make, maybe additionally providing some information on the xenomai installation dir / kernel source path, and some binary pops up, ready to run or insmod. I would furthermore suggest to have some kind of hello-xenomai.c in every skin example folder that is intended as the beginners tutorial, providing an elementary framework to start hacking. Reorganising the testsuites is another topic. I currently have no idea how to sort things. We have some stuff under sim/skins/*/testsuite that only builds for the simulator, there is src/testsuite, intended to be distributed and installed on the target, and we may want to have a larger testsuite that can be downloaded separately (my concern is also about driver tests here). Anyone any ideas/comments on this? Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] Re: Test, benchmark, demo frameworks
Jan Kiszka wrote: Hi all, Jim raised these issues nicely to a generic level. I would like to pick it up and add some thoughts. Jim Cromie wrote: ... FWIW, I noted that xeno-test is not running these: - switchbench - switchtest - irqbench Im not sure they belong in xeno-test though, since they dont appear to produce output that shows good vs bad performance, only an informal 'sanity' check. Including switchtest depends on if xeno-test should also do some elementary stability tests. This can be derived from performance tests as well, but Gilles' switchtest does it for the various switching constellations more systematically. elementary stability says make it 1st test. before longer running latency tests. TBD.. Including irqbench is more tricky as real hardware and a second box are always involved here (so far it only works over null-modem, need to be extended to some GPIOs or parallel port). Regarding the output of the various benchmarks I would like to cite myself here: https://mail.gna.org/public/xenomai-core/2006-06/msg00195.html [And as one of the major xeno-test contributors, you may feel included by the term test team. ;)] LOL at 2nd-to-last paragraph. wrt data collection, any updates on LTT or relayfs ? iirc LTT was split to create relayfs and LTT++, but the latter is WIP. With them, data-collection becomes comparatively limitless. also, Niklaus will be happy to hear I feel ownership (ie guilt) about a xeno-test bug where workloads get orphaned by middle 'workload-manager' shell not catching a terminating condition and cleaning up. Im not thrilled about bashing my way thru this jobctl problem, but I'll knuckle down someday (soon?), reduce it to an context-free bash script/apparatus for us to kick tires on busybox, etc.. Then fold into xeno-test and submit And technically, dont regression tests have to yield a PASS / FAIL decision ? ;-) Simple regular output is a good idea whenever the result is simple to express. A fatally crashing switch test due to broken support on arch XYZ will make it hard to issue FAIL... :) True, but that tells us something, doesnt it ? presume a regression test that prints this -- 1..4 ok 1 - Creating test program ok 2 - Test program runs, no error not ok 3 - infinite loop # TODO halting problem unsolved not ok 4 - infinite loop 2 # TODO halting problem unsolved we can know: - prog expects to complete 4 tests, and does so. (no segfault) - fails 2 of them - and which ones. http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap has a nice code sample, which, at its core, is: #include tap.h plan_tests(4); ok(0, Creating test prog); ok(some_function(), Test program runs, no error\n); ... I havent looked, but it looks purely header / macros / static inlines. Theres a full TAP model, but we can use just the basics. http://search.cpan.org/dist/Test-Harness/lib/Test/Harness/TAP.pod#Got_spare_tuits%3F One aspect we might reject is the rule about other print output starting with # . such other output is allowed by test harness, which complains otherwize. Speaking more broadly, there are 3 possible kinds of test-progs - regression tests PASS / FAIL - either by exit(rc), or by printf( %s\n, rc ? not-ok : ok) see perl's regression test suite ( 100k separate tests ) usually test details, are not tutorial Have you checked what is already under sim/skins/*/testsuite? I must confess I don't know if it is easily compilable for non-simulated execution as well. The best thing would be a test framework that builds both for the simulator and for real usage on the target. sheepishly never have tried the sim, beyond 1,2x, punted on some dependency issues.. Obviously thats no longer sufficient :-} - performance tests progs stress xenomai, print performance data should help see performance problems, expose bugs xeno-test aims to collect performance data performance data must be expressive (switchtest is perhaps insufficient here) See my note above. I think some approach with a generic data collection suite + various data generators would be really fantastic! Just takes some brain(s) to design it and some hands to hack it... Taking the vision apart (for inspection), we have : - xeno-test - shell based, semi-primitive, captures logs while running: machine-factors probes/reports, and performance tests workload management (semi-broken) semi-functional data delivery service (environmental challenges) email delivery wont work for me, others w/o local mail set up. - Niklaus' ruby-on-rails ideas (his xeno-test++ code to list was tantalizing, but Ill admit, havent looked since :-( - big issue - server side availability - klive.org python based client server system uses twisted library - very network apps centric. implies server-side sophistication maybe even extensible, perhaps such that we could piggyback on their
