Hi everybody

After working hard to get my ruby script as clean as possible, I would like to 
submit it to you for additional comments.
The attached ruby script must be placed at scripts/maint/test-xeno-test.rb.
It basically calls xeno-test many times with different options, creating log 
files called scripts/maint/logs/test_xeno_test.log.[0..n] (not attached). A 
short evaluation of the results is saved as scripts/maint/logs/runtest.log 
(attached).

In order to minimize the time need for a run I run the tests always with "-T 
1", except a single test with "-T 5" (to be sure, that -T <x> works).

Running xeno-test in less than 5 seconds was considered an error. I have not 
yet a "expected failure" feature for cases like passing a wrong option "-O" 
to xeno-test.

In my opinion this script correctly show the following deficiencies of the 
actual version of xeno-test:
- options -v and -U do not work correctly
- the load process "dd" are not always killed correctly

The options "-m" and "-M" are not tested, as they had a timeout behaviour 
which resisted my initial attempts to enforce a maximal timeout of 120 
seconds. Maybe I will have sometime a better idea how to deal with this 
error. "xeno-test -M [EMAIL PROTECTED]" succeded only if a manually killed 
either the dd jobs or entered Ctrl-C.

The error flagged as " 2 FAILED: with ''  47 + 5 not smaller <   50 seconds 
with '-T 5'." is a false positive, as this error didn't show up during other 
runs (but I think there was a backup job running).

If anybody has other experiences with xeno-test and thinks that my test 
reports false positives or false negatives, I would appreciate a feedback.
Suggestions/critics for improvement are always welcome.

If now serious errors are found in test-xeno-test.rb, I would appreciate if it 
could be checked into the xenomai trunk. Places other than scripts/maint 
would need a small change to the source code.

Once agreed upon this test-tool I would like to continue like this:
1) Switch to busybox compatible shell syntax. (No regression compared to the 
actual version, but not necessarily all bugs fixed). This patch is almost 
ready, but I would like to check-in the test first.
2) Fix the remaining bugs in xeno-test
3) Port all (or at least the most used) xeno-test feature to the busybox. The 
send mail feature will probably not be tested/fixed by me, as my busybox 
system does not have a mail program. Maybe the same for saving to a log file.
4) Add some improvement like better arguments passing to the individual tests, 
e.g. using OPT_$TOOL environment variables.

Steps 2) and 3) have no particular order.

I do not intend to add test-xeno-test.rb to my buildbot, as I consider this a 
test that should be run manually. But integrating a "xeno-test -m" step in 
some buildbots is on my todo list.

Best regards

-- 
Niklaus Giger

Attachment: test-xeno-test.rb
Description: application/ruby

Fri Jun 16 21:20:09 CEST 2006: started run
Fri Jun 16 21:20:10 CEST 2006:  0 FAILED:   1 seconds. opts <-O> returned too fast
Fri Jun 16 21:20:11 CEST 2006:  0 FAILED: dd not killed 
    >>> niklaus  25929     1 64 21:11 pts/1    00:05:25 dd if /dev/zero of /dev/null
Fri Jun 16 21:20:59 CEST 2006:  1 PASSED:  47 seconds. opts <> returned 0
Fri Jun 16 21:20:59 CEST 2006:  1 FAILED: dd not killed 
    >>> niklaus  26401     1 63 21:20 pts/4    00:00:27 dd if /dev/zero of /dev/null
Fri Jun 16 21:21:49 CEST 2006:  2 PASSED:  50 seconds. opts <-T 5> returned 0
Fri Jun 16 21:21:49 CEST 2006:  2 FAILED: dd not killed 
    >>> niklaus  26991     1 71 21:21 pts/4    00:00:34 dd if /dev/zero of /dev/null
Fri Jun 16 21:22:22 CEST 2006:  3 PASSED:  33 seconds. opts <-p 200> returned 0
Fri Jun 16 21:22:22 CEST 2006:  3 FAILED: dd not killed 
    >>> niklaus  27585     1 73 21:21 pts/4    00:00:23 dd if /dev/zero of /dev/null
Fri Jun 16 21:23:16 CEST 2006:  4 PASSED:  54 seconds. opts <-w 2> returned 0
Fri Jun 16 21:23:16 CEST 2006:  4 FAILED: dd not killed 
    >>> niklaus  28176     1 36 21:22 pts/4    00:00:19 dd if /dev/zero of /dev/null
Fri Jun 16 21:24:55 CEST 2006:  5 PASSED:  99 seconds. opts <-d /dev/hda9> returned 0
Fri Jun 16 21:24:55 CEST 2006:  5 FAILED: dd not killed 
    >>> niklaus  28790     1 13 21:23 pts/4    00:00:13 dd if /dev/hda9 of /dev/null
Fri Jun 16 21:25:18 CEST 2006:  6 PASSED:  23 seconds. opts <-w /bin/dd /dev/hda9> returned 0
Fri Jun 16 21:25:58 CEST 2006:  7 PASSED:  40 seconds. opts <-P 'echo marker'> returned 0
Fri Jun 16 21:25:58 CEST 2006:  7 FAILED: dd not killed 
    >>> niklaus  29993     1 68 21:25 pts/4    00:00:26 dd if /dev/zero of /dev/null
Fri Jun 16 21:26:29 CEST 2006:  8 PASSED:  30 seconds. opts <-L> returned 0
Fri Jun 16 21:27:00 CEST 2006:  9 PASSED:  32 seconds. opts <-N marker> returned 0
Fri Jun 16 21:27:00 CEST 2006: 10 FAILED:   0 seconds. opts <-v> returned too fast
Fri Jun 16 21:27:01 CEST 2006: 11 FAILED:   1 seconds. opts <-U /tmp/xeno-test26256.0> returned too fast
Fri Jun 16 21:27:32 CEST 2006: 12 PASSED:  31 seconds. opts <-D '%Ymarker%Mmarker%D'> returned 0
Fri Jun 16 21:27:33 CEST 2006: 12 FAILED: dd not killed 
    >>> niklaus  31893     1 79 21:27 pts/4    00:00:23 dd if /dev/zero of /dev/null
Fri Jun 16 21:27:33 CEST 2006:  9 PASSED: -N marker: ["/usr/xenomai/bin/marker-060616.212629"]
Fri Jun 16 21:27:33 CEST 2006: 11 PASSED: -U /tmp/xeno-test26256.0: ["/tmp/xeno-test26256.0"]
Fri Jun 16 21:27:33 CEST 2006:  8 PASSED: -L: ["/tmp/test-2.6.14-r-060616.212559"]
Fri Jun 16 21:27:33 CEST 2006:  2 FAILED: with ''  47 + 5 not smaller <   50 seconds with '-T 5'.
_______________________________________________
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core

Reply via email to