Philippe Gerum wrote:



This is a partial roadmap for the project, composed of the currently

Ah! I just _knew_ you would jump in as expected. The teasing worked :o)

well done ! Its the mark of a great leader to get folks to do what he wants,
while making them think its their idea ;-)

(and I imagine thats why you ccd Takis too :-)

[lots of snippage, thruout]


LiveCD has a few weaknesses though:

- cant test platforms w/o cdrom


I also think that's a serious issue. Aside of the hw availability problem (e.g. non-x86 eval boards), having to burn the CD is one step too many when time is a scarce resource. It often prevents to run it as a fast check procedure even in the absence of any noticeable problem. IOW, you won't burn a CD to run the tests unless you are really stuck with some issue. So a significant part of the interest of having a generic testsuite is lost: you just don't discover potential problems before the serious breakage is already in the wild.

One thing that would help expand LiveCD's usefullness is to be able to :

- mount pirt.iso in loopback on a host (my laptop),
- export it via NFS to box-under-test,
- use pxelinux to feed LiveCD's kernel(s?)  to box when it boots.

I tried to do this, and IIRC ran into trouble with absolute symlinks
from /etc.ro to /etc.   The absoluteness fouls things when the ISO
is mounted on forex: /media/cd.

I poked a bit at trying to convince NFS to resolve them as if they
were used within a chroot jail, but I dont know enough about that.



- manual re-entry of data is tedious,
- no collection of platform data (available for automation)
- spotty info about cpu, memory, mobo, etc

which is largely user-supplied, so it can be wrong.

- no unattended test (still true?)


- unfiltered preposterous data. Sometimes, data sent are just rubbish because of well-known hw-related dysfunctioning or misuse of the LiveCD. This perturbates the results uselessly.

Any ideas on how to reject these outliers ?
(defer til we have statistical analysis in place ?)

- difficulties so far to really get a sensible digested information out of the zillions of results, aside of very general figures (e.g. best performer). But this is more an issue of lack of data post-processors than of the LiveCD infrastructure itself.

yep.  And we *need* platform data to start to categorize them by platform,
important config choices, etc. We should see narrower ranges of results,
and be more able to reject the junk.

<snip>

Additionally, LiveCD is a really great tool when it comes to help people figuring out whether their respective box or brain have a problem with the tested software, i.e. by automatically providing a sane software (kernel+rtos) configuration and the proper way to run it quite easily, a number of people could determine if their current lack of luck comes from their software configuration, or rather from a more serious problem.

yeah.  pre-built world saves a lot of early thrashing.



- testsuite/cruncher ?


The cruncher measures the impact of using the interrupt shield, but this setting is now configured out by default since a majority of people don't currently need it. Shield cost/performances are still useful to know though.

OK. adding 1 call to cruncher is simple. Over time we *may* collect enough data to make some A (shields up!) vs B (shields down!) comparisons. But I dont see the data to distinguish A, B - dont we need the xeno/ipipe equivalent
of /proc/config.gz to do this ?

wrt testsuite/README cruncher notes, is this useful info ?

(manual insmods here...)
soekris:/usr/realtime/2.6.14-ski9-v1/testsuite/cruncher# cruncher
Calibrating cruncher...11773, done -- ideal computation time = 10023 us.
1000 samples, 1000 hz freq (pid=4183, policy=SCHED_FIFO, prio=99)
--------
Nanosleep jitter: min = 60 us, max = 192 us, avg = 77 us
Execution jitter: min = 39 us (0%), max = 72 us (0%), avg = 51 us (0%)
--------
Segmentation fault

soekris:/usr/realtime/2.6.14-ski9-v1/testsuite/cruncher# run
*
*
* Type ^C to stop this application.
*
*
Calibrating cruncher...11769, done -- ideal computation time = 10018 us.
1000 samples, 1000 hz freq (pid=4260, policy=SCHED_FIFO, prio=99)
--------
Nanosleep jitter: min = 62 us, max = 195 us, avg = 79 us
Execution jitter: min = 46 us (0%), max = 77 us (0%), avg = 57 us (0%)
--------



2. send your results to xenomai.testout-at-gmail.com
Obviously, an official gna.org ML might be more appropriate.


Will appear soon.

should this wait til xeno-test is upgraded to produce good data ?
ie prevent early bogus data from being submitted.


<snip>

As said before, the problem that currently exists with LiveCD's data, is that the results are cripled with irrelevant stuff, either because some people just tried it out over a simulator (ahem...), or had a serious hw-generated latency issue that basically made the whole run useless (mostly x86 issues: e.g. SMI stuff, legacy USB emulation, powermgmt, cpufreq artefacts etc.).

I added a few /proc/config.gz related checks for CPU-FREQ, X86-GENERIC,
can you suggest additional checks ?





4. xeno-test output parser

- /proc/ipipe/Linux-stats parse into pairs of IRQ => CPU0 prop-times
- such data is only comparable across kernels with eq IRQ maps
- currently wont handle CPU1, SMP data
- /proc/interrupts is slightly better parsed.
- no detail-parse at all for top-data, needed?


I'm not sure that per-process data would help, just because those are way too volatile and fragmented to be interpreted rationally over a long test period; maybe using per-subsystem data (e.g. /proc/sys crowd) at some point in time would better help.




prototype only, but its hackable (perl), and Im happy to graft all
sorts of horrible experiments on it provisionally to see whats useful.
Hopefully a plugin refactoring will become obvious wo too much work.


Warning people: JimC belongs to some kind of hybridization between a Perl Monger and a Real-timer; and the resulting entity is about to go wild... :o>

go off the deep end ?  into shark infested waters ?



Generally speaking, I guess that your idea is to collect sensible raw data first, and devise how to process combiantions of them later. Sounds ok for me, and I especially like the idea of providing a specialized ML for that which would be processed by a bot', since anyone would have unlimited access to the data, which might trigger some incentive for anyone to craft other/better digested figures.

yup.  inspired by LiveCD, and your reaction to it.


We should make sure to not base all the reasoning on a lo latency / hi cpufreq correlation: this just happens to be wrong, especially x86-wise. Actually, a lot of recent x86 platforms with insanely high CPU freqs are really out of luck when it comes to perform decently in real-time mode, just because the trend of "optimization" is just about killing any determinism one would expect from his hw, by various ugly tricks often aimed at making gamers happy.

pentium 4's   31 stage instruction-processing pipeline ? :-O

Im not suggesting its a good measure, but that it would make an interesting graph.
latency vs mhz,  with data-points colored per the CPU type.
K6 - navy-blue, K7-royal-blue, K8- sky-blue, P2 - lime-green, P3 - mint-green, P4 - forest green



I understand "the plan behind the plan" to be able to somehow predict that some particular sw / hw combo would work and help people figuring out which platform they might want to build their RT solution over using Xeno, and it would be quite an achievement to do that.

For the time being though, I'd suggest that we focus on gathering raw data and digest them according to a few simple metrics first; I'm pretty sure that once a sane and simple infrastructure to do that is in place, we should be able to flesh out the available results. As usual, the key issue is to make such process of producing and using this data becoming a routine; once people get used to something, they tend to improve it quite naturally.


agreed. Its all blue-sky dreaming atm, and subject to ongoing reality checks,
and ongoing discussion ( in little trickles )

jimc

Reply via email to