Hi Jim,
Jim Cromie wrote:
Philippe Gerum wrote:
This is a partial roadmap for the project, composed of the currently
Ah! I just _knew_ you would jump in as expected. The teasing worked :o)
o Web site.
Wiki ++ , eventually
Yes, separate issue to deal with Bruno and anyone who has some thoughts about
how to efficiently help people using Xeno, especially newcomers.
o Automated benchmarking.
- We are still considering the best way to do that; actually,
my take is that we would just need to bootstrap the thing and
flesh it out over time, writing one or two significant
benchmark tests to start with, choosing a tool to plot the
collected data and push the results to some web page for
public consumption on a regular basis, but so far, we did not
manage to spark this. It's still in the short-term plan,
though, because we currently have neither metrics nor data to
check for basics, and we deeply need both of them now.
ETA: Q4 2005.
A Xenomai Automatic Benchmarking plan
Goal is to test xenomai performance so we know when something breaks,
test it thoroughly enough that we can see / identify systematic,
generic, or
platform specific bottlenecks.
Yes; as a corollary, a tool to routinely, better and earlier check whether we
are on the good track or not regarding core updates without having to wait for a
blatant breakage to send us the wake up call.
Benchmarking
wrt bootstrap approach; scripts/xeno-test already runs 2
of 3 testsuite/* tests, and collects the results along with useful
platform data. If new testsuite/* stuff gets added, its trivial to
call them from xeno-test.
Indeed; xeno-test has always been meant to provide a simple client-side tool
available from regular Xenomai distros for running tests/benchmarks that could
be extended over time. The idea underlying this being that giving people easy
means to send us the test data output by plain simple standard stuff they find
in their Xeno distro is way more efficient than asking them to connect to some
site and download specialized tools to do that. The extra middle step is often
the one too many that kills the initial incentive to help.
Automatic
Automating the process is trickier than usual, due to need for
cross-compile (in some situations), NFS root mounts for remote boxes,
remote or scripted reboots, etc. Ive cobbled up a rube-goldberg
arrangement, which is out-of-scope for this message, will discuss all
that separately.
Characterization
RPM mentioned plotting, I take that to mean heavy use of graphs to
characterize and ultimately to predict xenomai performance over a
range of criteria, for any given platform.
At least characterize the current state, yes.
LiveCD had the right idea wrt this - collecting platform info and
performance data on any vanilla PC with a CD-ROM drive. And make this
data available on a website, allowing users to compare their results
with others done on similar platforms.
LiveCD has a few weaknesses though:
- cant test platforms w/o cdrom
I also think that's a serious issue. Aside of the hw availability problem (e.g.
non-x86 eval boards), having to burn the CD is one step too many when time is a
scarce resource. It often prevents to run it as a fast check procedure even in
the absence of any noticeable problem. IOW, you won't burn a CD to run the tests
unless you are really stuck with some issue. So a significant part of the
interest of having a generic testsuite is lost: you just don't discover
potential problems before the serious breakage is already in the wild.
- manual re-entry of data is tedious,
- no collection of platform data (available for automation)
- spotty info about cpu, memory, mobo, etc
- no unattended test (still true?)
- unfiltered preposterous data. Sometimes, data sent are just rubbish because of
well-known hw-related dysfunctioning or misuse of the LiveCD. This perturbates
the results uselessly.
- difficulties so far to really get a sensible digested information out of the
zillions of results, aside of very general figures (e.g. best performer). But
this is more an issue of lack of data post-processors than of the LiveCD
infrastructure itself.
These things could be readily fixed, but xeno-test already does
everything but the data upload.
The real value of LiveCD was the collection of data across hundreds of
different platforms, and its promise was that studying the data would
reveal the secrets of better performance on any platform.
Additionally, LiveCD is a really great tool when it comes to help people
figuring out whether their respective box or brain have a problem with the
tested software, i.e. by automatically providing a sane software (kernel+rtos)
configuration and the proper way to run it quite easily, a number of people
could determine if their current lack of luck comes from their software
configuration, or rather from a more serious problem.
A Plan (sort of)
