Farid Zaripov wrote:
-----Original Message-----
From: Martin Sebor [mailto:[EMAIL PROTECTED]
Sent: Monday, November 13, 2006 6:53 PM
To: [email protected]
Subject: Re: RFC: publishing nightly test results
Any comments or questions on this? Farid?
Not yet. I found it useful as for me.
I see "DATA" in State column in rows with results on MSVC. The tooltop
comment says
"Incomplete data". Can I help with this? The windows build
infrastructure produces the build
log (i.e. msvc-7.1-15s.html) and perhaps this log can be attached? The
size of this log file
is big due to HTML format and UNICODE characters (on MSVC8). We can
modify
makelog.wsf script to produce the ASCII text log file as well as html
one.
Andrew and I brainstormed about how to make it possible and easy
for others (such as yourself) to submit stdcxx test results so that
they could be published along with ours. I think we came up with a
pretty nifty idea. Here's how it would work:
1. Create a script, say stdcxx-test, that when executed will pull
down the latest stdcxx sources from svn (or perhaps use the
already downloaded sources instead), run one or more builds
(perhaps all possible builds by default) with a given compiler,
collecting the logs and results in the currently published form
(i.e., config.h, config.log, and build.log). Also have the script
compute the total number of warnings, tests and failed tests,
examples and failed examples, etc. When done with the builds,
have the script tar and gzip the logs into one tarball per build
and make each available to the world under a URL. Then send an
email (e.g., to stdcxx-dev) in a preset format outlining the
results of each build (each email could contain one or more
results). Something like:
Result: http://tinyurl.com/yfsrzf
OS: AIX 5.2 # any string
Compiler: VisuaAge C++ 6.0 # any string
Build Type: 11d # valid build type
Timestamp: 11/12 03:46 # date/time (optional)
Warnings: 127 # number of warnings
Examples: 11/132 # failed/total examples
Tests: 38/121 # failed/total tests
Locales: 156/161 # failed/total locales
Utilities: 0/3 # failed/total utilities
Branch: # optional branch name
Revision: 473590 # svn revision of sources
2. Create another script running periodically, say once or a few
times a day, as a cron job at apache.org that processes the
emails above (e.g., by reading the stdcxx-dev mbox archive,
http://tinyurl.com/ylo4ly), extracting the relevant data and
adding them to the published test matrix currently at
http://people.apache.org/~sebor/stdcxx/results/.
This way any user could run stdcxx-test on their machine and get
their test results published on the official result page. We could
also instruct the second script to only process results posted by
trusted parties to prevent pranksters from spoofing our results.
Thought? Comments?
Martin
