Re: [RFC] build result views

[Martin Sebor]

Mark Brown wrote:

Have you guys seen the Boost regression test page?
http://engineering.meta-comm.com/boost-regression/1_34_1/user/summary_release.html 


I saw at it a long time ago but not recently. It looks like they
have made some improvements since then.

Interestingly, the set of platforms indicates they're testing with
STLport but not with our implementation. Farid, do you have any idea
what it would take to get them to start testing with stdcxx (and
publish test results with it)? It looks like the sets of results
they publish now are all provided by volunteers. Could we offer to
do the testing with stdcxx for them?



I think there are some good ideas there. I like the links to
the test sources and to the compiler errors for the ones that
error out.


The links would definitely be useful. We need to come up with a way
to annotate the command lines in build logs so we can either easily
extract them or insert HTML anchors. Any ideas anyone?

I also like the annotations explaining the test failures -- for
example here: http://tinyurl.com/2532a8. We've been talking about
dealing with expected failures but this is something we haven't
considered doing. Seems like an interesting alternative to the
expected failure extension to the test driver that we discussed.

Martin



--Mark

Martin Sebor wrote:

Farid Zaripov wrote:

-Original Message-
From: Martin Sebor [mailto:[EMAIL PROTECTED] On Behalf Of Martin Sebor
Sent: Tuesday, November 13, 2007 6:18 AM
To: stdcxx-dev@incubator.apache.org
Subject: [RFC] build result views

I'm looking for feedback on the two sets of nightly result pages we 
currently publish:


   http://people.apache.org/~sebor/stdcxx/results/

and

   http://people.apache.org/~sebor/stdcxx/results/builds

Specifically, I'm wondering what would people think about replacing 
the first page with the second, or something like it. Do you find 
yourself using the first page more or do you prefer the second one, 
and why?


  The first page is useful to see the overall status of the library.
I'm using it just for sure that Windows builds are green :)


:) That's how I've been using it too. The danger of colorizing
entire builds the way we do on that page is in making it easy
to miss important failures occurring only in a small number of
tests. I.e., you might see green even when a critical piece of
the library is broken (recall the recent binary incompatibility
with XLC++ exception).

Anyway, sounds like adding colorization either to builds.html
or to the Logs and Columns table on each Multi-platform Test
Result View is one enhancement you'd like to see, correct?

What about data? Do you use any of the data from the colorized
page? E.g., the number of components (examples, tests, locales)
vs the number of those that failed? I personally don't think
they are terribly useful but adding them shouldn't be too hard.
I do plan on adding the duration column (with a lot more detail,
such as how long the library took to build in user, system, and
wall clock time, and the same for all examples, the test driver,
tests, and locales). Anything else?



  Of cource the second page is more convinient to see what
examples or tests are failed and how these fails dependent
from the build type.

  I would like to have the possibility to merge the results of the
multiple platforms into the one page, i.e.:
- all windows builds (MSVC and ICC);
- all MSVC builds;
- all ICC/windows builds;
- all ICC builds (windows and linux);
- all MSVC 8.0 build + all gcc 4.2.0 builds;


This is something I'd like to be able to do as well, and I have
in a small number of cases. It can easily be done by changing
the genxviews script to generate whatever combination of builds
you need (we should move the data out of the script to make it
possible to do this without modifying the script itself).

After making the changes you just run your custom genxviews like
this:

  $ genxviews  $HOME/public_html/stdcxx/results/builds.html

Of course, the ultimate implementation would let you do it on
demand (e.g., as you suggest below).

One thing to keep in mind is that the more builds you squeeze on
a page the harder it becomes to see them all at the same time. At
a certain point it starts to defeat the purpose of the page because
you end up scrolling it left and right to see the results for all
the platforms.


...

  It might be realized as the current page
(http://people.apache.org/~sebor/stdcxx/results/builds)
but with the checkboxes on the each line of the table and button
Next somewhere on the page.


That would be pretty cool. The only thing is that generating the
pages takes quite a bit of time (you can see how long in the Time
column), so you might have to wait a few minutes to get the results
for a custom selection. We could probably optimize it to just a few
seconds by pre-processing the individual logs so as not to make the
script work so hard.

Martin



If neither, how do you analyze build results and why do you find 
your 

Re: [RFC] build result views

[Mark Brown]

Have you guys seen the Boost regression test page?
http://engineering.meta-comm.com/boost-regression/1_34_1/user/summary_release.html
I think there are some good ideas there. I like the links to
the test sources and to the compiler errors for the ones that
error out.

--Mark

Martin Sebor wrote:

Farid Zaripov wrote:

-Original Message-
From: Martin Sebor [mailto:[EMAIL PROTECTED] On Behalf Of Martin Sebor
Sent: Tuesday, November 13, 2007 6:18 AM
To: stdcxx-dev@incubator.apache.org
Subject: [RFC] build result views

I'm looking for feedback on the two sets of nightly result pages we 
currently publish:


   http://people.apache.org/~sebor/stdcxx/results/

and

   http://people.apache.org/~sebor/stdcxx/results/builds

Specifically, I'm wondering what would people think about replacing 
the first page with the second, or something like it. Do you find 
yourself using the first page more or do you prefer the second one, 
and why?


  The first page is useful to see the overall status of the library.
I'm using it just for sure that Windows builds are green :)


:) That's how I've been using it too. The danger of colorizing
entire builds the way we do on that page is in making it easy
to miss important failures occurring only in a small number of
tests. I.e., you might see green even when a critical piece of
the library is broken (recall the recent binary incompatibility
with XLC++ exception).

Anyway, sounds like adding colorization either to builds.html
or to the Logs and Columns table on each Multi-platform Test
Result View is one enhancement you'd like to see, correct?

What about data? Do you use any of the data from the colorized
page? E.g., the number of components (examples, tests, locales)
vs the number of those that failed? I personally don't think
they are terribly useful but adding them shouldn't be too hard.
I do plan on adding the duration column (with a lot more detail,
such as how long the library took to build in user, system, and
wall clock time, and the same for all examples, the test driver,
tests, and locales). Anything else?



  Of cource the second page is more convinient to see what
examples or tests are failed and how these fails dependent
from the build type.

  I would like to have the possibility to merge the results of the
multiple platforms into the one page, i.e.:
- all windows builds (MSVC and ICC);
- all MSVC builds;
- all ICC/windows builds;
- all ICC builds (windows and linux);
- all MSVC 8.0 build + all gcc 4.2.0 builds;


This is something I'd like to be able to do as well, and I have
in a small number of cases. It can easily be done by changing
the genxviews script to generate whatever combination of builds
you need (we should move the data out of the script to make it
possible to do this without modifying the script itself).

After making the changes you just run your custom genxviews like
this:

  $ genxviews  $HOME/public_html/stdcxx/results/builds.html

Of course, the ultimate implementation would let you do it on
demand (e.g., as you suggest below).

One thing to keep in mind is that the more builds you squeeze on
a page the harder it becomes to see them all at the same time. At
a certain point it starts to defeat the purpose of the page because
you end up scrolling it left and right to see the results for all
the platforms.


...

  It might be realized as the current page
(http://people.apache.org/~sebor/stdcxx/results/builds)
but with the checkboxes on the each line of the table and button
Next somewhere on the page.


That would be pretty cool. The only thing is that generating the
pages takes quite a bit of time (you can see how long in the Time
column), so you might have to wait a few minutes to get the results
for a custom selection. We could probably optimize it to just a few
seconds by pre-processing the individual logs so as not to make the
script work so hard.

Martin



If neither, how do you analyze build results and why do you find your 
system preferable to what we have? What does your ideal result page 
look like? What data should it show and how should it be presented?


Farid.

RE: [RFC] build result views

[Farid Zaripov]

 -Original Message-
 From: Martin Sebor [mailto:[EMAIL PROTECTED] On Behalf Of Martin Sebor
 Sent: Tuesday, November 13, 2007 6:18 AM
 To: stdcxx-dev@incubator.apache.org
 Subject: [RFC] build result views
 
 I'm looking for feedback on the two sets of nightly result 
 pages we currently publish:
 
http://people.apache.org/~sebor/stdcxx/results/
 
 and
 
http://people.apache.org/~sebor/stdcxx/results/builds
 
 Specifically, I'm wondering what would people think about 
 replacing the first page with the second, or something like 
 it. Do you find yourself using the first page more or do you 
 prefer the second one, and why?

  The first page is useful to see the overall status of the library.
I'm using it just for sure that Windows builds are green :)

  Of cource the second page is more convinient to see what
examples or tests are failed and how these fails dependent
from the build type.

  I would like to have the possibility to merge the results of the
multiple platforms into the one page, i.e.:
- all windows builds (MSVC and ICC);
- all MSVC builds;
- all ICC/windows builds;
- all ICC builds (windows and linux);
- all MSVC 8.0 build + all gcc 4.2.0 builds;
...

  It might be realized as the current page
(http://people.apache.org/~sebor/stdcxx/results/builds)
but with the checkboxes on the each line of the table and button
Next somewhere on the page.

 If neither, how do you 
 analyze build results and why do you find your system 
 preferable to what we have? What does your ideal result page 
 look like? What data should it show and how should it be presented?

Farid.

Re: [RFC] build result views

[Martin Sebor]

Farid Zaripov wrote:

-Original Message-
From: Martin Sebor [mailto:[EMAIL PROTECTED] On Behalf Of Martin Sebor
Sent: Tuesday, November 13, 2007 6:18 AM
To: stdcxx-dev@incubator.apache.org
Subject: [RFC] build result views

I'm looking for feedback on the two sets of nightly result 
pages we currently publish:


   http://people.apache.org/~sebor/stdcxx/results/

and

   http://people.apache.org/~sebor/stdcxx/results/builds

Specifically, I'm wondering what would people think about 
replacing the first page with the second, or something like 
it. Do you find yourself using the first page more or do you 
prefer the second one, and why?


  The first page is useful to see the overall status of the library.
I'm using it just for sure that Windows builds are green :)


:) That's how I've been using it too. The danger of colorizing
entire builds the way we do on that page is in making it easy
to miss important failures occurring only in a small number of
tests. I.e., you might see green even when a critical piece of
the library is broken (recall the recent binary incompatibility
with XLC++ exception).

Anyway, sounds like adding colorization either to builds.html
or to the Logs and Columns table on each Multi-platform Test
Result View is one enhancement you'd like to see, correct?

What about data? Do you use any of the data from the colorized
page? E.g., the number of components (examples, tests, locales)
vs the number of those that failed? I personally don't think
they are terribly useful but adding them shouldn't be too hard.
I do plan on adding the duration column (with a lot more detail,
such as how long the library took to build in user, system, and
wall clock time, and the same for all examples, the test driver,
tests, and locales). Anything else?



  Of cource the second page is more convinient to see what
examples or tests are failed and how these fails dependent
from the build type.

  I would like to have the possibility to merge the results of the
multiple platforms into the one page, i.e.:
- all windows builds (MSVC and ICC);
- all MSVC builds;
- all ICC/windows builds;
- all ICC builds (windows and linux);
- all MSVC 8.0 build + all gcc 4.2.0 builds;


This is something I'd like to be able to do as well, and I have
in a small number of cases. It can easily be done by changing
the genxviews script to generate whatever combination of builds
you need (we should move the data out of the script to make it
possible to do this without modifying the script itself).

After making the changes you just run your custom genxviews like
this:

  $ genxviews  $HOME/public_html/stdcxx/results/builds.html

Of course, the ultimate implementation would let you do it on
demand (e.g., as you suggest below).

One thing to keep in mind is that the more builds you squeeze on
a page the harder it becomes to see them all at the same time. At
a certain point it starts to defeat the purpose of the page because
you end up scrolling it left and right to see the results for all
the platforms.


...

  It might be realized as the current page
(http://people.apache.org/~sebor/stdcxx/results/builds)
but with the checkboxes on the each line of the table and button
Next somewhere on the page.


That would be pretty cool. The only thing is that generating the
pages takes quite a bit of time (you can see how long in the Time
column), so you might have to wait a few minutes to get the results
for a custom selection. We could probably optimize it to just a few
seconds by pre-processing the individual logs so as not to make the
script work so hard.

Martin



If neither, how do you 
analyze build results and why do you find your system 
preferable to what we have? What does your ideal result page 
look like? What data should it show and how should it be presented?


Farid.