[chromium-dev] Re: Canary bot, now with formatted layout test results
This makes me very happy! 2009/10/15 Glenn Wilson gwil...@chromium.org: Hi Webkit gardeners, The Windows canary bot now generates formatted layout test results on each build that had unexpected failing layout tests, which means you can quickly see layout test failures, diffs, and upstream baselines without running the tool manually. The goal of this output is to help gardeners assess what may have changed, and how to best handle the failure. For example, here are the failures for build 13359: http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html To see the formatted layout test failures for a specific canary build, just click on the layout test results link in the archive webkit test results step, then click on the index-.html file. There's still more work to do, including getting the formatter working properly on the Mac Linux canaries, adding flakiness data, adding a link directly to the waterfall display, and more. Many thanks go to Ojan, Nicolas and all others who endured my many code reviews, and Eric Roman and Dimitri for the original spec and design. Please let me know what you think, and what changes you'd like to see. Glenn -- Erik Corry, Software Engineer Google Denmark ApS. CVR nr. 28 86 69 84 c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018 Copenhagen K, Denmark. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
Christmas came early for anyone working on WebKit! Thanks Glen. :-) On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.com wrote: This makes me very happy! 2009/10/15 Glenn Wilson gwil...@chromium.org: Hi Webkit gardeners, The Windows canary bot now generates formatted layout test results on each build that had unexpected failing layout tests, which means you can quickly see layout test failures, diffs, and upstream baselines without running the tool manually. The goal of this output is to help gardeners assess what may have changed, and how to best handle the failure. For example, here are the failures for build 13359: http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html To see the formatted layout test failures for a specific canary build, just click on the layout test results link in the archive webkit test results step, then click on the index-.html file. There's still more work to do, including getting the formatter working properly on the Mac Linux canaries, adding flakiness data, adding a link directly to the waterfall display, and more. Many thanks go to Ojan, Nicolas and all others who endured my many code reviews, and Eric Roman and Dimitri for the original spec and design. Please let me know what you think, and what changes you'd like to see. Glenn -- Erik Corry, Software Engineer Google Denmark ApS. CVR nr. 28 86 69 84 c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018 Copenhagen K, Denmark. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
Glenn, this tool is amazing! It really helped me yesterday on my 1st gardening shift. Re: upstream baselines without running the tool manually How do I do that? On Thu, Oct 15, 2009 at 10:01 AM, Jeremy Orlow jor...@chromium.org wrote: Christmas came early for anyone working on WebKit! Thanks Glen. :-) On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.com wrote: This makes me very happy! 2009/10/15 Glenn Wilson gwil...@chromium.org: Hi Webkit gardeners, The Windows canary bot now generates formatted layout test results on each build that had unexpected failing layout tests, which means you can quickly see layout test failures, diffs, and upstream baselines without running the tool manually. The goal of this output is to help gardeners assess what may have changed, and how to best handle the failure. For example, here are the failures for build 13359: http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html To see the formatted layout test failures for a specific canary build, just click on the layout test results link in the archive webkit test results step, then click on the index-.html file. There's still more work to do, including getting the formatter working properly on the Mac Linux canaries, adding flakiness data, adding a link directly to the waterfall display, and more. Many thanks go to Ojan, Nicolas and all others who endured my many code reviews, and Eric Roman and Dimitri for the original spec and design. Please let me know what you think, and what changes you'd like to see. Glenn -- Erik Corry, Software Engineer Google Denmark ApS. CVR nr. 28 86 69 84 c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018 Copenhagen K, Denmark. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
Actually, can you package this so it could be upstreamed to webkit.org?
On Thu, Oct 15, 2009 at 1:09 PM, Glenn Wilson gwil...@chromium.org wrote:
The tool lives in
src/webkit/tools/layout_tests/test_output_formatter{.bat,.sh}, and it can
actually be run against any builder, not just the canaries. Adding better
documentation is on the TODO list :)
(Also, re: adding link to waterfall, I think we can do this -- I'm
investigating now.)
On Thu, Oct 15, 2009 at 10:05 AM, Yaar Schnitman y...@chromium.orgwrote:
Glenn, this tool is amazing! It really helped me yesterday on my 1st
gardening shift.
Re: upstream baselines without running the tool manually
How do I do that?
On Thu, Oct 15, 2009 at 10:01 AM, Jeremy Orlow jor...@chromium.orgwrote:
Christmas came early for anyone working on WebKit! Thanks Glen. :-)
On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.comwrote:
This makes me very happy!
2009/10/15 Glenn Wilson gwil...@chromium.org:
Hi Webkit gardeners,
The Windows canary bot now generates formatted layout test results on
each
build that had unexpected failing layout tests, which means you can
quickly
see layout test failures, diffs, and upstream baselines without
running the
tool manually. The goal of this output is to help gardeners assess
what may
have changed, and how to best handle the failure.
For example, here are the failures for build 13359:
http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html
To see the formatted layout test failures for a specific canary build,
just
click on the layout test results link in the archive webkit test
results
step, then click on the index-.html file.
There's still more work to do, including getting the formatter working
properly on the Mac Linux canaries, adding flakiness data, adding a
link
directly to the waterfall display, and more. Many thanks go to Ojan,
Nicolas and all others who endured my many code reviews, and Eric
Roman and
Dimitri for the original spec and design.
Please let me know what you think, and what changes you'd like to see.
Glenn
--
Erik Corry, Software Engineer
Google Denmark ApS. CVR nr. 28 86 69 84
c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018
Copenhagen K, Denmark.
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
Yes, possibly. The tool is written explicitly for the Chromium builders'
output, so it would be plausible but non-trivial to generalize it.
It may not be as useful since I don't think they have a concept of an
'upstream' baseline. Those of you who do a lot of work upstream, would it
be worth upstreaming this?
Glenn
On Thu, Oct 15, 2009 at 12:44 PM, Marc-Antoine Ruel mar...@chromium.orgwrote:
Actually, can you package this so it could be upstreamed to webkit.org?
On Thu, Oct 15, 2009 at 1:09 PM, Glenn Wilson gwil...@chromium.orgwrote:
The tool lives in
src/webkit/tools/layout_tests/test_output_formatter{.bat,.sh}, and it can
actually be run against any builder, not just the canaries. Adding better
documentation is on the TODO list :)
(Also, re: adding link to waterfall, I think we can do this -- I'm
investigating now.)
On Thu, Oct 15, 2009 at 10:05 AM, Yaar Schnitman y...@chromium.orgwrote:
Glenn, this tool is amazing! It really helped me yesterday on my 1st
gardening shift.
Re: upstream baselines without running the tool manually
How do I do that?
On Thu, Oct 15, 2009 at 10:01 AM, Jeremy Orlow jor...@chromium.orgwrote:
Christmas came early for anyone working on WebKit! Thanks Glen. :-)
On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.comwrote:
This makes me very happy!
2009/10/15 Glenn Wilson gwil...@chromium.org:
Hi Webkit gardeners,
The Windows canary bot now generates formatted layout test results on
each
build that had unexpected failing layout tests, which means you can
quickly
see layout test failures, diffs, and upstream baselines without
running the
tool manually. The goal of this output is to help gardeners assess
what may
have changed, and how to best handle the failure.
For example, here are the failures for build 13359:
http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html
To see the formatted layout test failures for a specific canary
build, just
click on the layout test results link in the archive webkit test
results
step, then click on the index-.html file.
There's still more work to do, including getting the formatter
working
properly on the Mac Linux canaries, adding flakiness data, adding a
link
directly to the waterfall display, and more. Many thanks go to Ojan,
Nicolas and all others who endured my many code reviews, and Eric
Roman and
Dimitri for the original spec and design.
Please let me know what you think, and what changes you'd like to
see.
Glenn
--
Erik Corry, Software Engineer
Google Denmark ApS. CVR nr. 28 86 69 84
c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018
Copenhagen K, Denmark.
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
One could show baselines of all the other platforms instead of just an
upstream baseline.
I'm not really sure from the output what this tool does. So it's
difficult for me to tell how useful it would be to upstream.
-eric
On Thu, Oct 15, 2009 at 8:38 PM, Glenn Wilson gwil...@chromium.org wrote:
Yes, possibly. The tool is written explicitly for the Chromium builders'
output, so it would be plausible but non-trivial to generalize it.
It may not be as useful since I don't think they have a concept of an
'upstream' baseline. Those of you who do a lot of work upstream, would it
be worth upstreaming this?
Glenn
On Thu, Oct 15, 2009 at 12:44 PM, Marc-Antoine Ruel mar...@chromium.org
wrote:
Actually, can you package this so it could be upstreamed to webkit.org?
On Thu, Oct 15, 2009 at 1:09 PM, Glenn Wilson gwil...@chromium.org
wrote:
The tool lives in
src/webkit/tools/layout_tests/test_output_formatter{.bat,.sh}, and it can
actually be run against any builder, not just the canaries. Adding better
documentation is on the TODO list :)
(Also, re: adding link to waterfall, I think we can do this -- I'm
investigating now.)
On Thu, Oct 15, 2009 at 10:05 AM, Yaar Schnitman y...@chromium.org
wrote:
Glenn, this tool is amazing! It really helped me yesterday on my 1st
gardening shift.
Re: upstream baselines without running the tool manually
How do I do that?
On Thu, Oct 15, 2009 at 10:01 AM, Jeremy Orlow jor...@chromium.org
wrote:
Christmas came early for anyone working on WebKit! Thanks Glen. :-)
On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.com
wrote:
This makes me very happy!
2009/10/15 Glenn Wilson gwil...@chromium.org:
Hi Webkit gardeners,
The Windows canary bot now generates formatted layout test results
on each
build that had unexpected failing layout tests, which means you can
quickly
see layout test failures, diffs, and upstream baselines without
running the
tool manually. The goal of this output is to help gardeners assess
what may
have changed, and how to best handle the failure.
For example, here are the failures for build 13359:
http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html
To see the formatted layout test failures for a specific canary
build, just
click on the layout test results link in the archive webkit test
results
step, then click on the index-.html file.
There's still more work to do, including getting the formatter
working
properly on the Mac Linux canaries, adding flakiness data, adding
a link
directly to the waterfall display, and more. Many thanks go to
Ojan,
Nicolas and all others who endured my many code reviews, and Eric
Roman and
Dimitri for the original spec and design.
Please let me know what you think, and what changes you'd like to
see.
Glenn
--
Erik Corry, Software Engineer
Google Denmark ApS. CVR nr. 28 86 69 84
c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018
Copenhagen K, Denmark.
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
