A first version of the layout test flakiness dashboard is up.
http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html
Some of the key features:
- updates roughly as quickly as the bots have run the tests (no
post-processing, stdio parsing or crons)
- sortable by any column include "flakiness"
- builder + sort order permalinks (i.e. you can copy-paste URLs for a
given builder + sort order)
- highlights tests with missing expectations (i.e. the test passed, but
is marked only as failing)
- prints out test run times for tests that took > 1 second
- shows the expectations for a test on all platforms
Taking a look at the dashboard, you can immediately see a couple things:
1. A bunch of cases where we currently have the missing expectations for
a test. For example, LayoutTests/accessibility/plugin.html fails on Windows,
but is only listed as failing on the Mac.
2. We have a few dozen very slow tests that considerably slow down how
long the tests take to run.
How do you get this awesomeness for UI tests, unittests, etc? It's actually
*really* easy. Someone just needs to add something to the test runners for
those test types that spits out JSON files in the right format and copies
them to the appropriate place. Let me know if you're interested in adding
support for a new test type.
Known issues:
1. The JS that generates the page could stand to be faster.
2. The windows builders have a bunch of junk entries with windows style
paths. Ignore them for now, they'll gradually fall off the end of the tests
tracked by the dashboard. Only the tests with unix style paths should be
looked at.
If you have bugs or feature requests. Please file them at crbug.com and CC
me. One feature request that I have is to also highlight cases where we list
an expectation for a test that never happens (e.g. the test is listed as
"PASS CRASH FAIL", but has never crashed.
Ojan
--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---
