One final update. 1. The bogus results on windows are gone. 2. BUG*** now actually links to the bug. 3. Hides WONTFIX tests by default, with a checkbox to show them. 4. Linux release bot is now listed (debug coming soon) 5. The dashboard now shows (highlights in blue) all cases where a test either has expectations that never happen or doesn't have expectations that do happen (see the 'extra' and 'missing' columns).
The only work I still plan to do on it is to make it perform a bit better. Please file bugs if there are other additions that would be useful to you. Ojan On Tue, Aug 25, 2009 at 10:15 AM, Ojan Vafai <[email protected]> wrote: > One more thing (since people are asking), you can see all the platforms' > expectations for a test from the view for any builder. The ones that don't > apply to this builder are greyed out. For example, see > LayoutTests/fast/dom/HTMLObjectElement/object-as-frame.html on the "WebKit" > builder. > Ojan > > On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai <o...@chrom > ium.org> wrote: > >> A first version of the layout test flakiness dashboard is up. >> >> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html >> >> Some of the key features: >> >> - updates roughly as quickly as the bots have run the tests (no >> post-processing, stdio parsing or crons) >> - sortable by any column include "flakiness" >> - builder + sort order permalinks (i.e. you can copy-paste URLs for a >> given builder + sort order) >> - highlights tests with missing expectations (i.e. the test passed, >> but is marked only as failing) >> - prints out test run times for tests that took > 1 second >> - shows the expectations for a test on all platforms >> >> Taking a look at the dashboard, you can immediately see a couple things: >> >> 1. A bunch of cases where we currently have the missing expectations >> for a test. For example, LayoutTests/accessibility/plugin.html fails on >> Windows, but is only listed as failing on the Mac. >> 2. We have a few dozen very slow tests that considerably slow down how >> long the tests take to run. >> >> How do you get this awesomeness for UI tests, unittests, etc? It's >> actually *really* easy. Someone just needs to add something to the test >> runners for those test types that spits out JSON files in the right format >> and copies them to the appropriate place. Let me know if you're interested >> in adding support for a new test type. >> >> Known issues: >> >> 1. The JS that generates the page could stand to be faster. >> 2. The windows builders have a bunch of junk entries with windows >> style paths. Ignore them for now, they'll gradually fall off the end of >> the >> tests tracked by the dashboard. Only the tests with unix style paths >> should >> be looked at. >> >> If you have bugs or feature requests. Please file them at crbug.com and >> CC me. One feature request that I have is to also highlight cases where we >> list an expectation for a test that never happens (e.g. the test is listed >> as "PASS CRASH FAIL", but has never crashed. >> >> Ojan >> > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
