I lied. Another update worth spamming about. The dashboard now by default hides both WONTFIX tests and tests that match their expectations. This way, the dashboard represents a concrete list of work to make tests match their expectations. By default this list should be near-zero.
This also makes the dashboard load a ton faster since there's a lot less to render. There are links to show the WONTFIX tests and tests that match their expectations. Like everything else, it modifies your URL, so you can bookmark and copy/paste the URLs if you prefer a different view than the default. Ojan On Wed, Aug 26, 2009 at 5:12 PM, Ojan Vafai <[email protected]> wrote: > One final update. > > 1. The bogus results on windows are gone. > 2. BUG*** now actually links to the bug. > 3. Hides WONTFIX tests by default, with a checkbox to show them. > 4. Linux release bot is now listed (debug coming soon) > 5. The dashboard now shows (highlights in blue) all cases where a test > either has expectations that never happen or doesn't have expectations that > do happen (see the 'extra' and 'missing' columns). > > The only work I still plan to do on it is to make it perform a bit better. > Please file bugs if there are other additions that would be useful to you. > > Ojan > > On Tue, Aug 25, 2009 at 10:15 AM, Ojan Vafai <[email protected]> wrote: > >> One more thing (since people are asking), you can see all the platforms' >> expectations for a test from the view for any builder. The ones that don't >> apply to this builder are greyed out. For example, see >> LayoutTests/fast/dom/HTMLObjectElement/object-as-frame.html on the "WebKit" >> builder. >> Ojan >> >> On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai <o...@chrom >> ium.org> wrote: >> >>> A first version of the layout test flakiness dashboard is up. >>> >>> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html >>> >>> Some of the key features: >>> >>> - updates roughly as quickly as the bots have run the tests (no >>> post-processing, stdio parsing or crons) >>> - sortable by any column include "flakiness" >>> - builder + sort order permalinks (i.e. you can copy-paste URLs for a >>> given builder + sort order) >>> - highlights tests with missing expectations (i.e. the test passed, >>> but is marked only as failing) >>> - prints out test run times for tests that took > 1 second >>> - shows the expectations for a test on all platforms >>> >>> Taking a look at the dashboard, you can immediately see a couple things: >>> >>> 1. A bunch of cases where we currently have the missing expectations >>> for a test. For example, LayoutTests/accessibility/plugin.html fails on >>> Windows, but is only listed as failing on the Mac. >>> 2. We have a few dozen very slow tests that considerably slow down >>> how long the tests take to run. >>> >>> How do you get this awesomeness for UI tests, unittests, etc? It's >>> actually *really* easy. Someone just needs to add something to the test >>> runners for those test types that spits out JSON files in the right format >>> and copies them to the appropriate place. Let me know if you're interested >>> in adding support for a new test type. >>> >>> Known issues: >>> >>> 1. The JS that generates the page could stand to be faster. >>> 2. The windows builders have a bunch of junk entries with windows >>> style paths. Ignore them for now, they'll gradually fall off the end of >>> the >>> tests tracked by the dashboard. Only the tests with unix style paths >>> should >>> be looked at. >>> >>> If you have bugs or feature requests. Please file them at crbug.com and >>> CC me. One feature request that I have is to also highlight cases where we >>> list an expectation for a test that never happens (e.g. the test is listed >>> as "PASS CRASH FAIL", but has never crashed. >>> >>> Ojan >>> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
