Ojan is working on the tool for the layout tests. First bits are already checked in.
:DG< On Wed, Aug 5, 2009 at 10:10 PM, Eric Seidel<[email protected]> wrote: > Do we have a list of flakey tests? I feel like we used to have one... > > On Wed, Aug 5, 2009 at 9:44 PM, Peter Kasting <[email protected]> wrote: >> >> THIS MAIL APPLIES TO YOU >> Flakiness is growing. Smash it before it gets bigger, and keep it >> smashed. >> *** >> The MOST IMPORTANT section in this gigantic mail: >> PLEASE spend some of every workday (or each week at least, if you can't >> spare time each day) looking at test failures, flakiness, >> valgrind/purify/coverity bugs, crashes, and/or memory bugs. Make it a goal >> to get an average of one line in the test-expectations file removed each >> day. If you're a Googler, put it on your OKRs (now, not sometime tomorrow). >> * DON'T wait for someone to assign bugs to you or ask for your help >> * DON'T wait for a team fixit week (those haven't worked) >> * DON'T wait for someone else to solve the problems >> * DON'T wait until after your current project is finished >> * DON'T wait until you have worked on WebKit >> HELP, even if it's just a little, even if it's not your core competence. >> We currently have hundreds upon hundreds of failing or flaky tests. We can >> dramatically reduce this quickly but ONLY IF YOU HELP. This is an >> investment not only in the quality of Chrome but in the team's ability to >> move fast, so help here doesn't just improve the quality of Chrome, but also >> the derivative of the quality :) >> (If you do not know how to do anything above and need handholding, e-mail >> me and I will help you. It's OK to be ignorant.) >> *** >> Next, how you should help keep the tree green at all times: >> * If you ever look at the buildbot and see red, and there's no explanation >> in the build status, ask what's going on on #chromium. Ping the sheriffs >> specifically (they're listed in the upper-right corner). If you do not get >> an answer about ownership within a few minutes, close the tree (if you have >> the rights to) or ask someone to close it. THE TREE SHOULD NOT BE OPEN WITH >> RED THAT NO ONE OWNS. Help the sheriffs out with this -- they can't watch >> every second. Closed trees suck; unowned bustage sucks more. Be >> hard-nosed. >> * Yes, even purify, valgrind, and reliability bot redness. If you can't >> figure out what to do with these, try pinging erikkay for purify issues and >> huanr for reliability issues. (Not sure who a good general valgrind contact >> is.) >> * If you ever look at the buildbot and see orange ("unexpected pass"), >> especially in the WebKit LayoutTest bots, ping the WebKit sheriff (the >> calendar is linked from the top >> of http://dev.chromium.org/developers/how-tos/webkit-merge-1 ; I don't know >> whether it's world-readable). If he wasn't aware of it, agree between you >> on who will deal with it. Orange alone is not reason to close the tree, but >> it should NOT be ignored. >> * DON'T IGNORE TESTS BECAUSE THEY WENT GREEN ON THE NEXT CYCLE. If >> they're really fixed by someone's commit, that should be easy to determine. >> Otherwise, they're flaky, and we NEED to mark them as such, not just leave >> them. >> *** >> Finally, how to help if the LayoutTest bots are red or orange: >> (1) Try and determine if the test(s) are consistently passing/failing >> unexpectedly, or if they're flaky. Make sure you look at all the different >> bots to see which OSes are affected. >> (2) Update src/webkit/tools/layout_tests/test-expectations.txt. Look for >> the test(s) in question. Often, flaky tests will already be in there as >> failing or flaky for one OS, and need to have more added; or they will be >> marked flaky ("FAIL PASS") and need "CRASH" added. If they're not there, >> add a line. >> (3) Ensure the test(s) have a bug on file. Note the bug on the >> expectation. >> (4) If any tests are crashing (flaky or not), they're high-priority and >> someone needs to triage them. Today, dglazkov was WebKit sheriff and was >> having me mark these bugs as P1, Mstone-3, owner:dglazkov. I'm not sure >> whether the Right Thing is to assign them to the WebKit sheriff or still to >> him (feel free to comment, dglazkov!). Why are these P1? Because until we >> prove they can't affect Chrome itself, they potentially can, and Chrome >> crashes are always P1. They affect stability and security both. >> (5) If you have commit rights, go ahead and TBR test-expectations changes >> you're confident of. I even suggest using --force if the tree is closed. >> Updating expectations is like fixing bustage, it helps the tree go green >> faster and thus is almost always desirable. If you don't have commit >> rights, send your review to the WebKit sheriff. >> *** >> Your reward for reading this far: >> * At the end of the quarter, I will nominate for a peer bonus every >> Googler who puts something meaningful about flakiness/test failures/the >> other stuff above on their OKRs, accomplishes it, and sends me a note >> pointing that out. >> * At the end of the quarter, I will nominate for commit access every >> non-Googler who sends me a pointer to ten patches relating to the above >> items that they have posted for review, and who doesn't otherwise have some >> reason why they can't be nominated. >> If other people want to sweeten the pot somehow, feel free. >> PK >> > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
