On Wed, Apr 6, 2011 at 9:01 PM, Maciej Stachowiak <[email protected]> wrote: > > On Apr 6, 2011, at 7:39 PM, Dirk Pranke wrote: > >> >> There are also a number of bugs currently listed as blocking that I >> don't think really qualify. Unless told otherwise, I'm plannning to >> remove the blocking flag from the following on Monday 4/11 (if they >> haven't been fixed first): >> >> 57640 [GTK] overlapping drag&drop tests fail on NRWT >> 55909 new-run-webkit-tests --run-singly option is busted >> 55163 new-run-webkit-tests: enable multiple processes by default on Chromium >> Win >> 47240 new-run-webkit-tests: getting an "error 2" back from ImageDiff >> 37426 new-run-webkit-tests should use the ServerProcess abstraction in >> chromium.py >> 37007 fast/tokenizer/doctype-search-reset.html fails when run out of >> order (new-run-webkit-tests) >> 35359 fast/repaint/renderer-destruction-by-invalidateSelection-crash.html >> fails intermittently >> 35266 new-run-webkit-tests --platform=mac-leopard timeout limit should >> match run-webkit-tests >> 35049 http/tests/security/cross-frame-access-put.html fails >> intermittently under new-run-webkit-tests >> 35006 fast/dom/global-constructors.html is failing based on previous tests >> >> Also, just because I don't think they should block a cutover, I do >> still think they should be fixed, so don't worry about that :) > > I think the ones that represent tests newly failing or becoming flaky should > be fixed before cutting over. We wouldn't want to lose test coverage when we > do the switch, right? >
Hi Maciej, I'm not sure I understand you, but if I do, this is what I was attempting to talk about in the paragraph above, about expecting some tests to be flaky or failing under NRWT simply because NRWT isn't exactly identical to ORWT. NRWT may be exposing bugs in the code that ORWT didn't trigger (e.g., because tests ran in a slightly different order, or because of the concurrency issues). It may be that you're thinking that either we run the test and it fails, or we put the test in the Skipped file, because that was our only choice with ORWT. In the new system, we can mark the test as expected to fail in a particular way, but continue to run it (in order to ensure that the test doesn't get worse and maintaining coverage). Certainly running both systems in parallel for a while and shaking out bugs that the NRWT bots reveal prior to cutting over is a good idea, but I don't know that it's realistic to target all tests passing 100% of the time prior to cutover. Then again, it may be that I'm more used to Chromium bots where we have a large number of tests that aren't expected to pass for one reason or another, and the Apple Mac port will be more stable and easier to converge on. Does that address your concerns? And, just to be clear, I am not presuming to decide when anyone can or should cut over (besides Chromium, of course). It's up to the respective bot owners to decide to reconfigure their bots and switch over if and when they're ready to do so. I'm just trying to make it look appealing :) -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
