Hi all, I am getting increasingly close to having new-run-webkit-tests running correctly on the apple mac platform with full feature parity to old-run-webkit-tests. (And, of course, it continues to run on all of the Chromium bots as well). I am hoping to close the remaining issues blocking full parity over the next couple weeks.
You can track the issues where NRWT still falls behind ORWT here: https://bugs.webkit.org/show_bug.cgi?id=34984 Note that I expect some tests to fail or be flaky under NRWT when it is running multiple processes/threads at a time. Doing so can both expose test dependencies on the environment or change the order of tests that are run by a DumpRenderTree, and I don't consider those to be bugs that should stop people from using NRWT. The new tool has a "test_expectations" mechanism that can be used to track expected failures and we should use it (IMO). That said, if there are significant issues making the tool unstable, causing lots of tests to fail, or just missing functionality, now's the time to let me know. Also, for anyone building / maintaining webkit ports other than the chromium and apple ones, I strongly encourage you to take another look at getting NRWT running on your implementations. I hope to at least make some attempt to run some of the GTK and QT bots myself, but I'm not about to sign up to test every build variant personally :) Note that I believe that NRWT is fully usable today and people should seriously start using it. That said, I believe the following bugs should be fixed before we attempt to switch the apple mac port over. You may wish to wait for these fixes before switching your port over as well: 56731 new-run-webkit-tests doesn't support sample-on-timeout 56729 new-run-webkit-tests doesn't support WebKit2 55907 new-run-webkit-tests should upload crash logs 37739 new-run-webkit-tests does not report stderr output 37736 new-run-webkit-tests output is different from (and worse than) original run-webkit-tests There are also a number of issues keeping the apple win port from working at the moment that I plan to work on shortly. There are also a number of bugs currently listed as blocking that I don't think really qualify. Unless told otherwise, I'm plannning to remove the blocking flag from the following on Monday 4/11 (if they haven't been fixed first): 57640 [GTK] overlapping drag&drop tests fail on NRWT 55909 new-run-webkit-tests --run-singly option is busted 55163 new-run-webkit-tests: enable multiple processes by default on Chromium Win 47240 new-run-webkit-tests: getting an "error 2" back from ImageDiff 37426 new-run-webkit-tests should use the ServerProcess abstraction in chromium.py 37007 fast/tokenizer/doctype-search-reset.html fails when run out of order (new-run-webkit-tests) 35359 fast/repaint/renderer-destruction-by-invalidateSelection-crash.html fails intermittently 35266 new-run-webkit-tests --platform=mac-leopard timeout limit should match run-webkit-tests 35049 http/tests/security/cross-frame-access-put.html fails intermittently under new-run-webkit-tests 35006 fast/dom/global-constructors.html is failing based on previous tests Also, just because I don't think they should block a cutover, I do still think they should be fixed, so don't worry about that :) Lastly, over the next couple weeks I also plan to produce some better documentation for both how to use NRWT and how the code itself is structured for anyone that wants to hack on it or port it to new platforms. Comments definitely welcome, -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
