For web-platform-tests there is an additional issue; tests may be enabled but give a different result in e10s builds compared to non-e10s builds. Therefore compiling a list of disabled web-platform-tests in e10s is insufficient to spot all the differences in this case.
Instead, I recommend using the wptview tool which allows comparing the results of different web-platform-tests runs (or, indeed, any testsuite that uses structured logging). For people who are not interested in e10s specifically, this tool may still be useful when trying to examine the results of web-platform-tests runs. The tool aims at providing filtering of structured log results, based on status, path and test type. It takes one or more testsuite_raw.log files (e.g. wpt_raw.log, mochitest_raw.log) (obtained from Treeherder) and allows us to compare results, (eg: All tests where results from e10s != results from non-e10s build). It reads all tests, (including disabled tests). For example, to compare the W-2 and W-e10s-2 results from a recent run * Load http://jgraham.github.io/wptview/ * From treeherder download the wpt_raw.log artifacts for a W-2 and W-e10s-2 run on a build you are interested in. For the rest of this example I used [1] and [2] (loading from the log URL is on the roadmap). * In the wptview UI click on import, select the log for the W2 run and give the run a name like "W-2". Do the same for the W-e10s-2 log file. Note that importing files is still rather slow. http://ibin.co/2Sjznpoms8BW * Once the 2 logs have been imported, we get a list of all disabled tests in the e10s build, along with their results in the non-e10s build using a "By Result" filter saying "w-e10s-2 is SKIP". A number of test rows are rendered, each showing the test file, subtest (if any) and the Expected / Actual STATUS for each of the two runs. http://ibin.co/2Sk1vaguPyQ8 * If we wish to see all tests having a different status in w-2 and w-e10s-2, we add a filter like "w-2 is not results from w-e10s-2". Once again, a number of rows containing the filtered output are rendered. http://ibin.co/2Sk2R7ngwVWF Wptview is flexible and allows us to have a number of different complicated filters. For a demonstration, To obtain tests under /XMLHttpRequest/ where e10s and non e10s have the same result, and they have a status of ERROR or TIMEOUT, we need three filters. A path filter saying "Path starts with XMLHttpRequest", and two status filters saying "w-2 is results from w-e10s-2" and "w-2 is ERROR or TIMEOUT" (The + symbol adds the OR operator). http://ibin.co/2SlVPLNSvfnW wptview also addresses several other use cases that have hitherto required examining the logs by hand, such as looking for web-platform-tests that CRASH, looking for tests for a specific feature that don't pass, and comparing results from Servo and Gecko. Results are stored locally on an IndexedDB, so one may resume filtering tasks for previously imported logs. Test results could also be cleared using the "CLEAR ALL" button and started afresh. For feature requests and bug reports, please use the github issue tracker at https://github.com/jgraham/wptview/. If you have any questions, please ask me, martianwars, or jgraham, on #ateam. I hope you found this interesting! :) Cheers! [1] - W-2 wpt-raw.log <http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/27983329167951b69f6a451846d0c7e422bb0e589c407141737b40ef21f7a8aaf9347bc57928450fff6e64cc86c72970bb0325dbab258b43284a8008cf3e1885> [2] - W-e10s-2 wpt_raw.log <http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/d4372303775a5dfeb2d1d7c44d89a2b1dcb78d9ee2e7859fd6469b9a7518594a905a313f0bc947196bebc9a95f153804cd5344471126fd833a56d5498aa9cc5b> -- Kalpesh Krishna, Sophomore Undergraduate, Electrical Engineering, IIT Bombay _______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
