There's been significant set of changes to the test expectations lists recently. There seems to still be some confusion over the state of the world. So, hopefully I can clarify a bit. Firstly, there is now a lint option that will parse the test list for each platform (win/mac/linux) and each build target (debug/release). This way you can know that you at least got the syntax right. So, please consider running this when you make test list changes. It takes ~30 seconds on my machine and saves you from closing the tree. Also, if you feel moved to improve our tools a bit, add a pre-commit hook to gcl/git-cl<http://code.google.com/p/chromium/issues/detail?id=5339>to do this for us (please!).
$ ./run_webkit_tests.sh --lint-test-files 081210 11:04:34 run_webkit_tests.py:142 INFO Found: 10244 tests 081210 11:04:59 run_webkit_tests.py:614 INFO SUCCESS As for expecations, the new format is: METADATA : path_to_test = EXPECTATIONS METADATA and EXPECTATIONS are space-separated lists. Tests can have any combination of the following metadata associated with them (optional): SKIP : We don't want to run the test (perhaps because it hangs). DEBUG : Expectations apply only to the Debug build. RELEASE : Expectations apply only to the Release build. LINUX/WIN/MAC: Expectations apply only to the listed platform(s). GOOD EXAMPLES // Test fails for all platforms. LayoutTests/path/to/test.html = FAIL // Test fails for linux only. LINUX : LayoutTests/path/to/test.html = FAIL // Test times out for linux, fails on win-release but crashes on win-debug LINUX : LayoutTests/path/to/test.html = TIMEOUT WIN RELEASE : LayoutTests/path/to/test.html = FAIL WIN DEBUG : LayoutTests/path/to/test.html = CRASH BAD EXAMPLES // Fails on win-release but crashes on win-debug // This ends up defining two sets of expectations for win-debug. WIN : LayoutTests/path/to/test.html = FAIL WIN DEBUG : LayoutTests/path/to/test.html = CRASH // Fails on linux and times out on windows. // This won't throw an error, but loses information. // Now if windows starts failing, or linux starts timing out // we won't notice. LINUX WIN : LayoutTests/path/to/test.html = FAIL TIMEOUT So, as shown above, please use the most specific metadata available to you. If you have reason to believe a test fails on all platforms (e.g. it's a new test from a webkit merge or it started failing from a change to non-platform specific code) feel free to make it apply to all platforms. If there are ways to make any of this simpler, feel free to chip in. :) Ojan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Chromium-dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/chromium-dev?hl=en -~----------~----~----~----~------~----~------~--~---
