[chromium-dev] Re: layout test dashboard goofup
I put some more thought into this. Given that we only store a month's worth of data, it's not worth doing backups. Keeping around all the data (maybe a year's worth?) would be awesome though. I actually think that would not be too much work and would add value to the dashboard. At that point, doing backups seems more worthwhile. I'm happy to walk someone through how to make this happen. It really would not be a lot of work if you have a workable knowledge of Python and JS. Ojan On Wed, Oct 14, 2009 at 5:15 PM, Nicolas Sylvain nsylv...@chromium.orgwrote: On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai o...@google.com wrote: The data is stored in a single file per bot. For example, the webkit release bot's results are at http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. That file holds all the historical data for that bot and is copied over during the archive step of each run. We intentionally limit the number of results we keep in that file to 750 runs to keep filesize down. In my accidental checking, I changed 750 to 9. :( A little bit unrelated: This data, along with all the data on build.chromium.org, is replicated on at least 4 machines. It would be easy to recover the data if the server dies for example. We are also planning to do daily backups, but the data is huge. For example, we archive 25GB of new layout test results every day. Nicolas A trivial to implement backup would be to also copy the file to the archive location for just that run (same place as where we copy layout_test_results.zip), e.g. also copy it to http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. The downside is that this uses up disk space (e.g. the largest results.json file was 25mb before being clobbered). Another problem with backing up is that you'd also have to find a way to restore from backup that didn't lose data from runs that happened since the problem occurred. Merging the two files results.json should be pretty relatively trivial code, but it's all code that someone would need to write and test. While it sucks, I don't think backing up this data is worth the effort. It's a temporary productivity hit for the team, but we get enough new data to make reasonable decisions relatively quickly. Mistakes like this are very rare. It will become even more rare as coding work on the dashboard winds down. Feel free to have at it if you disagree. Creates the results.json file and it's content: trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py Copies the results.json file to the right place: trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py Ojan On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow jor...@chromium.orgwrote: I haven't actually gotten anything done on LocalStorage this week because I've been doing so many small side projects like this.but if it's a priority, sure. How about a cron job on some machine that ssh's via a cert into whatever machines the data is stored on, pulls it back, and dumps it into some dir? When we start filling up the hard drive, we can look at doing something smarter, deleting old data, or putting it somewhere like GFS. What server can I use and where's the data stored? On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin e...@chromium.org wrote: Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
I actually have a copy of the data from Tuesday at 2:30pm. If you need any information out of the results page, just let me know. Julie On Thu, Oct 15, 2009 at 10:58 AM, Ojan Vafai o...@google.com wrote: I put some more thought into this. Given that we only store a month's worth of data, it's not worth doing backups. Keeping around all the data (maybe a year's worth?) would be awesome though. I actually think that would not be too much work and would add value to the dashboard. At that point, doing backups seems more worthwhile. I'm happy to walk someone through how to make this happen. It really would not be a lot of work if you have a workable knowledge of Python and JS. Ojan On Wed, Oct 14, 2009 at 5:15 PM, Nicolas Sylvain nsylv...@chromium.orgwrote: On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai o...@google.com wrote: The data is stored in a single file per bot. For example, the webkit release bot's results are at http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. That file holds all the historical data for that bot and is copied over during the archive step of each run. We intentionally limit the number of results we keep in that file to 750 runs to keep filesize down. In my accidental checking, I changed 750 to 9. :( A little bit unrelated: This data, along with all the data on build.chromium.org, is replicated on at least 4 machines. It would be easy to recover the data if the server dies for example. We are also planning to do daily backups, but the data is huge. For example, we archive 25GB of new layout test results every day. Nicolas A trivial to implement backup would be to also copy the file to the archive location for just that run (same place as where we copy layout_test_results.zip), e.g. also copy it to http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. The downside is that this uses up disk space (e.g. the largest results.json file was 25mb before being clobbered). Another problem with backing up is that you'd also have to find a way to restore from backup that didn't lose data from runs that happened since the problem occurred. Merging the two files results.json should be pretty relatively trivial code, but it's all code that someone would need to write and test. While it sucks, I don't think backing up this data is worth the effort. It's a temporary productivity hit for the team, but we get enough new data to make reasonable decisions relatively quickly. Mistakes like this are very rare. It will become even more rare as coding work on the dashboard winds down. Feel free to have at it if you disagree. Creates the results.json file and it's content: trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py Copies the results.json file to the right place: trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py Ojan On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow jor...@chromium.orgwrote: I haven't actually gotten anything done on LocalStorage this week because I've been doing so many small side projects like this.but if it's a priority, sure. How about a cron job on some machine that ssh's via a cert into whatever machines the data is stored on, pulls it back, and dumps it into some dir? When we start filling up the hard drive, we can look at doing something smarter, deleting old data, or putting it somewhere like GFS. What server can I use and where's the data stored? On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin e...@chromium.org wrote: Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
I haven't actually gotten anything done on LocalStorage this week because I've been doing so many small side projects like this.but if it's a priority, sure. How about a cron job on some machine that ssh's via a cert into whatever machines the data is stored on, pulls it back, and dumps it into some dir? When we start filling up the hard drive, we can look at doing something smarter, deleting old data, or putting it somewhere like GFS. What server can I use and where's the data stored? On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin e...@chromium.org wrote: Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
The data is stored in a single file per bot. For example, the webkit release bot's results are at http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. That file holds all the historical data for that bot and is copied over during the archive step of each run. We intentionally limit the number of results we keep in that file to 750 runs to keep filesize down. In my accidental checking, I changed 750 to 9. :( A trivial to implement backup would be to also copy the file to the archive location for just that run (same place as where we copy layout_test_results.zip), e.g. also copy it to http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. The downside is that this uses up disk space (e.g. the largest results.json file was 25mb before being clobbered). Another problem with backing up is that you'd also have to find a way to restore from backup that didn't lose data from runs that happened since the problem occurred. Merging the two files results.json should be pretty relatively trivial code, but it's all code that someone would need to write and test. While it sucks, I don't think backing up this data is worth the effort. It's a temporary productivity hit for the team, but we get enough new data to make reasonable decisions relatively quickly. Mistakes like this are very rare. It will become even more rare as coding work on the dashboard winds down. Feel free to have at it if you disagree. Creates the results.json file and it's content: trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py Copies the results.json file to the right place: trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py Ojan On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow jor...@chromium.org wrote: I haven't actually gotten anything done on LocalStorage this week because I've been doing so many small side projects like this.but if it's a priority, sure. How about a cron job on some machine that ssh's via a cert into whatever machines the data is stored on, pulls it back, and dumps it into some dir? When we start filling up the hard drive, we can look at doing something smarter, deleting old data, or putting it somewhere like GFS. What server can I use and where's the data stored? On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin e...@chromium.org wrote: Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard goofup
On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai o...@google.com wrote: The data is stored in a single file per bot. For example, the webkit release bot's results are at http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. That file holds all the historical data for that bot and is copied over during the archive step of each run. We intentionally limit the number of results we keep in that file to 750 runs to keep filesize down. In my accidental checking, I changed 750 to 9. :( A little bit unrelated: This data, along with all the data on build.chromium.org, is replicated on at least 4 machines. It would be easy to recover the data if the server dies for example. We are also planning to do daily backups, but the data is huge. For example, we archive 25GB of new layout test results every day. Nicolas A trivial to implement backup would be to also copy the file to the archive location for just that run (same place as where we copy layout_test_results.zip), e.g. also copy it to http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. The downside is that this uses up disk space (e.g. the largest results.json file was 25mb before being clobbered). Another problem with backing up is that you'd also have to find a way to restore from backup that didn't lose data from runs that happened since the problem occurred. Merging the two files results.json should be pretty relatively trivial code, but it's all code that someone would need to write and test. While it sucks, I don't think backing up this data is worth the effort. It's a temporary productivity hit for the team, but we get enough new data to make reasonable decisions relatively quickly. Mistakes like this are very rare. It will become even more rare as coding work on the dashboard winds down. Feel free to have at it if you disagree. Creates the results.json file and it's content: trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py Copies the results.json file to the right place: trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py Ojan On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow jor...@chromium.org wrote: I haven't actually gotten anything done on LocalStorage this week because I've been doing so many small side projects like this.but if it's a priority, sure. How about a cron job on some machine that ssh's via a cert into whatever machines the data is stored on, pulls it back, and dumps it into some dir? When we start filling up the hard drive, we can look at doing something smarter, deleting old data, or putting it somewhere like GFS. What server can I use and where's the data stored? On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin e...@chromium.org wrote: Sounds like we've got a volunteer! :D :D :D On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow jor...@chromium.org wrote: I assume we're going to start backing this data up from now on? On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting pkast...@google.com wrote: On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai o...@google.com wrote: I accidentally checked in some test code (one number was wrong!) and clobbered all but 10 of the runs of data for each builder. There's no way to recover it. Do you moonlight for the Danger team at Microsoft? PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
I lied. Another update worth spamming about. The dashboard now by default hides both WONTFIX tests and tests that match their expectations. This way, the dashboard represents a concrete list of work to make tests match their expectations. By default this list should be near-zero. This also makes the dashboard load a ton faster since there's a lot less to render. There are links to show the WONTFIX tests and tests that match their expectations. Like everything else, it modifies your URL, so you can bookmark and copy/paste the URLs if you prefer a different view than the default. Ojan On Wed, Aug 26, 2009 at 5:12 PM, Ojan Vafai o...@chromium.org wrote: One final update. 1. The bogus results on windows are gone. 2. BUG*** now actually links to the bug. 3. Hides WONTFIX tests by default, with a checkbox to show them. 4. Linux release bot is now listed (debug coming soon) 5. The dashboard now shows (highlights in blue) all cases where a test either has expectations that never happen or doesn't have expectations that do happen (see the 'extra' and 'missing' columns). The only work I still plan to do on it is to make it perform a bit better. Please file bugs if there are other additions that would be useful to you. Ojan On Tue, Aug 25, 2009 at 10:15 AM, Ojan Vafai o...@chromium.org wrote: One more thing (since people are asking), you can see all the platforms' expectations for a test from the view for any builder. The ones that don't apply to this builder are greyed out. For example, see LayoutTests/fast/dom/HTMLObjectElement/object-as-frame.html on the WebKit builder. Ojan On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai o...@chrom ium.org wrote: A first version of the layout test flakiness dashboard is up. http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html Some of the key features: - updates roughly as quickly as the bots have run the tests (no post-processing, stdio parsing or crons) - sortable by any column include flakiness - builder + sort order permalinks (i.e. you can copy-paste URLs for a given builder + sort order) - highlights tests with missing expectations (i.e. the test passed, but is marked only as failing) - prints out test run times for tests that took 1 second - shows the expectations for a test on all platforms Taking a look at the dashboard, you can immediately see a couple things: 1. A bunch of cases where we currently have the missing expectations for a test. For example, LayoutTests/accessibility/plugin.html fails on Windows, but is only listed as failing on the Mac. 2. We have a few dozen very slow tests that considerably slow down how long the tests take to run. How do you get this awesomeness for UI tests, unittests, etc? It's actually *really* easy. Someone just needs to add something to the test runners for those test types that spits out JSON files in the right format and copies them to the appropriate place. Let me know if you're interested in adding support for a new test type. Known issues: 1. The JS that generates the page could stand to be faster. 2. The windows builders have a bunch of junk entries with windows style paths. Ignore them for now, they'll gradually fall off the end of the tests tracked by the dashboard. Only the tests with unix style paths should be looked at. If you have bugs or feature requests. Please file them at crbug.com and CC me. One feature request that I have is to also highlight cases where we list an expectation for a test that never happens (e.g. the test is listed as PASS CRASH FAIL, but has never crashed. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
On Tue, Sep 1, 2009 at 5:15 PM, Ojan Vafai o...@chromium.org wrote: I lied. Another update worth spamming about. The dashboard now by default hides both WONTFIX tests and tests that match their expectations. This way, the dashboard represents a concrete list of work to make tests match their expectations. By default this list should be near-zero. It seems like we should show tests whose expectations are flaky (i.e. contain more than one term) as well. After all, most of what we want to do is fix flaky tests, not just fix test expectations. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
On Tue, Sep 1, 2009 at 6:42 PM, Peter Kasting pkast...@google.com wrote: On Tue, Sep 1, 2009 at 5:15 PM, Ojan Vafai o...@chromium.org wrote: I lied. Another update worth spamming about. The dashboard now by default hides both WONTFIX tests and tests that match their expectations. This way, the dashboard represents a concrete list of work to make tests match their expectations. By default this list should be near-zero. It seems like we should show tests whose expectations are flaky (i.e. contain more than one term) as well. After all, most of what we want to do is fix flaky tests, not just fix test expectations. I agree. Already planning to do this as a third option of tests to show/hide, just haven't gotten around to it. I guess we should show them by default. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
One final update. 1. The bogus results on windows are gone. 2. BUG*** now actually links to the bug. 3. Hides WONTFIX tests by default, with a checkbox to show them. 4. Linux release bot is now listed (debug coming soon) 5. The dashboard now shows (highlights in blue) all cases where a test either has expectations that never happen or doesn't have expectations that do happen (see the 'extra' and 'missing' columns). The only work I still plan to do on it is to make it perform a bit better. Please file bugs if there are other additions that would be useful to you. Ojan On Tue, Aug 25, 2009 at 10:15 AM, Ojan Vafai o...@chromium.org wrote: One more thing (since people are asking), you can see all the platforms' expectations for a test from the view for any builder. The ones that don't apply to this builder are greyed out. For example, see LayoutTests/fast/dom/HTMLObjectElement/object-as-frame.html on the WebKit builder. Ojan On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai o...@chrom ium.org wrote: A first version of the layout test flakiness dashboard is up. http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html Some of the key features: - updates roughly as quickly as the bots have run the tests (no post-processing, stdio parsing or crons) - sortable by any column include flakiness - builder + sort order permalinks (i.e. you can copy-paste URLs for a given builder + sort order) - highlights tests with missing expectations (i.e. the test passed, but is marked only as failing) - prints out test run times for tests that took 1 second - shows the expectations for a test on all platforms Taking a look at the dashboard, you can immediately see a couple things: 1. A bunch of cases where we currently have the missing expectations for a test. For example, LayoutTests/accessibility/plugin.html fails on Windows, but is only listed as failing on the Mac. 2. We have a few dozen very slow tests that considerably slow down how long the tests take to run. How do you get this awesomeness for UI tests, unittests, etc? It's actually *really* easy. Someone just needs to add something to the test runners for those test types that spits out JSON files in the right format and copies them to the appropriate place. Let me know if you're interested in adding support for a new test type. Known issues: 1. The JS that generates the page could stand to be faster. 2. The windows builders have a bunch of junk entries with windows style paths. Ignore them for now, they'll gradually fall off the end of the tests tracked by the dashboard. Only the tests with unix style paths should be looked at. If you have bugs or feature requests. Please file them at crbug.com and CC me. One feature request that I have is to also highlight cases where we list an expectation for a test that never happens (e.g. the test is listed as PASS CRASH FAIL, but has never crashed. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
On Wed, Aug 26, 2009 at 5:12 PM, Ojan Vafai o...@chromium.org wrote: The only work I still plan to do on it is to make it perform a bit better. Please file bugs if there are other additions that would be useful to you. This is super-minor, but it might be nice to indicate, for the flakiness column, that the reading order is newer - older. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
One more thing (since people are asking), you can see all the platforms' expectations for a test from the view for any builder. The ones that don't apply to this builder are greyed out. For example, see LayoutTests/fast/dom/HTMLObjectElement/object-as-frame.html on the WebKit builder. Ojan On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai o...@chromium.org wrote: A first version of the layout test flakiness dashboard is up. http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html Some of the key features: - updates roughly as quickly as the bots have run the tests (no post-processing, stdio parsing or crons) - sortable by any column include flakiness - builder + sort order permalinks (i.e. you can copy-paste URLs for a given builder + sort order) - highlights tests with missing expectations (i.e. the test passed, but is marked only as failing) - prints out test run times for tests that took 1 second - shows the expectations for a test on all platforms Taking a look at the dashboard, you can immediately see a couple things: 1. A bunch of cases where we currently have the missing expectations for a test. For example, LayoutTests/accessibility/plugin.html fails on Windows, but is only listed as failing on the Mac. 2. We have a few dozen very slow tests that considerably slow down how long the tests take to run. How do you get this awesomeness for UI tests, unittests, etc? It's actually *really* easy. Someone just needs to add something to the test runners for those test types that spits out JSON files in the right format and copies them to the appropriate place. Let me know if you're interested in adding support for a new test type. Known issues: 1. The JS that generates the page could stand to be faster. 2. The windows builders have a bunch of junk entries with windows style paths. Ignore them for now, they'll gradually fall off the end of the tests tracked by the dashboard. Only the tests with unix style paths should be looked at. If you have bugs or feature requests. Please file them at crbug.com and CC me. One feature request that I have is to also highlight cases where we list an expectation for a test that never happens (e.g. the test is listed as PASS CRASH FAIL, but has never crashed. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafai o...@chromium.org wrote: A first version of the layout test flakiness dashboard is up. http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html AWESOME O RAMA Seriously, this has been critically needed, thanks tons. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: layout test dashboard
Great work, dude. Seriously good stuff. I'll be digging through this tomorrow. Now, how do I change the theme on this thing? ;P :DG On Mon, Aug 24, 2009 at 6:42 PM, Ojan Vafaio...@chromium.org wrote: A first version of the layout test flakiness dashboard is up. http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html Some of the key features: updates roughly as quickly as the bots have run the tests (no post-processing, stdio parsing or crons) sortable by any column include flakiness builder + sort order permalinks (i.e. you can copy-paste URLs for a given builder + sort order) highlights tests with missing expectations (i.e. the test passed, but is marked only as failing) prints out test run times for tests that took 1 second shows the expectations for a test on all platforms Taking a look at the dashboard, you can immediately see a couple things: A bunch of cases where we currently have the missing expectations for a test. For example, LayoutTests/accessibility/plugin.html fails on Windows, but is only listed as failing on the Mac. We have a few dozen very slow tests that considerably slow down how long the tests take to run. How do you get this awesomeness for UI tests, unittests, etc? It's actually *really* easy. Someone just needs to add something to the test runners for those test types that spits out JSON files in the right format and copies them to the appropriate place. Let me know if you're interested in adding support for a new test type. Known issues: The JS that generates the page could stand to be faster. The windows builders have a bunch of junk entries with windows style paths. Ignore them for now, they'll gradually fall off the end of the tests tracked by the dashboard. Only the tests with unix style paths should be looked at. If you have bugs or feature requests. Please file them at crbug.com and CC me. One feature request that I have is to also highlight cases where we list an expectation for a test that never happens (e.g. the test is listed as PASS CRASH FAIL, but has never crashed. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
