Re: [chromium-dev] code coverage on 3 platforms
Are these numbers generated by running all tests, eg unit, ui, browser, interactive .. ? -Scott On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] code coverage on 3 platforms
At this time, just the tests which run successfully on all 3 platforms. The list from chrome_tests.gypi: 'automated_ui_tests', '../app/app.gyp:app_unittests', '../base/base.gyp:base_unittests', '../ipc/ipc.gyp:ipc_tests', '../media/media.gyp:media_unittests', '../net/net.gyp:net_unittests', '../printing/printing.gyp:printing_unittests', 'unit_tests', I did not yet add unit test bundles which don't run on 3 platforms (interactive_ui_tests, browser_tests, courgette_tests). ui_tests runs on 3 platforms but chokes on OSX when run instrumented; I need to track down why. jrg On Mon, Jan 4, 2010 at 8:53 AM, Scott Violet s...@chromium.org wrote: Are these numbers generated by running all tests, eg unit, ui, browser, interactive .. ? -Scott On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] code coverage on 3 platforms
You are looking at older data. Look at numbers from Dec31 or later. I wasn't exaggerating when I said just barely got three platforms working in Q409. The low linux number problem was caused by a switch from scons to make as the default builder on all bots a few weeks ago. There were a number of trailing problems from this switch; for example, buildbot's scripts/master/slaves.py still referenced sconsbuild for any clobber=True configuration, so Linux clobber was broken. I could go on and on here but you probably don't care. In short I've gained a new respect for people who manage to hide the complexity of a build system behind a just works. jrg On Mon, Jan 4, 2010 at 9:07 AM, Mike Pinkerton pinker...@google.com wrote: The linux graphs say they're covering 0.25% of both source and tests. That can't be right, can it? On Mon, Jan 4, 2010 at 11:53 AM, Scott Violet s...@chromium.org wrote: Are these numbers generated by running all tests, eg unit, ui, browser, interactive .. ? -Scott On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Mike Pinkerton Mac Weenie pinker...@google.com -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] code coverage on 3 platforms
Good to see this. I don't see a reference to layout tests. Given that webkit is listed at 9% coverage, I'd guess that these aren't running either. Erik On Mon, Jan 4, 2010 at 10:00 AM, John Grabowski j...@chromium.org wrote: At this time, just the tests which run successfully on all 3 platforms. The list from chrome_tests.gypi: 'automated_ui_tests', '../app/app.gyp:app_unittests', '../base/base.gyp:base_unittests', '../ipc/ipc.gyp:ipc_tests', '../media/media.gyp:media_unittests', '../net/net.gyp:net_unittests', '../printing/printing.gyp:printing_unittests', 'unit_tests', I did not yet add unit test bundles which don't run on 3 platforms (interactive_ui_tests, browser_tests, courgette_tests). ui_tests runs on 3 platforms but chokes on OSX when run instrumented; I need to track down why. jrg On Mon, Jan 4, 2010 at 8:53 AM, Scott Violet s...@chromium.org wrote: Are these numbers generated by running all tests, eg unit, ui, browser, interactive .. ? -Scott On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] code coverage on 3 platforms
You are correct; I am not running layout tests on the coverage bots. Definitely something else to work on. jrg On Mon, Jan 4, 2010 at 3:46 PM, Erik Kay erik...@chromium.org wrote: Good to see this. I don't see a reference to layout tests. Given that webkit is listed at 9% coverage, I'd guess that these aren't running either. Erik On Mon, Jan 4, 2010 at 10:00 AM, John Grabowski j...@chromium.org wrote: At this time, just the tests which run successfully on all 3 platforms. The list from chrome_tests.gypi: 'automated_ui_tests', '../app/app.gyp:app_unittests', '../base/base.gyp:base_unittests', '../ipc/ipc.gyp:ipc_tests', '../media/media.gyp:media_unittests', '../net/net.gyp:net_unittests', '../printing/printing.gyp:printing_unittests', 'unit_tests', I did not yet add unit test bundles which don't run on 3 platforms (interactive_ui_tests, browser_tests, courgette_tests). ui_tests runs on 3 platforms but chokes on OSX when run instrumented; I need to track down why. jrg On Mon, Jan 4, 2010 at 8:53 AM, Scott Violet s...@chromium.org wrote: Are these numbers generated by running all tests, eg unit, ui, browser, interactive .. ? -Scott On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] code coverage on 3 platforms
WOOT! On Thu, Dec 31, 2009 at 9:07 PM, John Grabowski j...@chromium.org wrote: I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] code coverage on 3 platforms
I had an OKR to get code coverage working on 3 platforms for Chromium unit test bundles in Q409. If you're in PST, I made it! Dashboard (overview): http://build.chromium.org/buildbot/perf/dashboard/coverage.html Sample output for Windows: http://build.chromium.org/buildbot/coverage/xp-debug/35420/CHROMIUM/index.html Mac: http://build.chromium.org/buildbot/coverage/mac-debug/35422/CHROMIUM/index.html Linux: http://build.chromium.org/buildbot/coverage/linux-debug/35422/CHROMIUM/index.html Coverage is not perfect. For example, Mac coverage number generation is smart enough to exclude *_win.cc and *_linux.cc but can't tell base/file_version_info.cc and base/wmi_util.cc should be Windows-only. This is a trade-off to prevent a complete exclusion of files without unit tests at all. Lots of thanks to Randall and Bradley. jrg -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
