Re: [webkit-dev] WebKit unit test framework
Looks like folks don't have a strong opinion about this topic. I'd recommend just picking something that works and giving it a try. I suspect many different frameworks would all work fine. Adam On Mon, Apr 18, 2011 at 11:48 AM, Dirk Pranke dpra...@chromium.org wrote: I'd add maintained to your list, but otherwise it looks pretty good :) It would also be nice if the testing framework supported parallel or distributed execution, but not many do. -- Dirk On Mon, Apr 18, 2011 at 11:36 AM, David Levin le...@chromium.org wrote: Issue: There has been a long standing bug to add unit tests to WebKit (https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentioned on webkit-dev that it would be helpful in various cases. Landscape: Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: Compatible license with WebKit Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: Easy to write tests Hackable to suit our needs Well tested features (to support hackability/stability) Supports filtering of tests so you can run just the test you care about (and easily listing the tests). Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
I think having something the WebKit community owns and controls is preferred over importing and using a third-party library. So that makes me prefer TestWebKitAPI (or something built from/on it) over gtest. And TestWebKitAPI already has a very simple test for WTF::Vector — just begging to be expanded. On Apr 18, 2011, at 11:36 AM, David Levin wrote: Issue: There has been a long standing bug to add unit tests to WebKit (https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentioned on webkit-dev that it would be helpful in various cases. Landscape: Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: Compatible license with WebKit Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: Easy to write tests Hackable to suit our needs Well tested features (to support hackability/stability) Supports filtering of tests so you can run just the test you care about (and easily listing the tests). Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev — Timothy Hatcher ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
On Wed, Apr 20, 2011 at 3:00 PM, Timothy Hatcher timo...@apple.com wrote: I think having something the WebKit community owns and controls is preferred over importing and using a third-party library. So that makes me prefer TestWebKitAPI (or something built from/on it) over gtest. And TestWebKitAPI already has a very simple test for WTF::Vector — just begging to be expanded. I agree that this sounds attractive at first. However, it feels like there is a bigger picture. There is a lot of work already done in gtest which would be nice to have (-- some of which will likely be necessary). Here's a list right off the top of my head (from what I've seen of both): - TestWebKitAPI doesn't print out values when a test fails. (gtest has this support.) - I don't think there is any documentation for TestWebKitAPI (For, gtest there is http://code.google.com/p/googletest/wiki/Documentation). - I don't there are any test for the framework in TestWebKitAPI -- to be hackable in WebKit with confidence, this is needed. (gtest has extensive testing.) - TestWebKitAPI seems to only run one test at a time. When the ability is added to run all test, it would also be good to add the ability to run a set of test (gtest already has this). - In addition, gtest has a nice output (including a nice color output when supported by the terminal as well as output options which integrate better with automation -- see generating an xml report in http://code.google.com/p/googletest/wiki/AdvancedGuide). - The SetUp/TearDown functionality is a nice way isolate this type of stuff out of the way of tests. - In addition, the design of the api has gone through lots of discussion by concerned parties to work well. (This is more attention that we'd be able to expend on this.) Someone could add these items to TestWebKitAPI eventually and some of these items may never get done due to the cost/benefit ratio of doing them for something just used in WebKit project. In short, it seems to me that the effort to do any of this would be better invested in other places where there isn't already something that works for us. fwiw, we could go with what we do with bugzilla where we start with gtest and people change the code if needed. dave On Apr 18, 2011, at 11:36 AM, David Levin wrote: *Issue: *There has been a long standing bug to add unit tests to WebKit ( https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentionedhttp://lists.macosforge.org/pipermail/webkit-dev/2009-January/006359.htmlon webkit-dev that it would be helpful in various cases. *Landscape:* Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: - Compatible license with WebKit - Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: - Easy to write tests - Hackable to suit our needs - Well tested features (to support hackability/stability) - Supports filtering of tests so you can run just the test you care about (and easily listing the tests). - Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) - Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev — Timothy Hatcher ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
I am really not an expert on testing frameworks, and just put together something that met my needs (as has been the tradition in this project). That said, the only features I like about TestWebKitAPI is that I know how it works and can hack it to do what I want, and that it has the ability to run each test in its own invocation (I also like the color output from the tests, because it's is in color!) So, my questions for people who have used gtest is, Is it hackable? What kind of changes have you had success making? Is a death test as scary as it sounds? -Sam On Apr 18, 2011, at 11:36 AM, David Levin wrote: Issue: There has been a long standing bug to add unit tests to WebKit (https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentioned on webkit-dev that it would be helpful in various cases. Landscape: Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: Compatible license with WebKit Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: Easy to write tests Hackable to suit our needs Well tested features (to support hackability/stability) Supports filtering of tests so you can run just the test you care about (and easily listing the tests). Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
I believe both maruel and jcivelli have had experience contributing changes to gtest. While I wouldn't characterize its code as simple, I haven't had trouble understanding it. It is a fairly mature project, having been used internally at Google for ages. It seems to be fairly well maintained, and the code is clean to my eyes. Chances are good that it already has solutions for much of what you may wish of a unit testing framework. By the way, I was originally not in favor of using gtest for Chromium. It seemed too complicated at first blush. I had created a very simple testing framework that I liked for all the reasons you state below. That was ~5 years ago. However, I quickly became more than convinced that it was worth it to use an established tool for unit testing. It has so many nice features--features I didn't even know I would appreciate. It was also really easy to use. -Darin On Wed, Apr 20, 2011 at 4:01 PM, Sam Weinig wei...@apple.com wrote: I am really not an expert on testing frameworks, and just put together something that met my needs (as has been the tradition in this project). That said, the only features I like about TestWebKitAPI is that I know how it works and can hack it to do what I want, and that it has the ability to run each test in its own invocation (I also like the color output from the tests, because it's is in color!) So, my questions for people who have used gtest is, Is it hackable? What kind of changes have you had success making? Is a death test as scary as it sounds? -Sam On Apr 18, 2011, at 11:36 AM, David Levin wrote: *Issue: *There has been a long standing bug to add unit tests to WebKit ( https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentionedhttp://lists.macosforge.org/pipermail/webkit-dev/2009-January/006359.htmlon webkit-dev that it would be helpful in various cases. *Landscape:* Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: - Compatible license with WebKit - Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: - Easy to write tests - Hackable to suit our needs - Well tested features (to support hackability/stability) - Supports filtering of tests so you can run just the test you care about (and easily listing the tests). - Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) - Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
On Wed, Apr 20, 2011 at 4:01 PM, Sam Weinig wei...@apple.com wrote: Is a death test as scary as it sounds? :) Useful if you want to verify that the program crashes. fwiw, chromium uses this to verify that asserts fire in debug in particular scenarios. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] WebKit unit test framework
On Wed, Apr 20, 2011 at 4:59 PM, Darin Fisher da...@chromium.org wrote: I believe both maruel and jcivelli have had experience contributing changes to gtest. While I wouldn't characterize its code as simple, I haven't had trouble understanding it. It is a fairly mature project, having been used internally at Google for ages. It seems to be fairly well maintained, and the code is clean to my eyes. Chances are good that it already has solutions for much of what you may wish of a unit testing framework. By the way, I was originally not in favor of using gtest for Chromium. It seemed too complicated at first blush. I had created a very simple testing framework that I liked for all the reasons you state below. That was ~5 years ago. However, I quickly became more than convinced that it was worth it to use an established tool for unit testing. It has so many nice features--features I didn't even know I would appreciate. It was also really easy to use. -Darin On Wed, Apr 20, 2011 at 4:01 PM, Sam Weinig wei...@apple.com wrote: So, my questions for people who have used gtest is, Is it hackable? What kind of changes have you had success making? Darin's email said it so well that I hate to follow up with this, but in the interest of full disclosure, I'll include this info even though it may make folks feel less comfortable. (Before Darin's email) I talked to two people from chromium land who did changes to it. I believe those changes were to add support for FLAKY_ and FAILS_. Here's what those two people had to say to me about hacking gtest: Person 1: The codebase was somewhat hackable. The people maintaining it were not welcoming though. Most of the patches I tried to send ended up being R- because it was not important for them. However, they ended up writing a plugin API for it, and with that API it is a lot easier to make the needed improvement to gtest without having to do gtest changes. Person 2: gtest makes heavy use of templates and has many levels of indirection both of which made the code more difficult to deal with. Using it is fine. Hacking on it is definitely harder. In short, if we really need to make changes in gtest itself, it sounds pretty possible to do them. Getting them upstreamed may be harder, but we could always fork if needed. dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] WebKit unit test framework
*Issue: *There has been a long standing bug to add unit tests to WebKit ( https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentionedhttp://lists.macosforge.org/pipermail/webkit-dev/2009-January/006359.htmlon webkit-dev that it would be helpful in various cases. *Landscape:* Surveying WebKit, it is looks like there are at least three testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools), QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI has been used so far (as far as I can tell) for testing core WebKit items like WTF (though I was unaware of TestWebKitAPI until Friday). It seems like a good way to think about the issue of which to use in general in WebKit would be to decide on what would be desired in our framework and then see how each matches up. Here's my take on this. (It may be biased toward what I am familiar with but I welcome others to add their own criteria.) Criteria Musts: - Compatible license with WebKit - Builds/Can be built on the many platforms and build systems supported by WebKit (ideally without extra installs). Useful: - Easy to write tests - Hackable to suit our needs - Well tested features (to support hackability/stability) - Supports filtering of tests so you can run just the test you care about (and easily listing the tests). - Supports writing out values when there is test failure. (For example, if the is verifying that A == B but that is not true, then the values of A and B should be printed.) - Well documented thanks, dave ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
