Re: [whatwg] Cross browser automated tests
On May 20, 2008, at 7:51 PM, Ian Hickson wrote: On Wed, 21 May 2008, Sylvain Pasche wrote: I'm working on an experimental project to build a set of cross browser automated tests. The idea would be to have a repository of browser independent automated tests. Existing test suites could be imported into it. Would WHATWG be interested in such a project? I think that would be great; if you do go ahead with this, you are welcome to use the WHATWG wiki and other resources. I encourage you to use the implementors mailing list for this: http://www.whatwg.org/mailing-list#imps To meet these goals, such an API should be as unobtrusive as possible: * One .js file to include I recommend not embedding any JS, but instead requiring that the following two lines be used to report results (or something like them): if (parent.reportResults) parent.reportResults(...); ...and using iframes to embed the tests one after another. I use something similar for my performance tests: http://www.hixie.ch/tests/adhoc/perf/ This makes the test even easier to maintain, and also makes it a lot easier to reimplement the harness or to share tests between hardnesses. This approach would not be so good for the WebKit regression test harness, which loads pages one at a time in an offscreen window without any use of frames. Our regression test framework can run large numbers of tests significantly faster than any browser-based harness I know of, and it would be nice if any shared pool of regression tests could be compatible with it. WebKit's test setup is actually pretty advanced and can test many things that one might expect to require user interaction or a central web server, such as event handling, back/forward history, window opening, cross-site security, drag and drop, XMLHttpRequest in the face of different server behavior, etc. Regards, Maciej
[whatwg] Cross browser automated tests
Hi, I'm working on an experimental project to build a set of cross browser automated tests. The idea would be to have a repository of browser independent automated tests. Existing test suites could be imported into it. These tests could be run automatically on today's available browsers with the results published. Spec writers or web developers could check what feature is implemented on what browser and browser vendor could use that for their internal regression tests. Would WHATWG be interested in such a project? As a starting point, the testing environment and API would need to be defined. I'm thinking of the following requirements: * Possible for browser vendors to integrate these tests in their testing frameworks * Minimal modifications necessary for migrating existing non automated test cases like the ones on http://www.hixie.ch/tests or http://tc.labs.opera.com/. They should still report PASS/FAIL messages when run in a browser like they do now, which is convenient for testing test cases individually. To meet these goals, such an API should be as unobtrusive as possible: * One .js file to include * A set of functions that can be called for performing the checks and life cycle functions (end of tests, ...) Once this is defined, existing non automated test cases could be modified and imported in the repository. That could be lot of work if I'm looking at all the tests linked from http://wiki.whatwg.org/wiki/Test_cases, but I think there would be lots of benefit. The other big part of the work is to develop the framework for running these tests automatically and reporting the results. I've already done some experiments by running automated tests from Mozilla and WebKit. The results are visible on http://www.browsertests.org/. I started a thread on Mozilla [1] and WebKit lists [2]. There were some interesting discussions with Jeff Walden on the Mozilla list already. Sylvain [1] http://groups.google.com/group/mozilla.dev.quality/browse_thread/thread/b2a959c7547b9877 [2] https://lists.webkit.org/pipermail/webkit-dev/2008-May/003943.html
Re: [whatwg] Cross browser automated tests
Sylvain Pasche wrote: As a starting point, the testing environment and API would need to be defined. I'm thinking of the following requirements: * Possible for browser vendors to integrate these tests in their testing frameworks Hrm, I neglected to mention this on the other thread, but this requirement can't fly. It discourages creating libraries for common testing functions (or even just for non-testing functions, e.g. how Mochitests all have access to the entire MochiKit framework) and using them declaratively (hardcoded script versus dynamic addition with callback schemes), which just makes life harder if you're writing tests. Jeff -- Life would be so much easier if humans had a natural affinity for remembering 128-bit integers.
Re: [whatwg] Cross browser automated tests
On Wed, 21 May 2008, Sylvain Pasche wrote: I'm working on an experimental project to build a set of cross browser automated tests. The idea would be to have a repository of browser independent automated tests. Existing test suites could be imported into it. Would WHATWG be interested in such a project? I think that would be great; if you do go ahead with this, you are welcome to use the WHATWG wiki and other resources. I encourage you to use the implementors mailing list for this: http://www.whatwg.org/mailing-list#imps To meet these goals, such an API should be as unobtrusive as possible: * One .js file to include I recommend not embedding any JS, but instead requiring that the following two lines be used to report results (or something like them): if (parent.reportResults) parent.reportResults(...); ...and using iframes to embed the tests one after another. I use something similar for my performance tests: http://www.hixie.ch/tests/adhoc/perf/ This makes the test even easier to maintain, and also makes it a lot easier to reimplement the harness or to share tests between hardnesses. HTH, -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Cross browser automated tests
Collin, James, and I have been working on cross-browser security testing as well. We've implemented a subset of the WebKit LayoutTestController for Firefox and ran the WebKit security LayoutTests on both Firefox 2 and Firefox 3 RC1. http://crypto.stanford.edu/websec/cross-testing/ There are a number of interesting failures, which we are investigating. On Tue, May 20, 2008 at 7:51 PM, Ian Hickson [EMAIL PROTECTED] wrote: On Wed, 21 May 2008, Sylvain Pasche wrote: To meet these goals, such an API should be as unobtrusive as possible: * One .js file to include I recommend not embedding any JS, but instead requiring that the following two lines be used to report results (or something like them): if (parent.reportResults) parent.reportResults(...); The WebKit LayoutTestController might be a reasonable starting point for a testing API. Just calling parent.reportResults might be limiting for security testing because a frame might not be able to access its parent. For example, the LayoutTestController has a globally readable and writable boolean flag that helps cross-domain frames signal that the test is complete. ...and using iframes to embed the tests one after another. This is also limiting for security tests because some behavior must be tested in a top-level frame. This makes the test even easier to maintain, and also makes it a lot easier to reimplement the harness or to share tests between hardnesses. One of our goals for the project is to integrate with Firefox's testing harness. Ideally, the Firefox source tree could check out the WebKit LayoutTests unmodified and run them as part of their per-checkin regression testing. Adam
