Thomas, > > I've been thinking about the way we could use Selenium to run > > unit tests,automating the whole process. The testRunner app is > > great but it has to be run manually. > > > But in general, this could be exactly the app you drive through > Selenium. Would be a matter of "clicking" tree nodes and checking > for the right outcome...
I would prefer something a bit easier to drive, without having to write all the locaters etc which might change with different versions of the test runner and/or qooxdoo. > > We can evaluate arbitrary code using the Selenium assertEval > > command. ... snip ... > Interesting approach. You could in principle pass your unit test > classes code to assertEval and call their methods, but this is > inconvenient since you have to maintain your test code in an HTML > table, rather than a proper class file. Also, the unit test throw > exceptions on failure which might be an issue in an assertEval. I used Selenium IDE just as a short demo of the concept. We're actually writing our Selenium tests in Python which is much more powerful for this sort of thing. > Or, you could mimic the TestRunner and pass code to assertEval that > just instantiates test classes (included from a normal class file) > and runs their "test*" methods, catching exceptions and making > corresponding "outputs" (e.g. <div>'s on the hosting web page). > Those could be easily checked for in your Selenium test script. I want the test classes to just return a result string that can be checked directly in our unit tests. That is much easier to see and adjust. > But now you are already close to the Testrunner itself, so it might > be worth a try to Selenium-control the Testrunner directly, the > biggest challenge probably being to catch and evaluate test results > from the "Test Results" output pane. Again, too much like hard work! > > One advantage of this is that we could add some simple methods to > > our classes to enable the current state to be inspected while > > running our whole application test. Therefore in addition to > > testing what's on the screen we can verify other information > > internal to the application. > > > Every unit test can do that, and many qooxdoo unit tests already do > it. Yes, but I suppose in this case it's a bit more than unit tests - it's providing information in addition to that shown on the screen which is accessible to the testing code. > (Actually, unit tests lend themselves much better to return > values and internal state than GUI updates). Yes, as I said above we would prefer just to verify a result string. > > But also we can then build up and run arbitrary unit tests > > created in our Python environment for running Selenium tests. > > > > However that separates the unit tests from the code which isn't > > very good, > > and it would be difficult to debug the unit tests. > > > > Indeed. If you were to avoid the Testrunner for Selenium tests, I > would rather take my second approach, maintain proper Javascript > unit test classes and drive them through a separate Javascript > layer that replaces the Testrunner framework (instantiating, > invoking, catching exceptions). Sounds like you've got the idea :-). > > If the test runner tests were incorporated in a "make source" > > build (perhaps with "make source-test") then we could invoke them > > from a source version using injected code from Selenium; > > alternatively if they were included in a "make build-test" > > version as well then we could test the build compilation > > (probably quite important given the compression that is > > done). > > Mh, I'm not sure I get the whole picture but I'll throw in a couple > of remarks, see if it rings something with you... I'm imagining that there is a unit test class for every class that I create. There may be multiple tests in each unit test class. A normal "make build" would exclude these unit test classes but I might want to run them in a build environment because there may be compilation errors; or the wrong code included in a variant. So we do a "make build-test" and then the whole application can be driven from Selenium, or if needed, the unit tests can be invoked from Selenium using the Javascript wrapper you were talking about. > > A rough outline of what is needed is: > > > > - test runner testcases and basic stuff incorporated into main > > qooxdoo code tree. > > > The trunk version has qx.dev.unit.*, that probably covers that. OK, I haven't really looked at writing my own unit tests in JS yet so will take your word on that. Though the page on unit tests says that the test class has to be derived from testrunner.TestCase which wouldn't be available in a standard build. I guess it's different in 0.8. > > - normal source and build targets exclude any unit testing code. > > You can still include them if you like. How? I would want to exclude them normally because they will bloat code in a big way. > > - additional make targets "source-test" and "build-test" which > > include the unit testing code but don't invoke it. The user could > > add an arbitrary interface to run them from their UI, using qooxdoo > > variants. > > The new generator will allow you to define arbitrary targets (aka > "jobs" in the new speak) so it will be easy for anybody to produce > various "source" and "build" versions of their app, with as many > classes included as you wish for. I was hoping for a standard build target which included the Javascript wrapper for unit tests as well as any standard unit tests in the code. Unit tests could be specified with a given naming or folder convention: for example in our Python code all unit tests for a given folder are in a sub-folder "unit-test". So we could have abling.util.unit.CRangeManager which is a unit testing class for the abling.util.CRangeManager class. That way it's easy to tell which classes have unit tests and which don't, and provide a report as part of the build process. Or if unit tests are derived from a standard unit testing class they could be put anywhere, I guess. > > - add a standard API for invoking the test cases from Selenium > > more easily. > > Right, we could probably provide boilerplate code that runs > standard qooxdoo unit test classes in a "Selenium-friendly" way > (e.g. with the Simulator project) ... That's a nice thought :). Yep, the wrapper that you mentioned before. > > - "make test-source" and "make test" remain as they are to create > > the testRunner application for manual testing. As above, Selenium > > API created for running any individual test, or group of tests so > > that can be automated as well (maybe just the same as when using > > the application). > > Or run 'make simulator' and have a (possible non-GUI) app generated > that includes app classes, unit test classes and the > Simulator-friendly 'runner' classes. You could run this page > immediately through e.g. Selenium RC, the tests start onload and > the SRC client script checks the output of the <div>s. Doesn't > sound so bad, does it?! I would prefer that the output is not through divs but is returned directly to Selenium as strings. > The important point here is that the unit test classes do all the > real work, interacting with the application classes. Yep. > Ok, that would be my first brain dump so far... Sounds good. I think we're on the same wavelength. Hugh ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ qooxdoo-devel mailing list qooxdoo-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
