Comments inline.
On Thu, Jul 4, 2013 at 3:27 AM, Smith, Peter <pet...@fast.au.fujitsu.com>wrote: > Hi devs, > > > > I would like to understand better about the actual intent of the various > cordova test suites for v2.x. > > > > (This question was already asked to PhoneGap Forum last week but got not > replies) > > > > From what I can work out, the testing of cordova (for Android) is shared > between a number of projects: > > * cordova-js > > * cordova-android > > * cordova-mobile-spec > > correct. > > > Each project has test suites, and my understanding from reading/running > those various suites is as follows: > > > > cordova-js tests > > * ~400 jasmine test case > > * Intent appears to be to exercise various parts of the cordova.js core > > * Looks like focus is for testing framework things: like channels and > modules and message processing etc > > * Specifically, the plugin coverage is not tested much in this suite. I > am guessing it is assumed that plugins are covered by > cordova-mobile-spec? > Correct! Even more: In 3.0, what plugin tests that do exists here will be removed. > > > > codova-mobile-spec test > > * This project is nothing but a test suite > > * Lots of manual/auto tests with fairly clear intent of being 100% the > cordova javascript device APIs > > * 257 auto jasmine tests; 20-ish manual test buttons > > * By our analysis the API coverage is currently something closer to 70%. > > > * Seems strange to not expect 100% success - it risks real bugs falling > through cracks if everybody assumes fail is normal. Apparently this is a > known issue on dev list. > true :( > > * Also includes benchmark test cases (I haven't looked at these - they > are not part of the auto-tests - any documentation on them?) > These are pretty much still a work in progress. They are not run unless someone is working on performance. > > * Some manual tests also: Presumably separated from auto-tests when a > user is required to visually confirm something happened to judge > pass/fail? > exactly. > > > > cordova-android tests > > * Tests are split into manual and JUnit-style tests > > * There are about 20 of each. (Seems very small number - coverage of > Java classes is only about 10%). > > * Most JUnit test cases seem focussed on behaviour of the > CordovaWebView. It looks like lots of Java code does not get touched by > test cases at all. > Yep, we'd like this suite to be far more extensive. > > * There seems no explicit tests for any plugins. Omitted deliberately > assuming cordova-mobile-spec will work through all this code, or to do? > Omitted deliberately I think. > > * There seems no explicit test for the Java-side of the message bridge. > To do? > There are bridge tests in the mobile-spec that test them. > > * Manual tests suffer lots of fails in my environment - so much so that > it's not even clear if these manual tests are expected to work at all, > or are they just a bunch of ad-hoc test fragments which were useful once > but have since fallen on hard times > The intention is that all manual ones are run for each release, and that they should pass. > > > > -- > > > > Please help us to understand the intent of each suite. From what I wrote > above, what is correct/incorrect? What's missing? > > > > Basically we see a variety of test cases (some manual and some > automatic) across a number of projects, and are just trying to > understand how the whole jigsaw all fits together, and therefore how to > judge which bits might be missing. > > > > Then there are more questions regarding testing prior releases: Which of > these test suites actually get run prior to a release? Just the > auto-ones? What is the required pass/error results for the release to > proceed? > > > > Thanks, > > Peter. > >
