Testing process: this is important :) Let's hash that out in this thread, and document over at http://wiki.apache.org/cordova/RunningTests
Here's my version, please add anything that you think I'm missing. For me it's as simple as: 1. Run the qunit tests in mobile-spec. This is a baseline. I have never seen it pass 100% but below 50 tests failing (less than 10%) seems to be what the average across platforms is. Our goal is to reduce this as much as possible. 2. Run the "manual" tests in mobile-spec. These are the individual links/pages showing up when you run mobile-spec. I would like to see the manual tests refined a bit more to have expected behaviour mapped out more obviously. Something like: - you run the accelerometer manual tests. - these tests ask you to lie the device flat on a table and hit a button. We assert that the various x/y/z accel values line up to what we expect. - do the same for various other device positions. - rinse and repeat for all of our APIs. I had Becky express concerns that mobile-spec is not a complete test suite and "needs improvement." Please, if you feel this is the case, it is *extremely* important to commit patches to the test suite! It is our only tool that allows us to say with confidence "we are ok to release." Moving forward, with cordova-js lined up as dropping in the next few releases, as well as the embeddable webview + rename changes coming into the native side on all of our platform implementations, the test suite needs to be our go-to method for determining what works and what doesn't. If you feel it is lacking: patch it. Let's get this process nailed down in our wiki and in this thread so our next release goes smoother. On 12-02-27 1:35 PM, "Simon MacDonald" <simon.macdon...@gmail.com> wrote: >Well running mobile spec is only part of the regression I usually do. >Typically I'll run the autotests with the previous version then again with >the latest version and find the differences. By doing that this release I >was able to fix the issue with the DirectoryReader on Android. Then I'll >run through the manual tests in mobile spec to make sure that things are >still working. Then I start going through the examples on >docs.phonegap.combecause it is really embarrassing when the sample >code in our docs doesn't >run. > >I've been doing this for a number of releases now and it helps me get a >feel for how stable we are. With all the moving parts for 1.5 I don't feel >like we are as stable as 1.4 and don't think we should tag tomorrow as >there are some more things I want to look at. > >Simon Mac Donald >http://hi.im/simonmacdonald > > >On Mon, Feb 27, 2012 at 4:30 PM, Joe Bowser <bows...@gmail.com> wrote: > >> What regression process are you using? I've personally have never seen >> mobile-spec pass on Android, so I'm curious as to what the criteria is. >> >> Joe >> >> On Mon, Feb 27, 2012 at 1:24 PM, Simon MacDonald >> <simon.macdon...@gmail.com>wrote: >> >> > I do not feel comfortable tagging 1.5.0 tomorrow. I'm still finding >>bugs >> in >> > regression. >> > >> > Simon Mac Donald >> > http://hi.im/simonmacdonald >> > >> > >> > On Mon, Feb 27, 2012 at 4:20 PM, Joe Bowser <bows...@gmail.com> wrote: >> > >> > > JIRA says that this is supposed to happen tomorrow. >> > > >> > > On Mon, Feb 27, 2012 at 1:18 PM, Simon MacDonald >> > > <simon.macdon...@gmail.com>wrote: >> > > >> > > > Wait, when are we supposed to be releasing 1.5? >> > > > >> > > > Simon Mac Donald >> > > > http://hi.im/simonmacdonald >> > > > >> > > > >> > > > On Mon, Feb 27, 2012 at 4:10 PM, Steven Gill >><stevengil...@gmail.com >> > >> > > > wrote: >> > > > >> > > > > I haven't pulled the release candidate together yet, so feel >>free >> to >> > > > retag. >> > > > > >> > > > > -Steve >> > > > > >> > > > > On Mon, Feb 27, 2012 at 12:58 PM, Becky Gibson < >> > gibson.be...@gmail.com >> > > > > >wrote: >> > > > > >> > > > > > Has anything been done with the build? I would like to add >>the >> > > > > > <type>Entry.toURL() api and deprecate to.URI() into the >>release >> > > > > candidate. >> > > > > > Is ready to go, just need to retag if this change is to go >>in. >> > > > > > >> > > > > > -becky >> > > > > > >> > > > > > >> > > > > > On Mon, Feb 27, 2012 at 12:35 PM, Becky Gibson < >> > > gibson.be...@gmail.com >> > > > > > >wrote: >> > > > > > >> > > > > > > iOS is tagged for 1.5.0rc1 >> > > > > > > >> > > > > > > >> > > > > > > On Sat, Feb 25, 2012 at 12:31 AM, Simon MacDonald < >> > > > > > > simon.macdon...@gmail.com> wrote: >> > > > > > > >> > > > > > >> One test case was using escapeURL when it should have used >> > > escapeURI >> > > > > so >> > > > > > I >> > > > > > >> fixed it and checked it in. Also, in common-js there was a >>bug >> > in >> > > > the >> > > > > > >> readEntries method where it was referencing this.fullPath >>when >> > it >> > > > > should >> > > > > > >> have been this.path. I fixed that as well. >> > > > > > >> >> > > > > > >> I've regenerated the cordova.android.js file and checked >>it in >> > > along >> > > > > > with >> > > > > > >> a >> > > > > > >> version bump for Android to 1.5.0rc1. >> > > > > > >> >> > > > > > >> Simon Mac Donald >> > > > > > >> http://hi.im/simonmacdonald >> > > > > > >> >> > > > > > >> >> > > > > > >> On Fri, Feb 24, 2012 at 6:53 PM, Joe Bowser < >> bows...@gmail.com> >> > > > > wrote: >> > > > > > >> >> > > > > > >> > Which test case? >> > > > > > >> > >> > > > > > >> > On Fri, Feb 24, 2012 at 12:23 PM, Simon MacDonald < >> > > > > > >> > simon.macdon...@gmail.com >> > > > > > >> > > wrote: >> > > > > > >> > >> > > > > > >> > > I will not be tagging Android today. There are a few >> > > previously >> > > > > > >> passing >> > > > > > >> > > test cases in mobile-spec that are now failing. Will >>tag >> > once >> > > > the >> > > > > > >> problem >> > > > > > >> > > is resolved. >> > > > > > >> > > >> > > > > > >> > > Simon Mac Donald >> > > > > > >> > > http://hi.im/simonmacdonald >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > On Fri, Feb 24, 2012 at 1:54 PM, Steve Gill < >> > > > > stevengil...@gmail.com >> > > > > > > >> > > > > > >> > > wrote: >> > > > > > >> > > >> > > > > > >> > > > Hey guys, >> > > > > > >> > > > >> > > > > > >> > > > Lets tag 1.5.0rc1 today. Post here once your repo is >> > tagged. >> > > > > > >> > > > >> > > > > > >> > > > -Steve >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > On Feb 22, 2012, at 9:31 AM, Brian LeRoux >><b...@brian.io> >> > > wrote: >> > > > > > >> > > > >> > > > > > >> > > > > notes from call this am >> > > > > > >> > > > > >> > > > > > >> > > > > - aim to release 1.5 on tues >> > > > > > >> > > > > - cordova js unlikely for wp, ios, blackberry >> > > > > > >> > > > > - cordova rename from phonegap on track >> > > > > > >> > > > > - coho release tool now does gpg signing: need >>input >> > from >> > > > > > mentors >> > > > > > >> on >> > > > > > >> > > > > cutting official apache release >> > > > > > >> > > > > - plugins are breaking: create a wiki page about >> > updating >> > > > > ---but >> > > > > > >> > > > > plugins have bigger problems which we aim to >>address >> > > > sometime >> > > > > > >> between >> > > > > > >> > > > > now and 2.x >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >>
