I’m 100% down for a test matrix. My idea for CI triggers and schedule is as follows: - on commit to a tool, run its “npm test” - on commit to a platform, run its “npm test” - on commit to a plugin, run its “npm test”, create mobilespec, and run “cordova build” - 2 times a day, run E2E
For E2E, how should we combine released and master code? We have tools, platforms, and plugins, each of which can be either at released version, or from master. Care to start a discuss on this? Kindly, Dmitry > On Jul 13, 2015, at 5:24 PM, Nikhil Khandelwal <[email protected]> wrote: > > Agree to all of this. Currently, our CI system runs all plugin tests on every > commit. Creating a test matrix like you mention below would be a good > improvement. > > Thanks, > Nikhil > > > -----Original Message----- > From: Jesse [mailto:[email protected]] > Sent: Monday, July 13, 2015 2:41 PM > To: [email protected] > Subject: Re: [CI] Medic Update > > This particular issue has been addressed. [1] Ideally a failed test does not > break a build, I think we need to address this in the test framework. I will > look into this. > > I see a couple problems with the current system: > a. All plugins are tested together, meaning a commit to any one of our > multitude of plugins can break everything. > b. Dependencies are NOT tested. Again, We only test ALL the plugins together, > so if pluginA requires pluginB but does not declare the dependency, we are > already installing both, so we don't see an error. > > I think we need better granularity in our testing, so we can see where > something is broken immediately. > 1. A commit to any particular plugin repo should trigger a test of that > plugin for each platform. > 2. A commit to any particular platform should trigger the platforms tests, > and queue all individual plugin tests for that platform. > 3. A commit to lib/cli/plugman should trigger all the individual platform > tests, and queue all individual plugin tests ... > > > [1] > https://git-wip-us.apache.org/repos/asf?p=cordova-plugin-media.git;a=commit;h=5f8738c5267be70562df9714e6f3971790b37d19 > > > My team is hiring! > @purplecabbage > risingj.com > > On Mon, Jul 13, 2015 at 2:24 PM, Nikhil Khandelwal <[email protected]> > wrote: > >> Looks like this commit [1] breaks mobilespec on all platforms. Test >> failure log: >> >> >> https://ci.apache.org/builders/cordova-android-osx/builds/31/steps/run >> ning-tests/logs/stdio >> >> Logs here: >> https://ci.apache.org/builders/cordova-android-osx/builds/31/steps/gat >> hering-logs/logs/stdio >> >> W/PluginManager( 970): THREAD WARNING: exec() call to Media.release >> blocked the main thread for 18ms. Plugin should use >> CordovaInterface.getThreadPool(). >> I/chromium( 970): [INFO:CONSOLE(115)] "Error in Success callbackId: >> Media1993552855 : TypeError: Wrong type for parameter "src" of Media: >> Expected String, but got Undefined.", source: >> file:///android_asset/www/plugins/cordova-plugin-test-framework/www/ma >> in.js >> (115) >> >> Looks like we need to do: new Media("dummy") to prevent failure. We >> really need CI on PR to validate changes like these. >> >> Thanks, >> Nikhil >> >> [1] >> https://github.com/apache/cordova-plugin-media/commit/072656de2a12e6e8 >> deb94ce8ce3fad9eb9ce42a3 >> >> Thanks, >> Nikhil >> >> >> -----Original Message----- >> From: Shazron [mailto:[email protected]] >> Sent: Monday, July 13, 2015 1:28 PM >> To: [email protected] >> Subject: Re: [CI] Medic Update >> >> Since all the builds report the same error "didn't get test results: >> timed out", Perhaps the CouchDB server is not running? >> >> CouchDB server: >> http://cordova-vm.apache.org:5984 >> >> I verified that mobilespec for iOS builds and runs (with failing tests). >> >> >> On Mon, Jul 13, 2015 at 1:01 AM, Dmitry Blotsky >> <[email protected]> >> wrote: >> >>> Hi folks, >>> >>> TL;DR: all builds are failing; please help investigate >>> >>> Last week we had some CI downtime due to the Apache buildbot master >>> machine being rebuilt. Since then, we’ve had some successful runs >>> and some hickups getting resolved. Current status is: >>> - nightly builds now fire unconditionally at 00:01 GMT >>> - changes are still not arriving from GitHub (tracked by INFRA-9979) >>> - status reporting to GitHub is still in progress (tracked by >>> INFRA-9811 and CB-9243) >>> >>> (Unfortunate) current status: >>> - all builds are failing! >>> >>> Let’s get the build back up. Please run mobilespec on your own >>> machines with the latest code of all plugins and help resolve what’s >>> making the builds fail. >>> >>> Kindly, >>> Dmitry >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >>
