No no not on every test result. On every test build "fail". So far I have the following definitions for failures:
- any bin/create script failures - any compilation failures - any device deployment failures (may want to revisit this one) - any device mobile-spec time outs (I.e. Has not finish running the test suite in 5 minutes). All of these get forwarded to a "errors" database. Any mobile-spec test results that complete go to a "results" database. Ci.cordova.io renders the contents of the "results" database. Right now, it doesn't show anything or do anything with errors, but it does subscribe to new errors, so it knows of them. The question is: how to display them? Or, how to notify of them? Email is an easy one. Additionally, errors also have different levels of granularity: - platform - model - model OS version Compilation or bin/create failures are just tagged with a platform (I.e. Not specific to a device). Deployment to a device and specific mobile-spec time-outs ARE specific to a device. So theres a high-level overview of the ci setup :) On 2/22/13 2:34 PM, "Michal Mocny" <[email protected]> wrote: >Chromium try-jobs may re-start a test n-times before pinging people in >order to account for janky tests.. But that means increased hardware >needs. > >-Michal > > >On Fri, Feb 22, 2013 at 5:24 PM, Shazron <[email protected]> wrote: > >> For iOS, this might get too chatty if we send emails to the list since >> every run will produce an error because of the Location permissions >>timeout >> thing that we can't overcome. So if we do N commits a day (assuming >> ci.cordova.io builds on every new commit), we will get N emails. I >>suppose >> we could filter out timeouts as 'warnings' or something (not sure if >>that >> is supported on jasmine) >> >> >> On Fri, Feb 22, 2013 at 12:41 PM, Filip Maj <[email protected]> wrote: >> >> > Sounds like a good idea! I've got related issues set up: >> > >> > - On a timeout for running mobile-spec (which would have caught this >> > error), post an ERROR https://github.com/filmaj/medic/issues/20 >> > - review how errors are shown in the dashboard >> > https://github.com/filmaj/medic/issues/22 >> > >> > Perhaps I add one that the dashboard triggers an e-mail to the dev >>list >> > every time medic experiences an error? >> > >> > On 2/22/13 12:37 PM, "Michal Mocny" <[email protected]> wrote: >> > >> > >+100 >> > > >> > > >> > >On Fri, Feb 22, 2013 at 3:35 PM, Shazron <[email protected]> wrote: >> > > >> > >> I love http://ci.cordova.io :) >> > >> Feature request: e-mail a committer on failure >> > >> >> > >> >> > >> On Fri, Feb 22, 2013 at 12:31 PM, Filip Maj <[email protected]> wrote: >> > >> >> > >> > I was noticing the device wall was running tests on all iOS >>devices >> > >>fine >> > >> > except for the 5.1.1 iPod. >> > >> > >> > >> > Now I need medic to catch this issue on show it up on the >> dashboard.. >> > >> > Always work to do :) >> > >> > >> > >> > On 2/22/13 12:22 PM, "Andrew Grieve" <[email protected]> >>wrote: >> > >> > >> > >> > >Thanks for the catch! (and useful stack trace) >> > >> > > >> > >> > >Looks like I accidentally checked in a change I shouldn't have: >> > >> > > >> > >> > >- [self evalJsHelper:js]; >> > >> > >+ [self evalJsHelper2:js]; >> > >> > > >> > >> > > >> > >> > > >> > >> > >On Fri, Feb 22, 2013 at 3:15 PM, Filip Maj <[email protected]> >>wrote: >> > >> > > >> > >> > >> This is the output I get about 20 or so tests into the full >> jasmine >> > >> unit >> > >> > >> test: >> > >> > >> >> > >> > >> 2013-02-22 12:12:58.748 cordovaExample[4857:3107] bool >> > >> > >> _WebTryThreadLock(bool), 0x2aeb40: Tried to obtain the web >>lock >> > >>from a >> > >> > >> thread other than the main thread or the web thread. This may >>be >> a >> > >> > >>result >> > >> > >> of calling to UIKit from a secondary thread. Crashing now... >> > >> > >> 1 _ZL17_WebTryThreadLockb >> > >> > >> 2 WebThreadLock >> > >> > >> 3 -[UIWebView stringByEvaluatingJavaScriptFromString:] >> > >> > >> 4 -[CDVCommandDelegateImpl evalJsHelper2:] >> > >> > >> 5 -[CDVCommandDelegateImpl sendPluginResult:callbackId:] >> > >> > >> 6 __22-[CDVContacts search:]_block_invoke_2 >> > >> > >> 7 -[CDVAddressBookHelper createAddressBook:] >> > >> > >> 8 __22-[CDVContacts search:]_block_invoke >> > >> > >> 9 _dispatch_call_block_and_release >> > >> > >> 10 _dispatch_worker_thread2 >> > >> > >> 11 _pthread_wqthread >> > >> > >> 12 start_wqthread >> > >> > >> >> > >> > >> >> > >> > >> Passes fine on 6.0.1 devices (both iphone5 and iPod). >> > >> > >> >> > >> > >> Fails on ipod4,1 running 5.1.1. >> > >> > >> >> > >> > >> >> > >> > >> > >> > >> > >> >> > >> > >>
