Re: [Wikitech-l] QUnit testing in Jenkins
On Tue, Mar 5, 2013 at 4:12 AM, Krinkle krinklem...@gmail.com wrote: As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins. Great news! I won't go in detail about what PhantomJS is, but in short: It is a headless WebKit browser. Meaning, it doesn't render pixels to a screen on the server side, but it does behave like a fully valid browser environment as if it were rendering it to a screen (CSS is being parsed, the DOM is there, stylesheets are active, retrieving computed styles, ajax requests can be made etc.). For more information, see [2]. PhantomJS can render pixels, but I am not sure if it does it by default, or only when requested. The reason I know it can render pixels is that you can take a screen shot while driving PhantomJS with Selenium. Screenshots from a headless browser. Think about it for a minute. :) Željko ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On Tue, Mar 5, 2013 at 4:12 AM, Krinkle krinklem...@gmail.com wrote: However, unlike php-checkstyle, our QUnit tests are actually passing From console[1]: 02:50:21 Testing http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1f/index.php?title=Special:JavaScriptTest/qunitExceptionthrown by test.module1: expected 02:50:21 Error: expected 02:50:24 OK 02:50:24 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors. This is strange. The console says there was a problem: Exception thrown by test.module1: expected and Error: expected But then it says everything is fine: Done, without errors. Željko -- [1] https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
From console[1]: 02:50:21 Testing http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1f/index.php?title=Special:JavaScriptTest/qunitExceptionthrown by test.module1: expected 02:50:21 Error: expected 02:50:24 OK 02:50:24 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors. This is strange. The console says there was a problem: Exception thrown by test.module1: expected and Error: expected But then it says everything is fine: Done, without errors. Željko -- [1] https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console I'm not sure how to navigate to the source that defines that test but I suspect it's just using an expected exception test: http://docs.jquery.com/QUnit/raises ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On Mar 5, 2013, at 3:49 PM, Dan Andreescu dandree...@wikimedia.org wrote: From console[1]: 02:50:21 Testing http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1f/index.php?title=Special:JavaScriptTest/qunitExceptionthrown by test.module1: expected 02:50:21 Error: expected 02:50:24 OK 02:50:24 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors. This is strange. The console says there was a problem: Exception thrown by test.module1: expected and Error: expected But then it says everything is fine: Done, without errors. Željko -- [1] https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console I'm not sure how to navigate to the source that defines that test but I suspect it's just using an expected exception test: http://docs.jquery.com/QUnit/raises No, it is neither. Remember you're looking at the console log. Which, in this case is being written two from lots of sources: * jenkins - stdout * grunt/qunit - stdout * phantomjs - console.log() The part cited here is mostly qunit's output (the dotted progress line), but when calling console.log from within the javascript it is forwarded to the jenkins console. A console log is harmless and no reason for alarm. If it were an actual exception, it wouldn't be tolerated. In this case it is coming from the unit tests that asserts that mw.loader sets a module's state to error if it's javascript bundle throws an exception. mw.loader executes the bundle in a try/catch and logs any exceptions to the console after which it sets state=error and triggers the error callbacks. Right now grunt/qunit forwards the phantomjs console straight to the main output. It would be prettier if it would display it in a way that more clearly says phantomjs console. I requested this 3 months ago: https://github.com/gruntjs/grunt-contrib-qunit/pull/6 Execute the tests in your own browser and you'll see the same data in your browser's console (e.g. Chrome Dev Tools). -- Krinkle ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On Mon, Mar 4, 2013 at 8:07 PM, Ori Livneh o...@wikimedia.org wrote: Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed. This is fantastic. Thanks, Timo. Indeed - this is a great milestone. Thanks for all your work getting this out the door, Timo! :-) Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On 03/04/2013 07:12 PM, Krinkle wrote: Things this will catch are basically everything else. Any runtime error that we can't detect in static analysis but will fail no matter what browser you're in, such as: * misspelled identifiers or syntax errors * issues with ResourceLoader (mw.loader) * issues with AJAX * any code failures that result in exceptions * the obvious (catching failures/regressions in our QUnit tests) This is really a great step forward. Thanks, Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] QUnit testing in Jenkins
Hey all, As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins. Example: * https://gerrit.wikimedia.org/r/52177 * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/ * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed. It is run by using PhantomJS[2] and we're using grunt-contrib-qunit[3][4] to abstract the logic: * starting phantomjs * pointing it to a url * hooking into javascript engine to register callbacks to QUnit events * progress indicator in cli * return the proper exit code I won't go in detail about what PhantomJS is, but in short: It is a headless WebKit browser. Meaning, it doesn't render pixels to a screen on the server side, but it does behave like a fully valid browser environment as if it were rendering it to a screen (CSS is being parsed, the DOM is there, stylesheets are active, retrieving computed styles, ajax requests can be made etc.). For more information, see [2]. Just to point out the obvious, this doesn't catch issues specific to certain browsers (e.g. syntax only breaking in older EcmaScript 3 engines, or code that incorrectly relies on HTML5 DOM APIs that exist in latest WebKit but not in Internet Explorer or Firefox). Those we will catch once we also run this on multiple operating systems and in more than 1 browser (which is the next chapter in implementing the continuous integration workflow[5]). Things this will catch are basically everything else. Any runtime error that we can't detect in static analysis but will fail no matter what browser you're in, such as: * misspelled identifiers or syntax errors * issues with ResourceLoader (mw.loader) * issues with AJAX * any code failures that result in exceptions * the obvious (catching failures/regressions in our QUnit tests) Even code that doesn't have unit tests. The code execution alone should result an uncaught exception, which we can now get our hands on since we actually execute the javascript in a real browser. This includes anything related to ResourceLoader, since we don't just execute the unit tests in a browser, we load them from MediaWiki's core/index.php entry point (Special:JavaScriptTest, to be specific). Similar to how we have the php-checkstyle job currently, the QUnit job is in non-voting mode. However, unlike php-checkstyle, our QUnit tests are actually passing, but we're not letting it vote yet to see how it behaves over the next 24 hours. If it is stable, we'll make it voting (like phplint, jshint and phpunit are already). So, next time you read the jenkins-job comment, look for the QUnit job. Happy testing, -- Krinkle [1] https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing [2] PhantomJS: http://phantomjs.org/ [3] node-phantomjs (npm wrapper with nom-install hook) https://github.com/Obvious/phantomjs [4] grunt-contrib-qunit https://github.com/gruntjs/grunt-contrib-qunit [5] https://www.mediawiki.org/wiki/Continuous_integration/Workflow_specification ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On Mar 4, 2013 7:13 PM, Krinkle krinklem...@gmail.com wrote: Hey all, As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins. Awesome, thanks Krinkle! Greg ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
On Monday, March 4, 2013 at 7:12 PM, Krinkle wrote: Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed. This is fantastic. Thanks, Timo. -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] QUnit testing in Jenkins
Le 04/03/13 19:12, Krinkle wrote: As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins. Congratulations Timo on this huge improvement to the continuous integration system \O/ -- Antoine hashar Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
