Added: buildr/site/testing.html URL: http://svn.apache.org/viewvc/buildr/site/testing.html?rev=1448571&view=auto ============================================================================== --- buildr/site/testing.html (added) +++ buildr/site/testing.html Thu Feb 21 10:52:11 2013 @@ -0,0 +1,255 @@ +<html> + <head> + <meta content='text/html;charset=UTF-8' http-equiv='Content-Type'> + <title>buildr — Testing</title> + <style type='text/css'> + @import 'css/default.css'; + @import 'css/syntax.css'; + </style> + <style media='print' type='text/css'> + @import 'css/print.css'; + </style> + <meta content='Official Buildr documentation from the people in the know' name='subject'> + <link href='images/favicon.png' rel='shortcut icon'> + </head> + <body> + <div id='wrap'> + <div id='header'> + <a href='http://buildr.apache.org/' title="Build like you code"><img alt='buildr' src='images/buildr.png'></a> + <div class='tagline'>Build like you code</div> + </div> + <div id='pages'> + <ol class='toc'> + <li>Start Here + <ol class="toc"> + <li><a href='index.html'>Welcome</a></li> + <li><a href='quick_start.html'>Quick Start</a></li> + <li><a href='installing.html'>Installing & Running</a></li> + <li><a href='http://cwiki.apache.org/confluence/display/BUILDR/Index'>Community Wiki</a></li> + </ol> + </li> + <li>Using Buildr + <ol class="toc"> + <li><a href='buildr.pdf'>This Guide (PDF)</a></li> + <li><a href='projects.html'>Projects</a></li> + <li><a href='building.html'>Building</a></li> + <li><a href='artifacts.html'>Artifacts</a></li> + <li><a href='packaging.html'>Packaging</a></li> + <li><a href='testing.html'>Testing</a></li> + <li><a href='releasing.html'>Releasing</a></li> + <li><a href='settings_profiles.html'>Settings/Profiles</a></li> + <li><a href='languages.html'>Languages</a></li> + <li><a href='more_stuff.html'>More Stuff</a></li> + <li><a href='extending.html'>Extending Buildr</a></li> + <li><a href='http://cwiki.apache.org/confluence/display/BUILDR/Buildr+HowTos'>How-Tos</a></li> + </ol> + </li> + <li>Reference + <ol class="toc"> + <li><a href='rdoc'>API</a></li> + <li><a href='http://docs.rubyrake.org'>Rake</a></li> + <li><a href='http://antwrap.rubyforge.org'>Antwrap</a></li> + <li><a href='http://cwiki.apache.org/confluence/display/BUILDR/Common+Problems+and+Solutions'>Troubleshooting</a></li> + </ol> + </li> + <li>Get Involved + <ol class="toc"> + <li><a href='download.html'>Download</a></li> + <li><a href='mailing_lists.html'>Mailing Lists</a></li> + <li><a href='http://www.twitter.com/buildr'>Twitter</a></li> + <li><a href='http://issues.apache.org/jira/browse/Buildr'>Issues/Bugs</a></li> + <li><a href='https://builds.apache.org/view/A-F/view/Buildr'>CI Jobs</a></li> + <li><a href='contributing.html'>Contributing</a></li> + </ol> + </li> + <li> + <form action='http://www.google.com/cse' id='searchbox_003673095760649043902:ogcgrzu0l2y'> + <input name='cx' type='hidden' value='003673095760649043902:ogcgrzu0l2y'> + <input name='cof' type='hidden' value='FORID:0'> + <input name='q' size='20' type='text'> + <input name='sa' type='submit' value='Search'> + <img alt='Google Custom Search' src='http://www.google.com/coop/images/google_custom_search_smnar.gif'> + </form> + </li> + <li>The Buildr Book + <p><a href='http://www.amazon.com/Buildr/dp/1442160942/'><img src='images/1442160941-frontcover.jpg' style='width:120px'></a></p> + <p>Based on the Buildr documentation, available from <a href='http://www.amazon.com/Buildr/dp/1442160942/'>Amazon</a> and + <a href='http://www.createspace.com/3382199'>CreateSpace</a></p> + </li> + </ol> + </div> + <div id='content'> + <h1 id='testing'>Testing</h1> + <ol class="toc"><li><a href="#writing">Writing Tests</a></li><li><a href="#ignoring">Excluding Tests and Ignoring Failures</a></li><li><a href="#running">Running Tests</a></li><li><a href="#integration">Integration Tests</a></li><li><a href="#setup_teardown">Using Setup and Teardown</a></li><li><a href="#checks">Testing Your Build</a></li><li><a href="#bdd">Behaviour-Driven Development</a></li></ol> + <p>Untested code is broken code, so we take testing seriously. Off the bat you get to use either JUnit or TestNG for writing unit tests and integration tests. And you can also add your own framework, or even script tests using Ruby. But first, let’s start with the basics.</p> +<h2 id="writing">Writing Tests</h2> +<p>Each project has a <code>TestTask</code> that you can access using the <code>test</code> method. <code>TestTask</code> reflects on the fact that each project has one task responsible for getting the tests to run and acting on the results. But in fact there are several tasks that do all the work, and a <code>test</code> task coordinates all of that.</p> +<p>The first two tasks to execute are <code>test.compile</code> and <code>test.resources</code>. They work similar to <code>compile</code> and <code>resources</code>, but uses a different set of directories. For example, Java tests compile from the <code>src/test/java</code> directory into the <code>target/test/classes</code> directory, while resources are copied from <code>src/test/resources</code> into <code>target/test/resources</code>.</p> +<p>The <code>test.compile</code> task will run the <code>compile</code> task first, then use the same dependencies to compile the test classes. That much you already assumed. It also adds the test framework (e.g. JUnit, TestNG) and JMock to the dependency list. Less work for you.</p> +<p>If you need more dependencies, the best way to add them is by calling <code>test.with</code>. This method adds dependencies to both <code>compile.dependencies</code> (for compiling) and <code>test.dependencies</code> (for running). You can manage these two dependency lists separately, but using <code>test.with</code> is good enough in more cases.</p> +<p>Once compiled, the <code>test</code> task runs all the tests.</p> +<p>Different languages use different test frameworks. You can find out more about available test frameworks in the <a href="languages.html">Languages</a> section.</p> +<h2 id="ignoring">Excluding Tests and Ignoring Failures</h2> +<p>If you have a lot of tests that are failing or just hanging there collecting dusts, you can tell Buildr to ignore them. You can either tell Buildr to only run specific tests, for example:</p> +<div class="highlight"><pre><code class="ruby"><span class="nb">test</span><span class="o">.</span><span class="n">include</span> <span class="s1">'com.acme.tests.passing.*'</span> +</code></pre> +</div> +<p>Or tell it to exclude specific tests, for example:</p> +<div class="highlight"><pre><code class="ruby"><span class="nb">test</span><span class="o">.</span><span class="n">exclude</span> <span class="s1">'*FailingTest'</span><span class="p">,</span> <span class="s1">'*FailingWorseTest'</span> +</code></pre> +</div> +<p>Note that we’re always using the package qualified class name, and you can use star (<code>*</code>) to substitute for any set of characters.</p> +<p>When tests fail, Buildr fails the <code>test</code> task. This is usually a good thing, but you can also tell Buildr to ignore failures by resetting the <code>:fail_on_failure</code> option:</p> +<div class="highlight"><pre><code class="ruby"><span class="nb">test</span><span class="o">.</span><span class="n">using</span> <span class="ss">:fail_on_failure</span><span class="o">=></span><span class="kp">false</span> +</code></pre> +</div> +<p>Besides giving you a free pass to ignore failures, you can use it for other causes, for example, as a gentle reminder:</p> +<div class="highlight"><pre><code class="ruby"><span class="nb">test</span> <span class="k">do</span> + <span class="nb">warn</span> <span class="s2">"Did you forget something?"</span> <span class="k">if</span> <span class="nb">test</span><span class="o">.</span><span class="n">tests</span><span class="o">.</span><span class="n">nil?</span> <span class="o">||</span> <span class="nb">test</span><span class="o">.</span><span class="n">tests</span><span class="o">.</span><span class="n">empty?</span> +<span class="k">end</span> +</code></pre> +</div> +<p>The <code>tests</code> collection holds the names of all classes with tests, if any. And there’s <code>classes</code>, which holds the names of all test classes. We’ll let you imagine creative use for these two.</p> +<h2 id="running">Running Tests</h2> +<p>It’s a good idea to run tests every time you change the source code, so we wired the <code>build</code> task to run the <code>test</code> task at the end of the build. And conveniently enough, the <code>build</code> task is the default task, so another way to build changes in your code and run your tests:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr +</code></pre> +</div> +<p>That only works with the local <code>build</code> task and any local task that depends on it, like <code>package</code>, <code>install</code> and <code>upload</code>. Each project also has its own <code>build</code> task that does not invoke the <code>test</code> task, so <code>buildr build</code> will run the tests cases, but <code>buildr foo:build</code> will not.</p> +<p>While it’s a good idea to always run your tests, it’s not always possible. There are two ways you can get <code>build</code> to not run the <code>test</code> task. You can set the environment variable <code>test</code> to <code>no</code> (but <code>skip</code> and <code>off</code> will also work). You can do that when running Buildr:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span><span class="o">=</span>no +</code></pre> +</div> +<p>Or set it once in your environment:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span><span class="nb">export </span><span class="nv">TEST</span><span class="o">=</span>no +<span class="nv">$ </span>buildr +</code></pre> +</div> +<p>If you’re feeling really adventurous, you can also disable tests from your Buildfile or <code>buildr.rb</code> file, by setting <code>options.test = false</code>. We didn’t say it’s a good idea, we’re just giving you the option.</p> +<p>The <code>test</code> task is just smart enough to run all the tests it finds, but will accept include/exclude patterns. Often enough you’re only working on one broken test and you only want to run that one test. Better than changing your Buildfile, you can run the <code>test</code> task with a pattern. For example:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:KillerAppTest +</code></pre> +</div> +<p>Buildr will then run only tests that match the pattern <code>KillerAppTest</code>. It uses pattern matching, so <code>test:Foo</code> will run <code>com.acme.FooTest</code> and <code>com.acme.FooBarTest</code>. With Java, you can use this to pick a class name, or a package name to run all tests in that package, or any such combination. In fact, you can specify several patterns separated with commas. For example:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:FooTest,BarTest +</code></pre> +</div> +<p>Buildr forcefully runs all tests that match the pattern. If you want to re-run all tests even if your sources have not changed, you can execute:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:* +</code></pre> +</div> +<p>You can exclude tests by preceeding them with a minus sign (‘-’):</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:-Bar +</code></pre> +</div> +<p>The above would run all tests except those with a name containing <code>Bar</code>. Exclusions can be combined with inclusions:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:Foo,-Bar +</code></pre> +</div> +<p>Buildr would then run tests with names containing <code>Foo</code> but not <code>Bar</code>.</p> +<p>As you probably noticed, Buildr will stop your build at the first test that fails. We think it’s a good idea, except when it’s not. If you’re using a continuous build system, you’ll want a report of all the failed tests without stopping at the first failure. To make that happen, set the environment variable <code>test</code> to “all”, or the Buildr <code>options.test</code> option to <code>:all</code>. For example:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr package <span class="nb">test</span><span class="o">=</span>all +</code></pre> +</div> +<p>We’re using <code>package</code> and not <code>build</code> above. When using a continuous build system, you want to make sure that packages are created, contain the right files, and also run the integration tests.</p> +<p>During development, if you want to re-run only tests that have failed during the last test execution, you can execute:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span>:failed +</code></pre> +</div> +<p>One last note on running tests. By default when you run tests, Buildr will automatically run all transitive test dependencies. This mean if you run “buildr test” inside project <code>bar</code> and <code>bar</code> depends on project <code>foo</code>, Buildr will first run tests in project <code>foo</code> if there have been any changes affecting <code>foo</code> that haven’t been taken into account yet. This behavior often surprises people, especially when they are trying to get things done and only care about tests in <code>bar</code> at that moment. For those times when you’d like to focus your testing on specific projects, Buildr has the <code>only</code> option that will only run tests for projects specified on the command line,</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr <span class="nb">test</span><span class="o">=</span>only bar:test +</code></pre> +</div> +<h2 id="integration">Integration Tests</h2> +<p>So far we talked about unit tests. Unit tests are run in isolation on the specific project they test, in an isolated environment, generally with minimal setup and teardown. You get a sense of that when we told you tests run after the <code>build</code> task, and include JMock in the dependency list.</p> +<p>In contrast, integration tests are run with a number of components, in an environment that resembles production, often with more complicates setup and teardown procedures. In this section we’ll talk about the differences between running unit and integration tests.</p> +<p>You write integration tests much the same way as you write unit tests, using <code>test.compile</code> and <code>test.resources</code>. However, you need to tell Buildr that your tests will execute during integration test. To do so, add the following line in your project definition:</p> +<div class="highlight"><pre><code class="ruby"><span class="nb">test</span><span class="o">.</span><span class="n">using</span> <span class="ss">:integration</span> +</code></pre> +</div> +<p>Typically you’ll use unit tests in projects that create internal modules, such as JARs, and integration tests in projects that create components, such as WARs and EARs. You only need to use the <code>:integration</code> option with the later.</p> +<p>To run integration tests on the current project:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr integration +</code></pre> +</div> +<p>You can also run specific tests cases, for example:</p> +<div class="highlight"><pre><code class="sh"><span class="nv">$ </span>buildr integration:ClientTest +</code></pre> +</div> +<p>If you run the <code>package</code> task (or any task that depends on it, like <code>install</code> and <code>upload</code>), Buildr will first run the <code>build</code> task and all its unit tests, and then create the packages and run the integration tests. That gives you full coverage for your tests and ready to release packages. As with unit tests, you can set the environment variable <code>test</code> to “no” to skip integration tests, or “all” to ignore failures.</p> +<h2 id="setup_teardown">Using Setup and Teardown</h2> +<p>Some tests need you to setup an environment before they run, and tear it down afterwards. The test frameworks (JUnit, TestNG) allow you to do that for each test. Buildr provides two additional mechanisms for dealing with more complicated setup and teardown procedures.</p> +<p>Integration tests run a setup task before the tests, and a teardown task afterwards. You can use this task to setup a Web server for testing your Web components, or a database server for testing persistence. You can access either task by calling <code>integration.setup</code> and <code>integration.teardown</code>. For example:</p> +<div class="highlight"><pre><code class="ruby"><span class="n">integration</span><span class="o">.</span><span class="n">setup</span> <span class="p">{</span> <span class="n">server</span><span class="o">.</span><span class="n">start</span> <span class="p">;</span> <span class="n">server</span><span class="o">.</span><span class="n">deploy</span> <span class="p">}</span> +<span class="n">integration</span><span class="o">.</span><span class="n">teardown</span> <span class="p">{</span> <span class="n">server</span><span class="o">.</span><span class="n">stop</span> <span class="p">}</span> +</code></pre> +</div> +<p>Depending on your build, you may want to enhance the setup/teardown tasks from within a project, for example, to populate the database with data used by that project’s test, or from outside the project definition, for example, to start and stop the Web server.</p> +<p>Likewise, each project has its own setup and teardown tasks that are run before and after tests for that specific project. You can access these tasks using <code>test.setup</code> and <code>test.teardown</code>.</p> +<h2 id="checks">Testing Your Build</h2> +<p>So you got the build running and all the tests pass, binaries are shipping when you find out some glaring omissions. The license file is empty, the localized messages for Japanese are missing, the <span class="caps">CSS</span> files are not where you expect them to be. The fact is, some errors slip by unit and integration tests. So how do we make sure the same mistake doesn’t happen again?</p> +<p>Each project has a <code>check</code> task that runs just after packaging. You can use this task to verify that your build created the files you wanted it to create. And to make it extremely convenient, we introduced the notion of expectations.</p> +<p>You use the <code>check</code> method to express and expectation. Buildr will then run all these expectations against your project, and fail at the first expectation that doesn’t match. An expectation says three things. Let’s look at a few examples:</p> +<div class="highlight"><pre><code class="ruby"><span class="n">check</span> <span class="n">package</span><span class="p">(</span><span class="ss">:war</span><span class="p">),</span> <span class="s1">'should exist'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">exist</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">package</span><span class="p">(</span><span class="ss">:war</span><span class="p">),</span> <span class="s1">'should contain a manifest'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">contain</span><span class="p">(</span><span class="s1">'META-INF/MANIFEST.MF'</span><span class="p">)</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">package</span><span class="p">(</span><span class="ss">:war</span><span class="p">)</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="s1">'WEB-INF'</span><span class="p">),</span> <span class="s1">'should contain files'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should_not</span> <span class="n">be_empty</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">package</span><span class="p">(</span><span class="ss">:war</span><span class="p">)</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="s1">'WEB-INF/classes'</span><span class="p">),</span> <span class="s1">'should contain classes'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">contain</span><span class="p">(</span><span class="s1">'**/*.class'</span><span class="p">)</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">package</span><span class="p">(</span><span class="ss">:war</span><span class="p">)</span><span class="o">.</span><span class="n">entry</span><span class="p">(</span><span class="s1">'META-INF/MANIFEST'</span><span class="p">),</span> <span class="s1">'should have license'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">contain</span><span class="p">(</span><span class="sr">/Copyright (C) 2007/</span><span class="p">)</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">file</span><span class="p">(</span><span class="s1">'target/classes'</span><span class="p">),</span> <span class="s1">'should contain class files'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">contain</span><span class="p">(</span><span class="s1">'**/*.class'</span><span class="p">)</span> +<span class="k">end</span> +<span class="n">check</span> <span class="n">file</span><span class="p">(</span><span class="s1">'target/classes/killerapp/Code.class'</span><span class="p">),</span> <span class="s1">'should exist'</span> <span class="k">do</span> + <span class="n">it</span><span class="o">.</span><span class="n">should</span> <span class="n">exist</span> +<span class="k">end</span> +</code></pre> +</div> +<p>The first argument is the subject, or the project if you skip the first argument. The second argument is the description, optional, but we recommend using it. The method <code>it</code> returns the subject.</p> +<p>You can also write the first expectation like this:</p> +<div class="highlight"><pre><code class="ruby"><span class="n">check</span> <span class="k">do</span> + <span class="n">package</span><span class="p">(</span><span class="ss">:jar</span><span class="p">)</span><span class="o">.</span><span class="n">should</span> <span class="n">exist</span> +<span class="k">end</span> +</code></pre> +</div> +<p>We recommend using the subject and description, they make your build easier to read and maintain, and produce better error messages.</p> +<p>There are two methods you can call on just about any object, called <code>should</code> and <code>should_not</code>. Each method takes an argument, a matcher, and executes that matcher. If the matcher returns false, <code>should</code> fails. You can figure out what <code>should_not</code> does in the same case.</p> +<p>Buildr provides the following matchers:</p> +<table> + <tr> + <th>Method </th> + <th>Checks that … </th> + </tr> + <tr> + <td> <code>exist</code> </td> + <td> Given a file task checks that the file (or directory) exists. </td> + </tr> + <tr> + <td> <code>empty</code> </td> + <td> Given a file task checks that the file (or directory) is empty. </td> + </tr> + <tr> + <td> <code>contain</code> </td> + <td> Given a file task referencing a file, checks its contents, using string or regular expression. For a file task referencing a directory, checks that it contains the specified files; global patterns using <code>*</code> and <code>**</code> are allowed. </td> + </tr> +</table> +<p>All these matchers operate against a file task. If you run them against a ZipTask (including <span class="caps">JAR</span>, <span class="caps">WAR</span>, etc) or a TarTask, they can also check the contents of the archive. And as you can see in the examples above, you can also run them against a path in an archive, checking its contents as if it was a directory, or against an entry in an archive, checking the content of that file.</p> +<p class="note">The <code>package</code> method returns a package task based on packaging type, identifier, group, version and classifier. The last four are inferred, but if you create a package with different specifications (for example, you specify a classifier) your checks must call <code>package</code> with the same qualifying arguments to return the very same package task.</p> +<p>Buildr expectations are based on RSpec. <a href="http://rspec.info/";>RSpec</a> is the behavior-driven development framework we use to test Buildr itself. Check the RSpec documentation if want to see all the supported matchers, or want to write your own.</p> +<h2 id="bdd">Behaviour-Driven Development</h2> +<p>Buildr supports several Behaviour-Driven Development(<span class="caps">BDD</span>) frameworks for testing your projects. Buildr follows each framework naming conventions, searching for files under the <code>src/spec/{lang}</code> directory.</p> +<p>You can learn more about each <span class="caps">BDD</span> framework in the <a href="languages.html">Languages</a> section.</p> +<p>Next, let’s talk about <a href="settings_profiles.html">customizing your environment and using profiles</a></p> + </div> + <div id='footer'>Copyright © 2007-2010 The Apache Software Foundation</div> + </div> + </body> +</html>
