Revision: 4252
Author: jussi.ao.malinen
Date: Sun Oct 17 13:17:22 2010
Log: added a new wiki page with examples of better test writing
http://code.google.com/p/robotframework/source/detail?r=4252
Added:
/wiki/HowToWriteGoodTestCasesWithExamples.wiki
=======================================
--- /dev/null
+++ /wiki/HowToWriteGoodTestCasesWithExamples.wiki Sun Oct 17 13:17:22 2010
@@ -0,0 +1,343 @@
+#summary Guidelines for writing good test cases using Robot Framework
+
+<wiki:toc max_depth="2" />
+
+= Introduction =
+
+ * How to write tests in high-level, not how to interact with the SUT
+ * Most important guideline is keeping test cases as easy to read as
possible.
+ * For more information about this subject, you might want to take a look
at Dale H. Emery's great article about writing maintainable test cases:
http://cwd.dhemery.com/2009/11/wmaat/.
+
+
+= Naming =
+
+== Test suite names ==
+
+ * Names should be as describing as possible
+ * Can be relatively long but over 40 characters starts to be too much
+
+== Test case names ==
+
+ * Should be describing similarly as test suite names
+ * If a suite contain many similar tests, their names can be shorter
+<table>
+<tr><th>InvalidLoginShouldFail.txt</th><th>InvalidLoginShouldFail.txt</th></tr>
+<tr><td>
+{{{
+*** Test Cases ***
+Login With Empty Password Should Fail
+Login With Empty Username Should Fail
+Login With Empty Username And Password Should Fail
+Login With Invalid Username Should Fail
+Login With Invalid Password Should Fail
+Login With Invalid Username And Invalid Password Should Fail
+}}}
+</td>
+<td>
+{{{
+*** Test Cases ***
+Empty Password
+Empty Username
+Empty Username And Password
+Invalid Username
+Invalid Password
+Invalid Username And Invalid Password
+}}}
+</td></tr>
+</table>
+
+== Keyword names ==
+
+ * Should be describing and clear
+ * Should explain what a keyword does, not how it does it
+ * Can be on very different abstraction levels
+
+<table>
+<tr><td>
+{{{
+*** Keywords ***
+Input Valid Username and Valid Password And Click Login Button
+}}}
+</td>
+<td>
+{{{
+*** Keywords ***
+Login With Valid Credentials
+}}}
+</td></tr>
+</table>
+
+== Naming setup and teardown ==
+
+ * Try to use name that describes what is done
+ * Possibly an existing keyword
+ * More abstract names acceptable if setup/teardown contain unrelated steps
+ * Listing steps in name is duplication and a maintenance problem (e.g.
`Login to system, add user, activate alarms and check balance`)
+ * May need to use something generic like `Initialize system`
+ * Everyone working with these tests should always understand the name
+
+<table>
+<tr><td>
+{{{
+*** Settings ***
+Test Setup Login to system, add user, activate alarms and check balance
+}}}
+</td>
+<td>
+{{{
+*** Settings ***
+Test Setup Initialize System
+}}}
+</td></tr>
+</table>
+
+= Documentation =
+
+== Test suite documentation ==
+
+ * Often a good idea to add documentation to suites containing tests
+ * Higher level suites don't need documentation that often
+ * Should contain background information, why tests are created, etc.
+ * Don't just repeat test suite name
+ * Better to not have at all if not needed
+ * Don't include too much details about test cases
+ * Tests should be clear enough to
+ * Duplicate information is waste and maintenance problem
+ * Can contain links to more information
+
+<table>
+<tr><th>InvalidLoginShouldFail.txt</th><th>InvalidLoginShouldFail.txt</th></tr>
+<tr><td>
+{{{
+*** Settings ***
+Documentation Checks that invalid login fails
+}}}
+</td>
+<td>
+{{{
+*** Settings ***
+}}}
+</td></tr>
+</table>
+
+
+== Test case documentation ==
+
+ * Test cases normally don't need documentation
+ * Suite's name and documentation and test's name should give enough
background information
+ * Test cases' structure should be clear enough without documentation
or other comments
+ * Sometimes useful so don't be afraid to use it
+
+<table>
+<tr><th>ValidLogin.txt</th><th>ValidLogin.txt</th></tr>
+<tr><td>
+{{{
+*** Test Cases ***
+Valid Login
+ [Documentation] Opens a browser to login url, inputs username
+ ... and password and checks the welcome page is open
+ Open browser ${LOGIN URL} ${BROWSER}
+ Input text u_field ${VALID USERNAME}
+ Input text p_field ${VALID PASSWORD}
+ Click Button button_sbm
+ Title should be Welcome Page
+ [Teardown] Close Browser
+}}}
+</td>
+<td>
+{{{
+*** Test Cases ***
+Valid Login
+ Open login page
+ Input username ${VALID USERNAME}
+ Input password ${VALID PASSWORD}
+ Submit Credentials
+ Welcome page should be open
+ [Teardown] Close Browser
+}}}
+</td></tr>
+</table>
+
+
+== User keyword documentation ==
+
+ * Normally not needed
+ * Good name and clear structure should be enough
+ * Can be used to explain arguments and return values
+ * Visible in resource file documentations generated with libdoc.py
+
+
+= Test suite structure =
+
+ * Test in a suite should be related to each others
+ * Same setup/teardown is a good indicator
+ * Max 10 test cases unless they are data-driven
+ * Tests should be independent
+ * Initialization should be done using setup/teardown
+ * Sometimes dependencies between tests cannot be avoided
+ * It can be, for example, too time taking to initialize all tests
separately
+ * Never have long chains of dependent tests (max 4-5)
+ * Consider verifying the status of the previous test using `${PREV
TEST STATUS}` variable
+
+
+= Test case structure =
+
+ * Test case should be easy to understand
+ * One test case should be testing one thing
+ * _Things_ can be small (part of single feature) or large (end-to-end)
+ * Select suitable abstraction level
+ * Only include information that is relevant for the test case
+ * Two kinds of test cases
+ * Workflow tests
+ * Data-driven tests
+
+== Workflow tests ==
+
+ * Generally has these phases:
+ * Preconditions (optional, often in setup)
+ * Action (do something to the system)
+ * Verification (must have one!)
+ * Cleanup (optional, always in teardown to make sure it is executed)
+ * Keywords describe what test does
+ * Use clear keyword names and suitable abstraction level
+ * Should contain enough information to run manually
+ * Should never need documentation or commenting to explain what the
test does
+ * Different tests can have different abstraction levels
+ * Tests for a detailed functionality are more precise
+ * End-to-end tests can be on very high level
+ * One test should use only one abstraction level
+ * Different styles for different tests
+ * More technical tests for lower level details and integration tests
+ * "Executable specifications" act as requirements
+ * Use domain language
+ * Everyone (including customer/product owner) should always
understand
+ * No complex logic
+ * No for loops or if/else constructs
+ * Use variable assignments with care
+ * Test cases should not look like scripts
+ * Max 10 steps, preferably less
+
+== Data-driven tests ==
+
+ * One high-level keyword per test
+ * Different arguments create different tests
+ * The keyword often contains similar workflow as workflow tests and is
created in the same test case file
+ * Possible, and recommended, to name column headings
+ * If lots of test needed, consider generating them based on a separate
model
+
+{{{
+*** Test Cases *** USERNAME PASSWORD
+Invalid username invalid ${VALID PASSWORD}
+Invalid password ${VALID USERNAME} invalid
+Invalid both invalid invalid
+Empty username ${EMPTY} ${VALID PASSWORD}
+Empty password ${VALID USERNAME} ${EMPTY}
+Empty both ${EMPTY} ${EMPTY}
+
+*** Keywords ***
+Invalid Login
+ [Arguments] ${username} ${password}
+ Input username ${username}
+ Input password ${password}
+ Submit Credentials
+ Error page should be open
+}}}
+
+= User keywords =
+
+ * Should be easy to understand
+ * Same rules as with workflow tests
+ * Can be on different abstraction levels
+ * Can contain some programming logic (for loops, if/else)
+ * Especially on lower level keywords
+ * Complex logic in libraries rather than in user keywords
+
+
+= Variables =
+
+ * Encapsulate long and/or complicated values
+ * Pass information between keywords
+
+== Variable naming ==
+
+ * Clear but not too long names
+ * Can use comments in variable table to document them more
+ * Use case consistently
+ * Lower case with variables only available inside certain test or
keyword
+ * Upper case with others (global, suite or test level)
+ * Both space and underscore can be used as word separator
+ * Recommended to list also variables that are set dynamically in the
variable table
+ * Set typically using `Set Global/Suite/Test Variable` keywords
+ * The initial value should explain where/how the real value is set
+
+{{{
+*** Variables ***
+${LOGIN URL} http://localhost:7272/
+${BROWSER} firefox
+
+*** Keywords ***
+Title Should Begin With
+ [Arguments] ${beginning}
+ ${title} = Get Title
+ Should Start With ${title} ${beginning}
+}}}
+
+== Passing and returning values ==
+
+ * Common approach is to return values from keywords, assign them to
variables and then pass them as arguments to other keywords
+ * Clear and easy to follow approach
+ * Looks like programming
+ * Alternative approach is using `Set Test Variable` keyword
+ * No need to have any programming style in test case level
+ * More complex to follow, reusing keywords harder
+ * Avoid below test case level
+
+<table>
+<tr><th>.txt</th><th>ValidLogin.txt</th></tr>
+<tr><td>
+{{{
+*** Test Cases ***
+Withdraw from account
+ ${status} = Withdraw from account 50$
+ Withdraw should have succeeded ${status}
+
+*** Keywords ***
+Withdraw from account
+ [arguments] ${amount}
+ ${status} = Withdraw from user account ${USER} ${amount}
+ [return] ${status}
+
+Withdraw should have succeeded
+ [arguments] ${status}
+ Should be equal ${status} SUCCESS
+}}}
+</td>
+<td>
+{{{
+*** Test Cases ***
+Withdraw from account
+ Withdraw from account 50$
+ Withdraw should have succeeded
+
+*** Keywords ***
+Withdraw from account
+ [arguments] ${amount}
+ ${status} = Withdraw from user account ${USER} ${amount}
+ Set Test Variable ${status}
+
+Withdraw should have succeeded
+ Should be equal ${status} SUCCESS
+}}}
+</td></tr>
+</table>
+
+= Avoid sleeping =
+
+ * Sleeping is very fragile
+ * Safety margins cause too long sleeps on average
+ * Instead of sleep, use keyword that polls has certain action occurred
+ * Should have a maximum time to wait
+ * Keyword names often starts with `Wait Until ...`
+ * Possible to wrap other keywords inside built-in keyword `Wait Until
Keyword Succeeds`
+ * Sometimes sleeping is easiest but use with care
+ * Never use in keywords that are used often by tests or other keywords
