Revision: 3711
Author: pekka.klarck
Date: Mon Jun 7 01:11:48 2010
Log: documented test template functionality (issue 500)
http://code.google.com/p/robotframework/source/detail?r=3711
Modified:
/trunk/doc/userguide/src/Appendices/AvailableSettings.txt
/trunk/doc/userguide/src/CreatingTestData/CreatingTestCases.txt
=======================================
--- /trunk/doc/userguide/src/Appendices/AvailableSettings.txt Thu May 27
13:27:50 2010
+++ /trunk/doc/userguide/src/Appendices/AvailableSettings.txt Mon Jun 7
01:11:48 2010
@@ -56,14 +56,16 @@
+-----------------+--------------------------------------------------------+
| Test | A synonym for Test
Teardown. |
| Postcondition
| |
+
+-----------------+--------------------------------------------------------+
+ | Test Template | Used for specifying a default `template
keyword`_ |
+ | | for test
cases. |
+-----------------+--------------------------------------------------------+
| Test Timeout | Used for specifying a default `test case
timeout`_. |
+-----------------+--------------------------------------------------------+
-
+
__ `Test suite documentation`_
__ `Documenting resource files`_
-
Test Case table
~~~~~~~~~~~~~~~
@@ -88,10 +90,11 @@
| [Precondition] | A synonym for
[Setup]. |
+-----------------+--------------------------------------------------------+
| [Postcondition] | A synonym for
[Teardown]. |
+
+-----------------+--------------------------------------------------------+
+ | [Template] | Used for specifying a `template
keyword`_. |
+-----------------+--------------------------------------------------------+
| [Timeout] | Used for specifying a `test case
timeout`_. |
+-----------------+--------------------------------------------------------+
-
Keyword table
~~~~~~~~~~~~~
@@ -113,4 +116,3 @@
+-----------------+--------------------------------------------------------+
| [Timeout] | Used for specifying a `user keyword
timeout`_. |
+-----------------+--------------------------------------------------------+
-
=======================================
--- /trunk/doc/userguide/src/CreatingTestData/CreatingTestCases.txt Sun Apr
25 21:39:58 2010
+++ /trunk/doc/userguide/src/CreatingTestData/CreatingTestCases.txt Mon
Jun 7 01:11:48 2010
@@ -9,7 +9,6 @@
:depth: 2
:local:
-
Test case syntax
~~~~~~~~~~~~~~~~
@@ -55,7 +54,6 @@
\ Should Be Equal ${value}
Expected value
================== =========================== ==================
===============
-
Settings in the Test Case table
'''''''''''''''''''''''''''''''
@@ -74,7 +72,11 @@
`[Setup]`:opt:, `[Teardown]`:opt:
Specify `test setup and teardown`_. Have also synonyms
:opt:`[Precondition]` and :opt:`[Postcondition]`,
- respectively.
+ respectively.
+
+`[Template]`:opt:
+ Specifies the `template keyword`_ to use. The test itself will contain
only
+ data to use as arguments to that keyword.
`[Timeout]`:opt:
Used for setting a `test case timeout`_. Timeouts_ are discussed in
@@ -106,7 +108,10 @@
`Test Setup`:opt:, `Test Teardown`:opt:
The default values for `test setup and teardown`_. Have also synonyms
:opt:`Test Precondition` and :opt:`Test Postcondition`,
- respectively.
+ respectively.
+
+`Test Template`:opt:
+ The default `template keyword`_ to use.
`Test Timeout`:opt:
The default value for `test case timeout`_. Timeouts_ are discussed in
@@ -131,7 +136,6 @@
documentation generated by generic documentation tools such as
:prog:`javadoc`.
-
__ `User keyword arguments`_
__ `Keyword arguments`_
@@ -326,7 +330,6 @@
\ Execute command ls ${options} ${path}
============= ================= ===================== ============
============
-
Arguments embedded to keyword names
'''''''''''''''''''''''''''''''''''
@@ -336,7 +339,6 @@
__ `Embedding arguments into keyword name`_
-
Test case name and documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -393,7 +395,6 @@
environment and user information in the last example above, is often
better specified using tags_.
-
Tagging test cases
~~~~~~~~~~~~~~~~~~
@@ -510,7 +511,6 @@
| | Remove Tags | smoke |
req-* |
+---------------+-----------------+---------------------+------------------------+
-
Test setup and teardown
~~~~~~~~~~~~~~~~~~~~~~~
@@ -537,7 +537,6 @@
:opt:`[Setup]` or :opt:`[Teardown]` setting means having no
setup or teardown.
-
.. table:: Test setup and teardown examples
:class: example
@@ -571,7 +570,6 @@
\ [Teardown] ${TEARDOWN} \
================== =============== ===================
==================
-
Often when creating use-case-like test cases, the terms *precondition*
and *postcondition* are preferred over the terms setup and
teardown. Robot Framework supports also this terminology, so that a
@@ -600,6 +598,67 @@
__ `Suite setup and teardown`_
+Test templates
+~~~~~~~~~~~~~~
+
+Test templates convert the normal `keyword-driven`_ test cases into
+`data-driven`_ tests. When the body of the normal test case is
+constructed from keywords and their possible arguments, test cases
+with template only have data that is used as arguments for the
+template keyword. This is illustrated by the following example test
+cases that are functionally fully identical.
+
+.. table:: Using test template
+ :class: example
+
+ =================== =============== ================ ===============
+ Test Case Action Argument Argument
+ =================== =============== ================ ===============
+ Normal test case Example keyword first argument second argument
+ \
+ Templated test case [Template] Example keyword
+ \ first argument second argument
+ =================== =============== ================ ===============
+
+As the example above illustrates, it is possible to specify the
+template for an individual test case using the :opt:`[Template]`
+setting. An alternative approach is using the :opt:`Test Template`
+setting in the Setting table, in which case the template is applied
+for all test cases in that test case file. The :opt:`[Template]`
+setting overrides the possible template set in the Setting table, and
+an empty value for :opt:`[Template]` means that the test has no
+template even when :opt:`Test Template` is used.
+
+If a templated test case has multiple data rows in its body, like in
+the example below, the template is applied for all the rows. This
+means that the same keyword is executed multiple times, once with data
+on each row. Templated tests are also special so that all the rounds
+are executed even if there are failures. It is possible to use this
+kind of `continue on failure`_ mode with normal tests too, but with
+the templated tests the mode is on automatically.
+
+.. table:: Using test template with multiple data rows
+ :class: example
+
+ =================== =============== ================ ===============
+ Test Case Action Argument Argument
+ =================== =============== ================ ===============
+ Templated test case [Template] Example keyword
+ \ first round 1 first round 2
+ \ second round 1 second round 2
+ \ third round 1 third round 2
+ =================== =============== ================ ===============
+
+The main use case for test templates is reducing duplication with
+data-driven tests. Instead of repeating the same keyword with all the
+tests in a file, it is possible to use it only once in the Setting
+table. This usage is illustrated more thoroughly in the next section.
+
+.. note:: Test templates is a new feature in Robot Framework 2.5.
+
+.. note:: It is currently not possibly to specify the template keyword
+ using variables. This limitation may be lifted in the future
+ versions.
Different test case styles
~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -609,62 +668,104 @@
keyword-driven or behavior-driven style. Data-driven style can be used to
test
the same workflow with varying input data.
-
Keyword-driven style
''''''''''''''''''''
-Workflow tests, such as the :name:`Valid Login` test described earlier_,
are
-constructed from several keywords. Their normal structure is that first the
-system is taken into the initial state (:name:`Open Login Page` in the
-:name:`Valid Login` example), then something is done to the system
-(:name:`Input Name`, :name:`Input Password`, :name:`Submit Credentials`),
and
-finally it is verified that the system behaved as expected (:name:`Welcome
Page
-Should Be Open`).
+Workflow tests, such as the :name:`Valid Login` test described
+earlier_, are constructed from several keywords and their possible
+arguments. Their normal structure is that first the system is taken
+into the initial state (:name:`Open Login Page` in the :name:`Valid
+Login` example), then something is done to the system (:name:`Input
+Name`, :name:`Input Password`, :name:`Submit Credentials`), and
+finally it is verified that the system behaved as expected
+(:name:`Welcome Page Should Be Open`).
.. _earlier: example-tests_
-
Data-driven style
'''''''''''''''''
-Another style to write test cases is the *data-driven* approach where test
-cases contain only one higher-level keyword, normally created as `user
-keywords`_. These kinds of tests are very useful when there is a need to
test
-the same workflow with different input or output data. This is
demonstrated by
-the example below.
-
-.. table:: Example test cases using data-driven approach
+Another style to write test cases is the *data-driven* approach where
+test cases use only one higher-level keyword, normally created as a
+`user keyword`_, that hides the actual test workflow. These tests are
+very useful when there is a need to test the same scenario with
+different input and/or output data. It would be possible to repeat the
+same keyword with every test, but the `test template`_ functionality
+allows specifying the keyword to use only once.
+
+.. table:: Data-driven testing example
:class: example
+-------------------+-------------------------+---------+---------+
- | Test Case | |User Name|Password |
+ | Setting | Value | Value | Value |
+===================+=========================+=========+=========+
- | Invalid User Name | Login With Invalid | invalid | mode |
- | | Credentials Should Fail | | |
+ | Test Template | Login with invalid | | |
+ | | credentials should fail | | |
+-------------------+-------------------------+---------+---------+
- | Invalid Password | Login With Invalid | demo | invalid |
- | | Credentials Should Fail | | |
- +-------------------+-------------------------+---------+---------+
- | Invalid User Name | Login With Invalid | invalid | whatever|
- | And Password | Credentials Should Fail | | |
- +-------------------+-------------------------+---------+---------+
- | Empty User Name | Login With Invalid | | mode |
- | | Credentials Should Fail | | |
- +-------------------+-------------------------+---------+---------+
- | Empty Password | Login With Invalid | demo | \\ |
- | | Credentials Should Fail | | |
- +-------------------+-------------------------+---------+---------+
- | Empty User Name | Login With Invalid | | \\ |
- | And Password | Credentials Should Fail | | |
- +-------------------+-------------------------+---------+---------+
-
-Note that in the example above, column headers have been changed to
-match the data. This is possible because on the first row other cells
-except the first one `are ignored`__. Additionally, backslash is used
-for escaping_ empty cells at the end of the table.
-
-__ `Ignored data`_
-
+
+.. table::
+ :class: example
+
+ +-------------------+-----------+-----------+---------+
+ | Test Case | User Name | Password | |
+ +===================+===========+===========+=========+
+ | Invalid User Name | invalid | ${VALID | |
+ | | | PASSWORD} | |
+ +-------------------+-----------+-----------+---------+
+ | Invalid Password | ${VALID | invalid | |
+ | | USER} | | |
+ +-------------------+-----------+-----------+---------+
+ | Invalid User Name | invalid | whatever | |
+ | And Password | | | |
+ +-------------------+-----------+-----------+---------+
+ | Empty User Name | ${EMPTY} | ${VALID | |
+ | | | PASSWORD} | |
+ +-------------------+-----------+-----------+---------+
+ | Empty Password | ${VALID | ${EMPTY} | |
+ | | USER} | | |
+ +-------------------+-----------+-----------+---------+
+ | Empty User Name | ${EMPTY} | ${EMPTY} | |
+ | And Password | | | |
+ +-------------------+-----------+-----------+---------+
+
+The above example has six separate tests, one for each invalid
+user/password combination, and the example below illustrates how to
+have only one test with all the combinations. When using `test
+templates`_, all the rounds in a test are executed even if there are
+failures, so there is no real functional difference between these two
+styles. In the above example separate combinations are named so it is
+easier to see what they test, but having potentially large number of
+these tests may mess-up statistics. Which style to use depends on the
+context and personal preferences.
+
+.. table:: Data-driven test with multiple data variations
+ :class: example
+
+ +-------------------+---------------+-------------------+---------+
+ | Test Case | User Name | Password | |
+ +===================+===============+===================+=========+
+ | Invalid Password | [Template] | Login with invalid| |
+ | | | credentials should| |
+ | | | fail | |
+ +-------------------+---------------+-------------------+---------+
+ | | invalid | ${VALID PASSWORD} | |
+ +-------------------+---------------+-------------------+---------+
+ | | ${VALID USER} | invalid | |
+ +-------------------+---------------+-------------------+---------+
+ | | invalid | whatever | |
+ +-------------------+---------------+-------------------+---------+
+ | | ${EMPTY} | ${VALID PASSWORD} | |
+ +-------------------+---------------+-------------------+---------+
+ | | ${VALID USER} | ${EMPTY} | |
+ +-------------------+---------------+-------------------+---------+
+ | | ${EMPTY} | ${EMPTY} | |
+ +-------------------+---------------+-------------------+---------+
+
+.. tip:: In both of the above examples, column headers have been
+ changed to match the data. This is possible because on the
+ first row other cells except the first one `are ignored`__.
+
+__ `Ignored data`_
Behavior-driven style
'''''''''''''''''''''
@@ -706,13 +807,9 @@
:name:`Welcome page should be open` could also used as :name:`And welcome
page
should be open`.
-.. note:: This feature is supported in Robot Framework 2.1.2 and newer.
-
-
Embedding data to keywords
``````````````````````````
When writing concrete examples it is useful to be able pass actual data to
keyword implementations. User keywords support this by allowing `embedding
arguments into keyword name`_.
-
