hi,
In a project I am working on there is a separate automation team.
This team gets test descriptions from the manual QA people
including steps how to implement each step:
Test A
Step 1 - Setup a smadge and set it to operation Blue
Step 2 - Setup another smadge and set it to operation Black
Step 3 - Put a table between the two smadges
Step 4 - Launch a zbong from each smadge towards the table at level 10
Step 5 - Check if the table is full of food
Step 5.1 - Check if there are at least 3 types of eggs
Step 5.2 - Check if there are at least 4 types of bread
...
Step 5.19 - Check if there are both blue and black berries
Step 6 - Launch a zbong from each smadge towards the table at level 1
Step 7 - Check if the table is empty now
Steps might have an arbitrary depth of sub steps.
Once I get such description I implement each step. Usually each step breaks
down into several "technical" steps. Such as
Step 1 - Setup a smadge and set it to operation Blue
TS 1 - Setup an empty smadgeholder
TS 2 - Put an empty smadge on the smadgeholder
TS 3 - Copy the blue smadge definition file to the smadgeholder
TS 4 - Load the smadge definition file
...
These technical steps could also have a hierarchy of arbitrary depth of
sub steps. We write functions (later maybe methods) to implement
each step and each "technical step".
The people who run the tests expect to get some HTML report that says
"Test A is successful" though they would also settle if sometimes it says
"Test A failed".
Then they want to be able to click on the link "Test A" and see the Steps
and drill down all the way to the lowest level of technical steps.
It is important for them as they want to "debug" the test runs and
tell if a failure was due to bad test implementation,
temporary network failure, broken smadgeholder, or a real bug in the
implementation of a smadge. Occasionally they event want to execute the
test manually step by step. Having such a drilldown will help them create
the test.
Can I use TAP for this? Can TAP be used to represent such hierarchy?
Is there a module already doing something like this even without the HTML
report?
Am I missing something in the concept?
I thought of creating something I could call MyTest::Hierarchy that would do
$t = MyTest::Hierarchy->new;
$t->start_test("Test A");
$t->start_step("Step 1");
$t->start_ts("TS 1");
...
$t->ok(1); # call to Test::Builder
$t->is_num($received, $expected); # call to Test::Builder
$t->end_ts(); # marks the end of the latest
Technical Step
The TAP representation might be simple indentation of the output
(both the diag messages and the ok 1 prints).
In addition on each ok() call they might want to print several values,
as if there were several titles. I am not sure if an xml output would
not be better than TAP to represent all this.
Gabor
