In the test wg meetings I've mentioned the goals I have for optimizing the
effort required to develop test documentation and coverage. In summary,
* Tests should be self-documenting - no "test plan" should be needed
beyond the entries in the test database and the comments in the tests
* Tests should include specific (identified by an ID) test assertions,
which provide all the information necessary to understand the steps of the
test, beyond a general description. For example, for the test
the header below.
* The test assertions can be managed by some database if that's
necessary and as effective as a simple flat file. For now, a flat file will do
and they can be further described as needed on a wiki. See
test-assertions<https://wiki.opnfv.org/display/models/test-assertions> on the
Models wiki as an example. With the flat file approach can use simple bash
scripts to change (by sed etc) the IDs as needed (e.g. as they get renamed,
split, merged, etc... as typically will happen as tests are developed).
* Test coverage can be assessed by processing the set of test scripts
to pull out the referenced assertions, and comparing them to the test assertion
database. Or we can develop the test coverage map by adding assertion pass/fail
reports (for the discrete assertions in addition to the overall test) to the
test results database (recommended).
I'd like to get your feedback on this approach. The bottom line goal is that we
have test documentation and coverage info with the least development and
# What this is: Deployment test for the Tacker Hello World blueprint.
# Status: work in progress, planned for OPNFV Danube release.
# Use Case Description: A single-node simple python web server, connected to
# two internal networks (private and admin), and accessible via a floating IP.
# Based upon the OpenStack Tacker project's "tosca-vnfd-hello-world" blueprint,
# as extended for testing of more Tacker-supported features as of OpenStack
# models-joid-001 | models-apex-001 (installation of OPNFV system)
# Test Steps and Assertions:
# 1) bash vHello_Tacker.sh tacker-cli setup|start|run|stop|clean]
# models-tacker-001 (Tacker installation in a docker container on the
# models-nova-001 (Keypair creation)
# 2) bash vHello_Tacker.sh tacker-cli start
# models-tacker-002 (VNFD creation)
# models-tacker-003 (VNF creation)
# models-tacker-vnfd-001 (config_drive creation)
# models-tacker-vnfd-002 (artifacts creation)
# models-tacker-vnfd-003 (user_data creation)
# models-vhello-001 (vHello VNF creation)
# 3) bash vHello_Tacker.sh tacker-cli stop
# models-tacker-004 (VNF deletion)
# models-tacker-005 (VNFD deletion)
# models-tacker-vnfd-004 (artifacts deletion)
# 4) bash vHello_Tacker.sh tacker-cli clean
# TODO: add assertions
# After step 1, Tacker is installed and active in a docker container, and the
# test blueprint etc are prepared in a shared virtual folder /tmp/tacker.
# After step 2, the VNF is running and verified.
# After step 3, the VNF is deleted and the system returned to step 1 post-state.
# After step 4, the system returned to test pre-state.
# Cleanup: bash vHello_Tacker.sh tacker-cli clean
Bryan Sullivan | AT&T
opnfv-tech-discuss mailing list