Hi all,

In the test wg meetings I've mentioned the goals I have for optimizing the 
effort required to develop test documentation and coverage. In summary,

*         Tests should be self-documenting - no "test plan" should be needed 
beyond the entries in the test database and the comments in the tests

*         Tests should include specific (identified by an ID) test assertions, 
which provide all the information necessary to understand the steps of the 
test, beyond a general description. For example, for the test 
vHello_Tacker.sh<https://git.opnfv.org/models/tree/tests/vHello_Tacker.sh> see 
the header below.

*         The test assertions can be managed by some database if that's 
necessary and as effective as a simple flat file. For now, a flat file will do 
and they can be further described as needed on a wiki. See 
test-assertions<https://wiki.opnfv.org/display/models/test-assertions> on the 
Models wiki as an example. With the flat file approach can use simple bash 
scripts to change (by sed etc) the IDs as needed (e.g. as they get renamed, 
split, merged, etc... as typically will happen as tests are developed).

*         Test coverage can be assessed by processing the set of test scripts 
to pull out the referenced assertions, and comparing them to the test assertion 
database. Or we can develop the test coverage map by adding assertion pass/fail 
reports (for the discrete assertions in addition to the overall test) to the 
test results database (recommended).

I'd like to get your feedback on this approach. The bottom line goal is that we 
have test documentation and coverage info with the least development and 
maintenance effort.

# What this is: Deployment test for the Tacker Hello World blueprint.
# Status: work in progress, planned for OPNFV Danube release.
# Use Case Description: A single-node simple python web server, connected to
# two internal networks (private and admin), and accessible via a floating IP.
# Based upon the OpenStack Tacker project's "tosca-vnfd-hello-world" blueprint,
# as extended for testing of more Tacker-supported features as of OpenStack
# Mitaka.
# Pre-State:
# models-joid-001 | models-apex-001 (installation of OPNFV system)
# Test Steps and Assertions:
# 1) bash vHello_Tacker.sh tacker-cli setup|start|run|stop|clean]
#   models-tacker-001 (Tacker installation in a docker container on the 
#   models-nova-001 (Keypair creation)
# 2) bash vHello_Tacker.sh tacker-cli start
#   models-tacker-002 (VNFD creation)
#   models-tacker-003 (VNF creation)
#   models-tacker-vnfd-001 (config_drive creation)
#   models-tacker-vnfd-002 (artifacts creation)
#   models-tacker-vnfd-003 (user_data creation)
#   models-vhello-001 (vHello VNF creation)
# 3) bash vHello_Tacker.sh tacker-cli stop
#   models-tacker-004 (VNF deletion)
#   models-tacker-005 (VNFD deletion)
#   models-tacker-vnfd-004 (artifacts deletion)
# 4) bash vHello_Tacker.sh tacker-cli clean
#   TODO: add assertions
# Post-State:
# After step 1, Tacker is installed and active in a docker container, and the
# test blueprint etc are prepared in a shared virtual folder /tmp/tacker.
# After step 2, the VNF is running and verified.
# After step 3, the VNF is deleted and the system returned to step 1 post-state.
# After step 4, the system returned to test pre-state.
# Cleanup: bash vHello_Tacker.sh tacker-cli clean

Bryan Sullivan | AT&T

opnfv-tech-discuss mailing list

Reply via email to