apitest is now functional and I've written most basic tests in apitest xml
format.
This includes ssh_user_test, nfs_home tests and more.
Two kind of file are involved:
the .apt files (test files)
and the .apb files (batch files).
The .apt and .apb are all stored in /usr/lib/oscar/testing/wizard_tests/
This can be changed, but if we create sub-directories, then we'll need to use
hard-coded paths in .apt and .apb files (could be achieved in the make install
or in the packaging)
Templates for scripts and batches are already written and available in
/usr/lib/oscar/tests/wizard_tests/
The benefit of this testing method are the following:
- No more test_cluster script that do multiple stuffs like testing ssh,
creating a test user, performing root tests then user tests.
- No more user_tests and root_tests. All are the same, the uid or login name
under which the script must be run is fixed inside the script (qwml parameter).
- No more need to have a hard-coded list of packages with run priorities and
sort functions. Now dependencies are handled from the batch files by apitest.
- Now tests can be run at anytime. Thus at the beginning of each step, some
batches can be run. This is usefull if wizard was stopped, system modified and
wizard restarted. Thus before starting a step, we can recheck that previous
work done is still there and only start the step if a green result is returned.
- those tests can include the system-sanity check.
- unified testing (system-sanity, root_tests, user_tests, install_cluster
builtin tests, ...)
The cons:
- Need to port old opkg test scripts to new format (not that difficult. At
worse a .apt can run the old script just like systemd can run old init scripts.
NOW the status.
- Most if not all non_opkg tests are ported (ssh user test, home mounts, ...)
plus other like ipv6 test, selinux test, ....
- All skeletons for before_step_#.apb have been created. only before_step_0.apb
and before_step_7.apb are functional. (maybe incomplete, but they give a good
idea of what can be the whole stuff).
Some helper scripts have been created to ease writing scripts like:
- create_oscartst.sh => Creates the oscartst user if it doesn't exists.
- oscar_nodes.sh => lists all the nodes (usefull for non perl scripts)
And the TODO:
- Port all opkg testing scripts to apt and apb scripts (as I say, a call to the
old script can be made in the meantime)
=> All tests batches should have at least 2 tests (if relevant): install test
and validation test. The 1st one would test setup and second one would try to
run some simple code and see if it works)
=> usefull for example in ssh test. install test is used in system-sanity
and validation test is used later.
- Add new scripts like c3.apb inside before_step_7.apb (complete cluster setup)
that would depend on all_nodes_up.apt of course. (c3.apb would run
c3_config.apt and c3_validation.apt for example)
- Modify wizard_gui to call apitest -f
/usr/lib/oscar/testing/wizard_tests/before_step_#.apb and only perform the step
code if result is ok (or popup error and not perform the step).
Note: the before_step_9.apb is in fact the test batch run by clicking the
test cluster setup while the before_step_8.apb is the test ran before running
step 8. (which are not the same tests)
- modify /usr/bin/system-sanity to use apitest.
The only remaining question (that can be changed later) is:
the directory structure to hosts the test files.
the choices are:
- The almost flat structure:
- PRO: dependencies are easy to define and a single .apt can be in multiple
batches without the need to hard-code paths
- CONS: even though once written you don't need to look often here, there
will be quite a lot of files. (IMHO, the good solution and similar to
/usr/lib/systemd/system/ were some dedicated sub directory exists.)
- The structure: one directory for each batch
- PRO easy to navigate
- CONS: difficult to write dependancies, may need patching the /usr/bin/opkgc
command so deps are fixed at packaging time. Also difficult to reuse the same
test in two batch scripts. Should it be copied in both sub directories or
linked or using a hard-codded dep?
Sorry for so long reading, but this was to present actual state of oscar
testing.
Best regards,
Olivier.
--
Olivier LAHAYE
CEA DRT/LIST/DM2I/DIR
------------------------------------------------------------------------------
Sponsored by Intel(R) XDK
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk
_______________________________________________
Oscar-devel mailing list
Oscar-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oscar-devel
