On 14 October 2015 at 08:41, Christophe Milard <[email protected] > wrote:
> > > On 13 October 2015 at 18:41, Stuart Haslam <[email protected]> > wrote: > >> Update sections describing how a specific platform may skip tests by >> marking them as inactive. >> >> Signed-off-by: Stuart Haslam <[email protected]> >> --- >> doc/implementers-guide/implementers-guide.adoc | 104 >> +++++++++++++++++-------- >> 1 file changed, 71 insertions(+), 33 deletions(-) >> >> diff --git a/doc/implementers-guide/implementers-guide.adoc >> b/doc/implementers-guide/implementers-guide.adoc >> index 090d4e5..0c11079 100644 >> --- a/doc/implementers-guide/implementers-guide.adoc >> +++ b/doc/implementers-guide/implementers-guide.adoc >> @@ -1,4 +1,4 @@ >> -OpenDataPlane (ODP) Implementers-Guide >> +OpenDataPlane (ODP) Implementers Guide >> ======================================= >> :toc: >> >> @@ -70,12 +70,12 @@ This grouping defines tests that are expected to be >> executable and succeed on an >> They are written in plain C code, and may only use functions defined in >> the standard libC library (besides the ODP functions being tested, of >> course). >> No other languages (like scripting) are allowed as their usage would >> make assumptions on the platform capability. >> >> -This area is located at: '<ODP_ROOT>/test/validation/' >> +This area is located at: 'test/validation/' >> >> The ODP API itself is ordered by module, where each module groups the >> set of ODP API functions related to the same "topic". >> Examples of modules includes "classification" (API functions dealing >> with ingres packets classification), time (functions dealing with time, >> excluding timers which have their own module), timer,... >> The complete module list can be seen at: >> http://docs.opendataplane.org/linux-generic-doxygen-html/modules.html[ODP >> Modules] + >> -Within the platform agnostic area, the tests are also grouped by >> modules, matching the ODP API modules: '<ODP_ROOT>/test/validation/' mainly >> contains a list of directories matching each module name (as defined by the >> doxygen "@defgroup" or "@ingroup" statement present in each API ".h" file). >> +Within the platform agnostic area, the tests are also grouped by >> modules, matching the ODP API modules: 'test/validation/' mainly contains a >> list of directories matching each module name (as defined by the doxygen >> "@defgroup" or "@ingroup" statement present in each API ".h" file). >> >> Within each of these directories, a library (called >> "libtest<module>.la") and its associated ".h" file (called "<module>.h") >> defines all the test functions for this module as well as few other >> functions to initialize, terminate, and group the tests. >> An executable called "<module>_main*", is also built. It is permissible >> to generate more than one executable to cover the functionality in the test >> library for the module. >> @@ -87,38 +87,36 @@ The obvious illustration of this is for module "init" >> whose functions are requir >> >> There is a "Makefile.am" located at the top of the platform agnostic >> area. Its role is limited to the construction of the different test >> libraries and the "<module>_main*" executables. No tests are run from this >> area when "make check" is performed. >> >> -A convenience library '.../test/validation/libodptests.la' (and its >> associated .h file, '.../test/validation/odptest.h') regrouping all tests >> symbols of all modules may be built in the future. (The superlib) >> - >> -C_UNIT >> +CUnit >> ^^^^^^ >> -Within a given test executable C_UNIT is used to run the different >> tests. The usage of C_UNIT implies the following structure: >> +Within a given test executable CUnit is used to run the different tests. >> The usage of CUnit implies the following structure: >> >> * Tests are simple C functions. >> -* Tests are grouped in arrays called test suites. Each test suite can be >> associated with a suite initialization/termination function(s), called by >> C_UNIT before and after the whole suite is ran. >> -* An array of test suites (and associated init/term functions) defines >> the test registry ran by the test executable. >> +* Tests are grouped in arrays called test suites. Each test suite can be >> associated with a suite initialization/termination function(s), called by >> CUnit before and after the whole suite is run. >> +* An array of test suites (and associated init/term functions) defines >> the test registry run by the test executable. >> >> -Moreover, two extra functions can be used to initialize/terminate the >> test executable (these are not part of C_UNIT). + >> -A test executable return success (0) if every tests of each suite >> succeed. >> +Moreover, two extra functions can be used to initialize/terminate the >> test executable (these are not part of CUnit). + >> +A test executable return success (0) if every test of each suite succeed. >> >> -More details about http://cunit.sourceforge.net/doc/index.html[C_Unit >> users guide] >> +More details about http://cunit.sourceforge.net/doc/index.html[CUnit >> users guide] >> >> [[anchor-1]] >> Module test and naming convention >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> >> -* Tests, i.e. C functions which are used in CUNIT test suites are named: >> +* Tests, i.e. C functions which are used in CUnit test suites are named: >> *<Module>_test_+++*+++* + >> where the suffix idendify the test. >> >> -* Test arrays, i.e. arrays of CU_TestInfo, listing the test functions >> belonging to a suite, are called: >> +* Test arrays, i.e. arrays of odp_testinfo_t, listing the test functions >> belonging to a suite, are called: >> *<Module>_suite+++[_*]+++* + >> where the possible suffix can be used if many suites are declared. >> >> -* CUNIT suite init and termination functions are called: >> +* CUnit suite init and termination functions are called: >> *<Module>+++_suite[_*]_init()+++* and >> *<Module>+++_suite[_*]_term()+++* respectively. + >> where the possible extra middle pattern can be used if many suites >> are declared. >> >> -* Suite arrays, i.e. arrays of CU_SuiteInfo used in executables (C_UNIT >> registry) are called: >> +* Suite arrays, i.e. arrays of odp_suiteinfo_t used in executables >> (CUnit registry) are called: >> *<Module>+++_suites[_*]+++* + >> where the possible suffix identifies the executable using it, if many. >> >> @@ -134,14 +132,14 @@ All the above symbols are part of the generated >> libtest<Module>.la libraries. Th >> >> Platform specific >> ~~~~~~~~~~~~~~~~~ >> -These tests are located under '<ODP_ROOT>/platform/<platform>/test'. >> There is one such area for each platform implementing ODP. >> +These tests are located under 'platform/<platform>/test'. There is one >> such area for each platform implementing ODP. >> This location will be referred as <PLATFORM_SPECIFIC> in the rest of >> this document. >> >> The normal case >> ^^^^^^^^^^^^^^^ >> -If the considered platform needs nothing specific to be tested this >> directory will contain a single "Makefile.am". >> -This "Makefile.am" then only lists the executables to be run on "make >> check" (in the automake TEST variable): when the platform has nothing >> specific to it, this just list the "<module>_main+++[_*]+++" executables, >> picked from the platform agnostic area. >> -For the linux-generic platform, most tested modules fall into this >> category: currently, the >> '<ODP_ROOT>/platform/linux-generic/test/Makefile.am' looks as follows: >> +If the considered platform needs no platform specific tests, this >> directory simply needs to contain a single Makefile.am listing each of the >> executables (named <module>_main) built from the platform agnostic area. >> The executables are listed in the automake TEST variable and will therefore >> be run on "make check". >> + >> +For the linux-generic platform, most tested modules fall into this >> category: currently, the 'platform/linux-generic/test/Makefile.am' looks as >> follows: >> >> [source,am] >> ---- >> @@ -179,13 +177,11 @@ endif >> >> With the exception for module pktio, all other modules testing just >> involves calling the platform agnostic <module>_main executables (in >> test/validation). >> >> -When no platform specific testing is required, the >> '<PLATFORM_SPECIFIC>/Makefile.am' is used to list the tests executables to >> be run only, as these tests are actually built from the platform agnostic >> side by the 'test/validation/Makefile.am' (and subdirectories). >> '<PLATFORM_SPECIFIC>/Makefile.am' is involved in building only when >> platform specific tests exists, as discussed below. >> - >> Using other languages >> ^^^^^^^^^^^^^^^^^^^^^ >> -The pktio module, above, is actually tested using a bash script. This >> script is needed to set up the interfaces used by the tests. The pktio_run >> script actually eventually calls the platform agnostic >> 'test/validation/pktio/pktio_main' after setting up the interfaces needed >> by the tests. >> -Notice that the path to the script is >> '<PLATFORM_SPECIFIC>/pktio/pktio_run', i.e. it is private to this platform. >> Any languages supported by the tested platform can be used there, as it >> will not impact other platforms. >> -The platform "private" executables (such as this script), of course, >> must also return one of the return code expected by the automake test >> harness (0 for success, 77 for inconclusive, other values for errors). >> +The pktio module, above, is actually tested using a bash script. This >> script is needed to set up the interfaces used by the tests. The pktio_run >> script eventually calls the platform agnostic >> 'test/validation/pktio/pktio_main' after setting up the interfaces needed >> by the tests. >> +Notice that the path to the script, 'pktio/pktio_run', is relative so is >> private to this platform. Any languages supported by the tested platform >> can be used there, as it will not impact other platforms. >> +The platform "private" executables (such as this script), of course, >> must also return one of the return code expected by the automake test >> harness (0 for success, 77 for skipped, other values for errors). >> > > It is not the fact that the path is relative that makes it platform > specific: It is the fact that it "points" to the platform side > > > >> >> Defining test wrappers >> ^^^^^^^^^^^^^^^^^^^^^^ >> @@ -232,14 +228,56 @@ Defining platform specific tests >> Sometimes, it may be necessary to call platform specific system calls to >> check some functionality: For instance, testing odp_cpumask_* could involve >> checking the underlying system CPU mask. On linux, such a test would >> require using the CPU_ISSET macro, which is linux specific. Such a test >> would be written in '<PLATFORM_SPECIFIC>/cpumask/...' The contents of this >> directory would be very similar to the contents of the platform agnostic >> side cpu_mask tests (including a Makefile.am...), but platform specific >> test would be written there. >> '<PLATFORM_SPECIFIC>/Makefile.am' would then trigger the building of the >> platform specific tests (by listing their module name in SUBDIRS and >> therefore calling the appropriate Makefile.am) and then it would call both >> the platform agnostic executable(s) and the platform specific test >> executable. >> >> -Skipping tests during development >> -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> -During ODP development, it may be useful to skip some test. This can be >> achieved by creating a new test executable (still on the platform side), >> picking up the required tests from the platform agnostic libtest<module>.la. >> +Marking tests as inactive >> +^^^^^^^^^^^^^^^^^^^^^^^^^ >> +The general policy is that a full run of the validation suite (a "make >> check") must pass at all times. However a particular platform may have one >> or more test cases that are known to be unimplemented either during >> development or permanently, so to avoid these test cases being reported as >> failures it's useful to be able to skip them. This can be achieved by >> creating a new test executable (still on the platform side), giving the >> platform specific initialisation code the opportunity to modify the >> registered tests in order to mark unwanted tests as inactive while leaving >> the remaining tests active. It's important that the unwanted tests are >> still registered with the test framework to allow the fact that they're not >> being tested to be recorded. >> + >> +The odp_cunit_update() function is intended for this purpose, it is used >> to modify the properties of previously registered tests, for example to >> mark them as inactive. Inactive tests are registered with the test >> framework but aren't executed and will be recorded as inactive in test >> reports. >> + >> +In 'test/validation/foo/foo.c', define all tests for the 'foo' module; >> > > That semicolumn (";") should be a column (":"), right? > > + >> +[source,c] >> +------------------ >> +odp_testinfo_t foo_tests[] = { >> + ODP_TEST_INFO(foo_test_a), >> + ODP_TEST_INFO(foo_test_b), >> + ODP_TEST_INFO_NULL >> +}; >> + >> +odp_suiteinfo_t foo_suites[] = { >> + {"Foo", foo_suite_init, foo_suite_term, foo_tests}, >> + ODP_SUITE_INFO_NULL >> +}; >> +------------------ >> + >> +In 'platform/<platform>/test/foo/foo_main.c', register all the tests >> defined in the 'foo' module, then mark a single specific test case as >> inactive; >> > > That semicolumn (";") should be a column (":"), right? > > >> + >> +[source,c] >> +------------------ >> +static odp_testinfo_t foo_tests_updates[] = { >> + ODP_TEST_INFO_INACTIVE(foo_test_b), >> + ODP_TEST_INFO_NULL >> +}; >> + >> +static odp_suiteinfo_t foo_suites_updates[] = { >> + {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, >> + ODP_SUITE_INFO_NULL >> +}; >> + >> +int pktio_main(void) >> > ... and this should be "foo" (Am I commenting the wrong patch? Didn't you correct that once already)? Christophe > +{ >> + int ret = odp_cunit_register(foo_suites); >> + >> + if (ret == 0) >> + ret = odp_cuint_update(foo_suites_updates); >> > > Should be "cunit", not "cuint" > > >> + >> + if (ret == 0) >> + ret = odp_cunit_run(); >> >> -The top Makefile would then call only the platform specific executable, >> hence skipping the tests which have been omitted. >> + return ret; >> +} >> +------------------ >> >> -TIP: You probably want to copy the platform-agnostic module main >> function and prune it from the undesired tests when writing your own >> platform specific main, for a given module. >> +So 'foo_test_a' will be executed and 'foo_test_b' is inactive. >> >> -Permanently skipping test >> -^^^^^^^^^^^^^^^^^^^^^^^^^^ >> -If a platform wants to permanently skip a test (i.e. a part of the ODP >> API is and will not be supported on that platform), it is recommended to >> use the function odp_cunit_TBD() to removed the tests or suite from the >> list of tests. This gives a chance to the test environment to trace this >> removal. >> +It's expected that early in the development cycle of a new >> implementation the inactive list will be quite long, but it should shrink >> over time as more parts of the API are implemented. >> -- >> 2.1.1 >> >> >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
