It is possible to have tests run over multiple test families if the respective modules can be loaded/unloaded serially on a test to test basis. The kernel patch [1] separated out the PAPR specific ndtest module on its own as "ndtest.ko".
So with [1], we have two family specific modules(nfit_test & ndtest) which can be loaded one after the other and have the test results in one go without a need for kernel recompilation with different kernel configs. The series ensures all the known failures/unsupported/irrelevant tests to be skipped on ndtest to avoid all "known" chances of false negatives. This is achieved using a skip file containing the list of tests to be skipped on the given platform. The PAPR specific tests to be skipped are updated in the skip_PAPR.js file. A meson config option(-Dtest-families=INTEL,PAPR default) is added. One can just limit the tests to INTEL or PAPR alone if required. An example run would look like, =========================================== 1/72 ndctl:ndctl+INTEL / libndctl OK 2.79s 2/72 ndctl:ndctl+INTEL / dsm-fail OK 0.70s 3/72 ndctl:ndctl+INTEL / create.sh OK 0.80s 4/72 ndctl:ndctl+INTEL / clear.sh OK 15.04s 5/72 ndctl:ndctl+INTEL / pmem-errors.sh OK 14.97s 6/72 ndctl:dax+INTEL / daxdev-errors.sh OK 12.33s 7/72 ndctl:dax+INTEL / multi-dax.sh OK 0.67s 8/72 ndctl:ndctl+INTEL / btt-check.sh OK 16.20s 9/72 ndctl:ndctl+INTEL / label-compat.sh OK 0.82s 10/72 ndctl:ndctl+INTEL / sector-mode.sh OK 1.01s 11/72 ndctl:ndctl+INTEL / inject-error.sh OK 14.52s 12/72 ndctl:ndctl+INTEL / btt-errors.sh OK 24.01s 13/72 ndctl:ndctl+INTEL / hugetlb OK 0.09s 14/72 ndctl:ndctl+INTEL / btt-pad-compat.sh OK 1.78s 15/72 ndctl:ndctl+INTEL / firmware-update.sh FAIL 10.33s exit status 1 >>> TEST_PATH=/home/sbhat/code/ndctl/build/test NDCTL_TEST_FAMILY=INTEL >>> DAXCTL=/home/sbhat/code/ndctl/build/daxctl/daxctl >>> NDCTL=/home/sbhat/code/ndctl/build/ndctl/ndctl MALLOC_PERTURB_=181 >>> DATA_PATH=/home/sbhat/code/ndctl/test /home/sbhat /code/ndctl/test/firmware-update.sh 16/72 ndctl:ndctl+INTEL / ack-shutdown-count-set OK 0.26s 17/72 ndctl:ndctl+INTEL / rescan-partitions.sh OK 7.84s 18/72 ndctl:ndctl+INTEL / inject-smart.sh OK 1.89s 19/72 ndctl:ndctl+INTEL / monitor.sh OK 13.22s 20/72 ndctl:ndctl+INTEL / max_extent_ns OK 1.10s 21/72 ndctl:ndctl+INTEL / pfn-meta-errors.sh OK 14.62s 22/72 ndctl:ndctl+INTEL / track-uuid.sh OK 0.78s 23/72 ndctl:ndctl+INTEL / pmem-ns FAIL 0.26s (exit status 237 or signal 109 SIGinvalid) >>> TEST_PATH=/home/sbhat/code/ndctl/build/test MALLOC_PERTURB_=140 >>> NDCTL_TEST_FAMILY=INTEL DAXCTL=/home/sbhat/code/ndctl/build/daxctl/daxctl >>> NDCTL=/home/sbhat/code/ndctl/build/ndctl/ndctl >>> DATA_PATH=/home/sbhat/code/ndctl/test /home/sbhat /code/ndctl/build/test/pmem-ns 24/72 ndctl:dax+INTEL / sub-section.sh SKIP 0.15s exit status 77 25/72 ndctl:dax+INTEL / dax-dev SKIP 0.03s exit status 77 26/72 ndctl:dax+INTEL / dax-ext4.sh SKIP 0.05s exit status 77 27/72 ndctl:dax+INTEL / dax-xfs.sh SKIP 0.05s exit status 77 28/72 ndctl:ndctl+INTEL / align.sh SKIP 0.09s exit status 77 29/72 ndctl:dax+INTEL / device-dax SKIP 0.04s exit status 77 30/72 ndctl:dax+INTEL / revoke-devmem OK 0.03s 31/72 ndctl:dax+INTEL / device-dax-fio.sh SKIP 0.06s exit status 77 32/72 ndctl:dax+INTEL / daxctl-devices.sh SKIP 0.07s exit status 77 33/72 ndctl:dax+INTEL / daxctl-create.sh SKIP 0.02s exit status 77 34/72 ndctl:dax+INTEL / dm.sh FAIL 0.12s exit status 1 >>> TEST_PATH=/home/sbhat/code/ndctl/build/test NDCTL_TEST_FAMILY=INTEL >>> DAXCTL=/home/sbhat/code/ndctl/build/daxctl/daxctl >>> NDCTL=/home/sbhat/code/ndctl/build/ndctl/ndctl MALLOC_PERTURB_=115 >>> DATA_PATH=/home/sbhat/code/ndctl/test /home/sbhat /code/ndctl/test/dm.sh 35/72 ndctl:dax+INTEL / mmap.sh SKIP 0.05s exit status 77 36/72 ndctl:ndctl+INTEL / security.sh SKIP 0.38s exit status 77 37/72 ndctl:ndctl+PAPR / libndctl OK 2.63s 38/72 ndctl:ndctl+PAPR / dsm-fail OK 1.41s 39/72 ndctl:ndctl+PAPR / create.sh OK 1.06s 40/72 ndctl:ndctl+PAPR / clear.sh SKIP 0.12s exit status 77 41/72 ndctl:ndctl+PAPR / pmem-errors.sh SKIP 0.16s exit status 77 42/72 ndctl:dax+PAPR / daxdev-errors.sh SKIP 0.08s exit status 77 43/72 ndctl:dax+PAPR / multi-dax.sh OK 1.03s 44/72 ndctl:ndctl+PAPR / btt-check.sh OK 17.13s 45/72 ndctl:ndctl+PAPR / label-compat.sh SKIP 0.26s exit status 77 46/72 ndctl:ndctl+PAPR / sector-mode.sh OK 1.26s 47/72 ndctl:ndctl+PAPR / inject-error.sh SKIP 0.20s exit status 77 48/72 ndctl:ndctl+PAPR / btt-errors.sh SKIP 1.42s exit status 77 49/72 ndctl:ndctl+PAPR / hugetlb OK 0.16s 50/72 ndctl:ndctl+PAPR / btt-pad-compat.sh OK 3.06s 51/72 ndctl:ndctl+PAPR / firmware-update.sh SKIP 0.90s exit status 77 52/72 ndctl:ndctl+PAPR / ack-shutdown-count-set SKIP 0.09s exit status 77 53/72 ndctl:ndctl+PAPR / rescan-partitions.sh OK 8.11s 54/72 ndctl:ndctl+PAPR / inject-smart.sh OK 1.37s 55/72 ndctl:ndctl+PAPR / monitor.sh OK 7.63s 56/72 ndctl:ndctl+PAPR / max_extent_ns OK 1.41s 57/72 ndctl:ndctl+PAPR / pfn-meta-errors.sh SKIP 0.14s exit status 77 58/72 ndctl:ndctl+PAPR / track-uuid.sh OK 1.07s 59/72 ndctl:ndctl+PAPR / pmem-ns OK 0.54s 60/72 ndctl:dax+PAPR / sub-section.sh SKIP 0.27s exit status 77 61/72 ndctl:dax+PAPR / dax-dev SKIP 0.04s exit status 77 62/72 ndctl:dax+PAPR / dax-ext4.sh SKIP 0.35s exit status 77 63/72 ndctl:dax+PAPR / dax-xfs.sh SKIP 0.39s exit status 77 64/72 ndctl:ndctl+PAPR / align.sh SKIP 0.46s exit status 77 65/72 ndctl:dax+PAPR / device-dax SKIP 0.04s exit status 77 66/72 ndctl:dax+PAPR / revoke-devmem OK 0.04s 67/72 ndctl:dax+PAPR / device-dax-fio.sh SKIP 0.32s exit status 77 68/72 ndctl:dax+PAPR / daxctl-devices.sh SKIP 0.40s exit status 77 69/72 ndctl:dax+PAPR / daxctl-create.sh SKIP 0.48s exit status 77 70/72 ndctl:dax+PAPR / dm.sh SKIP 0.48s exit status 77 71/72 ndctl:dax+PAPR / mmap.sh SKIP 0.50s exit status 77 72/72 ndctl:ndctl+PAPR / security.sh SKIP 0.22s exit status 77 Ok: 37 Expected Fail: 0 Fail: 3 # Not from PAPR family Unexpected Pass: 0 Skipped: 32 Timeout: 0 =========================================== Note the meson suite string here "ndctl+INTEL|PAPR" indicating the test family. The first 6 patches in the series move the code with little/no change to current behaviour in isolation while fixing existing bugs in preparation for running the tests for INTEL and PAPR families one after the other. The last 3 patches fix the known failures on ndtest and they depend on the kernel patch [2]. These 3 patches are repost of previously posted patches with rebase. The last patches are included here in this series for completeness. The patches are posted on github repos below Kernel - https://github.com/shivaprasadbhat/linux/tree/ndtest-runall ndctl - https://github.com/shivaprasadbhat/ndctl/tree/ndctl-test-all References: [1] - https://patchwork.kernel.org/project/linux-nvdimm/patch/165025849452.2974989.6131145460098517848.st...@lep8c.aus.stglabs.ibm.com/ [2] - https://patchwork.kernel.org/project/linux-nvdimm/patch/165027233876.3035289.4353747702027907365.st...@lep8c.aus.stglabs.ibm.com/ --- Shivaprasad G Bhat (9): test/common: Ensure to unload test modules test: core: Fix module unload failures test: Unload the nfit module during cleanup test: Introduce skip file to skip irrelevant tests test: Assign provider name based on the test family test: Enable PAPR test family tests after INTEL family tests test/libndctl: Enable libndctl tests on ndtest test/inject-smart: Enable inject-smart tests on ndtest test/monitor.sh: Partially skip monitor test on ndtest meson.build | 10 +++ meson_options.txt | 2 + ndctl/bat.c | 4 +- ndctl/test.c | 4 +- test.h | 10 ++- test/ack-shutdown-count-set.c | 17 ++-- test/btt-check.sh | 3 +- test/btt-errors.sh | 3 +- test/btt-pad-compat.sh | 3 +- test/clear.sh | 4 +- test/common | 57 ++++++++++++- test/core.c | 140 +++++++++++++++++++++++++++++++- test/create.sh | 4 +- test/dax-dev.c | 2 +- test/dax-pmd.c | 2 +- test/daxdev-errors.sh | 6 +- test/device-dax.c | 2 +- test/dsm-fail.c | 33 +++++--- test/firmware-update.sh | 3 +- test/inject-error.sh | 3 +- test/inject-smart.sh | 17 ++-- test/label-compat.sh | 4 +- test/libndctl.c | 60 ++++++++++++-- test/list-smart-dimm.c | 33 +++++++- test/max_available_extent_ns.sh | 3 +- test/meson.build | 59 ++++++++------ test/monitor.sh | 14 +++- test/multi-dax.sh | 4 +- test/pfn-meta-errors.sh | 3 +- test/pmem-errors.sh | 3 +- test/pmem_namespaces.c | 29 +++---- test/rescan-partitions.sh | 3 +- test/revoke-devmem.c | 2 +- test/sector-mode.sh | 4 +- test/security.sh | 3 +- test/skip_PAPR.js | 32 ++++++++ test/track-uuid.sh | 3 +- 37 files changed, 447 insertions(+), 141 deletions(-) create mode 100644 test/skip_PAPR.js --
