On Wed, Oct 22, 2008 at 08:55:13PM +0000, Adam Litke wrote:
> The libhugetlbfs test suite has various tests for which a failure (under
> certain circumstances) is expected and not indicative of an error. It would
> be desirable to specially annotate these cases to eliminate the need for the
> tester to know which failures are known and which could be bugs.
>
> This patch creates some test harness infrastructure that can be used to
> achieve this goal. How does it work? If a test case needs expected failure
> validation, a shell script (named <testcase>.sh) is placed in the tests/
> directory. The test invocation in run_tests.sh is changed to execute the
> shell script instead of the c program directly. The shell script must
> perform any required discovery, execute the real test case, and interpret the
> result.
>
> Comments?
>
> Signed-off-by: Adam Litke <[EMAIL PROTECTED]>
> ---
>
> tests/Makefile | 18 +++++++++++++++++-
> tests/hugetests.h | 2 ++
> tests/run_tests.sh | 8 ++++++++
> 3 files changed, 27 insertions(+), 1 deletions(-)
>
>
> diff --git a/tests/Makefile b/tests/Makefile
> index 27e5dd5..5fb926f 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -16,6 +16,7 @@ LDSCRIPT_TESTS = zero_filesize_segment
> HUGELINK_TESTS = linkhuge linkhuge_nofd linkshare
> HUGELINK_RW_TESTS = linkhuge_rw
> STRESS_TESTS = mmap-gettest mmap-cow shm-gettest shm-getraw shm-fork
> +WRAPPERS =
> HELPERS = get_hugetlbfs_path
> HELPER_LIBS = libheapshrink.so
> BADTOOLCHAIN = bad-toolchain.sh
> @@ -60,13 +61,14 @@ DEPFILES = $(LIB_TESTS:%=%.d) $(NOLIB_TESTS:%=%.d)
> $(HUGELINK_TESTS:%=%.d) \
> $(HELPERS:%=%.d) testutils.d
>
> ALLTESTS = $(foreach DIR,$(OBJDIRS),$(TESTS:%=$(DIR)/%))
> +ALLWRAPPERS = $(foreach DIR,$(OBJDIRS),$(DIR)/.wrappers)
> ALLHELPERS = $(foreach DIR,$(OBJDIRS),$(HELPERS:%=$(DIR)/%))
> ALLHELPERLIBS = $(foreach DIR,$(OBJDIRS),$(HELPER_LIBS:%=$(DIR)/%))
> ifdef CC64
> ALLTESTS += $(TESTS_64:%=obj64/%)
> endif
>
> -all: $(ALLTESTS) $(ALLHELPERS) $(ALLHELPERLIBS)
> +all: $(ALLTESTS) $(ALLHELPERS) $(ALLHELPERLIBS) $(ALLWRAPPERS)
>
> shmoverride_linked.c: shmoverride_unlinked.c
> ln -s shmoverride_unlinked.c shmoverride_linked.c
> @@ -185,6 +187,20 @@ $(HELPERS:%=obj64/%): %: %.o
> @$(VECHO) LD64 "(helper)" $@
> $(CC64) $(LDFLAGS) $(LDFLAGS64) -o $@ $^ $(LDLIBS) -lhugetlbfs
>
> +obj32/.wrappers: $(ALLTESTS)
> + @$(VECHO) LN "(test wrappers)" obj32/
> + @for obj in $(WRAPPERS); do \
> + [ -f obj32/$$obj ] && ln -sf ../$$obj.sh obj32/$$obj.sh ; \
> + done ; true
> + @touch $@
> +
> +obj64/.wrappers: $(ALLTESTS)
> + @$(VECHO) LN "(test wrappers)" obj64/
> + @for obj in $(WRAPPERS); do \
> + [ -f obj64/$$obj ] && ln -s ../$$obj.sh obj64/$$obj.sh ; \
> + done ; true
> + @touch $@
> +
> clean:
> @$(VECHO) CLEAN "(tests)"
> rm -f *~ *.o *.so *.a *.d core a.out
> diff --git a/tests/hugetests.h b/tests/hugetests.h
> index 270923b..b860244 100644
> --- a/tests/hugetests.h
> +++ b/tests/hugetests.h
> @@ -31,6 +31,8 @@
> #define RC_PASS 0
> #define RC_CONFIG 1
> #define RC_FAIL 2
> +#define RC_XFAIL 3 /* Expected Failure */
> +#define RC_XPASS 4 /* Unexpected Pass */
I don't think it makes sense to use X for both does it? Do we need both
of these? Is not the result simply not the expected result in either
case? RC_NOT_EXPECTED or RC_UNEXPECTED or something?
> #define RC_BUG 99
>
> #define FOURGB (1UL << 32)
> diff --git a/tests/run_tests.sh b/tests/run_tests.sh
> index 5b2a955..19cd9ee 100755
> --- a/tests/run_tests.sh
> +++ b/tests/run_tests.sh
> @@ -19,6 +19,8 @@ for BITS in 32 64; do
> tot_config[$BITS]=0
> tot_signal[$BITS]=0
> tot_strange[$BITS]=0
> + tot_xpass[$BITS]=0
> + tot_xfail[$BITS]=0
> tot_skip[$BITS]=0
> done
>
> @@ -110,6 +112,10 @@ run_test_bits () {
> tot_config[$BITS]=$[tot_config[$BITS] + 1]
> elif [ "$rc" == "2" ]; then
> tot_fail[$BITS]=$[tot_fail[$BITS] + 1]
> + elif [ "$rc" == "3" ]; then
> + tot_xfail[$BITS]=$[tot_xfail[$BITS] + 1]
> + elif [ "$rc" == "4" ]; then
> + tot_xpass[$BITS]=$[tot_xpass[$BITS] + 1]
> elif [ "$rc" -gt 127 ]; then
> tot_signal[$BITS]=$[tot_signal[$BITS] + 1]
> else
> @@ -421,5 +427,7 @@ echo -e "* PASS:
> ${tot_pass[32]}\t${tot_pass[64]}"
> echo -e "* FAIL: ${tot_fail[32]}\t${tot_fail[64]}"
> echo -e "* Killed by signal: ${tot_signal[32]}\t${tot_signal[64]}"
> echo -e "* Bad configuration: ${tot_config[32]}\t${tot_config[64]}"
> +echo -e "* Expected FAIL: ${tot_xfail[32]}\t${tot_xfail[64]}"
> +echo -e "* Unexpected PASS: ${tot_xpass[32]}\t${tot_xpass[64]}"
> echo -e "* Strange test result: ${tot_strange[32]}\t${tot_strange[64]}"
> echo -e "**********"
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Libhugetlbfs-devel mailing list
> Libhugetlbfs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
