On Thu, 2008-07-31 at 20:47 +0100, Mel Gorman wrote:
> libhugetlbfs is able to run a variety of regression tests against the
> currently running kernel. Some of these tests may fail due to known
> reasons such as the kernel being too old. It is unreasonable to expect an
> ordinary user of the library to determine that a FAIL is really an "expected
> fail". This patch introduces a framework where verification scripts can be
> added called verifyresults-TESTNAME.sh. It gets passed the return value
> of the regression script and returns 0 if the output is expected and 1
> otherwise. This in combination with a message can be used to report to the
> user if a FAILed test is due to expected reasons.
>
> Signed-off-by: Mel Gorman <[EMAIL PROTECTED]>
> ---
> tests/run_tests.sh | 29 ++++++++++++++++++++++++++++-
> 1 files changed, 28 insertions(+), 1 deletions(-)
>
> diff --git a/tests/run_tests.sh b/tests/run_tests.sh
> index c98179d..a08eecb 100755
> --- a/tests/run_tests.sh
> +++ b/tests/run_tests.sh
> @@ -55,7 +55,34 @@ run_test_bits () {
>
> if [ -d obj$BITS ]; then
> echo -n "$@ ($BITS): "
> - PATH="obj$BITS:$PATH"
> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:../obj$BITS:obj$BITS" $ENV "$@"
> + OUTPUT=`PATH="obj$BITS:$PATH"
> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:../obj$BITS:obj$BITS" $ENV "$@"`
> + RETVAL=$?
> +
> + # Figure out the name of the testcase we just ran
> + echo $1 | grep = > /dev/null
> + while [ $? -eq 0 ]; do
> + shift 1
> + echo $1 | grep = > /dev/null
> + done
> +
> + # Run a postprocessing script if it exists. Given the return value
> + # of the regression test, a post-processing script returns 0 if the
> + # result is expected and non-zero otherwise. If the user should be
> + # informed, it may print out a reason for the decision. For example,
> + # a post-processing script can detect that a FAIL is expected due
> + # to an old kernel and print that explanation so the user does not
> + # report the FAIL
> + if [ -e verifyresults-$1.sh ]; then
> + REASON=`./verifyresults-$1.sh $RETVAL`
> + if [ "$REASON" != "" ]; then
> + if [ $? -eq 0 ]; then
> + OUTPUT="EXPECTED $OUTPUT ($REASON)"
> + else
> + OUTPUT="UNEXPECTED $OUTPUT ($REASON)"
> + fi
> + fi
> + fi
> + echo "$OUTPUT"
> fi
> }
Not to make this more complicated, but I think we only want the script
to call attention to _false_ positives or _false_ negatives. ie. An
EXPECTED PASS is not meaningful. Neither is an UNEXPECTED FAIL. I
guess making the scripts only return non-zero when the situation is
noteworthy is the way to fix this.
Perhaps we could have the script return 0 when the result is not
noteworthy, 1 when it's an expected failure, and 2 when it's an
unexpected pass. Then the above messages can be made more descriptive:
if [ $? -eq 1 ]; then
OUTPUT="EXPECTED FAILURE ($REASON)"
elif [ $? -eq 2 ]; then
OUTPUT="UNEXPECTED PASS ($REASON)"
fi
...
--
Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Libhugetlbfs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
