Hi Frederic,
Op 19-10-2018 om 11:51 schreef Frederic Lecaille:
The idea of the script sounds good to me. About the script itself it
is a nice work which could be a good start.
Thanks.
Just a few details below.
Note that "cat $file | grep something" may be shortened by "grep
something $file". It would also be interesting to avoid creating
temporary files as most as possible (at least testflist.lst is not
necessary I think).
TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //')
OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //')
may be shortened by these lines:
{ read TARGET; read OPTIONS; } << EOF
$(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //')
EOF
Thanks, I've changed this.
which is portable, or something similar.
sed 's/.*=//'| sed 's/,.*//'
may be shortened by
sed -e 's/.*=//' -e 's/,.*//'
Thanks, I've changed this as well.
Also note, there are some cases where options are enabled without
appearing in OPTIONS variable value.
For instance if you compile haproxy like that:
$ make TARGET=linux2628
the support for the thread is enabled without appearing in OPTIONS
variable value. I am not sure this is an issue at this time.
That could become an issue. but should be easy to solve adding a 'fake'
option perhaps to check against.. Or adding a separate check perhaps I'm
not sure yet.
Perhaps we could could only one line for the required options and
excluded targets like that:
#EXCLUDED_TARGETS=freebsd,dos,windows ;)
#REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB
Added this option, but then i like the option of excluding single
targets and having some comment behind it explaining the reason.. But i
guess if one would want to know some comments above the setting could
also 'explain' why that target is currently not 'valid' to run the test
on. Should i remove the settings for the 'single' option/target.?
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?
Regards,
PiBa-NL (Pieter)
#!/usr/bin/env sh
if [ "$1" == "--help" ]; then
cat << EOF
### run-regtests.sh ###
Running run-regtests.sh --help shows this information about how to use it
Run without parameters to run all tests in the current folder (including
subfolders)
run-regtests.sh
Provide paths to run tests from (including subfolders):
run-regtests.sh ./tests1 ./tests2
Parameters:
--j <NUM>, To run varnishtest with multiple jobs / threads for a faster
overall result
run-regtests.sh ./fasttest --j 16
--v, to run verbose
run-regtests.sh --v, disables the default varnishtest 'quiet' parameter
--varnishtestparams <ARGS>, passes custom ARGS to varnishtest
run-regtests.sh --varnishtestparams "-n 10"
--f, force deleting old /tmp/*.vtc results without asking
Including text below into a .vtc file will check for its requirements
related to haproxy's target and compilation options
# Below targets are not capable of completing this test succesfully
#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd
#EXCLUDE_TARGETS=dos,freebsd,windows
# Below option is required to complete this test succesfully
#REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in.
#REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA
# To define a range of versions that a test can run with:
#REQUIRE_VERSION=0.0
#REQUIRE_VERSION_BELOW=99.9
EOF
return
fi
_startswith() {
_str="$1"
_sub="$2"
echo "$_str" | grep "^$_sub" >/dev/null 2>&1
}
_findtests() {
#find "$1" -name "*.vtc" | while read i; do
IFS='
'
set -f
for i in $( find "$1" -name "*.vtc" ); do
#echo "TESTcheck '$i'"
skiptest=
require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')"
require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e
's/.*=//')"
require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')"
exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//'),"
if [ -n "$require_version" ]; then
if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ];
then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo " REASON: this test requires at least version: $require_version"
skiptest=1
fi
fi
if [ -n "$require_version_below" ]; then
if [ $(_version "$HAPROXY_VERSION") -ge $(_version
"$require_version_below") ]; then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo " REASON: this test requires a version below:
$require_version_below"
skiptest=1
fi
fi
if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then
echo " Skip $i because exclude_targets"
echo " REASON: exclude_targets '$exclude_targets' contains '$TARGET'"
skiptest=1
fi
#echo "REQUIRE_OPTIONS : $require_options"
for requiredoption in $(echo $require_options | tr "," "
" ); do
if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ]
then
echo " Skip $i because option $requiredoption not found"
echo -n " REASON: "
echo -n "$required" | sed 's/.*,//' | sed -e 's/^[[:space:]]//'
echo
skiptest=1
fi
done
for required in "$(grep "#REQUIRE_OPTION=" "$i")";
do
if [ -z "$required" ]
then
continue
fi
requiredoption=$(echo "$required" | sed -e 's/.*=//' -e 's/,.*//')
if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ]
then
echo " Skip $i because option $requiredoption not found"
echo -n " REASON: "
echo "$required" | sed 's/.*,//' | sed -e 's/^[[:space:]]//'
skiptest=1
fi
done
testtarget=$(grep "#EXCLUDE_TARGET=" "$i")
if [ "$( echo "$testtarget" | grep "#EXCLUDE_TARGET=$TARGET," )" ]
then
echo " Skip $i because: TARGET = $TARGET"
echo -n " REASON: "
echo "$testtarget" | sed 's/.*,//' | sed -e 's/^[[:space:]]//'
skiptest=1
fi
if [ -z $skiptest ]; then
echo " Add test: $i"
testlist="$testlist
$i"
fi
done
}
_process() {
jobcount=""
verbose="-q"
# for
while [ ${#} -gt 0 ]; do
if _startswith "$1" "-"; then
case "${1}" in
--j)
jobcount="$2"
shift
;;
--varnishtestparams)
varnishtestparams="$2"
shift
;;
--v)
verbose=""
;;
--f)
forcetest=1
;;
*)
echo "Unknown parameter : $1"
return 1
;;
esac
else
_findtests "$1"
pathwasset=1
fi
shift 1
done
if [ -z $pathwasset ]; then
# no path was given, find all tests under current path
_findtests ./
fi
}
_version() {
echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }';
}
echo ""
echo "########################## Preparing to run tests
##########################"
{ read HAPROXY_VERSION; read TARGET; read OPTIONS; } << EOF
$(haproxy -vv |grep 'HA-Proxy version\|TARGET\|OPTIONS' | sed 's/.* = //')
EOF
HAPROXY_VERSION=$(echo $HAPROXY_VERSION | awk -F" " '{ printf("%s", $3); }')
echo "Testing with haproxy version: $(echo $HAPROXY_VERSION)"
echo "########################## Gathering tests to run
##########################"
testlist=""
pathwasset=
forcetest=0
_process "$@";
echo "########################## Cleaning old test results
##########################"
echo "Check for old /tmp/vtc.* result files"
if [ -n "$( find /tmp/ -name "vtc.*" )" ]; then
if [ $forcetest != 1 ]; then
echo "Old varnishtest temporary results found, these need to be deleted,
run with --f to automatically approve"
echo -n "Continue? Y/N: "
read answer
if [ $answer != "Y" ]; then
echo "aborting"
return
fi
fi
echo "deleting old /tmp/vtc.* results"
find /tmp/ -name "vtc.*" -print0 | xargs -r0 -- rm -r
fi
echo "########################## Starting varnishtest
##########################"
_vtresult=0
if [ -n "$testlist" ]; then
if [ -n "$jobcount" ]; then
jobcount="-j $jobcount"
fi
varnishtest $varnishtestparams $verbose $jobcount -l -k -t 10 $testlist
_vtresult=$?
#echo "result: $_vtresult"
else
echo "No tests found that meet the required criteria"
fi
if [ $_vtresult != 0 ]
then
echo "########################## Gathering failed results
##########################"
for i in $(find /tmp/ -name "vtc.*");
do
echo "###### $(cat $i/INFO) ######"
echo "## test results in: $i"
cat $i/LOG | grep -- ----
done
exit 1
fi
exit 0
########################## Preparing to run tests ##########################
Testing with haproxy version: 1.9-dev4-1ff7633
########################## Gathering tests to run ##########################
Add test: ./haproxy_test_OK_20180831/lua/b00002.vtc
Add test: ./haproxy_test_OK_20180831/lua/h00001.vtc
Add test: ./haproxy_test_OK_20180831/lua/b00000.vtc
Add test: ./haproxy_test_OK_20180831/lua/b00001.vtc
Add test: ./haproxy_test_OK_20180831/log/b00000.vtc
Add test: ./haproxy_test_OK_20180831/reload/b00001-server-state-file.vtc
Add test: ./haproxy_test_OK_20180831/ssl/b00000.vtc
Add test: ./haproxy_test_OK_20180831/stick-table/b00001.vtc
Add test: ./haproxy_test_OK_20180831/stick-table/b00000.vtc
Add test: ./test/b00000-loadtest.vtc
Skip ./test/b00002.vtc because: TARGET = freebsd
REASON: abns sockets are not available on freebsd
Add test: ./test/b00003-cpu.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/server/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/ssl/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/stick-table/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/stick-table/b00001.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/connection/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/log/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/lua/b00001.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/lua/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/lua/b00003.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/lua/h00001.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/lua/b00002.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/seamless-reload/b00000.vtc
Add test: ./work/haproxy-1ff7633/reg-tests/spoe/b00000.vtc
Add test: ./PB-TEST/h2-ssl-backend.vtc
Add test: ./PB-TEST/a02021.vtc
Skip ./PB-TEST/run-regtest-test-skiptests/skip_version_below.vtc because
option haproxy is version: 1.9-dev4-1ff7633
REASON: this test requires a version below: 0.0-dev0
Skip ./PB-TEST/run-regtest-test-skiptests/skip_exclude target.vtc because:
TARGET = freebsd
REASON: abns sockets are not available on freebsd
Skip ./PB-TEST/run-regtest-test-skiptests/skip_require_option.vtc because
option THIS_IS_NO_OPTION not found
REASON: this test needs THIS_IS_NO_OPTION compiled in.
Skip ./PB-TEST/run-regtest-test-skiptests/skip_require_options.vtc because
option THIS_IS_NO_OPTION not found
REASON: this test needs THIS_IS_NO_OPTION compiled in.
Skip ./PB-TEST/run-regtest-test-skiptests/skip_require_options.vtc because
option THAT_IS_NO_OPTION not found
REASON: this test needs THIS_IS_NO_OPTION compiled in.
Skip ./PB-TEST/run-regtest-test-skiptests/skip_exclude_targets.vtc because
exclude_targets
REASON: exclude_targets ',dos,freebsd,windows,' contains 'freebsd'
Skip ./PB-TEST/run-regtest-test-skiptests/skip_version.vtc because option
haproxy is version: 1.9-dev4-1ff7633
REASON: this test requires at least version: 99.99-dev99
Add test: ./PB-TEST/h2-windowsize-basics.vtc
Add test: ./PB-TEST/b00000-loadtest-40000-connections-SLOW.vtc
Add test: ./PB-TEST/cache.vtc
Add test: ./varnish_regtest/t02005_WORKS.vtc
########################## Cleaning old test results ##########################
Check for old /tmp/vtc.* result files
Old varnishtest temporary results found, these need to be deleted, run with --f
to automatically approve
Continue? Y/N: Y
deleting old /tmp/vtc.* results
########################## Starting varnishtest ##########################
# top TEST ./PB-TEST/h2-ssl-backend.vtc FAILED (0.279) exit=2
########################## Gathering failed results ##########################
###### Test case: ./PB-TEST/h2-ssl-backend.vtc ######
## test results in: /tmp/vtc.64837.140ed3dd
---- s4 0.2 HTTP rx failed (fd:10 read: Connection reset by peer)
