Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/23/18 1:12 AM, PiBa-NL wrote:
Hi Frederic,
Hi Pieter,
I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..
Strangely on my PC with both bash and dash I do not have to change
IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read"
internal command.
Reading version,target and options works fine indeed without, however
the loop over test files fails if any .vtc file has a space character in
its filename. Or should we 'forbid' that with documentation.? Or is
there another better workaround for that?
We will never use VTC files with spaces in their filenames.
I do not think it is a good idea to build TESTDIR like that:
TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')"
TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME
mkdir -p $TESTDIR
What if we run the tests several times at the same time?
Well they would have to run at the same second. Not sure if that would
be wise to do.. But mktemp now solved that i guess :) at least for the
directory name part..
Please have a look to mkstemp utility.
Without the 's' right? Done, combined with the rundatetime which i do
like, so its 'readable' and unique, best of both ways i think?.
Yes.
Remaining details:
cat $i/LOG | grep --
should be replaced by
grep -- $i/LOG
I guess ill never learn do do this right the first time around ;). Fixed.
Note that you script is plenty of '\r' characters with no newline
character at the end of file position:
Not sure what the 'correct' way would be. I think there is a CR LF
everywhere at the moment? And the scripts hashbang tries to point to
'sh', will this be a issue.? (acme.sh does the same, and seems to be run
an lots of systems..) And if so what can i best do to avoid issues?
Yes there are \r\n characters in place of \n end of line character for
each line.
You can check that with hexdump or od:
$ hexdump -c run-regtests.sh | head
000 # ! / u s r / b i n / e n v s
010 h \r \n \r \n i f [ " $ 1 " =
020 " - - h e l p " ] ; t h e
030 n \r \n c a t < < E O F \r
040 \n # # # r u n - r e g t e s t
050 s . s h # # # \r \n R u n n
060 i n g r u n - r e g t e s t s
070 . s h - - h e l p s h o w s
080 t h i s i n f o r m a t i o
090 n a b o u t h o w t o u
There are chances you are using a text file editor on a non-Unix system?
You can remove them like that before
cat run-regtests.sh | tr -d '\r' >run-regtests.sh.ok
Also note that some shells do not like == operator (at line 3):
Used a single = now.
When I do not set both HAPROY_PROGRAM I get this output with a script
with a successful result.
Checks added to avoid this issue for both haproxy and varnishtest so
they are checked to exist.
Next round :).
Shoulb be the last ;)
Thanks a lot Pieter.
Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic,
I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..
Strangely on my PC with both bash and dash I do not have to change
IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read"
internal command.
Reading version,target and options works fine indeed without, however
the loop over test files fails if any .vtc file has a space character in
its filename. Or should we 'forbid' that with documentation.? Or is
there another better workaround for that?
I do not think it is a good idea to build TESTDIR like that:
TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')"
TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME
mkdir -p $TESTDIR
What if we run the tests several times at the same time?
Well they would have to run at the same second. Not sure if that would
be wise to do.. But mktemp now solved that i guess :) at least for the
directory name part..
Please have a look to mkstemp utility.
Without the 's' right? Done, combined with the rundatetime which i do
like, so its 'readable' and unique, best of both ways i think?.
Remaining details:
cat $i/LOG | grep --
should be replaced by
grep -- $i/LOG
I guess ill never learn do do this right the first time around ;). Fixed.
Note that you script is plenty of '\r' characters with no newline
character at the end of file position:
Not sure what the 'correct' way would be. I think there is a CR LF
everywhere at the moment? And the scripts hashbang tries to point to
'sh', will this be a issue.? (acme.sh does the same, and seems to be run
an lots of systems..) And if so what can i best do to avoid issues?
Also note that some shells do not like == operator (at line 3):
Used a single = now.
When I do not set both HAPROY_PROGRAM I get this output with a script
with a successful result.
Checks added to avoid this issue for both haproxy and varnishtest so
they are checked to exist.
Next round :).
Regards,
PiBa-NL (Pieter)
#!/usr/bin/env sh
if [ "$1" = "--help" ]; then
cat << EOF
### run-regtests.sh ###
Running run-regtests.sh --help shows this information about how to use it
Run without parameters to run all tests in the current folder (including
subfolders)
run-regtests.sh
Provide paths to run tests from (including subfolders):
run-regtests.sh ./tests1 ./tests2
Parameters:
--j , To run varnishtest with multiple jobs / threads for a faster
overall result
run-regtests.sh ./fasttest --j 16
--v, to run verbose
run-regtests.sh --v, disables the default varnishtest 'quiet' parameter
--varnishtestparams , passes custom ARGS to varnishtest
run-regtests.sh --varnishtestparams "-n 10"
Including text below into a .vtc file will check for its requirements
related to haproxy's target and compilation options
# Below targets are not capable of completing this test succesfully
#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd
#EXCLUDE_TARGETS=dos,freebsd,windows
# Below option is required to complete this test succesfully
#REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in.
#REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA
# To define a range of versions that a test can run with:
#REQUIRE_VERSION=0.0
#REQUIRE_VERSION_BELOW=99.9
Configure environment variables to set the haproxy and varnishtest binaries
to use
setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy
setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest
EOF
return
fi
_startswith() {
_str="$1"
_sub="$2"
echo "$_str" | grep "^$_sub" >/dev/null 2>&1
}
_findtests() {
#find "$1" -name "*.vtc" | while read i; do
set -f
for i in $( find "$1" -name "*.vtc" ); do
#echo "TESTcheck '$i'"
skiptest=
require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')"
require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e
's/.*=//')"
require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')"
exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//'),"
if [ -n "$require_version" ]; then
if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ];
then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test requires at least version: $require_version"
skiptest=1
fi
fi
if [ -n "$require_version_below" ]; then
if [ $(_version "$HAPROXY_VERSION") -ge $(_version
"$require_version_below") ]; then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test requires a version below:
$require_version_below"
skiptest=1
fi
fi
if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then
echo " Skip $i because exclude_targets"
echo "REASON: exclude_targets '$exclude_targets' contains '$TARGET'"
skiptest=1
fi
#echo "REQUIRE_OPTIONS :
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/19/18 10:08 PM, PiBa-NL wrote:
Hi Frederic, Willy,
Hi Pieter,
Thank you a lot again for this work Pieter.
Hello Pieter,
Do you intend to finalize this script? We would like to use it in
haproxy sources.
Note that varnishtest already uses TMPDIR variable in place of /tmp if
it is set in the environment.
Thanks again.
Fred.
Thanks for your advices and comments, to be honest i haven't looked at
the script for several days, got distracted by other things ;). So sorry
for late reply.
No problem. Sorry for my late reply too ;)
Just cleaned it up a bit. I guess its ready for another review.
I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..
Strangely on my PC with both bash and dash I do not have to change
IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read"
internal command.
I've tried to incorporate all suggestions. Lemme know if/what i missed :)
I do not think it is a good idea to build TESTDIR like that:
TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')"
TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME
mkdir -p $TESTDIR
What if we run the tests several times at the same time?
Please have a look to mkstemp utility.
Remaining details:
cat $i/LOG | grep --
should be replaced by
grep -- $i/LOG
Note that you script is plenty of '\r' characters with no newline
character at the end of file position:
$ sh run-regtests.sh
: not founds.sh: 2: run-regtests.sh:
run-regtests.sh: 58: run-regtests.sh: Syntax error: word unexpected
(expecting "do")
flecaille@fl-haproxy:~/src/haproxy
$ bash -x run-regtests.sh
+ $'\r'
run-regtests.sh: line 2: $'\r': command not found
run-regtests.sh: line 47: syntax error near unexpected token `$'{\r''
'un-regtests.sh: line 47: `_startswith() {
Also note that some shells do not like == operator (at line 3):
$ HAPROXY_PROGRAM=~/src/haproxy/haproxy
VARNISHTEST_PROGRAM=~/src/varnish-cache-trunk/bin/varnishtest/varnishtest
sh run-regtests.sh
run-regtests.sh: 3: [: unexpected operator
When I do no set both HAPROY_PROGRAM I get this output with a script
with a successful result:
$ sh run-regtests.sh
## Preparing to run tests ##
run-regtests.sh.ok: 177: run-regtests.sh.ok: haproxy: not found
Testing with haproxy version:
run-regtests.sh.ok: 192: [: =: unexpected operator
run-regtests.sh.ok: 196: [: =: unexpected operator
run-regtests.sh.ok: 200: [: =: unexpected operator
run-regtests.sh.ok: 204: [: =: unexpected operator
run-regtests.sh.ok: 208: [: =: unexpected operator
run-regtests.sh.ok: 212: [: =: unexpected operator
run-regtests.sh.ok: 216: [: =: unexpected operator
run-regtests.sh.ok: 220: [: =: unexpected operator
run-regtests.sh.ok: 224: [: =: unexpected operator
run-regtests.sh.ok: 228: [: =: unexpected operator
run-regtests.sh.ok: 232: [: =: unexpected operator
run-regtests.sh.ok: 236: [: =: unexpected operator
run-regtests.sh.ok: 240: [: =: unexpected operator
run-regtests.sh.ok: 244: [: =: unexpected operator
run-regtests.sh.ok: 248: [: =: unexpected operator
Target :
Options :
## Gathering tests to run ##
Skip ./h2-keepalive-backend.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./urlp.regex.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/ssl/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/spoe/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/log/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/stick-table/b1.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/stick-table/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/server/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/seamless-reload/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/connection/b0.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/lua/b1.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
Skip ./reg-tests/lua/b3.vtc because exclude_targets
REASON: exclude_targets ',,' contains ''
$ echo $?
0
Regards.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Willy,
Hello Pieter,
Do you intend to finalize this script? We would like to use it in
haproxy sources.
Note that varnishtest already uses TMPDIR variable in place of /tmp if
it is set in the environment.
Thanks again.
Fred.
Thanks for your advices and comments, to be honest i haven't looked at
the script for several days, got distracted by other things ;). So sorry
for late reply.
Just cleaned it up a bit. I guess its ready for another review.
I still have a '
' newline, with the IFS= but the \n and \012 didnt seem to work there..
I've tried to incorporate all suggestions. Lemme know if/what i missed :)
Regards,
PiBa-NL (Pieter)
#!/usr/bin/env sh
if [ "$1" == "--help" ]; then
cat << EOF
### run-regtests.sh ###
Running run-regtests.sh --help shows this information about how to use it
Run without parameters to run all tests in the current folder (including
subfolders)
run-regtests.sh
Provide paths to run tests from (including subfolders):
run-regtests.sh ./tests1 ./tests2
Parameters:
--j , To run varnishtest with multiple jobs / threads for a faster
overall result
run-regtests.sh ./fasttest --j 16
--v, to run verbose
run-regtests.sh --v, disables the default varnishtest 'quiet' parameter
--varnishtestparams , passes custom ARGS to varnishtest
run-regtests.sh --varnishtestparams "-n 10"
Including text below into a .vtc file will check for its requirements
related to haproxy's target and compilation options
# Below targets are not capable of completing this test succesfully
#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd
#EXCLUDE_TARGETS=dos,freebsd,windows
# Below option is required to complete this test succesfully
#REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in.
#REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA
# To define a range of versions that a test can run with:
#REQUIRE_VERSION=0.0
#REQUIRE_VERSION_BELOW=99.9
Configure environment variables to set the haproxy and varnishtest binaries
to use
setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy
setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest
EOF
return
fi
_startswith() {
_str="$1"
_sub="$2"
echo "$_str" | grep "^$_sub" >/dev/null 2>&1
}
_findtests() {
#find "$1" -name "*.vtc" | while read i; do
IFS='
'
set -f
for i in $( find "$1" -name "*.vtc" ); do
#echo "TESTcheck '$i'"
skiptest=
require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')"
require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e
's/.*=//')"
require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')"
exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//'),"
if [ -n "$require_version" ]; then
if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ];
then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test requires at least version: $require_version"
skiptest=1
fi
fi
if [ -n "$require_version_below" ]; then
if [ $(_version "$HAPROXY_VERSION") -ge $(_version
"$require_version_below") ]; then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test requires a version below:
$require_version_below"
skiptest=1
fi
fi
if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then
echo " Skip $i because exclude_targets"
echo "REASON: exclude_targets '$exclude_targets' contains '$TARGET'"
skiptest=1
fi
#echo "REQUIRE_OPTIONS : $require_options"
for requiredoption in $(echo $require_options | tr "," "\012" ); do
if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ]
then
echo " Skip $i because option $requiredoption not found"
echo -n "REASON: "
echo -n "$required" | sed -e 's/.*,//' -e 's/^[[:space:]]//'
echo
skiptest=1
fi
done
for required in "$(grep "#REQUIRE_OPTION=" "$i")";
do
if [ -z "$required" ]
then
continue
fi
requiredoption=$(echo "$required" | sed -e 's/.*=//' -e 's/,.*//')
if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ]
then
echo " Skip $i because option $requiredoption not found"
echo -n "REASON: "
echo "$required" | sed -e 's/.*,//' -e 's/^[[:space:]]//'
skiptest=1
fi
done
testtarget=$(grep "#EXCLUDE_TARGET=" "$i")
if [ "$( echo "$testtarget" | grep "#EXCLUDE_TARGET=$TARGET," )" ]
then
echo " Skip $i because: TARGET = $TARGET"
echo -n "REASON: "
echo "$testtarget" | sed -e 's/.*,//' -e 's/^[[:space:]]//'
skiptest=1
fi
if [ -z $skiptest ]; then
echo " Add test: $i"
testlist="$testlist
$i"
fi
done
}
_process() {
jobcount=""
verbose="-q"
#
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/8/18 6:11 AM, Willy Tarreau wrote:
Hi guys,
On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote:
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Very good work.
I missed this message last week (I'm not often on the list these days
I confess), but this tool is awesome!
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
No.
Rename the old vtc files like that before running the tests:
bname=$(basename $i)
mv /tmp/$bname /tmp/old.$bname
with $i in $(find /tmp/ -type d -name "vtc.*")
or something like that.
It would be nice to support TMPDIR so that we can force it in a script
instead of /tmp. What I *think* could be done by default would be to
delete the old.vtc.* files there before running the tests so that we
always keep only the last results by default. I found hundreds of vtc
directories in /tmp on my laptop and that can become problematic, even
worse on shared development machines.
please *add -type d* to the find command to list the temporary varnishtest
directories.
Well, then add -maxdepth 1 or 2 because you certainly don't want to run
find into my /tmp which contains complete distros to chroot into!
And we should only keep temporaries directory of tests which have failed
with -l varnishtest option currently enabled option. They are all the
/tmp/vtc.* directories found after having run the tests.
Interesting.
So the script should list the remaining temporary directories after having
run the tests. There is no need to parse the log file of this directories.
Then I'd like to see a small change, which would be that :
1) we rely on a user-provided base directory instead of /tmp, such as
${TMPDIR:-/tmp} since TMPDIR is very commonly defined ;
Pieter,
Note that varnishtest already uses TMPDIR variable in place of /tmp if
it is set in the environment.
Regards,
Fred
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/8/18 6:11 AM, Willy Tarreau wrote:
Hi guys,
On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote:
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Very good work.
I missed this message last week (I'm not often on the list these days
I confess), but this tool is awesome!
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
No.
Rename the old vtc files like that before running the tests:
bname=$(basename $i)
mv /tmp/$bname /tmp/old.$bname
with $i in $(find /tmp/ -type d -name "vtc.*")
or something like that.
It would be nice to support TMPDIR so that we can force it in a script
instead of /tmp. What I *think* could be done by default would be to
delete the old.vtc.* files there before running the tests so that we
always keep only the last results by default. I found hundreds of vtc
directories in /tmp on my laptop and that can become problematic, even
worse on shared development machines.
please *add -type d* to the find command to list the temporary varnishtest
directories.
Well, then add -maxdepth 1 or 2 because you certainly don't want to run
find into my /tmp which contains complete distros to chroot into!
And we should only keep temporaries directory of tests which have failed
with -l varnishtest option currently enabled option. They are all the
/tmp/vtc.* directories found after having run the tests.
Interesting.
So the script should list the remaining temporary directories after having
run the tests. There is no need to parse the log file of this directories.
Then I'd like to see a small change, which would be that :
1) we rely on a user-provided base directory instead of /tmp, such as
${TMPDIR:-/tmp} since TMPDIR is very commonly defined ;
2) we create our own directory there using mkstemp and we display this
name when starting the tests ;
3) everything is created under this temporary directory, probably using
the test's basename, so that we don't have a mixture of old and new
tests at the same place, instead we have a single directory per run ;
4) we continue to delete successful entries from these directories,
and we can rmdir -p the temporary directory so that it completely
disappears if everything was successful.
It will then make it very convenient to figure what tests systematically
fail and compare the errors between runs.
We cannot rely on the fact there is no bug in varnishtest logging code ;)
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?
Yes a few details.
It seems there are remaining useless piped sed processes (sed | sed) or
perhaps I have missed something.
There is no need to use awk to remove sections of lines. Have a look to cut
command which is more lightweight.
It depends if there are multiple spaces or not.
tr command accepts escaped characters as '\n' for new line character which
is more readable than '
' ;)
Be careful, "\n" is a gnu-ism or something like this. The only portable
way to use \n is to use \012.
For me with these remaining modifications/cleanup this script could be added
to haproxy sources.
And please make the varnishtest program configurable as is done in the
makefile. This will help those of us who installed it in a location not
present in the path, and will also help support multiple versions during
development.
Thanks again for this good work Pieter.
Seconded!
Hello Pieter,
Do you intend to finalize this script? We would like to use it in
haproxy sources.
Thanks again.
Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi guys,
On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote:
> > New 'version' of the script attached.
> > It now supports a set of parameters to modify its behavior a little. And
> > also checking for a 'version requirement'. So a H2 test doesn't have to
> > fail on 1.7.
>
> Very good work.
I missed this message last week (I'm not often on the list these days
I confess), but this tool is awesome!
> > Should i 'ask' to delete old test result.? Or would there be a other
> > better way to keep previous results separated from the current run?
>
> No.
>
> Rename the old vtc files like that before running the tests:
>
> bname=$(basename $i)
> mv /tmp/$bname /tmp/old.$bname
>
> with $i in $(find /tmp/ -type d -name "vtc.*")
>
> or something like that.
It would be nice to support TMPDIR so that we can force it in a script
instead of /tmp. What I *think* could be done by default would be to
delete the old.vtc.* files there before running the tests so that we
always keep only the last results by default. I found hundreds of vtc
directories in /tmp on my laptop and that can become problematic, even
worse on shared development machines.
> please *add -type d* to the find command to list the temporary varnishtest
> directories.
Well, then add -maxdepth 1 or 2 because you certainly don't want to run
find into my /tmp which contains complete distros to chroot into!
> And we should only keep temporaries directory of tests which have failed
> with -l varnishtest option currently enabled option. They are all the
> /tmp/vtc.* directories found after having run the tests.
Interesting.
> So the script should list the remaining temporary directories after having
> run the tests. There is no need to parse the log file of this directories.
Then I'd like to see a small change, which would be that :
1) we rely on a user-provided base directory instead of /tmp, such as
${TMPDIR:-/tmp} since TMPDIR is very commonly defined ;
2) we create our own directory there using mkstemp and we display this
name when starting the tests ;
3) everything is created under this temporary directory, probably using
the test's basename, so that we don't have a mixture of old and new
tests at the same place, instead we have a single directory per run ;
4) we continue to delete successful entries from these directories,
and we can rmdir -p the temporary directory so that it completely
disappears if everything was successful.
It will then make it very convenient to figure what tests systematically
fail and compare the errors between runs.
> We cannot rely on the fact there is no bug in varnishtest logging code ;)
>
> > If you could give it another look i would appreciate that. Are there any
> > things that need to be added/changed about it before considering to add
> > it to haproxy sources branch?
>
> Yes a few details.
>
> It seems there are remaining useless piped sed processes (sed | sed) or
> perhaps I have missed something.
>
> There is no need to use awk to remove sections of lines. Have a look to cut
> command which is more lightweight.
It depends if there are multiple spaces or not.
> tr command accepts escaped characters as '\n' for new line character which
> is more readable than '
> ' ;)
Be careful, "\n" is a gnu-ism or something like this. The only portable
way to use \n is to use \012.
> For me with these remaining modifications/cleanup this script could be added
> to haproxy sources.
And please make the varnishtest program configurable as is done in the
makefile. This will help those of us who installed it in a location not
present in the path, and will also help support multiple versions during
development.
> Thanks again for this good work Pieter.
Seconded!
Willy
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/28/18 8:01 PM, PiBa-NL wrote:
Hi Frederic,
Hello Pieter,
Sorry again for this late reply.
Op 19-10-2018 om 11:51 schreef Frederic Lecaille:
The idea of the script sounds good to me. About the script itself it
is a nice work which could be a good start.
Thanks.
Just a few details below.
Note that "cat $file | grep something" may be shortened by "grep
something $file". It would also be interesting to avoid creating
temporary files as most as possible (at least testflist.lst is not
necessary I think).
TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //')
OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //')
may be shortened by these lines:
{ read TARGET; read OPTIONS; } << EOF
$(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //')
EOF
Thanks, I've changed this.
which is portable, or something similar.
sed 's/.*=//'| sed 's/,.*//'
may be shortened by
sed -e 's/.*=//' -e 's/,.*//'
Thanks, I've changed this as well.
Also note, there are some cases where options are enabled without
appearing in OPTIONS variable value.
For instance if you compile haproxy like that:
$ make TARGET=linux2628
the support for the thread is enabled without appearing in OPTIONS
variable value. I am not sure this is an issue at this time.
That could become an issue. but should be easy to solve adding a 'fake'
option perhaps to check against.. Or adding a separate check perhaps I'm
not sure yet.
With one test like that after having parsed OPTIONS, it would suffice at
this time :
if [ $TARGET = linux2628 ] ; then
OPTIONS="$OPTIONS USE_xxx USE_xxx"
fi
with USE_xxx the listed compilation options found in the Makefile file:
ifeq ($(TARGET),linux2628)
# This is for standard Linux >= 2.6.28 with netfilter, epoll, tproxy
and splice
USE_NETFILTER = implicit
USE_POLL= implicit
USE_EPOLL = implicit
USE_TPROXY = implicit
USE_CRYPT_H = implicit
USE_LIBCRYPT= implicit
USE_LINUX_SPLICE= implicit
USE_LINUX_TPROXY= implicit
USE_ACCEPT4 = implicit
USE_FUTEX = implicit
USE_CPU_AFFINITY= implicit
ASSUME_SPLICE_WORKS= implicit
USE_DL = implicit
USE_RT = implicit
USE_THREAD = implicit
else
Perhaps we could could only one line for the required options and
excluded targets like that:
#EXCLUDED_TARGETS=freebsd,dos,windows ;)
#REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB
Added this option, but then i like the option of excluding single
targets and having some comment behind it explaining the reason.. But i
guess if one would want to know some comments above the setting could
also 'explain' why that target is currently not 'valid' to run the test
on. Should i remove the settings for the 'single' option/target.?
No. It is ok at this time.
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Very good work.
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
No.
Rename the old vtc files like that before running the tests:
bname=$(basename $i)
mv /tmp/$bname /tmp/old.$bname
with $i in $(find /tmp/ -type d -name "vtc.*")
or something like that.
please *add -type d* to the find command to list the temporary
varnishtest directories.
And we should only keep temporaries directory of tests which have failed
with -l varnishtest option currently enabled option. They are all the
/tmp/vtc.* directories found after having run the tests.
So the script should list the remaining temporary directories after
having run the tests. There is no need to parse the log file of this
directories. We cannot rely on the fact there is no bug in varnishtest
logging code ;)
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?
Yes a few details.
It seems there are remaining useless piped sed processes (sed | sed) or
perhaps I have missed something.
There is no need to use awk to remove sections of lines. Have a look to
cut command which is more lightweight.
tr command accepts escaped characters as '\n' for new line character
which is more readable than '
' ;)
For me with these remaining modifications/cleanup this script could be
added to haproxy sources.
Thanks again for this good work Pieter.
Regards,
Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic,
Op 19-10-2018 om 11:51 schreef Frederic Lecaille:
The idea of the script sounds good to me. About the script itself it
is a nice work which could be a good start.
Thanks.
Just a few details below.
Note that "cat $file | grep something" may be shortened by "grep
something $file". It would also be interesting to avoid creating
temporary files as most as possible (at least testflist.lst is not
necessary I think).
TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //')
OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //')
may be shortened by these lines:
{ read TARGET; read OPTIONS; } << EOF
$(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //')
EOF
Thanks, I've changed this.
which is portable, or something similar.
sed 's/.*=//'| sed 's/,.*//'
may be shortened by
sed -e 's/.*=//' -e 's/,.*//'
Thanks, I've changed this as well.
Also note, there are some cases where options are enabled without
appearing in OPTIONS variable value.
For instance if you compile haproxy like that:
$ make TARGET=linux2628
the support for the thread is enabled without appearing in OPTIONS
variable value. I am not sure this is an issue at this time.
That could become an issue. but should be easy to solve adding a 'fake'
option perhaps to check against.. Or adding a separate check perhaps I'm
not sure yet.
Perhaps we could could only one line for the required options and
excluded targets like that:
#EXCLUDED_TARGETS=freebsd,dos,windows ;)
#REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB
Added this option, but then i like the option of excluding single
targets and having some comment behind it explaining the reason.. But i
guess if one would want to know some comments above the setting could
also 'explain' why that target is currently not 'valid' to run the test
on. Should i remove the settings for the 'single' option/target.?
New 'version' of the script attached.
It now supports a set of parameters to modify its behavior a little. And
also checking for a 'version requirement'. So a H2 test doesn't have to
fail on 1.7.
Should i 'ask' to delete old test result.? Or would there be a other
better way to keep previous results separated from the current run?
If you could give it another look i would appreciate that. Are there any
things that need to be added/changed about it before considering to add
it to haproxy sources branch?
Regards,
PiBa-NL (Pieter)
#!/usr/bin/env sh
if [ "$1" == "--help" ]; then
cat << EOF
### run-regtests.sh ###
Running run-regtests.sh --help shows this information about how to use it
Run without parameters to run all tests in the current folder (including
subfolders)
run-regtests.sh
Provide paths to run tests from (including subfolders):
run-regtests.sh ./tests1 ./tests2
Parameters:
--j , To run varnishtest with multiple jobs / threads for a faster
overall result
run-regtests.sh ./fasttest --j 16
--v, to run verbose
run-regtests.sh --v, disables the default varnishtest 'quiet' parameter
--varnishtestparams , passes custom ARGS to varnishtest
run-regtests.sh --varnishtestparams "-n 10"
--f, force deleting old /tmp/*.vtc results without asking
Including text below into a .vtc file will check for its requirements
related to haproxy's target and compilation options
# Below targets are not capable of completing this test succesfully
#EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd
#EXCLUDE_TARGETS=dos,freebsd,windows
# Below option is required to complete this test succesfully
#REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in.
#REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA
# To define a range of versions that a test can run with:
#REQUIRE_VERSION=0.0
#REQUIRE_VERSION_BELOW=99.9
EOF
return
fi
_startswith() {
_str="$1"
_sub="$2"
echo "$_str" | grep "^$_sub" >/dev/null 2>&1
}
_findtests() {
#find "$1" -name "*.vtc" | while read i; do
IFS='
'
set -f
for i in $( find "$1" -name "*.vtc" ); do
#echo "TESTcheck '$i'"
skiptest=
require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')"
require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e
's/.*=//')"
require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')"
exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//'),"
if [ -n "$require_version" ]; then
if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ];
then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test requires at least version: $require_version"
skiptest=1
fi
fi
if [ -n "$require_version_below" ]; then
if [ $(_version "$HAPROXY_VERSION") -ge $(_version
"$require_version_below") ]; then
echo " Skip $i because option haproxy is version: $HAPROXY_VERSION"
echo "REASON: this test re
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/19/2018 12:43 AM, PiBa-NL wrote:
Hi Frederic,
Hi Pieter,
Do you have a little time to take a look at (the idea of) the script?
The idea of the script sounds good to me. About the script itself it is
a nice work which could be a good start.
Just a few details below.
Note that "cat $file | grep something" may be shortened by "grep
something $file". It would also be interesting to avoid creating
temporary files as most as possible (at least testflist.lst is not
necessary I think).
TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //')
OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //')
may be shortened by these lines:
{ read TARGET; read OPTIONS; } << EOF
$(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //')
EOF
which is portable, or something similar.
sed 's/.*=//'| sed 's/,.*//'
may be shortened by
sed -e 's/.*=//' -e 's/,.*//'
Also note, there are some cases where options are enabled without
appearing in OPTIONS variable value.
For instance if you compile haproxy like that:
$ make TARGET=linux2628
the support for the thread is enabled without appearing in OPTIONS
variable value. I am not sure this is an issue at this time.
Perhaps we could could only one line for the required options and
excluded targets like that:
#EXCLUDED_TARGETS=freebsd,dos,windows ;)
#REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB
Thank you a lot.
Thanks in advance :).
Op 3-10-2018 om 21:52 schreef PiBa-NL:
Hi Frederic,
Made a little script (attached)..
What do you think ? Could this work for current 'requirements'?
Regards,
PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Do you have a little time to take a look at (the idea of) the script? Thanks in advance :). Op 3-10-2018 om 21:52 schreef PiBa-NL: Hi Frederic, Made a little script (attached).. What do you think ? Could this work for current 'requirements'? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 2-10-2018 om 8:58 schreef Frederic Lecaille: At this time I do not know how to skip the tests which are not supported due to disable features at compilation time without running a "haproxy -vv" command before launching the VTC files. Made a little script (attached).. It would allow for the .vtc files to contain what 'requirements' they have, by including lines like these: #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #REQUIRE_OPTION=OPENSSL, this test needs ssl compiled in. Which could then generate output like listed below. It searches for all testcases, validates their requirements, and then puts them in a list. After which it calls varnishtest with quiet and simultaneous job parameters to allow for a short turnaround time.. What do you think ? Could this work for current 'requirements'? Or is something different preferred? I though about running each test directly from the script, and while that allows per test OK/Fail response logging for and a nicer overall summery, it takes more time to run. So i went for automatically creating a list of tests and letting varnishtest itself handle the running of multiple tests at once. Thoughts appreciated :). Regards, PiBa-NL (Pieter) root@freebsd11:/usr/ports/net/haproxy-devel # ./run-regtests.sh ## Preparing to run tests ## HA-Proxy version 1.8.14-52e4d43 2018/09/20 Copyright 2000-2018 Willy Tarreau ## Gathering tests to run ## Skip ./haproxy_test_OK_20180831/lua/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./haproxy_test_OK_20180831/lua/h1.vtc Add test: ./haproxy_test_OK_20180831/lua/b0.vtc Add test: ./haproxy_test_OK_20180831/lua/b1.vtc Add test: ./haproxy_test_OK_20180831/loadtest/b0-loadtest.vtc Add test: ./haproxy_test_OK_20180831/log/b0.vtc Skip ./haproxy_test_OK_20180831/reload/b1-server-state-file.vtc because: TARGET = freebsd REASON: abns sockets are not available on freebsd Skip ./haproxy_test_OK_20180831/ssl/b0.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./haproxy_test_OK_20180831/stick-table/b1.vtc Add test: ./haproxy_test_OK_20180831/stick-table/b0.vtc Add test: ./test/b0-loadtest.vtc Skip ./test/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./test/b3-cpu.vtc Add test: ./work/haproxy-1.8-52e4d43/reg-tests/server/b0.vtc Add test: ./work/haproxy-1.8-52e4d43/reg-tests/stick-table/h0.vtc Skip ./work/haproxy-1.8-52e4d43/reg-tests/lua/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./work/haproxy-1.8-52e4d43/reg-tests/lua/h1.vtc ## Starting varnishtest ## # top TEST ./test/b3-cpu.vtc FAILED (5.281) exit=2 1 tests failed, 0 tests skipped, 11 tests passed root@freebsd11:/usr/ports/net/haproxy-devel # #!/usr/bin/env sh echo "## Preparing to run tests ##" haproxy -v #varnishtest -j 16 -k -t 20 ./work/haproxy-*/reg-tests/*/*.vtc > ./mytest-result.log 2>&1 #varnishtest -j 16 -k -t 20 ./haproxy_test_OK_20180831/*/*.vtc >> ./mytest-result.log 2>&1 #cat ./mytest-result.log #echo "" >> ./mytest-result.log #haproxy -vv >> ./mytest-result.log TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') echo "TARGET : $TARGET" echo "OPTIONS : $OPTIONS" haproxy -v > testresult_summary.log haproxy -v > testresult_detailed.log echo -n > testlist.lst echo "## Gathering tests to run ##" for i in $(find ./ -name "*.vtc"); do optionismissing= for required in "$(cat $i | grep REQUIRE_OPTION)"; do if [ -z "$required" ] then continue fi requiredoption=$(echo "$required" | sed 's/.*=//' | sed 's/,.*//') #echo " Test requires: $requiredoption" if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ] then echo -n "Skip $i because option $requiredoption not found! REASON: " echo "$required" | sed 's/.*,//' optionismissing=1 fi done if [ $optionismissing ] then continue fi testtarget=$(cat $i | grep "#EXCLUDE_TARGET") #echo "testtarget : $testtarget" if [ "$( echo "$testtarget" | grep "#EXCLUDE_TARGET=$TARGET," )" ] then echo "Skip $i because: TARGET = $TARGET" echo -n " REASON: " echo "$testtarget" | sed 's/.*,//' continue fi echo
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/01/2018 08:59 PM, PiBa-NL wrote: Hi Frederic, Op 1-10-2018 om 16:09 schreef Frederic Lecaille: - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. Should i send a patch changing that? Or can you fix it easily ;) Yes, feel free to send a patch for that. - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. Afaik the 'check' on the server line is failing to send a GET / request.. My other mail has a bit more ideas about that.. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. I think creating a list of tests that could be run on FreeBSD will take a lot of maintenance and i assume 'most' tests will actually be runnable on most OS's. So having short list with exceptions is probably easier to maintain. In my opinion every test should be run on every os unless there is some good reason not to run it (abns/splicing/stuff..). And if possible the reason to exclude a specific test should be described (a single line of text could be enough). I did not mention to create new tests for each OSes. All VTC files would be created in directories below reg-tests directories. They would be the union of all VTC files supported by all the OSes supported by HAProxy. reg-tests directory would contain directories for each OSes with symbolic links to the directories previously mentioned, if they contain VTC files supported by this OS. Something like that: reg-tests log abns linux log -> ../log abns -> ../abns freebsd log -> ../log The even bigger issue that Willy raised is with compilation options not including lua/ssl/gzip/threads/ stuff, and having lots of tests fail on those, while they should get skipped if they rely on a feature not compiled. I discussed this also a bit with Willy in the other mail thread ( https://www.mail-archive.com/haproxy@formilux.org/msg31345.html ), but i don't think we have defined the perfect way to do it yet.. So that part of the question still stands ;) .. Whats the best way to skip tests that are not applicable? At this time I do not know how to skip the tests which are not supported due to disable features at compilation time without running a "haproxy -vv" command before launching the VTC files. Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 1-10-2018 om 16:09 schreef Frederic Lecaille: - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. Should i send a patch changing that? Or can you fix it easily ;) - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. Afaik the 'check' on the server line is failing to send a GET / request.. My other mail has a bit more ideas about that.. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. I think creating a list of tests that could be run on FreeBSD will take a lot of maintenance and i assume 'most' tests will actually be runnable on most OS's. So having short list with exceptions is probably easier to maintain. In my opinion every test should be run on every os unless there is some good reason not to run it (abns/splicing/stuff..). And if possible the reason to exclude a specific test should be described (a single line of text could be enough). The even bigger issue that Willy raised is with compilation options not including lua/ssl/gzip/threads/ stuff, and having lots of tests fail on those, while they should get skipped if they rely on a feature not compiled. I discussed this also a bit with Willy in the other mail thread ( https://www.mail-archive.com/haproxy@formilux.org/msg31345.html ), but i don't think we have defined the perfect way to do it yet.. So that part of the question still stands ;) .. Whats the best way to skip tests that are not applicable? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 1-10-2018 om 16:19 schreef Frederic Lecaille: On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) This test relies on timeout values to produce the correct syslog output. Relying on timing values for reg testing is perhaps a bad idea as the results are not deterministic. Indeed here we want the 5ms timeout client to expire before it reads the server response after having waited for 20ms. I propose to remove this test. Regarding /log/b0.vtc i think the test itself is 'mostly' okay. If we keep the check i do think its 'check' on the server line should be changed by one of the following options: - removing 'check' - check to different port to a s2 server that does not use txresp - add 'option httpchk' and repeat s1 twice.. Any of the changes above would 'fix' the biggest issue with the test as far as i see.. But indeed having short timeouts could cause on occasion cause random failures if for example multiple tests are run simultaneously.. The test for a decent log line with a 'client timout' flag is probably something that should be tested, perhaps the c1 delay could be set to something longer to 'enforce' the 'timeout client 5' would always hit first even when stuff starts competing for (cpu) resources.? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) This test relies on timeout values to produce the correct syslog output. Relying on timing values for reg testing is perhaps a bad idea as the results are not deterministic. Indeed here we want the 5ms timeout client to expire before it reads the server response after having waited for 20ms. I propose to remove this test.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, Hi, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. With a few small changes (attached) i can run all tests like this, get the following result: varnishtest -l ./work/haproxy-ss-20180901/reg-tests/*/*.vtc # top TEST ./work/haproxy-ss-20180901/reg-tests/connection/b0.vtc passed (0.142) # top TEST ./work/haproxy-ss-20180901/reg-tests/log/b0.vtc passed (0.136) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b0.vtc passed (0.121) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b1.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b2.vtc passed (0.186) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b3.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/h1.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/seamless-reload/b0.vtc passed (0.143) # top TEST ./work/haproxy-ss-20180901/reg-tests/spoe/b0.vtc passed (0.011) # top TEST ./work/haproxy-ss-20180901/reg-tests/ssl/b0.vtc passed (0.146) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b0.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b1.vtc passed (0.122) Above would be good :).. But well.. needs changing or skipping some tests.. I would like to know next time that 'all' testable reg-tests are working properly when making a new build to take into my production system. If some tests fail by design (on this platform), it takes more administration to figure out if that was okay or not. Please advice :) Regards, PiBa-NL (Pieter)
