Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/23/18 1:12 AM, PiBa-NL wrote: Hi Frederic, Hi Pieter, I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. Strangely on my PC with both bash and dash I do not have to change IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read" internal command. Reading version,target and options works fine indeed without, however the loop over test files fails if any .vtc file has a space character in its filename. Or should we 'forbid' that with documentation.? Or is there another better workaround for that? We will never use VTC files with spaces in their filenames. I do not think it is a good idea to build TESTDIR like that: TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')" TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME mkdir -p $TESTDIR What if we run the tests several times at the same time? Well they would have to run at the same second. Not sure if that would be wise to do.. But mktemp now solved that i guess :) at least for the directory name part.. Please have a look to mkstemp utility. Without the 's' right? Done, combined with the rundatetime which i do like, so its 'readable' and unique, best of both ways i think?. Yes. Remaining details: cat $i/LOG | grep -- should be replaced by grep -- $i/LOG I guess ill never learn do do this right the first time around ;). Fixed. Note that you script is plenty of '\r' characters with no newline character at the end of file position: Not sure what the 'correct' way would be. I think there is a CR LF everywhere at the moment? And the scripts hashbang tries to point to 'sh', will this be a issue.? (acme.sh does the same, and seems to be run an lots of systems..) And if so what can i best do to avoid issues? Yes there are \r\n characters in place of \n end of line character for each line. You can check that with hexdump or od: $ hexdump -c run-regtests.sh | head 000 # ! / u s r / b i n / e n v s 010 h \r \n \r \n i f [ " $ 1 " = 020 " - - h e l p " ] ; t h e 030 n \r \n c a t < < E O F \r 040 \n # # # r u n - r e g t e s t 050 s . s h # # # \r \n R u n n 060 i n g r u n - r e g t e s t s 070 . s h - - h e l p s h o w s 080 t h i s i n f o r m a t i o 090 n a b o u t h o w t o u There are chances you are using a text file editor on a non-Unix system? You can remove them like that before cat run-regtests.sh | tr -d '\r' >run-regtests.sh.ok Also note that some shells do not like == operator (at line 3): Used a single = now. When I do not set both HAPROY_PROGRAM I get this output with a script with a successful result. Checks added to avoid this issue for both haproxy and varnishtest so they are checked to exist. Next round :). Shoulb be the last ;) Thanks a lot Pieter. Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. Strangely on my PC with both bash and dash I do not have to change IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read" internal command. Reading version,target and options works fine indeed without, however the loop over test files fails if any .vtc file has a space character in its filename. Or should we 'forbid' that with documentation.? Or is there another better workaround for that? I do not think it is a good idea to build TESTDIR like that: TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')" TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME mkdir -p $TESTDIR What if we run the tests several times at the same time? Well they would have to run at the same second. Not sure if that would be wise to do.. But mktemp now solved that i guess :) at least for the directory name part.. Please have a look to mkstemp utility. Without the 's' right? Done, combined with the rundatetime which i do like, so its 'readable' and unique, best of both ways i think?. Remaining details: cat $i/LOG | grep -- should be replaced by grep -- $i/LOG I guess ill never learn do do this right the first time around ;). Fixed. Note that you script is plenty of '\r' characters with no newline character at the end of file position: Not sure what the 'correct' way would be. I think there is a CR LF everywhere at the moment? And the scripts hashbang tries to point to 'sh', will this be a issue.? (acme.sh does the same, and seems to be run an lots of systems..) And if so what can i best do to avoid issues? Also note that some shells do not like == operator (at line 3): Used a single = now. When I do not set both HAPROY_PROGRAM I get this output with a script with a successful result. Checks added to avoid this issue for both haproxy and varnishtest so they are checked to exist. Next round :). Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" = "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 Configure environment variables to set the haproxy and varnishtest binaries to use setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$require_version" ]; then if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires at least version: $require_version" skiptest=1 fi fi if [ -n "$require_version_below" ]; then if [ $(_version "$HAPROXY_VERSION") -ge $(_version "$require_version_below") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires a version below: $require_version_below" skiptest=1 fi fi if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then echo " Skip $i because exclude_targets" echo "REASON: exclude_targets '$exclude_targets' contains '$TARGET'" skiptest=1 fi #echo "REQUIRE_OPTIONS :
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/19/18 10:08 PM, PiBa-NL wrote: Hi Frederic, Willy, Hi Pieter, Thank you a lot again for this work Pieter. Hello Pieter, Do you intend to finalize this script? We would like to use it in haproxy sources. Note that varnishtest already uses TMPDIR variable in place of /tmp if it is set in the environment. Thanks again. Fred. Thanks for your advices and comments, to be honest i haven't looked at the script for several days, got distracted by other things ;). So sorry for late reply. No problem. Sorry for my late reply too ;) Just cleaned it up a bit. I guess its ready for another review. I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. Strangely on my PC with both bash and dash I do not have to change IFS value to parse HAPROXY_VERSION, TARGET and OPTIONS with "read" internal command. I've tried to incorporate all suggestions. Lemme know if/what i missed :) I do not think it is a good idea to build TESTDIR like that: TESTRUNDATETIME="$(date '+%Y-%m-%d_%H-%M-%S')" TESTDIR=${TMPDIR:-/tmp}/varnishtest_haproxy/$TESTRUNDATETIME mkdir -p $TESTDIR What if we run the tests several times at the same time? Please have a look to mkstemp utility. Remaining details: cat $i/LOG | grep -- should be replaced by grep -- $i/LOG Note that you script is plenty of '\r' characters with no newline character at the end of file position: $ sh run-regtests.sh : not founds.sh: 2: run-regtests.sh: run-regtests.sh: 58: run-regtests.sh: Syntax error: word unexpected (expecting "do") flecaille@fl-haproxy:~/src/haproxy $ bash -x run-regtests.sh + $'\r' run-regtests.sh: line 2: $'\r': command not found run-regtests.sh: line 47: syntax error near unexpected token `$'{\r'' 'un-regtests.sh: line 47: `_startswith() { Also note that some shells do not like == operator (at line 3): $ HAPROXY_PROGRAM=~/src/haproxy/haproxy VARNISHTEST_PROGRAM=~/src/varnish-cache-trunk/bin/varnishtest/varnishtest sh run-regtests.sh run-regtests.sh: 3: [: unexpected operator When I do no set both HAPROY_PROGRAM I get this output with a script with a successful result: $ sh run-regtests.sh ## Preparing to run tests ## run-regtests.sh.ok: 177: run-regtests.sh.ok: haproxy: not found Testing with haproxy version: run-regtests.sh.ok: 192: [: =: unexpected operator run-regtests.sh.ok: 196: [: =: unexpected operator run-regtests.sh.ok: 200: [: =: unexpected operator run-regtests.sh.ok: 204: [: =: unexpected operator run-regtests.sh.ok: 208: [: =: unexpected operator run-regtests.sh.ok: 212: [: =: unexpected operator run-regtests.sh.ok: 216: [: =: unexpected operator run-regtests.sh.ok: 220: [: =: unexpected operator run-regtests.sh.ok: 224: [: =: unexpected operator run-regtests.sh.ok: 228: [: =: unexpected operator run-regtests.sh.ok: 232: [: =: unexpected operator run-regtests.sh.ok: 236: [: =: unexpected operator run-regtests.sh.ok: 240: [: =: unexpected operator run-regtests.sh.ok: 244: [: =: unexpected operator run-regtests.sh.ok: 248: [: =: unexpected operator Target : Options : ## Gathering tests to run ## Skip ./h2-keepalive-backend.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./urlp.regex.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/ssl/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/spoe/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/log/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/stick-table/b1.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/stick-table/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/server/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/seamless-reload/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/connection/b0.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/lua/b1.vtc because exclude_targets REASON: exclude_targets ',,' contains '' Skip ./reg-tests/lua/b3.vtc because exclude_targets REASON: exclude_targets ',,' contains '' $ echo $? 0 Regards.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Willy, Hello Pieter, Do you intend to finalize this script? We would like to use it in haproxy sources. Note that varnishtest already uses TMPDIR variable in place of /tmp if it is set in the environment. Thanks again. Fred. Thanks for your advices and comments, to be honest i haven't looked at the script for several days, got distracted by other things ;). So sorry for late reply. Just cleaned it up a bit. I guess its ready for another review. I still have a ' ' newline, with the IFS= but the \n and \012 didnt seem to work there.. I've tried to incorporate all suggestions. Lemme know if/what i missed :) Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" == "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 Configure environment variables to set the haproxy and varnishtest binaries to use setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy setenv VARNISHTEST_PROGRAM /usr/local/bin/varnishtest EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do IFS=' ' set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$require_version" ]; then if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires at least version: $require_version" skiptest=1 fi fi if [ -n "$require_version_below" ]; then if [ $(_version "$HAPROXY_VERSION") -ge $(_version "$require_version_below") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires a version below: $require_version_below" skiptest=1 fi fi if [ -n "$( echo "$exclude_targets" | grep ",$TARGET," )" ]; then echo " Skip $i because exclude_targets" echo "REASON: exclude_targets '$exclude_targets' contains '$TARGET'" skiptest=1 fi #echo "REQUIRE_OPTIONS : $require_options" for requiredoption in $(echo $require_options | tr "," "\012" ); do if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ] then echo " Skip $i because option $requiredoption not found" echo -n "REASON: " echo -n "$required" | sed -e 's/.*,//' -e 's/^[[:space:]]//' echo skiptest=1 fi done for required in "$(grep "#REQUIRE_OPTION=" "$i")"; do if [ -z "$required" ] then continue fi requiredoption=$(echo "$required" | sed -e 's/.*=//' -e 's/,.*//') if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ] then echo " Skip $i because option $requiredoption not found" echo -n "REASON: " echo "$required" | sed -e 's/.*,//' -e 's/^[[:space:]]//' skiptest=1 fi done testtarget=$(grep "#EXCLUDE_TARGET=" "$i") if [ "$( echo "$testtarget" | grep "#EXCLUDE_TARGET=$TARGET," )" ] then echo " Skip $i because: TARGET = $TARGET" echo -n "REASON: " echo "$testtarget" | sed -e 's/.*,//' -e 's/^[[:space:]]//' skiptest=1 fi if [ -z $skiptest ]; then echo " Add test: $i" testlist="$testlist $i" fi done } _process() { jobcount="" verbose="-q" #
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/8/18 6:11 AM, Willy Tarreau wrote: Hi guys, On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote: New 'version' of the script attached. It now supports a set of parameters to modify its behavior a little. And also checking for a 'version requirement'. So a H2 test doesn't have to fail on 1.7. Very good work. I missed this message last week (I'm not often on the list these days I confess), but this tool is awesome! Should i 'ask' to delete old test result.? Or would there be a other better way to keep previous results separated from the current run? No. Rename the old vtc files like that before running the tests: bname=$(basename $i) mv /tmp/$bname /tmp/old.$bname with $i in $(find /tmp/ -type d -name "vtc.*") or something like that. It would be nice to support TMPDIR so that we can force it in a script instead of /tmp. What I *think* could be done by default would be to delete the old.vtc.* files there before running the tests so that we always keep only the last results by default. I found hundreds of vtc directories in /tmp on my laptop and that can become problematic, even worse on shared development machines. please *add -type d* to the find command to list the temporary varnishtest directories. Well, then add -maxdepth 1 or 2 because you certainly don't want to run find into my /tmp which contains complete distros to chroot into! And we should only keep temporaries directory of tests which have failed with -l varnishtest option currently enabled option. They are all the /tmp/vtc.* directories found after having run the tests. Interesting. So the script should list the remaining temporary directories after having run the tests. There is no need to parse the log file of this directories. Then I'd like to see a small change, which would be that : 1) we rely on a user-provided base directory instead of /tmp, such as ${TMPDIR:-/tmp} since TMPDIR is very commonly defined ; Pieter, Note that varnishtest already uses TMPDIR variable in place of /tmp if it is set in the environment. Regards, Fred
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 11/8/18 6:11 AM, Willy Tarreau wrote: Hi guys, On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote: New 'version' of the script attached. It now supports a set of parameters to modify its behavior a little. And also checking for a 'version requirement'. So a H2 test doesn't have to fail on 1.7. Very good work. I missed this message last week (I'm not often on the list these days I confess), but this tool is awesome! Should i 'ask' to delete old test result.? Or would there be a other better way to keep previous results separated from the current run? No. Rename the old vtc files like that before running the tests: bname=$(basename $i) mv /tmp/$bname /tmp/old.$bname with $i in $(find /tmp/ -type d -name "vtc.*") or something like that. It would be nice to support TMPDIR so that we can force it in a script instead of /tmp. What I *think* could be done by default would be to delete the old.vtc.* files there before running the tests so that we always keep only the last results by default. I found hundreds of vtc directories in /tmp on my laptop and that can become problematic, even worse on shared development machines. please *add -type d* to the find command to list the temporary varnishtest directories. Well, then add -maxdepth 1 or 2 because you certainly don't want to run find into my /tmp which contains complete distros to chroot into! And we should only keep temporaries directory of tests which have failed with -l varnishtest option currently enabled option. They are all the /tmp/vtc.* directories found after having run the tests. Interesting. So the script should list the remaining temporary directories after having run the tests. There is no need to parse the log file of this directories. Then I'd like to see a small change, which would be that : 1) we rely on a user-provided base directory instead of /tmp, such as ${TMPDIR:-/tmp} since TMPDIR is very commonly defined ; 2) we create our own directory there using mkstemp and we display this name when starting the tests ; 3) everything is created under this temporary directory, probably using the test's basename, so that we don't have a mixture of old and new tests at the same place, instead we have a single directory per run ; 4) we continue to delete successful entries from these directories, and we can rmdir -p the temporary directory so that it completely disappears if everything was successful. It will then make it very convenient to figure what tests systematically fail and compare the errors between runs. We cannot rely on the fact there is no bug in varnishtest logging code ;) If you could give it another look i would appreciate that. Are there any things that need to be added/changed about it before considering to add it to haproxy sources branch? Yes a few details. It seems there are remaining useless piped sed processes (sed | sed) or perhaps I have missed something. There is no need to use awk to remove sections of lines. Have a look to cut command which is more lightweight. It depends if there are multiple spaces or not. tr command accepts escaped characters as '\n' for new line character which is more readable than ' ' ;) Be careful, "\n" is a gnu-ism or something like this. The only portable way to use \n is to use \012. For me with these remaining modifications/cleanup this script could be added to haproxy sources. And please make the varnishtest program configurable as is done in the makefile. This will help those of us who installed it in a location not present in the path, and will also help support multiple versions during development. Thanks again for this good work Pieter. Seconded! Hello Pieter, Do you intend to finalize this script? We would like to use it in haproxy sources. Thanks again. Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi guys, On Wed, Nov 07, 2018 at 09:37:14AM +0100, Frederic Lecaille wrote: > > New 'version' of the script attached. > > It now supports a set of parameters to modify its behavior a little. And > > also checking for a 'version requirement'. So a H2 test doesn't have to > > fail on 1.7. > > Very good work. I missed this message last week (I'm not often on the list these days I confess), but this tool is awesome! > > Should i 'ask' to delete old test result.? Or would there be a other > > better way to keep previous results separated from the current run? > > No. > > Rename the old vtc files like that before running the tests: > > bname=$(basename $i) > mv /tmp/$bname /tmp/old.$bname > > with $i in $(find /tmp/ -type d -name "vtc.*") > > or something like that. It would be nice to support TMPDIR so that we can force it in a script instead of /tmp. What I *think* could be done by default would be to delete the old.vtc.* files there before running the tests so that we always keep only the last results by default. I found hundreds of vtc directories in /tmp on my laptop and that can become problematic, even worse on shared development machines. > please *add -type d* to the find command to list the temporary varnishtest > directories. Well, then add -maxdepth 1 or 2 because you certainly don't want to run find into my /tmp which contains complete distros to chroot into! > And we should only keep temporaries directory of tests which have failed > with -l varnishtest option currently enabled option. They are all the > /tmp/vtc.* directories found after having run the tests. Interesting. > So the script should list the remaining temporary directories after having > run the tests. There is no need to parse the log file of this directories. Then I'd like to see a small change, which would be that : 1) we rely on a user-provided base directory instead of /tmp, such as ${TMPDIR:-/tmp} since TMPDIR is very commonly defined ; 2) we create our own directory there using mkstemp and we display this name when starting the tests ; 3) everything is created under this temporary directory, probably using the test's basename, so that we don't have a mixture of old and new tests at the same place, instead we have a single directory per run ; 4) we continue to delete successful entries from these directories, and we can rmdir -p the temporary directory so that it completely disappears if everything was successful. It will then make it very convenient to figure what tests systematically fail and compare the errors between runs. > We cannot rely on the fact there is no bug in varnishtest logging code ;) > > > If you could give it another look i would appreciate that. Are there any > > things that need to be added/changed about it before considering to add > > it to haproxy sources branch? > > Yes a few details. > > It seems there are remaining useless piped sed processes (sed | sed) or > perhaps I have missed something. > > There is no need to use awk to remove sections of lines. Have a look to cut > command which is more lightweight. It depends if there are multiple spaces or not. > tr command accepts escaped characters as '\n' for new line character which > is more readable than ' > ' ;) Be careful, "\n" is a gnu-ism or something like this. The only portable way to use \n is to use \012. > For me with these remaining modifications/cleanup this script could be added > to haproxy sources. And please make the varnishtest program configurable as is done in the makefile. This will help those of us who installed it in a location not present in the path, and will also help support multiple versions during development. > Thanks again for this good work Pieter. Seconded! Willy
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/28/18 8:01 PM, PiBa-NL wrote: Hi Frederic, Hello Pieter, Sorry again for this late reply. Op 19-10-2018 om 11:51 schreef Frederic Lecaille: The idea of the script sounds good to me. About the script itself it is a nice work which could be a good start. Thanks. Just a few details below. Note that "cat $file | grep something" may be shortened by "grep something $file". It would also be interesting to avoid creating temporary files as most as possible (at least testflist.lst is not necessary I think). TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') may be shortened by these lines: { read TARGET; read OPTIONS; } << EOF $(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //') EOF Thanks, I've changed this. which is portable, or something similar. sed 's/.*=//'| sed 's/,.*//' may be shortened by sed -e 's/.*=//' -e 's/,.*//' Thanks, I've changed this as well. Also note, there are some cases where options are enabled without appearing in OPTIONS variable value. For instance if you compile haproxy like that: $ make TARGET=linux2628 the support for the thread is enabled without appearing in OPTIONS variable value. I am not sure this is an issue at this time. That could become an issue. but should be easy to solve adding a 'fake' option perhaps to check against.. Or adding a separate check perhaps I'm not sure yet. With one test like that after having parsed OPTIONS, it would suffice at this time : if [ $TARGET = linux2628 ] ; then OPTIONS="$OPTIONS USE_xxx USE_xxx" fi with USE_xxx the listed compilation options found in the Makefile file: ifeq ($(TARGET),linux2628) # This is for standard Linux >= 2.6.28 with netfilter, epoll, tproxy and splice USE_NETFILTER = implicit USE_POLL= implicit USE_EPOLL = implicit USE_TPROXY = implicit USE_CRYPT_H = implicit USE_LIBCRYPT= implicit USE_LINUX_SPLICE= implicit USE_LINUX_TPROXY= implicit USE_ACCEPT4 = implicit USE_FUTEX = implicit USE_CPU_AFFINITY= implicit ASSUME_SPLICE_WORKS= implicit USE_DL = implicit USE_RT = implicit USE_THREAD = implicit else Perhaps we could could only one line for the required options and excluded targets like that: #EXCLUDED_TARGETS=freebsd,dos,windows ;) #REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB Added this option, but then i like the option of excluding single targets and having some comment behind it explaining the reason.. But i guess if one would want to know some comments above the setting could also 'explain' why that target is currently not 'valid' to run the test on. Should i remove the settings for the 'single' option/target.? No. It is ok at this time. New 'version' of the script attached. It now supports a set of parameters to modify its behavior a little. And also checking for a 'version requirement'. So a H2 test doesn't have to fail on 1.7. Very good work. Should i 'ask' to delete old test result.? Or would there be a other better way to keep previous results separated from the current run? No. Rename the old vtc files like that before running the tests: bname=$(basename $i) mv /tmp/$bname /tmp/old.$bname with $i in $(find /tmp/ -type d -name "vtc.*") or something like that. please *add -type d* to the find command to list the temporary varnishtest directories. And we should only keep temporaries directory of tests which have failed with -l varnishtest option currently enabled option. They are all the /tmp/vtc.* directories found after having run the tests. So the script should list the remaining temporary directories after having run the tests. There is no need to parse the log file of this directories. We cannot rely on the fact there is no bug in varnishtest logging code ;) If you could give it another look i would appreciate that. Are there any things that need to be added/changed about it before considering to add it to haproxy sources branch? Yes a few details. It seems there are remaining useless piped sed processes (sed | sed) or perhaps I have missed something. There is no need to use awk to remove sections of lines. Have a look to cut command which is more lightweight. tr command accepts escaped characters as '\n' for new line character which is more readable than ' ' ;) For me with these remaining modifications/cleanup this script could be added to haproxy sources. Thanks again for this good work Pieter. Regards, Fred.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 19-10-2018 om 11:51 schreef Frederic Lecaille: The idea of the script sounds good to me. About the script itself it is a nice work which could be a good start. Thanks. Just a few details below. Note that "cat $file | grep something" may be shortened by "grep something $file". It would also be interesting to avoid creating temporary files as most as possible (at least testflist.lst is not necessary I think). TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') may be shortened by these lines: { read TARGET; read OPTIONS; } << EOF $(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //') EOF Thanks, I've changed this. which is portable, or something similar. sed 's/.*=//'| sed 's/,.*//' may be shortened by sed -e 's/.*=//' -e 's/,.*//' Thanks, I've changed this as well. Also note, there are some cases where options are enabled without appearing in OPTIONS variable value. For instance if you compile haproxy like that: $ make TARGET=linux2628 the support for the thread is enabled without appearing in OPTIONS variable value. I am not sure this is an issue at this time. That could become an issue. but should be easy to solve adding a 'fake' option perhaps to check against.. Or adding a separate check perhaps I'm not sure yet. Perhaps we could could only one line for the required options and excluded targets like that: #EXCLUDED_TARGETS=freebsd,dos,windows ;) #REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB Added this option, but then i like the option of excluding single targets and having some comment behind it explaining the reason.. But i guess if one would want to know some comments above the setting could also 'explain' why that target is currently not 'valid' to run the test on. Should i remove the settings for the 'single' option/target.? New 'version' of the script attached. It now supports a set of parameters to modify its behavior a little. And also checking for a 'version requirement'. So a H2 test doesn't have to fail on 1.7. Should i 'ask' to delete old test result.? Or would there be a other better way to keep previous results separated from the current run? If you could give it another look i would appreciate that. Are there any things that need to be added/changed about it before considering to add it to haproxy sources branch? Regards, PiBa-NL (Pieter) #!/usr/bin/env sh if [ "$1" == "--help" ]; then cat << EOF ### run-regtests.sh ### Running run-regtests.sh --help shows this information about how to use it Run without parameters to run all tests in the current folder (including subfolders) run-regtests.sh Provide paths to run tests from (including subfolders): run-regtests.sh ./tests1 ./tests2 Parameters: --j , To run varnishtest with multiple jobs / threads for a faster overall result run-regtests.sh ./fasttest --j 16 --v, to run verbose run-regtests.sh --v, disables the default varnishtest 'quiet' parameter --varnishtestparams , passes custom ARGS to varnishtest run-regtests.sh --varnishtestparams "-n 10" --f, force deleting old /tmp/*.vtc results without asking Including text below into a .vtc file will check for its requirements related to haproxy's target and compilation options # Below targets are not capable of completing this test succesfully #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #EXCLUDE_TARGETS=dos,freebsd,windows # Below option is required to complete this test succesfully #REQUIRE_OPTION=OPENSSL, this test needs OPENSSL compiled in. #REQUIRE_OPTIONS=ZLIB,OPENSSL,LUA # To define a range of versions that a test can run with: #REQUIRE_VERSION=0.0 #REQUIRE_VERSION_BELOW=99.9 EOF return fi _startswith() { _str="$1" _sub="$2" echo "$_str" | grep "^$_sub" >/dev/null 2>&1 } _findtests() { #find "$1" -name "*.vtc" | while read i; do IFS=' ' set -f for i in $( find "$1" -name "*.vtc" ); do #echo "TESTcheck '$i'" skiptest= require_version="$(grep "#REQUIRE_VERSION=" "$i" | sed -e 's/.*=//')" require_version_below="$(grep "#REQUIRE_VERSION_BELOW=" "$i" | sed -e 's/.*=//')" require_options="$(grep "#REQUIRE_OPTIONS=" "$i" | sed -e 's/.*=//')" exclude_targets=",$(grep "#EXCLUDE_TARGETS=" "$i" | sed -e 's/.*=//')," if [ -n "$require_version" ]; then if [ $(_version "$HAPROXY_VERSION") -lt $(_version "$require_version") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test requires at least version: $require_version" skiptest=1 fi fi if [ -n "$require_version_below" ]; then if [ $(_version "$HAPROXY_VERSION") -ge $(_version "$require_version_below") ]; then echo " Skip $i because option haproxy is version: $HAPROXY_VERSION" echo "REASON: this test re
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/19/2018 12:43 AM, PiBa-NL wrote: Hi Frederic, Hi Pieter, Do you have a little time to take a look at (the idea of) the script? The idea of the script sounds good to me. About the script itself it is a nice work which could be a good start. Just a few details below. Note that "cat $file | grep something" may be shortened by "grep something $file". It would also be interesting to avoid creating temporary files as most as possible (at least testflist.lst is not necessary I think). TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') may be shortened by these lines: { read TARGET; read OPTIONS; } << EOF $(./haproxy -vv |grep 'TARGET\|OPTIONS' | sed 's/.* = //') EOF which is portable, or something similar. sed 's/.*=//'| sed 's/,.*//' may be shortened by sed -e 's/.*=//' -e 's/,.*//' Also note, there are some cases where options are enabled without appearing in OPTIONS variable value. For instance if you compile haproxy like that: $ make TARGET=linux2628 the support for the thread is enabled without appearing in OPTIONS variable value. I am not sure this is an issue at this time. Perhaps we could could only one line for the required options and excluded targets like that: #EXCLUDED_TARGETS=freebsd,dos,windows ;) #REQUIRED_OPTIONS=OPENSSL,LUA,ZLIB Thank you a lot. Thanks in advance :). Op 3-10-2018 om 21:52 schreef PiBa-NL: Hi Frederic, Made a little script (attached).. What do you think ? Could this work for current 'requirements'? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Do you have a little time to take a look at (the idea of) the script? Thanks in advance :). Op 3-10-2018 om 21:52 schreef PiBa-NL: Hi Frederic, Made a little script (attached).. What do you think ? Could this work for current 'requirements'? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 2-10-2018 om 8:58 schreef Frederic Lecaille: At this time I do not know how to skip the tests which are not supported due to disable features at compilation time without running a "haproxy -vv" command before launching the VTC files. Made a little script (attached).. It would allow for the .vtc files to contain what 'requirements' they have, by including lines like these: #EXCLUDE_TARGET=freebsd, abns sockets are not available on freebsd #REQUIRE_OPTION=OPENSSL, this test needs ssl compiled in. Which could then generate output like listed below. It searches for all testcases, validates their requirements, and then puts them in a list. After which it calls varnishtest with quiet and simultaneous job parameters to allow for a short turnaround time.. What do you think ? Could this work for current 'requirements'? Or is something different preferred? I though about running each test directly from the script, and while that allows per test OK/Fail response logging for and a nicer overall summery, it takes more time to run. So i went for automatically creating a list of tests and letting varnishtest itself handle the running of multiple tests at once. Thoughts appreciated :). Regards, PiBa-NL (Pieter) root@freebsd11:/usr/ports/net/haproxy-devel # ./run-regtests.sh ## Preparing to run tests ## HA-Proxy version 1.8.14-52e4d43 2018/09/20 Copyright 2000-2018 Willy Tarreau ## Gathering tests to run ## Skip ./haproxy_test_OK_20180831/lua/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./haproxy_test_OK_20180831/lua/h1.vtc Add test: ./haproxy_test_OK_20180831/lua/b0.vtc Add test: ./haproxy_test_OK_20180831/lua/b1.vtc Add test: ./haproxy_test_OK_20180831/loadtest/b0-loadtest.vtc Add test: ./haproxy_test_OK_20180831/log/b0.vtc Skip ./haproxy_test_OK_20180831/reload/b1-server-state-file.vtc because: TARGET = freebsd REASON: abns sockets are not available on freebsd Skip ./haproxy_test_OK_20180831/ssl/b0.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./haproxy_test_OK_20180831/stick-table/b1.vtc Add test: ./haproxy_test_OK_20180831/stick-table/b0.vtc Add test: ./test/b0-loadtest.vtc Skip ./test/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./test/b3-cpu.vtc Add test: ./work/haproxy-1.8-52e4d43/reg-tests/server/b0.vtc Add test: ./work/haproxy-1.8-52e4d43/reg-tests/stick-table/h0.vtc Skip ./work/haproxy-1.8-52e4d43/reg-tests/lua/b2.vtc because option OPENSSL not found REASON: this test needs ssl compiled in. Add test: ./work/haproxy-1.8-52e4d43/reg-tests/lua/h1.vtc ## Starting varnishtest ## # top TEST ./test/b3-cpu.vtc FAILED (5.281) exit=2 1 tests failed, 0 tests skipped, 11 tests passed root@freebsd11:/usr/ports/net/haproxy-devel # #!/usr/bin/env sh echo "## Preparing to run tests ##" haproxy -v #varnishtest -j 16 -k -t 20 ./work/haproxy-*/reg-tests/*/*.vtc > ./mytest-result.log 2>&1 #varnishtest -j 16 -k -t 20 ./haproxy_test_OK_20180831/*/*.vtc >> ./mytest-result.log 2>&1 #cat ./mytest-result.log #echo "" >> ./mytest-result.log #haproxy -vv >> ./mytest-result.log TARGET=$(haproxy -vv | grep "TARGET = " | sed 's/.*= //') OPTIONS=$(haproxy -vv | grep "OPTIONS = " | sed 's/.*= //') echo "TARGET : $TARGET" echo "OPTIONS : $OPTIONS" haproxy -v > testresult_summary.log haproxy -v > testresult_detailed.log echo -n > testlist.lst echo "## Gathering tests to run ##" for i in $(find ./ -name "*.vtc"); do optionismissing= for required in "$(cat $i | grep REQUIRE_OPTION)"; do if [ -z "$required" ] then continue fi requiredoption=$(echo "$required" | sed 's/.*=//' | sed 's/,.*//') #echo " Test requires: $requiredoption" if [ -z "$( echo "$OPTIONS" | grep "USE_$requiredoption=1" )" ] then echo -n "Skip $i because option $requiredoption not found! REASON: " echo "$required" | sed 's/.*,//' optionismissing=1 fi done if [ $optionismissing ] then continue fi testtarget=$(cat $i | grep "#EXCLUDE_TARGET") #echo "testtarget : $testtarget" if [ "$( echo "$testtarget" | grep "#EXCLUDE_TARGET=$TARGET," )" ] then echo "Skip $i because: TARGET = $TARGET" echo -n " REASON: " echo "$testtarget" | sed 's/.*,//' continue fi echo
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 10/01/2018 08:59 PM, PiBa-NL wrote: Hi Frederic, Op 1-10-2018 om 16:09 schreef Frederic Lecaille: - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. Should i send a patch changing that? Or can you fix it easily ;) Yes, feel free to send a patch for that. - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. Afaik the 'check' on the server line is failing to send a GET / request.. My other mail has a bit more ideas about that.. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. I think creating a list of tests that could be run on FreeBSD will take a lot of maintenance and i assume 'most' tests will actually be runnable on most OS's. So having short list with exceptions is probably easier to maintain. In my opinion every test should be run on every os unless there is some good reason not to run it (abns/splicing/stuff..). And if possible the reason to exclude a specific test should be described (a single line of text could be enough). I did not mention to create new tests for each OSes. All VTC files would be created in directories below reg-tests directories. They would be the union of all VTC files supported by all the OSes supported by HAProxy. reg-tests directory would contain directories for each OSes with symbolic links to the directories previously mentioned, if they contain VTC files supported by this OS. Something like that: reg-tests log abns linux log -> ../log abns -> ../abns freebsd log -> ../log The even bigger issue that Willy raised is with compilation options not including lua/ssl/gzip/threads/ stuff, and having lots of tests fail on those, while they should get skipped if they rely on a feature not compiled. I discussed this also a bit with Willy in the other mail thread ( https://www.mail-archive.com/haproxy@formilux.org/msg31345.html ), but i don't think we have defined the perfect way to do it yet.. So that part of the question still stands ;) .. Whats the best way to skip tests that are not applicable? At this time I do not know how to skip the tests which are not supported due to disable features at compilation time without running a "haproxy -vv" command before launching the VTC files. Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 1-10-2018 om 16:09 schreef Frederic Lecaille: - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. Should i send a patch changing that? Or can you fix it easily ;) - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. Afaik the 'check' on the server line is failing to send a GET / request.. My other mail has a bit more ideas about that.. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. I think creating a list of tests that could be run on FreeBSD will take a lot of maintenance and i assume 'most' tests will actually be runnable on most OS's. So having short list with exceptions is probably easier to maintain. In my opinion every test should be run on every os unless there is some good reason not to run it (abns/splicing/stuff..). And if possible the reason to exclude a specific test should be described (a single line of text could be enough). The even bigger issue that Willy raised is with compilation options not including lua/ssl/gzip/threads/ stuff, and having lots of tests fail on those, while they should get skipped if they rely on a feature not compiled. I discussed this also a bit with Willy in the other mail thread ( https://www.mail-archive.com/haproxy@formilux.org/msg31345.html ), but i don't think we have defined the perfect way to do it yet.. So that part of the question still stands ;) .. Whats the best way to skip tests that are not applicable? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi Frederic, Op 1-10-2018 om 16:19 schreef Frederic Lecaille: On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) This test relies on timeout values to produce the correct syslog output. Relying on timing values for reg testing is perhaps a bad idea as the results are not deterministic. Indeed here we want the 5ms timeout client to expire before it reads the server response after having waited for 20ms. I propose to remove this test. Regarding /log/b0.vtc i think the test itself is 'mostly' okay. If we keep the check i do think its 'check' on the server line should be changed by one of the following options: - removing 'check' - check to different port to a s2 server that does not use txresp - add 'option httpchk' and repeat s1 twice.. Any of the changes above would 'fix' the biggest issue with the test as far as i see.. But indeed having short timeouts could cause on occasion cause random failures if for example multiple tests are run simultaneously.. The test for a decent log line with a 'client timout' flag is probably something that should be tested, perhaps the c1 delay could be set to something longer to 'enforce' the 'timeout client 5' would always hit first even when stuff starts competing for (cpu) resources.? Regards, PiBa-NL (Pieter)
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) This test relies on timeout values to produce the correct syslog output. Relying on timing values for reg testing is perhaps a bad idea as the results are not deterministic. Indeed here we want the 5ms timeout client to expire before it reads the server response after having waited for 20ms. I propose to remove this test.
Re: reg-test failures on FreeBSD, how to best adapt/skip some tests?
On 09/11/2018 04:51 PM, PiBa-NL wrote: Hi List, Hi, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? Correct. I agree I did not think to replace this part specific to Linux. - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) Perhaps the syslog traces could give us more information about what is happening here. - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? Perhaps we should use the TARGET value to select the VTC files directories which should be selected for the OSes. By default for linux all VTC files in reg-tests directory should be run (found with find command without -L option, so that not to follow the symbolic link). For instance for freebsd OS we would create reg-tests/freebsd directory with symbolic links to the linux reg-tests subdirectories it supports. With a few small changes (attached) i can run all tests like this, get the following result: varnishtest -l ./work/haproxy-ss-20180901/reg-tests/*/*.vtc # top TEST ./work/haproxy-ss-20180901/reg-tests/connection/b0.vtc passed (0.142) # top TEST ./work/haproxy-ss-20180901/reg-tests/log/b0.vtc passed (0.136) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b0.vtc passed (0.121) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b1.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b2.vtc passed (0.186) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b3.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/h1.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/seamless-reload/b0.vtc passed (0.143) # top TEST ./work/haproxy-ss-20180901/reg-tests/spoe/b0.vtc passed (0.011) # top TEST ./work/haproxy-ss-20180901/reg-tests/ssl/b0.vtc passed (0.146) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b0.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b1.vtc passed (0.122) Above would be good :).. But well.. needs changing or skipping some tests.. I would like to know next time that 'all' testable reg-tests are working properly when making a new build to take into my production system. If some tests fail by design (on this platform), it takes more administration to figure out if that was okay or not. Please advice :) Regards, PiBa-NL (Pieter)
reg-test failures on FreeBSD, how to best adapt/skip some tests?
Hi List, I was wondering how to best run the reg-tests that are 'valid' for FreeBSD. There are a 2 tests that use abns@ sockets, which seem not available on FreeBSD. Also 1 test is failing for a reason i'm not totally sure if its totally expected to or not.. - /connection/b0.vtc probably does not 'really' need abns@ sockets, so changing to unix@ would make it testable on more platforms? - /log/b0.vtc Not exactly sure why this fails/why it was supposed to work. It either produces a timeout, or the s1 server fails to read the request which the tcp-healthcheck does not send.. *** s1 0.0 accepted fd 5 127.0.0.1 23986 ** s1 0.0 === rxreq s1 0.0 HTTP rx failed (fd:5 read: Connection reset by peer) - /seamless-reload/b0.vtc This one specifically mentions testing a abns@ socket functionality. so changing it to a unix@ socket likely changes the test in such a way its no longer testing what it was meant for.. What would be the best way to skip this test on FreeBSD? With a few small changes (attached) i can run all tests like this, get the following result: varnishtest -l ./work/haproxy-ss-20180901/reg-tests/*/*.vtc # top TEST ./work/haproxy-ss-20180901/reg-tests/connection/b0.vtc passed (0.142) # top TEST ./work/haproxy-ss-20180901/reg-tests/log/b0.vtc passed (0.136) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b0.vtc passed (0.121) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b1.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b2.vtc passed (0.186) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/b3.vtc passed (0.133) # top TEST ./work/haproxy-ss-20180901/reg-tests/lua/h1.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/seamless-reload/b0.vtc passed (0.143) # top TEST ./work/haproxy-ss-20180901/reg-tests/spoe/b0.vtc passed (0.011) # top TEST ./work/haproxy-ss-20180901/reg-tests/ssl/b0.vtc passed (0.146) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b0.vtc passed (0.120) # top TEST ./work/haproxy-ss-20180901/reg-tests/stick-table/b1.vtc passed (0.122) Above would be good :).. But well.. needs changing or skipping some tests.. I would like to know next time that 'all' testable reg-tests are working properly when making a new build to take into my production system. If some tests fail by design (on this platform), it takes more administration to figure out if that was okay or not. Please advice :) Regards, PiBa-NL (Pieter) From 89a5fd48c11dba6a30f468e86e5a7d4bdab6b986 Mon Sep 17 00:00:00 2001 From: PiBa-NL Date: Tue, 11 Sep 2018 16:19:21 +0200 Subject: [PATCH] fix freebsd reg-tests it has no abns@ socket ability.. also the log/b0.vtc test fails on the health-check either timing out, or failing to read the request from the client when its only perfoming a tcp connection check. --- reg-tests/connection/b0.vtc | 4 ++-- reg-tests/log/b0.vtc | 3 ++- reg-tests/seamless-reload/b0.vtc | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/reg-tests/connection/b0.vtc b/reg-tests/connection/b0.vtc index 3a873848..a2b74f73 100644 --- a/reg-tests/connection/b0.vtc +++ b/reg-tests/connection/b0.vtc @@ -36,14 +36,14 @@ haproxy h1 -conf { listen http bind-process 1 -bind abns@http accept-proxy name ssl-offload-http +bind /tmp/http accept-proxy name ssl-offload-http option forwardfor listen ssl-offload-http option httplog bind-process 2-4 bind "fd@${ssl}" ssl crt ${testdir}/common.pem ssl no-sslv3 alpn h2,http/1.1 -server http abns@http send-proxy +server http /tmp/http send-proxy } -start diff --git a/reg-tests/log/b0.vtc b/reg-tests/log/b0.vtc index f0ab7ea1..423c7e7f 100644 --- a/reg-tests/log/b0.vtc +++ b/reg-tests/log/b0.vtc @@ -24,7 +24,7 @@ feature ignore_unknown_macro server s1 { rxreq txresp -} -start +} -repeat 2 -start syslog Slg_1 -level notice { recv @@ -50,6 +50,7 @@ frontend fe1 default_backendbe_app backend be_app +option httpchk server app1 ${s1_addr}:${s1_port} check } -start diff --git a/reg-tests/seamless-reload/b0.vtc b/reg-tests/seamless-reload/b0.vtc index 498e0c61..e8507523 100644 --- a/reg-tests/seamless-reload/b0.vtc +++ b/reg-tests/seamless-reload/b0.vtc @@ -25,10 +25,10 @@ haproxy h1 -W -conf { listen testme bind "fd@${testme}" -server test_abns_server abns@wpproc1 send-proxy-v2 +server test_abns_server /tmp/wpproc1 send-proxy-v2 frontend test_abns -bind abns@wpproc1 accept-proxy +bind /tmp/wpproc1 accept-proxy http-request deny deny_status 200 } -start -- 2.18.0.windows.1