From: Frédéric Lécaille <flecai...@haproxy.com> This is a first patch to switch from varnishtest to new standalone varnish cache reg testing program: vtest.
More information may be found here: https://github.com/vtest/VTest https://varnish-cache.org/docs/trunk/reference/varnishtest.html https://varnish-cache.org/docs/trunk/reference/vtc.html --- doc/regression-testing.txt | 78 +++++++++++++++++++++++++--------------------- reg-tests/README | 69 ++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 69 deletions(-) diff --git a/doc/regression-testing.txt b/doc/regression-testing.txt index 2a96b0cd..77ebaecb 100644 --- a/doc/regression-testing.txt +++ b/doc/regression-testing.txt @@ -1,74 +1,82 @@ - +---------------------------------------------+ - | HAProxy regression testing with varnishtest | - +---------------------------------------------+ + +---------------------------------------+ + | HAProxy regression testing with vtest | + +---------------------------------------+ The information found in this file are a short starting guide to help you to write VTC (Varnish Test Case) scripts (or VTC files) for haproxy regression testing. Such VTC files are currently used to test Varnish cache application developed by Poul-Henning Kamp. A very big thanks you to him for having helped you to add -our haproxy C modules to varnishtest tool. +our haproxy C modules to vtest tool. Note that vtest was formally developed for +varnish cache reg testing and was named varnishtest. vtest is an haproxy specific +version of varnishtest program which reuses the non varnish cache specific code. A lot of general information about how to write VTC files may be found in 'man/vtc.7' -manual. It is *highly* recommended to read this manual before asking. This -documentation only deals with the varnishtest support for haproxy. +manual of varnish cache sources directory or directly on the web here: + https://varnish-cache.org/docs/trunk/reference/vtc.html -varnishtest installation +It is *highly* recommended to read this manual before asking to haproxy ML. This +documentation only deals with the vtest support for haproxy. + + +vtest installation ------------------------ -To use varnishtest you will have to download and compile the recent Varnish cache -sources found at https://github.com/varnishcache/varnish-cache. +To use vtest you will have to download and compile the recent vtest +sources found at https://github.com/vtest/VTest. -To compile Varnish cache : +To compile vtest: - $ ./autogen.sh - $ ./configure - $ make + $ cd VTest + $ make vtest -The varnishtest sources may be found in 'bin/varnishtest' directory. -'bin/varnishtest/tests' is plenty of VTC files for Varnish cache. After having -compiled these sources, the varnishtest executable location is -'bin/varnishtest/varnishtest'. +Note that varnishtest may be also compiled but not without the varnish cache +sources already compiled: -varnishtest is able to search for the haproxy executable file it is supposed to -launch in the PATH environment variable. To force the executable to be used by -varnishtest, the HAPROXY_PROGRAM environment variable for varnishtest may be -typically set as follows: + $ VARNISH_SRC=<...> make varnishtest - $ HAPROXY_PROGRAM=~/srcs/haproxy/haproxy varnishtest ... +After having compiled these sources, the vtest executable location is at the +root of the vtest sources directory. -varnistest exectution +vtest exectution --------------------- -varnishtest program comes with interesting options. The most interesting are: +vtest is able to search for the haproxy executable file it is supposed to +launch thanks to the PATH environment variable. To force the executable to be used by +vtest, the HAPROXY_PROGRAM environment variable for vtest may be +typically set as follows: + + $ HAPROXY_PROGRAM=~/srcs/haproxy/haproxy vtest ... + +vtest program comes with interesting options. The most interesting are: -t Timeout in seconds to abort the test if some launched program - -v By default, varnishtest does not dump the outputs of processus it launched + -v By default, vtest does not dump the outputs of processus it launched when the test passes. With this option the outputs are dumped even when the test passes. -L to always keep the temporary VTC directories. -l to keep the temporary VTC directories only when the test fails. -About haproxy when launched by varnishtest, -d option is enabled by default. +About haproxy, when launched by vtest, -d option is enabled by default. How to write VTC files ---------------------- -A VTC file must start with a "varnishtest" command line followed by a descriptive -line enclosed by double quotes. This is not specific to the VTC files for haproxy. +A VTC file must start with a "varnishtest" or "vtest" command line followed by a +descriptive line enclosed by double quotes. This is not specific to the VTC files +for haproxy. The VTC files for haproxy must also contain a "feature ignore_unknown_macro" line if any macro is used for haproxy in this file. This is due to the fact that -varnishtest parser code for haproxy commands generates macros the varnishtest -parser code for varnish has no knowledge of. This line prevents varnishtest from -failing in such cases. Since bbc34e2 varnish commit, as a "cli" macro is -automatically generated, this "feature ignore_unknown_macro" is mandatory -for each VTC file for haproxy. +vtest parser code for haproxy commands generates macros the vtest +parser code for varnish cache has no knowledge of. This line prevents vtest from +failing in such cases. As a "cli" macro automatically generated, this +"feature ignore_unknown_macro" is mandatory for each VTC file for haproxy. -To make varnishtest capable of testing haproxy, two new VTC commands have been +To make vtest capable of testing haproxy, two new VTC commands have been implemented: "haproxy" and "syslog". "haproxy" is used to start haproxy processus. "syslog" is used to start syslog servers (at this time, only used by haproxy). @@ -99,7 +107,7 @@ the configuration file is temporarily written. A 'stats.sock' UNIX socket is also created in this directory. There is no need to declare such stats sockets in the -conf {...} section. The name of the parent directory of the haproxy instances working directories is stored in 'tmpdir'. In -fact this is the working directory of the current varnishtest processus. +fact this is the working directory of the current vtest processus. There also exists 'testdir' macro which is the parent directory of the VTC file. It may be useful to use other files located in the same directory than the current diff --git a/reg-tests/README b/reg-tests/README index 5427ad69..f83ec60e 100644 --- a/reg-tests/README +++ b/reg-tests/README @@ -1,63 +1,64 @@ - * Regression testing for HAProxy with varnishtest * + * Regression testing for HAProxy with VTest * -This little README file is about how to compile and run varnishtest test case files (VTC files) to test HAProxy for any regression. +This little README file is about how to compile and run vtest test case files (VTC files) +to test HAProxy for any regression. -To do so, you will have to compile varnishtest program sources which comes with -Varnish cache application. varnishtest is a very useful program which has been -developed to test Varnish cache application. varnishtest has been modified in -collaboration with Varnish cache conceptor Poul-Henning Kamp to support HAProxy in -addition to Varnish cache. +To do so, you will have to compile vtest program sources which depends on +Varnish cache application sources. vtest, formerly varnishtest, is a very useful +program which has been developed to test Varnish cache application. vtest has been +modified in collaboration with Varnish cache conceptor Poul-Henning Kamp to support +HAProxy in addition to Varnish cache. -* varnishtest compilation * +* vtest compilation * - - Clone recent Varnish cache sources which may be found in this GitHub repository - https://github.com/varnishcache/varnish-cache: + $ git clone https://github.com/vtest/VTest - $ git clone https://github.com/varnishcache/varnish-cache + $ cd VTest - - Compile Varnish cache sources: + $ make vtest - $ cd varnish-cache && ./autogen.sh && ./configure && make - - Then varnishtest program may be found in bin/varnishtest directory. - VTC files for Varnish cache may be found in bin/varnishtest/tests directory. - The Varnish cache manuals are located in 'man' directory. You will have to - have a look at varnishtest(7) and vtc(7) manuals. + Then vtest program may be found at the root directory of vtest sources directory. + The Varnish cache manuals are located in 'man' directory of Varnish cache sources + directory. You will have to have a look at varnishtest(7) and vtc(7) manuals to + use vtest. Some information may also be found in doc/regression-testing.txt in HAProxy sources. + Note that VTC files for Varnish cache may be found in bin/varnishtest/tests directory + of Varnish cache sources directory which may be found here: + https://github.com/varnishcache/varnish-cache + -* varnishtest execution * +* vtest execution * You must set HAPROXY_PROGRAM environment variable to give the location - of the HAProxy program to test to varnishtest: + of the HAProxy program to test to vtest: - $ HAPROXY_PROGRAM=<my haproxy program> varnishtest ... + $ HAPROXY_PROGRAM=<my haproxy program> vtest ... The HAProxy VTC files found in HAProxy sources may be run with the reg-tests - Makefile target. You must set the VARNISHTEST_PROGRAM environment variable to - give the location of the varnishtest program which has been previously compiled. + Makefile target. You must set the VTEST_PROGRAM environment variable to + give the location of the vtest program which has been previously compiled. + + $ VTEST_PROGRAM=<my vtest program> make reg-tests - $ VARNISHTEST_PROGRAM=<my varnishtest program> HAPROXY_PROGRAM=<my haproxy program> make reg-tests + "reg-tests" Makefile target run scripts/run-regtest.sh script. + To get more information about this script run it with --help option. - Note that varnishtest is run with -t5 and -l option. -l option is to keep - keep varnishtest temporary directory in case of failed test cases. core files + Note that vtest is run with -t10 and -l option. -l option is to keep + keep vtest temporary directory in case of failed test cases. core files may be found in this directory (if enabled by ulimit). -* varnishtest patches for HAProxy VTC files * +* vtest patches for HAProxy VTC files * - When producing a patch to add a VCT regression testing file to reg-tests directory, + When producing a patch to add a VTC regression testing file to reg-tests directory, please follow these simple rules: - - Add the commit number on the first line. - - Then, copy and paste the commit log. - - If your VTC file needs others files, use the same basename as that of the VTC - file. + - If your VTC file needs others files, if possible, use the same basename as that + of the VTC file, - Put these files in a directory with the same name as the code area concerned by the bug ('peers', 'lua', 'acl' etc). - - -- 2.11.0
