Hi ML,

Here are two patches for haproxy reg testing.

Note that we have recently added an new feature to varnishtest so that to send commands to the CLI without running a shell, socat etc (https://varnish-cache.org/docs/trunk/reference/vtc.html#haproxy). This breaks reg-tests/spoe/h00000.vtc reg test case (fixed by the first patch).

We can send commands to the CLI as follows:

    haproxy h1 -conf {...}

    haproxy h1 -cli {
        send "show info"
        expect ~ "something to expect"
    } -wait

With the 2nd patch we changed the prefix of the VTC files from 'h' letter to 'b' (as "bug") and added a new LEVEL 4 to run all the VTC files prefixed with 'b' letter. This VTC files are in relation with bugs they help to reproduce (and prevent from coming back).

So, for now on, if you run reg-tests, as there is no more LEVEL 1 (default LEVEL) VTC files, no VTC file will be run.

To run the LEVEL 4 (VTC files for already fixed bugs) you will have to use such a command:

$ VARNISHTEST_PROGRAM=~/src/varnish-cache-trunk/bin/varnishtest/varnishtest LEVEL=4 make reg-tests

>From 35486065bf63a47d17ba65b94d7b42b08d958abb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= <flecai...@haproxy.com>
Date: Wed, 22 Aug 2018 10:41:33 +0200
Subject: [PATCH 2/2] REGTEST/MINOR: Add a new class of regression testing

Add LEVEL #4 regression testing files which is dedicated to
VTC files in relation with bugs they help to reproduce.
At the date of this commit, all VTC files are LEVEL 4 VTC files.
 Makefile                                             | 7 +++++++
 reg-tests/log/{h00000.vtc => b00000.vtc}             | 0
 reg-tests/lua/{h00000.lua => b00000.lua}             | 0
 reg-tests/lua/{h00000.vtc => b00000.vtc}             | 2 +-
 reg-tests/seamless-reload/{h00000.vtc => b00000.vtc} | 0
 reg-tests/spoe/{h00000.vtc => b00000.vtc}            | 0
 reg-tests/ssl/{h00000.vtc => b00000.vtc}             | 0
 reg-tests/stick-table/{h00000.vtc => b00000.vtc}     | 0
 8 files changed, 8 insertions(+), 1 deletion(-)
 rename reg-tests/log/{h00000.vtc => b00000.vtc} (100%)
 rename reg-tests/lua/{h00000.lua => b00000.lua} (100%)
 rename reg-tests/lua/{h00000.vtc => b00000.vtc} (97%)
 rename reg-tests/seamless-reload/{h00000.vtc => b00000.vtc} (100%)
 rename reg-tests/spoe/{h00000.vtc => b00000.vtc} (100%)
 rename reg-tests/ssl/{h00000.vtc => b00000.vtc} (100%)
 rename reg-tests/stick-table/{h00000.vtc => b00000.vtc} (100%)

diff --git a/Makefile b/Makefile
index 817161f7..c4923995 100644
--- a/Makefile
+++ b/Makefile
@@ -999,6 +999,11 @@ update-version:
 	echo "$(SUBVERS)" > SUBVERS
 	echo "$(VERDATE)" > VERDATE
+# Target to run the regression testing script files.
+# LEVEL 1 scripts are dedicated to pure haproxy compliance tests (prefixed with 'h' letter).
+# LEVEL 2 scripts are slow scripts (prefixed with 's' letter).
+# LEVEL 3 scripts are low interest scripts (prefixed with 'l' letter).
+# LEVEL 4 scripts are in relation with bugs they help to reproduce (prefixed with 'b' letter).
 	@if [ ! -x "$(VARNISHTEST_PROGRAM)" ]; then \
 		echo "Please make the VARNISHTEST_PROGRAM variable point to the location of the varnishtest program."; \
@@ -1011,6 +1016,8 @@ reg-tests:
 	   EXPR='s*.vtc'; \
 	elif [ $$LEVEL = 3 ] ; then \
 	   EXPR='l*.vtc'; \
+	elif [ $$LEVEL = 4 ] ; then \
+	   EXPR='b*.vtc'; \
 	fi ; \
 	if [ -n "$$EXPR" ] ; then \
 	   find reg-tests -type f -name "$$EXPR" -print0 | \
diff --git a/reg-tests/log/h00000.vtc b/reg-tests/log/b00000.vtc
similarity index 100%
rename from reg-tests/log/h00000.vtc
rename to reg-tests/log/b00000.vtc
diff --git a/reg-tests/lua/h00000.lua b/reg-tests/lua/b00000.lua
similarity index 100%
rename from reg-tests/lua/h00000.lua
rename to reg-tests/lua/b00000.lua
diff --git a/reg-tests/lua/h00000.vtc b/reg-tests/lua/b00000.vtc
similarity index 97%
rename from reg-tests/lua/h00000.vtc
rename to reg-tests/lua/b00000.vtc
index 2b2ffb0e..4229eeb0 100644
--- a/reg-tests/lua/h00000.vtc
+++ b/reg-tests/lua/b00000.vtc
@@ -40,7 +40,7 @@ server s1 -repeat 2 {
 haproxy h1 -conf {
-        lua-load ${testdir}/h00000.lua
+        lua-load ${testdir}/b00000.lua
     frontend fe1
         mode http
diff --git a/reg-tests/seamless-reload/h00000.vtc b/reg-tests/seamless-reload/b00000.vtc
similarity index 100%
rename from reg-tests/seamless-reload/h00000.vtc
rename to reg-tests/seamless-reload/b00000.vtc
diff --git a/reg-tests/spoe/h00000.vtc b/reg-tests/spoe/b00000.vtc
similarity index 100%
rename from reg-tests/spoe/h00000.vtc
rename to reg-tests/spoe/b00000.vtc
diff --git a/reg-tests/ssl/h00000.vtc b/reg-tests/ssl/b00000.vtc
similarity index 100%
rename from reg-tests/ssl/h00000.vtc
rename to reg-tests/ssl/b00000.vtc
diff --git a/reg-tests/stick-table/h00000.vtc b/reg-tests/stick-table/b00000.vtc
similarity index 100%
rename from reg-tests/stick-table/h00000.vtc
rename to reg-tests/stick-table/b00000.vtc

>From 971de0c29c683d629eae0f3a2e52715933f18463 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= <flecai...@haproxy.com>
Date: Tue, 21 Aug 2018 15:29:24 +0200
Subject: [PATCH 1/2] REGTEST/MINOR: Missing mandatory "ignore_unknown_macro".

Since bbc34e2 varnish commit (for varnishtest), a new "cli"
macro is automatically created for each VTC script to dialog with
the CLI. Consequently, as this macro is unknown from higher level
code for varnishtest, it makes the scripts fail if we
we do not ask varnishtest to disregard the unknown macros.
To prevent this, from now on, for each VTC file for haproxy we MUST add
"feature ignore_unknown_macro" line to do so. This is mandatory
 doc/regression-testing.txt | 4 +++-
 reg-tests/spoe/h00000.vtc  | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/regression-testing.txt b/doc/regression-testing.txt
index 68845997..1b8cdb22 100644
--- a/doc/regression-testing.txt
+++ b/doc/regression-testing.txt
@@ -64,7 +64,9 @@ The VTC files for haproxy must also contain a "feature ignore_unknown_macro" lin
 if any macro is used for haproxy in this file. This is due to the fact that
 varnishtest parser code for haproxy commands generates macros the varnishtest
 parser code for varnish has no knowledge of. This line prevents varnishtest from
-failing in such cases.
+failing in such cases. Since bbc34e2 varnish commit, as a "cli" macro is
+automatically generated, this "feature ignore_unknown_macro" is mandatory
+for each VTC file for haproxy.
 To make varnishtest capable of testing haproxy, two new VTC commands have been
 implemented: "haproxy" and "syslog". "haproxy" is used to start haproxy processus.
diff --git a/reg-tests/spoe/h00000.vtc b/reg-tests/spoe/h00000.vtc
index b4ac8774..76b41d60 100644
--- a/reg-tests/spoe/h00000.vtc
+++ b/reg-tests/spoe/h00000.vtc
@@ -7,6 +7,8 @@
 varnishtest "SPOE bug: missing configuration file"
+feature ignore_unknown_macro
 haproxy h1 -conf-BAD {} {
         timeout connect 5000ms

Reply via email to