On 1/19/19 8:53 AM, Willy Tarreau wrote:
Hi Lukas,

On Fri, Jan 18, 2019 at 12:43:34PM +0100, Lukas Tribus wrote:

currently we have 4 reg-tests in haproxy-1.8, backported due to the
actual bugfix commit, which included a test. We also have a broken
symbolic link in reg-tests/lua/common.pem, which causes at least some
confusion [1].

This is getting dirty :-/

We don't have any test infrastructure in haproxy-1.8 (Makefile,
reg-tests/README) afaik.

So my question is, where do we go from here? Dropping them all, and
remove the test part in future backports? Or backport the actual
infrastructure so that 1.8 can be tested?

I was interested in backporting them to 1.8 once we have more experience
with them and they're better organized, so that we avoid backporting
reorg patches. I'd say we've made quite some progress now and we could
possibly backport them. But I wouldn't be surprised if we'd soon rename
many of them again since the relation between the level and the prefix
letter has to be looked up into the makefile each time, so probably this
is something we should improve.

Note that a "reg-tests-help" makefile target dump the list of LEVELs:

$ make reg-tests-help

To launch the reg tests for haproxy, first export to your environment VTEST_PROGRAM variable to point to your vtest program:
    $ export VTEST_PROGRAM=/opt/local/bin/vtest
    $ setenv VTEST_PROGRAM /opt/local/bin/vtest

The same thing may be done to set your haproxy program with HAPROXY_PROGRAM but with ./haproxy as default value.

To run all the tests:
    $ make reg-tests

You can also set the programs to be used on the command line:
    $ VTEST_PROGRAM=<...> HAPROXY_PROGRAM=<...> make reg-tests

To run tests with specific levels:
    $ LEVEL=1,3,4   make reg-tests  #list of levels
    $ LEVEL=1-3,5-6 make reg-tests  #list of range of levels

About the levels:
LEVEL 1 scripts are dedicated to pure haproxy compliance tests (prefixed with 'h' letter).
    LEVEL 2 scripts are slow scripts (prefixed with 's' letter).
    LEVEL 3 scripts are low interest scripts (prefixed with 'l' letter).
LEVEL 4 scripts are in relation with bugs they help to reproduce (prefixed with 'b' letter). LEVEL 5 scripts are scripts triggering known broken behaviors for which there is still no fix (prefixed with 'k' letter). LEVEL 6 scripts are experimental, typically used to develop new scripts (prefixed with 'e' lettre).

On my side I always run all the VTC files. When I add a new reg test, it can be tested like that without using LEVEL:

    $ make reg-tests <my new VTC file>

We could set the level with strings:

h*.vtc -> haproxy
s*.vtc -> slow
l*.vtc -> low   (perhaps this one should be removed).
b*.vtx -> bug
k*.vtc -> broken
e*.vtc -> exp

only list of levels could be permitted:

    $ LEVEL=haproxy,bug make reg-tests ...

As there is no more level notion here, perhaps we should rename LEVEL environment variable to VTC_TYPES, REGTEST_TYPES or something else.

Reply via email to