Hi Pieter, On Sun, Sep 30, 2018 at 12:05:14AM +0200, PiBa-NL wrote: > Hi Willy, > > I thought lets give those reg-test another try :) as its easy to run and > dev3 just came out. > All tests pass on my FreeBSD system, except this one, new reg-test attached. > > Pretty much the same test as previously send, but now with only 4 x 10 > connections. Which should be fine for conntrack and sysctls (i hope..). It > seems those stats numbers are 'off', or is my expected value not as fixed as > i thought it would be?
Well, at least it works fine on 1.8 and not on 1.9-dev3 so I think you spotted a regression that we have to analyse. However, I'd like to merge the fix before merging the regtest otherwise it will kill the reg-test feature until we manage to get the issue fixed! I'm also seeing that you rely on threads, I think I noticed another test involving threads. Probably that we should have a specific directory for these ones that we can disable completely when threads are not enabled, otherwise this will also destroy tests (and make them extremely slow due to varnishtest waiting for the timeout if haproxy refuses to parse the config). I think that we should think a bit forward based on these tests. We must not let varnishtest stop on the first error but rather just log it. Then at the end we could produce a report of successes and failures that would be easy to diff from the previous (or expected) one. That will be particularly useful when running the tests on older releases. As an example, I had to run your test manually on 1.8 because for I-don't-know- what-reason, the one about the proxy protocol now fails while it used to work fine last week for the 1.8.14 release. That's a shame that we can't complete tests just because one randomly fails. Thanks, Willy
