Hi Willy,
Op 30-9-2018 om 7:46 schreef Willy Tarreau:
Hi Pieter,
On Sun, Sep 30, 2018 at 12:05:14AM +0200, PiBa-NL wrote:
Hi Willy,
I thought lets give those reg-test another try :) as its easy to run and
dev3 just came out.
All tests pass on my FreeBSD system, except this one, new reg-test attached.
Pretty much the same test as previously send, but now with only 4 x 10
connections. Which should be fine for conntrack and sysctls (i hope..). It
seems those stats numbers are 'off', or is my expected value not as fixed as
i thought it would be?
Well, at least it works fine on 1.8 and not on 1.9-dev3 so I think you
spotted a regression that we have to analyse.
Indeed it works with 1.8, so in that regard i 'think' the test itself is
correct.. Also when disabling threads, or running only 1 client, it
still works.. Then both CumConns CumReq show 11 for the first stats result.
However, I'd like to merge
the fix before merging the regtest otherwise it will kill the reg-test
feature until we manage to get the issue fixed!
I'm not fully sure i agree on that.. While i understand that failing
reg-tests can be a pita while developing (if you run them regulary) the
fact is that currently existing tests can already already start to fail
after some major redesign of the code, a few mails back (different
mailthread) i tested like 10 commits in a row and they all suffered from
different failing tests, that would imho not be a reason to remove those
tests, and they didnt stop development.
I'm also seeing that you rely on threads, I think I noticed another test
involving threads. Probably that we should have a specific directory for
these ones that we can disable completely when threads are not enabled,
otherwise this will also destroy tests (and make them extremely slow due
to varnishtest waiting for the timeout if haproxy refuses to parse the
config).
A specific directory will imho not work. How should it be called?
/threaded_lua_with_ssl_using_kqueue_scheduler_on_freebsd_without_absn_for_haproxy_1.9_and_higher/
?
Having varnishtest fail while waiting for a feature that was not
compiled is indeed undesirable as well. So some 'smart' way of defining
'requirements' for a test will be needed so they can gracefully skip if
not applicable.. I'm not sure myself how that way should look though..
On one side i think the .vtc itself might be the place to define what
requirements it has, on the other the other a separate list/script
including logic of what tests to run could be nice.. But then who is
going to maintain that one..
I think that we should think a bit forward based on these tests. We must
not let varnishtest stop on the first error but rather just log it.
varnishtest can continue on error with -k
Using this little mytest.sh script at the moment, this runs all tests
and only failed tests produce a lot of logging..:
haproxy -v
varnishtest -j 16 -k -t 20 ./work/haproxy-*/reg-tests/*/*.vtc >
./mytest-result.log 2>&1
varnishtest -j 16 -k -t 20 ./haproxy_test_OK_20180831/*/*.vtc >>
./mytest-result.log 2>&1
cat ./mytest-result.log
echo "" >> ./mytest-result.log
haproxy -vv >> ./mytest-result.log
There is also the -q parameter, but then it doesn't log anymore what
tests passed and would only the failed tests will produce 1 log line..
(i do like to log what tests where executed though..)
Then
at the end we could produce a report of successes and failures that would
be easy to diff from the previous (or expected) one. That will be
particularly useful when running the tests on older releases. As an
example, I had to run your test manually on 1.8 because for I-don't-know-
what-reason, the one about the proxy protocol now fails while it used to
work fine last week for the 1.8.14 release. That's a shame that we can't
complete tests just because one randomly fails.
You can continue tests. ( -k ) But better write it out to a logfile
then, or perhaps combine with -l which leaves the /tmp/.vtc folder..
Thanks,
Willy
Regards,
PiBa-NL (Pieter)
