On Sun, Sep 30, 2018 at 07:59:34PM +0200, PiBa-NL wrote: > Indeed it works with 1.8, so in that regard i 'think' the test itself is > correct.. Also when disabling threads, or running only 1 client, it still > works.. Then both CumConns CumReq show 11 for the first stats result.
Hmmm for me it fails even without threads. That was the first thing I tried when meeting the error in fact. But I need to dig deeper. > > However, I'd like to merge > > the fix before merging the regtest otherwise it will kill the reg-test > > feature until we manage to get the issue fixed! > I'm not fully sure i agree on that.. While i understand that failing > reg-tests can be a pita while developing (if you run them regulary) the fact > is that currently existing tests can already already start to fail after > some major redesign of the code, a few mails back (different mailthread) i > tested like 10 commits in a row and they all suffered from different failing > tests, that would imho not be a reason to remove those tests, and they didnt > stop development. The reason is that for now we have no way to let the tests fail gracefully and report what is OK and what is not. So any error that's in the way will lead to an absolutely certain behaviour from everyone : nobody will run the tests anymore since the result will be known. Don't get me wrong, I'm willing to get as many tests as we can, but 1) we have to be sure these tests only fail for regressions and not for other reasons, and 2) we must be sure that these tests do not prevent other ones from being run nor make it impossible to observe the progress on other ones. We're still at the beginning with reg tests, and as you can see we have not even yet sorted out the requirements for some of them like threads or Lua or whatever else. I'm just asking that we don't create tests faster than we can sort them out, that's all. This probably means that we really have to work on these two main areas which are test prerequisites and synthetic reports of what worked and what failed. Ideas and proposals on this are welcome, but to be honest I can't spend as much time as I'd want on this for now given how late we are on all what remains to be done, so I really welcome discussions and help on the subject between the various actors. Thanks, Willy
