Hi Pieter, On Mon, Jan 07, 2019 at 09:24:24PM +0100, PiBa-NL wrote: > For 1 part its always been broken (needing the short mailer timeout to send > all expected mails), for the other part, at least until 1.8.14 it used to > NOT send thousands of mails so that would be a regression in the current 1.9 > version that should get fixed on a shorter term.
OK, thanks for clarifying. Indeed a fix is needed then. > > I'm hesitant between > > merging this in the slow category or the broken one. My goal with "broken" > > was to keep the scripts that trigger broken behaviours that need to be > > addressed, rather than keep broken scripts. > Indeed keeping broken scripts wouldn't be help-full in the long run, unless > there is still the intent to fix them. It isn't what the makefile says about > 'LEVEL 5' though. It says its for 'broken scripts' and to quickly disable > them, not as you write here for scripts that show 'broken haproxy behavior'. I know :-) For me the "broken" scripts should be the experimental ones, those for known broken haproxy behavior definitely are worth keeping because once the issue is fixed they can be renamed and included in the standard test series. It's just that until a bug is fixed, reminding in loops about it is counter-productive. Especially when these ones cause time-outs or whatever issue some of our bugs used to cause! > > My goal is to make sure we > > never consider it normal to have failures in the regular test suite, > > otherwise you know how it becomes, just like compiler warnings, people > > say "oh I didn't notice this new error in the middle of all other ones". > Agreed, though i will likely fall into repeat some day, apology in advance > ;).. No, you're welcome! > I guess we could 'fix' the regtest by specifying the 'timeout mail > 200', that would fix it for 1.7 and 1.8.. And might help for 1.9 > regressiontests and to get it fixed to at least not send thousands of mails. > We might forget about the short time requirement then though, which seems > strange as well. And the test wouldn't be 1.6 compatible as it doesn't have > that setting at all. I honestly have no idea. My goal is that this bug gets fixed at least if it's a regression. > > Thus probably the best thing to do is to use it at level 5 so that it's > > easier to work on the bug without triggering false positives when doing > > regression testing. > > > > What's your opinion ? > > With a changed description for 'level 5' being 'shows broken haproxy > behavior, to be fixed in a future release' i think it would fit in there > nicely. Can you change the starting letter of the .vtc test (and the .lua > and reference to that) to 'k' during committing? Or shall i re-send it? I can do it, don't worry. > p.s. What do you think about the 'naming' of the test? > 'k_healthcheckmail.vtc' or 'k00000.vtc' personally i don't think the > 'numbering' of tests makes them easier to use.?. I've been thinking about such things as well. As I mentioned earlier, I definitely think that our reg-testing suite's organisation will continue to evolve. It's something new for this project and we're discovering a number of unexpected side effects that progressively make us adapt, and as long as we can continue to issue "make regtest", I think everyone will be OK. Cheers, Willy