On 9 February 2018 at 20:46, Ville Voutilainen <ville.voutilai...@gmail.com> wrote: >> But I am asking to do a minimal investigation. In most cases of blacklisting, >> the test has been failing for days, if not months. Spending an hour or two to >> understand why it's failing and whether it's something that only happens in >> the CI should be the norm. > > The problem is that a large amount of tests have been failing, for > weeks. In some cases, > months. In some cases, for a year. That results in a restage storm, > and carves a doubt > in every submitter's mind whether CI failures are something to really > bother about beyond > hitting the restage button. I doubt either of us thinks that to be optimal.
Also, our blacklist patch submitters *do* minimal investigation and more. There's been a whole shebang of suggested flaky tests, which have been deemed to not be caused by flakiness in tests, after proper investigation. And another matter to consider is that both because of CI infra problems and flaky tests, it's been next to impossible to get anything into e.g. qtbase for two weeks. Why some of those flaky tests seem to be hit more often isn't exactly known, but our statistics information does support their being flaky, and we've seen every one of these tests fail before in spurious manners, we just haven't done anything to it, due to some extent some people adamantly opposing blacklisting in general, and some demanding that there's a thorough investigation almost tantamount to 100% proof that a test is flaky before a possibly temporary blacklisting is even considered. It's also perhaps worth realizing that the current flakiness fixes and blacklists haven't gone in, because flaky tests here and there and everywhere prevent integrating those fixes. _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development