Hi, In the past two weeks we've made some "interesting" discoveries regarding our test coverage, specifically regarding qtbase, but the same applies also to qt5.git:
* Our networking stack is only tested on OS X 10.10. Due to what seems like a bug in the test planner, the tests are not fully included on the Windows configurations. Regarding linux, see the second bullet below. * We do not run _any_ tests on Linux for the dev branch of qtbase and qt5.git. We actually run the tests, but the results are ignored - therefore test failures on Linux do not block bad changes from integrating. Any change in qtbase that accidentally breaks for example qtscript on Linux will go into qt5.git undetected. As to how it happened that we do not do testing on Linux anymore: We've had enforcing test coverage with our Ubuntu 12.04 configurations. Then we added Ubuntu 14.04 but with ignore-failures mode (called "insignificant" in the system). Then in the beginning of February this year we removed the 12.04 configurations from the CI. When that happened nobody had checked if the Ubuntu 14.04 configurations were a proper replacement, i.e. if they would enforce test failures. In addition we've had OpenSuSE and RHEL 6.6 configurations added to the CI as well, but those were also in ignore-failures mode, despite RHEL 6.6 being the configuration we're using for our binary packages. Tests were failing, but nobody looked at the concrete results and fixed the tests. In the 5.4/5.5/5.5.0 branches the situation is actually similar, but meanwhile some fixes helped to regain some coverage. In the future we must avoid these situations through much more thorough review of changes to the CI system configuration. At the same time we are not having this ignore-failures mode in the new CI system anymore. If some configuration gets added, the tests have to pass. Otherwise we cannot add it and we cannot claim support for a certain OS/compiler configuration. Additionally we're taking this also as a sign to re-prioritize CI system related work to bring qtbase and the rest of the dev branches of qt5 modules over to the new CI system much sooner. I've been doing repeated builds of qtbase dev in the new CI and with a larger set of blacklisted tests - I'll send a separate email about this - and I would like to switch the module over this week. We will run tests on all non-cross compiling Linux configurations (Ubuntu 14.04, OpenSUSE 13.01 and RHEL 6.6) and if a test fails, the change doesn't go in. Generally we have the same matrix of os/compiler combinations as for the dev branch in the current CI system. Re-prioritization means for us that we'll do this switch sooner in order to maintain a certain level of quality, at the expense of not having a public web frontend and other features in the CI system available before the switch but some later point in time. There are a few more lessons to learn from this, but one step at a time :). Simon _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
