On 1 July 2015 at 08:03, Jürgen E. <[email protected]> wrote: > Hi Sandro, > > On Tue, 30. Jun 2015 at 23:05:47 +0200, Sandro Mani wrote: >> Just one curiosity: are unit tests run on all main platforms before >> release? > > They are run every night - currently for different version of debian/ubuntu, > windows and osx. > > But there virtually always have been a lot of tests (see dash[0]; starting at > the HF in Lyon), that fail at least on some of the platforms (including only > some of the debian/ubuntu version), so that's not a clear indicator for real > bugs. Hence the outcome has never been reliable enough to mean anything > without taking a closer look at the results. > > Matthias fixed a bunch of the tests, when Travis was introduced. But still > disabled a couple of the flaky tests just in the travis configuraton and also > added some new control images to make some others pass.
I've been meaning to post an update on this, so may as well do it now... Following up Matthias' work on Travis, he and I (+some friendly goblins) have been doing a lot of work to get the test suite behaving well on other platforms. This has mostly consisted of making the test suites more robust so they behave nicely cross platform, plus a small amount of identifying and addressing real issues. We're getting getting much closer, and comparing dash from the start of 2015 [1] to yesterday [2] shows just how much progress has been made. (Today's builds should look even better). Here's where we currently stand: - On linux platforms almost all of the tests should pass. The exception is qgis_legendrenderertest, which requires a bit of tweaking to handle different sized renderered/control images. There's some (potentially real issues) failures on 2.11.0-wheezy-amd64, which I still need to investigate and are probably related to dependency packaging issues on that distro. - Again, the tests should mostly pass on OSX, with the exception again of the legend renderer and possibly the server tests (I haven't been able to get a server build running yet on OSX to verify). Unfortunately there seems to be major issues with dakcarto's nightly builds which are causing additional failures - I'm going to chase this up with Larry and see if he can fix the build environment to clear these up. The unfortunate thing is that the OSX tests DID correctly flag the regression addressed in cc13c57, but the failure was buried in the noise of other failing tests. Lastly.... Windows. We're getting stacks of failures on the nightly Windows builds. I think there's two reasons for this: 1. A valid issue regarding font style names on Windows. Basically, the test suite relies on a reference font "QGIS Vera Sans Roman". It looks like there's an issue with font style names in Windows, where "Roman" is incorrectly shown as "Normal", "Oblique" as "Italic", etc. You can see this even with standard Windows fonts like Calibri - under Windows all the style names are wrong, but they're fine under other platforms. This potentially breaks compatibility of project files between Windows and other platforms, and also causes all tests which rely on a font to fail. I *think* this is an upstream issue - from my testing it looks like we aren't the only Qt app affected by it. 2. Locale issues with the test suite. The Windows builds are done using a locale which uses , instead of . for decimal places, and many tests aren't expecting this. We could blindly force the locale for the tests, but I'd rather address this so that the test suite can actually test to see if there's underlying issues with the methods on different locales. So there we go: Linux = all good OSX = mostly good Windows = a bunch of real issues to address All told, I'm happy with the direction the test suite is going and the shape it's in. When we finally get through this backlog of failing tests it should be easy to keep an eye on dash and address regressions as they occur. We'll even be ready when Travis gives us the option of OSX/Windows automated builds. Nyall [1] http://dash.orfeo-toolbox.org/index.php?project=QGIS&date=2015-01-02 [2] http://dash.orfeo-toolbox.org/index.php?project=QGIS&date=2015-06-30 _______________________________________________ Qgis-developer mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/qgis-developer
