Hi, On Thursday 06 August 2009, Andrew Manson wrote: > Hi everyone , > > I'm a Marble developer and have just started to think seriously about unit > tests and test driven development, possibly a bit late in the game but > better late then never! I was surprised to see that we already had some > unit tests in Marble but what was worse was that the core developers ( > including myself ) were "Surprised to see" that more than 50% of the tests > failed. This is why I've come to you lot, hopefully to spark some > discussion and get some things done ;)
Nice, I'd also like to get things going in this regard. > The shocking thing about ^^^ is that the core developers were completely > unaware that the unit tests were failing, which i find a bit odd when there > are some pretty good Dashboard programs out there that can display and > notify about the results of unit tests. I would really like to see one of > these programs being used in a big way in KDE sooner rather than later so > thats why I spend most of yesterday chatting with someone that knows much > more about this stuff than I do. Sorry in advance if this turns out to be a > very long email. > > Over the course of yesterday's discussions we have identified two major > "styles" of unit test dashboard system. 1) a distributed model using > something like CTest and CDash to display the results or 2) a centralised > model where the build system is "part" of the *dedicated* unit test running > system and the dashboard, like buildbot and CruiseControl . Each have their > merits and hopefully we can discuss which one would be "more right" for > KDE. > > For the centralised model we have a few organisational problems, mainly > that we would need a (dedicated ) server to be provided so that we could > run the build and display the results. This may not be so difficulty > because I know that "Dirk" has a system like this setup already where the > build results are displayed here: http://ktown.kde.org/~dirk/dashboard/ but > this currently only works for displaying the actual result of the build and > does not include unit tests. Using this model to incorporate unit tests > shouldn't be too hard but it might cause an organisational nightmare for > the sysadmins ( who have a hard enough time already ). On the other side of > that, if we used a buildbot system and some of the new cool buzzwords in > computer science like "distributed virtualised cloud computing" we could > make a really cool system that would be able to check the build of KDE on > Linux, Windows and Mac. This would be pretty cool but like i said an > organisational nightmare. > > The other possibility is that we could use the distributed unit test > reporting model that is currently incorporated by the CTest and CDash > system. This is favorable for a number of reasons: > 1) we are currently using CMake so adding CTest support is *easy*! > 2) we don't have to have a centralised build system because any time the > unit tests are run on any computer the results are submitted to the CDash > dashboard Nevertheless having some big central machine ideally running a bunch of virtual machines, i.e. one with Windows, with OSX, FreeBSD and Solaris would be nice :-) I guess this would mean an Apple server... > 3) to set up the CDash system we would only need to add a few > CMake variables to our CMakeList.txt files and we will be submitting > results to a database in no time > 4) from what I hear the kind people at http://www.cdash.org/CDash/index.php > have already offered to host the CDash installation so our sysadmins would > be able to take it easy. Yes, Kitware offers free hosting for free projects at my.cdash.org. I have already set up dashboards for kdelibs, kdepimlibs, kdesupport and automoc. I still need to document this somewhere, and I need help with running the builds :-) > There are a lot of good points for using the CDash system but there is one > pretty big problem with it that may render our test results somewhat > useless. The fact that we are now starting to use the QtTest system makes > things very easy for us but it means that each QtTest executable will be > regarded as a single test by the CTest system. This is conceptually wrong > because each QtTest executable contains many sub tests that can fail or > pass independently of the single QtTest executable. Currently the CTest > system only creates a test result on the test executable level which means > that the results may not give any direct information as to why the test > failed. Some may say that this is only a small detail and results on the > executable level are "good enough" but if we are building a KDE-wide build > system we should at least try and get it as close to perfect as we possibly > can, or at least move in that direction. This problem could possibly be > fixed with a patch to the CTest system but that would require some effort > by someone far smarter than myself ;) I don't think it would be too hard. And you are not the first one with this issue. Not using ctest/cdash because of this one issue would be stupid, instead we (the people interested in testing) should contribute and write a patch :-) I can commit to the cmake (ctest) cvs, so this wouldn't be a big problem. Not sure whether it would also require modifications in cdash. I don't have commit access there, but Julien, the main developer, is usually very responsive. Alex P.S. I'm offline now until Tuesday _______________________________________________ Kde-buildsystem mailing list [email protected] https://mail.kde.org/mailman/listinfo/kde-buildsystem
