At 12:52 PM 6/2/2006, Ralf Habacker wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Hi all, > >while porting qt-dbus to cmake, we encountered some problems with test >applications result display. > >qt-dbus uses the QtTestLib framework for test applications, which >produces by default output in the following manner > >C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal >********* Start testing of tst_Hal ********* >Config: Using QTest library 4.1.1, Qt 4.1.2 >PASS : tst_Hal::initTestCase() ><snip> >FAIL... >FAIL... >PASS : tst_Hal::cleanupTestCase() >Totals: 2 passed, 2 failed, 0 skipped >********* Finished testing of tst_Hal ********* > >QtTestLib based test application has more options for example xml >output. See the complete list of options below. > >C:\Daten\kde4\kdesupport\qt-dbus-build>bin\tst_hal -help > Usage: bin\tst_hal [options] [testfunctions[:testdata]]... > By default, all testfunction will be run. > > options: > -functions : Returns a list of current testfunctions > -xml : Outputs results as XML document > -lightxml : Outputs results as stream of XML tags > -o filename: Writes all output into a file > -v1 : Print enter messages for each testfunction > -v2 : Also print out each QVERIFY/QCOMPARE/QTEST > -vs : Print every signal emitted > -eventdelay ms : Set default delay for mouse and keyboard simulation >to ms milliseconds > -keydelay ms : Set default delay for keyboard simulation to ms >milliseconds > -mousedelay ms : Set default delay for mouse simulation to ms >milliseconds > -keyevent-verbose : Turn on verbose messages for keyboard simulation > -maxwarnings n : Sets the maximum amount of messages to output. > 0 means unlimited, default: 2000 > -help : This help > > >ctest based test applications produces an output like shown below > >Running tests... >Start processing tests >Test project > 1/ 6 Testing tst_qdbusconnection ***Failed > 2/ 6 Testing tst_qdbusmarshall ***Failed > 3/ 6 Testing tst_qdbusxmlparser Passed > 4/ 6 Testing tst_qdbusinterface ***Failed > 5/ 6 Testing tst_qdbusabstractadaptor Passed > 6/ 6 Testing tst_hal ***Failed > >The question to the cmake people is now how to integrate the QTestLib >based tests into ctest so that more informations are available as >mentioned by Thiago > >>>> Would be better if CTest integrated with QtTestLib to find out >>>> *which* tests are available and, therefore, report the error >>>> count by test function, not test program. > >Some topics: > >- - output layout, may be similar to: > >1/ 6 Testing tst_qdbusconnection 5 tests passed, 10 tests ***Failed >2/ 6 Testing tst_qdbusmarshall 8 tests passed 0 failed > >or in a more detailed mode > > 1/ 6 Testing tst_qdbusconnection 6 tests > PASS : tst_Hal::initTestCase() > FAIL! : tst_Hal::getDevices() 'reply.type() = > QDBusMessage::ReplyMessage' returned FALSE. () > FAIL! : tst_Hal::lock() Compared values are not the same > Actual (spy.count): 0 > Expected (3): 3 > PASS : tst_Hal::cleanupTestCase() > > 2/ 6 Testing tst_qdbusmarshall 8 tests passed > > >- - How should cmake detect that the test application uses QtTestlib ? >- -> by a parameter for ENABLE_TESTING(qttestlib) > >- - How should cmake get the number of available test functions ? >- -> by counting the number of lines provided by the -function option or >by a specific command line switch which return the number of tests > >Are there other proposals and suggestions ? > >Additional requirements to the QtTestLib are welcome too as Thiago >Macieira mentioned > >>>>If anything is required of QtTestLib, now is the time to ask too. > >I think that such a support would affect whole KDE sources because there >are already QtTestLib based test in kdelibs and will be too in other >package.
You can always do ctest -V and get a more verbose output right now. We could change ctest to maybe have some regular expressions that can match output and produce output. ADD_TEST(foo EXTRA_OUTPUT_MATCH ".*tests passed.* failed") Then we could change QtTestLib to print out a line like this: 5 tests passed, 10 tests Then ctest would look for the extra output match in the tests output, and print out something like this: 1/ 6 Testing tst_qdbusconnection 5 tests passed, 10 tests ***Failed 2/ 6 Testing tst_qdbusmarshall 8 tests passed 0 failed I suppose the other option is to modify ctest to explicitly look for QtTestLib output. I would like to try to come up with a solution that would work for other projects as well. -Bill _______________________________________________ Kde-buildsystem mailing list [email protected] https://mail.kde.org/mailman/listinfo/kde-buildsystem
