Re: [Development] Code coverage statistics online
Bill, this is corrected now: http://download.froglogic.com/public/qt5-squishcoco-report/QtBase/source_543.html#line467 Regards Am 08.10.2012 um 12:26 schrieb Sébastien Fricker fric...@froglogic.com: Thanks Bill, this is effectively not correct, the lines should be marks as executed. Let me have a deeper look on it. On 08.10.2012 12:19, bill.k...@nokia.com wrote: Off-the-mailing-list… “QString QSqlDriver::sqlStatement(StatementType type, const QString tableName,” function, I think the marking is wrong. It says: “never executed (the execution status of this line is deducted): s.reserve(128);” for example, but it shows that by logic, that should be executed, as the first case of the switch statement below is… evaluated: i rec.count() TRUE FALSE yes Evaluation Count:128 yes Evaluation Count:64 Now, this may just be optimisation performed by the compiler, but it could also be erroneous. If it’s the first, I’d probably turn off for testing (there’s debug dumping functions that aren’t showing as being hit either). I really do think it’s the first, but I’d like it to be checked out in case it’s actually a bug or corner case. Thanks, Bill. -- Bill King Test Engineer Nokia, Guidance Team From: development-bounces+bill.king=nokia@qt-project.org [mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of ext Sébastien Fricker Sent: Sunday, 7 October 2012 9:08 AM To: development@qt-project.org Subject: Re: [Development] Code coverage statistics online One possible request: entry marking in the source reports, ie: QString QMimeType::filterString() const From: http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html Was the test entered but the if lines not tested? Another example: void QMimeTypePrivate::clear() It would be more understandable if the entry point was highlighted, instead of the exit point of the block. MfG, Bill. This is now available. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
Much more instantly-readable, thankyou very much :) -- Bill King Test Engineer Nokia, Guidance Team From: development-bounces+bill.king=nokia@qt-project.org [mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of ext Sébastien Fricker Sent: Sunday, 7 October 2012 9:08 AM To: development@qt-project.org Subject: Re: [Development] Code coverage statistics online One possible request: entry marking in the source reports, ie: QString QMimeType::filterString() const From: http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html Was the test entered but the if lines not tested? Another example: void QMimeTypePrivate::clear() It would be more understandable if the entry point was highlighted, instead of the exit point of the block. MfG, Bill. This is now available. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
One possible request: entry marking in the source reports, ie: QString QMimeType::filterString() const From: http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html Was the test entered but the if lines not tested? Another example: void QMimeTypePrivate::clear() It would be more understandable if the entry point was highlighted, instead of the exit point of the block. MfG, Bill. This is now available.___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
Hi, providing a code coverage analysis of Qt is a living project and so we would be happy of any feedback of the Qt developers. Issues and suggestions can be addressed to the Squish Coco forum: http://www.froglogic.com/forum/viewforum.php?f=10 We found that it is better to not put issues on metrics in the general Qt developer list, and so we prefer that people use a dedicated forum. Sébastien___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
On Wednesday 26. September 2012 20.06.21 Harri Porten wrote: Feedback of any kind is welcome. Nice! Jędrek ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
Overall the coverage was less than I expected, so I started reading the source files list starting from lowest coverage and found some surprising results, e.g. qmimedata.cpp 0% coverage, qsettings.cpp nearly 0% coverage although I know there are tests for these. In the execution list at http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html indeed it seems that tst_QSettings, tst_QMimeData are missing from the list :( I suppose there are others missing too. Any idea what might have caused them to be excluded from the results? If the tests are not present it can be: 1) the unit tests are not compiling. 2) the unit tests are not instrumented (should not be the case if part of QtBase) 3) make check does not execute the unit test. 4) results are not collected Let me have a look on it. Rohan, this analysis is the first one provided, and I you know better than everybody that maintaining the test suite is not an easy job. That's why it is important that if something is not correct that the community report it. This is the only way to get accurate measurements. Sébastien ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
Would you include add-ons in the reports too? Actually we start with QtBase. Of course in the future, the goal is to provide an analysis on the whole Qt source. Anyway to get run output logs from failed tests? Good suggestion, will be present in the next daily coverage report. Sébastien ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Code coverage statistics online
Awesome work guys :) One possible request: entry marking in the source reports, ie: QString QMimeType::filterString() const From: http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html Was the test entered but the if lines not tested? Another example: void QMimeTypePrivate::clear() It would be more understandable if the entry point was highlighted, instead of the exit point of the block. MfG, Bill. -- Bill King Test Engineer Nokia, Guidance Team From: development-bounces+bill.king=nokia@qt-project.org [mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of ext Sébastien Fricker Sent: Thursday, 27 September 2012 10:15 AM To: development@qt-project.org Subject: Re: [Development] Code coverage statistics online Overall the coverage was less than I expected, so I started reading the source files list starting from lowest coverage and found some surprising results, e.g. qmimedata.cpp 0% coverage, qsettings.cpp nearly 0% coverage although I know there are tests for these. In the execution list at http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.htmlhttp://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html indeed it seems that tst_QSettings, tst_QMimeData are missing from the list :( I suppose there are others missing too. Any idea what might have caused them to be excluded from the results? If the tests are not present it can be: 1) the unit tests are not compiling. 2) the unit tests are not instrumented (should not be the case if part of QtBase) 3) make check does not execute the unit test. 4) results are not collected Let me have a look on it. Rohan, this analysis is the first one provided, and I you know better than everybody that maintaining the test suite is not an easy job. That's why it is important that if something is not correct that the community report it. This is the only way to get accurate measurements. Sébastien ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] Code coverage statistics online
Hi! We started publishing code coverage results of Qt 5 unit test runs as produced by our tool Squish Coco: http://download.froglogic.com/public/qt5-squishcoco-report/ (results) http://download.froglogic.com/public/effectively-testing-qt5-using-squishcoco/ (setup and example) From that link you can browse to the coverage data of libraries, functions and source files. We chose a decision/coverage analysis method. In comparison to line coverage this is a bit harder to grasp and gives different numbers but for a developer it's the more useful. I plan to follow up with a study of a small number of exemplaric cases accompanied with suggested patches to tests and maybe code. And to make things easier to understand in the mostly-static HTML output we'll also improve the coloring of instrumented branches soon. Provided that the cron job works the results will be updated daily. If anyone has an idea how this could be integrated more tightly with the automated Qt project test results please let us know. Caroline Chao was already using a predecessor of the tool for the same purpose within the Nokia build infrastructure btw. Feedback of any kind is welcome. Harri. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
