Matt, The tree map visualization looks great! A great tool to pick out long running tests (if the fonts of the label could be scaled accordingly, that would be even better) !
Looks like the code coverage loss is not much. +1 on your patches. -Xiaoxiao On Tue, Sep 24, 2013 at 10:15 PM, Matt McCormick <[email protected] > wrote: > Hi Brad, > > > My ITK valgrind it taking too long. Just some test are just too long. > How would I disable these tests on a dashboard script? > > These tests could be disabled with > > set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG) > > > Also any idea what the change is coverage is? > > Yes, the decrease is about 0.26 %. Here are my local results: > > Default run 1: > Covered LOC: 123682 > Not covered LOC: 22557 > Total LOC: 146239 > Percentage Coverage: 84.58% > > Default run 2: > Covered LOC: 123671 > Not covered LOC: 22570 > Total LOC: 146241 > Percentage Coverage: 84.57% > > -LE RUNS_LONG: > Covered LOC: 123306 > Not covered LOC: 22942 > Total LOC: 146248 > Percentage Coverage: 84.31% > > Thanks, > Matt > > > > On Sep 20, 2013, at 6:40 PM, Matt McCormick <[email protected]> > wrote: > > > >> I did a visualization on test times [1] and it turns out that less > >> than 1% of the tests (0.97%) take of half (48.7%) of the test times! > >> > >> It would be good to add a "RUNS_LONG" label [2] to these tests so that > >> developers can run the test suite relatively quickly with > >> > >> ctest -LE RUNS_LONG > >> > >> and the Visual Studio Debug Nightly builds can exclude them with > >> > >> set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG) > >> > >> These tests are mostly integration tests that test complicated > >> algorithms on semi-realistic data. They have value, but leaving them > >> out in these situations should not do much harm relative to their time > >> overhead. > >> > >> Thanks, > >> Matt > >> > >> [1] http://review.source.kitware.com/#/c/12733/ > >> [2] http://review.source.kitware.com/#/c/12734/ > >> > >> > >> On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <[email protected]> > wrote: > >>> In general, we do want to make sure the tests produce "correct" > results and > >>> make sure there are no regressions. By having the Release build run a > proper > >>> set of iterations, the developer can verify the correctness. > >>> > >>> That said, using a lower resolution input dataset could probably help > most > >>> the these tests and the release/debug times would both be reduced. > >>> > >>> > >>> > >>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <[email protected]> wrote: > >>>> > >>>> > >>>>> I think it is OK to drop the iterations in the Debug tests. We are > still > >>>>> exercising the code. If we > >>>>> need alternate baselines, we will still catch exceptions. This will > also > >>>>> allow more valgrind tests > >>>>> to run to completion. > >>>> > >>>> > >>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word > “Fast” in > >>>> the test name... Perhaps we should at the very least consider a > rename. 😊 > >>>> > >>>> If it’s ok to drop the iterations in the Debug tests, then why not > drop > >>>> the iterations in the Release tests, too? There is something to be > said for > >>>> the “same” code being run for the test in both Debug and Release. If > it’s > >>>> not the same, then I should definitely be able to tell just by > glancing at > >>>> the source code for a test that there are Debug/non-Debug > differences... > >>>> > >>>> I would say each individual test should be able to run in *seconds*, > not > >>>> minutes or hours. Ideally, less than a second, but I realize that’s > overly > >>>> ambitious with some ITK code. > >>>> > >>>> However, for the main purpose I have in mind here, namely reducing the > >>>> time burden on my volunteer machine, I would be absolutely *ecstatic* > if all > >>>> of the 20 tests I listed in the original email were able to run in > under 4 > >>>> or 5 minutes each. > >>>> > >>>> Let me know if I can help out in some additional way (beyond just > >>>> continuing to submit the build to CDash...) > >>>> > >>>> > >>>> Thanks for the discussion, I appreciate it > >>>> D > >>>> > >>> > >>> > >>> > >>> > >>> -- > >>> Unpaid intern in BillsBasement at noware dot com > >>> > >>> _______________________________________________ > >>> Powered by www.kitware.com > >>> > >>> Visit other Kitware open-source projects at > >>> http://www.kitware.com/opensource/opensource.html > >>> > >>> Kitware offers ITK Training Courses, for more information visit: > >>> http://kitware.com/products/protraining.php > >>> > >>> Please keep messages on-topic and check the ITK FAQ at: > >>> http://www.itk.org/Wiki/ITK_FAQ > >>> > >>> Follow this link to subscribe/unsubscribe: > >>> http://www.itk.org/mailman/listinfo/insight-developers > >>> > >> _______________________________________________ > >> Powered by www.kitware.com > >> > >> Visit other Kitware open-source projects at > >> http://www.kitware.com/opensource/opensource.html > >> > >> Kitware offers ITK Training Courses, for more information visit: > >> http://kitware.com/products/protraining.php > >> > >> Please keep messages on-topic and check the ITK FAQ at: > >> http://www.itk.org/Wiki/ITK_FAQ > >> > >> Follow this link to subscribe/unsubscribe: > >> http://www.itk.org/mailman/listinfo/insight-developers > > > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Kitware offers ITK Training Courses, for more information visit: > http://kitware.com/products/protraining.php > > Please keep messages on-topic and check the ITK FAQ at: > http://www.itk.org/Wiki/ITK_FAQ > > Follow this link to subscribe/unsubscribe: > http://www.itk.org/mailman/listinfo/insight-developers > -- --------------------------------------------- *Xiaoxiao Liu*, Ph.D. R & D Engineer Kitware Inc <http://www.kitware.com/>. Clifton Park, NY Phone: (518) 881-4924 or (518) 371-3971 x124
_______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Kitware offers ITK Training Courses, for more information visit: http://kitware.com/products/protraining.php Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ Follow this link to subscribe/unsubscribe: http://www.itk.org/mailman/listinfo/insight-developers
