I did a visualization on test times [1] and it turns out that less than 1% of the tests (0.97%) take of half (48.7%) of the test times!
It would be good to add a "RUNS_LONG" label [2] to these tests so that developers can run the test suite relatively quickly with ctest -LE RUNS_LONG and the Visual Studio Debug Nightly builds can exclude them with set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG) These tests are mostly integration tests that test complicated algorithms on semi-realistic data. They have value, but leaving them out in these situations should not do much harm relative to their time overhead. Thanks, Matt [1] http://review.source.kitware.com/#/c/12733/ [2] http://review.source.kitware.com/#/c/12734/ On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <[email protected]> wrote: > In general, we do want to make sure the tests produce "correct" results and > make sure there are no regressions. By having the Release build run a proper > set of iterations, the developer can verify the correctness. > > That said, using a lower resolution input dataset could probably help most > the these tests and the release/debug times would both be reduced. > > > > On Fri, Aug 2, 2013 at 10:49 AM, David Cole <[email protected]> wrote: >> >> >> > I think it is OK to drop the iterations in the Debug tests. We are still >> > exercising the code. If we >> > need alternate baselines, we will still catch exceptions. This will also >> > allow more valgrind tests >> > to run to completion. >> >> >> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in >> the test name... Perhaps we should at the very least consider a rename. 😊 >> >> If it’s ok to drop the iterations in the Debug tests, then why not drop >> the iterations in the Release tests, too? There is something to be said for >> the “same” code being run for the test in both Debug and Release. If it’s >> not the same, then I should definitely be able to tell just by glancing at >> the source code for a test that there are Debug/non-Debug differences... >> >> I would say each individual test should be able to run in *seconds*, not >> minutes or hours. Ideally, less than a second, but I realize that’s overly >> ambitious with some ITK code. >> >> However, for the main purpose I have in mind here, namely reducing the >> time burden on my volunteer machine, I would be absolutely *ecstatic* if all >> of the 20 tests I listed in the original email were able to run in under 4 >> or 5 minutes each. >> >> Let me know if I can help out in some additional way (beyond just >> continuing to submit the build to CDash...) >> >> >> Thanks for the discussion, I appreciate it >> D >> > > > > > -- > Unpaid intern in BillsBasement at noware dot com > > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Kitware offers ITK Training Courses, for more information visit: > http://kitware.com/products/protraining.php > > Please keep messages on-topic and check the ITK FAQ at: > http://www.itk.org/Wiki/ITK_FAQ > > Follow this link to subscribe/unsubscribe: > http://www.itk.org/mailman/listinfo/insight-developers > _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Kitware offers ITK Training Courses, for more information visit: http://kitware.com/products/protraining.php Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ Follow this link to subscribe/unsubscribe: http://www.itk.org/mailman/listinfo/insight-developers
