Re: [CMake] Benchmarking with CMake
On Friday, November 13, 2015 14:41:35 Nagy-Egri Máté Ferenc via CMake wrote: > Thank you all for your resonses, > > My use case is that I have a set of executables that function as unit tests. > Currently I use the return value of the .exe as a means of indicating > success, though I know there is way to match std::cout against a regex for > more complex tests. However, the timings of the tests are not always > suitable for benchmarking, because I only want the timings of a particular > part of the tests. Their initialization times are comparable to their > runtimes. While I could tune a parameter so that the init phase is > negligable, but then benchmarking would take for ages. I want to write > benchmark executables (similar to unit tests) that write a single number to > the console and nothing else. Run some command (make bench?, cbench?) which > runs, these tests, fetches the sole number from the std::cout and prints > those numbers as output instead of the total run time presented by ctest. > > @Peter: I also encountered Google Benchmark. I have already seen it’s output > and is perfectly fine. I am however a bit reluctant to grab another > dependency that all my consumers would be advised to install. CMake is a > fairly full blown suite and I was hoping to find something that could > provide similar results. > > @Matt: Seems nice. Python is not much a dependency, but VTK is less evident > to be present on all machines. > > @Alexander: I still have to look into this, because information seems a > little scarce on doing it. It seems to me, that CDash has to be installed > somewhere to be able visualize the data. Yes. > Is there a way to generate the > HTML-r for local consumption instead of submitting somewhere? (Ideally > bypassing a local installation of CDash) I don't think so. Alex -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Benchmarking with CMake
Thank you all for your resonses, My use case is that I have a set of executables that function as unit tests. Currently I use the return value of the .exe as a means of indicating success, though I know there is way to match std::cout against a regex for more complex tests. However, the timings of the tests are not always suitable for benchmarking, because I only want the timings of a particular part of the tests. Their initialization times are comparable to their runtimes. While I could tune a parameter so that the init phase is negligable, but then benchmarking would take for ages. I want to write benchmark executables (similar to unit tests) that write a single number to the console and nothing else. Run some command (make bench?, cbench?) which runs, these tests, fetches the sole number from the std::cout and prints those numbers as output instead of the total run time presented by ctest. @Peter: I also encountered Google Benchmark. I have already seen it’s output and is perfectly fine. I am however a bit reluctant to grab another dependency that all my consumers would be advised to install. CMake is a fairly full blown suite and I was hoping to find something that could provide similar results. @Matt: Seems nice. Python is not much a dependency, but VTK is less evident to be present on all machines. @Alexander: I still have to look into this, because information seems a little scarce on doing it. It seems to me, that CDash has to be installed somewhere to be able visualize the data. Is there a way to generate the HTML-r for local consumption instead of submitting somewhere? (Ideally bypassing a local installation of CDash) I know, I have a lot of problems. Ultimately I will choose one of the above, I was just looking for the one that is the least painful. I am very reluctant on obtaining dependencies. CMake is such a good tool, it seems strange to me that no direct benchmarking support has been added yet. Any ideas? Feladó: Nagy-Egri Máté Ferenc via CMake Elküldve: 2015. november 9., hétfő 23:27 Címzett: CMake felhasználók Tárgy: [CMake] Benchmarking with CMake Hi! Can anyone tell me if CMake (or CTest) can be customized in a way to produce decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I have not found any examples of CTest being (ab)used in this way. Is my feeling correct, that this should be something like CBench? Cheers, Máté -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Benchmarking with CMake
On Monday, November 09, 2015 23:21:58 Nagy-Egri Máté Ferenc via CMake wrote: > Hi! > > Can anyone tell me if CMake (or CTest) can be customized in a way to produce > decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I > have not found any examples of CTest being (ab)used in this way. > > Is my feeling correct, that this should be something like CBench? depending on what you want... You can make your test output tags, and cdash will create graphs from the measured numbers. I think they can also be exported to csv. Alex -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] Benchmarking with CMake
Hi Mate, This blog post has an example of parsing and visualizing CTestCostData.txt. Although, that file was not created for that purpose: http://www.kitware.com/blog/home/post/556 Cheers, Matt On Mon, Nov 9, 2015 at 5:21 PM, Nagy-Egri Máté Ferencwrote: > Hi! > > > > Can anyone tell me if CMake (or CTest) can be customized in a way to produce > decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I > have not found any examples of CTest being (ab)used in this way. > > > > Is my feeling correct, that this should be something like CBench? > > > > Cheers, > > Máté > > > > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
[CMake] Benchmarking with CMake
Hi! Can anyone tell me if CMake (or CTest) can be customized in a way to produce decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I have not found any examples of CTest being (ab)used in this way. Is my feeling correct, that this should be something like CBench? Cheers, Máté -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake