Re: [CMake] Benchmarking with CMake

[Alexander Neundorf]

On Friday, November 13, 2015 14:41:35 Nagy-Egri Máté Ferenc via 
CMake wrote:
> Thank you all for your resonses,
> 
> My use case is that I have a set of executables that function as unit 
tests.
> Currently I use the return value of the .exe as a means of indicating
> success, though I know there is way to match std::cout against a 
regex for
> more complex tests. However, the timings of the tests are not always
> suitable for benchmarking, because I only want the timings of a 
particular
> part of the tests. Their initialization times are comparable to their
> runtimes. While I could tune a parameter so that the init phase is
> negligable, but then benchmarking would take for ages. I want to write
> benchmark executables (similar to unit tests) that write a single 
number to
> the console and nothing else. Run some command (make bench?, 
cbench?) which
> runs, these tests, fetches the sole number from the std::cout and 
prints
> those numbers as output instead of the total run time presented by 
ctest.
> 
> @Peter: I also encountered Google Benchmark. I have already seen 
it’s output
> and is perfectly fine. I am however a bit reluctant to grab another
> dependency that all my consumers would be advised to install. 
CMake is a
> fairly full blown suite and I was hoping to find something that could
> provide similar results.
> 
> @Matt: Seems nice. Python is not much a dependency, but VTK is 
less evident
> to be present on all machines.
> 
> @Alexander: I still have to look into this, because information seems a
> little scarce on doing it. It seems to me, that CDash has to be installed
> somewhere to be able visualize the data. 

Yes.

> Is there a way to generate the
> HTML-r for local consumption instead of submitting somewhere? 
(Ideally
> bypassing a local installation of CDash)

I don't think so.

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Benchmarking with CMake

[Nagy-Egri Máté Ferenc via CMake]

Thank you all for your resonses,

My use case is that I have a set of executables that function as unit tests. 
Currently I use the return value of the .exe as a means of indicating success, 
though I know there is way to match std::cout against a regex for more complex 
tests. However, the timings of the tests are not always suitable for 
benchmarking, because I only want the timings of a particular part of the 
tests. Their initialization times are comparable to their runtimes. While I 
could tune a parameter so that the init phase is negligable, but then 
benchmarking would take for ages. I want to write benchmark executables 
(similar to unit tests) that write a single number to the console and nothing 
else. Run some command (make bench?, cbench?) which runs, these tests, fetches 
the sole number from the std::cout and prints those numbers as output instead 
of the total run time presented by ctest.

@Peter: I also encountered Google Benchmark. I have already seen it’s output 
and is perfectly fine. I am however a bit reluctant to grab another dependency 
that all my consumers would be advised to install. CMake is a fairly full blown 
suite and I was hoping to find something that could provide similar results.

@Matt: Seems nice. Python is not much a dependency, but VTK is less evident to 
be present on all machines.

@Alexander: I still have to look into this, because information seems a little 
scarce on doing it. It seems to me, that CDash has to be installed somewhere to 
be able visualize the data. Is there a way to generate the HTML-r for local 
consumption instead of submitting somewhere? (Ideally bypassing a local 
installation of CDash)

I know, I have a lot of problems. Ultimately I will choose one of the above, I 
was just looking for the one that is the least painful. I am very reluctant on 
obtaining dependencies. CMake is such a good tool, it seems strange to me that 
no direct benchmarking support has been added yet.

Any ideas?


Feladó: Nagy-Egri Máté Ferenc via CMake
Elküldve: 2015. november 9., hétfő 23:27
Címzett: CMake felhasználók
Tárgy: [CMake] Benchmarking with CMake


Hi!

Can anyone tell me if CMake (or CTest) can be customized in a way to produce 
decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I 
have not found any examples of CTest being (ab)used in this way.

Is my feeling correct, that this should be something like CBench?

Cheers,
Máté



-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Benchmarking with CMake

[Alexander Neundorf]

On Monday, November 09, 2015 23:21:58 Nagy-Egri Máté Ferenc via 
CMake wrote:
> Hi!
> 
> Can anyone tell me if CMake (or CTest) can be customized in a way to 
produce
> decent benchmark output? An .xlsx perhaps, or something GnuPlot 
friendly? I
> have not found any examples of CTest being (ab)used in this way.
> 
> Is my feeling correct, that this should be something like CBench?


depending on what you want...
You can make your test output  tags, and cdash will 
create graphs from the measured numbers. I think they can also be 
exported to csv.

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Benchmarking with CMake

[Matt McCormick]

Hi Mate,

This blog post has an example of parsing and visualizing
CTestCostData.txt.  Although, that file was not created for that
purpose:

  http://www.kitware.com/blog/home/post/556

Cheers,
Matt

On Mon, Nov 9, 2015 at 5:21 PM, Nagy-Egri Máté Ferenc  wrote:
> Hi!
>
>
>
> Can anyone tell me if CMake (or CTest) can be customized in a way to produce
> decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I
> have not found any examples of CTest being (ab)used in this way.
>
>
>
> Is my feeling correct, that this should be something like CBench?
>
>
>
> Cheers,
>
> Máté
>
>
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake