Re: [CMake] CTest question

[T.Sariyski]

Hi Eric,

 

I am sorry I did not state the question clear enough. Let me first say that 
tests does not depend on each other, and  running with or without –parallel 
does not make much of difference.

 

Here is how I set the tests . This is a Fortran code, it does not return. 
ADD_TEST command is a call to CMAKE, which (i) executes the Fortran code, and 
(ii) executes a python script, a metrics of how close the solution is to the 
reference data. I have a PASS_REGULAR_EXPRESSION property,  passRegex, 
associated with the test. When a test fails, the output is that the required 
regular expression ${passRegex} is not found. A test and the python script are 
attached. Please let me know if more information is needed. 

 

Thanks in advance,

Ted

 

 

CMAKE_MINIMUM_REQUIRED(VERSION 2.8)

project(project_name Fortran ) 

enable_testing()

 

set (SRUN_COMMAND /bin/sh -c "cd ${ CMAKE_CURRENT_BINARY_DIR } &&  
$ ${PROJECT_NAME}")

add_custom_target(${PROJECT_NAME}_srun COMMAND ${SRUN_COMMAND})

set(TARGET_NAME ${PROJECT_NAME}_serial)

add_test(NAME ${TARGET_NAME}

   WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}

   COMMAND ${CMAKE_COMMAND} 

 -D "PRE_DELETE_OUTPUT_FILES:BOOL=true" 

 -D "OUTPUT_FILES:STRING=${files_to_delete}" 

 -D "PROGRAM1:STRING=${SRUN_COMMAND}" 

 -D 
"PROGRAM2:STRING=python;${assert_case};${hs_work_file};${hs_gold_file};0"   
  

 -P "run_and_assert.cmake")

 

set(passRegex "CALCULATION TERMINATED BECAUSE STOP TIME ACHIEVED" "ctest: 
SUCCESS")

set_property(TEST ${TARGET_NAME}

 PROPERTY PASS_REGULAR_EXPRESSION "${passRegex}")



 

/> cat run_and_assert.cmake

…

EXECUTE_PROCESS(COMMAND ${PROGRAM1}

RESULT_VARIABLE error

ERROR_FILE err.log

OUTPUT_FILE out.log

)

IF(error)

MESSAGE(FATAL_ERROR "Executing \"${PROGRAM1}\" FAIED! error=${error}")

ENDIF()

 

EXECUTE_PROCESS(COMMAND ${PROGRAM2}

RESULT_VARIABLE status)

IF(status)

MESSAGE(SEND_ERROR "Executing \"${PROGRAM2}\" FAILED! status=${status}")

ENDIF()

 

 

 

From: Eric Noulard [mailto:eric.noul...@gmail.com] 
Sent: Wednesday, May 17, 2017 10:54 PM
To: T.Sariyski <tzsari...@gmail.com>
Cc: CMake ML <cmake@cmake.org>
Subject: Re: [CMake] CTest question

 

We lack information on how your test are failing. If they all timeout then may 
be you should setup a proper timeout for the test. The actual execution speed 
of a test may depend on the load of your machine or the network (depending on 
what the test are actually doing).

This may be even more sensible if your test are run in parallel. Do you run 
ctest with --parallel option? 

 

Do your tests depends on each other? If so did you setup test dependencies?

 

Le 17 mai 2017 10:59 PM, "T.Sariyski" <tzsari...@gmail.com 
<mailto:tzsari...@gmail.com> > a écrit :

Hi,

 

I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ 
and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the 
number of these false failures is about one third. I guess that the problem is 
in the way I set my tests, but I have no clue where to look, or how to ‘debug’ 
the ctest run.

Any advice is highly appreciated.

 

Thanks,

Ted


--

Powered by www.kitware.com <http://www.kitware.com> 

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] CTest question

[Eric Noulard]

We lack information on how your test are failing. If they all timeout then
may be you should setup a proper timeout for the test. The actual execution
speed of a test may depend on the load of your machine or the network
(depending on what the test are actually doing).
This may be even more sensible if your test are run in parallel. Do you run
ctest with --parallel option?

Do your tests depends on each other? If so did you setup test dependencies?

Le 17 mai 2017 10:59 PM, "T.Sariyski"  a écrit :

> Hi,
>
>
>
> I wonder what is the reason that sometimes ctest ends with numerous
> ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At
> times the number of these false failures is about one third. I guess that
> the problem is in the way I set my tests, but I have no clue where to look,
> or how to ‘debug’ the ctest run.
>
> Any advice is highly appreciated.
>
>
>
> Thanks,
>
> Ted
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

[CMake] CTest question

[T.Sariyski]

Hi,

 

I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ 
and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the 
number of these false failures is about one third. I guess that the problem is 
in the way I set my tests, but I have no clue where to look, or how to ‘debug’ 
the ctest run.

Any advice is highly appreciated.

 

Thanks,

Ted

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] CTest question...

[BRM]

It was reporting that it could not find the executable.
However, this was certainly a PEBKAC issue (do'h) - I forgot to have an 
ADD_EXECUTABLE() for the test program - or even an 
CREATE_TEST_SOURCE_LIST()...though so it shouldn't have been finding it. Though 
I am not sure what the best method is...I don't want these executables 
installed when I do a 'make install' later...while I am no where near needing 
to think about the install process (far too much to go before the project will 
be ready for that), it would be good to avoid doing it the 
wrong-even-if-it-works way.

Thanks for the tip on the ADD_TEST()...I was simply trying to follow the 
referenced WIKI page - it made no mention of needing to define the 
EXECUTE_OUTPUT_PATH variable...It probably should be amended to just read as 
you mentioned.

I've now got it working. Thanks!

Ben



- Original Message 
From: Eric NOULARD [EMAIL PROTECTED]
To: cmake@cmake.org
Sent: Wednesday, November 19, 2008 7:52:49 AM
Subject: Re: [CMake] CTest question...

Le Tue, 18 Nov 2008 15:53:25 -0800 (PST),
BRM [EMAIL PROTECTED] a écrit :

 Any info on this? Any better resources?

I think we miss informations about your particular configuration.
I do use ADD_TEST with no trouble.

See more question below: 

 From: BRM [EMAIL PROTECTED]
 To: CMake cmake@cmake.org
 Sent: Monday, November 17, 2008 3:14:53 AM
 Subject: [CMake] CTest question...
 
 I'd like to build a series of test using the CTest functionality in
 CMake. I have added ENABLE_TESTING() to the main CMakeLists, and was
 hoping to put the tests in their own sub-directories - as follows:
 
 /
 /Tests
 /Tests/object
 /Tests/object/child-objects
 
 I added the Tests directory in the root's CMakeLists.txt. I've
 created my first test (/Tests/object/objectTest) and tried adding it
 via ADD_TEST(objectTest ${EXECUTABLE_OUTPUT_PATH}/objectTest}) 

if objectTest is a target built in /Tests/object/objectTest
then you don't need the extra ${EXECUTABLE_OUTPUT_PATH} just put:

ADD_TEST(objectTest objectTest)

or in order to avoid name clash (I never experienced that but who knows)

ADD_TEST(objectTest_test objectTest)

 - no
 parameters needed (as described by
 http://www.vtk.org/Wiki/CMake_Testing_With_CTest). However, it's
 unable to find the test when I run make test.

What is the exact error shown by make test.
Do you get the same error when running

ctest 

 Note: I tried doing a MESSAGE() to find out the value of
 EXECUTABLE_OUTPUT_PATH and it seems to be NULL. Is that reall the
 right variable?

I don't think EXECUTABLE_OUTPUT_PATH is set by CMake unless you
decide to set it yourself. The default behavior of cmake is to put
built executable in CMAKE_CURRENT_BINARY_DIR.

Did you set EXECUTABLE_OUTPUT_PATH is your main CMakeLists.txt?

 The documentation on ENABLE_TESTING() mentions that CTest expects
 files in build's root directory. Does that also apply to ADD_TEST()?
 (no mention in ADD_TEST()'s documentation of this).

This is true for me but I do not use absolute file specification with
ADD_TEST, my ADD_TEST use target name previsouly built with
ADD_EXECUTABLE with no problem.


-- 
Erk
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest question...

[Eric Noulard]

2008/11/20 BRM [EMAIL PROTECTED]:
 It was reporting that it could not find the executable.
 However, this was certainly a PEBKAC issue (do'h) - I forgot to have an 
 ADD_EXECUTABLE() for the test program - or even an 
 CREATE_TEST_SOURCE_LIST()...though so it shouldn't have been finding it. 
 Though I am not sure what the best method is...I don't want these executables 
 installed when I do a 'make install' later...

You may ADD_EXECUTABLE without
corresponding INSTALL
this way built executable won't be installed at all.
This is usually the case with test executable.

 while I am no where near needing to think about the install process (far too 
 much to go before the project will be ready for that), it would be good to 
 avoid doing it the wrong-even-if-it-works way.

 Thanks for the tip on the ADD_TEST()...I was simply trying to follow the 
 referenced WIKI page -

 it made no mention of needing to define the EXECUTE_OUTPUT_PATH variable...
 It probably should be amended to just read as you mentioned.

Go ahead with the sharing of your experience.
It's a Wiki, open an account and add more valuable informations :=)

 I've now got it working. Thanks!

Good point.
-- 
Erk
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest question...

[BRM]

Any info on this? Any better resources?

TIA,

Ben



- Original Message 
From: BRM [EMAIL PROTECTED]
To: CMake cmake@cmake.org
Sent: Monday, November 17, 2008 3:14:53 AM
Subject: [CMake] CTest question...

I'd like to build a series of test using the CTest functionality in CMake.
I have added ENABLE_TESTING() to the main CMakeLists, and was hoping to put the 
tests in their own sub-directories - as follows:

/
/Tests
/Tests/object
/Tests/object/child-objects

I added the Tests directory in the root's CMakeLists.txt. I've created my first 
test (/Tests/object/objectTest) and tried adding it via ADD_TEST(objectTest 
${EXECUTABLE_OUTPUT_PATH}/objectTest}) - no parameters needed (as described by 
http://www.vtk.org/Wiki/CMake_Testing_With_CTest). However, it's unable to find 
the test when I run make test.

Note: I tried doing a MESSAGE() to find out the value of EXECUTABLE_OUTPUT_PATH 
and it seems to be NULL. Is that reall the right variable?

The documentation on ENABLE_TESTING() mentions that CTest expects files in 
build's root directory. Does that also apply to ADD_TEST()? (no mention in 
ADD_TEST()'s documentation of this).

Advice appreciated.

TIA,

Ben

___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest question...

[Eric NOULARD]

Le Tue, 18 Nov 2008 15:53:25 -0800 (PST),
BRM [EMAIL PROTECTED] a écrit :

 Any info on this? Any better resources?

I think we miss informations about your particular configuration.
I do use ADD_TEST with no trouble.

See more question below: 

 From: BRM [EMAIL PROTECTED]
 To: CMake cmake@cmake.org
 Sent: Monday, November 17, 2008 3:14:53 AM
 Subject: [CMake] CTest question...
 
 I'd like to build a series of test using the CTest functionality in
 CMake. I have added ENABLE_TESTING() to the main CMakeLists, and was
 hoping to put the tests in their own sub-directories - as follows:
 
 /
 /Tests
 /Tests/object
 /Tests/object/child-objects
 
 I added the Tests directory in the root's CMakeLists.txt. I've
 created my first test (/Tests/object/objectTest) and tried adding it
 via ADD_TEST(objectTest ${EXECUTABLE_OUTPUT_PATH}/objectTest}) 

if objectTest is a target built in /Tests/object/objectTest
then you don't need the extra ${EXECUTABLE_OUTPUT_PATH} just put:

ADD_TEST(objectTest objectTest)

or in order to avoid name clash (I never experienced that but who knows)

ADD_TEST(objectTest_test objectTest)

 - no
 parameters needed (as described by
 http://www.vtk.org/Wiki/CMake_Testing_With_CTest). However, it's
 unable to find the test when I run make test.

What is the exact error shown by make test.
Do you get the same error when running

ctest 
 
 Note: I tried doing a MESSAGE() to find out the value of
 EXECUTABLE_OUTPUT_PATH and it seems to be NULL. Is that reall the
 right variable?

I don't think EXECUTABLE_OUTPUT_PATH is set by CMake unless you
decide to set it yourself. The default behavior of cmake is to put
built executable in CMAKE_CURRENT_BINARY_DIR.

Did you set EXECUTABLE_OUTPUT_PATH is your main CMakeLists.txt?

 The documentation on ENABLE_TESTING() mentions that CTest expects
 files in build's root directory. Does that also apply to ADD_TEST()?
 (no mention in ADD_TEST()'s documentation of this).

This is true for me but I do not use absolute file specification with
ADD_TEST, my ADD_TEST use target name previsouly built with
ADD_EXECUTABLE with no problem.


-- 
Erk
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

RE: [CMake] ctest question

[Kedzierski, Artur CIV NAVSURFWARCENDIV CORONA]

I've ran into similar problem.
Instead of keeping the original files as golden results, I 
wanted to keep MD5 checksums of these files. There are two reasons 
for it: disk space constraints and different file format
support. The generated files can take up a lot of disk space. If
a MD5 checksum difference is detected, I can roll back a revision
and generate the original file.  Also, I wanted to perform similar
test on software that generates images (jpeg, etc.).
Because these scheme has to work on Linux and Windows, I've
wrote python script that computes file's MD5 checksum and compares
it to given golden result. This is something that I've done (CheckMD5.py
is the script mentioned before):

FIND_PACKAGE(PytonInterp)
IF(PYTHONINTERP_FOUND)
ADD_TEST(test1 my_program ${CMAKE_BINARY_DIR}/test.txt)
ADD_TEST(test2 ${PYTHON_EXECUTABLE}
${PROJECT_SOURCE_DIR}/CheckMD5.py ${CMAKE_BINARY_DIR}/test.txt
1a3fdcea)

ENDIF(PYTHONINTERP_FOUND)

That did the trick. However, now Dart2 shows reports about
possible memory leaks inside of python interpreter. Has anybody 
else did something similar? It would be if CMake contained
macro like this:
ADD_MD5_TEST(testname generatedfile md5checksum)


--
Artur Kedzierski 

 -Original Message-
 From: [EMAIL PROTECTED] 
 [mailto:[EMAIL PROTECTED] On 
 Behalf Of Juan Sanchez
 Sent: Sunday, September 23, 2007 12:49
 To: CMake ML
 Subject: [CMake] ctest question
 
 I want run a test program and pipe its results to a file.  I 
 then want to compare this file to the golden results using diff.
 
 Does anyone have a macro or cookbook example for doing this?
 
 Thank you,
 
 Juan
 
 
 ___
 CMake mailing list
 CMake@cmake.org
 http://www.cmake.org/mailman/listinfo/cmake
 
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] ctest question

[Mathieu MARACHE]

2007/9/25, Juan Sanchez [EMAIL PROTECTED]:
 Hi Alan,

 I also have floating point results I want to account for.  I'm thinking
 about writing a diff script for numerical results which uses an absolute
 and relative error tolerance.  This would account for the difference in
 transcendentals and other math functions between processors and math
 libraries.  I always disable 80 bit extended precision on linux since
 the results are non-deterministic with respect to compiler settings.

diff with numerical precision concerns is ndiff :
http://www.math.utah.edu/~beebe/software/ndiff

HTH

 Regards,

 Juan


 Alan W. Irwin wrote:
  On 2007-09-24 10:05-0500 Juan Sanchez wrote:
 
  Hello Alan,
 
  From your example, what in this statement that causes the test to run
  when I type make test?
 
  ADD_TEST(my_first_test diff -q goldenfile testfile)
 
  All it says is to run diff.  How do I tell it to generate the testfile
  from another executable?  How do I tell this executable to run only when
  I type make test and not a moment before?
 
  Hi Juan:
 
  In the above simple example diff is run only when you run ctest (or I
  guess make test although I don't use that). So you could do something like
 
  ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/create_testfile; diff -q 
  goldenfile testfile)
 
  subject to escaping of ; which I can never get right until I experiment.
  (This general command-line approach of separating commands with ; only
  works on Unix, I believe.)
 
  Then the create_testfile executable is run at ctest time to create testfile
  and then diff is run immediately afterwards (which appears to be what you
  want).
 
  A better approach would be to put everything you want including the diff
  into a configurable script, e.g.,
 
  ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/test1.sh)
 
  Note, in this case, the script is configured using CONFIGURE_FILE at
  cmake time (basically by substituting CMake-defined variables when needed),
  but run only at ctest time.
 
  Our tests don't use diff (because postscript PLplot results are slightly
  platform/compiler version dependent because of floating-point rounding
  issues), but we do use a configurable scripting approach to generate our
  test plots, see
  http://plplot.svn.sourceforge.net/viewvc/plplot/trunk/test/CMakeLists.txt?view=log
 
  I don't recommend you wade through _all_ of that CMake logic and bash script
  logic since it is so specific to our PLplot needs (and also its pretty
  voluminous/hierarchical since it deals with hundreds of test plots), but I
  have given you the above starting reference in case you have trouble
  configuring test scripts for yourself using CONFIGURE_FILE.
 
  Alan
  __
  Alan W. Irwin
 
  Astronomical research affiliation with Department of Physics and Astronomy,
  University of Victoria (astrowww.phys.uvic.ca).
 
  Programming affiliations with the FreeEOS equation-of-state implementation
  for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
  package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
  Linux Links project (loll.sf.net); and the Linux Brochure Project
  (lbproject.sf.net).
  __
 
  Linux-powered Science
  __
 
 


 --
 Juan Sanchez
 [EMAIL PROTECTED]
 800-538-8450 Ext. 54395
 512-602-4395


 ___
 CMake mailing list
 CMake@cmake.org
 http://www.cmake.org/mailman/listinfo/cmake



-- 
Mathieu
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] ctest question

[Mathieu MARACHE]

2007/9/25, Juan Sanchez [EMAIL PROTECTED]:
 Hi Alan,

 I also have floating point results I want to account for.  I'm thinking
 about writing a diff script for numerical results which uses an absolute
 and relative error tolerance.  This would account for the difference in
 transcendentals and other math functions between processors and math
 libraries.  I always disable 80 bit extended precision on linux since
 the results are non-deterministic with respect to compiler settings.

diff with numerical precision concerns is ndiff :
http://www.math.utah.edu/~beebe/software/ndiff

HTH

 Regards,

 Juan


 Alan W. Irwin wrote:
  On 2007-09-24 10:05-0500 Juan Sanchez wrote:
 
  Hello Alan,
 
  From your example, what in this statement that causes the test to run
  when I type make test?
 
  ADD_TEST(my_first_test diff -q goldenfile testfile)
 
  All it says is to run diff.  How do I tell it to generate the testfile
  from another executable?  How do I tell this executable to run only when
  I type make test and not a moment before?
 
  Hi Juan:
 
  In the above simple example diff is run only when you run ctest (or I
  guess make test although I don't use that). So you could do something like
 
  ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/create_testfile; diff -q 
  goldenfile testfile)
 
  subject to escaping of ; which I can never get right until I experiment.
  (This general command-line approach of separating commands with ; only
  works on Unix, I believe.)
 
  Then the create_testfile executable is run at ctest time to create testfile
  and then diff is run immediately afterwards (which appears to be what you
  want).
 
  A better approach would be to put everything you want including the diff
  into a configurable script, e.g.,
 
  ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/test1.sh)
 
  Note, in this case, the script is configured using CONFIGURE_FILE at
  cmake time (basically by substituting CMake-defined variables when needed),
  but run only at ctest time.
 
  Our tests don't use diff (because postscript PLplot results are slightly
  platform/compiler version dependent because of floating-point rounding
  issues), but we do use a configurable scripting approach to generate our
  test plots, see
  http://plplot.svn.sourceforge.net/viewvc/plplot/trunk/test/CMakeLists.txt?view=log
 
  I don't recommend you wade through _all_ of that CMake logic and bash script
  logic since it is so specific to our PLplot needs (and also its pretty
  voluminous/hierarchical since it deals with hundreds of test plots), but I
  have given you the above starting reference in case you have trouble
  configuring test scripts for yourself using CONFIGURE_FILE.
 
  Alan
  __
  Alan W. Irwin
 
  Astronomical research affiliation with Department of Physics and Astronomy,
  University of Victoria (astrowww.phys.uvic.ca).
 
  Programming affiliations with the FreeEOS equation-of-state implementation
  for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
  package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
  Linux Links project (loll.sf.net); and the Linux Brochure Project
  (lbproject.sf.net).
  __
 
  Linux-powered Science
  __
 
 


 --
 Juan Sanchez
 [EMAIL PROTECTED]
 800-538-8450 Ext. 54395
 512-602-4395


 ___
 CMake mailing list
 CMake@cmake.org
 http://www.cmake.org/mailman/listinfo/cmake



-- 
Mathieu
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] ctest question

[Juan Sanchez]

Awesome.  I love that guys work on bibtex and bibtools.

Juan

Mathieu MARACHE wrote:
 2007/9/25, Juan Sanchez [EMAIL PROTECTED]:
 Hi Alan,

 I also have floating point results I want to account for.  I'm thinking
 about writing a diff script for numerical results which uses an absolute
 and relative error tolerance.  This would account for the difference in
 transcendentals and other math functions between processors and math
 libraries.  I always disable 80 bit extended precision on linux since
 the results are non-deterministic with respect to compiler settings.
 
 diff with numerical precision concerns is ndiff :
 http://www.math.utah.edu/~beebe/software/ndiff
 
 HTH
 
 Regards,

 Juan


 Alan W. Irwin wrote:
 On 2007-09-24 10:05-0500 Juan Sanchez wrote:

 Hello Alan,

 From your example, what in this statement that causes the test to run
 when I type make test?

 ADD_TEST(my_first_test diff -q goldenfile testfile)

 All it says is to run diff.  How do I tell it to generate the testfile
 from another executable?  How do I tell this executable to run only when
 I type make test and not a moment before?
 Hi Juan:

 In the above simple example diff is run only when you run ctest (or I
 guess make test although I don't use that). So you could do something like

 ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/create_testfile; diff -q 
 goldenfile testfile)

 subject to escaping of ; which I can never get right until I experiment.
 (This general command-line approach of separating commands with ; only
 works on Unix, I believe.)

 Then the create_testfile executable is run at ctest time to create testfile
 and then diff is run immediately afterwards (which appears to be what you
 want).

 A better approach would be to put everything you want including the diff
 into a configurable script, e.g.,

 ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/test1.sh)

 Note, in this case, the script is configured using CONFIGURE_FILE at
 cmake time (basically by substituting CMake-defined variables when needed),
 but run only at ctest time.

 Our tests don't use diff (because postscript PLplot results are slightly
 platform/compiler version dependent because of floating-point rounding
 issues), but we do use a configurable scripting approach to generate our
 test plots, see
 http://plplot.svn.sourceforge.net/viewvc/plplot/trunk/test/CMakeLists.txt?view=log

 I don't recommend you wade through _all_ of that CMake logic and bash script
 logic since it is so specific to our PLplot needs (and also its pretty
 voluminous/hierarchical since it deals with hundreds of test plots), but I
 have given you the above starting reference in case you have trouble
 configuring test scripts for yourself using CONFIGURE_FILE.

 Alan
 __
 Alan W. Irwin

 Astronomical research affiliation with Department of Physics and Astronomy,
 University of Victoria (astrowww.phys.uvic.ca).

 Programming affiliations with the FreeEOS equation-of-state implementation
 for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
 package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
 Linux Links project (loll.sf.net); and the Linux Brochure Project
 (lbproject.sf.net).
 __

 Linux-powered Science
 __



___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] ctest question

[Juan Sanchez]

Hi Alan,

I also have floating point results I want to account for.  I'm thinking
about writing a diff script for numerical results which uses an absolute
and relative error tolerance.  This would account for the difference in
transcendentals and other math functions between processors and math
libraries.  I always disable 80 bit extended precision on linux since
the results are non-deterministic with respect to compiler settings.

Regards,

Juan


Alan W. Irwin wrote:
 On 2007-09-24 10:05-0500 Juan Sanchez wrote:
 
 Hello Alan,

 From your example, what in this statement that causes the test to run
 when I type make test?

 ADD_TEST(my_first_test diff -q goldenfile testfile)

 All it says is to run diff.  How do I tell it to generate the testfile
 from another executable?  How do I tell this executable to run only when
 I type make test and not a moment before?
 
 Hi Juan:
 
 In the above simple example diff is run only when you run ctest (or I
 guess make test although I don't use that). So you could do something like
 
 ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/create_testfile; diff -q 
 goldenfile testfile)
 
 subject to escaping of ; which I can never get right until I experiment.
 (This general command-line approach of separating commands with ; only
 works on Unix, I believe.)
 
 Then the create_testfile executable is run at ctest time to create testfile
 and then diff is run immediately afterwards (which appears to be what you
 want).
 
 A better approach would be to put everything you want including the diff
 into a configurable script, e.g.,
 
 ADD_TEST(my_first_test ${CMAKE_CURRENT_BINARY_DIR}/test1.sh)
 
 Note, in this case, the script is configured using CONFIGURE_FILE at
 cmake time (basically by substituting CMake-defined variables when needed),
 but run only at ctest time.
 
 Our tests don't use diff (because postscript PLplot results are slightly
 platform/compiler version dependent because of floating-point rounding
 issues), but we do use a configurable scripting approach to generate our
 test plots, see
 http://plplot.svn.sourceforge.net/viewvc/plplot/trunk/test/CMakeLists.txt?view=log
 
 I don't recommend you wade through _all_ of that CMake logic and bash script
 logic since it is so specific to our PLplot needs (and also its pretty
 voluminous/hierarchical since it deals with hundreds of test plots), but I
 have given you the above starting reference in case you have trouble
 configuring test scripts for yourself using CONFIGURE_FILE.
 
 Alan
 __
 Alan W. Irwin
 
 Astronomical research affiliation with Department of Physics and Astronomy,
 University of Victoria (astrowww.phys.uvic.ca).
 
 Programming affiliations with the FreeEOS equation-of-state implementation
 for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
 package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
 Linux Links project (loll.sf.net); and the Linux Brochure Project
 (lbproject.sf.net).
 __
 
 Linux-powered Science
 __
 
 


-- 
Juan Sanchez
[EMAIL PROTECTED]
800-538-8450 Ext. 54395
512-602-4395


___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

[CMake] ctest question

[Juan Sanchez]

I want run a test program and pipe its results to a file.  I then want
to compare this file to the golden results using diff.

Does anyone have a macro or cookbook example for doing this?

Thank you,

Juan


___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest Question

[William A. Hoffman]

At 04:51 PM 6/29/2006, Scott Amort wrote:
William A. Hoffman wrote:
 make test should do the trick.
 You can also run ctest directly in the build tree.
 What you have should work.  You might have to move the
 ENABLE_TESTING to the top of the project.  Also you can
 look for the DartTesting.  You can look for this file
 in your build tree and make sure it looks good: 
 DartTestfile.txt

Thanks Bill - I just needed to move ENABLE_TESTING up to the project
root.  Works now!

However, it does compile the testing executables during the `make'
command, and only runs them during the `make test'.  If I make a change
to one of the tests, I have to run `make' again, then `make test'...
`make test' alone doesn't see that changes were made to the sources.
Certainly only a minor complaint, but is there a way to restrict the
compiling/link/executing of tests to the `make test' command?

You may want to look at the ctest --build-and-test option.
We use it in cmake itself.  It will run cmake and do a build on
a test that is not include in the parent build tree.

-Bill

___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest Question

[Scott Amort]

William A. Hoffman wrote:
  You may want to look at the ctest --build-and-test option.
 We use it in cmake itself.  It will run cmake and do a build on
 a test that is not include in the parent build tree.

Hi Bill,

I will look into this.  Thanks again for your assistance!

Best,
Scott
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

[CMake] CTest Question

[Scott Amort]

Hi All,

I have one last hurdle in converting my project over to CMake - unit
testing.  I don't quite understand the CTest framework, and it doesn't
seem to be doing what I expect.  In my tests subdirectory, I have this
CMakeLists.txt:

#for testing
ENABLE_TESTING()

#build testing executables
ADD_EXECUTABLE(ExceptionTest ExceptionTest.cpp)

#add tests
ADD_TEST(exception_test ExceptionTest)

Currently, I generate various unit tests (using CxxTest) that result in
a series of .hpp and .cpp files in my tests subdirectory (e.g.
ExceptionTest).  With the older autotools setup, I could execute `make
check', which would then run these tests.  I expected that with CMake, I
could do a `make test' and get the same result, but it does not seem to
be working this way (although it does build and link the executable).
It would also be preferable that the testing sources aren't compiled
with the project sources, but only when a `make test' or equivalent
command is run.

Is CTest the right framework to be using?  Am I misunderstanding how to
use it?  There is not much documentation on the subject.

Thanks for any assistance!

Best,
Scott
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] CTest Question

[William A. Hoffman]

At 01:03 PM 6/29/2006, Scott Amort wrote:
Hi All,

I have one last hurdle in converting my project over to CMake - unit
testing.  I don't quite understand the CTest framework, and it doesn't
seem to be doing what I expect.  In my tests subdirectory, I have this
CMakeLists.txt:

#for testing
ENABLE_TESTING()

#build testing executables
ADD_EXECUTABLE(ExceptionTest ExceptionTest.cpp)

#add tests
ADD_TEST(exception_test ExceptionTest)

Currently, I generate various unit tests (using CxxTest) that result in
a series of .hpp and .cpp files in my tests subdirectory (e.g.
ExceptionTest).  With the older autotools setup, I could execute `make
check', which would then run these tests.  I expected that with CMake, I
could do a `make test' and get the same result, but it does not seem to
be working this way (although it does build and link the executable).
It would also be preferable that the testing sources aren't compiled
with the project sources, but only when a `make test' or equivalent
command is run.

Is CTest the right framework to be using?  Am I misunderstanding how to
use it?  There is not much documentation on the subject.

Thanks for any assistance!

Have you looked at the Wiki:

http://www.cmake.org/Wiki/CMake_Testing_With_CTest#Introduction

make test should do the trick.
You can also run ctest directly in the build tree.
What you have should work.  You might have to move the
ENABLE_TESTING to the top of the project.  Also you can
look for the DartTesting.  You can look for this file
in your build tree and make sure it looks good: 
DartTestfile.txt

-Bill


-Bill

___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake