Hi Eric,
I noticed something that may relates to the problem. Like I said in my previous mail, add_test is a call to cmake: ADD_TEST(NAME ${TARGET_NAME} … COMMAND ${CMAKE_COMMAND} -DPROGRAM1 -DPROGRAM2 … Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically? Here is what I noticed. If a test is executed from its own binary directory, ‘ctest -VV’, it always pass. If the same test is executed from the main binary directory, ‘ctest -VV -R zamod’, it fails because PROGRAM2 is executed before PROGRAM1 is finished. Attached are the screen outputs for both runs. Thanks, Ted ================================================================ cd CTEST/zamod s1002807:zamod tesari$ ctest -VV UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl Test project /Users/tesari/build/gnu.dbg/CTEST/zamod Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph... Checking test dependency graph end test 1 Start 1: zamod_serial 1: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake" 1: Test timeout computed to be: 9.99988e+06 1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" 1: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" 1: /Users/tesari/kodiak/trunk/benchmark/Scripts 1: Number of arguments: 4 arguments. 1: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0'] 1: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3 arguments: 1: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk 1: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk 1: 3 0 1: zamod Q0.4 PASS 1: zamod Q0.5 PASS 1: Kodiak ctest: SUCCESS 1/1 Test #1: zamod_serial ..................... Passed 5.73 sec ================================================================ s1002807:gnu.dbg tesari$ ctest -VV -R zamod UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl Test project /Users/tesari/build/gnu.dbg Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph... Checking test dependency graph end test 113 Start 113: zamod_serial 113: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake" 113: Test timeout computed to be: 1500 113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" 113: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" 113: /Users/tesari/kodiak/trunk/benchmark/Scripts 113: Number of arguments: 4 arguments. 113: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0'] 113: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3 arguments: 113: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk 113: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk 113: 3 0 113: /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk does not exist! From: Eric Noulard [mailto:eric.noul...@gmail.com] Sent: Wednesday, May 17, 2017 10:54 PM To: T.Sariyski <tzsari...@gmail.com <mailto:tzsari...@gmail.com> > Cc: CMake ML <cmake@cmake.org <mailto:cmake@cmake.org> > Subject: Re: [CMake] CTest question We lack information on how your test are failing. If they all timeout then may be you should setup a proper timeout for the test. The actual execution speed of a test may depend on the load of your machine or the network (depending on what the test are actually doing). This may be even more sensible if your test are run in parallel. Do you run ctest with --parallel option? Do your tests depends on each other? If so did you setup test dependencies? Le 17 mai 2017 10:59 PM, "T.Sariyski" < <mailto:tzsari...@gmail.com> tzsari...@gmail.com> a écrit : Hi, I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the number of these false failures is about one third. I guess that the problem is in the way I set my tests, but I have no clue where to look, or how to ‘debug’ the ctest run. Any advice is highly appreciated. Thanks, Ted -- Powered by www.kitware.com <http://www.kitware.com> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake