Yup. ctest -N and ctest without -N differ in test ordering. Congratulations! You've reported the first regression bug in CMake 2.8.4....! :-)
I suspect the change for this bug [ http://public.kitware.com/Bug/view.php?id=11561 ] has introduced this issue. I'll make a new issue that we can hopefully address fairly soon. Thanks, David P.S. ======= The same build tree with a non -N ctest run gives me this: $ ctest Test project /Users/davidcole/tmp/b6 Start 27: enum 1/52 Test #27: enum ............................. Passed 0.01 sec Start 28: set_extent 2/52 Test #28: set_extent ....................... Passed 3.66 sec Start 29: getname 3/52 Test #29: getname .......................... Passed 0.06 sec Start 30: vfd 4/52 Test #30: vfd .............................. Passed 0.03 sec Start 31: ntypes 5/52 Test #31: ntypes ........................... Passed 0.17 sec Start 32: dangle 6/52 Test #32: dangle ........................... Passed 0.18 sec Start 33: dtransform 7/52 Test #33: dtransform ....................... Passed 0.02 sec Start 34: reserved 8/52 Test #34: reserved ......................... Passed 0.00 sec Start 35: cross_read 9/52 Test #35: cross_read ....................... Passed 0.02 sec Start 36: freespace 10/52 Test #36: freespace ........................ Passed 0.03 sec Start 37: mf 11/52 Test #37: mf ............................... Passed 0.09 sec ... On Thu, Feb 17, 2011 at 10:41 PM, Allen D Byrne <b...@hdfgroup.org> wrote: > David, > > I tried ctest -N ---- it lies! > > Here is the beginning of the test.log showing the problem (peviously the > tests were in test number order) : > > Start testing: Feb 17 21:13 CST > > ---------------------------------------------------------- > > 435/660 Testing: H5COPY-clear-ext-links > > 436/660 Testing: H5COPY-ext_link > > 437/660 Testing: H5COPY-ext_link_f > > 438/660 Testing: H5COPY-ext_dangle_noobj > > 439/660 Testing: H5COPY-DIFF_ext_dangle_noobj > > 440/660 Testing: H5COPY-ext_dangle_noobj_f > > 441/660 Testing: H5COPY-ext_dangle_nofile > > 442/660 Testing: H5COPY-DIFF_ext_dangle_nofile > > 443/660 Testing: H5COPY-ext_dangle_nofile_f > > 444/660 Testing: H5COPY-ext_link_group > > 445/660 Testing: H5COPY-ext_link_group_f > > 446/660 Testing: H5COPY-H5LS_h5copy_extlinks_src > > 447/660 Testing: H5STAT-h5stat_help1 > > 448/660 Testing: H5STAT-h5stat_help2 > > 449/660 Testing: H5STAT-h5stat_filters > > 450/660 Testing: H5STAT-h5stat_filters-file > > 451/660 Testing: H5STAT-h5stat_filters-F > > 452/660 Testing: H5STAT-h5stat_filters-d > > 453/660 Testing: H5STAT-h5stat_filters-g > > 454/660 Testing: H5STAT-h5stat_filters-dT > > 414/660 Testing: H5COPY-DIFF_grp_nested > > 415/660 Testing: H5COPY-simple_group > > 416/660 Testing: H5COPY-DIFF_simple_group > > 417/660 Testing: H5COPY-grp_rename > > 418/660 Testing: H5COPY-DIFF_grp_rename > > 419/660 Testing: H5COPY-grp_dsets_rename > > 420/660 Testing: H5COPY-DIFF_grp_dsets_rename > > 421/660 Testing: H5COPY-A_B1_simple > > 422/660 Testing: H5COPY-DIFF_A_B1_simple > > 423/660 Testing: H5COPY-A_B2_simple2 > > 424/660 Testing: H5COPY-DIFF_A_B2_simple2 > > 425/660 Testing: H5COPY-C_D_simple > > 426/660 Testing: H5COPY-DIFF_C_D_simple > > 427/660 Testing: H5COPY-E_F_grp_dsets > > 428/660 Testing: H5COPY-DIFF_E_F_grp_dsets > > 429/660 Testing: H5COPY-G_H_grp_nested > > 430/660 Testing: H5COPY-DIFF_G_H_grp_nested > > 431/660 Testing: H5COPY-H5LS_h5copytst > > 432/660 Testing: H5COPY-clear-refs > > 433/660 Testing: H5COPY-region_ref > > 434/660 Testing: H5COPY-H5LS_h5copy_ref > > 476/660 Testing: H5DUMP-thlink-4 > > 477/660 Testing: H5DUMP-thlink-5 > > 478/660 Testing: H5DUMP-tcomp-1 > > 479/660 Testing: H5DUMP-tcomp-2 > > 480/660 Testing: H5DUMP-tcomp-4 > > 481/660 Testing: H5DUMP-tnestcomp-1 > > 482/660 Testing: H5DUMP-tall-1 > > 483/660 Testing: H5DUMP-tall-2 > > 484/660 Testing: H5DUMP-tall-3 > > 485/660 Testing: H5DUMP-tloop-1 > > 486/660 Testing: H5DUMP-tstr-1 > > 487/660 Testing: H5DUMP-tstr-2 > > 488/660 Testing: H5DUMP-tsaf > > 489/660 Testing: H5DUMP-tvldtypes1 > > 490/660 Testing: H5DUMP-tvldtypes2 > > 491/660 Testing: H5DUMP-tvldtypes3 > > 492/660 Testing: H5DUMP-tvldtypes4 > > 493/660 Testing: H5DUMP-tvldtypes5 > > 494/660 Testing: H5DUMP-tvlstr > > 495/660 Testing: H5DUMP-tarray1 > > 455/660 Testing: H5STAT-h5stat_filters-UD > > 456/660 Testing: H5STAT-h5stat_filters-UT > > 457/660 Testing: H5STAT-h5stat_tsohm > > 458/660 Testing: H5STAT-h5stat_newgrat > > 459/660 Testing: H5STAT-h5stat_newgrat-UG > > 460/660 Testing: H5STAT-h5stat_newgrat-UA > > 461/660 Testing: H5DUMP-tgroup-1 > > 462/660 Testing: H5DUMP-tgroup-2 > > 463/660 Testing: H5DUMP-tdset-1 > > 464/660 Testing: H5DUMP-tdset-2 > > 465/660 Testing: H5DUMP-tattr-1 > > 466/660 Testing: H5DUMP-tattr-2 > > 467/660 Testing: H5DUMP-tattr-3 > > 468/660 Testing: H5DUMP-tnamed_dtype_attr > > 469/660 Testing: H5DUMP-tslink-1 > > 470/660 Testing: H5DUMP-tudlink-1 > > 471/660 Testing: H5DUMP-tslink-2 > > 472/660 Testing: H5DUMP-tudlink-2 > > 473/660 Testing: H5DUMP-thlink-1 > > 474/660 Testing: H5DUMP-thlink-2 > > 475/660 Testing: H5DUMP-thlink-3 > > 352/660 Testing: H5REPACK-DIFF_chunk_compa > > 353/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_compa > > 354/660 Testing: H5REPACK-chunk_conti > > 355/660 Testing: H5REPACK-DIFF_chunk_conti > > 356/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_conti > > 357/660 Testing: H5REPACK-chunk_18x13 > > 358/660 Testing: H5REPACK-DIFF_chunk_18x13 > > 359/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_18x13 > > 360/660 Testing: H5REPACK-contig_small_compa > > 361/660 Testing: H5REPACK-DIFF_contig_small_compa > > 362/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_compa > > 363/660 Testing: H5REPACK-contig_small_fixed_compa > > 364/660 Testing: H5REPACK-DIFF_contig_small_fixed_compa > > 365/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_fixed_compa > > 366/660 Testing: H5REPACK-native_fill > > 367/660 Testing: H5REPACK-DIFF_native_fill > > 368/660 Testing: H5REPACK-native_attr > > 369/660 Testing: H5REPACK-DIFF_native_attr > > 370/660 Testing: H5REPACK-layout_long_switches > > 371/660 Testing: H5REPACK-DIFF_layout_long_switches > > 331/660 Testing: H5REPACK-DIFF_compa > > 332/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-compa > > 333/660 Testing: H5REPACK-dset_compa_conti > > 334/660 Testing: H5REPACK-DIFF_dset_compa_conti > > 335/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_conti > > 336/660 Testing: H5REPACK-dset_compa_chunk > > 337/660 Testing: H5REPACK-DIFF_dset_compa_chunk > > 338/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_chunk > > 339/660 Testing: H5REPACK-dset_compa_compa > > 340/660 Testing: H5REPACK-DIFF_dset_compa_compa > > 341/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_compa > > 342/660 Testing: H5REPACK-dset_conti_compa > > 343/660 Testing: H5REPACK-DIFF_dset_conti_compa > > 344/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_compa > > 345/660 Testing: H5REPACK-dset_conti_chunk > > 346/660 Testing: H5REPACK-DIFF_dset_conti_chunk > > 347/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_chunk > > 348/660 Testing: H5REPACK-dset_conti_conti > > 349/660 Testing: H5REPACK-DIFF_dset_conti_conti > > 350/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_conti > > 351/660 Testing: H5REPACK-chunk_compa > > 393/660 Testing: H5COPY-chunk > > 394/660 Testing: H5COPY-DIFF_chunk > > 395/660 Testing: H5COPY-compact > > 396/660 Testing: H5COPY-DIFF_compact > > 397/660 Testing: H5COPY-compound > > 398/660 Testing: H5COPY-DIFF_compound > > 399/660 Testing: H5COPY-compressed > > 400/660 Testing: H5COPY-DIFF_compressed > > 401/660 Testing: H5COPY-named_vl > > 402/660 Testing: H5COPY-DIFF_named_vl > > 403/660 Testing: H5COPY-nested_vl > > 404/660 Testing: H5COPY-DIFF_nested_vl > > 405/660 Testing: H5COPY-simple_top > > 406/660 Testing: H5COPY-DIFF_simple_top > > 407/660 Testing: H5COPY-rename > > 408/660 Testing: H5COPY-DIFF_rename > > 409/660 Testing: H5COPY-grp_empty > > 410/660 Test: H5COPY-DIFF_grp_empty > > 411/660 Testing: H5COPY-grp_dsets > > 412/660 Testing: H5COPY-DIFF_grp_dsets > > 413/660 Testing: H5COPY-grp_nested > > 372/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_long_switches > > 373/660 Testing: H5REPACK-layout_short_switches > > 374/660 Test: H5REPACK-DIFF_layout_short_switches > > 375/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_short_switches > > 376/660 Testing: H5REPACK-global_filters > > 377/660 Testing: H5REPACK-DIFF_global_filters > > 378/660 Testing: H5REPACK-add_userblock > > 379/660 Testing: H5REPACK-DIFF_add_userblock > > 380/660 Testing: H5REPACK-add_alignment > > 381/660 Testing: H5REPACK-DIFF_add_alignment > > 382/660 Testing: H5REPACK-upgrade_layout > > 383/660 Testing: H5REPACK-DIFF_upgrade_layout > > 384/660 Testing: H5REPACK-gt_mallocsize > > 385/660 Testing: H5REPACK-DIFF_gt_mallocsize > > 386/660 Testing: H5REPACK-committed_dt > > 387/660 Testing: H5REPACK-DIFF_committed_dt > > 388/660 Testing: H5REPACK-bug1814 > > 389/660 Testing: H5REPACK-DIFF_bug1814 > > 390/660 Testing: H5COPY-clear-objects > > 391/660 Testing: H5COPY-simple > > 392/660 Testing: H5COPY-DIFF_simple > > 600/660 Testing: H5DUMP-XML-thlink.h5 > > 601/660 Testing: H5DUMP-XML-tloop.h5 > > 602/660 Testing: H5DUMP-XML-tloop2.h5 > > 603/660 Testing: H5DUMP-XML-tmany.h5 > > 604/660 Testing: H5DUMP-XML-tnestedcomp.h5 > > 605/660 Testing: H5DUMP-XML-tcompound_complex.h5 > > Allen > >> After building HDF5 v 1.8.6 on my Mac, I get this: > >> > >> $ ctest -N > >> Test project /Users/davidcole/tmp/b6 > >> Test #1: testhdf5 > >> Test #2: accum > >> Test #3: lheap > >> Test #4: ohdr > >> Test #5: stab > >> Test #6: gheap > >> Test #7: pool > >> Test #8: hyperslab > >> Test #9: istore > >> Test #10: bittests > >> Test #11: dt_arith > >> Test #12: dtypes > >> Test #13: dsets > >> Test #14: cmpd_dset > >> Test #15: extend > >> Test #16: external > >> Test #17: objcopy > >> Test #18: links > >> Test #19: unlink > >> Test #20: big > >> Test #21: mtime > >> Test #22: fillval > >> Test #23: mount > >> Test #24: flush1 > >> Test #25: flush2 > >> Test #26: app_ref > >> Test #27: enum > >> Test #28: set_extent > >> Test #29: getname > >> Test #30: vfd > >> Test #31: ntypes > >> Test #32: dangle > >> Test #33: dtransform > >> Test #34: reserved > >> Test #35: cross_read > >> Test #36: freespace > >> Test #37: mf > >> Test #38: btree2 > >> Test #39: fheap > >> Test #40: error_test > >> Test #41: err_compat > >> Test #42: tcheck_version > >> Test #43: testmeta > >> Test #44: cache > >> Test #45: cache_api > >> Test #46: ttsafe > >> Test #47: h5perf_serial > >> Test #48: chunk > >> Test #49: iopipe > >> Test #50: overhead > >> Test #51: perf_meta > >> Test #52: zip_perf > >> > >> Total Tests: 52 > >> > >> I'm not familiar enough with HDF5 to know if this is correct or not. > >> Is this the same order you're seeing? > >> > >> Or can you tell me your expected order? I'll try to analyze the source > >> tree and figure out what's going on, but what you expect will tell me > >> something, too. > >> > >> > >> Thanks, > >> David > >> > >> > >> > >> On Thu, Feb 17, 2011 at 5:10 PM, Allen D Byrne <b...@hdfgroup.org> wrote: > >> > Just using standard gcc/gfortran on the fedora distro. It is the 64bit > >> > version? > >> > > >> > You can grab one of our recent src tarballs from > >> > > >> > "http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.6/"; > >> > > >> > Allen > >> > > >> > PS. I will be out of touch for the next 36 hours > >> > > >> >> On Thu, Feb 17, 2011 at 4:20 PM, Allen D Byrne <b...@hdfgroup.org> >> >> wrote: > >> > > >> >> > This was from a clean build folder. And I just want the test to run >> >> > in > >> >> > the > >> > > >> >> > order defined in the CMakeLists.txt files. Also never had set any >> >> > COST > >> > > >> >> > properties. The surprise was that this never happened before. > >> > > >> >> > > >> > > >> >> > Allen > >> > > >> >> > > >> > > >> >> >> On Thu, Feb 17, 2011 at 4:08 PM, David Cole <david.c...@kitware.com> > >> > > >> >> >> wrote: > >> > > >> >> > > >> > > >> >> >> > On Thu, Feb 17, 2011 at 3:57 PM, Allen D Byrne <b...@hdfgroup.org> > >> > > >> >> >> > wrote: > >> > > >> >> > > >> > > >> >> >> >> I just installed 2.8.4 and on my fedora 14 linux box the ctest -D > >> > > >> >> > > >> > > >> >> >> >> Experimental stage decided to run the tests in some unknown to me > >> > > >> >> >> >> order. The > >> > > >> >> > > >> > > >> >> >> >> previous versions always ran them in order of definition. Any >> >> >> >> thing > >> >> >> >> I > >> > > >> >> >> >> can do > >> > > >> >> > > >> > > >> >> >> >> to get the ordering back? > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> >> Allen > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> >> _______________________________________________ > >> > > >> >> > > >> > > >> >> >> >> Powered by www.kitware.com > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> >> Visit other Kitware open-source projects at > >> > > >> >> > > >> > > >> >> >> >> http://www.kitware.com/opensource/opensource.html > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> >> Please keep messages on-topic and check the CMake FAQ at: > >> > > >> >> > > >> > > >> >> >> >> http://www.cmake.org/Wiki/CMake_FAQ > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> >> Follow this link to subscribe/unsubscribe: > >> > > >> >> > > >> > > >> >> >> >> http://www.cmake.org/mailman/listinfo/cmake > >> > > >> >> > > >> > > >> >> >> >> > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > Delete the file "Testing/Temporary/CTestCostData.txt" in your >> >> >> > build > >> > > >> >> > > >> > > >> >> >> > tree. It saves run times from run to run and orders them as >> >> >> > "slowest > >> > > >> >> > > >> > > >> >> >> > first" on subsequent runs. > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > A clean build (no CostData file) without any parallel testing, and > >> > > >> >> > > >> > > >> >> >> > without any test COST properties defined should give you the top >> >> >> > to > >> > > >> >> > > >> > > >> >> >> > bottom ordering that you're used to. > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > Saving this data from run to run helps us schedule parallel >> >> >> > testing > >> > > >> >> > > >> > > >> >> >> > jobs efficiently on subsequent runs. > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > HTH, > >> > > >> >> > > >> > > >> >> >> > David > >> > > >> >> > > >> > > >> >> >> > > >> > > >> >> > > >> > > >> >> >> > >> > > >> >> > > >> > > >> >> >> > >> > > >> >> > > >> > > >> >> >> Having said all that: if your goal is to order certain tests with > >> > > >> >> > > >> > > >> >> >> respect to each other, see the help for the test property "DEPENDS": > >> > > >> >> > > >> > > >> >> >> http://cmake.org/cmake/help/cmake-2-8-docs.html#prop_test:DEPENDS > >> > > >> >> > > >> > > >> >> >> > >> > > >> >> > >> > > >> >> So there is no "Testing/Temporary/CTestCostData.txt" file at the time > >> > > >> >> you call "ctest -D Experimental"? > >> > > >> >> > >> > > >> >> Can you point me to a source tree that I can build that demonstrates > >> > > >> >> this behavior? > >> > > >> >> > >> > > >> >> If you are not running tests in parallel and you have no "cost data" > >> > > >> >> (in the form of a cached file or properties set) then the tests should > >> > > >> >> run in the same top-to-bottom order that they always have. > >> > > >> >> > >> > > >> >> > >> > > >> >> Thx, > >> > > >> >> David > >> > > >> >> > >> _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
