Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

On 27.10.2013 15:48, David Cole wrote:
It's the tests in all the other projects in the world, that have not 
taken advantage of parallel testing yet that you will have to be 
concerned about. (For a notable example, the CDash project cannot use 
parallel testing. It's tests are highly dependent on order, and can 
only be done serially as the tests in its suite do not use the DEPENDS 
property to specify order, they rely on the default ordering. You can 
verify this by running a CDash dashboard with any proposed build of 
CMake/ctest...)


In serial testing, the default ordering of the tests should match the 
order of the add_test calls encountered during CMakeLists processing.



HTH,
David C.



Thanks David!

That makes a lot of sense.
I merged the topic to next.

Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[David Cole]

I just realized that CTest only prioritized failed tests in parallel
test runs since it only modified COSTs in that case. So Casey was 

quite

right noticing that there is a behavior change here.

I created a topic "restore-serial-failed-test-order" which limits 

that

behavior to parallel test runs.
Should anyone prefer that behavior I could merge it to next.


I think that would be wise for backwards compatibility. The recent 
email discussions about CMake's /STACK argument change without a policy 
highlight the importance of this.




Pro/con the current behavior in next:
+ Consistent sorting for serial and parallel test runs
- There may be users who expect a fixed order of tests in serial runs
- There may be tests in CMake that expect a fixed order of tests in
serial runs(?)
+ When bug fixing or implementing (in case of TDD) the previously 

failed
tests are those that a user is waiting for the most so it may make 

sense

to have them run first (I assume that is why it was implemented for
parallel testing)


It's not the tests in CMake that we have to worry about. We can always 
adapt the test suite of CMake to account for new behavior if necessary.


It's the tests in all the other projects in the world, that have not 
taken advantage of parallel testing yet that you will have to be 
concerned about. (For a notable example, the CDash project cannot use 
parallel testing. It's tests are highly dependent on order, and can 
only be done serially as the tests in its suite do not use the DEPENDS 
property to specify order, they rely on the default ordering. You can 
verify this by running a CDash dashboard with any proposed build of 
CMake/ctest...)


In serial testing, the default ordering of the tests should match the 
order of the add_test calls encountered during CMakeLists processing.



HTH,
David C.

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

I just realized that CTest only prioritized failed tests in parallel 
test runs since it only modified COSTs in that case. So Casey was quite 
right noticing that there is a behavior change here.


I created a topic "restore-serial-failed-test-order" which limits that 
behavior to parallel test runs.

Should anyone prefer that behavior I could merge it to next.

Pro/con the current behavior in next:
+ Consistent sorting for serial and parallel test runs
- There may be users who expect a fixed order of tests in serial runs
- There may be tests in CMake that expect a fixed order of tests in 
serial runs(?)
+ When bug fixing or implementing (in case of TDD) the previously failed 
tests are those that a user is waiting for the most so it may make sense 
to have them run first (I assume that is why it was implemented for 
parallel testing)


Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Brad King]

On 10/24/2013 10:06 AM, Nils Gladitz wrote:
> I think I can probably fix that by replacing the insert with a manual loop.
> SSH access is being blocked here so I probably wont get to fix it before 
> I get home but looks like the client only builds nightly anyway :)

Please also address the VS 6 build failure:

 http://open.cdash.org/viewBuildError.php?buildid=3073355

Thanks,
-Brad
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Thanks Bill,

I think I can probably fix that by replacing the insert with a manual loop.
SSH access is being blocked here so I probably wont get to fix it before 
I get home but looks like the client only builds nightly anyway :)


Nils


On 10/24/2013 03:40 PM, Bill Hoffman wrote:

On 10/24/2013 8:58 AM, Nils Gladitz wrote:

Had some trouble with my new test and multi configuration generators at
first but the incomming dashboard results look like I got that fixed 
now :)

Thanks again!

Looks like there was a build failure here:

http://open.cdash.org/viewBuildError.php?buildid=3071938

-Bill


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/24/2013 8:58 AM, Nils Gladitz wrote:

Had some trouble with my new test and multi configuration generators at
first but the incomming dashboard results look like I got that fixed now :)
Thanks again!

Looks like there was a build failure here:

http://open.cdash.org/viewBuildError.php?buildid=3071938

-Bill
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

On 10/23/2013 08:19 PM, Bill Hoffman wrote:

Looks good to me.  Feel free to merge into next.

-Bill

Had some trouble with my new test and multi configuration generators at 
first but the incomming dashboard results look like I got that fixed now :)

Thanks again!

Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/23/2013 2:01 PM, Casey Goodlett wrote:

Thanks for the explanation.  In that case, the change looks good to me.


Looks good to me.  Feel free to merge into next.

-Bill

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Casey Goodlett]

Thanks for the explanation.  In that case, the change looks good to me.


On Wed, Oct 23, 2013 at 1:43 PM, Nils Gladitz  wrote:

> On 23.10.2013 19:28, Casey Goodlett wrote:
>
>>
>> Bill and Nils
>>
>> Does this change the logic to run previously failed tests first or was
>> that already the case?  I am not sure whether or not this is a better
>> behavior.  Can running failed tests first be considered a separate
>> modification rather than part of the fix to running serial?
>>
>>  First of thanks for testing!
>
> Vanilla CTest 2.8.12 prioritizes failed test by changing their COST to
> some float maximum (only if a parallel test run is being performed) before
> generating the sorted test cost list.
>
> I tried to duplicate that behavior but did so without modifying the actual
> COST.
>
> Nils
>



-- 
Casey B. Goodlett, Ph.D.
R&D Engineer
Kitware, Inc. - North Carolina Office
http://www.kitware.com
(919) 969-6990 x310
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

On 23.10.2013 19:28, Casey Goodlett wrote:


Bill and Nils
Does this change the logic to run previously failed tests first or was 
that already the case?  I am not sure whether or not this is a better 
behavior.  Can running failed tests first be considered a separate 
modification rather than part of the fix to running serial?



First of thanks for testing!

Vanilla CTest 2.8.12 prioritizes failed test by changing their COST to 
some float maximum (only if a parallel test run is being performed) 
before generating the sorted test cost list.


I tried to duplicate that behavior but did so without modifying the 
actual COST.


Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Casey Goodlett]

Bill and Nils,

I tried this on my large test suite under linux, and it seems to work.  I
checked that the serial tests in my suite were still run alone (although in
my particular setup that happened to already occur).  I also did not notice
any violation of the RESOURCE_LOCK constraints which are also present in my
suite.

I get reduced overall runtime as this change seems to do a better job of
getting the dependent tests run early.  I did not evaluate this enough to
make a quantitative statement.

Does this change the logic to run previously failed tests first or was that
already the case?  I am not sure whether or not this is a better behavior.
 Can running failed tests first be considered a separate modification
rather than part of the fix to running serial?

Looks like a good change overall.

Thank you


On Wed, Oct 23, 2013 at 9:15 AM, Bill Hoffman wrote:

> On 10/23/2013 3:53 AM, Nils Gladitz wrote:
>
>> Hello Bill,
>>
>> Would you approve a merge of the topic into next?
>> Who else should I be asking for approval?
>> e.g. is there a CTest specific maintainer?
>>
>> Thanks in advance.
>>
> I am traveling right now at our NC Kitware office.  There is someone there
> that has a complicated system he wanted to try your changes on. He should
> be able to get to that today.  He also did some work on this issue in the
> past.  As soon as we try it with that code, we can merge it to next.
>  Thanks for the work, and sorry for the slow response.
>
> -Bill
>
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/**
> opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/**CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/cgi-**bin/mailman/listinfo/cmake-**developers
>



-- 
Casey B. Goodlett, Ph.D.
R&D Engineer
Kitware, Inc. - North Carolina Office
http://www.kitware.com
(919) 969-6990 x310
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

On 23.10.2013 15:15, Bill Hoffman wrote:
I am traveling right now at our NC Kitware office.  There is someone 
there that has a complicated system he wanted to try your changes on. 
He should be able to get to that today.  He also did some work on this 
issue in the past.  As soon as we try it with that code, we can merge 
it to next.  Thanks for the work, and sorry for the slow response.

Sounds great.

Thanks!

Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/23/2013 3:53 AM, Nils Gladitz wrote:

Hello Bill,

Would you approve a merge of the topic into next?
Who else should I be asking for approval?
e.g. is there a CTest specific maintainer?

Thanks in advance.
I am traveling right now at our NC Kitware office.  There is someone 
there that has a complicated system he wanted to try your changes on. 
He should be able to get to that today.  He also did some work on this 
issue in the past.  As soon as we try it with that code, we can merge it 
to next.  Thanks for the work, and sorry for the slow response.


-Bill
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello Bill,

Would you approve a merge of the topic into next?
Who else should I be asking for approval?
e.g. is there a CTest specific maintainer?

Thanks in advance.

Nils
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/21/2013 2:39 PM, Nils Gladitz wrote:

I haven't see that fail on my local test runs.
Is it sporadic or does it fail for you consistently?

Never mind on this one.  Looks like it fails for me without your changes.

-Bill

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

On 21.10.2013 19:04, Bill Hoffman wrote:


It would also be good to merge your test case 
http://pastebin.com/3zXsTrem into its own test or one of these:
I created a new test case " CTestTestSerialInDepends" which now detects 
the issue but because of the sporadic nature of the issue may falsly 
report success.
I think it is still usefull though since the probability of detection is 
high and increases with the number of test machines and test runs.


Nils

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello Bill,

thanks for checking up on the changes!

On 21.10.2013 19:04, Bill Hoffman wrote:


Can you add a few comments into the code that describe what you are 
doing so it will be easier to follow?   Maybe a block comment that 
describes the high level approach?



I've added comments to the code blocks in CreateTestCostList().


// This should work:
if(this->Tests[test].size() == 0)

Changed.



It would also be good to merge your test case 
http://pastebin.com/3zXsTrem into its own test or one of these:


CTestTestParallel.
I am not yet entirely sure how to do that since the test case (and the 
underlying behavior in vanilla cmake) is sporadic and still needs some 
machine detectable failure condition but I'll try looking into it.




I am also getting a test failure here:

190: Test project C:/Users/hoffman/Work/My 
Builds/cmake-ninja/Tests/CTestTestStopTime

190: Start 1: TestSleep
190: 1/2 Test #1: TestSleep    Passed 29.99 sec
190: Start 2: ShouldNotRun
190: 2/2 Test #2: ShouldNotRun .***Failed 30.00 sec
190:
190: 50% tests passed, 1 tests failed out of 2
190:
190: Total Test time (real) =  60.00 sec
190:
190: The following tests FAILED:
190:  2 - ShouldNotRun (Failed)
1/1 Test #190: CTestTestStopTime ***Failed Required 
regular expression not found.Regex=[The stop time has been passed

] 60.67 sec


I haven't see that fail on my local test runs.
Is it sporadic or does it fail for you consistently?

Nils

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/18/2013 6:50 PM, Nils Gladitz wrote:

Hello Bill,

I've updated the topic with (I think) improved sorting.
Tests are now prioritized by depth within the dependency graph and
within each depth level they are further ordered by cost.
This gives better total test time for my test case and preserves
correctness in relation to RUN_SERIAL
Can you add a few comments into the code that describe what you are 
doing so it will be easier to follow?   Maybe a block comment that 
describes the high level approach?



Also, small change, I don't think we need this copy anymore:
 // copy the depend tests locally because when
  // a test is finished it will be removed from the depend list
  // and we don't want to be iterating a list while removing from it
  TestSet depends = this->Tests[test];
  size_t totalDepends = depends.size();

// This should work:
if(this->Tests[test].size() == 0)


It would also be good to merge your test case 
http://pastebin.com/3zXsTrem into its own test or one of these:


CTestTestParallel.

I am also getting a test failure here:

190: Test project C:/Users/hoffman/Work/My 
Builds/cmake-ninja/Tests/CTestTestStopTime

190: Start 1: TestSleep
190: 1/2 Test #1: TestSleep    Passed   29.99 sec
190: Start 2: ShouldNotRun
190: 2/2 Test #2: ShouldNotRun .***Failed   30.00 sec
190:
190: 50% tests passed, 1 tests failed out of 2
190:
190: Total Test time (real) =  60.00 sec
190:
190: The following tests FAILED:
190:  2 - ShouldNotRun (Failed)
1/1 Test #190: CTestTestStopTime ***Failed  Required 
regular expression not found.Regex=[The stop time has been passed

] 60.67 sec


Thanks.

-Bill

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello Bill,

I've updated the topic with (I think) improved sorting.
Tests are now prioritized by depth within the dependency graph and 
within each depth level they are further ordered by cost.
This gives better total test time for my test case and preserves 
correctness in relation to RUN_SERIAL:


   Test project /home/ngladitz/src/test/build
Start 4: i_want_to_be_alone
   1/4 Test #4: i_want_to_be_alone ...   Passed3.03 sec
Start 1: i_like_company
Start 3: i_have_dependencies
Start 2: i_like_company_too
   2/4 Test #1: i_like_company ...   Passed3.02 sec
   3/4 Test #2: i_like_company_too ...   Passed3.02 sec
   4/4 Test #3: i_have_dependencies ..   Passed3.02 sec

   100% tests passed, 0 tests failed out of 4

   Total Test time (real) =   6.06 sec


Nils

On 18.10.2013 00:03, Nils Gladitz wrote:
StartTest() returns false without running the test if there are any 
dependencies remaining.


Nils

On 17.10.2013 23:57, Bill Hoffman wrote:
On Thu, Oct 17, 2013 at 5:31 PM, Nils Gladitz > wrote:


Hello Bill, thanks for looking into this!

All four tests in my example are still run (as you can see in the
test output I provided).
This includes the dependent test.

What I did change is that all tests (from an dependency or
otherwise) are run from StartNextTests() which ensures that
serial tests run alone rather than from StartTest() which did not
seem to account for this.

I am not saying it won't run all of the tests.  I am saying there is 
a danger it will ignore the depends with your change and run a depend 
before the test that depends on it.  I don't see it using the depends 
in the sort.  Maybe a better fix would be to make sure the sort takes 
the depends into consideration.


Or, just change the depend starting in StartTest to make sure it does 
not start a serial test with another test running.


-Bill




--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

StartTest() returns false without running the test if there are any 
dependencies remaining.


Nils

On 17.10.2013 23:57, Bill Hoffman wrote:
On Thu, Oct 17, 2013 at 5:31 PM, Nils Gladitz > wrote:


Hello Bill, thanks for looking into this!

All four tests in my example are still run (as you can see in the
test output I provided).
This includes the dependent test.

What I did change is that all tests (from an dependency or
otherwise) are run from StartNextTests() which ensures that serial
tests run alone rather than from StartTest() which did not seem to
account for this.

I am not saying it won't run all of the tests.  I am saying there is a 
danger it will ignore the depends with your change and run a depend 
before the test that depends on it.  I don't see it using the depends 
in the sort.  Maybe a better fix would be to make sure the sort takes 
the depends into consideration.


Or, just change the depend starting in StartTest to make sure it does 
not start a serial test with another test running.


-Bill


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On Thu, Oct 17, 2013 at 5:31 PM, Nils Gladitz  wrote:

> Hello Bill, thanks for looking into this!
>
> All four tests in my example are still run (as you can see in the test
> output I provided).
> This includes the dependent test.
>
> What I did change is that all tests (from an dependency or otherwise) are
> run from StartNextTests() which ensures that serial tests run alone rather
> than from StartTest() which did not seem to account for this.
>
I am not saying it won't run all of the tests.  I am saying there is a
danger it will ignore the depends with your change and run a depend before
the test that depends on it.  I don't see it using the depends in the sort.
 Maybe a better fix would be to make sure the sort takes the depends into
consideration.

Or, just change the depend starting in StartTest to make sure it does not
start a serial test with another test running.

-Bill
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello Bill, thanks for looking into this!

All four tests in my example are still run (as you can see in the test 
output I provided).

This includes the dependent test.

What I did change is that all tests (from an dependency or otherwise) 
are run from StartNextTests() which ensures that serial tests run alone 
rather than from StartTest() which did not seem to account for this.


Nils


On 17.10.2013 23:19, Bill Hoffman wrote:

On 10/17/2013 4:41 PM, Nils Gladitz wrote:


I pushed my topic "ctest-fix-run-serial" to stage.
I am taking a look.   It has been a long time since I looked at this 
code.  However, it seems like your patch removes the running of 
dependent tests.


old code did this:
StartTest(int test)
 ...
 foreach d in depend tests of test
startTests


New code removes that.   I guess it depends on SortedTests having 
taken depends into consideration, which I am not seeing.  However, I 
might be missing something


What was your thinking on this?

-Bill

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html


Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ


Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Bill Hoffman]

On 10/17/2013 4:41 PM, Nils Gladitz wrote:


I pushed my topic "ctest-fix-run-serial" to stage.
I am taking a look.   It has been a long time since I looked at this 
code.  However, it seems like your patch removes the running of 
dependent tests.


old code did this:
StartTest(int test)
 ...
 foreach d in depend tests of test
startTests


New code removes that.   I guess it depends on SortedTests having taken 
depends into consideration, which I am not seeing.  However, I might be 
missing something


What was your thinking on this?

-Bill

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers

[cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello,

I'm trying to fix: http://public.kitware.com/Bug/view.php?id=14484
which I reported earlier.

I pushed my topic "ctest-fix-run-serial" to stage.

Could some of you take a look at my changes?
They seem to fix the issue for me and don't seem to break any of the 
existing tests but
the code is foreign to me and I am not aware of the history and 
potential pitfalls that go with it.


I use an updated test case that I attached to this mail (I hope it gets 
preserved in the mailing list) otherwise I hope at least this paste link 
survives: http://pastebin.com/3zXsTrem


It sets up 4 Tests:
i_like_company
i_like_company_too
i_have_dependencies (depends on "i_want_to_be_alone)
i_want_to_be_alone (required to run serial)

Each test consumes 3 seconds.

With vanilla 2.8.12 a parallel test run (e.g. ctest -j16) gives me:

   Test project /home/ngladitz/src/test/build
Start 1: i_like_company
Start 4: i_want_to_be_alone
Start 2: i_like_company_too
   1/4 Test #1: i_like_company ...   Passed3.01 sec
   2/4 Test #4: i_want_to_be_alone ...   Passed3.01 sec
   3/4 Test #2: i_like_company_too ...   Passed3.01 sec
Start 3: i_have_dependencies
   4/4 Test #3: i_have_dependencies ..   Passed3.02 sec

   100% tests passed, 0 tests failed out of 4

   Total Test time (real) =   6.05 sec

Note how the serial test "i_want_to_be_alone" starts and ends roughly at 
the same time as "i_like_company" and "i_like_comany_too".


With my branch I get:

   Test project /home/ngladitz/src/test/build
Start 1: i_like_company
Start 2: i_like_company_too
   1/4 Test #1: i_like_company ...   Passed3.02 sec
   2/4 Test #2: i_like_company_too ...   Passed3.02 sec
Start 4: i_want_to_be_alone
   3/4 Test #4: i_want_to_be_alone ...   Passed3.03 sec
Start 3: i_have_dependencies
   4/4 Test #3: i_have_dependencies ..   Passed3.02 sec

   100% tests passed, 0 tests failed out of 4

   Total Test time (real) =   9.08 sec

Note how the whole test run takes 9 rather than 6 seconds.

The test runs with vanilla are sporadic and sometimes give the correct 
results consistently for a while which makes this difficult to test 
consistently.


I am grateful for any guidance and insight you may provide.

Thanks in advance.

Nils
if(TEST_NAME)
set(CTEST_RUN_CURRENT_SCRIPT 0)
ctest_sleep(3)
return()
endif()

cmake_minimum_required(VERSION 2.8.12)

enable_testing()

function(my_add_test NAME COST)
add_test(${NAME} 
${CMAKE_CTEST_COMMAND} -DTEST_NAME=${NAME} 
-S ${CMAKE_CURRENT_LIST_FILE})
set_tests_properties(${NAME} PROPERTIES COST ${COST})
endfunction()

my_add_test(i_like_company 1000)
my_add_test(i_like_company_too 0)

my_add_test(i_have_dependencies 1000)
set_tests_properties(i_have_dependencies PROPERTIES 
DEPENDS "i_want_to_be_alone")

my_add_test(i_want_to_be_alone 100)
set_tests_properties(i_want_to_be_alone PROPERTIES RUN_SERIAL 1)
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers