Re: [CMake] Parallel builds and auto generated header files

[Isaiah Norton]

I use essentially the technique in the answer below, except with
`add_custom_target` instead of `add_custom_command`, so that the
hash-generation is always run regardless of whether the temp file is
successfully deleted:

https://stackoverflow.com/a/41750028/508431

I also use a dummy `add_custom_command` which depends on GITVER_FILE as in
the above. IIRC the reason was that in my use case GITVER_FILE is passed as
part of a sources list to a macro, and defining the custom command was the
only way I could find to properly enforce an ordering dependency in
parallel builds:

# dummy output so we can depend on it in the external macro, because CMake.
add_custom_command(OUTPUT "${GITVER_FILE}"
  DEPENDS _GEN_GITVER)

(I think it was fine running `add_executable`/`add_library` directly in the
current CMakeList, but ordering didn't work when passing to a macro -- at
least in the version of cmake where I developed this)

On Mon, Jul 23, 2018 at 2:50 PM Andrew Fuller  wrote:

> Using configure_file() at CMake time will appear to work at first glance,
> but you'll wind up with stale information when you change revisions without
> any CMake change.  CMake won't re-run automatically and thus your header
> file won't get updated.  You'll need to do it at build time to ensure it's
> always up-to-date.
>
> How do the translation units obtain the generated header?  There needs to
> be some form of dependency between the generated header and the
> consumer(s).  If you add the generated header as an input to another
> target, then CMake should ensure the file is generated before processing
> the dependant target.
>
> Also note that AFAICT add_custom_command will not re-run if the output
> file already exists (unless invalidated by a dependency).  Since you're
> grabbing dynamic information like the commit hash, then you likely want it
> to run _every_ time.  add_custom_target might be better suited for your
> needs.
> --
> *From:* CMake  on behalf of Robert Dailey <
> rcdailey.li...@gmail.com>
> *Sent:* July 19, 2018 8:05:06 AM
> *To:* CMake
> *Subject:* [CMake] Parallel builds and auto generated header files
>
> I have a Version.hpp file that I have a custom command tied to which
> basically runs CMake in script mode to perform configure_file() on it
> during build time. The reason it does this is because it builds
> Version.hpp using dynamic information, such as defining a macro with
> the current SHA1 being built, etc.
>
> I notice in parallel builds, this header file gets built too late,
> because of parallel builds. What if a file includes the header and its
> translation unit is built before Version.hpp is generated by the
> custom command? Would it be better/simpler to do configure_file() at
> configuration time instead of at build time as a custom command?
> Here's the logic (some variables are defined elsewhere, but should
> give you the gist):
>
>
> set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
> set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
> set( version_file_out ${generated_source_dir}/Main/Version.cpp )
>
> add_custom_command(
> OUTPUT ${version_file_out}
> COMMAND ${CMAKE_COMMAND}
> -D IN_FILE=${version_file_in}
> -D OUT_FILE=${version_file_out}
> -D GIT_EXECUTABLE=${GIT_EXECUTABLE}
> -D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
> -P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
> WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
> )
>
> source_group( Generated FILES ${version_file_out} )
> list( APPEND source_files ${version_file_out} )
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMak

Re: [CMake] Parallel builds and auto generated header files

[Andrew Fuller]

Using configure_file() at CMake time will appear to work at first glance, but 
you'll wind up with stale information when you change revisions without any 
CMake change.  CMake won't re-run automatically and thus your header file won't 
get updated.  You'll need to do it at build time to ensure it's always 
up-to-date.

How do the translation units obtain the generated header?  There needs to be 
some form of dependency between the generated header and the consumer(s).  If 
you add the generated header as an input to another target, then CMake should 
ensure the file is generated before processing the dependant target.

Also note that AFAICT add_custom_command will not re-run if the output file 
already exists (unless invalidated by a dependency).  Since you're grabbing 
dynamic information like the commit hash, then you likely want it to run 
_every_ time.  add_custom_target might be better suited for your needs.

From: CMake  on behalf of Robert Dailey 

Sent: July 19, 2018 8:05:06 AM
To: CMake
Subject: [CMake] Parallel builds and auto generated header files

I have a Version.hpp file that I have a custom command tied to which
basically runs CMake in script mode to perform configure_file() on it
during build time. The reason it does this is because it builds
Version.hpp using dynamic information, such as defining a macro with
the current SHA1 being built, etc.

I notice in parallel builds, this header file gets built too late,
because of parallel builds. What if a file includes the header and its
translation unit is built before Version.hpp is generated by the
custom command? Would it be better/simpler to do configure_file() at
configuration time instead of at build time as a custom command?
Here's the logic (some variables are defined elsewhere, but should
give you the gist):


set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
set( version_file_out ${generated_source_dir}/Main/Version.cpp )

add_custom_command(
OUTPUT ${version_file_out}
COMMAND ${CMAKE_COMMAND}
-D IN_FILE=${version_file_in}
-D OUT_FILE=${version_file_out}
-D GIT_EXECUTABLE=${GIT_EXECUTABLE}
-D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
-P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)

source_group( Generated FILES ${version_file_out} )
list( APPEND source_files ${version_file_out} )
--

Powered by www.kitware.com<http://www.kitware.com>

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[CMake] Parallel builds and auto generated header files

[Robert Dailey]

I have a Version.hpp file that I have a custom command tied to which
basically runs CMake in script mode to perform configure_file() on it
during build time. The reason it does this is because it builds
Version.hpp using dynamic information, such as defining a macro with
the current SHA1 being built, etc.

I notice in parallel builds, this header file gets built too late,
because of parallel builds. What if a file includes the header and its
translation unit is built before Version.hpp is generated by the
custom command? Would it be better/simpler to do configure_file() at
configuration time instead of at build time as a custom command?
Here's the logic (some variables are defined elsewhere, but should
give you the gist):


set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
set( version_file_out ${generated_source_dir}/Main/Version.cpp )

add_custom_command(
OUTPUT ${version_file_out}
COMMAND ${CMAKE_COMMAND}
-D IN_FILE=${version_file_in}
-D OUT_FILE=${version_file_out}
-D GIT_EXECUTABLE=${GIT_EXECUTABLE}
-D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
-P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)

source_group( Generated FILES ${version_file_out} )
list( APPEND source_files ${version_file_out} )
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Bill Hoffman]

On 9/16/2015 2:06 PM, Greg Marr wrote:

This part here is inaccurate, at least with the current CMake and
Visual Studio: "To set the flag you will have to edit the CMake cache
with the cmake-gui and add it to the CMAKE_CXX_FLAGS and the
CMAKE_C_FLAGS."

This is all we do in the CMake file that we include in all our
projects: add_compile_options(/MP$ENV{NUMBER_OF_PROCESSORS})

If you don't have a NUMBER_OF_PROCESSORS environment variable, you
just get /MP.

The oversubscription part is still true, but I've never seen randomly
bad object files.


I was writing it from the point of view of NOT having to edit the 
project.  Seems like more of a per build decision to me.  As there could 
be issues, the most reliable way to build is to not use /MP which is why 
CMake does not just do this by default.  So, I would not want to 
recommend that projects make it a default.  But, you can change the 
flags either by editing the CMakeLists.txt files or by editing the 
CMakeCache.txt file to change the flags in the build.


-Bill

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Greg Marr]

-Original Message-
From: CMake [mailto:cmake-boun...@cmake.org] On Behalf Of Bill Hoffman
Sent: Wednesday, September 16, 2015 2:00 PM
To: cmake@cmake.org
Subject: Re: [CMake] Parallel builds

On 9/16/2015 1:47 PM, J Decker wrote:
> I see yer right.
> Details get foggy after years.
> well that's something to add:)
>
All covered my blog.  :)

This part here is inaccurate, at least with the current CMake and Visual Studio:
"To set the flag you will have to edit the CMake cache with the cmake-gui 
and add it to the CMAKE_CXX_FLAGS and the CMAKE_C_FLAGS."

This is all we do in the CMake file that we include in all our projects:
add_compile_options(/MP$ENV{NUMBER_OF_PROCESSORS})

If you don't have a NUMBER_OF_PROCESSORS environment variable, you just get /MP.

The oversubscription part is still true, but I've never seen randomly bad 
object files.

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Bill Hoffman]

On 9/16/2015 1:47 PM, J Decker wrote:

I see yer right.
Details get foggy after years.
well that's something to add:)


All covered my blog.  :)

-Bill
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[J Decker]

On Wed, Sep 16, 2015 at 10:39 AM, Greg Marr  wrote:
> From: J Decker [mailto:d3c...@gmail.com]
> Sent: Wednesday, September 16, 2015 1:30 PM
> To: Greg Marr
> Cc: Petr Bena; Cory Quammen; cmake@cmake.org; Bill Hoffman
> Subject: Re: [CMake] Parallel builds
>
> /mp has no effect on cmake projects since each file is compiled with a
> separate command... it relies more on msbuild /m: option...
>
> from 
> https://msdn.microsoft.com/en-us/library/bb385193.aspx?f=255&MSPPError=-2147217396
>
> cl /MP7 a.cpp b.cpp c.cpp d.cpp e.cpp
>
> but cl is invoked for each source already; multiple sources aren't 
> passed to cl.
>
> That's not true for the Visual Studio generator.  CMake generates project 
> files, and Visual Studio builds them just like any other project file, 
> including using the /MP flag.
>
I see yer right.
Details get foggy after years.
well that's something to add :)
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Greg Marr]

From: J Decker [mailto:d3c...@gmail.com] 
Sent: Wednesday, September 16, 2015 1:30 PM
To: Greg Marr
Cc: Petr Bena; Cory Quammen; cmake@cmake.org; Bill Hoffman
Subject: Re: [CMake] Parallel builds

/mp has no effect on cmake projects since each file is compiled with a
separate command... it relies more on msbuild /m: option...

from 
https://msdn.microsoft.com/en-us/library/bb385193.aspx?f=255&MSPPError=-2147217396

cl /MP7 a.cpp b.cpp c.cpp d.cpp e.cpp

but cl is invoked for each source already; multiple sources aren't 
passed to cl.

That's not true for the Visual Studio generator.  CMake generates project 
files, and Visual Studio builds them just like any other project file, 
including using the /MP flag.

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[J Decker]

On Wed, Sep 16, 2015 at 9:17 AM, Greg Marr  wrote:
> On Wed, Sep 16, 2015 at 5:55 PM, Cory Quammen  
> wrote:
>> VTK exposes the /MP compiler flag in its CMake configuration.
>>
>> Here is the relevant code from
>> http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkDetermineCompilerFlags.cmake;h=1398050afb34ff8c0a74137d847c19a6f63b12e9;hb=HEAD
>>
>> 126 # Enable /MP flag for Visual Studio 2008 and greator
>> 132 SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} 
>> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
>> 133 SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} 
>> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
>
> If you're just going to set it to the number of processors on the machine, 
> you might as well just use /MP, which means use the number of processors on 
> the machine.  The only time you need a number is when you want to use a 
> different number of parallel processes, such as only using half your capacity 
> because something else is using the other half, or leaving one or two cores 
> open for GUI work.
>
/mp has no effect on cmake projects since each file is compiled with a
separate command... it relies more on msbuild /m: option...

from 
https://msdn.microsoft.com/en-us/library/bb385193.aspx?f=255&MSPPError=-2147217396

cl /MP7 a.cpp b.cpp c.cpp d.cpp e.cpp

but cl is invoked for each source already; multiple sources aren't passed to cl.



> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Greg Marr]

On Wed, Sep 16, 2015 at 5:55 PM, Cory Quammen  wrote:
> VTK exposes the /MP compiler flag in its CMake configuration.
>
> Here is the relevant code from
> http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkDetermineCompilerFlags.cmake;h=1398050afb34ff8c0a74137d847c19a6f63b12e9;hb=HEAD
>
> 126 # Enable /MP flag for Visual Studio 2008 and greator
> 132 SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} 
> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
> 133 SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} 
> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")

If you're just going to set it to the number of processors on the machine, you 
might as well just use /MP, which means use the number of processors on the 
machine.  The only time you need a number is when you want to use a different 
number of parallel processes, such as only using half your capacity because 
something else is using the other half, or leaving one or two cores open for 
GUI work.

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Raymond Wan]

On Wed, Sep 16, 2015 at 11:44 PM, Petr Bena  wrote:
> I would prefer a way to get cmake to generate solution files for MSVC
> in a way that these options are enabled by default. Right now I need
> to do this by hand and when I run cmake again, it overwrites the
> solution files I modified and restore back the settings that disallow
> the multithreaded builds. Basically I would like to know if there is
> some option I would add to CMakeLists.txt that would make it do so. I
> understand that this may not possible with current version of CMake
> though.


I sort of don't see this as being CMake's problem.  Suppose your
computer has 4 cores.  Do you mean you would like CMake to detect that
there are 4 cores and to change the Makefile automatically?  Someone
showed this with two Makefiles [1] -- it's an interesting solution if
you want to try to make the same pair of Makefiles with CMake.  (I
have no idea how, but I guess it's possible?)

I don't know what is the Windows equivalent, but on Linux, I've
aliased "make" to "make -j 4".  So, what you describe about "solution
files" generated by CMake that get overwritten isn't really an issue.

Or maybe I've completely misunderstood you?  Do you want CMake to
detect the number of threads and then to pass that to (a) the program
that would be generated [i.e., for its execution] or (b) the
compilation of such a program?  I've assumed you meant (b)...

Ray

[1]  
http://stackoverflow.com/questions/2527496/how-can-i-write-a-makefile-to-auto-detect-and-parallelize-the-build-with-gnu-mak
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[J Decker]

On Wed, Sep 16, 2015 at 8:44 AM, Petr Bena  wrote:
> I know that I can pass -j parameter to GNU make as well as there is
> some sort of option to run MSVC compiler by hand and give that similar
> parameter, but that's not exactly what I want.
>
> I would prefer a way to get cmake to generate solution files for MSVC
> in a way that these options are enabled by default. Right now I need
> to do this by hand and when I run cmake again, it overwrites the
> solution files I modified and restore back the settings that disallow
> the multithreaded builds. Basically I would like to know if there is
> some option I would add to CMakeLists.txt that would make it do so. I
> understand that this may not possible with current version of CMake
> though.
>
What modification do you make?  I have to do none myself to get it;
because again that's a setting that's outside of the project/solution
files.  It's something the IDE has internally to pass to MSBUILD as a
parameter (/m:)

> Thanks
>
> On Wed, Sep 16, 2015 at 5:28 PM, Bill Hoffman  
> wrote:
>> http://www.kitware.com/blog/home/post/434
>>
>> -Bill
>>
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Petr Bena]

This seems to be the precise answer to my question! Thank you

On Wed, Sep 16, 2015 at 5:55 PM, Cory Quammen  wrote:
> Hi Petr,
>
> VTK exposes the /MP compiler flag in its CMake configuration.
>
> Here is the relevant code from
> http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkDetermineCompilerFlags.cmake;h=1398050afb34ff8c0a74137d847c19a6f63b12e9;hb=HEAD
>
> 126 # Enable /MP flag for Visual Studio 2008 and greator
> 127 IF(MSVC_VERSION GREATER 1400)
> 128   SET(CMAKE_CXX_MP_FLAG OFF CACHE BOOL "Build with /MP flag enabled")
> 129   SET(PROCESSOR_COUNT "$ENV{NUMBER_OF_PROCESSORS}")
> 130   SET(CMAKE_CXX_MP_NUM_PROCESSORS CACHE ${PROCESSOR_COUNT} "The maximum
> number of processes for the /MP flag")
> 131   IF (CMAKE_CXX_MP_FLAG)
> 132 SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}
> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
> 133 SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}
> /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
> 134   ENDIF ()
> 135 ENDIF()
>
> Hope that helps,
> Cory
>
> On Wed, Sep 16, 2015 at 11:44 AM, Petr Bena  wrote:
>>
>> I know that I can pass -j parameter to GNU make as well as there is
>> some sort of option to run MSVC compiler by hand and give that similar
>> parameter, but that's not exactly what I want.
>>
>> I would prefer a way to get cmake to generate solution files for MSVC
>> in a way that these options are enabled by default. Right now I need
>> to do this by hand and when I run cmake again, it overwrites the
>> solution files I modified and restore back the settings that disallow
>> the multithreaded builds. Basically I would like to know if there is
>> some option I would add to CMakeLists.txt that would make it do so. I
>> understand that this may not possible with current version of CMake
>> though.
>>
>> Thanks
>>
>> On Wed, Sep 16, 2015 at 5:28 PM, Bill Hoffman 
>> wrote:
>> > http://www.kitware.com/blog/home/post/434
>> >
>> > -Bill
>> >
>> >
>> > --
>> >
>> > Powered by www.kitware.com
>> >
>> > Please keep messages on-topic and check the CMake FAQ at:
>> > http://www.cmake.org/Wiki/CMake_FAQ
>> >
>> > Kitware offers various services to support the CMake community. For more
>> > information on each offering, please visit:
>> >
>> > CMake Support: http://cmake.org/cmake/help/support.html
>> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> > CMake Training Courses: http://cmake.org/cmake/help/training.html
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://public.kitware.com/mailman/listinfo/cmake
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>
>
>
>
> --
> Cory Quammen
> R&D Engineer
> Kitware, Inc.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Cory Quammen]

Hi Petr,

VTK exposes the /MP compiler flag in its CMake configuration.

Here is the relevant code from
http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=CMake/vtkDetermineCompilerFlags.cmake;h=1398050afb34ff8c0a74137d847c19a6f63b12e9;hb=HEAD

126

# Enable /MP flag for Visual Studio 2008 and greator
127

IF(MSVC_VERSION GREATER 1400)
128

  SET(CMAKE_CXX_MP_FLAG OFF CACHE BOOL "Build with /MP flag enabled")
129

  SET(PROCESSOR_COUNT "$ENV{NUMBER_OF_PROCESSORS}")
130

  SET(CMAKE_CXX_MP_NUM_PROCESSORS CACHE ${PROCESSOR_COUNT} "The
maximum number of processes for the /MP flag")
131

  IF (CMAKE_CXX_MP_FLAG)
132

SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
133

SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} /MP${CMAKE_CXX_MP_NUM_PROCESSORS}")
134

  ENDIF ()
135

ENDIF()

Hope that helps,
Cory

On Wed, Sep 16, 2015 at 11:44 AM, Petr Bena  wrote:

> I know that I can pass -j parameter to GNU make as well as there is
> some sort of option to run MSVC compiler by hand and give that similar
> parameter, but that's not exactly what I want.
>
> I would prefer a way to get cmake to generate solution files for MSVC
> in a way that these options are enabled by default. Right now I need
> to do this by hand and when I run cmake again, it overwrites the
> solution files I modified and restore back the settings that disallow
> the multithreaded builds. Basically I would like to know if there is
> some option I would add to CMakeLists.txt that would make it do so. I
> understand that this may not possible with current version of CMake
> though.
>
> Thanks
>
> On Wed, Sep 16, 2015 at 5:28 PM, Bill Hoffman 
> wrote:
> > http://www.kitware.com/blog/home/post/434
> >
> > -Bill
> >
> >
> > --
> >
> > Powered by www.kitware.com
> >
> > Please keep messages on-topic and check the CMake FAQ at:
> > http://www.cmake.org/Wiki/CMake_FAQ
> >
> > Kitware offers various services to support the CMake community. For more
> > information on each offering, please visit:
> >
> > CMake Support: http://cmake.org/cmake/help/support.html
> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
> > CMake Training Courses: http://cmake.org/cmake/help/training.html
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Follow this link to subscribe/unsubscribe:
> > http://public.kitware.com/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>



-- 
Cory Quammen
R&D Engineer
Kitware, Inc.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitwa

Re: [CMake] Parallel builds

[Petr Bena]

I know that I can pass -j parameter to GNU make as well as there is
some sort of option to run MSVC compiler by hand and give that similar
parameter, but that's not exactly what I want.

I would prefer a way to get cmake to generate solution files for MSVC
in a way that these options are enabled by default. Right now I need
to do this by hand and when I run cmake again, it overwrites the
solution files I modified and restore back the settings that disallow
the multithreaded builds. Basically I would like to know if there is
some option I would add to CMakeLists.txt that would make it do so. I
understand that this may not possible with current version of CMake
though.

Thanks

On Wed, Sep 16, 2015 at 5:28 PM, Bill Hoffman  wrote:
> http://www.kitware.com/blog/home/post/434
>
> -Bill
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Bill Hoffman]

http://www.kitware.com/blog/home/post/434

-Bill

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[J Decker]

Tools->Option->Projects and soltuions->Build and Run at the top is
'maximum number of parallel project builds.  This is external setting
to what CMake generates so in the IDE that's what it uses.

>From the command line...

C:\tools\unix\cmake\bin\cmake.exe --build . --config Debug --target
INSTALL -- /m:4 /v:m

the /m:4 flag is number of processes to parallel build. and that works.

---
now if that's not what you mean and you mean 'Multi-threaded DLL'(/MD)
vs 'multi-threaded'(/MT) which selects the runtime to use cmake by
default generates /MD.

This is set in

CMAKE_${lang}_FLAGS_DEBUG_INIT

which you can do

if( ${CMAKE_BUILD_TYPE} MATCHES "[dD][eE][bB][uU][gG]" )
   string( REPLACE "/MDd" "/MTd" CMAKE_C_FLAGS_INIT ${CMAKE_C_FLAGS_INIT} )
   string( REPLACE "/MDd" "/MTd" CMAKE_CXX_FLAGS_INIT ${CMAKE_CXX_FLAGS_INIT} )
else()
string( REPLACE "/MD" "/MT" CMAKE_C_FLAGS_INIT ${CMAKE_C_FLAGS_INIT} )
string( REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_INIT ${CMAKE_CXX_FLAGS_INIT} )
endif()

something like that


On Wed, Sep 16, 2015 at 4:30 AM, Petr Bena  wrote:
> There is /MP compiler flag that I need to enable for every single
> project to be precise.
>
> On Wed, Sep 16, 2015 at 1:26 PM, Petr Bena  wrote:
>> Hi,
>>
>> Is it possible to enable multithreaded builds by default in Visual
>> Studio projects? It seems that cmake generates the solution files in a
>> way that they are forbidden, so even if global preference is to use
>> multithreaded builds, the compiler never does that for sln files
>> create by cmake, because multithreaded builds are not allowed on
>> project level.
>>
>> Is there any way to enable them so that I don't have to do that by
>> hand all time?
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds

[Petr Bena]

There is /MP compiler flag that I need to enable for every single
project to be precise.

On Wed, Sep 16, 2015 at 1:26 PM, Petr Bena  wrote:
> Hi,
>
> Is it possible to enable multithreaded builds by default in Visual
> Studio projects? It seems that cmake generates the solution files in a
> way that they are forbidden, so even if global preference is to use
> multithreaded builds, the compiler never does that for sln files
> create by cmake, because multithreaded builds are not allowed on
> project level.
>
> Is there any way to enable them so that I don't have to do that by
> hand all time?
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

[CMake] Parallel builds

[Petr Bena]

Hi,

Is it possible to enable multithreaded builds by default in Visual
Studio projects? It seems that cmake generates the solution files in a
way that they are forbidden, so even if global preference is to use
multithreaded builds, the compiler never does that for sln files
create by cmake, because multithreaded builds are not allowed on
project level.

Is there any way to enable them so that I don't have to do that by
hand all time?
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Parallel builds for CDash targets (Continuous/Nightly/Experimental)?

[ddresser]

I found the answer to my question in this post

http://cmake.3232098.n2.nabble.com/CTest-make-Experimental-Nightly-on-N-procs-with-jN-td7584598.html

which says to run ctest directly with the -j flag instead of running make.

Sorry for the noise.



--
View this message in context: 
http://cmake.3232098.n2.nabble.com/Parallel-builds-for-CDash-targets-Continuous-Nightly-Experimental-tp7585255p7585256.html
Sent from the CMake mailing list archive at Nabble.com.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Bill Hoffman]

Michael Wild wrote:


On 2. Dec, 2009, at 16:36 , Sean McBride wrote:


On 12/2/09 10:10 AM, Bill Hoffman said:


Well, as Dave mentioned the -j flag only works with gmake.  It does not
work with Xcode, as you tried here:  :)


http://www.cdash.org/CDash/viewNotes.php?buildid=484307

Which is why nothing built for that build. :)


Doh!  I blindly changed all 30 of my scripts, should not have changed
that one obviously.  So in the Xcode case, how does it build?  Does it
invoke xcodebuild?  If so, that will already use many cores, so that's 
great.


Cheers,



xcodebuild also takes the -parallelizeTargets option.



It actually runs cmakexbuild which is a very thin wrapper for xcodebuild 
that has less verbose output, but the options should pass in just fine.


-Bill
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Michael Wild]

On 2. Dec, 2009, at 16:36 , Sean McBride wrote:


On 12/2/09 10:10 AM, Bill Hoffman said:

Well, as Dave mentioned the -j flag only works with gmake.  It does  
not

work with Xcode, as you tried here:  :)


http://www.cdash.org/CDash/viewNotes.php?buildid=484307

Which is why nothing built for that build. :)


Doh!  I blindly changed all 30 of my scripts, should not have changed
that one obviously.  So in the Xcode case, how does it build?  Does it
invoke xcodebuild?  If so, that will already use many cores, so  
that's great.


Cheers,



xcodebuild also takes the -parallelizeTargets option.

Michael

___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Sean McBride]

On 12/2/09 10:10 AM, Bill Hoffman said:

>Well, as Dave mentioned the -j flag only works with gmake.  It does not 
>work with Xcode, as you tried here:  :)
>
>
>http://www.cdash.org/CDash/viewNotes.php?buildid=484307
>
>Which is why nothing built for that build. :)

Doh!  I blindly changed all 30 of my scripts, should not have changed
that one obviously.  So in the Xcode case, how does it build?  Does it
invoke xcodebuild?  If so, that will already use many cores, so that's great.

Cheers,

-- 

Sean McBride, B. Eng s...@rogue-research.com
Rogue Researchwww.rogue-research.com 
Mac Software Developer  Montréal, Québec, Canada


___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Bill Hoffman]

Sean McBride wrote:

On 12/1/09 12:30 PM, Bill Hoffman said:


Is that better/worse/equivalent to David's suggestion of:

set(CTEST_BUILD_FLAGS -j4)  ?


Worse most likely. :)

Both should work.


Thanks.  If our dashboards are all red tomorrow, you'll know why. :)

Well, as Dave mentioned the -j flag only works with gmake.  It does not 
work with Xcode, as you tried here:  :)



http://www.cdash.org/CDash/viewNotes.php?buildid=484307

Which is why nothing built for that build. :)

-Bill
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Sean McBride]

On 12/1/09 12:30 PM, Bill Hoffman said:

>> Is that better/worse/equivalent to David's suggestion of:
>> 
>> set(CTEST_BUILD_FLAGS -j4)  ?
>> 
>
>Worse most likely. :)
>
>Both should work.

Thanks.  If our dashboards are all red tomorrow, you'll know why. :)

-- 

Sean McBride, B. Eng s...@rogue-research.com
Rogue Researchwww.rogue-research.com 
Mac Software Developer  Montréal, Québec, Canada


___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Bill Hoffman]

Sean McBride wrote:

On 11/30/09 3:52 PM, Bill Hoffman said:


David Cole wrote:

In the script, prior to the ctest_build call, do:
set(CTEST_BUILD_FLAGS -j4)

(only works with make that supports -j, obviously...)


We often do this in our scripts:

set(CTEST_BUILD_COMMAND "make -j4 -i")
MAKECOMMAND:STRING=/usr/bin/make -i -j4


Is that better/worse/equivalent to David's suggestion of:

set(CTEST_BUILD_FLAGS -j4)  ?



Worse most likely. :)

Both should work.

-Bill
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[Bill Hoffman]

David Cole wrote:

In the script, prior to the ctest_build call, do:
set(CTEST_BUILD_FLAGS -j4)

(only works with make that supports -j, obviously...)


We often do this in our scripts:

set(CTEST_BUILD_COMMAND "make -j4 -i")
MAKECOMMAND:STRING=/usr/bin/make -i -j4

I don't think we will add the make tool smarts into ctest_build to 
handle this as it depends on which make you are actually using.


-Bill
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds (ctest_build() ) for dashboards ?

[David Cole]

In the script, prior to the ctest_build call, do:
set(CTEST_BUILD_FLAGS -j4)

(only works with make that supports -j, obviously...)


On Mon, Nov 30, 2009 at 3:05 PM, Alexander Neundorf  wrote:

> Hi,
>
> I'm running some dashboard builds and didn't find a way how to make the
> builds
> (via ctest_build()) run parallel, e.g. make -4.
> For ctest_test() there is a PARALLEL_LEVEL option, I didn't find this for
> ctest_build().
> Is it possible to do this ?
>
> Alex
> ___
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

[CMake] parallel builds (ctest_build() ) for dashboards ?

[Alexander Neundorf]

Hi,

I'm running some dashboard builds and didn't find a way how to make the builds 
(via ctest_build()) run parallel, e.g. make -4.
For ctest_test() there is a PARALLEL_LEVEL option, I didn't find this for 
ctest_build().
Is it possible to do this ?

Alex
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Alexander Neundorf]

On Thursday 12 March 2009, Michael Jackson wrote:
> I think that is a CDT 5.x thing, I think. I have it on my CDT 5.x
> installation but didn't remember having it on CDT 4. If there is a way
> to set that "checkbox" during the CDT project file generation then CDT
> will figure out how many compile threads to use.
>   Here is a screen shot from my installation:

What effect does this checkbox have on the project files (.project 
and .cproject), i.e. how is this information stored in them ?
Also, does "optimal" mean it will run "make -j" or will eclipse try to 
determine how many cores there are and then run "make -j X" ?

Also, it would be nice if you could put this in the cmake bug tracker.

Alex
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[hl.ichaus]

Am Donnerstag, 12. März 2009 09:37:12 wrote Adolfo Rodríguez:
> We currently use three of CMake's generators: "Unix Makefiles",
> "KDevelop3", and "Eclipse CDT4 - Unix Makefiles", and we're interested in
> doing parallel builds. 

I use the environment variable MAKEFLAGS

before starting eclipse, i do a 
export MAKEFLAGS="-j15 --no-print-directory"

(I use icecream to do parallel build - instead of 5-10 minutes to do a full 
build, it takes me <1 Minute *smile*).

i guess it should also work to set the environment variable in "Make 
Preferences".

By theway, i also use the environment variable "VERBOSE=1", because eclipse 
parses the command line of the compiler - right? (not sure about this).

j. holzer
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Adolfo Rodríguez]

On Thu, Mar 12, 2009 at 10:56 PM, Alexander Neundorf <
a.neundorf-w...@gmx.net> wrote:

> On Thursday 12 March 2009, Bill Hoffman wrote:
> > Alexander Neundorf wrote:
> > >> I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an
> > >> issue?)
> > >
> > > I don't think so.
> > > We could add some special variable, like
> > > CMAKE_MAKEFILE_PROJECT_FLAGS or CMAKE_MAKEFILE_PARALLELITY, and use
> this
> > > in the makefile-based generators to set the "-jx" parameter for make.
> But
> > > then again, this shouldn't be hardcoded in the CMakeLists.txt, so it
> > > could only be a cache variable.
> > > Or should cmake try to determine automatically how many cores there are
> > > and add "-jx" automatically ? E.g. using sysconf(_SC_NPROCESSORS_ONLN)
> ?
> > > This could also be disabled by default and enabled e.g. via an option
> > > like CMAKE_AUTOMATIC_PARALLEL_MAKE_INVOCATION or something like this.
> >
> > This sounds like an Eclipse issue...  Does this work:
> >
> > http://dev.eclipse.org/newslists/news.eclipse.tools.cdt/msg15742.html
>
> From that link:
>
> > project properties -> C/C++ Build -> Behaviour tab
>
> I can't find this here.
> I have Project -> Properties -> C/C++ Make project
> and then there is nothing with parallel builds.
>
> Can you see that option ?


I have compared two Eclipse projects, one generated manually using custom
Makefiles, and another one genereated by CMake with the "Eclipse CDT4 - Unix
Makefiles" generator.
Effectively, the former _has_ the *project properties -> C/C++ Build ->
Behaviour *tab, but the CMake project does _not_. Could this be related to
the way CMake creates an Eclipse project? I used CDT5 and not 4, could it be
version related?

As Alex points out, the closest thing that is available is *Project ->
Properties -> C/C++ Make project*. There, in the Make Builder tab you can
set the Build command and add the -j# option. This is a per-project setting
and will only work if you build your project by doing *Project->Build
Project* or *Project->Build All*. It turns out that I had played with this
parameter before, and didn't seem to work because I was invoking directly
the individual targets (all, foo) from the Make Targets tab of the C/C++
perspective (and these targets have separate build settings).

As I'm quite ignorant on all things Eclipse, could it be possible to use the
Build Command specified in *Project -> Properties -> C/C++ Make project* for
each individual target? so that if it changes, this change will be
propagated. Right now I'm under the impression that they are decoupled.

Thanks,

Adolfo

P.S. The current workaround seems a good-enough solution for now.

>
> Alex
> ___
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>



-- 
Adolfo Rodríguez Tsouroukdissian

Robotics engineer
PAL ROBOTICS S.L
http://www.pal-robotics.com
Tel. +34.93.414.53.47
Fax.+34.93.209.11.09
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Alexander Neundorf]

On Thursday 12 March 2009, Bill Hoffman wrote:
> Alexander Neundorf wrote:
> >> I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an
> >> issue?)
> >
> > I don't think so.
> > We could add some special variable, like
> > CMAKE_MAKEFILE_PROJECT_FLAGS or CMAKE_MAKEFILE_PARALLELITY, and use this
> > in the makefile-based generators to set the "-jx" parameter for make. But
> > then again, this shouldn't be hardcoded in the CMakeLists.txt, so it
> > could only be a cache variable.
> > Or should cmake try to determine automatically how many cores there are
> > and add "-jx" automatically ? E.g. using sysconf(_SC_NPROCESSORS_ONLN) ?
> > This could also be disabled by default and enabled e.g. via an option
> > like CMAKE_AUTOMATIC_PARALLEL_MAKE_INVOCATION or something like this.
>
> This sounds like an Eclipse issue...  Does this work:
>
> http://dev.eclipse.org/newslists/news.eclipse.tools.cdt/msg15742.html

>From that link:

> project properties -> C/C++ Build -> Behaviour tab

I can't find this here.
I have Project -> Properties -> C/C++ Make project
and then there is nothing with parallel builds.

Can you see that option ?

Alex
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Bill Hoffman]

Alexander Neundorf wrote:


I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an issue?)


I don't think so.
We could add some special variable, like
CMAKE_MAKEFILE_PROJECT_FLAGS or CMAKE_MAKEFILE_PARALLELITY, and use this in 
the makefile-based generators to set the "-jx" parameter for make.
But then again, this shouldn't be hardcoded in the CMakeLists.txt, so it could 
only be a cache variable.
Or should cmake try to determine automatically how many cores there are and 
add "-jx" automatically ? E.g. using sysconf(_SC_NPROCESSORS_ONLN) ?
This could also be disabled by default and enabled e.g. via an option like 
CMAKE_AUTOMATIC_PARALLEL_MAKE_INVOCATION or something like this.




This sounds like an Eclipse issue...  Does this work:

http://dev.eclipse.org/newslists/news.eclipse.tools.cdt/msg15742.html

How is this done with non-cmake based Eclipse projects?

-Bill
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Timothy M. Shead]

Alexander Neundorf wrote:

On Thursday 12 March 2009, Adolfo Rodríguez wrote:

We currently use three of CMake's generators: "Unix Makefiles",
"KDevelop3", and "Eclipse CDT4 - Unix Makefiles", and we're interested in
doing parallel builds. Doing so with makefiles is trivial (make -j#), as
well as with KDevelop3 (only _one_ parameter must be edited in the build
options of the project). However Eclipse is still eluding me (I must
confess that I'm quite new to it).
My current hack is to edit individual targets and change the name of the
target from, say 'all' to '-j# all', and it works. However, I haven't found
a project-wide policy that if edited once, will affect all targets. I
cannot set the default build command, because apparently CMake generates
each target with a custom build command.

Any clues from the CMake-Eclipse experts?

I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an issue?)


I don't think so.
We could add some special variable, like
CMAKE_MAKEFILE_PROJECT_FLAGS or CMAKE_MAKEFILE_PARALLELITY, and use this in 
the makefile-based generators to set the "-jx" parameter for make.
But then again, this shouldn't be hardcoded in the CMakeLists.txt, so it could 
only be a cache variable.
Or should cmake try to determine automatically how many cores there are and 
add "-jx" automatically ? E.g. using sysconf(_SC_NPROCESSORS_ONLN) ?
This could also be disabled by default and enabled e.g. via an option like 
CMAKE_AUTOMATIC_PARALLEL_MAKE_INVOCATION or something like this.


I believe that this must be configurable - using distcc, developers may 
want to build with -jx > the number of local cores.


Cheers,
Tim

--
Timothy M. Shead
Data Analysis & Visualization (1424)
Sandia National Laboratories
505-284-0139

___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds and Eclipse CDT4 generator

[Alexander Neundorf]

On Thursday 12 March 2009, Adolfo Rodríguez wrote:
> We currently use three of CMake's generators: "Unix Makefiles",
> "KDevelop3", and "Eclipse CDT4 - Unix Makefiles", and we're interested in
> doing parallel builds. Doing so with makefiles is trivial (make -j#), as
> well as with KDevelop3 (only _one_ parameter must be edited in the build
> options of the project). However Eclipse is still eluding me (I must
> confess that I'm quite new to it).
> My current hack is to edit individual targets and change the name of the
> target from, say 'all' to '-j# all', and it works. However, I haven't found
> a project-wide policy that if edited once, will affect all targets. I
> cannot set the default build command, because apparently CMake generates
> each target with a custom build command.
>
> Any clues from the CMake-Eclipse experts?
>
> I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an issue?)

I don't think so.
We could add some special variable, like
CMAKE_MAKEFILE_PROJECT_FLAGS or CMAKE_MAKEFILE_PARALLELITY, and use this in 
the makefile-based generators to set the "-jx" parameter for make.
But then again, this shouldn't be hardcoded in the CMakeLists.txt, so it could 
only be a cache variable.
Or should cmake try to determine automatically how many cores there are and 
add "-jx" automatically ? E.g. using sysconf(_SC_NPROCESSORS_ONLN) ?
This could also be disabled by default and enabled e.g. via an option like 
CMAKE_AUTOMATIC_PARALLEL_MAKE_INVOCATION or something like this.

Alex
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

[CMake] Parallel builds and Eclipse CDT4 generator

[Adolfo Rodríguez]

We currently use three of CMake's generators: "Unix Makefiles", "KDevelop3",
and "Eclipse CDT4 - Unix Makefiles", and we're interested in doing parallel
builds. Doing so with makefiles is trivial (make -j#), as well as with
KDevelop3 (only _one_ parameter must be edited in the build options of the
project). However Eclipse is still eluding me (I must confess that I'm quite
new to it).
My current hack is to edit individual targets and change the name of the
target from, say 'all' to '-j# all', and it works. However, I haven't found
a project-wide policy that if edited once, will affect all targets. I cannot
set the default build command, because apparently CMake generates each
target with a custom build command.

Any clues from the CMake-Eclipse experts?

I'm using CMake 2.6.2 and Eclipse CDT5 (not CDT4, could this be an issue?)

Thanks,

-- 
Adolfo Rodríguez Tsouroukdissian

Robotics engineer
PAL ROBOTICS S.L
http://www.pal-robotics.com
Tel. +34.93.414.53.47
Fax.+34.93.209.11.09
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[Bill Hoffman]

Alan W. Irwin wrote:

On 2008-02-16 12:38-0500 Bill Hoffman wrote:

[...]parallel ctest is on the todo list for this year.  Most likely  
it will be ctest -j N.   We can not depend on the make system to do 
the parallel stuff, and ctest -j would be useful for all generators.


That's excellent news!  The execution of the PLplot tests actually take
about the same time to run as (non-parallel) building of PLplot itself. Now
we are doing our builds in parallel, convenient parallel testing will be
most welcome as well, and I look forward to trying out that new ctest
functionality once you are ready.



As a work around until we get it done, you can do something like this:

ctest -I1,10
ctest -I11,20

And run each of those in a different shell.

-Bill
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[Alan W. Irwin]

On 2008-02-16 12:38-0500 Bill Hoffman wrote:

[...]parallel ctest is on the todo list for this year.  Most likely  it 
will be ctest -j N.   We can not depend on the make system to do the parallel 
stuff, and ctest -j would be useful for all generators.


That's excellent news!  The execution of the PLplot tests actually take
about the same time to run as (non-parallel) building of PLplot itself. Now
we are doing our builds in parallel, convenient parallel testing will be
most welcome as well, and I look forward to trying out that new ctest
functionality once you are ready.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[Bill Hoffman]

David Cole wrote:




Parallel tests are only going to be possible if you can guarantee that 
all tests are independent of all other tests, or if you specify explicit 
dependencies between tests somehow. (Which would introduce the same sort 
of complexity you complain about above for build steps...)


That said, parallel ctest is on the todo list for this year.  Most 
likely  it will be ctest -j N.   We can not depend on the make system to 
do the parallel stuff, and ctest -j would be useful for all generators.


-Bill
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[David Cole]

On 2/15/08, Alan W. Irwin <[EMAIL PROTECTED]> wrote:
>
> David, I may be leaping to conclusions here, but it appears from what you
> have said that it is a reasonable hypothesis that the ctest commmand works
> through the cmake generator backend.


True


When that is "make" (the default for
> Unix systems), then it further appears you can use a parallel version of
> make.  That would be a wonderful feature to have for ctest.


You can and it is. :-)

Also, when you use Visual Studio 2005 or 2008, builds happen in parallel
automatically according to your options, just as they do when you initiate
them through the UI. No need for a custom MAKECOMMAND.


However, assuming the above suppositions are all true, I don't particularly
> like the somewhat clumsy hardwired mechanism above which won't work very
> well for machines with more than two processors or for systems where you
> want to do your tests in parallel, but you might not want to do your build
> in parallel (since parallel builds are difficult to get right with cmake
> for
> projects with complicated dependencies).


You cannot do the tests in parallel via ctest. It runs all the tests it's
going to run sequentially. And although parallel builds may be difficult to
"get right," you are much more likely to get them right if you run nightly
dashboards using this technique and observing the results every day. When
there's a dependency issue, it usually shows up as a sporadic dashboard
failure unrelated to the changes made for the day


One way to do this smoothly is allow ctest to pass any arguments to
> the backend that the user desires.
>
> Also, didn't "make test" work for early versions of cmake-2.4.x?  I find
> it
> is a no-op now.  (I was thinking that "make -j N test" might be a slick
> way
> to get parallel tests.)


Parallel tests are only going to be possible if you can guarantee that all
tests are independent of all other tests, or if you specify explicit
dependencies between tests somehow. (Which would introduce the same sort of
complexity you complain about above for build steps...)

"make test" should still work. What happens when you do "ctest -N" in the
binary directory? "make test" should be equivalent to running "ctest" from
the binary directory... If it seems to be a no-op, perhaps you do not have
an ENABLE_TESTING() call in your CMakeLists.txt?

HTH,
David
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[Alan W. Irwin]

On 2008-02-15 18:44-0500 David Cole wrote:


One of the VTK dashboards does it by giving a custom MAKECOMMAND to the
initial cache:MAKECOMMAND:STRING=/usr/bin/make -i -j 2

See here for the full script:
http://www.vtk.org/Testing/Sites/hythloth.kitware/Linux-gcc41/20080215-0300-Nightly/Notes.html


David, I may be leaping to conclusions here, but it appears from what you
have said that it is a reasonable hypothesis that the ctest commmand works
through the cmake generator backend.  When that is "make" (the default for
Unix systems), then it further appears you can use a parallel version of
make.  That would be a wonderful feature to have for ctest.

However, assuming the above suppositions are all true, I don't particularly
like the somewhat clumsy hardwired mechanism above which won't work very
well for machines with more than two processors or for systems where you
want to do your tests in parallel, but you might not want to do your build
in parallel (since parallel builds are difficult to get right with cmake for
projects with complicated dependencies).

One way to do this smoothly is allow ctest to pass any arguments to
the backend that the user desires.

Also, didn't "make test" work for early versions of cmake-2.4.x?  I find it
is a no-op now.  (I was thinking that "make -j N test" might be a slick way
to get parallel tests.)

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] parallel builds for ctest

[David Cole]

One of the VTK dashboards does it by giving a custom MAKECOMMAND to the
initial cache:MAKECOMMAND:STRING=/usr/bin/make -i -j 2

See here for the full script:
http://www.vtk.org/Testing/Sites/hythloth.kitware/Linux-gcc41/20080215-0300-Nightly/Notes.html


HTH,
David


On 2/15/08, James Bigler <[EMAIL PROTECTED]> wrote:
>
> Is it possible to get ctest to do parallel builds on systems that support
> it?
>
> I'm using a configuration file that looks something like this:
>
> # Where the source code lives
> SET (CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/..")
> SET (CTEST_BINARY_DIRECTORY "${CTEST_SOURCE_DIRECTORY}/build-ctest-sse")
>
> # Make sure we always reconfigure cmake stuff from scratch and don't
> # rely on previously built libraries
> SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
>
> SET (CTEST_CMAKE_COMMAND "cmake")
> SET (CTEST_CVS_COMMAND "svn")
>
> # A smoke test only builds the code and doesn't run any tests, so we
> # exclude all tests here
> #SET (CTEST_COMMAND "ctest -D Nightly")
> SET (CTEST_COMMAND "ctest -D Experimental")
>
> SET(CTEST_INITIAL_CACHE "
>BUILDNAME:STRING=SSE
>BUILD_TESTING:BOOL=ON
> ")
>
>
> James
> ___
> CMake mailing list
> CMake@cmake.org
> http://www.cmake.org/mailman/listinfo/cmake
>
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

[CMake] parallel builds for ctest

[James Bigler]

Is it possible to get ctest to do parallel builds on systems that support it?

I'm using a configuration file that looks something like this:

# Where the source code lives
SET (CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/..")
SET (CTEST_BINARY_DIRECTORY "${CTEST_SOURCE_DIRECTORY}/build-ctest-sse")

# Make sure we always reconfigure cmake stuff from scratch and don't
# rely on previously built libraries
SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)

SET (CTEST_CMAKE_COMMAND "cmake")
SET (CTEST_CVS_COMMAND "svn")

# A smoke test only builds the code and doesn't run any tests, so we
# exclude all tests here
#SET (CTEST_COMMAND "ctest -D Nightly")
SET (CTEST_COMMAND "ctest -D Experimental")

SET(CTEST_INITIAL_CACHE "
  BUILDNAME:STRING=SSE
  BUILD_TESTING:BOOL=ON
")


James
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2008-02-06 22:37-0500 Bill Hoffman wrote:


Alan W. Irwin wrote:


No file level depends are done mostly a build time.  This is a performance 
issue.  Some generators like VS IDE do file level depends by themselves. 
With the makefiles cmake computes the depends, but at build time not cmake 
time.  The custom command stuff output input is known at cmake time, and 
maybe enough for what you want.


Probably, since it is usually there where the build-system developer makes
mistakes in the dependencies.

But if you have a file foo.c with #include 
, cmake does not know that foo.c depends on foo.h until build time.


Right, but hopefully those automatically generated depends would be ok.

Alan.
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2008-02-06 21:04-0500 Brad King wrote:


Alan W. Irwin wrote:

On 2007-12-14 09:53-0800 Alan W. Irwin wrote:


On 2007-12-14 10:32-0500 Brad King wrote:


CMake employs a 2-level make recursion system that is independent of 
the

directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That 
determines

the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up 
to

date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be 
built
simultaneously and you get race conditions causing the double 
evaluations.


CMake traces through the dependencies of custom commands in each 
target.

When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but 
then

the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.


That's fine.  Your combined explanation now makes sense and completely
confirms my working hypothesis that the make recursion system of CMake 
is
responsible for the parallel build issues I was encountering.  I hope I 
can
work around these PLplot parallel build issues (note the double copy 
issue

was only the most obvious one) by using extra target dependencies.  The
problem is that parallel build issues tend to appear and disappear 
depending
on load, the N level (for -j N), and hardware.  Thus, even if a whole 
flock

of PLplot developers confirm success for parallel builds, there could be
some subtle dependency issue left that we have missed, and some user 
down

the road is going to come up with a combination of load, N, and hardware
that triggers the parallel build problem because of that dependency 
issue.

As a PLplot developer, I don't like being in such an uncertain position!


I thought it important to resurrect this two-month old thread because 
today
I _finally_ got success (at least no obvious issues, see comment below) 
with
parallel builds of PLplot on my particular platform.  That's the good 
news.


The bad news is it took so much effort.  Plplot is not that big a piece of
software, but there are a large number of different components with 
complex
dependencies between them.  Therefore I had several tries in the two 
months
to get parallel builds to work that failed miserably.  This last 
successful

effort of getting "make -J N" to work for many different N values took at
least several days of isolating the problem by enabling/disabling various
PLplot components until I was finally able to find and fix the last two
dependency issues that showed up on my system.

Even worse news is I caught the last problem only by accident. That 
problem

only showed up intermittently for N = 4 for a very specific PLplot
configuration.  N=2 and N=8 never showed any problems for that 
configuration

for my two-processor hardware!  So from that experience it is unlikely I
caught all issues.

To help to sort out such difficult dependency issues with CMake (which
affect parallel builds on Unix system and I understand also certain kinds 
of

builds on Windows), I have a feature request I would like to discuss here
before I make a formal feature request on the kitware bug system.


I already made one for this:

http://public.kitware.com/Bug/view.php?id=6285


That is great that you are considering automatically putting in target
depends if two targets depend on the same file.  That new feature would
address the original issue that started this thread, and I am all in favour
of this feature for that reason.

However, during my dependency hell I discovered other issues with the PLplot
depends such as missing dependencies between custom commands.  Those missing
dependencies didn't matter for the non-parallel build case because the order
of the custom commands was deliberately chosen (back in our autotools days
and simply copied to our CMake build system without much thought) so that
the files were built in the correct order, but of course that doesn't happen
for parallel builds.  So some sort of output that emphasizes targets or
files without many depends (which mean they are suspects for missing
dependencies) is needed as well.  Bill's idea of adding file depends to the
graphviz output file would probably satisfy that need since isolated
files/targets would really stand out.

Alan

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Bill Hoffman]

Alan W. Irwin wrote:



My first interpretation was "that" referred to graphviz, but in fact the
file was produced at cmake time, and it was a simple matter to process 
it by

hand using the "dot" command-line tool (even though I had never heard of
that tool or graphviz before). "gv" has errors for both the ps and pdf
results, but I think that is because the latest gv is extra careful about
non-standard ps and pdf files.  xpdf could understand the pdf output, but I
have to say the result is black with dependency lines to a frightening
extent. I can send the pdf file to Brad and/or you off-list if either of 
you

is interested in being frightened by the PLplot dependencies as well.  :-)

Seriously, I am fairly impressed with the graphviz result, and adding in
the file depends would add a lot of value to the result.

If your "that" refers to file depends instead of graphviz, I don't
understand your comment since surely file depend information is 
available at

cmake time?



No file level depends are done mostly a build time.  This is a 
performance issue.  Some generators like VS IDE do file level depends by 
themselves.  With the makefiles cmake computes the depends, but at build 
time not cmake time.  The custom command stuff output input is known at 
cmake time, and maybe enough for what you want.  But if you have a file 
foo.c with #include , cmake does not know that foo.c depends on 
foo.h until build time.


-Bill
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2008-02-06 21:05-0500 Bill Hoffman wrote:

Could we have a cmake command-line option to evaluate/diagnose the 
complete
list of file and target dependencies as understood by cmake? You could 
start

with a print out of complete target dependency chains and file dependency
chains as cmake understands them.  As part of that printout it would be
useful to highlight files or targets that are built with few dependencies
since that might be a sign of missing dependencies.  And also highlight
chains of file depends that include files that are part of other chains of
file depends. You could put in some error analysis as well (in case two
targets which do not target-depend on each other file-depend on the same
file, for example.)

Anyhow, as I went through this dependency hell for PLplot I kept wishing 
for

such a diagnostic tool, and I think it would be useful for others as well
that are dealing with projects like PLplot with complex dependency chains
spread over quite a few different directories.

What do you think?



You could try this:

cmake --graphviz=[file]   = Generate graphviz of dependencies.

It will only show the target level stuff.  It would be another project to get 
the file level depend stuff to show up.  The problem is that is done at build
   

time and not a cmake time.


My first interpretation was "that" referred to graphviz, but in fact the
file was produced at cmake time, and it was a simple matter to process it by
hand using the "dot" command-line tool (even though I had never heard of
that tool or graphviz before). "gv" has errors for both the ps and pdf
results, but I think that is because the latest gv is extra careful about
non-standard ps and pdf files.  xpdf could understand the pdf output, but I
have to say the result is black with dependency lines to a frightening
extent. I can send the pdf file to Brad and/or you off-list if either of you
is interested in being frightened by the PLplot dependencies as well.  :-)

Seriously, I am fairly impressed with the graphviz result, and adding in
the file depends would add a lot of value to the result.

If your "that" refers to file depends instead of graphviz, I don't
understand your comment since surely file depend information is available at
cmake time?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Bill Hoffman]

Alan W. Irwin wrote:

On 2007-12-14 09:53-0800 Alan W. Irwin wrote:


On 2007-12-14 10:32-0500 Brad King wrote:



CMake employs a 2-level make recursion system that is independent of the
directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That determines
the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up to
date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be built
simultaneously and you get race conditions causing the double 
evaluations.


CMake traces through the dependencies of custom commands in each target.
When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but then
the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.


That's fine.  Your combined explanation now makes sense and completely
confirms my working hypothesis that the make recursion system of CMake is
responsible for the parallel build issues I was encountering.  I hope 
I can
work around these PLplot parallel build issues (note the double copy 
issue

was only the most obvious one) by using extra target dependencies.  The
problem is that parallel build issues tend to appear and disappear 
depending
on load, the N level (for -j N), and hardware.  Thus, even if a whole 
flock

of PLplot developers confirm success for parallel builds, there could be
some subtle dependency issue left that we have missed, and some user down
the road is going to come up with a combination of load, N, and hardware
that triggers the parallel build problem because of that dependency 
issue.

As a PLplot developer, I don't like being in such an uncertain position!


I thought it important to resurrect this two-month old thread because today
I _finally_ got success (at least no obvious issues, see comment below) 
with

parallel builds of PLplot on my particular platform.  That's the good news.

The bad news is it took so much effort.  Plplot is not that big a piece of
software, but there are a large number of different components with complex
dependencies between them.  Therefore I had several tries in the two months
to get parallel builds to work that failed miserably.  This last successful
effort of getting "make -J N" to work for many different N values took at
least several days of isolating the problem by enabling/disabling various
PLplot components until I was finally able to find and fix the last two
dependency issues that showed up on my system.

Even worse news is I caught the last problem only by accident. That problem
only showed up intermittently for N = 4 for a very specific PLplot
configuration.  N=2 and N=8 never showed any problems for that 
configuration

for my two-processor hardware!  So from that experience it is unlikely I
caught all issues.

To help to sort out such difficult dependency issues with CMake (which
affect parallel builds on Unix system and I understand also certain 
kinds of

builds on Windows), I have a feature request I would like to discuss here
before I make a formal feature request on the kitware bug system.

Could we have a cmake command-line option to evaluate/diagnose the complete
list of file and target dependencies as understood by cmake? You could 
start

with a print out of complete target dependency chains and file dependency
chains as cmake understands them.  As part of that printout it would be
useful to highlight files or targets that are built with few dependencies
since that might be a sign of missing dependencies.  And also highlight
chains of file depends that include files that are part of other chains of
file depends. You could put in some error analysis as well (in case two
targets which do not target-depend on each other file-depend on the same
file, for example.)

Anyhow, as I went through this dependency hell for PLplot I kept wishing 
for

such a diagnostic tool, and I think it would be useful for others as well
that are dealing with projects like PLplot with complex dependency chains
spread over quite a few different directories.

What do you think?



You could try this:

 cmake --graphviz=[file]   = Generate graphviz of dependencies.

It will only show the target level stuff.  It would be another project 
to get the file level depend stuff to 

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:

On 2007-12-14 09:53-0800 Alan W. Irwin wrote:


On 2007-12-14 10:32-0500 Brad King wrote:



CMake employs a 2-level make recursion system that is independent of the
directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That determines
the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up to
date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be built
simultaneously and you get race conditions causing the double 
evaluations.


CMake traces through the dependencies of custom commands in each target.
When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but then
the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.


That's fine.  Your combined explanation now makes sense and completely
confirms my working hypothesis that the make recursion system of CMake is
responsible for the parallel build issues I was encountering.  I hope 
I can
work around these PLplot parallel build issues (note the double copy 
issue

was only the most obvious one) by using extra target dependencies.  The
problem is that parallel build issues tend to appear and disappear 
depending
on load, the N level (for -j N), and hardware.  Thus, even if a whole 
flock

of PLplot developers confirm success for parallel builds, there could be
some subtle dependency issue left that we have missed, and some user down
the road is going to come up with a combination of load, N, and hardware
that triggers the parallel build problem because of that dependency 
issue.

As a PLplot developer, I don't like being in such an uncertain position!


I thought it important to resurrect this two-month old thread because today
I _finally_ got success (at least no obvious issues, see comment below) 
with

parallel builds of PLplot on my particular platform.  That's the good news.

The bad news is it took so much effort.  Plplot is not that big a piece of
software, but there are a large number of different components with complex
dependencies between them.  Therefore I had several tries in the two months
to get parallel builds to work that failed miserably.  This last successful
effort of getting "make -J N" to work for many different N values took at
least several days of isolating the problem by enabling/disabling various
PLplot components until I was finally able to find and fix the last two
dependency issues that showed up on my system.

Even worse news is I caught the last problem only by accident. That problem
only showed up intermittently for N = 4 for a very specific PLplot
configuration.  N=2 and N=8 never showed any problems for that 
configuration

for my two-processor hardware!  So from that experience it is unlikely I
caught all issues.

To help to sort out such difficult dependency issues with CMake (which
affect parallel builds on Unix system and I understand also certain 
kinds of

builds on Windows), I have a feature request I would like to discuss here
before I make a formal feature request on the kitware bug system.


I already made one for this:

http://public.kitware.com/Bug/view.php?id=6285

-Brad
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-14 09:53-0800 Alan W. Irwin wrote:


On 2007-12-14 10:32-0500 Brad King wrote:



CMake employs a 2-level make recursion system that is independent of the
directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That determines
the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up to
date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be built
simultaneously and you get race conditions causing the double evaluations.

CMake traces through the dependencies of custom commands in each target.
When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but then
the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.


That's fine.  Your combined explanation now makes sense and completely
confirms my working hypothesis that the make recursion system of CMake is
responsible for the parallel build issues I was encountering.  I hope I can
work around these PLplot parallel build issues (note the double copy issue
was only the most obvious one) by using extra target dependencies.  The
problem is that parallel build issues tend to appear and disappear depending
on load, the N level (for -j N), and hardware.  Thus, even if a whole flock
of PLplot developers confirm success for parallel builds, there could be
some subtle dependency issue left that we have missed, and some user down
the road is going to come up with a combination of load, N, and hardware
that triggers the parallel build problem because of that dependency issue.
As a PLplot developer, I don't like being in such an uncertain position!


I thought it important to resurrect this two-month old thread because today
I _finally_ got success (at least no obvious issues, see comment below) with
parallel builds of PLplot on my particular platform.  That's the good news.

The bad news is it took so much effort.  Plplot is not that big a piece of
software, but there are a large number of different components with complex
dependencies between them.  Therefore I had several tries in the two months
to get parallel builds to work that failed miserably.  This last successful
effort of getting "make -J N" to work for many different N values took at
least several days of isolating the problem by enabling/disabling various
PLplot components until I was finally able to find and fix the last two
dependency issues that showed up on my system.

Even worse news is I caught the last problem only by accident. That problem
only showed up intermittently for N = 4 for a very specific PLplot
configuration.  N=2 and N=8 never showed any problems for that configuration
for my two-processor hardware!  So from that experience it is unlikely I
caught all issues.

To help to sort out such difficult dependency issues with CMake (which
affect parallel builds on Unix system and I understand also certain kinds of
builds on Windows), I have a feature request I would like to discuss here
before I make a formal feature request on the kitware bug system.

Could we have a cmake command-line option to evaluate/diagnose the complete
list of file and target dependencies as understood by cmake? You could start
with a print out of complete target dependency chains and file dependency
chains as cmake understands them.  As part of that printout it would be
useful to highlight files or targets that are built with few dependencies
since that might be a sign of missing dependencies.  And also highlight
chains of file depends that include files that are part of other chains of
file depends. You could put in some error analysis as well (in case two
targets which do not target-depend on each other file-depend on the same
file, for example.)

Anyhow, as I went through this dependency hell for PLplot I kept wishing for
such a diagnostic tool, and I think it would be useful for others as well
that are dealing with projects like PLplot with complex dependency chains
spread over quite a few different directories.

What do you think?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
f

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:

So let me rephrase the question.  Are the CMake developers happy with the
present state of the dependencies system or are you considering some major
changes there because of such issues as the difficulties in getting 
parallel

builds to work properly for projects like PLplot which (necessarily) have
complicated chains of dependencies?


I personally don't have any problems with the current state.  I do 
parallel builds of big projects all the time.  There will be no major 
changes.  If you want to submit a feature request to the bug tracker 
you're welcome to do so.


-Brad
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-15 12:57-0500 Brad King wrote:


Alan W. Irwin wrote:

Well, it turns out I had to add four different target dependencies to the
CMake-based PLplot build system to get rid of the parallel build problems 
I
was having on my Core Duo box.  One of them was pretty subtle so I missed 
it

for my first review of the dependencies. Nevertheless, these changes were
not as extensive as I thought they would be so there is some hope that I
didn't miss anything that will show up as strange parallel build problems 
for PLplot on other machines.


Great, I'm glad you got it working.


Well, I thought so, but my previous test was without the (docbook)
documentation build.  Now, that I have included that, the parallel build 
errors out.  For the last few hours I have been going through the

complicated dependencies in our documentation build, but I just cannot see
what is causing the trouble.  Perhaps if I sleep on it, it will become
obvious tomorrow.



Is that complete rework actually going to happen for 2.6.x or is it 
currently just a gleam in the CMake developer's eyes?


To what message are you referring?


I was sure I remembered a discussion of reworking the CMake depends system
on this list in the past year, but I have been unable to find it so perhaps
I was misremembering (or perhaps my searching skills are not good enough).

So let me rephrase the question.  Are the CMake developers happy with the
present state of the dependencies system or are you considering some major
changes there because of such issues as the difficulties in getting parallel
builds to work properly for projects like PLplot which (necessarily) have
complicated chains of dependencies?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:

Well, it turns out I had to add four different target dependencies to the
CMake-based PLplot build system to get rid of the parallel build problems I
was having on my Core Duo box.  One of them was pretty subtle so I 
missed it

for my first review of the dependencies. Nevertheless, these changes were
not as extensive as I thought they would be so there is some hope that I
didn't miss anything that will show up as strange parallel build 
problems for PLplot on other machines.


Great, I'm glad you got it working.

Is that complete rework actually going to happen for 2.6.x or is it 
currently just a gleam in the CMake developer's eyes?


To what message are you referring?

-Brad
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-14 09:53-0800 Alan W. Irwin wrote:


Obviously, CMake 2.4.x users are stuck with these file dependency issues and
their workarounds, but for obvious reasons and especially for the parallel
build case I hope the complete rework of the CMake dependency system that
has been mentioned previously on list will remove these limitations.


Well, it turns out I had to add four different target dependencies to the
CMake-based PLplot build system to get rid of the parallel build problems I
was having on my Core Duo box.  One of them was pretty subtle so I missed it
for my first review of the dependencies. Nevertheless, these changes were
not as extensive as I thought they would be so there is some hope that I
didn't miss anything that will show up as strange parallel build problems 
for PLplot on other machines.


I am still interested in the answer to the question below.

Alan


Is that complete rework actually going to happen for 2.6.x or is it currently
just a gleam in the CMake developer's eyes?


__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-14 12:49-0500 Bill Hoffman wrote:

You might also want to consider visual studio builds.  It will build two 
targets at the same time if there is no dependency between them, and would 
have the same issue.


Currently, we have had no reports about such problems.  However, our windows
developers (and users) tend to use just the core of PLplot mostly because
that was all that was available in the past for our previous home-brew build
system for windows, and installing the extra libraries needed for the rest
of PLplot (additional language interfaces and additional plot device and
plot file drivers) can be an issue for windows users.  Thus, it is quite
possible our windows developers have so far fortuitously skated by the
issue, and that testing of a complete PLplot build on windows would show
similar dependency issues to what I am getting now with parallel builds of a
complete PLplot under Linux.

Hopefully, a complete dependency review and deploying the appropriate
workarounds will make all these PLplot parallel build (and potentially
windows build) problems go away, but I am definitely concerned the review
might miss some complex target- or file-dependency chains that may only
cause intermittent and difficult-to-reproduce parallel build problems.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-14 10:32-0500 Brad King wrote:


Alan W. Irwin wrote:

I am struggling with understanding the recursive make system that
CMake normally employs


CMake employs a 2-level make recursion system that is independent of the
directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That determines
the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up to
date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be built
simultaneously and you get race conditions causing the double evaluations.

CMake traces through the dependencies of custom commands in each target.
When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but then
the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.


That's fine.  Your combined explanation now makes sense and completely
confirms my working hypothesis that the make recursion system of CMake is
responsible for the parallel build issues I was encountering.  I hope I can
work around these PLplot parallel build issues (note the double copy issue
was only the most obvious one) by using extra target dependencies.  The
problem is that parallel build issues tend to appear and disappear depending
on load, the N level (for -j N), and hardware.  Thus, even if a whole flock
of PLplot developers confirm success for parallel builds, there could be
some subtle dependency issue left that we have missed, and some user down
the road is going to come up with a combination of load, N, and hardware
that triggers the parallel build problem because of that dependency issue.
As a PLplot developer, I don't like being in such an uncertain position!  I
wonder if really large projects such as KDE have attempted to deal with
parallel build issues for CMake or whether they have just given up on
parallel builds because the symptoms can be so intermittent and
non-reproducible.

I was well aware of the CMake file dependency limitation for separate
directories before, but I did not realize that limitation extends to file
dependencies in the _same_ directory when using parallel builds.  I
personally think these limitations of your 2-level make recursion system are
pretty ugly since they require CMake users to deploy nonintuitive (at least
from the make perspective) additional target-level dependencies as a
workaround for these issues. The problem is especially pernicious for the
parallel builds case since the symptoms are inherently difficult to
reproduce.

Obviously, CMake 2.4.x users are stuck with these file dependency issues and
their workarounds, but for obvious reasons and especially for the parallel
build case I hope the complete rework of the CMake dependency system that
has been mentioned previously on list will remove these limitations.

Is that complete rework actually going to happen for 2.6.x or is it currently
just a gleam in the CMake developer's eyes?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Bill Hoffman]

You might also want to consider visual studio builds.  It will build two 
targets at the same time if there is no dependency between them, and 
would have the same issue.


-Bill
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:
> I am struggling with understanding the recursive make system that
> CMake normally employs

CMake employs a 2-level make recursion system that is independent of the
directory structure.  The first level never builds anything...it just
evaluates target-level dependencies with phony targets.  That determines
the order in which targets must be built.  The second level is the
build.make for each target.  This is where file-level dependencies are
evaluated.

In your example the file1...fileN rules are showing up in target1's
build.make and target2's build.make but they should never be evaluated
in the second target.  They are pulled in through the additional_file
rule's dependencies on them (see below), but they should always be up to
date if target2 doesn't build until after target1 finishes.  Then only
the additional_file rule will be invoked.  However if there is no
dependency from target2->target1 then both build.make files may be built
simultaneously and you get race conditions causing the double evaluations.

CMake traces through the dependencies of custom commands in each target.
 When it is constructing target2 it doesn't know that target1 will also
provide rules for the files.  If you place the targets in different
directories it would not be able to make this extra connection, but then
the build would not work correctly unless you add the target-level
dependency.  Any further explanation here will just duplicate my
previous message so I'll stop.

-Brad
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-13 18:45-0800 Alan W. Irwin wrote:


Brad, I am struggling with understanding the recursive make system that
CMake normally employs so I am having trouble following the complete
Makefile logic that my simple example creates.  However,
CMakeFiles/tclIndex_examples_tcl2.dir/build.make generated by my simple
CMake example seems to follow the above OPTION A scenario.  Indeed, if I 
execute

that Makefile directly from the commmand line, e.g.,

make -f CMakeFiles/tclIndex_examples_tcl2.dir/build.make \
CMakeFiles/tclIndex_examples_tcl2.dir/clean
make -j 2 -f CMakeFiles/tclIndex_examples_tcl2.dir/build.make \
CMakeFiles/tclIndex_examples_tcl2.dir/build

there are never double copy problems, while if I run

make clean
make -j 2

there are always double copy problems.  (You should try this for yourself to
be sure you can replicate my experience.) So my current working hypothesis
is there is a parallel build issue for OPTION A that CMake artificially
introduces when it recursively invokes make (i.e., the result of the
above "make -j 2" command).


That last sentence was poorly written.  Replace it with the following:

So my current working hypothesis is there is a parallel build issue for
OPTION A that CMake artificially introduces through its generated recursive
make system.  That generated recursive make system is invoked with the above
"make -j 2" command, but bypassed with the "make -j 2 -f CMakeFiles/..."
command above.

Alan

__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-13 19:15-0500 Brad King wrote:


Alan W. Irwin wrote:

So the rule seems to be that parallel builds do not work if there are two
or more separate custom targets that file depend directly or indirectly
(via
some custom command file dependency chain) on the same output files.

Another way of summarizing these results is that file depends must be
minimized and/or custom target depends maximized in order for parallel
builds to work properly.


This is correct.


My guess is I should be able to work around this CMake issue by appropriate
changes to the PLplot build system although I have a number of these
parallel build issues and the copy problem was only the most obvious.

I do regard this as a CMake issue.  Normally, the shoe is on the other
foot,
and the build system developer is desperately trying to make sure that all
the CMake file depends are obviously in place locally rather than depending
on a long easily-broken chain of dependencies to do it for them in a
minimalist way.  So the big question is whether CMake can be modified so
that minimalist file depends and/or maximal (and unintuitive) target
depends
are not required in order for parallel builds to work properly.


I don't see how CMake can automatically fix this.  If two targets think
they know how to build the same file how is CMake supposed to know which
one is the correct target to build first?


I am not completely convinced by that reasoning.  Let me create an abstract
case in Makefile terms that we can discuss (at first) strictly from the GNU
make point of view.  Putting my simple test case in Makefile terms we have
the following rules:

all: target1 target2

file1:
custom command to create file1

fileN:
custom command to create fileN

target1: file1, file2,fileN

additional_file: file1, file2,fileN
custom command to create additional_file

There are two alternatives for the target2 dependencies

Either

target2: additional_file  (OPTION A)

or

target2: target1 additional_file (OPTION B)

all, target1, and target2 arephony targets that do not correspond
to actual files.

OPTION A is what my simple example (and current PLplot) uses.  OPTION B
is one fixup you discussed where you made target2 depend on target1.

I know OPTION A always works for serial builds. The reason is each target
knows independently how to build what it needs.  target1 file-depends
directly on the files from a variety of file1 through fileN custom rules. So
it knows how to build exactly what it needs.  target2 (with OPTION A)
file-depends on additional_file which file depends on the file1-fileN rules.
So target2 knows how to build exactly what it needs as well. Thus, if you
remove target1, target2 would build without problems and vice versa.

Of course once you introduce parallel builds it gets complicated, and I
would appreciate your input on that.  I have always assumed that if one
processor was busy creating file1 through the target1 chain of dependencies
the make command kept track that a build of file1 was in progress so the
other processor would not attempt to duplicate that build regardless of
whether it needed it via the target1 or target2 dependency chains.  Indeed,
OPTION A works well for parallel builds (see below where I non-recursively
invoke the appropriate Makefile that is generated by CMake).

You claim that OPTION B must be used for parallel builds (at least if I have
understood you correctly and if my mental model of how cmake dependencies
translate to Makefile dependencies is correct).  I just don't see the
necessity of OPTION B for parallel builds for non-recursive Makefiles, but I
am willing to be educated. :-)

If GNU make does parallel builds without problems for non-recursive OPTION A
(which appears to be the case, see below), then my concern is that CMake is
introducing some additional make issues via the make recursion that it
normally employs that screws up parallel builds and OPTION B is simply a
workaround that bypasses that recursion issue.

Brad, I am struggling with understanding the recursive make system that
CMake normally employs so I am having trouble following the complete
Makefile logic that my simple example creates.  However,
CMakeFiles/tclIndex_examples_tcl2.dir/build.make generated by my simple
CMake example seems to follow the above OPTION A scenario.  Indeed, if I execute
that Makefile directly from the commmand line, e.g.,

make -f CMakeFiles/tclIndex_examples_tcl2.dir/build.make \
CMakeFiles/tclIndex_examples_tcl2.dir/clean
make -j 2 -f CMakeFiles/tclIndex_examples_tcl2.dir/build.make \
CMakeFiles/tclIndex_examples_tcl2.dir/build

there are never double copy problems, while if I run

make clean
make -j 2

there are always double copy problems.  (You should try this for yourself to
be sure you can replicate my experience.) So my current working hypothesis
is there is a parallel build issue for OPTION A that CMake artificially
introduces when it recursively invokes make (i.e., the res

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:
> So the rule seems to be that parallel builds do not work if there are two
> or more separate custom targets that file depend directly or indirectly
> (via
> some custom command file dependency chain) on the same output files.
> 
> Another way of summarizing these results is that file depends must be
> minimized and/or custom target depends maximized in order for parallel
> builds to work properly.

This is correct.

> My guess is I should be able to work around this CMake issue by appropriate
> changes to the PLplot build system although I have a number of these
> parallel build issues and the copy problem was only the most obvious.
> 
> I do regard this as a CMake issue.  Normally, the shoe is on the other
> foot,
> and the build system developer is desperately trying to make sure that all
> the CMake file depends are obviously in place locally rather than depending
> on a long easily-broken chain of dependencies to do it for them in a
> minimalist way.  So the big question is whether CMake can be modified so
> that minimalist file depends and/or maximal (and unintuitive) target
> depends
> are not required in order for parallel builds to work properly.

I don't see how CMake can automatically fix this.  If two targets think
they know how to build the same file how is CMake supposed to know which
one is the correct target to build first?

Consider this:

1.) Split your example into two separate directories
2.) Put the first-level custom commands in a target in one dir
3.) Put the second-level custom commands in a target in the other dir
4.) Try to build

Without explicit dependence between the two targets there is no way the
target in the second directory can know it must wait for the target in
the first directory to build.  It doesn't even know about the
first-level custom command rules or that the input to one if its
second-level custom commands is a generated file.  The second
directory's target will try to build and complain that input files are
missing.

This problem arises because CMake does not track file-level dependencies
globally.  It actually can't because some target build environments like
the VS and Xcode IDEs do not provide this capability.  File-level
dependencies must be divided into targets.

In your case the target-level dependency that must be added is a logical
high-level statement of dependence: "I need all the 'tcl_examples' to be
ready before I use them for anything"

-Brad
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-13 17:07-0500 Brad King wrote:


Alan W. Irwin wrote:

So just keeping narrowly focussed on that fragment there is only one "ALL"
custom target and ADD_DEPENDENCIES would not help since it only works on
targets.  Thus, I doubt there is anything locally wrong with dependencies
there.  It is possible some other dependency is making a dependency pattern
that triggers the bug, but I should know more about that when I have a
simpler example that triggers the bug (or not).


I was able to reproduce the problem with the code below.  It is fixed by
uncommenting the ADD_DEPENDENCIES line.  You must be putting those
output files into another target that does not depend on the
tcl_examples target.  Perhaps the make_documentation target?

-Brad


PROJECT(FOO)

FOREACH(f f1 f2 f3 f4 f5 f6 f7 f8 f9)
 LIST(APPEND DEPS ${FOO_BINARY_DIR}/${f})
 ADD_CUSTOM_COMMAND(OUTPUT ${FOO_BINARY_DIR}/${f}
   COMMAND echo ${f} > ${FOO_BINARY_DIR}/${f}
   )
ENDFOREACH(f)
ADD_CUSTOM_TARGET(examples ALL DEPENDS ${DEPS})
ADD_CUSTOM_TARGET(examples2 ALL DEPENDS ${DEPS})
#ADD_DEPENDENCIES(examples2 examples)


Good example, Brad!

Working from the PLplot case, I came up with another simple test case
(complete tarball attached including the required small files to be copied
for those who want to play with it).

In the PLplot case (and also the attached simple test case) there is
an additional custom command that file depends on the copied files.  In
addition there is a custom target that depends on the additional custom
command output file, and a custom target that depends on the copied files.

So the rule seems to be that parallel builds do not work if there are two
or more separate custom targets that file depend directly or indirectly (via
some custom command file dependency chain) on the same output files.

Another way of summarizing these results is that file depends must be
minimized and/or custom target depends maximized in order for parallel
builds to work properly.

My guess is I should be able to work around this CMake issue by appropriate
changes to the PLplot build system although I have a number of these
parallel build issues and the copy problem was only the most obvious.

I do regard this as a CMake issue.  Normally, the shoe is on the other foot,
and the build system developer is desperately trying to make sure that all
the CMake file depends are obviously in place locally rather than depending
on a long easily-broken chain of dependencies to do it for them in a
minimalist way.  So the big question is whether CMake can be modified so
that minimalist file depends and/or maximal (and unintuitive) target depends
are not required in order for parallel builds to work properly.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__

test_parallel.tar.gz
Description: complete CMake test case for bad parallel builds
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Brad King]

Alan W. Irwin wrote:
> So just keeping narrowly focussed on that fragment there is only one "ALL"
> custom target and ADD_DEPENDENCIES would not help since it only works on
> targets.  Thus, I doubt there is anything locally wrong with dependencies
> there.  It is possible some other dependency is making a dependency pattern
> that triggers the bug, but I should know more about that when I have a
> simpler example that triggers the bug (or not).

I was able to reproduce the problem with the code below.  It is fixed by
uncommenting the ADD_DEPENDENCIES line.  You must be putting those
output files into another target that does not depend on the
tcl_examples target.  Perhaps the make_documentation target?

-Brad


PROJECT(FOO)

FOREACH(f f1 f2 f3 f4 f5 f6 f7 f8 f9)
  LIST(APPEND DEPS ${FOO_BINARY_DIR}/${f})
  ADD_CUSTOM_COMMAND(OUTPUT ${FOO_BINARY_DIR}/${f}
COMMAND echo ${f} > ${FOO_BINARY_DIR}/${f}
)
ENDFOREACH(f)
ADD_CUSTOM_TARGET(examples ALL DEPENDS ${DEPS})
ADD_CUSTOM_TARGET(examples2 ALL DEPENDS ${DEPS})
#ADD_DEPENDENCIES(examples2 examples)
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Bryan O'Sullivan]

Alan W. Irwin wrote:

> It was good to hear that make -j N normally works with CMake.

Yes indeed.  I frequently run make -j70 across a 35-host dual-CPU
cluster using distcc, and every time I've updated CMake's files, it's
correctly rebuilt the makefiles before continuing.

http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-13 15:39-0500 Bill Hoffman wrote:


Alan W. Irwin wrote:

My obvious next step is to try and make a simple CMake example that 
reliably

reproduces the bug, but this is such an important bug (at least for those
with access to multiprocessors who want to use parallel builds) that I
thought the above result was worth reporting immediately since it tends to
point the finger at something CMake is doing rather than some bug in GNU
make.



We use -j N builds all the time at Kitware for VTK, ParaView and CMake.  It 
is however, possible to create input to CMake that will not work in a 
parallel environment.   A simple example would be the best way to figure out 
if there is a way around the issue you are having.  One thing you might want 
to look at is the add_dependancy command, and make sure that your custom 
targets are built in some order.  From your email, I am not exactly sure what 
targets are involved and what files are created at what time.


It was good to hear that make -j N normally works with CMake.

To answer your question, from the CMake language fragment in the first
e-mail on this issue, there is a custom command (with OUTPUT signature with
full pathname) for each file to be copied, and then an overall "ALL" custom
target that file depends (with full pathname) on those OUTPUT files.

So just keeping narrowly focussed on that fragment there is only one "ALL"
custom target and ADD_DEPENDENCIES would not help since it only works on
targets.  Thus, I doubt there is anything locally wrong with dependencies
there.  It is possible some other dependency is making a dependency pattern
that triggers the bug, but I should know more about that when I have a
simpler example that triggers the bug (or not).

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Bill Hoffman]

Alan W. Irwin wrote:

My obvious next step is to try and make a simple CMake example that 
reliably

reproduces the bug, but this is such an important bug (at least for those
with access to multiprocessors who want to use parallel builds) that I
thought the above result was worth reporting immediately since it tends to
point the finger at something CMake is doing rather than some bug in GNU
make.



We use -j N builds all the time at Kitware for VTK, ParaView and CMake. 
 It is however, possible to create input to CMake that will not work in 
a parallel environment.   A simple example would be the best way to 
figure out if there is a way around the issue you are having.  One thing 
you might want to look at is the add_dependancy command, and make sure 
that your custom targets are built in some order.  From your email, I am 
not exactly sure what targets are involved and what files are created at 
what time.


-Bill
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-12 17:10-0800 Alan W. Irwin wrote:


A set of custom rules to copy files from the source tree to the build tree
is screwing up for parallel builds on Debian testing with cmake 2.4.7.



Here is part of the "make -j 2" output:

make -f examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build.make 
examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build

[...]

make[2]: *** [examples/tcl/x05] Error 1
make[2]: Leaving directory `/home/software/plplot_cvs/HEAD/build_dir'
/usr/bin/cmake -E cmake_progress_report 
/home/software/plplot_cvs/HEAD/build_dir/CMakeFiles make[1]: *** 
[examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs


Note, the above make command is generated recursively from the overall
"make -j 2" command.  I have no idea how the -j option was propagated in
that case, but I assume it was via some Makefile variable.

Now here is the really strange part.  If I directly run

make -f examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build.make \
examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/clean
make -j 2 -f examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build.make \
examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build

from the command line there are no extra copying operations and no errors.
Also, all other attempts to reproduce this bug from a hand-crafted Makefile
have failed.
However, the double copy is reliably reproduced in the build tree by

make clean
make -j 2

and intermittently that above sequence produces other "*** Waiting for
unfinished jobs..." errors as well, but those are more complicated than
the simple double copy errors I have documented so I won't go into details.

In sum, so far it appears I need CMake-generated Makefiles that are
recursively run with "make -j 2" in order to see parallel build errors.
Thus, my working hypothesis is these bad parallel build results are due to
some CMake error (a Makefile variable that is set or propagated
incorrectly?) in the way it does make recursion.

My obvious next step is to try and make a simple CMake example that reliably
reproduces the bug, but this is such an important bug (at least for those
with access to multiprocessors who want to use parallel builds) that I
thought the above result was worth reporting immediately since it tends to
point the finger at something CMake is doing rather than some bug in GNU
make.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

On 2007-12-12 17:10-0800 Alan W. Irwin wrote:


A set of custom rules to copy files from the source tree to the build tree
is screwing up for parallel builds on Debian testing with cmake 2.4.7. The
parallel builds are done with "make -j 2" on a core duo system (Intel E6550
2.33 MHz).


Before anybody else spots that, I have to say that system sure would be
slow!  I meant GHz, of course.  :-)

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__

Linux-powered Science
__
___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

[CMake] Parallel builds do not work correctly when using "cmake -E copy" to copy files

[Alan W. Irwin]

A set of custom rules to copy files from the source tree to the build tree
is screwing up for parallel builds on Debian testing with cmake 2.4.7. The
parallel builds are done with "make -j 2" on a core duo system (Intel E6550
2.33 MHz).  I am not sure whether the issue is something wrong with our
CMake code, a bug in CMake, or a bug in GNU make, and I would appreciate
your comments to help sort out the possibilities.

Here is the cmake stanza to set up the custom rules and overall custom
target that depends on those rules:

# Copy file and scripts to the binary directory if different to the 
# source directory. Needed for ctest, but also so the tclIndex file

# is generated in the binary tree not the source tree.
if(NOT CMAKE_CURRENT_BINARY_DIR STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
  set(tclIndex_WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
  set(tclIndex_DEPENDS)
  foreach(file ${tcl_SCRIPTS} ${tcl_FILES})
set(
tclIndex_DEPENDS
${tclIndex_DEPENDS}
${CMAKE_CURRENT_BINARY_DIR}/${file}
)
add_custom_command(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${file}
COMMAND ${CMAKE_COMMAND} -E copy
${CMAKE_CURRENT_SOURCE_DIR}/${file} ${CMAKE_CURRENT_BINARY_DIR}
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${file}
)
  endforeach(file ${tcl_SCRIPTS} ${tcl_FILES})
  add_custom_target(tcl_examples ALL DEPENDS ${tclIndex_DEPENDS})
else(NOT CMAKE_CURRENT_BINARY_DIR STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
  set(tclIndex_DEPENDS ${tcl_FILES} ${tcl_SCRIPTS})
  set(tclIndex_WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
endif(NOT CMAKE_CURRENT_BINARY_DIR STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")

For the case of separate build tree, the resulting Makefile stanzas look
like the following :

examples/tcl/CMakeFiles/tclIndex_examples_tcl: examples/tcl/x01
examples/tcl/CMakeFiles/tclIndex_examples_tcl: examples/tcl/x02
...etc.

examples/tcl/x01: /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x01
$(CMAKE_COMMAND) -E cmake_progress_report 
/home/software/plplot_cvs/HEAD/build_dir/CMakeFiles $(CMAKE_PROGRESS_1)
@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold 
"Generating x01"
cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && 
/usr/bin/cmake -E copy /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x01 
/home/software/plplot_cvs/HEAD/build_dir/examples/tcl

examples/tcl/x02: /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x02
$(CMAKE_COMMAND) -E cmake_progress_report 
/home/software/plplot_cvs/HEAD/build_dir/CMakeFiles $(CMAKE_PROGRESS_1)
@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold 
"Generating x02"
cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && 
/usr/bin/cmake -E copy /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x02 
/home/software/plplot_cvs/HEAD/build_dir/examples/tcl
...etc.

This works fine for serial builds (make -j 1), but for parallel builds 
(make -j 2) there is some problem with the interpretation of the

dependencies which means the copy commands are sometimes duplicated.

x01 and x02 were copied correctly, but the x03, x04, and x05 files
were copied twice with the latter double copy causing a severe error.

Here is part of the "make -j 2" output:

make -f examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build.make 
examples/tcl/CMakeFiles/tclIndex_examples_tcl.dir/build
make[2]: Entering directory `/home/software/plplot_cvs/HEAD/build_dir'
/usr/bin/cmake -E cmake_progress_report /home/software/plplot_cvs/HEAD/build_dir/CMakeFiles 
[ 22%] Generating x01

cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && /usr/bin/cmake -E 
copy /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x01 
/home/software/plplot_cvs/HEAD/build_dir/examples/tcl
/usr/bin/cmake -E cmake_progress_report /home/software/plplot_cvs/HEAD/build_dir/CMakeFiles 
[ 22%] make[2]: Leaving directory `/home/software/plplot_cvs/HEAD/build_dir'

/usr/bin/cmake -E cmake_progress_report 
/home/software/plplot_cvs/HEAD/build_dir/CMakeFiles  18
Generating x02
cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && /usr/bin/cmake -E 
copy /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x02 
/home/software/plplot_cvs/HEAD/build_dir/examples/tcl
[ 23%] Built target make_documentation
make -f examples/tcl/CMakeFiles/tcl_examples.dir/build.make 
examples/tcl/CMakeFiles/tcl_examples.dir/build
/usr/bin/cmake -E cmake_progress_report /home/software/plplot_cvs/HEAD/build_dir/CMakeFiles 
make[2]: Entering directory `/home/software/plplot_cvs/HEAD/build_dir'
/usr/bin/cmake -E cmake_progress_report /home/software/plplot_cvs/HEAD/build_dir/CMakeFiles 
[ 23%] [ 23%] Generating x03

cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && /usr/bin/cmake -E 
copy /home/software/plplot_cvs/HEAD/plplot_cmake/examples/tcl/x03 
/home/software/plplot_cvs/HEAD/build_dir/examples/tcl
Generating x03
cd /home/software/plplot_cvs/HEAD/build_dir/examples/tcl && /usr/bin/cmake -E 
c

Re: [CMake] parallel builds

[Bill Hoffman]

Darby J Van Uitert wrote:

Hello,

How can I have the build portion of running ctest -D Experimental use 
the -j option and do a parallel build? I normally do a make -j4 and 
want to do something similar when building an Experimental (or Nightly 
or Continuous build for that matter). Thanks. Sorry if this is obvious.

Set the following cache variable for the project:

MAKECOMMAND:STRING=gmake -i -j4

-Bill



___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake

[CMake] parallel builds

[Darby J Van Uitert]

Hello,

How can I have the build portion of running ctest -D Experimental use 
the -j option and do a parallel build? I normally do a make -j4 and want 
to do something similar when building an Experimental (or Nightly or 
Continuous build for that matter). Thanks. Sorry if this is obvious.


  -darby j

--
Darby J Van Uitert
SCI Institute
University of Utah
Email: [EMAIL PROTECTED]
Phone: (301) 528-8469


___
CMake mailing list
CMake@cmake.org
http://www.cmake.org/mailman/listinfo/cmake