Re: [CMake] How to get custom commands to build in parallel ?
/MP flag is specifically for compiling object files in parallel so it
won't cause multiple custom commands to be run in parallel as each one
is a msbuild custom build item task. I don't know exactly how msbuild
treats custom tasks such as these, but they might be implicitly
serial.
On Wed, Sep 24, 2014 at 3:27 AM, Glenn Coombs wrote:
> I am already using /MP and the equivalent to the /maxcpucount. It is the
> /MP one that is important in this context as I am only concerned with
> improving the speed of a single project out of the many in my solution. I
> believe the problem is with cmake rather than with Visual Studio as I can
> select any of the other projects in the same solution and build them in
> isolation and they will build their object files in parallel. For some
> reason cmake isn't seeing the custom commands as something it can run in
> parallel.
>
> --
> Glenn
>
> On 23 September 2014 16:20, Robert Maynard
> wrote:
>>
>> If you are using VS2013 you should look at the /maxcpucount flag which
>> allows msbuild to build multiple projects at the same time. You will
>> have to manually balance /MP and /maxcpucount as they cause a P*C
>> number of processes to execute.
>>
>> On Tue, Sep 23, 2014 at 11:05 AM, Glenn Coombs
>> wrote:
>> > I have the following code in one of my CMakeLists.txt files:
>> >
>> > set(feature_files_h)
>> > set(feature_files_cpp)
>> >
>> > macro(create_feature_files NAME)
>> > add_custom_command(
>> > OUTPUT ${FEATURES_OUTPUT_DIR}/${NAME}.cpp
>> > ${FEATURES_OUTPUT_DIR}/${NAME}.h
>> > DEPENDS ${FEATURES_INPUT_DIR}/${NAME}.txt
>> > WORKING_DIRECTORY ${SIM_ROOT}/tools/features
>> > COMMAND perl ${FEATURES_PERL_PATH} "-s${FEATURES_INPUT_DIR}"
>> > "-d${FEATURES_OUTPUT_DIR}" "-f${NAME}"
>> > )
>> > list(APPEND feature_files_h ${FEATURES_OUTPUT_DIR}/${NAME}.h)
>> > list(APPEND feature_files_cpp ${FEATURES_OUTPUT_DIR}/${NAME}.cpp)
>> > endmacro()
>> >
>> > create_feature_files(example_features)
>> > create_feature_files(fme_core_features)
>> > create_feature_files(fme_features)
>> > create_feature_files(front_end_features)
>> > create_feature_files(inloop_filter_features)
>> >
>> > add_library(${PROJECT_NAME} STATIC ${feature_files_cpp}
>> > ${feature_files_h}
>> > ${SIM_ROOT}/tools/features/cpp-code/cfeatures.cpp)
>> >
>> > which creates a library from an existing source file called
>> > cfeatures.cpp
>> > and from a bunch of auto-generated .cpp and .h files. The whole thing
>> > works
>> > fine and runs the perl script to create each of the auto-generated files
>> > before creating the library. The only problem I have with it is that it
>> > is
>> > quite slow and I would like to speed it up by having it run the perl
>> > commands to create the auto-generated files in parallel. Currently,
>> > each of
>> > the auto-generated cpp files is generated sequentially, then they are
>> > all
>> > compiled in parallel and the library created in the final step.
>> >
>> > I'm using cmake 3.0.0 with Visual Studio 2013 Express Edition. And I've
>> > added the /MP option so that Visual Studio will do parallel builds.
>> > I've
>> > also set the "Projects and Solutions=>Build and Run=>maximum number of
>> > parallel project builds" option to 4. This works well with all the
>> > normal
>> > cpp files in other projects and I easily see all 4 cpu cores maxxed out.
>> > But when building this project only 1 cpu core is used.
>> >
>> > Can anybody tell me how to change it so that the auto-geneated files are
>> > created in parallel as well ?
>> >
>> > --
>> > Glenn
>> >
>> >
>> > --
>> >
>> > Powered by www.kitware.com
>> >
>> > Please keep messages on-topic and check the CMake FAQ at:
>> > http://www.cmake.org/Wiki/CMake_FAQ
>> >
>> > Kitware offers various services to support the CMake community. For more
>> > information on each offering, please visit:
>> >
>> > CMake Support: http://cmake.org/cmake/help/support.html
>> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> > CMake Training Courses: http://cmake.org/cmake/help/training.html
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://public.kitware.com/mailman/listinfo/cmake
>
>
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] How to get custom commands to build in parallel ?
I am already using /MP and the equivalent to the /maxcpucount. It is the
/MP one that is important in this context as I am only concerned with
improving the speed of a single project out of the many in my solution. I
believe the problem is with cmake rather than with Visual Studio as I can
select any of the other projects in the same solution and build them in
isolation and they will build their object files in parallel. For some
reason cmake isn't seeing the custom commands as something it can run in
parallel.
--
Glenn
On 23 September 2014 16:20, Robert Maynard
wrote:
> If you are using VS2013 you should look at the /maxcpucount flag which
> allows msbuild to build multiple projects at the same time. You will
> have to manually balance /MP and /maxcpucount as they cause a P*C
> number of processes to execute.
>
> On Tue, Sep 23, 2014 at 11:05 AM, Glenn Coombs
> wrote:
> > I have the following code in one of my CMakeLists.txt files:
> >
> > set(feature_files_h)
> > set(feature_files_cpp)
> >
> > macro(create_feature_files NAME)
> > add_custom_command(
> > OUTPUT ${FEATURES_OUTPUT_DIR}/${NAME}.cpp
> > ${FEATURES_OUTPUT_DIR}/${NAME}.h
> > DEPENDS ${FEATURES_INPUT_DIR}/${NAME}.txt
> > WORKING_DIRECTORY ${SIM_ROOT}/tools/features
> > COMMAND perl ${FEATURES_PERL_PATH} "-s${FEATURES_INPUT_DIR}"
> > "-d${FEATURES_OUTPUT_DIR}" "-f${NAME}"
> > )
> > list(APPEND feature_files_h ${FEATURES_OUTPUT_DIR}/${NAME}.h)
> > list(APPEND feature_files_cpp ${FEATURES_OUTPUT_DIR}/${NAME}.cpp)
> > endmacro()
> >
> > create_feature_files(example_features)
> > create_feature_files(fme_core_features)
> > create_feature_files(fme_features)
> > create_feature_files(front_end_features)
> > create_feature_files(inloop_filter_features)
> >
> > add_library(${PROJECT_NAME} STATIC ${feature_files_cpp}
> ${feature_files_h}
> > ${SIM_ROOT}/tools/features/cpp-code/cfeatures.cpp)
> >
> > which creates a library from an existing source file called cfeatures.cpp
> > and from a bunch of auto-generated .cpp and .h files. The whole thing
> works
> > fine and runs the perl script to create each of the auto-generated files
> > before creating the library. The only problem I have with it is that it
> is
> > quite slow and I would like to speed it up by having it run the perl
> > commands to create the auto-generated files in parallel. Currently,
> each of
> > the auto-generated cpp files is generated sequentially, then they are all
> > compiled in parallel and the library created in the final step.
> >
> > I'm using cmake 3.0.0 with Visual Studio 2013 Express Edition. And I've
> > added the /MP option so that Visual Studio will do parallel builds. I've
> > also set the "Projects and Solutions=>Build and Run=>maximum number of
> > parallel project builds" option to 4. This works well with all the
> normal
> > cpp files in other projects and I easily see all 4 cpu cores maxxed out.
> > But when building this project only 1 cpu core is used.
> >
> > Can anybody tell me how to change it so that the auto-geneated files are
> > created in parallel as well ?
> >
> > --
> > Glenn
> >
> >
> > --
> >
> > Powered by www.kitware.com
> >
> > Please keep messages on-topic and check the CMake FAQ at:
> > http://www.cmake.org/Wiki/CMake_FAQ
> >
> > Kitware offers various services to support the CMake community. For more
> > information on each offering, please visit:
> >
> > CMake Support: http://cmake.org/cmake/help/support.html
> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
> > CMake Training Courses: http://cmake.org/cmake/help/training.html
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Follow this link to subscribe/unsubscribe:
> > http://public.kitware.com/mailman/listinfo/cmake
>
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Re: [CMake] How to get custom commands to build in parallel ?
If you are using VS2013 you should look at the /maxcpucount flag which
allows msbuild to build multiple projects at the same time. You will
have to manually balance /MP and /maxcpucount as they cause a P*C
number of processes to execute.
On Tue, Sep 23, 2014 at 11:05 AM, Glenn Coombs wrote:
> I have the following code in one of my CMakeLists.txt files:
>
> set(feature_files_h)
> set(feature_files_cpp)
>
> macro(create_feature_files NAME)
> add_custom_command(
> OUTPUT ${FEATURES_OUTPUT_DIR}/${NAME}.cpp
> ${FEATURES_OUTPUT_DIR}/${NAME}.h
> DEPENDS ${FEATURES_INPUT_DIR}/${NAME}.txt
> WORKING_DIRECTORY ${SIM_ROOT}/tools/features
> COMMAND perl ${FEATURES_PERL_PATH} "-s${FEATURES_INPUT_DIR}"
> "-d${FEATURES_OUTPUT_DIR}" "-f${NAME}"
> )
> list(APPEND feature_files_h ${FEATURES_OUTPUT_DIR}/${NAME}.h)
> list(APPEND feature_files_cpp ${FEATURES_OUTPUT_DIR}/${NAME}.cpp)
> endmacro()
>
> create_feature_files(example_features)
> create_feature_files(fme_core_features)
> create_feature_files(fme_features)
> create_feature_files(front_end_features)
> create_feature_files(inloop_filter_features)
>
> add_library(${PROJECT_NAME} STATIC ${feature_files_cpp} ${feature_files_h}
> ${SIM_ROOT}/tools/features/cpp-code/cfeatures.cpp)
>
> which creates a library from an existing source file called cfeatures.cpp
> and from a bunch of auto-generated .cpp and .h files. The whole thing works
> fine and runs the perl script to create each of the auto-generated files
> before creating the library. The only problem I have with it is that it is
> quite slow and I would like to speed it up by having it run the perl
> commands to create the auto-generated files in parallel. Currently, each of
> the auto-generated cpp files is generated sequentially, then they are all
> compiled in parallel and the library created in the final step.
>
> I'm using cmake 3.0.0 with Visual Studio 2013 Express Edition. And I've
> added the /MP option so that Visual Studio will do parallel builds. I've
> also set the "Projects and Solutions=>Build and Run=>maximum number of
> parallel project builds" option to 4. This works well with all the normal
> cpp files in other projects and I easily see all 4 cpu cores maxxed out.
> But when building this project only 1 cpu core is used.
>
> Can anybody tell me how to change it so that the auto-geneated files are
> created in parallel as well ?
>
> --
> Glenn
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
[CMake] How to get custom commands to build in parallel ?
I have the following code in one of my CMakeLists.txt files:
set(feature_files_h)
set(feature_files_cpp)
macro(create_feature_files NAME)
add_custom_command(
OUTPUT ${FEATURES_OUTPUT_DIR}/${NAME}.cpp
${FEATURES_OUTPUT_DIR}/${NAME}.h
DEPENDS ${FEATURES_INPUT_DIR}/${NAME}.txt
WORKING_DIRECTORY ${SIM_ROOT}/tools/features
COMMAND perl ${FEATURES_PERL_PATH} "-s${FEATURES_INPUT_DIR}"
"-d${FEATURES_OUTPUT_DIR}" "-f${NAME}"
)
list(APPEND feature_files_h ${FEATURES_OUTPUT_DIR}/${NAME}.h)
list(APPEND feature_files_cpp ${FEATURES_OUTPUT_DIR}/${NAME}.cpp)
endmacro()
create_feature_files(example_features)
create_feature_files(fme_core_features)
create_feature_files(fme_features)
create_feature_files(front_end_features)
create_feature_files(inloop_filter_features)
add_library(${PROJECT_NAME} STATIC ${feature_files_cpp} ${feature_files_h}
${SIM_ROOT}/tools/features/cpp-code/cfeatures.cpp)
which creates a library from an existing source file called cfeatures.cpp
and from a bunch of auto-generated .cpp and .h files. The whole thing
works fine and runs the perl script to create each of the auto-generated
files before creating the library. The only problem I have with it is that
it is quite slow and I would like to speed it up by having it run the perl
commands to create the auto-generated files in parallel. Currently, each
of the auto-generated cpp files is generated sequentially, then they are
all compiled in parallel and the library created in the final step.
I'm using cmake 3.0.0 with Visual Studio 2013 Express Edition. And I've
added the /MP option so that Visual Studio will do parallel builds. I've
also set the "Projects and Solutions=>Build and Run=>maximum number of
parallel project builds" option to 4. This works well with all the normal
cpp files in other projects and I easily see all 4 cpu cores maxxed out.
But when building this project only 1 cpu core is used.
Can anybody tell me how to change it so that the auto-geneated files are
created in parallel as well ?
--
Glenn
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
