Re: Parallel cmake (diff to review)
Hey, dude- Our make passes -j through MAKEFLAGS. The way it handles recursive invocation is documented. Read the end of make(1): Thanks for the cluestick! I apologize for not seeing that paragraph earlier. I'm corresponding with Amit privately to try and track down the root cause. Best, Kent
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
I think we now have the critical path to libreoffice building quickly enough
that it's not going to reduce the overall dpb build time (the problem
DPB_PROPERTIES=parallel intends to solve is waiting for libreoffice to finish
when everything else is done. Moving to gmake may actually increase the overall
time (gmake uses crazy amounts of cpu at times).
David Coppa dco...@gmail.com wrote:
On Thu, Dec 13, 2012 at 7:01 AM, Vadim Zhukov persg...@gmail.com
wrote:
13.12.2012 4:22 пользователь Amit Kulkarni amitk...@gmail.com
написал:
I thought you guys were talking about building cmake proper in
parallel.
We did. cmake proper first builds a minimal bootstrap cmake,
then
rebuilds itself with it, so getting cmake proper to build in
parallel
*is* the same problem as getting any other cmake-using port to
build
in parallel.
Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building
without
gmake.
The problem isn't that make -j fails with cmake. The build
succeeds
just fine. The problem is that with our make there is no
parallelism.
It's as if the -j was ignored.
It's likely that cmake decides (arbitrarily) things don't work
without
gmake.
Since there is some recursive makefiles involved, it probably
strips the
extra stuff early on...
i could not see any gmake specific code when i grepped in the cmake
codebase.
i can confirm what naddy@ sees, when i cd ${WRKBUILD} make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are
used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.
The problem was already made clear: GNU Make propagates -j to
subcalls,
our - does not. The latter is by design, IIRC (if I'm wrong here,
then,
probably, espie@ will use his cluestick to teach me not to write
about
things I understand badly), to avoid extra subprocesses being run:
suppose
that make -j 4 runs four make -j 3, then each runs three make -j
2...
That's GNU Make's way, IIRC, and it's broken by design.
Unfortunately, three
is no easy way to fix this. The best option I see (and it's probably
wrong)
is to create socket in /tmp on the initial make(1) invocation, and
pass its
path to subprocesses through environment variable. Through this
socket, each
sub-make could request the right to start one or more jobs, and wait
until
master make process answers. But I'm not ready to prepare any
patches
implementing such functionality now: too much time is being spent on
fixing
KDE breakage (due to upstream lazyness, my own stupidity and a lots
of
inaccuracy from both sides).
Btw, I still think it's not so bad to use gmake *only* for cmake (not
for all the cmake-based ports!) to speed up dpb builds...
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 06:21:43PM -0600, Amit Kulkarni wrote:
I thought you guys were talking about building cmake proper in parallel.
We did. cmake proper first builds a minimal bootstrap cmake, then
rebuilds itself with it, so getting cmake proper to build in parallel
*is* the same problem as getting any other cmake-using port to build
in parallel.
Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building without
gmake.
The problem isn't that make -j fails with cmake. The build succeeds
just fine. The problem is that with our make there is no parallelism.
It's as if the -j was ignored.
It's likely that cmake decides (arbitrarily) things don't work without
gmake.
Since there is some recursive makefiles involved, it probably strips the
extra stuff early on...
i could not see any gmake specific code when i grepped in the cmake codebase.
i can confirm what naddy@ sees, when i cd ${WRKBUILD} make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.
No, our make is not ignoring -j, and it's passed to submakes. Using standard
posix mechanisms. That is, it's passed through the environment, using
MAKEFLAGS.
There are two ways to defeat that mechanism: either by explicitly wiping
out the environment, or by passing another -j somewhere.
cmake is probably doing either of those.
If you think make does not pass -j to submakes by default, you're confused,
again, or you explain yourself poorly.
Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 09:32:23AM -0600, Kent R. Spillner wrote: Hey, dude- On Dec 12, 2012, at 7:51, David Coppa dco...@gmail.com wrote: Thanks. I'll wait for useful pointers... I don't think CMake does anything specifically to handle parallel recursive builds. It works with GNU make because when invoked with -jX the top level gmake sets some environment variables (it adds -j to MAKEFLAGS, but i think it uses some others for job control, too) so sub-makes pick that up in parallel builds and CMake doesn't need to explicitly write -j$((X - 1)) in targets invoking sub-makes. How does our make handle parallel job control between recursive invocations? Our make passes -j through MAKEFLAGS. The way it handles recursive invocation is documented. Read the end of make(1): In parallel mode, -j n only limits the number of direct children of make. During recursive invocations, each level may multiply the total number of processes by n. However, make includes some heuristics to try to prevent catastrophic behavior: if a command is marked as expensive, or preceded by `+', or seems to invoke a program that looks sufficiently like `make', make will assume recursive invocation, and not start any new process until said command has finished running. Thus the number of processes run directly or indirectly by make will increase linearly with each level of recursion instead of exponentially.
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
No, our make is not ignoring -j, and it's passed to submakes. Using standard posix mechanisms. That is, it's passed through the environment, using MAKEFLAGS. There are two ways to defeat that mechanism: either by explicitly wiping out the environment, or by passing another -j somewhere. cmake is probably doing either of those. If you think make does not pass -j to submakes by default, you're confused, again, or you explain yourself poorly. cmake is destroying the env by setting MAKEFLAGS=, that is what i saw in git master yesterday...
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Thu, Dec 13, 2012 at 3:39 PM, Amit Kulkarni amitk...@gmail.com wrote: No, our make is not ignoring -j, and it's passed to submakes. Using standard posix mechanisms. That is, it's passed through the environment, using MAKEFLAGS. There are two ways to defeat that mechanism: either by explicitly wiping out the environment, or by passing another -j somewhere. cmake is probably doing either of those. If you think make does not pass -j to submakes by default, you're confused, again, or you explain yourself poorly. cmake is destroying the env by setting MAKEFLAGS=, that is what i saw in git master yesterday... Already tried with removing that chunk: it does not solve the problem :(
Re: Parallel cmake (diff to review)
On Tue, 11 Dec 2012, Amit Kulkarni wrote:
I tried with USE_GMAKE = Yes and it drives all cores
so I think please add CONFIGURE_ARGS = --parallel=${MAKE_JOBS} and
USE_GMAKE = Yes and then both configure and build will go parallel.
...
The use of GNU make like this seems bogus. I take it CMake
isn't doing something right.
as far back as 2008 I seem to remember cmake having implicit
dependency on gmake (i tried it on on Solaris,FreeBSD and then
OpenBSD). my builds would always parallel build with gmake but not
with plain os make. i still haven't figured out why. maybe david can
chime in with more...
perhaps
http://cmake.3232098.n2.nabble.com/parallel-make-j-N-td7193296.html
which leads to
http://public.kitware.com/Bug/view.php?id=12882
Hi again,
Since I've tried all the weapons in my arsenal and found nothing
useful, I think I can be ok with Amit's approach.
I did the usual tests and it doesn't seem to introduce any regressions
after all...
Summarizing:
I've added --parallel=${MAKE_JOBS}.
Added --verbose that prints some nice informations, like
System: OpenBSD
Doing parallel make: 2
Removed USE_GROFF which is not needed.
Added a XXX to the comment.
Thoughts? OKs?
Index: Makefile
===
RCS file: /cvs/ports/devel/cmake/Makefile,v
retrieving revision 1.73
diff -u -p -r1.73 Makefile
--- Makefile11 Dec 2012 11:51:39 - 1.73
+++ Makefile12 Dec 2012 11:42:10 -
@@ -8,7 +8,7 @@ HOMEPAGE = http://www.cmake.org/
CATEGORIES = devel
COMMENT = portable build system
DISTNAME = cmake-2.8.10.2
-REVISION = 0
+REVISION = 1
MASTER_SITES = ${HOMEPAGE}files/v2.8/
MAINTAINER = David Coppa dco...@openbsd.org
@@ -29,12 +29,16 @@ CONFIGURE_STYLE = simple
CONFIGURE_ARGS = --prefix=${PREFIX} \
--datadir=/share/cmake \
--docdir=/share/doc/cmake \
- --system-libs
+ --parallel=${MAKE_JOBS} \
+ --system-libs \
+ --verbose
PORTHOME = ${WRKDIR}
SEPARATE_BUILD = Yes
-USE_GROFF =Yes
+
+# XXX: GNU make enables dpb parallel builds
+USE_GMAKE =Yes
REGRESS_TARGET = test
Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 12:48:09PM +0100, David Coppa wrote:
On Tue, 11 Dec 2012, Amit Kulkarni wrote:
I tried with USE_GMAKE = Yes and it drives all cores
so I think please add CONFIGURE_ARGS = --parallel=${MAKE_JOBS} and
USE_GMAKE = Yes and then both configure and build will go parallel.
...
The use of GNU make like this seems bogus. I take it CMake
isn't doing something right.
as far back as 2008 I seem to remember cmake having implicit
dependency on gmake (i tried it on on Solaris,FreeBSD and then
OpenBSD). my builds would always parallel build with gmake but not
with plain os make. i still haven't figured out why. maybe david can
chime in with more...
perhaps
http://cmake.3232098.n2.nabble.com/parallel-make-j-N-td7193296.html
which leads to
http://public.kitware.com/Bug/view.php?id=12882
Hi again,
Since I've tried all the weapons in my arsenal and found nothing
useful, I think I can be ok with Amit's approach.
I did the usual tests and it doesn't seem to introduce any regressions
after all...
Summarizing:
I've added --parallel=${MAKE_JOBS}.
Added --verbose that prints some nice informations, like
System: OpenBSD
Doing parallel make: 2
Removed USE_GROFF which is not needed.
Added a XXX to the comment.
Thoughts? OKs?
USE_GMAKE is _not_ the way to go. What next, add USE_GMAKE to all ports
using cmake ? Our make -j itself works, one just need to find why cmake
doesnt take benefit from it.
Landry
Re: Parallel cmake (diff to review)
12.12.2012 15:55 пользователь David Coppa dco...@gmail.com написал:
On Tue, 11 Dec 2012, Amit Kulkarni wrote:
I tried with USE_GMAKE = Yes and it drives all cores
so I think please add CONFIGURE_ARGS = --parallel=${MAKE_JOBS}
and
USE_GMAKE = Yes and then both configure and build will go
parallel.
...
The use of GNU make like this seems bogus. I take it CMake
isn't doing something right.
as far back as 2008 I seem to remember cmake having implicit
dependency on gmake (i tried it on on Solaris,FreeBSD and then
OpenBSD). my builds would always parallel build with gmake but not
with plain os make. i still haven't figured out why. maybe david can
chime in with more...
perhaps
http://cmake.3232098.n2.nabble.com/parallel-make-j-N-td7193296.html
which leads to
http://public.kitware.com/Bug/view.php?id=12882
Hi again,
Since I've tried all the weapons in my arsenal and found nothing
useful, I think I can be ok with Amit's approach.
I did the usual tests and it doesn't seem to introduce any regressions
after all...
Summarizing:
I've added --parallel=${MAKE_JOBS}.
Added --verbose that prints some nice informations, like
System: OpenBSD
Doing parallel make: 2
Removed USE_GROFF which is not needed.
Added a XXX to the comment.
Thoughts? OKs?
Until this go in (if ever), could please anyone make clear a few things?
1. USE_GMAKE=Yes in CMake's port enables parallel build of CMake itself
only, or does it equate to setting USE_GMAKE=Yes to all CMake-based ports
too? IIRC, CMake does not care, thus we can use GMake to build CMake itself
and our make(1) for building CMake-based ports, but I could be wrong.
2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
to achieve parallel building (at least until problem with our make and
CMake is fixed)?
Index: Makefile
===
RCS file: /cvs/ports/devel/cmake/Makefile,v
retrieving revision 1.73
diff -u -p -r1.73 Makefile
--- Makefile11 Dec 2012 11:51:39 - 1.73
+++ Makefile12 Dec 2012 11:42:10 -
@@ -8,7 +8,7 @@ HOMEPAGE = http://www.cmake.org/
CATEGORIES = devel
COMMENT = portable build system
DISTNAME = cmake-2.8.10.2
-REVISION = 0
+REVISION = 1
MASTER_SITES = ${HOMEPAGE}files/v2.8/
MAINTAINER = David Coppa dco...@openbsd.org
@@ -29,12 +29,16 @@ CONFIGURE_STYLE = simple
CONFIGURE_ARGS = --prefix=${PREFIX} \
--datadir=/share/cmake \
--docdir=/share/doc/cmake \
- --system-libs
+ --parallel=${MAKE_JOBS} \
+ --system-libs \
+ --verbose
PORTHOME = ${WRKDIR}
SEPARATE_BUILD = Yes
-USE_GROFF =Yes
+
+# XXX: GNU make enables dpb parallel builds
+USE_GMAKE =Yes
REGRESS_TARGET = test
Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 1:09 PM, Vadim Zhukov persg...@gmail.com wrote: Until this go in (if ever), could please anyone make clear a few things? 1. USE_GMAKE=Yes in CMake's port enables parallel build of CMake itself only, or does it equate to setting USE_GMAKE=Yes to all CMake-based ports too? IIRC, CMake does not care, thus we can use GMake to build CMake itself and our make(1) for building CMake-based ports, but I could be wrong. CMake itself only. No need to set USE_GMAKE=Yes for CMake-based ports. 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want to achieve parallel building (at least until problem with our make and CMake is fixed)? I don't know but I suspect the answer is yes.
Re: Parallel cmake (diff to review)
On 2012/12/12 15:09, Vadim Zhukov wrote: 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want to achieve parallel building (at least until problem with our make and CMake is fixed)? in testing, it seems so. weird though, I was pretty sure I'd seen an improvement from doing this in the past, so either I was mistaken (not unlikely ;) or something changed.
Re: Parallel cmake (diff to review)
2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want to achieve parallel building (at least until problem with our make and CMake is fixed)? in testing, it seems so. weird though, I was pretty sure I'd seen an improvement from doing this in the past, so either I was mistaken (not unlikely ;) or something changed. i asked cmake-develop...@cmake.org for help in figuring out which area of code to look at.
Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 2:44 PM, Amit Kulkarni amitk...@gmail.com wrote: 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want to achieve parallel building (at least until problem with our make and CMake is fixed)? in testing, it seems so. weird though, I was pretty sure I'd seen an improvement from doing this in the past, so either I was mistaken (not unlikely ;) or something changed. i asked cmake-develop...@cmake.org for help in figuring out which area of code to look at. Thanks. I'll wait for useful pointers...
using cmake in parallel builds, Re: Parallel cmake (diff to review)
Very bad subject line, I almost did not read the thread, because I thought you guys were talking about building cmake proper in parallel. Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. I haven't looked too closely, I have NO IDEA which of the two issues we're talking about (besides that, it might be a good idea to have a generator that caters a bit more to our make, specifically, include subfile is now part of posix. As far as make -j is concerned, the main issue left is matching file paths to targets. So if you see make saying hey, I don't know how to build that or target foo unaccounted for, that's probably it.
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Wed, 12 Dec 2012, Marc Espie wrote: Very bad subject line, I almost did not read the thread, because I thought you guys were talking about building cmake proper in parallel. Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. I think this is the issue (cmake not writing correct makefiles for parallel building) I haven't looked too closely, I have NO IDEA which of the two issues we're talking about (besides that, it might be a good idea to have a generator that caters a bit more to our make, specifically, include subfile is now part of posix. As far as make -j is concerned, the main issue left is matching file paths to targets. So if you see make saying hey, I don't know how to build that or target foo unaccounted for, that's probably it. Attached you have one of the generated Makefiles (/usr/ports/pobj/cmake-2.8.10.2/build-amd64/Makefile) cheers, David # CMAKE generated file: DO NOT EDIT! # Generated by Unix Makefiles Generator, CMake Version 2.8 # Default target executed when no arguments are given to make. default_target: all .PHONY : default_target #= # Special targets provided by cmake. # Disable implicit rules so canonical targets will work. .SUFFIXES: # Remove some rules from gmake that .SUFFIXES does not remove. SUFFIXES = .SUFFIXES: .hpux_make_needs_suffix_list # Produce verbose output by default. VERBOSE = 1 # Suppress display of executed commands. $(VERBOSE).SILENT: # A target that is always out of date. cmake_force: .PHONY : cmake_force #= # Set environment variables for the build. # The shell in which to execute make rules. SHELL = /bin/sh # The CMake executable. CMAKE_COMMAND = /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64/Bootstrap.cmk/cmake # The command to remove a file. RM = /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64/Bootstrap.cmk/cmake -E remove -f # Escaping for special characters. EQUALS = = # The top-level source directory on which CMake was run. CMAKE_SOURCE_DIR = /home/dcoppa/ports/pobj/cmake-2.8.10.2/cmake-2.8.10.2 # The top-level build directory on which CMake was run. CMAKE_BINARY_DIR = /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64 #= # Targets provided globally by CMake. # Special rule for the target edit_cache edit_cache: @echo Running interactive CMake command-line interface... Bootstrap.cmk/cmake -i . .PHONY : edit_cache # Special rule for the target edit_cache edit_cache/fast: edit_cache .PHONY : edit_cache/fast # Special rule for the target install install: preinstall @echo Install the project... bin/cmake -P cmake_install.cmake .PHONY : install # Special rule for the target install install/fast: preinstall/fast @echo Install the project... bin/cmake -P cmake_install.cmake .PHONY : install/fast # Special rule for the target install/local install/local: preinstall @echo Installing only the local directory... bin/cmake -DCMAKE_INSTALL_LOCAL_ONLY=1 -P cmake_install.cmake .PHONY : install/local # Special rule for the target install/local install/local/fast: install/local .PHONY : install/local/fast # Special rule for the target install/strip install/strip: preinstall @echo Installing the project stripped... bin/cmake -DCMAKE_INSTALL_DO_STRIP=1 -P cmake_install.cmake .PHONY : install/strip # Special rule for the target install/strip install/strip/fast: install/strip .PHONY : install/strip/fast # Special rule for the target list_install_components list_install_components: @echo Available install components are: \Unspecified\ .PHONY : list_install_components # Special rule for the target list_install_components list_install_components/fast: list_install_components .PHONY : list_install_components/fast # Special rule for the target package package: preinstall @echo Run CPack packaging tool... /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64/bin/cpack --config ./CPackConfig.cmake .PHONY : package # Special rule for the target package package/fast: package .PHONY : package/fast # Special rule for the target package_source package_source: @echo Run CPack packaging tool for source... /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64/bin/cpack --config ./CPackSourceConfig.cmake /home/dcoppa/ports/pobj/cmake-2.8.10.2/build-amd64/CPackSourceConfig.cmake .PHONY : package_source # Special rule for the target package_source package_source/fast: package_source .PHONY : package_source/fast # Special rule for the target rebuild_cache rebuild_cache: @echo Running CMake to regenerate build system... Bootstrap.cmk/cmake -H$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) .PHONY :
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
Marc Espie: I thought you guys were talking about building cmake proper in parallel. We did. cmake proper first builds a minimal bootstrap cmake, then rebuilds itself with it, so getting cmake proper to build in parallel *is* the same problem as getting any other cmake-using port to build in parallel. Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. The problem isn't that make -j fails with cmake. The build succeeds just fine. The problem is that with our make there is no parallelism. It's as if the -j was ignored. -- Christian naddy Weisgerber na...@mips.inka.de
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 04:52:17PM +0100, Christian Weisgerber wrote: Marc Espie: I thought you guys were talking about building cmake proper in parallel. We did. cmake proper first builds a minimal bootstrap cmake, then rebuilds itself with it, so getting cmake proper to build in parallel *is* the same problem as getting any other cmake-using port to build in parallel. Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. The problem isn't that make -j fails with cmake. The build succeeds just fine. The problem is that with our make there is no parallelism. It's as if the -j was ignored. It's likely that cmake decides (arbitrarily) things don't work without gmake. Since there is some recursive makefiles involved, it probably strips the extra stuff early on...
Re: Parallel cmake (diff to review)
Hey, dude- On Dec 12, 2012, at 7:51, David Coppa dco...@gmail.com wrote: Thanks. I'll wait for useful pointers... I don't think CMake does anything specifically to handle parallel recursive builds. It works with GNU make because when invoked with -jX the top level gmake sets some environment variables (it adds -j to MAKEFLAGS, but i think it uses some others for job control, too) so sub-makes pick that up in parallel builds and CMake doesn't need to explicitly write -j$((X - 1)) in targets invoking sub-makes. How does our make handle parallel job control between recursive invocations? Best, Kent
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
Hey, dude- On Dec 12, 2012, at 9:30, Marc Espie es...@nerim.net wrote: Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. The second one is the problem here. CMake doesn't do anything special for parallel builds, it just leans on gmake's environment variable fu for parallel job control. (besides that, it might be a good idea to have a generator that caters a bit more to our make, specifically, include subfile is now part of posix. I think that's the right way to fix this problem. Best, Kent
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Wed, Dec 12, 2012 at 9:48 AM, Kent R. Spillner kspill...@acm.org wrote: Hey, dude- On Dec 12, 2012, at 9:30, Marc Espie es...@nerim.net wrote: Anyways, there are TWO distinct points: - problems with make -j. - cmake not writing correct makefiles for parallel building without gmake. The second one is the problem here. CMake doesn't do anything special for parallel builds, it just leans on gmake's environment variable fu for parallel job control. (besides that, it might be a good idea to have a generator that caters a bit more to our make, specifically, include subfile is now part of posix. I think that's the right way to fix this problem. Best, Kent here is the mail from Brad King On 12/12/2012 08:41 AM, Amit Kulkarni wrote: On OpenBSD, we want to build cmake and some critical cmake based ports in parallel using our dpb (distributed ports builder). We pass j=N in a MAKE_JOBS environment variable when building cmake and cmake based ports using our Makefile hierarchy. It seems that when we use gmake, cmake builds in parallel, i.e with -j option. Plain bsd make fails here... http://www.mail-archive.com/ports@openbsd.org/msg44397.html Our porters want to track down this issue and have it built in parallel, so is the environment variable to pass j=N flag called MAKEFLAGS? A pointer would be appreciated before digging into the source. CMake does not require GNU make on any platform. The Makefile we generate will work with any make tool, even for parallel builds. For projects other than CMake itself, CMake simply generates the build files. Then it is up to you to run make -j $N if you want to build the result in parallel. For CMake itself there is the bootstrap script (and a configure alias for it). This script's only job is to build a minimal part of CMake with just enough to run on CMake itself to generate build files like any other CMake-based project. After bootstrap is done it is still up to you to run make -j $N to build CMake in parallel. What remains is to get the bootstrap build to work in parallel too. That is what the --parallel option to the bootstrap script requests. That script tries to detect whether the make tool supports parallel builds. Only if that detection fails would the script choose not to pass -j $N to make.
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
I thought you guys were talking about building cmake proper in parallel.
We did. cmake proper first builds a minimal bootstrap cmake, then
rebuilds itself with it, so getting cmake proper to build in parallel
*is* the same problem as getting any other cmake-using port to build
in parallel.
Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building without gmake.
The problem isn't that make -j fails with cmake. The build succeeds
just fine. The problem is that with our make there is no parallelism.
It's as if the -j was ignored.
It's likely that cmake decides (arbitrarily) things don't work without gmake.
Since there is some recursive makefiles involved, it probably strips the
extra stuff early on...
i could not see any gmake specific code when i grepped in the cmake codebase.
i can confirm what naddy@ sees, when i cd ${WRKBUILD} make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
13.12.2012 4:22 пользователь Amit Kulkarni amitk...@gmail.com написал:
I thought you guys were talking about building cmake proper in
parallel.
We did. cmake proper first builds a minimal bootstrap cmake, then
rebuilds itself with it, so getting cmake proper to build in parallel
*is* the same problem as getting any other cmake-using port to build
in parallel.
Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building without
gmake.
The problem isn't that make -j fails with cmake. The build succeeds
just fine. The problem is that with our make there is no parallelism.
It's as if the -j was ignored.
It's likely that cmake decides (arbitrarily) things don't work without
gmake.
Since there is some recursive makefiles involved, it probably strips the
extra stuff early on...
i could not see any gmake specific code when i grepped in the cmake
codebase.
i can confirm what naddy@ sees, when i cd ${WRKBUILD} make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.
The problem was already made clear: GNU Make propagates -j to subcalls,
our - does not. The latter is by design, IIRC (if I'm wrong here, then,
probably, espie@ will use his cluestick to teach me not to write about
things I understand badly), to avoid extra subprocesses being run: suppose
that make -j 4 runs four make -j 3, then each runs three make -j 2...
That's GNU Make's way, IIRC, and it's broken by design. Unfortunately,
three is no easy way to fix this. The best option I see (and it's probably
wrong) is to create socket in /tmp on the initial make(1) invocation, and
pass its path to subprocesses through environment variable. Through this
socket, each sub-make could request the right to start one or more jobs,
and wait until master make process answers. But I'm not ready to prepare
any patches implementing such functionality now: too much time is being
spent on fixing KDE breakage (due to upstream lazyness, my own stupidity
and a lots of inaccuracy from both sides).
Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)
On Thu, Dec 13, 2012 at 7:01 AM, Vadim Zhukov persg...@gmail.com wrote:
13.12.2012 4:22 пользователь Amit Kulkarni amitk...@gmail.com написал:
I thought you guys were talking about building cmake proper in
parallel.
We did. cmake proper first builds a minimal bootstrap cmake, then
rebuilds itself with it, so getting cmake proper to build in parallel
*is* the same problem as getting any other cmake-using port to build
in parallel.
Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building without
gmake.
The problem isn't that make -j fails with cmake. The build succeeds
just fine. The problem is that with our make there is no parallelism.
It's as if the -j was ignored.
It's likely that cmake decides (arbitrarily) things don't work without
gmake.
Since there is some recursive makefiles involved, it probably strips the
extra stuff early on...
i could not see any gmake specific code when i grepped in the cmake
codebase.
i can confirm what naddy@ sees, when i cd ${WRKBUILD} make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.
The problem was already made clear: GNU Make propagates -j to subcalls,
our - does not. The latter is by design, IIRC (if I'm wrong here, then,
probably, espie@ will use his cluestick to teach me not to write about
things I understand badly), to avoid extra subprocesses being run: suppose
that make -j 4 runs four make -j 3, then each runs three make -j 2...
That's GNU Make's way, IIRC, and it's broken by design. Unfortunately, three
is no easy way to fix this. The best option I see (and it's probably wrong)
is to create socket in /tmp on the initial make(1) invocation, and pass its
path to subprocesses through environment variable. Through this socket, each
sub-make could request the right to start one or more jobs, and wait until
master make process answers. But I'm not ready to prepare any patches
implementing such functionality now: too much time is being spent on fixing
KDE breakage (due to upstream lazyness, my own stupidity and a lots of
inaccuracy from both sides).
Btw, I still think it's not so bad to use gmake *only* for cmake (not
for all the cmake-based ports!) to speed up dpb builds...
