Re: [uClinux-dev] parallel compiles
Jivin Mike Frysinger lays it down ...
On Wednesday 06 February 2008, David McCullough wrote:
is there a reason for forcing -j1 in sub-packages ? we've dropped the
-j1 in {lib,user}/Makefile for the DIRS target and generally had great
success. you can really feel the difference on an 8 or 16 proc system.
i also replaced the -j1 for the _only targets with -j$(HOST_NCPU).
Interesting result, do you have an measured build times with and
without the change ? It would be nice to know, and if the gains
justify the fixing of any potentially broken sub dirs :-)
just doing a simple `time make AnalogDevices/BF537-STAMP_default` here ...
- with NON_SMP_BUILD set: ~8min
- without NON_SMP_BUILD set (and the -j1 pieces removed): ~1.5min
pretty consistent here ... +/-10seconds
If you have a patch, send it in and I'll try it here and see how bad it
is for our builds, if it's relatively isolated I am happy to fix up
some bits and include it, anything that speeds builds is good,
i'm pushing this for our next-next release (so current development tree).
only noticed net-tools failing so far (but i posted a patch for that)
---
Do not force -j1 in subdirs and use -j$(HOST_NCPU) for _only targets.
Signed-off-by: Mike Frysinger vap...@gentoo.org
Applied. I needed to fix:
new-wave/user/samba/source/makefile (disable parallel builds)
new-wave/user/pciutils/Makefile (fixed deps)
new-wave/user/nessus/Makefile (fixed deps)
For completeness, this reduces my complete build time for a large target
from 25.3 minutes to 21.4 minutes, which was enough to put it in,
Thanks,
Davidm
---
diff --git a/lib/Makefile b/lib/Makefile
index 0de9193..e9e9372 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -136,16 +136,16 @@ all:
# the actual dir target
.PHONY: $(DIRS_y)
$(DIRS_y):
- [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -j1 -C $@ ) || exit
$$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_lib; true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
##
diff --git a/user/Makefile b/user/Makefile
index d452260..18b8f00 100644
--- a/user/Makefile
+++ b/user/Makefile
@@ -440,16 +440,16 @@ asterisk-addons: mysql asterisk
miniupnpd : iptables
$(sort $(dir_y) $(dir_p)): config
- [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -j1 -C $@ ) ||
exit $$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_user $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_user $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_user;
true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
romfs:
for i in $(sort $(dir_y)) $(dir_p) ; do \
--
David McCullough, david_mccullo...@securecomputing.com, Ph:+61 734352815
McAfee - SnapGear http://www.snapgear.comhttp://www.uCdot.org
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
Re: [uClinux-dev] parallel compiles
On Thursday 07 February 2008 00:47:13 Mike Frysinger wrote: On Wednesday 06 February 2008, David McCullough wrote: If you have a patch, send it in and I'll try it here and see how bad it is for our builds, if it's relatively isolated I am happy to fix up some bits and include it, anything that speeds builds is good, i'm pushing this for our next-next release (so current development tree). only noticed net-tools failing so far (but i posted a patch for that) --- Do not force -j1 in subdirs and use -j$(HOST_NCPU) for _only targets. ping ... -mike signature.asc Description: This is a digitally signed message part. ___ uClinux-dev mailing list uClinux-dev@uclinux.org http://mailman.uclinux.org/mailman/listinfo/uclinux-dev This message was resent by uclinux-dev@uclinux.org To unsubscribe see: http://mailman.uclinux.org/mailman/options/uclinux-dev
Re: [uClinux-dev] parallel compiles
Jivin Mike Frysinger lays it down ...
On Wednesday 06 February 2008, David McCullough wrote:
is there a reason for forcing -j1 in sub-packages ? we've dropped the
-j1 in {lib,user}/Makefile for the DIRS target and generally had great
success. you can really feel the difference on an 8 or 16 proc system.
i also replaced the -j1 for the _only targets with -j$(HOST_NCPU).
Interesting result, do you have an measured build times with and
without the change ? It would be nice to know, and if the gains
justify the fixing of any potentially broken sub dirs :-)
just doing a simple `time make AnalogDevices/BF537-STAMP_default` here ...
- with NON_SMP_BUILD set: ~8min
- without NON_SMP_BUILD set (and the -j1 pieces removed): ~1.5min
pretty consistent here ... +/-10seconds
I was hoping for the with -j1 and without -j1 numbers,
comparing against NON_SMP_BUILD=1 is not really fair ;-)
If you have a patch, send it in and I'll try it here and see how bad it
is for our builds, if it's relatively isolated I am happy to fix up
some bits and include it, anything that speeds builds is good,
i'm pushing this for our next-next release (so current development tree).
only noticed net-tools failing so far (but i posted a patch for that)
Patch is small enough, I try it if you don't get time to compare the
before an after.
Cheers,
Davidm
---
Do not force -j1 in subdirs and use -j$(HOST_NCPU) for _only targets.
Signed-off-by: Mike Frysinger [EMAIL PROTECTED]
---
diff --git a/lib/Makefile b/lib/Makefile
index 0de9193..e9e9372 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -136,16 +136,16 @@ all:
# the actual dir target
.PHONY: $(DIRS_y)
$(DIRS_y):
- [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -j1 -C $@ ) || exit
$$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_lib; true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
##
diff --git a/user/Makefile b/user/Makefile
index d452260..18b8f00 100644
--- a/user/Makefile
+++ b/user/Makefile
@@ -440,16 +440,16 @@ asterisk-addons: mysql asterisk
miniupnpd : iptables
$(sort $(dir_y) $(dir_p)): config
- [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -j1 -C $@ ) ||
exit $$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_user $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_user $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_user;
true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
romfs:
for i in $(sort $(dir_y)) $(dir_p) ; do \
--
David McCullough, [EMAIL PROTECTED], Ph:+61 734352815
Secure Computing - SnapGear http://www.uCdot.org http://www.cyberguard.com
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
Re: [uClinux-dev] parallel compiles
Jivin Mike Frysinger lays it down ...
On Thursday 07 February 2008, David McCullough wrote:
On Wednesday 06 February 2008, David McCullough wrote:
is there a reason for forcing -j1 in sub-packages ? we've dropped
the -j1 in {lib,user}/Makefile for the DIRS target and generally had
great success. you can really feel the difference on an 8 or 16 proc
system. i also replaced the -j1 for the _only targets with
-j$(HOST_NCPU).
Interesting result, do you have an measured build times with and
without the change ? It would be nice to know, and if the gains
justify the fixing of any potentially broken sub dirs :-)
just doing a simple `time make AnalogDevices/BF537-STAMP_default` here
... - with NON_SMP_BUILD set: ~8min
- without NON_SMP_BUILD set (and the -j1 pieces removed): ~1.5min
pretty consistent here ... +/-10seconds
I was hoping for the with -j1 and without -j1 numbers,
comparing against NON_SMP_BUILD=1 is not really fair ;-)
how is it not fair ? the NON_SMP_BUILD var forces -j1 ...
NON_SMP_BUILD=1 means there are no parallel builds at all, everything is
done sequentially, and while useful for finding build errors etc, it
is a lot slower than the default partially-paralleled uClinux-dist build
process.
The default in the uClinux-dist is to build multiple lib/user directories
at the same time, but each directory is built single threaded (-j1).
i didnt feel like
locally modifying the Makefiles to get the same result
patch -R ;-)
Cheers,
Davidm
--
David McCullough, [EMAIL PROTECTED], Ph:+61 734352815
Secure Computing - SnapGear http://www.uCdot.org http://www.cyberguard.com
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
[uClinux-dev] parallel compiles
is there a reason for forcing -j1 in sub-packages ? we've dropped the -j1 in
{lib,user}/Makefile for the DIRS target and generally had great success. you
can really feel the difference on an 8 or 16 proc system. i also replaced
the -j1 for the _only targets with -j$(HOST_NCPU).
if a package fails to build in parallel, i think the correct answer is fix the
package in question. if that is too burdensome, you can always force single
jobs in the one package directory rather than forcing everyone else.
-mike
signature.asc
Description: This is a digitally signed message part.
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
Re: [uClinux-dev] parallel compiles
Jivin Mike Frysinger lays it down ...
is there a reason for forcing -j1 in sub-packages ? we've dropped the -j1 in
{lib,user}/Makefile for the DIRS target and generally had great success. you
can really feel the difference on an 8 or 16 proc system. i also replaced
the -j1 for the _only targets with -j$(HOST_NCPU).
Interesting result, do you have an measured build times with and
without the change ? It would be nice to know, and if the gains
justify the fixing of any potentially broken sub dirs :-)
if a package fails to build in parallel, i think the correct answer is fix
the
package in question. if that is too burdensome, you can always force single
jobs in the one package directory rather than forcing everyone else.
It was merely an incremental approach, we have/had quite a lot of older
packages in there and biting things off in small chunks keeps the build
machine running happily ;-).
If you have a patch, send it in and I'll try it here and see how bad it
is for our builds, if it's relatively isolated I am happy to fix up
some bits and include it, anything that speeds builds is good,
Cheers,
Davidm
--
David McCullough, [EMAIL PROTECTED], Ph:+61 734352815
Secure Computing - SnapGear http://www.uCdot.org http://www.cyberguard.com
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
Re: [uClinux-dev] parallel compiles
On Wednesday 06 February 2008, David McCullough wrote:
is there a reason for forcing -j1 in sub-packages ? we've dropped the
-j1 in {lib,user}/Makefile for the DIRS target and generally had great
success. you can really feel the difference on an 8 or 16 proc system.
i also replaced the -j1 for the _only targets with -j$(HOST_NCPU).
Interesting result, do you have an measured build times with and
without the change ? It would be nice to know, and if the gains
justify the fixing of any potentially broken sub dirs :-)
just doing a simple `time make AnalogDevices/BF537-STAMP_default` here ...
- with NON_SMP_BUILD set: ~8min
- without NON_SMP_BUILD set (and the -j1 pieces removed): ~1.5min
pretty consistent here ... +/-10seconds
If you have a patch, send it in and I'll try it here and see how bad it
is for our builds, if it's relatively isolated I am happy to fix up
some bits and include it, anything that speeds builds is good,
i'm pushing this for our next-next release (so current development tree).
only noticed net-tools failing so far (but i posted a patch for that)
---
Do not force -j1 in subdirs and use -j$(HOST_NCPU) for _only targets.
Signed-off-by: Mike Frysinger [EMAIL PROTECTED]
---
diff --git a/lib/Makefile b/lib/Makefile
index 0de9193..e9e9372 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -136,16 +136,16 @@ all:
# the actual dir target
.PHONY: $(DIRS_y)
$(DIRS_y):
- [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -j1 -C $@ ) || exit
$$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_lib $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_lib $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_lib; true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
##
diff --git a/user/Makefile b/user/Makefile
index d452260..18b8f00 100644
--- a/user/Makefile
+++ b/user/Makefile
@@ -440,16 +440,16 @@ asterisk-addons: mysql asterisk
miniupnpd : iptables
$(sort $(dir_y) $(dir_p)): config
- [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -j1 -C $@ ) ||
exit $$?
+ [ ! -d $@ ] || ( touch $@/.sgbuilt_user $(MAKE) -C $@ ) || exit $$?
%_only:
- touch $(@:_only=)/.sgbuilt_user $(MAKE) -j1 -C $(@:_only=)
+ touch $(@:_only=)/.sgbuilt_user $(MAKE) -j$(HOST_NCPU) -C $(@:_only=)
%_clean:
$(MAKE) -j1 -C $(@:_clean=) clean; rm -f $(@:_clean=)/.sgbuilt_user;
true
%_romfs:
- $(MAKE) -j1 -C $(@:_romfs=) romfs
+ $(MAKE) -C $(@:_romfs=) romfs
romfs:
for i in $(sort $(dir_y)) $(dir_p) ; do \
___
uClinux-dev mailing list
uClinux-dev@uclinux.org
http://mailman.uclinux.org/mailman/listinfo/uclinux-dev
This message was resent by uclinux-dev@uclinux.org
To unsubscribe see:
http://mailman.uclinux.org/mailman/options/uclinux-dev
