Re: HEADS UP multi processor compilations for everyone
On Wed, 25 Mar 2009 12:57:51 -0400 Coleman Kane cok...@freebsd.org wrote: On Wed, 2009-03-25 at 19:30 +0300, Dmitry Marakasov wrote: * Pav Lucistnik (p...@freebsd.org) wrote: This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. This has worked fine for me for countless years, except where the vendor's Makefiles were not parallel-safe. This has been my trick to get larger things (like mysql or xorg-server) to make in parallel. It *did* work. If this has changed, then it definitely warrants mention in UPDATING. Then it must have worked all these years by pure chance :) Be the way, could anyone clarify how this works? My idea was that passing -j to port Makefile does nothing, as make/gmake on vendor's Makefile is ran without any -j flags - you get usual singlethreaded build. However, I have a broken port, which uses gmake and something like that: sometarget: (cd xxx; make) and that fails with -j (make: illegal option -- -). So is there some magic with recursive make calls and -j? When processing a Makefile, make's that support concurrent operation look for targets that will execute the $(MAKE) program. Whenever a compliant make is run (make or gmake), if it detects $(MAKE) in a rule then it will automatically expand that rule into a child process that has some sort of magical connection to the parent process. The connection allows the different make processes to share the same pool of process count resources amongst each other. I am not sure what the implementation is, but this communication mechanism allows child make processes called with $(MAKE) ... from inside a Makefile to globally only use N children (from -j N), and otherwise block until more free jobs are available amongst their shared job pool. I hope that's clear... Probably more so than the explanation given on GNU make's manual. -- Coleman Kane Indeed, that is why (cd xxx; make) fails where gmake is running this command. The way GNU make and BSD make passes -j related options are different. They are not compatible. If I remember correctly, if GNU make calls BSD make (which is an error of writing Makefiles in most of vendor codes), the differences are significant and fails. Usually, changing make to $(MAKE) fixes the problems. Why did it build when -j was not supplied. That is because makefile rule was not GNU makefile specific such that BSD make could also executed it without problems. With -j switch presented, these two make commands became incompatible each other. I hope this helps if you guys haven't found this facts; I hope I had replied earlier. Regards, Hiro ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Dmitry Marakasov píše v pá 27. 03. 2009 v 15:48 +0300:
* Pav Lucistnik (p...@freebsd.org) wrote:
second
also improves MAKE_JOBS_* handling, shortening it a bit and exposing
MAKE_JOBS_NUMBER to the ports, so it can be used for other build systems
without having to parse out -j from _MAKE_JOBS (and defaults to 1 if
jobs support is disabled).
This one I cannot accept, because it adds back != call I made a big
effort to avoid. We can't have one != call per port during building
INDEX.
Understood. I still think it'd be nice to expose number of jobs as a
plain number to the ports.
If you figure it out without != call, let me know.
Also, [ x != x${BUILD_FAIL_MESSAGE} ] thing seems to be a bit unsafe,
and inconsistent to IGNORE/BROKEN/... vars, in which we don't use
quotes.
It's consistent with CONFIGURE_FAIL_MESSAGE. Why are you removing the
parenthesis around the ${ECHO_CMD} ${BUILD_FAIL_MESSAGE}, BTW?
Because they are present in do-configure target too - should they be
removed there too?
I just didn't like them :)
Good enough reason for me :)
Are they really needed around single command?
No idea.
--
Pav Lucistnik p...@oook.cz
p...@freebsd.org
On real UNIX, /usr/bin/more prints -More-.
signature.asc
Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
Dmitry Marakasov píše v pá 27. 03. 2009 v 04:02 +0300:
* Pav Lucistnik (p...@freebsd.org) wrote:
Btw, this change broke build failures. If vendor's make fails,
.build_done.xxx._usr_local is still created in work and $? = 0 as if it
have succeeded.
Can you give me a hard example?
Test port consisting of a sole Makefile attached.
I've made two patches for bsd.port.mk.
First one fixes this issue by adding some false's to do-build,
Ah right, I have stolen do-configure targets and missed the ${FALSE}.
Will fix.
second
also improves MAKE_JOBS_* handling, shortening it a bit and exposing
MAKE_JOBS_NUMBER to the ports, so it can be used for other build systems
without having to parse out -j from _MAKE_JOBS (and defaults to 1 if
jobs support is disabled).
This one I cannot accept, because it adds back != call I made a big
effort to avoid. We can't have one != call per port during building
INDEX.
Also, [ x != x${BUILD_FAIL_MESSAGE} ] thing seems to be a bit unsafe,
and inconsistent to IGNORE/BROKEN/... vars, in which we don't use
quotes.
It's consistent with CONFIGURE_FAIL_MESSAGE. Why are you removing the
parenthesis around the ${ECHO_CMD} ${BUILD_FAIL_MESSAGE}, BTW?
Because they are present in do-configure target too - should they be
removed there too?
--
Pav Lucistnik p...@oook.cz
p...@freebsd.org
Pain clots and unformed lice pat this train.
signature.asc
Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
* Pav Lucistnik (p...@freebsd.org) wrote:
second
also improves MAKE_JOBS_* handling, shortening it a bit and exposing
MAKE_JOBS_NUMBER to the ports, so it can be used for other build systems
without having to parse out -j from _MAKE_JOBS (and defaults to 1 if
jobs support is disabled).
This one I cannot accept, because it adds back != call I made a big
effort to avoid. We can't have one != call per port during building
INDEX.
Understood. I still think it'd be nice to expose number of jobs as a
plain number to the ports.
Also, [ x != x${BUILD_FAIL_MESSAGE} ] thing seems to be a bit unsafe,
and inconsistent to IGNORE/BROKEN/... vars, in which we don't use
quotes.
It's consistent with CONFIGURE_FAIL_MESSAGE. Why are you removing the
parenthesis around the ${ECHO_CMD} ${BUILD_FAIL_MESSAGE}, BTW?
Because they are present in do-configure target too - should they be
removed there too?
I just didn't like them :) Are they really needed around single command?
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
* Pav Lucistnik (p...@freebsd.org) wrote: Btw, this change broke build failures. If vendor's make fails, .build_done.xxx._usr_local is still created in work and $? = 0 as if it have succeeded. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
On Thu, Mar 26, 2009 at 10:46 AM, Dmitry Marakasov amd...@amdmi3.ru wrote: * Pav Lucistnik (p...@freebsd.org) wrote: Btw, this change broke build failures. If vendor's make fails, .build_done.xxx._usr_local is still created in work and $? = 0 as if it have succeeded. Another thing that may have failed to have been mentioned earlier: if a python package uses easy_install, there's a potential for corrupted data and missing install registries, so everything that uses easy_install cannot be parallelized at this time. This is a bug in easy_install that I need to provide a quick hack to fix upstream because it's affecting my group at Cisco right now too. Thanks, -Garrett ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Dmitry Marakasov píše v čt 26. 03. 2009 v 20:46 +0300: * Pav Lucistnik (p...@freebsd.org) wrote: Btw, this change broke build failures. If vendor's make fails, .build_done.xxx._usr_local is still created in work and $? = 0 as if it have succeeded. Can you give me a hard example? -- Pav Lucistnik p...@oook.cz p...@freebsd.org Eat when you are hungry, sleep when you are tired. Chase butterflies when you want some fun. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
Garrett Cooper píše v čt 26. 03. 2009 v 11:55 -0700: On Thu, Mar 26, 2009 at 10:46 AM, Dmitry Marakasov amd...@amdmi3.ru wrote: * Pav Lucistnik (p...@freebsd.org) wrote: Btw, this change broke build failures. If vendor's make fails, .build_done.xxx._usr_local is still created in work and $? = 0 as if it have succeeded. Another thing that may have failed to have been mentioned earlier: if a python package uses easy_install, there's a potential for corrupted data and missing install registries, so everything that uses easy_install cannot be parallelized at this time. This is a bug in easy_install that I need to provide a quick hack to fix upstream because it's affecting my group at Cisco right now too. Simple fix - don't mark easy_install port with SAFE flag. That's why it's whitelist. -- Pav Lucistnik p...@oook.cz p...@freebsd.org If God is perfect, why did He create discontinuous functions? signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
* Pav Lucistnik (p...@freebsd.org) wrote:
Btw, this change broke build failures. If vendor's make fails,
.build_done.xxx._usr_local is still created in work and $? = 0 as if it
have succeeded.
Can you give me a hard example?
Test port consisting of a sole Makefile attached.
I've made two patches for bsd.port.mk.
First one fixes this issue by adding some false's to do-build, second
also improves MAKE_JOBS_* handling, shortening it a bit and exposing
MAKE_JOBS_NUMBER to the ports, so it can be used for other build systems
without having to parse out -j from _MAKE_JOBS (and defaults to 1 if
jobs support is disabled).
Also, [ x != x${BUILD_FAIL_MESSAGE} ] thing seems to be a bit unsafe,
and inconsistent to IGNORE/BROKEN/... vars, in which we don't use
quotes.
---
r...@hades:test# make
=== Extracting for test-1.0
=== Patching for test-1.0
=== Configuring for test-1.0
(echo all:; echo false)
/usr/home/amdmi3/projects/ports/test/work/Makefile
=== Building for test-1.0
false
*** Error code 1
Stop in /usr/home/amdmi3/projects/ports/test/work.
r...@hades:test# echo $?
0
r...@hades:test# ls work
.build_done.test._usr_local
.configure_done.test._usr_local
.extract_done.test._usr_local
.patch_done.test._usr_local
Makefile
---
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru
# New ports collection makefile for:test
# Date created: 27 Mar 2009
# Whom: Dmitry Marakasov amd...@freebsd.org
#
# $FreeBSD$
#
PORTNAME= test
PORTVERSION=1.0
CATEGORIES= sysutils
MASTER_SITES= #none
DISTFILES= #none
MAINTAINER= amd...@freebsd.org
COMMENT=Empty comment
NO_WRKSUBDIR= yes
do-configure:
(echo all:; echo false) ${WRKSRC}/Makefile
.include bsd.port.mk
Index: bsd.port.mk
===
RCS file: /home/amdmi3/projects/freebsd/FreeBSD.cvs/ports/Mk/bsd.port.mk,v
retrieving revision 1.614
diff -u -r1.614 bsd.port.mk
--- bsd.port.mk 22 Mar 2009 10:28:53 - 1.614
+++ bsd.port.mk 27 Mar 2009 00:40:57 -
@@ -3693,16 +3693,18 @@
@(cd ${BUILD_WRKSRC}; if ! ${SETENV} ${MAKE_ENV} ${GMAKE} ${MAKE_FLAGS} ${MAKEFILE} ${_MAKE_JOBS} ${MAKE_ARGS} ${ALL_TARGET}; then \
if [ x != x${BUILD_FAIL_MESSAGE} ] ; then \
${ECHO_MSG} === Compilation failed unexpectedly.; \
- (${ECHO_CMD} ${BUILD_FAIL_MESSAGE}) | ${FMT} 75 79 ; \
+ ${ECHO_CMD} ${BUILD_FAIL_MESSAGE} | ${FMT} 75 79 ; \
fi; \
- fi)
+ fi; \
+ false)
.else
@(cd ${BUILD_WRKSRC}; if ! ${SETENV} ${MAKE_ENV} ${MAKE} ${MAKE_FLAGS} ${MAKEFILE} ${_MAKE_JOBS} ${MAKE_ARGS} ${ALL_TARGET}; then \
if [ x != x${BUILD_FAIL_MESSAGE} ] ; then \
${ECHO_MSG} === Compilation failed unexpectedly.; \
- (${ECHO_CMD} ${BUILD_FAIL_MESSAGE}) | ${FMT} 75 79 ; \
+ ${ECHO_CMD} ${BUILD_FAIL_MESSAGE} | ${FMT} 75 79 ; \
fi; \
- fi)
+ fi; \
+ false)
.endif
.endif
Index: bsd.port.mk
===
RCS file: /home/amdmi3/projects/freebsd/FreeBSD.cvs/ports/Mk/bsd.port.mk,v
retrieving revision 1.614
diff -u -r1.614 bsd.port.mk
--- bsd.port.mk 22 Mar 2009 10:28:53 - 1.614
+++ bsd.port.mk 27 Mar 2009 01:00:38 -
@@ -2177,18 +2177,17 @@
# Multiple make jobs support
.if defined(DISABLE_MAKE_JOBS) || defined(MAKE_JOBS_UNSAFE)
_MAKE_JOBS=#
-.else
-.if defined(MAKE_JOBS_SAFE) || defined(FORCE_MAKE_JOBS)
-.if defined(MAKE_JOBS_NUMBER)
-_MAKE_JOBS=-j${MAKE_JOBS_NUMBER}
-.else
-_MAKE_JOBS=-j`${SYSCTL} -n kern.smp.cpus`
+.elif defined(MAKE_JOBS_SAFE) || defined(FORCE_MAKE_JOBS)
+.if !defined(MAKE_JOBS_NUMBER)
+MAKE_JOBS_NUMBER!= ${SYSCTL} -n kern.smp.cpus
.endif
+_MAKE_JOBS=-j${MAKE_JOBS_NUMBER}
.if defined(FORCE_MAKE_JOBS)
BUILD_FAIL_MESSAGE+= You have chosen to use multiple make jobs
(parallelization) for all ports. This port was not tested for this setting.
Please remove FORCE_MAKE_JOBS and retry the build before reporting the failure
to the maintainer.
.endif
.endif
-.endif
+
+MAKE_JOBS_NUMBER?= 1
PTHREAD_CFLAGS?=
PTHREAD_LIBS?= -pthread
@@ -3693,16 +3692,18 @@
@(cd ${BUILD_WRKSRC}; if ! ${SETENV} ${MAKE_ENV} ${GMAKE} ${MAKE_FLAGS}
${MAKEFILE} ${_MAKE_JOBS} ${MAKE_ARGS} ${ALL_TARGET}; then \
if [ x != x${BUILD_FAIL_MESSAGE} ] ; then \
${ECHO_MSG} === Compilation failed unexpectedly.; \
- (${ECHO_CMD} ${BUILD_FAIL_MESSAGE}) | ${FMT} 75 79 ; \
+ ${ECHO_CMD} ${BUILD_FAIL_MESSAGE} | ${FMT} 75 79 ; \
fi; \
- fi)
+ fi; \
+ false)
.else
@(cd ${BUILD_WRKSRC}; if ! ${SETENV} ${MAKE_ENV} ${MAKE} ${MAKE_FLAGS}
${MAKEFILE} ${_MAKE_JOBS} ${MAKE_ARGS} ${ALL_TARGET}; then \
if [ x != x${BUILD_FAIL_MESSAGE} ] ; then \
Re: HEADS UP multi processor compilations for everyone
Pav Lucistnik p...@freebsd.org writes:
If you are FreeBSD port maintainer:
I'm not one.
Nothing changes for you, if you don't want. If you want to enable the
use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block
somewhere below dependency declarations. If you know your port does not
handle -jX well, and want to disable it from using -jX even when user
forces this feature, use MAKE_JOBS_UNSAFE=yes. And that's all to it.
Not all ports build using make/gmake. Wouldn't it be better to export
the number of parallel processes so maintainer can decide whether to use
it in his port build system? For example
Index: devel/boost/Makefile
===
RCS file: /home/csup/ports/devel/boost/Makefile,v
retrieving revision 1.42
diff -u -p -r1.42 Makefile
--- devel/boost/Makefile 20 Feb 2009 01:13:49 - 1.42
+++ devel/boost/Makefile 25 Mar 2009 06:12:06 -
@@ -17,6 +17,7 @@ COMMENT= Free peer-reviewed portable C++
USE_BZIP2= yes
USE_LDCONFIG= yes
+MAKE_JOBS_SAFE= yes
WRKSRC= ${WRKDIR}/${PORTNAME}_${PORTVERSION:S/./_/g}
.if defined(WITH_PYTHON) || defined (WITH_PYSTE)
@@ -87,6 +88,7 @@ PLIST_SUB+= BOOST_PYSTE=@comment
BOOST_TOOLS= gcc
BJAM_OPTIONS= --layout=system
+BJAM_OPTIONS+= -j${_MAKE_JOBS:S/-j//}
.if defined (WITH_VERBOSE_BUILD)
BJAM_OPTIONS+= -d2
.endif
Is this completely discouraged?
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Anonymous píše v st 25. 03. 2009 v 09:26 +0300: Pav Lucistnik p...@freebsd.org writes: If you are FreeBSD port maintainer: I'm not one. Nothing changes for you, if you don't want. If you want to enable the use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block somewhere below dependency declarations. If you know your port does not handle -jX well, and want to disable it from using -jX even when user forces this feature, use MAKE_JOBS_UNSAFE=yes. And that's all to it. Not all ports build using make/gmake. Wouldn't it be better to export the number of parallel processes so maintainer can decide whether to use it in his port build system? For example Is this completely discouraged? I suppose you can use internal variable _MAKE_JOBS directly. Why are you stripping -j just to add it back again? -- Pav Lucistnik p...@oook.cz p...@freebsd.org Any Palm app requiring an 90+ page manual has lost its vision. -- words about DateBk4 on Action Names list signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
Pav Lucistnik p...@freebsd.org writes: Anonymous pí¹e v st 25. 03. 2009 v 09:26 +0300: Pav Lucistnik p...@freebsd.org writes: If you are FreeBSD port maintainer: I'm not one. Nothing changes for you, if you don't want. If you want to enable the use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block somewhere below dependency declarations. If you know your port does not handle -jX well, and want to disable it from using -jX even when user forces this feature, use MAKE_JOBS_UNSAFE=yes. And that's all to it. Not all ports build using make/gmake. Wouldn't it be better to export the number of parallel processes so maintainer can decide whether to use it in his port build system? For example Is this completely discouraged? I suppose you can use internal variable _MAKE_JOBS directly. Why are you stripping -j just to add it back again? Oh, so you're not against the idea. The substitution was to be able to easy replace it with something else. FYI, that example went to ports/133054. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Anonymous píše v st 25. 03. 2009 v 12:51 +0300: FYI, that example went to ports/133054. Cool. Just a side note -- boost is having a major overhaul, so you might want to coordinate with the folks who currently do the work, too. -- Pav Lucistnik p...@oook.cz p...@freebsd.org Like 'Do Notte Buye Betamacks.' That was a prediction for 1972. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
* Pav Lucistnik (p...@freebsd.org) wrote: This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. This has worked fine for me for countless years, except where the vendor's Makefiles were not parallel-safe. This has been my trick to get larger things (like mysql or xorg-server) to make in parallel. It *did* work. If this has changed, then it definitely warrants mention in UPDATING. Then it must have worked all these years by pure chance :) Be the way, could anyone clarify how this works? My idea was that passing -j to port Makefile does nothing, as make/gmake on vendor's Makefile is ran without any -j flags - you get usual singlethreaded build. However, I have a broken port, which uses gmake and something like that: sometarget: (cd xxx; make) and that fails with -j (make: illegal option -- -). So is there some magic with recursive make calls and -j? -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
* Pav Lucistnik (p...@freebsd.org) wrote: Great, that was just about time. My question is whenther that would be enabled on pointyhat. I fear that error logs may become far less readable with output from multiple commands mixed together. We may also consider using -P option for BSD make for that reason (althrough there's no equivalent for GNU make). -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Dmitry Marakasov píše v st 25. 03. 2009 v 19:36 +0300: * Pav Lucistnik (p...@freebsd.org) wrote: Great, that was just about time. My question is whenther that would be enabled on pointyhat. I fear that error logs may become far less readable with output from multiple commands mixed together. pointyhat observes the same rules and conditions as the end user, for obvious reasons. So ports marked 'safe' will be parallelized, others will not. -- Pav Lucistnik p...@oook.cz p...@freebsd.org It's the classic Microsoft security-bulletin formula: The vulnerability is important (never dangerous); you have nothing to fear and no reason to regret trusting us; we have no intention of apologizing for it or even explaining it adequately; now go get your patch, shut up, and be grateful nothing bad has happened. -- The Register signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
On Wed, 2009-03-25 at 19:30 +0300, Dmitry Marakasov wrote: * Pav Lucistnik (p...@freebsd.org) wrote: This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. This has worked fine for me for countless years, except where the vendor's Makefiles were not parallel-safe. This has been my trick to get larger things (like mysql or xorg-server) to make in parallel. It *did* work. If this has changed, then it definitely warrants mention in UPDATING. Then it must have worked all these years by pure chance :) Be the way, could anyone clarify how this works? My idea was that passing -j to port Makefile does nothing, as make/gmake on vendor's Makefile is ran without any -j flags - you get usual singlethreaded build. However, I have a broken port, which uses gmake and something like that: sometarget: (cd xxx; make) and that fails with -j (make: illegal option -- -). So is there some magic with recursive make calls and -j? When processing a Makefile, make's that support concurrent operation look for targets that will execute the $(MAKE) program. Whenever a compliant make is run (make or gmake), if it detects $(MAKE) in a rule then it will automatically expand that rule into a child process that has some sort of magical connection to the parent process. The connection allows the different make processes to share the same pool of process count resources amongst each other. I am not sure what the implementation is, but this communication mechanism allows child make processes called with $(MAKE) ... from inside a Makefile to globally only use N children (from -j N), and otherwise block until more free jobs are available amongst their shared job pool. I hope that's clear... Probably more so than the explanation given on GNU make's manual. -- Coleman Kane signature.asc Description: This is a digitally signed message part
Re: HEADS UP multi processor compilations for everyone
Pav Lucistnik wrote: Two days ago, I have checked in probably most requested feature of last few years. Ports framework now systematically supports building ports on multiple processing cores. It is achieved by passing -jX flag to make(1) running on vendor code. Thanks for this very useful addition! To clarify: this is about making individual ports in parallel (make -j on each), not building multiple ports in parallel? signature.asc Description: OpenPGP digital signature
Re: HEADS UP multi processor compilations for everyone
Great work! Pav Lucistnik wrote: Two days ago, I have checked in probably most requested feature of last few years. Ports framework now systematically supports building ports on multiple processing cores. It is achieved by passing -jX flag to make(1) running on vendor code. Of course not all ports handle this well, experimental run on pointyhat with this flag globally enabled turned up shy of 400 failures. Because of that, the feature was designed as a whitelist. Individual ports need to be enabled, and indeed, fellow developers took on and already started adding required declarations to popular ports like Firefox and others. If you are FreeBSD ports user: You don't need to do anything to enable the new feature. Whitelisted ports will automatically make use of all processors available in your computer. If you want, for some reasons, to disable this feature, put DISABLE_MAKE_JOBS=yes to your /etc/make.conf. By default, the level of parallelization will be equal to a number of processing cores in your machine. If you want to override this number, use for example MAKE_JOBS_NUMBER=6, again in /etc/make.conf. And if you are extra brave, or you want to check out all the yet unmarked ports, if they will build, you can define FORCE_MAKE_JOBS=yes in /etc/make.conf. Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ If you are FreeBSD port maintainer: Nothing changes for you, if you don't want. If you want to enable the use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block somewhere below dependency declarations. If you know your port does not handle -jX well, and want to disable it from using -jX even when user forces this feature, use MAKE_JOBS_UNSAFE=yes. And that's all to it. Regards! //Niclas ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
On Tue, Mar 24, 2009 at 10:28 AM, Niclas Zeising niclas.zeis...@gmail.com wrote: Great work! Indeed. Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ Can it be added to UPDATING as well? -- Glen Barber 570.328.0318 ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Niclas Zeising píše v út 24. 03. 2009 v 15:28 +0100: Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ I will document it soon, thinking The Handbook would be best place. -- Pav Lucistnik p...@oook.cz p...@freebsd.org East or west, ~ is best. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
On Tue, Mar 24, 2009 at 03:04:41PM +0100, Ivan Voras wrote: Pav Lucistnik wrote: Two days ago, I have checked in probably most requested feature of last few years. Ports framework now systematically supports building ports on multiple processing cores. It is achieved by passing -jX flag to make(1) running on vendor code. Thanks for this very useful addition! To clarify: this is about making individual ports in parallel (make -j on each), not building multiple ports in parallel? Correct. The latter was being worked on in last years SoC. The author of that (David Forsythe) recently posted something about it. The subject of his post is Parallel builds, PKG_DBDIR locks. -- WXS ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
On Tue, 2009-03-24 at 15:54 +0100, Pav Lucistnik wrote: Niclas Zeising píše v út 24. 03. 2009 v 15:28 +0100: Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ I will document it soon, thinking The Handbook would be best place. Definitely add it to UPDATING too. This will allow people who typically do something like make configure make -j3 to now know that they don't have to. It will also allow others to know why ports compilation on their multi-core boxes suddenly uses a lot more CPU time. BTW, Good work, Pav! Thank you for taking the time to do what so many of us wanted but wouldn't do. -- Coleman Kane signature.asc Description: This is a digitally signed message part
Re: HEADS UP multi processor compilations for everyone
Coleman Kane píše v út 24. 03. 2009 v 10:58 -0400: On Tue, 2009-03-24 at 15:54 +0100, Pav Lucistnik wrote: Niclas Zeising píše v út 24. 03. 2009 v 15:28 +0100: Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ I will document it soon, thinking The Handbook would be best place. Definitely add it to UPDATING too. This will allow people who typically do something like make configure make -j3 to now know that they This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. don't have to. It will also allow others to know why ports compilation on their multi-core boxes suddenly uses a lot more CPU time. Same CPU time, less wall time, more CPU utilization :) -- Pav Lucistnik p...@oook.cz p...@freebsd.org If God didn't mean for us to juggle, tennis balls wouldn't come three to a can. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
On Tue, 2009-03-24 at 16:19 +0100, Pav Lucistnik wrote: Coleman Kane píše v út 24. 03. 2009 v 10:58 -0400: On Tue, 2009-03-24 at 15:54 +0100, Pav Lucistnik wrote: Niclas Zeising píše v út 24. 03. 2009 v 15:28 +0100: Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ I will document it soon, thinking The Handbook would be best place. Definitely add it to UPDATING too. This will allow people who typically do something like make configure make -j3 to now know that they This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. This has worked fine for me for countless years, except where the vendor's Makefiles were not parallel-safe. This has been my trick to get larger things (like mysql or xorg-server) to make in parallel. It *did* work. If this has changed, then it definitely warrants mention in UPDATING. don't have to. It will also allow others to know why ports compilation on their multi-core boxes suddenly uses a lot more CPU time. Same CPU time, less wall time, more CPU utilization :) Thanks for the clarification... that's what I meant :) -- Coleman Kane signature.asc Description: This is a digitally signed message part
Re: HEADS UP multi processor compilations for everyone
Coleman Kane píše v út 24. 03. 2009 v 12:28 -0400: On Tue, 2009-03-24 at 16:19 +0100, Pav Lucistnik wrote: Coleman Kane píše v út 24. 03. 2009 v 10:58 -0400: On Tue, 2009-03-24 at 15:54 +0100, Pav Lucistnik wrote: Niclas Zeising píše v út 24. 03. 2009 v 15:28 +0100: Not to nitpick or be an annoyance, but you might want to document this in ports(7) or make.conf(5) (or both) so it doesn't get lost in the mail-lists or if people are not reading ports@ I will document it soon, thinking The Handbook would be best place. Definitely add it to UPDATING too. This will allow people who typically do something like make configure make -j3 to now know that they This would break very fast -- it's passing -j3 to port Makefile instead of vendor Makefile. This has worked fine for me for countless years, except where the vendor's Makefiles were not parallel-safe. This has been my trick to get larger things (like mysql or xorg-server) to make in parallel. It *did* work. If this has changed, then it definitely warrants mention in UPDATING. Then it must have worked all these years by pure chance :) -- Pav Lucistnik p...@oook.cz p...@freebsd.org 94 outdated ports on the box, 94 outdated ports. Portupgrade one, an hour 'til done, 82 outdated ports on the box. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: HEADS UP multi processor compilations for everyone
Pav Lucistnik p...@freebsd.org wrote: Ports framework now systematically supports building ports on multiple processing cores. It is achieved by passing -jX flag to make(1) running on vendor code. Of course not all ports handle this well, A word of caution: It is quite possible for a port to build fine with -jN and still fail with -jM, where N M. If it builds fine on a 64-core sparc64, it will build anywhere, but I expect some ports will work fine for, say, -j2 and fail for -j4. Or only fail *sometimes* due to a race. If you are FreeBSD port maintainer: Nothing changes for you, if you don't want. If you want to enable the use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block somewhere below dependency declarations. What is the goal here? Should all ports that can be safely built in parallel eventually have MAKE_JOBS_SAFE=yes? Or should we not bother with ports where the gain is likely to be minimal? -- Christian naddy Weisgerber na...@mips.inka.de ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: HEADS UP multi processor compilations for everyone
Christian Weisgerber píše v út 24. 03. 2009 v 18:09 +: If you are FreeBSD port maintainer: Nothing changes for you, if you don't want. If you want to enable the use of multiple cores in your port, add MAKE_JOBS_SAFE=yes to a block somewhere below dependency declarations. What is the goal here? Should all ports that can be safely built in parallel eventually have MAKE_JOBS_SAFE=yes? Ideally yes. Or should we not bother with ports where the gain is likely to be minimal? That would be sensible. -- Pav Lucistnik p...@oook.cz p...@freebsd.org As to floating eyes, let them float :). - r.g.r.a signature.asc Description: Toto je digitálně podepsaná část zprávy
