Re: make -jN build with portmaster
On Monday 26 January 2009 09:14:21 cpghost wrote: On Sun, Jan 25, 2009 at 10:41:05PM -0900, Mel wrote: On Sunday 25 January 2009 20:48:22 Doug Barton wrote: Josh Carroll wrote: What I do is the following via make.conf, I think this is a good solution. Given that incredible foot-shooting power of the -j stuff I am not inclined to add something like this to portmaster, not even as an advanced option. Yes, I understand the problem with that. The make.conf solution is good enough for now. ;) Given the fact that the build target is presumably -j safe (as far as the ports system is concerned), it would be nice to have a BUILD_JOBS in Mk/bsd.port.mk similar to INDEX_JOBS that is already there. Port maintainers then can also set WITHOUT_PARALLEL (or USE_PARALLEL=NO etc) for ports that break by themselves (f.e. www/lynx, editors/vim). portmaster should then have no problem setting BUILD_JOBS on request. That would be absolutely perfect! At least, big ports (www/firefox3 etc...) that take a long time to compile could use USE_PARALLEL=YES right now (or the solution with make.conf) if they are safe with -jN. The gazillion smallish ports could come later when maintainers have some time to follow up, but they are not really all that critical. PR Filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=131065 -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Sun, Jan 25, 2009 at 10:41:05PM -0900, Mel wrote: On Sunday 25 January 2009 20:48:22 Doug Barton wrote: Josh Carroll wrote: What I do is the following via make.conf, I think this is a good solution. Given that incredible foot-shooting power of the -j stuff I am not inclined to add something like this to portmaster, not even as an advanced option. Yes, I understand the problem with that. The make.conf solution is good enough for now. ;) Given the fact that the build target is presumably -j safe (as far as the ports system is concerned), it would be nice to have a BUILD_JOBS in Mk/bsd.port.mk similar to INDEX_JOBS that is already there. Port maintainers then can also set WITHOUT_PARALLEL (or USE_PARALLEL=NO etc) for ports that break by themselves (f.e. www/lynx, editors/vim). portmaster should then have no problem setting BUILD_JOBS on request. That would be absolutely perfect! At least, big ports (www/firefox3 etc...) that take a long time to compile could use USE_PARALLEL=YES right now (or the solution with make.conf) if they are safe with -jN. The gazillion smallish ports could come later when maintainers have some time to follow up, but they are not really all that critical. Memory may fail me, but wasn't there a GSoC project to parallelize the ports infrastructure? Or was that about building many different ports simultaneously instead of one port on multiple cores? -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
snip That would be absolutely perfect! +1 snip Memory may fail me, but wasn't there a GSoC project to parallelize the ports infrastructure? Or was that about building many different ports simultaneously instead of one port on multiple cores? -cpghost. http://wiki.freebsd.org/SummerOfCode2008#head-1abba39cb7f9d11ebef625f2287dbaebe6daf000 Ready to enter CVS: Probably not. -- Eitan Adler Security is increased by designing for the way humans actually behave. -Jakob Nielsen ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Monday 26 January 2009 14:56:10 Eitan Adler wrote: snip That would be absolutely perfect! +1 snip Memory may fail me, but wasn't there a GSoC project to parallelize the ports infrastructure? Or was that about building many different ports simultaneously instead of one port on multiple cores? -cpghost. http://wiki.freebsd.org/SummerOfCode2008#head-1abba39cb7f9d11ebef625f2287db aebe6daf000 Ready to enter CVS: Probably not. This is different: it allows multiple ports to be built at the same time, not 1 port with multiple make processes. -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
make -jN build with portmaster
To build ports in parallel on a 4 core machine, I usually do this manually: # cd /usr/ports/some/port # make configure make -j5 build make install clean because all steps except make build are not compatible with -jN (some ports don't work with -jN in the make build phase either, but they are quite rare). Now, is there a way to teach portmaster to build or rebuild ports this way? The only workaround for now is something like: # cd /usr/ports/some/port # make configure make -j5 build # portmaster -b -d -C name-of-some/port-in-pkg-var-db which doesn't clean up some/port/work before building, thus using the manual parallel step before. So, how can portmaster run 'make -jN build' instead of simple 'make build'? (I know about portmaster's -m option, but passing -jN to it won't work, as it would also try to apply this to other phases than make build). Thanks, -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Sun, Jan 25, 2009 at 4:44 PM, cpghost cpgh...@cordula.ws wrote:
To build ports in parallel on a 4 core machine, I usually
do this manually:
# cd /usr/ports/some/port
# make configure make -j5 build make install clean
because all steps except make build are not compatible
with -jN (some ports don't work with -jN in the make build
phase either, but they are quite rare).
Now, is there a way to teach portmaster to build or rebuild
ports this way? The only workaround for now is something
like:
What I do is the following via make.conf, which will work for
portmaster/portupgrade or manual builds:
# set MAKE_ARGS for the build target(s)
.if !(make(*install) || make(package))
MAKE_ARGS+=-j8
.endif
Then as you find ports that don't build properly, add an entry like this:
# some ports don't like -j8, so we can undo the MAKE_ARGS addition for those
.if ${.CURDIR:M*/multimedia/mplayer}
MAKE_ARGS:=${MAKE_ARGS:C/-j8//}
.endif
It's a bit of a hack, but I've had decent success with this. Enough
ports fail to build with -jX, that I'd never do the above on a
production machine, especially since it's possible for some sort of
silent error that produces an unpredictable binary. But for my home
machine, I've been pretty happy with it.
Regards,
Josh
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Sun, Jan 25, 2009 at 05:22:04PM -0500, Josh Carroll wrote:
On Sun, Jan 25, 2009 at 4:44 PM, cpghost cpgh...@cordula.ws wrote:
To build ports in parallel on a 4 core machine, I usually
do this manually:
# cd /usr/ports/some/port
# make configure make -j5 build make install clean
because all steps except make build are not compatible
with -jN (some ports don't work with -jN in the make build
phase either, but they are quite rare).
Now, is there a way to teach portmaster to build or rebuild
ports this way? The only workaround for now is something
like:
What I do is the following via make.conf, which will work for
portmaster/portupgrade or manual builds:
# set MAKE_ARGS for the build target(s)
.if !(make(*install) || make(package))
MAKE_ARGS+=-j8
.endif
Then as you find ports that don't build properly, add an entry like this:
# some ports don't like -j8, so we can undo the MAKE_ARGS addition for those
.if ${.CURDIR:M*/multimedia/mplayer}
MAKE_ARGS:=${MAKE_ARGS:C/-j8//}
.endif
That's a good hint indeed. I'll try it. Maybe I'll modify it a bit to
include ONLY the build target, because -jN fails on nearly every
other target, AFAICS.
It's a bit of a hack, but I've had decent success with this. Enough
ports fail to build with -jX, that I'd never do the above on a
production machine, especially since it's possible for some sort of
silent error that produces an unpredictable binary. But for my home
machine, I've been pretty happy with it.
Yes, that's true, and it could happen. But I don't remember having
had a port compile, yet silently fail, yet. (*knocking-on-wood*). ;)
Regards,
Josh
Thanks,
-cpghost.
--
Cordula's Web. http://www.cordula.ws/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Sun, 25 Jan 2009 23:46:54 +0100 cpghost cpgh...@cordula.ws wrote: That's a good hint indeed. I'll try it. Maybe I'll modify it a bit to include ONLY the build target, because -jN fails on nearly every other target, AFAICS. Don't forget that there are two different makes; BSD make does the ports infrastructure bit, and usually gmake (Gnu Make) does the actual build and install in the work directory. IIRC MAKE_ARGS only goes to the lower-level make (gmake), and aside from those ports that don't work with -j at all, the failure with other targets are caused by bsd make i.e. by trying to run the whole ports infrastructure with -j. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
Josh Carroll wrote:
What I do is the following via make.conf,
I think this is a good solution. Given that incredible foot-shooting
power of the -j stuff I am not inclined to add something like this to
portmaster, not even as an advanced option.
Doug
which will work for
portmaster/portupgrade or manual builds:
# set MAKE_ARGS for the build target(s)
.if !(make(*install) || make(package))
MAKE_ARGS+=-j8
.endif
Then as you find ports that don't build properly, add an entry like this:
# some ports don't like -j8, so we can undo the MAKE_ARGS addition for those
.if ${.CURDIR:M*/multimedia/mplayer}
MAKE_ARGS:=${MAKE_ARGS:C/-j8//}
.endif
It's a bit of a hack, but I've had decent success with this. Enough
ports fail to build with -jX, that I'd never do the above on a
production machine, especially since it's possible for some sort of
silent error that produces an unpredictable binary. But for my home
machine, I've been pretty happy with it.
Regards,
Josh
--
This .signature sanitized for your protection
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: make -jN build with portmaster
On Sunday 25 January 2009 20:48:22 Doug Barton wrote: Josh Carroll wrote: What I do is the following via make.conf, I think this is a good solution. Given that incredible foot-shooting power of the -j stuff I am not inclined to add something like this to portmaster, not even as an advanced option. Given the fact that the build target is presumably -j safe (as far as the ports system is concerned), it would be nice to have a BUILD_JOBS in Mk/bsd.port.mk similar to INDEX_JOBS that is already there. Port maintainers then can also set WITHOUT_PARALLEL (or USE_PARALLEL=NO etc) for ports that break by themselves (f.e. www/lynx, editors/vim). portmaster should then have no problem setting BUILD_JOBS on request. -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
