Re: can make -j be used for ports?
On 16/03/2011 10:56, Matthias Andree wrote: /etc/make.conf: FORCE_MAKE_JOBS=yes MAKE_JOBS_NUMBER=3# whatever you find useful. The key is that you need to figure how loaded the CPU is. As long as it is waiting for the disks (which it usually is), you can try to increase the number, but keep an eye on the RAM, you don't want the machine to go swapping in and out (although 8 GibiB are plenty). Thanks very much for this. -- John ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
Am 15.03.2011 23:44, schrieb Chuck Swiger:
On Mar 15, 2011, at 3:35 PM, Eitan Adler wrote:
[ ... ]
Yes. Ports which support parallel builds will have MAKE_JOBS_SAFE=yes set in
the port Makefile. It defaults to running -j with MAKE_JOBS_NUMBER=`${SYSCTL}
-n kern.smp.cpus`, but you can change that to some other # if you like.
No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used
internally when building a single port.
What is incorrect?
When the OP is asking if he can manually specify -j on the command line which
would end up
building multiple ports in parallel. This can not be done (primarily
because there is no locking done on ports)
It certainly wasn't clear to me that this is what the OP meant. If you:
cd /usr/ports/www/apache22
make -j 3
make FORCE_MAKE_JOBS=yes
and the -j argument will default to the number of CPU cores as provided
by the corresponding sysctl. If the port breaks with
FORCE_MAKE_JOBS=yes please file a PR with a patch fixing it, or if that
is too much of an effort, a patch to add a line MAKE_JOBS_UNSAFE= yes
(on a single line with a TAB between MAKE_JOBS_UNSAFE= and yes).
...what do you expect to happen, and how many ports would you expect to be
built at once?
(Building one port in parallel is supported, where the port itself is safe to
do so; building many at the same time is not. Supporting the former provides
more speed gain for many situations as compared to the latter; which doesn't
help at all if you are just installing or updating one port.)
I beg to differ on the speed assertion. I own a somewhat fast computer
(Phenom II X4 i. e. 4 x 2.5 GHz) and I find that a lot of serialization
takes place, particularly behind the configure phases and the
mtree/registration stuff.
In that respect, building non-dependent ports in parallel would be
rather useful to actually exploit SMP computers; however it would
require a dependency analysis so that only those ports build in parallel
that do not depend on one another, and I also fear that sometimes mtree
during installation might get in the way.
I seem to recall that Doug stated something to the extent he wouldn't be
doing it in portmaster, and I don't see any other tool that is similarly
mature so it would be worthwhile even trying that.
Best
--
Matthias Andree
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
Am 16.03.2011 03:38, schrieb John: On 15/03/2011 22:35, Eitan Adler wrote: No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used internally when building a single port. When the OP is asking if he can manually specify -j on the command line which would end up building multiple ports in parallel. This can not be done (primarily because there is no locking done on ports) Actually, he has it partially right, at least in the idea I was trying to convey. I'll explain again, because maybe I wasn't coherent enough previously: I have an amd x2 6000+ with 8GB RAM. It's a wonderful fast desktop. Not the fastest, but it's fast enough for me. What I want to do is this: 1. If I can speed things up, with *ports* as I have a dual cpu, I want to maybe run j2 or j3. I seek clarification which is logically best, because some literature says jn, others jn+1 where n is number of cores. kern.smp.cpus: 2 on my machine. Is there benefit setting this to 3,4,5? I want to know if there is perhaps a conf file or sysctl where I can specify this *for ports only.* - if not I'm happy to specify on the command line. It's just that the manual is a tad unclear about this. /etc/make.conf: FORCE_MAKE_JOBS=yes MAKE_JOBS_NUMBER=3# whatever you find useful. The key is that you need to figure how loaded the CPU is. As long as it is waiting for the disks (which it usually is), you can try to increase the number, but keep an eye on the RAM, you don't want the machine to go swapping in and out (although 8 GibiB are plenty). On an i7-920 with a truckload of RAM I've sometimes even run make -j12 or so (on Linux build jobs though) to actually get the CPU 100% busy. 2. Where I was being unclear I think is when I was talking about building the system. I have seen in the literature that -jn when installing world Breaks Things. This isn't an issue as I run RELEASE and so only either apply patches or make the system, so easy to specify, where I can, -j3 except in make installkernel and make installworld. I've also found that make -j4 -DNOCLEAN buildworld buildkernel works for me on 8.2. If RELENG_8 occasionally failed, I re-ran without -DNOCLEAN but still with -j4. Looks like most of the issues with parallel building have been shaken out of the world :-) -- Matthias Andree ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
Is this possible or am I being unreasonable, or both, or not?
This is unsupported, but you are not being unreasonable. This is a
much wanted feature.
Yes. Ports which support parallel builds will have MAKE_JOBS_SAFE=yes set in
the port Makefile. It defaults to running -j with
MAKE_JOBS_NUMBER=`${SYSCTL} -n kern.smp.cpus`, but you can change that to
some other # if you like.
No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used
internally when building a single port. When the OP is asking if he
can manually specify -j on the command line which would end up
building multiple ports in parallel. This can not be done (primarily
because there is no locking done on ports)
Certain utilities can make this process faster. For example portmaster
prefetches as much as it can,
--
Eitan Adler
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
On Mar 15, 2011, at 3:35 PM, Eitan Adler wrote:
[ ... ]
Yes. Ports which support parallel builds will have MAKE_JOBS_SAFE=yes set
in the port Makefile. It defaults to running -j with
MAKE_JOBS_NUMBER=`${SYSCTL} -n kern.smp.cpus`, but you can change that to
some other # if you like.
No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used
internally when building a single port.
What is incorrect?
When the OP is asking if he can manually specify -j on the command line which
would end up
building multiple ports in parallel. This can not be done (primarily
because there is no locking done on ports)
It certainly wasn't clear to me that this is what the OP meant. If you:
cd /usr/ports/www/apache22
make -j 3
...what do you expect to happen, and how many ports would you expect to be
built at once?
(Building one port in parallel is supported, where the port itself is safe to
do so; building many at the same time is not. Supporting the former provides
more speed gain for many situations as compared to the latter; which doesn't
help at all if you are just installing or updating one port.)
Regards,
--
-Chuck
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
What is incorrect? MAKE_JOBS_SAFE uses make -j on a single port (when in the WRKSRC directory) but does *not* build multiple ports at the same time, It certainly wasn't clear to me that this is what the OP meant. If you: cd /usr/ports/www/apache22 make -j 3 ...what do you expect to happen, and how many ports would you expect to be built at once? What *would* happen is that make would try to build multiple ports at once. This is not supported. (Building one port in parallel is supported, where the port itself is safe to do so; building many at the same time is not. ) This is what I said. This feature is automatic and does not require -j specified. On Tue, Mar 15, 2011 at 5:44 PM, David Forsythe dfors...@freebsd.org wrote: I had patches for this a couple of years ago when I worked on this problem for summer of code. What I did back then is surely stale, but if people really want it, I'd be happy to take another stab at it. I've seen that work: it looked quite good. Yes, I think this would be very useful and I think a lot of people would want it :-). -- Eitan Adler ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
On Mar 15, 2011, at 3:51 PM, Eitan Adler wrote: It certainly wasn't clear to me that this is what the OP meant. If you: cd /usr/ports/www/apache22 make -j 3 ...what do you expect to happen, and how many ports would you expect to be built at once? What *would* happen is that make would try to build multiple ports at once. Did you actually try and see what happens? I'm thinking no...since, unless something is very different, you ought to find it terminating as follows: # cd /usr/ports/www/apache22 ; make -j 3 === apache-2.2.17_1 depends on file: /usr/local/lib/libcrypto.so.7 - found === apache-2.2.17_1 depends on file: /usr/local/bin/perl5.12.3 - found === apache-2.2.17_1 depends on file: /usr/local/bin/autoconf-2.68 - found === apache-2.2.17_1 depends on package: libtool=2.4 - found === apache-2.2.17_1 depends on shared library: expat.6 - found === apache-2.2.17_1 depends on shared library: apr-1 - found === apache-2.2.17_1 depends on shared library: pcre.0 - found === apache-2.2.17_1 depends on shared library: iconv.3 - found === apache-2.2.17_1 depends on shared library: mysqlclient.15 - found === apache-2.2.17_1 depends on shared library: db-4.4.0 - found === apache-2.2.17_1 depends on shared library: sqlite3.8 - found === Configuring for apache-2.2.17_1 ...whereas just typing make proceeds with: checking for chosen layout... FreeBSD checking for working mkdir -p... yes checking build system type... i386-portbld-freebsd7.4 checking host system type... i386-portbld-freebsd7.4 checking target system type... i386-portbld-freebsd7.4 [ ...followed by an actual build... ] Regards, -- -Chuck ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
I had patches for this a couple of years ago when I worked on this
problem for summer of code. What I did back then is surely stale, but
if people really want it, I'd be happy to take another stab at it.
On Tue, Mar 15, 2011 at 3:35 PM, Eitan Adler li...@eitanadler.com wrote:
Is this possible or am I being unreasonable, or both, or not?
This is unsupported, but you are not being unreasonable. This is a
much wanted feature.
Yes. Ports which support parallel builds will have MAKE_JOBS_SAFE=yes set
in the port Makefile. It defaults to running -j with
MAKE_JOBS_NUMBER=`${SYSCTL} -n kern.smp.cpus`, but you can change that to
some other # if you like.
No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used
internally when building a single port. When the OP is asking if he
can manually specify -j on the command line which would end up
building multiple ports in parallel. This can not be done (primarily
because there is no locking done on ports)
Certain utilities can make this process faster. For example portmaster
prefetches as much as it can,
--
Eitan Adler
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
--
David
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
On 15/03/2011 22:35, Eitan Adler wrote: No, this is incorrect. The MAKE_JOBS_NUMBER and MAKE_JOBS_SAFE is used internally when building a single port. When the OP is asking if he can manually specify -j on the command line which would end up building multiple ports in parallel. This can not be done (primarily because there is no locking done on ports) Actually, he has it partially right, at least in the idea I was trying to convey. I'll explain again, because maybe I wasn't coherent enough previously: I have an amd x2 6000+ with 8GB RAM. It's a wonderful fast desktop. Not the fastest, but it's fast enough for me. What I want to do is this: 1. If I can speed things up, with *ports* as I have a dual cpu, I want to maybe run j2 or j3. I seek clarification which is logically best, because some literature says jn, others jn+1 where n is number of cores. kern.smp.cpus: 2 on my machine. Is there benefit setting this to 3,4,5? I want to know if there is perhaps a conf file or sysctl where I can specify this *for ports only.* - if not I'm happy to specify on the command line. It's just that the manual is a tad unclear about this. 2. Where I was being unclear I think is when I was talking about building the system. I have seen in the literature that -jn when installing world Breaks Things. This isn't an issue as I run RELEASE and so only either apply patches or make the system, so easy to specify, where I can, -j3 except in make installkernel and make installworld. thanks, -- John li...@reiteration.net ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
1.
I want to know if there is perhaps a conf file or sysctl where I can
specify this *for ports only.* - if not I'm happy to specify on the
command line. It's just that the manual is a tad unclear about this.
Yes - simply install ports as normal:
specifically unless you specifically tell the ports system otherwise
you will be the maximum amount of supported parallelization.
.if defined(MAKE_JOBS_SAFE) || defined(FORCE_MAKE_JOBS)
MAKE_JOBS_NUMBER?= `${SYSCTL} -n kern.smp.cpus`
_MAKE_JOBS?=-j${MAKE_JOBS_NUMBER}
causes ports to be build with the proper number of jobs for your
system. If you wish to override this number change MAKE_JOBS_NUMBER.
DO NOT manually set -j when running make install from the ports
directory. It will only hurt,
--
Eitan Adler
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: can make -j be used for ports?
On Mar 15, 2011, at 7:38 PM, John wrote: 1. If I can speed things up, with *ports* as I have a dual cpu, I want to maybe run j2 or j3. I seek clarification which is logically best, because some literature says jn, others jn+1 where n is number of cores. kern.smp.cpus: 2 on my machine. Is there benefit setting this to 3,4,5? If you have plenty of RAM, then setting it somewhat higher than #CPUs is likely to improve performance. Frankly, it's best if you measure things on your own machine against the software you care about building, and decide for yourself whether the difference matters. :-) I want to know if there is perhaps a conf file or sysctl where I can specify this *for ports only.* - if not I'm happy to specify on the command line. It's just that the manual is a tad unclear about this. Setting MAKE_JOBS_NUMBER in /etc/make.conf will only apply to things using the bsd.ports.mk Makefile, unless they've chosen to honor the same variable independently. In other words, you're unlikely to hurt things much by setting it to some reasonable value. Obtain some data and let us know if you run into surprises... Regards, -- -Chuck ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
