Re: speed up port compiling using RAM (tmpfs) ???
Wesley Shields <[EMAIL PROTECTED]> writes: > I think he is trying to get at a scenario where WRKDIR is on a seperate > disk from the one /usr/ports is on. There is no performance advantage in doing that. I can only see two possible reasons for pointing WRKDIRPREFIX to another disk: - insufficient space on /usr (but then again, you could just move the ports tree to that other disk and mount it on /usr/ports) - /usr/ports on NFS (I do that myself to facilitate testing changes on different platforms) DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Weird PCI interrupt delivery problem (resolution, sort of)
On Thu, Jan 19, 2006 at 10:17:39PM -0700, Scott Long wrote: > This points to a bus coherency problem. I wonder if your BIOS is > incorrectly setting the memory region of the apics as cachable. You'll > want to bug Baldwin about this. I CC-ed him on my post since he was working with me on the problem before. For some reason the Cc: header got wiped out when it went to the list (but I checked my server logs and it did deliver a copy of the message to him). Craig ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: On Thu, Jan 19, 2006 at 05:32:58PM -0500, Gary Thorpe wrote: Ashok Shrestha wrote: I mounted part of RAM as such: mdmfs -s 500m md /mnt Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf. It substantially reduces compile time by about 5-10 times. Thanx to all ur replies. -Ashok Shrestha An alternative is to try using the "-pipe" flag with GCC: this eliminates the need to use some temporary files by using a unix pipe for IPC. Setting another flag "-j 2" will allow 2 jobs to be done at the same time and should eliminate (by hiding) the I/O delays (this slows down the CPU slightly because of more task switches, but even with one job there are still lots of generated tasks). -j is not safe to use with port builds since many ported software contain race conditions in the build. Kris This effectively means that you cannot take advantage of SMP to compile FreeBSD's ports collection. That sounds like a big limitation...especially for people trying to speed up bulk builds. Is "-pipe" still a good idea? It should provide a significant performance boost, if the process is spending a lot of time doing disk I/O, without using up a lot of memory. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Gary Thorpe <[EMAIL PROTECTED]> writes: > This effectively means that you cannot take advantage of SMP to > compile FreeBSD's ports collection. That sounds like a big > limitation...especially for people trying to speed up bulk builds. We cannot be held responsible for race conditions in the Makefiles of third-party software. > Is "-pipe" still a good idea? It should provide a significant > performance boost, if the process is spending a lot of time doing disk > I/O, without using up a lot of memory. -pipe is fine. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Re: speed up port compiling using RAM (tmpfs) ???
>From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> >Gary Thorpe <[EMAIL PROTECTED]> writes: >> This effectively means that you cannot take advantage of SMP to >> compile FreeBSD's ports collection. That sounds like a big >> limitation...especially for people trying to speed up bulk builds. > >We cannot be held responsible for race conditions in the Makefiles of >third-party software. Well, maybe we can then build multiple ports in parallel. I guess the way to do it would be to run the top-level make with -j but then disable it when calling the makefiles of the individual ports. Not that I have any idea how to actually do that. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > >> This effectively means that you cannot take advantage of SMP to > >> compile FreeBSD's ports collection. That sounds like a big > >> limitation...especially for people trying to speed up bulk builds. > > > >We cannot be held responsible for race conditions in the Makefiles of > >third-party software. > > Well, maybe we can then build multiple ports in parallel. > I guess the way to do it would be to run the top-level make with > -j but then disable it when calling the makefiles of the > individual ports. Not that I have any idea how to actually > do that. It's harder than that, because you need to impose dependency information and mutual exclusion between different makes. e.g. they can't both be compiling the same port at the same time, which will happen if you just do the naive thing. Kris pgp6U8a9JihqQ.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > >-j is not safe to use with port builds since many ported software > >contain race conditions in the build. > > > >Kris > > This effectively means that you cannot take advantage of SMP to compile > FreeBSD's ports collection. That sounds like a big > limitation...especially for people trying to speed up bulk builds. Yeah, but what do you propose to do about it? We have 14000 ports that need to be inspected for build race conditions and fixed before you can turn on -j by default. Kris pgpbkaEytid09.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, 2006-Jan-20 14:47:00 -0500, Kris Kennaway wrote: >On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > >> >-j is not safe to use with port builds since many ported software >> >contain race conditions in the build. >> > >> >Kris >> >> This effectively means that you cannot take advantage of SMP to compile >> FreeBSD's ports collection. That sounds like a big >> limitation...especially for people trying to speed up bulk builds. > >Yeah, but what do you propose to do about it? We have 14000 ports >that need to be inspected for build race conditions and fixed before >you can turn on -j by default. There are some areas of the FreeBSD port building infrastructure that could be enhanced to make it possible to run multiple top-level makes in parallel, even if individual ports can't be build with '-j'. IMHO, the biggest problem (as des pointed out) is that there's nothing to prevent two makes attempting to build the same port (this can easily happen when both ports A and B depend on port C). One possible solution would be to create another status file in the work directory and hold a file lock (flock/lockf) on it whilst a make is in progress. Any parallel attempt to make that port would block. If the above was implemented, an enhancement would then be to process the port's dependency list in parallel, rather than serially. This would allow a make that was blocked on one dependency to continue with a different dependency. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sat, Jan 21, 2006 at 07:52:20AM +1100, Peter Jeremy wrote:5C > IMHO, the biggest problem (as des pointed out) is that there's nothing > to prevent two makes attempting to build the same port (this can > easily happen when both ports A and B depend on port C). One possible > solution would be to create another status file in the work directory > and hold a file lock (flock/lockf) on it whilst a make is in progress. > Any parallel attempt to make that port would block. > > If the above was implemented, an enhancement would then be to process > the port's dependency list in parallel, rather than serially. This > would allow a make that was blocked on one dependency to continue with > a different dependency. There have been a couple of half-hearted attempts to implement this, but no complete ones (see an old portmgr PR for details + discussion). I'd welcome a complete version :) Kris pgptDcR8KK7mK.pgp Description: PGP signature
Re: Weird PCI interrupt delivery problem (resolution, sort of)
On Friday 20 January 2006 10:27, Craig Boston wrote: > On Thu, Jan 19, 2006 at 10:17:39PM -0700, Scott Long wrote: > > This points to a bus coherency problem. I wonder if your BIOS is > > incorrectly setting the memory region of the apics as cachable. You'll > > want to bug Baldwin about this. > > I CC-ed him on my post since he was working with me on the problem > before. For some reason the Cc: header got wiped out when it went to > the list (but I checked my server logs and it did deliver a copy of the > message to him). Hmm, well, you can actually try the PAT patch if you are feeling brave as it maps all devices (including APICs) as uncacheable. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Weird PCI interrupt delivery problem (resolution, sort of)
On Fri, Jan 20, 2006 at 03:42:21PM -0500, John Baldwin wrote: > Hmm, well, you can actually try the PAT patch if you are feeling brave as it > maps all devices (including APICs) as uncacheable. Heh, took me a minute to find. I first found the one at http://people.freebsd.org/~jhb/patches/pat.patch but it maps devices as write-back. I'm guessing you mean to use the version in perforce? I'll give it a try tonight. Could hardy make things worse -- I just noticed that X now randomly locks up hard, ever since I bumped up the memory from 256Mb to 2G -- though text mode still works fine. (yes, I tried reverting all my local patches and testing the memory) Craig ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > >> This effectively means that you cannot take advantage of SMP to > > >> compile FreeBSD's ports collection. That sounds like a big > > >> limitation...especially for people trying to speed up bulk builds. > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > >third-party software. > > > > Well, maybe we can then build multiple ports in parallel. > > I guess the way to do it would be to run the top-level make with > > -j but then disable it when calling the makefiles of the > > individual ports. Not that I have any idea how to actually > > do that. > > It's harder than that, because you need to impose dependency > information and mutual exclusion between different makes. e.g. they > can't both be compiling the same port at the same time, which will > happen if you just do the naive thing. That's the part that "make -j" is supposed to take care of, since it should build in parallel only the targets independent of each other. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 04:54:33PM -0500, Sergey Babkin wrote: > Kris Kennaway wrote: > > > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > > >> This effectively means that you cannot take advantage of SMP to > > > >> compile FreeBSD's ports collection. That sounds like a big > > > >> limitation...especially for people trying to speed up bulk builds. > > > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > > >third-party software. > > > > > > Well, maybe we can then build multiple ports in parallel. > > > I guess the way to do it would be to run the top-level make with > > > -j but then disable it when calling the makefiles of the > > > individual ports. Not that I have any idea how to actually > > > do that. > > > > It's harder than that, because you need to impose dependency > > information and mutual exclusion between different makes. e.g. they > > can't both be compiling the same port at the same time, which will > > happen if you just do the naive thing. > > That's the part that "make -j" is supposed to take care of, > since it should build in parallel only the targets independent > of each other. If (as I said) you impose the correct dependency information. Currently there is no such information provided. kris pgpof8ZmKKyl2.pgp Description: PGP signature
RE: speed up port compiling using RAM (tmpfs) ???
> [mailto:[EMAIL PROTECTED] On Behalf Of Kris Kennaway > Sent: Friday, January 20, 2006 11:47 AM > To: Gary Thorpe > Cc: Wesley Shields; Ashok Shrestha; Brandon Flowers; Kris > Kennaway; Mike Meyer; freebsd-hackers@freebsd.org; Dag-Erling Sm?rgrav > Subject: Re: speed up port compiling using RAM (tmpfs) ??? > > > On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > > > >-j is not safe to use with port builds since many ported software > > >contain race conditions in the build. > > > > > >Kris > > > > This effectively means that you cannot take advantage of SMP to compile > > FreeBSD's ports collection. That sounds like a big > > limitation...especially for people trying to speed up bulk builds. > > Yeah, but what do you propose to do about it? We have 14000 ports > that need to be inspected for build race conditions and fixed before > you can turn on -j by default. > > Kris With a dual processor (even dual core?) is there a way to assign the big make to one processor and to assign the foreground activity (email, edits, etc) to another? Is SMP somehow smart enough to do this automagically? -gayn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 03:28:08PM -0800, Gayn Winters wrote: > > [mailto:[EMAIL PROTECTED] On Behalf Of Kris Kennaway > > Sent: Friday, January 20, 2006 11:47 AM > > To: Gary Thorpe > > Cc: Wesley Shields; Ashok Shrestha; Brandon Flowers; Kris > > Kennaway; Mike Meyer; freebsd-hackers@freebsd.org; Dag-Erling Sm?rgrav > > Subject: Re: speed up port compiling using RAM (tmpfs) ??? > > > > > > On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > > > > > >-j is not safe to use with port builds since many ported software > > > >contain race conditions in the build. > > > > > > > >Kris > > > > > > This effectively means that you cannot take advantage of SMP to > compile > > > FreeBSD's ports collection. That sounds like a big > > > limitation...especially for people trying to speed up bulk builds. > > > > Yeah, but what do you propose to do about it? We have 14000 ports > > that need to be inspected for build race conditions and fixed before > > you can turn on -j by default. > > > > Kris > > > With a dual processor (even dual core?) is there a way to assign the big > make to one processor and to assign the foreground activity (email, > edits, etc) to another? Is SMP somehow smart enough to do this > automagically? Yes, this is what will happen, more or less. Kris pgpZMaai7Amfh.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, 2006-Jan-20 16:54:33 -0500, Sergey Babkin wrote: >Kris Kennaway wrote: >> It's harder than that, because you need to impose dependency >> information and mutual exclusion between different makes. e.g. they >> can't both be compiling the same port at the same time, which will >> happen if you just do the naive thing. > >That's the part that "make -j" is supposed to take care of, >since it should build in parallel only the targets independent >of each other. It doesn't quite work like that. A single make execution will correctly parallelize independent targets (as long as the makefile dependency tree is correct - which is not true for all ports). The problem occurs when the targets are sub-makes. In this case, you have multiple sub-makes running in parallel with no knowledge of the dependency trees of the other sub-makes. Unless the top-level makefile has full knowledge of all the dependencies (which is not practical), it is quite likely that the sub-makes will collide. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: > > On Fri, Jan 20, 2006 at 04:54:33PM -0500, Sergey Babkin wrote: > > Kris Kennaway wrote: > > > > > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > > > >> This effectively means that you cannot take advantage of SMP to > > > > >> compile FreeBSD's ports collection. That sounds like a big > > > > >> limitation...especially for people trying to speed up bulk builds. > > > > > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > > > >third-party software. > > > > > > > > Well, maybe we can then build multiple ports in parallel. > > > > I guess the way to do it would be to run the top-level make with > > > > -j but then disable it when calling the makefiles of the > > > > individual ports. Not that I have any idea how to actually > > > > do that. > > > > > > It's harder than that, because you need to impose dependency > > > information and mutual exclusion between different makes. e.g. they > > > can't both be compiling the same port at the same time, which will > > > happen if you just do the naive thing. > > > > That's the part that "make -j" is supposed to take care of, > > since it should build in parallel only the targets independent > > of each other. > > If (as I said) you impose the correct dependency information. > Currently there is no such information provided. Ah, so we don't have any reliable information about dependencies between the ports either (not just between files inside each particular port)? Hm, I think it would present a problem even when building them sequentially. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 08:36:17PM -0500, Sergey Babkin wrote: > > If (as I said) you impose the correct dependency information. > > Currently there is no such information provided. > > Ah, so we don't have any reliable information about dependencies > between the ports either (not just between files inside each > particular port)? Hm, I think it would present a problem even > when building them sequentially. The port dependency checking of one port upon other ports doesn't really use the make(1) dependency system. Instead there is a shell script in bsd.port.mk that looks for the file listed in the *_DEPENDS variable and goes and explicitly spawns a new make to build the port if it is not found. This is fine for building ports serially, but it's not trivially parallelizable since it only has local knowledge: In order to do better you either have to: 1a) construct an enormous make(1) dependency web of the entire ports collection (in order to have global knowledge and be able to direct makes to avoid collisions), and 1b) somehow keep it up-to-date each time the ports tree is modified. or 2) Introduce mutual exclusion to prevent conflicts from two makes trying to operate on the same port at once. You also need to serialize in other places, e.g. modifying the database state in /var/db. This won't be as efficient as 1) since the second make will block instead of going off to build some other dependency in the meantime, which you could do with 1) if you could manage to implement it. Kris pgp6LclSDGMdd.pgp Description: PGP signature
Re: Config(8) dependency checking - first patches
: Notes and patches against 7-CURRENT are at : http://www.gsicomp.on.ca/projects/freebsd/configdep.html. How would you encode ed's dependencies? The ed driver only depends on mii when built with pccard enabled. Warner ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
