Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 02 April 2011 23:47:42 Neil Bothwick wrote: Yes. the script that I use to start up and enter the chroot for each system not only does the usual mounting of /dev/ and /proc in the chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with the real target. Make.conf has to be maintained manually, because there are settings in the two that need to be different, although I suppose I could split out the common settings, USE, CHOST etc, into a separate file and source that. In my case the chroot is identical in structure to the real target, apart from the number of cores, so I can copy make.conf into the chroot without risk. to those on its target? And do you nfs-mount only the PKGDIR, or the whole of /usr/portage/ ? Just PKGDIR and DISTDIR, I have an NFS exported directory that contains a global DISTDIR and individual PKGDIRS, as well as my and layman's overlays. I'm hoping not to have to use any overlays here, mostly because the target box is going to be a LAN server, so shouldn't need any cutting-edge versions of anything. My setup mounts /usr/portage over nfs from the target; it's going to contain the latest tree for rsync'ing clients from, so it's the master version. Interesting - many thanks. It's all getting quite involved. -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Sun, 3 Apr 2011 14:55:39 +0100, Peter Humphrey wrote: Yes. the script that I use to start up and enter the chroot for each system not only does the usual mounting of /dev/ and /proc in the chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with the real target. Make.conf has to be maintained manually, because there are settings in the two that need to be different, although I suppose I could split out the common settings, USE, CHOST etc, into a separate file and source that. In my case the chroot is identical in structure to the real target, apart from the number of cores, so I can copy make.conf into the chroot without risk. You probably don't want EMERGE_DEFAULT_OPTS=--usepkg in the chroot's make.conf. I also turn off the ELOG* functions in the chroot, as the emails it sends contain the wrong hostname, leading to much confusion. -- Neil Bothwick / For security reasons, all text in this mail is double-rot13 encrypted. / signature.asc Description: PGP signature
Re: [gentoo-user] distcc and crossdev, anyone?
On Sunday 03 April 2011 18:08:25 Neil Bothwick wrote: You probably don't want EMERGE_DEFAULT_OPTS=--usepkg in the chroot's make.conf. In fact I don't have it in either of them; so far I've been issuing manual parameters. When I've settled the process down I'll encapsulate it in scripts. I also turn off the ELOG* functions in the chroot, as the emails it sends contain the wrong hostname, leading to much confusion. Good idea. Logging isn't working for me yet either, but with any luck it will be. Thanks again -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Sunday 03 April 2011 18:24:51 Peter Humphrey wrote: Logging isn't working for me yet either, I should have said that e-mailing of logs isn't working. -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Fri, 1 Apr 2011 12:43:44 +0100, Peter Humphrey wrote: I've decided to revert to Neil's method (once I've shaken this infection off). Could you please confirm that the infection is in no way linked to me or my method :-O -- Neil Bothwick Always proofread carefully to see if you any words out. signature.asc Description: PGP signature
Re: [gentoo-user] distcc and crossdev, anyone?
Neil Bothwick wrote: On Fri, 1 Apr 2011 12:43:44 +0100, Peter Humphrey wrote: I've decided to revert to Neil's method (once I've shaken this infection off). Could you please confirm that the infection is in no way linked to me or my method :-O I hope you get rid of the infection but don't send it this way. I don't need a trip to the hospital again. Having folks check on you is nice but I would prefer hotel room service to a hospital nurse and a pricey Dr. o_O Of course, I would rather spend the money on building a computer too. More fun than wondering if you are about to meet your maker. Dale :-) :-)
Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 02 April 2011 09:57:57 Neil Bothwick wrote: Could you please confirm that the infection is in no way linked to me or my method :-O Gladly. Not sure what it's linked to, nor even what it is, but it doesn't half sap the energy. -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. Just to confirm, and to save me having to think more deeply than I'm able pro tem, does each chroot have identical make.conf and package.use to those on its target? And do you nfs-mount only the PKGDIR, or the whole of /usr/portage/ ? -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Sat, 2 Apr 2011 16:19:45 +0100, Peter Humphrey wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. Just to confirm, and to save me having to think more deeply than I'm able pro tem, does each chroot have identical make.conf and package.use Yes. the script that I use to start up and enter the chroot for each system not only does the usual mounting of /dev/ and /proc in the chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with the real target. Make.conf has to be maintained manually, because there are settings in the two that need to be different, although I suppose I could split out the common settings, USE, CHOST etc, into a separate file and source that. to those on its target? And do you nfs-mount only the PKGDIR, or the whole of /usr/portage/ ? Just PKGDIR and DISTDIR, I have an NFS exported directory that contains a global DISTDIR and individual PKGDIRS, as well as my and layman's overlays. -- Neil Bothwick We can sympathize with a child who is afraid of the dark, but the tragedy of life is that most people are afraid of the light. signature.asc Description: PGP signature
Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: On Fri, 17 Dec 2010 22:56:29 +, Peter Humphrey wrote: I've bought (against my better judgement) an Atom N270 box to be a LAN server, but it's a bit slow compared with the other boxes on the network. A big bit, actually - 69 minutes to compile a kernel compared with less than 9 minutes on this workstation. I thought I'd give distcc a go, but after reading the Gentoo distcc and crossdev guides and doing what they say I get no result. I might just as well not have made the effort. The Atom box just labours with the emerge without trying to send anything to the server box I've set up for the purpose. I've found there's just too much overhead with distcc, plus much of the work is still done locally. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. I've been experimenting with nfs-mounting the whole Atom file system to /target in a chroot on my workstation, then setting --root=/target and --config- root=/target on every portage command. I can't recommend it. Numerous packages require to be installed into both the chroot and the target. I suppose that's not too onerous, even though I haven't found a way to predict which packages will be affected, but I've found that, when I go back to the Atom box and emerge -pkuv world, a lot of the packages that should already have been upgraded haven't been, and I have to emerge them on the Atom box directly. The states of the target and the native chroot are neither consistent nor independent - it's a mess. It was a nice idea to enable portage to work in this way, but it's still full of holes. Maybe all packages need some extra configuring; I don't know. A lot more work is definitely needed by someone, at any rate. I've decided to revert to Neil's method (once I've shaken this infection off). -- Rgds Peter
Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. I'd like to try this, but I haven't yet found the right set of parameters: either I'm not exporting the PKGDIR properly or my fstab isn't right. I've followed this guide: http://en.gentoo- wiki.com/wiki/NFS. I get this on the workstation when trying to nfs-mount the exported PKGDIR: # mount /mnt/nfs mount.nfs: an incorrect mount option was specified Is there a secret incantation? -- Rgds Peter. Linux Counter 5290, 1994-04-23.
Re: [gentoo-user] distcc and crossdev, anyone?
On Thursday 23 December 2010 14:44:25 I wrote: I get this on the workstation when trying to nfs-mount the exported PKGDIR: # mount /mnt/nfs mount.nfs: an incorrect mount option was specified The system log on vt12 says bad mount option value specified: vers=4. Ah-hah! I thought. All I have to do is add vers=3 to the fstab options and it should work. It did. Sorry about the noise. -- Rgds Peter. Linux Counter 5290, 1994-04-23.
Re: [gentoo-user] distcc and crossdev, anyone?
On Tue, Dec 21, 2010 at 1:56 AM, Bill Longman bill.long...@gmail.comwrote: On 12/18/2010 07:15 AM, Peter Humphrey wrote: On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I expected that but I wanted to try it to see. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. Maybe I'll go this way instead. Thanks for the idea, which is similar to one from YoYo Siska three days ago. I had my Atom 330 running as a distcc client for a long time. I have several other speedy CPUs alongside it so it could spray plenty of compilation requests out its gigabit NIC to various much beefier machines. But as Neil stated, lots of the processing still occurs locally, so as you increase nodes, you need to decrease the amount of compilation done locally. With such a disparity between CPU, it takes less time overall to just do it the way Neil describes - make a chroot and then just build it with the intention that the slow CPUs will use the binary build. You still need lots of CPU to compile, so a slow machine will still compile slowly. If your client is a pokey 1.6GHz Atom and you're sending jobs to two quad core 3GHz CPUs on your subnet, you'll soon see that the Atom's load goes up toward 8 as it tries to bring those remote jobs back. So, the four threads on my 330 get completely filled up and it's dog slow. And it's even more painful when you use the preprocessor because the client must zip the compile construction before it ships it out, so you have even less CPU available for compilation (although you get some of that back). All said and done, my back-of-the-napkin and seat-of-the-pants calculation tells me that I still get a _minimum_ 25% reduction in overall compile times with distcc. That's my experience after using distcc for almost ten years with various configurations of network and CPUs. I have set up my system as Neil described chroots for different systems on a fast computer. I use this setup for my gentoo boxes I have and it has made my compilations fast(er). I tried to use distcc with one U2300 celeron and some amd 4x cpu and the amd didn't really compile, because the U2300 was a bottleneck, so I decided to chroot it and been happy ever since. I have been thinking about a tool that could automagically start the emerge on the remote system. I thought about just ssh in with a script. But I am on so many flaky Internet connections that it isn't reliable enough. Petri
Re: [gentoo-user] distcc and crossdev, anyone?
On Sat, Dec 18, 2010 at 2:44 AM, Stroller strol...@stellar.eclipse.co.uk wrote: On 17/12/2010, at 10:56pm, Peter Humphrey wrote: ... an Atom N270 box ... server, but it's a bit slow compared with the other boxes on the network. A big bit, actually - 69 minutes to compile a kernel compared with less than 9 minutes on this workstation. 9 minutes!?!? I'm flabbergasted. The machines I have around here, I consider 1 hour to compile a kernel pretty good. Actually I'm in the process of migrating to newer hardware, but I haven't tested kernel compilation times. [brag] real1m46.250s user11m54.140s sys 0m57.290s [/brag] Less than 2 minutes here ;) That is for make -j9 all on Core i7 920 (OC'ed to 3.5GHz) To be more on topic, I've never been able to figure out distcc to the point where I feel comfortable that I've done it correctly. I have a laptop where emerging a new release of KDE takes more than 1 day, and the above mentioned workstation where it takes an hour. Followed the wiki and I could see compilation happening on the remote machine, but it was few and far between. It usually seemed like using it was slower than not using it at all. I tried to set it to just not use the local machine for anything but was never able to get that to work. (I'm not sure if it's even possible?) I probably did something wrong or misunderstood some fundamental part of it, but I gave up on it long ago.
Re: [gentoo-user] distcc and crossdev, anyone?
On 12/18/2010 07:15 AM, Peter Humphrey wrote: On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I expected that but I wanted to try it to see. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. Maybe I'll go this way instead. Thanks for the idea, which is similar to one from YoYo Siska three days ago. I had my Atom 330 running as a distcc client for a long time. I have several other speedy CPUs alongside it so it could spray plenty of compilation requests out its gigabit NIC to various much beefier machines. But as Neil stated, lots of the processing still occurs locally, so as you increase nodes, you need to decrease the amount of compilation done locally. With such a disparity between CPU, it takes less time overall to just do it the way Neil describes - make a chroot and then just build it with the intention that the slow CPUs will use the binary build. You still need lots of CPU to compile, so a slow machine will still compile slowly. If your client is a pokey 1.6GHz Atom and you're sending jobs to two quad core 3GHz CPUs on your subnet, you'll soon see that the Atom's load goes up toward 8 as it tries to bring those remote jobs back. So, the four threads on my 330 get completely filled up and it's dog slow. And it's even more painful when you use the preprocessor because the client must zip the compile construction before it ships it out, so you have even less CPU available for compilation (although you get some of that back). All said and done, my back-of-the-napkin and seat-of-the-pants calculation tells me that I still get a _minimum_ 25% reduction in overall compile times with distcc. That's my experience after using distcc for almost ten years with various configurations of network and CPUs.
Re: [gentoo-user] distcc and crossdev, anyone?
On 17/12/2010, at 10:56pm, Peter Humphrey wrote: ... an Atom N270 box ... server, but it's a bit slow compared with the other boxes on the network. A big bit, actually - 69 minutes to compile a kernel compared with less than 9 minutes on this workstation. 9 minutes!?!? I'm flabbergasted. The machines I have around here, I consider 1 hour to compile a kernel pretty good. Actually I'm in the process of migrating to newer hardware, but I haven't tested kernel compilation times. Nevertheless: it's a server. Open a `tmux` session, start it compiling, go watch a movie. Stroller.
Re: [gentoo-user] distcc and crossdev, anyone?
On Fri, 17 Dec 2010 22:56:29 +, Peter Humphrey wrote: I've bought (against my better judgement) an Atom N270 box to be a LAN server, but it's a bit slow compared with the other boxes on the network. A big bit, actually - 69 minutes to compile a kernel compared with less than 9 minutes on this workstation. I thought I'd give distcc a go, but after reading the Gentoo distcc and crossdev guides and doing what they say I get no result. I might just as well not have made the effort. The Atom box just labours with the emerge without trying to send anything to the server box I've set up for the purpose. I've found there's just too much overhead with distcc, plus much of the work is still done locally. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. -- Neil Bothwick Top Oxymorons Number 1: Microsoft Works signature.asc Description: PGP signature
Re: [gentoo-user] distcc and crossdev, anyone?
On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote: I've found there's just too much overhead with distcc, plus much of the work is still done locally. I expected that but I wanted to try it to see. I have a couple of Atom boxes, a server and a netbook, and I've set up a chroot for each on my workstation. In the chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to both computers, then I emerge -k on the Atom box. Maybe I'll go this way instead. Thanks for the idea, which is similar to one from YoYo Siska three days ago. -- Rgds Peter. Linux Counter 5290, 1994-04-23.
Re: [gentoo-user] distcc and crossdev, anyone?
On Friday 17 December 2010 23:23:10 Jacob Todd wrote:
Could you post your distcc config files?
$ extract /etc/conf.d/distccd
DISTCCD_OPTS=
DISTCCD_EXEC=/usr/bin/distccd
DISTCCD_PIDFILE=/var/run/distccd/distccd.pid
DISTCCD_OPTS=${DISTCCD_OPTS} --port 3632
DISTCCD_OPTS=${DISTCCD_OPTS} --log-level critical
DISTCCD_OPTS=${DISTCCD_OPTS} --allow 192.168.2.0/24
DISTCCD_OPTS=${DISTCCD_OPTS} --listen 192.168.2.2
DISTCCD_OPTS=${DISTCCD_OPTS} -N 15
(Extract is just a mini-script to cut out comments.)
$ cat /etc/distcc/hosts
ostn.ethnet
Ostn is the box that's supposed to do the compilation, but the Atom
client box just doesn't bother trying distcc. If it had and I had an
error in my config I'd have got an error message.
$ grep distcc /etc/make.conf
DISTCC_DIR=${PORTAGE_TMPDIR}/.distcc
FEATURES=buildpkg ccache distcc fixpackages parallel-fetch userfetch
Maybe another of those features is incompatible with distcc. I'd also
have expected an error message in that case, but I just get a bog-
standard emerge process running locally.
--
Rgds
Peter. Linux Counter 5290, 1994-04-23.
Re: [gentoo-user] distcc and crossdev, anyone?
Could you post your distcc config files?
Re: [gentoo-user] distcc and crossdev, anyone?
On Friday 17 December 2010 22:56:29 Peter Humphrey wrote: Hello list, I've bought (against my better judgement) an Atom N270 box to be a LAN server, but it's a bit slow compared with the other boxes on the network. A big bit, actually - 69 minutes to compile a kernel compared with less than 9 minutes on this workstation. I thought I'd give distcc a go, but after reading the Gentoo distcc and crossdev guides and doing what they say I get no result. I might just as well not have made the effort. The Atom box just labours with the emerge without trying to send anything to the server box I've set up for the purpose. Are the Gentoo guides up to date? Hi I have a N270 netbook and use crossdev and distcc. It is definately usefull but for doing a kernel compile I havn't tried it. Do you see any improvement when using for emerges. Whats your setup like Al
