Re: [distcc] about distcc's scheduling

Wed, 25 Nov 2009 01:13:17 -0800 

Francois Berenger wrote:

Fergus Henderson wrote:

Have you tried using pump mode?  That helps a lot.
Also, if the issue is that distcc isn't running enough jobs on each host, try using the "host/JOBS" format in DISTCC_HOSTS, e.g. DISTCC_HOSTS="foo.riken.jp/20,cpp,lzo <http://foo.riken.jp/20,cpp,lzo> bar.riken.jp/40,cpp,lzo <http://bar.riken.jp/40,cpp,lzo>". This would allow up to 20 concurrent compiles on host foo.riken.jp <http://foo.riken.jp> and 40 concurrent compiles on host bar.riken.jp <http://bar.riken.jp>. 
(The ",cpp,lzo" part is required for pump mode.)


Thanks a lot! I am trying this right away.
I'll post my speedup results when I'm done.


Well, I managed to make our soft compile with:
distcc-server-3.1-1
distcc-3.1-1

However, pump seems to be for 32bit machines only:
t...@toto:x# pump scons -j24 bin mode=release
__________Using distcc-pump from /usr/bin
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/include_server/include_server.py", line 42, in ? 
    import distcc_pump_c_extensions
ImportError: /usr/lib/python2.4/site-packages/include_server/distcc_pump_c_extensions.so: wrong ELF class: ELFCLASS32 
__________Expected a socket at '/tmp/distcc-pump.E15442/socket'
__________Could not start distcc-pump include server
__________Warning: distcc-pump include server failed; running unpumped
__________Found 0 available distcc servers
Can someone publish x86_64 binary packages in the download section of the distcc website? 
Or at least the source RPM so I can to build for my systems.

Thanks a lot,
Francois.
At some point, depending on the size and structure of the software that you're trying to compile, and the way your Makefiles are arranged, you may run into limitations in the parallelizability of your build graph. For example, if your software has only three source files, then distcc won't be able to speed up compilation by more than a factor of three at very best. Similarly, if your software has 30 files, but they are split into 10 directories of three files each, and your Makefile(s) sequentialize the compilation of each directory, then you still won't get more than a factor of three speed up at very best.
We use scons. I am not so happy with it (like when it spends 30s before giving me back control of the command line when theres is _nothing_ to compile because everything is already done). However, it manages the dependencies far better than any Makefile I could have written by hand. 

Regards,
Francois.
On Mon, Nov 23, 2009 at 8:19 PM, Francois Berenger <[email protected] <mailto:[email protected]>> wrote: 

    Hello,

    Is there a way to use a pull scheduler instead of a push scheduler
    inside distcc?
I have many compiling hosts and when they join a compiling job, they are 
    not enough loaded, at least to my taste. Meaning: they could compile
    more things and I could wait less for them to finish.

    I think the problem is that the scheduler is pushing: the distcc
    server sends jobs to distcc compiling hosts.

    Instead, I would like this behavior: distcc compiling hosts ask and
    take jobs as long as they are not completely loaded with compilation
    jobs.
    The distcc server should distribute jobs to whom asks for.

    This should scale better.
If there is no way now to do so but you can give me some hint on how to 
    do so (where to look, what to modify, etc.), it might be helpful.

    Best regards,
    Francois.

    __
    distcc mailing list            http://distcc.samba.org/
    To unsubscribe or change options:
    https://lists.samba.org/mailman/listinfo/distcc




--
Fergus Henderson <[email protected] <mailto:[email protected]>>


__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc


__
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc

Reply via email to