Re: [Chandler-dev] Speeding up full builds

[Mike Taylor]

I almost never use -j when building other people's code (i.e. external libraries) because you don't know what portions of the code base can be safely built in parallel. It's tricky to do it with your own code and requires some serious testing to make sure everything has been assembled properly.

On Oct 31, 2006, at 1:37 AM, Heikki Toivonen wrote:

The other items you mentioned seem to be easy enough fixes (heck I've already implemented them :) but I have to wonder if we are looking at the right problem? Currently the full build only really rebuilds the whole set of code if some error happens during testing (because we do a make realclean). Is the majority of time spent during a "normal" build run in actual building or is it in running the tests twice? I'm thinking we need to clock different parts of the full build so we can find out where the slow spots are.

I realized that I have gathered quite a few ideas on how to speed up the full tinderboxes (or full builds actually), many of them when discussing

the problems with various people.

One reason of slow builds are simply bugs where we do the same thing
over and over when one time would do. For example, downloading some of
the sources when we already have them is taking unnecessary time.

We also do some operations in not so smart ways. For example, doing two
passes over some source directory to find and *.pyc and *.pyo and then
pipe the lists to rm - we could simply use a single find command.


system libraries instead of building them ourselves. For example, I've
already tested that I can build and run Chandler fine on Ubuntu using
the system ICU and OpenSSL libraries. (Chandler seemed to build and run
on the Mac with system OpenSSL but I got a scary-looking warning.)

make has the often overlooked -j switch to split the work into multiple

problem is that it can be tricky getting the build actually work. I
already know that full build won't work, but clean, sources and expand

could be built with -j even if we can't use it for all of them. The
problem with -j is also that if there is a build failure, trying to
decipher the build log is a nightmare (I found that using -j 2 gave me

the log reasonably readable).

I also know of distributed compiling, but I don't have any personal
experience with it.

Anyone know of other tips and tricks we could use, or has experience
with -j or distributed compiling?

--
  Heikki Toivonen


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev

Especially on Linux, but perhaps to some extent on Mac, we could utilize jobs. Especially on a multi-cpu system this can be a big time saver. The targets seem to work, and possibly some of the external libraries we use almost all of the -j benefits on my dual core system while still leaving

---
Bear

Build and Release Engineer
Open Source Applications Foundation (OSAF)
[EMAIL PROTECTED]
http://www.osafoundation.org

[EMAIL PROTECTED]
http://code-bear.com

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29

PGP.sig
Description: This is a digitally signed message part

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev