On 6/21/10 5:34 AM, Stefan Bodewig wrote:
This would allow for a little more concurrency than we can do on the
Zone or VM... of course we'd have to be able to address all of those
cores. Wonder whether Python has glue for Grand Central Dispatch?
Most of the weight in gump runs is inside the java processes; the other
half of the latency is svn checkouts/updates.
For the former, you'd need (a) the JVM to hook up GCD (which is for
apple to do) and (b) maven to do more stuff in parallel (which is on the
charts for maven 3 I think).
For the latter, IIRC we have code to run more checkouts in parallel, but
the code is buggy in gump2; and would mean a load increase on the svn
server which may not be a good thing?
If you log into one of the machines while Gump is running, the system
feels sluggish and any opration that hits the file system takes ages
which makes me fear we are I/O bound rather than CPU bound - making
those cores do more may not help too much in that case. I can certainly
be wrong.
You're absolutely right. Builds are almost always I/O bound and you'll
see a lot of CPU is actually iowait -- so the numbers in top are usually
misleading and most of CPU busy-ness is due to overhead of waiting for IO.
IIRC Gump's trunk supports parallel SCM checkouts but we've restricted
it to a maximum of one updater because Adam saw problems - it's been a
long time.
Oh, eh, yep, so that's what I remember too :)
In any case to take advantage of multicores it'd be good to re-implement
parallelism using python's multiprocess module.
Currently we don't support building things in parallel at all. Starting
several Ant or make builds in parallel would likely do what you expect,
but I don't know how mvn would deal with multiple processes accessing
the same local repository (and writing to it) in parallel.
I don't think there's any special code for it in maven, but nevertheless
I've never really seen an issue with that. A common hudson deployment is
to run 4-5 builds in parallel using a 'hudson' user that has one local
repository.
cheers,
Leo
PS: no I'm not dead just persistently e-mail overloaded :)
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscr...@gump.apache.org
For additional commands, e-mail: general-h...@gump.apache.org
