Re: Cocoon for Forrest (was Re: A Few Plans)
Stefan Bodewig wrote: David Crossley wrote: Stefan Bodewig wrote: - ask the Forrest and Lenya communities which parts of Cocoon 2.x they need, remove the Cocoon reactor and only build the parts needed, finally start building Forrest and Lenya again. For Forrest, we use Cocoon-2.1 branch (i.e. the Ant-based build) and various blocks. Is there still any development on the 2.1 branch at all (bug fixes, whatever) It is sporadic. There has been some desire expressed on the Cocoon dev list to keep maintaining it. However only some commit activity has followed. or would it make sense to consider it stale and just package up the jars Forrest needs? Perhaps we could do that until there is further activity at Cocoon. Would someone please refer me to a useful example of a packaged gump project. Where would the packaged jars be stored? If we want to keep building Cocoon, can we identify the blocks that are needed and restrict the build to those? Yes. We would need the core, and then Forrest has the blocks listed in our existing gump descriptor (but commented out). By the way, so that people are clear. The current Gump run happening for cocoon is the Maven-based build of Cocoon trunk (i.e. Cocoon-2.2) and there is not yet any Gump run for the cocoon3. -David - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: Parallelism (was Re: A Few Plans)
On 6/21/10 5:34 AM, Stefan Bodewig wrote: This would allow for a little more concurrency than we can do on the Zone or VM... of course we'd have to be able to address all of those cores. Wonder whether Python has glue for Grand Central Dispatch? Most of the weight in gump runs is inside the java processes; the other half of the latency is svn checkouts/updates. For the former, you'd need (a) the JVM to hook up GCD (which is for apple to do) and (b) maven to do more stuff in parallel (which is on the charts for maven 3 I think). For the latter, IIRC we have code to run more checkouts in parallel, but the code is buggy in gump2; and would mean a load increase on the svn server which may not be a good thing? If you log into one of the machines while Gump is running, the system feels sluggish and any opration that hits the file system takes ages which makes me fear we are I/O bound rather than CPU bound - making those cores do more may not help too much in that case. I can certainly be wrong. You're absolutely right. Builds are almost always I/O bound and you'll see a lot of CPU is actually iowait -- so the numbers in top are usually misleading and most of CPU busy-ness is due to overhead of waiting for IO. IIRC Gump's trunk supports parallel SCM checkouts but we've restricted it to a maximum of one updater because Adam saw problems - it's been a long time. Oh, eh, yep, so that's what I remember too :) In any case to take advantage of multicores it'd be good to re-implement parallelism using python's multiprocess module. Currently we don't support building things in parallel at all. Starting several Ant or make builds in parallel would likely do what you expect, but I don't know how mvn would deal with multiple processes accessing the same local repository (and writing to it) in parallel. I don't think there's any special code for it in maven, but nevertheless I've never really seen an issue with that. A common hudson deployment is to run 4-5 builds in parallel using a 'hudson' user that has one local repository. cheers, Leo PS: no I'm not dead just persistently e-mail overloaded :) - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: A Few Plans
Stefan Bodewig wrote: - ask the Forrest and Lenya communities which parts of Cocoon 2.x they need, remove the Cocoon reactor and only build the parts needed, finally start building Forrest and Lenya again. For Forrest, we use Cocoon-2.1 branch (i.e. the Ant-based build) and various blocks. -David - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Re: A Few Plans
On Jun 18, 2010, at 10:15 AM, Stefan Bodewig wrote: Hi, I just wanted to share a few plans I have short/midterm. Feel free to comment, pick tasks or add wishes. Honestly I have no idea how we could deal with the ever increasing build times as Gump grows, apart from some sort of distributed Gump which I wouldn't want to build on top of the current code base (I'd rather think in a tuple spaces architecture like Mnesia and Erlang or JavaSpaces and anything on the VM). When I come back from vacation, we'll be pressing into service a dual quad core Apple Xserve with 6Gb of memory. This would allow for a little more concurrency than we can do on the Zone or VM... of course we'd have to be able to address all of those cores. Wonder whether Python has glue for Grand Central Dispatch? S. mvnrepoproxy: - move the mapping of URL prefix to real repository into a properties file passed in on the command line. This way we can maintain the list of repositories we proxy in a single place which would create the properties and mvn settings files. Python: - go further with generalizing outputs, add POM as a type of output - allow outputs of different types to share the same id (jar and POM for example) - only publish POMs and JARs to the proxy - maybe add a mvn-install builder to simplify things like the camel-pom where the ids could be taken from the project/workspace and the version might even be read from the POM directly. - maybe add a RAT builder - generalize junitreport so it can be used to publish other types of reports (like RAT's) Metadata: - ask the Forrest and Lenya communities which parts of Cocoon 2.x they need, remove the Cocoon reactor and only build the parts needed, finally start building Forrest and Lenya again. - regularly publish POMs with the JARs so depending projects will pick up the new POMs as well as the new code. - split the Axis2 reactor. Reactor builds are simpler to maintain but fail for too many reasons IMHO. - the next logical projects to add (judging from dependencies downloaded by Maven) would be XBean (somewhere in Geronimo) and Felix. I'm afraid Felix and OSGi would mean a whole new can of worms, though. - try to get some communities engaged again, in particular it would be nice if we could get the Velocity and Portals communities to sort out the Velocity bridge problem. I'm not sure I have the energy required to do that right now. Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org -- Sander Temme scte...@apache.org PGP FP: FC5A 6FC6 2E25 2DFD 8007 EE23 9BB8 63B0 F51B B88A - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Cocoon for Forrest (was Re: A Few Plans)
On 2010-06-20, David Crossley wrote: Stefan Bodewig wrote: - ask the Forrest and Lenya communities which parts of Cocoon 2.x they need, remove the Cocoon reactor and only build the parts needed, finally start building Forrest and Lenya again. For Forrest, we use Cocoon-2.1 branch (i.e. the Ant-based build) and various blocks. Is there still any development on the 2.1 branch at all (bug fixes, whatever) or would it make sense to consider it stale and just package up the jars Forrest needs? If we want to keep building Cocoon, can we identify the blocks that are needed and restrict the build to those? Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
Parallelism (was Re: A Few Plans)
On 2010-06-20, Sander Temme wrote: On Jun 18, 2010, at 10:15 AM, Stefan Bodewig wrote: Hi, I just wanted to share a few plans I have short/midterm. Feel free to comment, pick tasks or add wishes. Honestly I have no idea how we could deal with the ever increasing build times as Gump grows, apart from some sort of distributed Gump which I wouldn't want to build on top of the current code base (I'd rather think in a tuple spaces architecture like Mnesia and Erlang or JavaSpaces and anything on the VM). When I come back from vacation, we'll be pressing into service a dual quad core Apple Xserve with 6Gb of memory. Sounds great - enjoy your vacation. This would allow for a little more concurrency than we can do on the Zone or VM... of course we'd have to be able to address all of those cores. Wonder whether Python has glue for Grand Central Dispatch? Right now Gump is a controlling process that spins of new processes, so doing more in parallel would mean run those processes in parallel - nothing Python would need to support. If you log into one of the machines while Gump is running, the system feels sluggish and any opration that hits the file system takes ages which makes me fear we are I/O bound rather than CPU bound - making those cores do more may not help too much in that case. I can certainly be wrong. IIRC Gump's trunk supports parallel SCM checkouts but we've restricted it to a maximum of one updater because Adam saw problems - it's been a long time. Currently we don't support building things in parallel at all. Starting several Ant or make builds in parallel would likely do what you expect, but I don't know how mvn would deal with multiple processes accessing the same local repository (and writing to it) in parallel. It may be possible to construct concurrency in a way that is more or less safe so that long running and self-contained builds like test-ant could be spun off but all mvn builds that accessed the same local repo would get serialized. Of course it would take somebody to write the code 8-) Stefan - To unsubscribe, e-mail: general-unsubscr...@gump.apache.org For additional commands, e-mail: general-h...@gump.apache.org
