Ok, here are some of the details I promised. Cc'ing Continuum as as
much of this code as possible is destined to them. The idea is that
we only should need to maintain a few extensions (which I spelled as
extention in the code, oops) for the things that are specific to us
or they don't want.
BASIC PARTS
--------------------------------------
Executing Agent (runs builds)
Producing Agent (creates builds)
Results Agent (aggregates build results)
Each agent consumes or produces on a queue or topic (or both). You
can have as many of each as you need or what. They can run in the
same process or separate processes.
All messages are Maps that are serialized to ObjectMessages onto
either the BUILD.TASKS queue or the BUILD.RESULTS topic.
Excecuting Agent
-------------------
Runs continuums various continuum componentry and "update-from-scm"
and "execute-builder" actions. When this is done, the code has been
updated and built. Has extendible pre and post checking of incoming
builds and outgoing build results. Specifically there is a
BuildAgentExtension (when spelling is fixed) that has a pre(Map
incoming) method and a post(Map incoming, Map outgoing) method.
Implemented Extensions so far:
- FileIncludeExtension: Checks the build request for a
configurable header format. Any headers matching that format are
assumed to point to files that should be read in and included as text
in the response. The file must be one under the project's work
directory.
- HeaderIncludeExtension: Any headers that match the configured
pattern are copied as-is from the build request to the response message.
Producing Agent
-------------------
Scans a directory for properties files that define builds to add to
the build queue. This part could use some work. The implementation
that uses properties files is fine for shoving lots of work onto a
queue in a really easy way, but this is where we also need some more
integration with continuum as a frontend. An implementation of A
Producing Agent could be an alternate implementation of the Continuum
build queue that just turns the build tasks into items that go onto
the jms build queue. Once you do something like that, you need to
consider how things work in Continuum in regards to it's single
threaded nature that seems to be assumed in places of the code.
Results Agent
-------------------
Very bare bones agent that consumes from teh BUILD.RESULTS topic and
invokes BuildResultsExtensions (when spelling is fixed). Performs no
other work aside from invoking the extensions. We can really go nuts
with extensions here. No limit to the ones we can creat to log,
track, compile, report, analyze or otherwise process the results data
in various ways.
The only BuildResultsExtension so far is one that scans for entries
in the results map matching a regex ("^include.*" is the standard).
It takes the value of that entry and writes it into a file. The name
of the file can be created using entries in the map as well. So for
example:
{project.id}/{build.name}-{date}.properties
Some things we could add or do
--------------------------------
- System status topic and agent
- A page to display system status (the status of machines online)
- Notification topic and agent that delegates to the notification
manager
- Something to browse the queue and displays the contents
- A results extension to record how long each build took (data
already available)
- Something to use the average time for a build definition and the
work in the queue and estimates the total hours of work to be done
the divides it by the number of machines online and estimates a
completion time.
- Make client connection a component so it can easily be shared
between agents
- Security: maybe ssh-keys for all agents. use SignedObject and
SealedObject for message integrity and host verification.
- Sort properties files when creating build definitions.
- Have DirectoryMonitor file updated events add definitions to the
queue. Allows for builds to be in tasks dir and us to just exec
'touch foo.properties' to queue up the builds inside.
- Add gbuild agent version to message headers
That's all I have time for now. Hope this can get some conversations
started on where to go from here.
-David
On Nov 22, 2005, at 2:44 AM, David Blevins wrote:
On Oct 26, 2005, at 2:01 AM, David Blevins wrote:
When I close my eyes and think big, I see a large federation
consisting of smaller groups of machines from individuals and
companies sharing some common building/testing infrastructure,
open to and co-maintained by members of the community projects,
building all our code all the time and testing it on every variety
of OS, VM and Database imaginable....
I'm happy to say...I so nailed this. We've got it, it works, it
rocks. It's running right now!
For those who haven't been following the gbuild commit logs over
the last two weeks, I've taken Continuum, ActiveMQ, and a whole lot
of long nights and days and created a system that can push
continuum builds onto a jms queue (activemq) where they are picked
up by GBuild Agents (slimmed and extended version of continuum)
running in the gbuild network. The results are then pushed onto a
topic where they are aggregated by another agent in the gbuild
network. Putting more work on the queue is as easy as dropping a
properties file in a directory.
At the moment, have the following machines chugging away on a mix
of tck and other work:
- stan.gbuild.org
- kyle.gbuild.org
- kenny.gbuild.org
- cartman.build.org
Going to get the Chariot Solutions boxes chef.gbuild.org,
timmy.gbuild.org, and jimmy.gbuild.org running tomorrow.
A really cool aspect of this is that we can throw more machines at
it in the middle of a run. It's just jms queue/consumer underneath
so new consumers can be added at anytime! The whole thing is also
really extendible and not specific to tck testing, so we could work
in performance testing and whip up results agents to analyze the
performance data. Sky is the limit!
Off to get some sleep finally... Will send more technical
information tomorrow. Just wanted to give everyone the good news.
Thanks again to everyone (Simula, Mergere, Chariot, Dain) for
donating hosting, boxes, and manpower to get GBuild going. Can't
wait to get the AMD boxes in the mix too. I'm really excited about
all this and can't imagine it going any better.
-David
