On Wed, Oct 22, 2008 at 8:37 PM, Emmanuel Venisse < [EMAIL PROTECTED]> wrote:
> Before to start on distributed builds (I want it in the next version), I > think we must work on some things before. > - the build system must be totally independant of the DB > - the build system must sent events for each steps so Continuum will know > the status and if the DB should be updated > - We must refactor the build system with a plugin mechanism like maven > (Olivier has started to work on a proposal) > Then we'll can play with distributed builds. > I added some comment below. > > On Wed, Oct 22, 2008 at 4:07 AM, Marica Tan <[EMAIL PROTECTED]> wrote: > > > Hi, > > > > Our team in exist came up with a requirements for distributed builds. > > > > *A. Distribution of work* > > > > Master and Slave Agent will communicate with each other through > > xml-rpc. > > I'm not sure xmlrpc is the best choice for the communication. > 1- the communication between the master and slaves must be secured > 2- when Continuum master is updated (new version), I think it can be good > to > update all slaves too > > A solution would can be to store the agent binaries in the master, so if > you > update the master, agent binaries will be updated too. For the > communication, the master would can open a ssh connection to the slave > machine and start the agent remotely, the agent can be installed by the > master automatically. > With that, the communication is totally secured, configured in the master > and the agent is always up-to-date with the master version. > IMHO, there may be other alternative for a secured communication. For instance, we can wrap xmlrpc with SSL to make it secured. I'm not sure if it's a good idea to use ssh just for a secured communication and it's not yet clear to me how to trigger the build using ssh. > > > > > 1. Master Agent - controls build via xml-rpc > > 1. slaves need to be added to the master server > > - attach the slave to a build environment as a type of > > "installation". Each slave reports a number of available > > installations. > > 2. check availability of slaves > > - slave is available if it's up and has no current build. > > - if slave is available then dispatch build to the slave. > > - if slave is unavailable then enqueue build on the master > > I'm not totally agree. With a build/slaves mechanism, you can choose to > delegate the build to an available slave, but if it's the first time a > build > is assigned for a specific project to this node, a full checkout will be > required on the node even if the checkout was already done on an other > node. > The disk usage is potentially the size of all projects X nb nodes. > Some users will want to use always the same node for all builds of a > project > or attach a specific node to a project group. > What about parallel build? We have another proposal for the parallel build. > > > > > 3. distribute work > > - select slave based on build environment (master) > > - build > > - pass current configuration > > - expect result (master)/return result (slave) > > - attach each result returned by the slave to each project in > > the group (add to database) > > 2. Slave Agent - is a dumb build agent that will run on another > > build server. It will only build the project and return the result. It > > also > > has it's own installation. > > - queried for available installations > > - start build, stop build > > - returns build result > > > > I think we'll need an "internal" remote repository used by all nodes. If > the > build definition run only "mvn clean install", the artifact will be stored > on the local repository of the node and other nodes will can't use the new > artifact if we don't use our remote repository. Yes, I agree. > > > > *B. UI/Configuration* > > > > 1. Aggregated view of distributed projects > > > Can you add more informations? Will it modify the aspect of the actual > pages? No. This is a new page listing all projects currently building and where they are being built. > > > > > 2. Slave server list (new page) > > 3. Build Environment > > - add slave selection > > - modify installations based on slave selection > > 4. distributed build configuration stored and editable in XML > > configuration file > > > > Limitations > > > > - credentials (s.a. svn credentials) are passed along if specified, but > > if server cache is used it will need to be done individually on the > > slaves > > - slaves can only be configured at the project group level, by a system > > administrator > > - project dependencies are not considered in distribution - > > interdependent projects should target the same slave server > > > > Future Enhancements > > > > 1. Policy-based distribution > > - next available > > - load balanced > > - target environment matching > > > > > > Comments, suggestions and/or violent reactions are welcome :) > > > Can you add the proposal in the wiki? > Thanks > > Emmanuel > > > > > > > > > Thanks, > > -- > > Marica > > >
