> -----Original Message----- > From: Jared Richardson [mailto:[EMAIL PROTECTED] > Sent: mercredi 23 f�vrier 2005 22:01 > To: Maven Users List > Subject: RE: Clustered Maven? >
[snip] > > > > From an architecture standpoint, I would prefer to let the > > different build > > machines grab a build job. It's hard for a central point to > > know the state > > of the different build machines whereas the machine itself is > > the best to > > know its state. This allows to mix machines with different CPUs, RAM, > > processors, etc. > > You get no failover this way. If machine X dies, the builds can't run > until someone restores the machine. That is not acceptable in our > environment. True, but does it matter that much for a build? I mean the build is taking source and producing something that can be scratched and rebuilt any time. Even if a machine fails you haven't lost anything. > > To give you a little more background, I'm thinking about a several hundred > developers, 300 projects (plus installers) and about 5 millions lines of > code. (SAS is big). http://www.pragmaticautomation.com/cgi- > bin/pragauto.cgi/Build/CCOnALargeScale.rdoc Note: I can't access the doc (access is forbidden). I think that with such a system having a perfect load-balancer (i.e. automatic load balancing) would be more important than the fact that a machine could crash (which would lead to nothing being lost as everything can always be rebuilt - You could even save logs for whatever is put in the build log queue and the result of the build so that you know which builds have not been processed. You could also have the machine that takes a build starts by logging the build in a log file so that you could replay the build if it crashes, etc). > > We could put builds 1 through 17 on a single machine, 18 through 30 on > another, I think having a "reverse" load balancer would improve a lot the overall efficiency of the build farm. If you put builds 1 through 17 on one machine and those builds happen to be simple builds or with not a lot of developers or whatever the machine will be under-used. If these builds are heavy then they the projects won't get built as often as they could, etc. That said, I'm not an expert in this domain so I'd love to be proved wrong and learn something in the process! :-) > etc, but if/when a single machine crashes, the recovery time > would become a real issue. It is if you have a controller that controls to which to load balance because it'll have to understand that the machine should be removed from the farm and not been given any job whereas with the "reverse" load balancer there's no logic to implement for this. > Also, you can't load balance this way. Someone > would end up "tuning" the load to get projects that run in parallel off of > the same machine. I don't understand this point. > > As to "knowing the state", if the proxy/manager issues the job and is > notified (return code? Good log file?) of the job's completion, it's > pretty easy to keep track of state. CPU, ram, etc becomes a side issue if > you just issue one build at a time to each box in the cluster. Once a box > is finished, you send another job. The faster boxes process more builds > and the slow ones process less. Automatic load balancing. Yes, true. The hard part is knowing to which machine to send the next build job so you need get some answer from the build machines. And you need to modify the scheduler if a new machine is added. But yes, I think both solutions have pros and cons. I haven't really implemented the "reverse" load balancer solution but I have always found it extremely elegant in term of architecture. It seems to me it has more pros than cons but maybe the devil is in the implementation details... :-) > > > > > > See > > http://blogs.codehaus.org/people/vmassol/archives/000937_unbre > > akable_builds. > > html to see what I mean (forget the unbreakable part as this > > is not our > > topic here - just look at the build queue and the machines). > > > Thanks. I had actually read that beofore. Your Binary Dependency Builds > entry > (http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependenc > y_builds.html) is what convinced me that Maven might be able to handle a > setup this big. Hey that's cool! I hope you'll like it... At my last project we had quite a big team too (hundred developers). The only thing that needs to be stressed is that the build has to have strong automated unit and functional tests. Thanks -Vincent _________________________________________________________________ Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en fran�ais ! Yahoo! Mail : http://fr.mail.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
