On Wed, 7 Jun 2006, Ben Bodley wrote:
Would a clustering file system really help here?
The short answer is depends very much on the clustering file system. For many the answer is clearly NO.
distcc is a good hack to solve compiling "compute" problems. If you want preprocessing, splint and linking sped up also then they are again all much of the same problem.
Splint for example requires unpreprocessed source since it understands and warns about on errors in preprocessor usage.
It sounds like you have a source server, possibly CVS/SVN. And clients doing development. I would guess you would have that same server handling a commit-trigger which would in turn build your tests then email if someone breaks something. Either that or sends a job email to a build server or similar?
That is one use case which I have running. The other use case is to check some reasonable subset (as fast as possible) before the developer commits. The main use case is the usual update/compile/link/debug cycle.
http://www.mosix.org is a kernel module for sharing "compute" resources among nodes. To increase the power of the build server and use cycles from desktop nodes that may help, not sure..
The reason why I'm pushing on some sort of clustering file system is the whole job queuing problem becomes fairly trivial if all CPU's have efficient access to the same filesystem. I mean I have literally thousands of processes per build to hand out. If the file system view for each process is CPU neutral, it is really easy. However if each CPU has different permissions and directory structure and everything is off on a single overloaded network server... nah, it's just not worth the effort. Hmm. I was just thinking, the problem is special in that I can guarantee that writes to the same file/directory from two different machines never happen and also reads after writes are guaranteed to only happen after the writing machine has closed the file. That makes for all kinds of huge simplifications that a general purpose clustering file system couldn't make. I better I could code it as a Ruby FuseFS in a couple of days... PS: When last did you look at the FuseFS list.... http://fuse.sourceforge.net/wiki/index.php/FileSystems John Carter Phone : (64)(3) 358 6639 Tait Electronics Fax : (64)(3) 359 4632 PO Box 1645 Christchurch Email : [EMAIL PROTECTED] New Zealand Carter's Clarification of Murphy's Law. "Things only ever go right so that they may go more spectacularly wrong later."
From this principle, all of life and physics may be deduced.
