-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/09/12 05:55 AM, Gregory M. Turner wrote: > > Note that, effectively, we have this already, and it's called > "portage". But one could certainly make a case for modularizing it > better, since, in truth, we are talking about a very common, very > abstract problem here which portage shares with any number of > batch-build systems. > > Such an engine could very well do exactly the right thing if it > were faced with a constraint that a certain part of a certain build > needed to proceed without parallelism due to limitations coming > from the build. > > Also, there are very large parts of most builds -- configure comes > to mind -- that don't parallelize even if, perhaps, they should. > In such cases, a really smart global parallelism arbiter could > easily respond by spawning more jobs from other builds. >
So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Advanced HPC systems (sys-cluster/torque along with an appropriate scheduler, for instance) can do such things with their jobs when the jobs are properly built; I could see portage being able to handle this as well given most of what is necessary is already known (ebuild phases, build system type (via eclass), etc). However, given the limitations already put on parallelism in terms of emerge order, etc, I could see this solution needing to be -very- complex and integration needing to occur on multiple levels. We'd also need to consider distcc (and other cluster-shared compilation methods if there are any??).. It would be an interesting project, though. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQhwAACgkQ2ugaI38ACPCKrgD+JNlHPUl7ETYDC6u3lYWRSz8J fpWC/puDfCYu51yNOVIA/0E+U6x9Ds8GV8r/RinkTqss3/fcd06w24GRvZOda3Mj =XONZ -----END PGP SIGNATURE-----
