I am looking into the practicality of using clusters containing ~5 cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build and test software (since even with ccache I am currently spending a lot of my time waiting for tests to complete as I develop my software). Such clusters would give you ~40 cores which is a lot of computer power for ~$500 or so. But, of course, the issue with clusters is how to use them efficiently?
For one 8-core M3 you would expect make -j<NNN> target to scale linearly with NNN up to the saturation limit of roughly NNN=8 (where target depends on ~hundreds of different and mostly independent tests). But my understanding from recent Google searching is that the -j option for GNU Make is not MPI-aware, i.e., only works for the cores for a single computer (node) in a cluster. So in a cluster you would have to assign the above task to just one node and simply accept that you cannot scale NNN up to the total number of cores in that cluster. I did find one reference to an effort (called the pmake project at Lawrence Livermore, see <https://computing.llnl.gov/code/pmake.html>) to modify GNU make to become MPI-aware so the above command would roughly scale NNN up to the number of cores in the cluster. But the single developer of the project seemed unable to overcome a bug in the result so that parallel make project was abandoned. I hope someone here can point me to a make variant (does Ninja qualify?) that does have the capability of scaling NNN up to the number of cores in the cluster since that would make clusters with a large number of cores become extremely useful for the purpose of quick testing by software developers. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
