Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Alan W. Irwin]

On 2018-01-12 13:34+0300 Konstantin Tokarev wrote:




12.01.2018, 12:58, "Alan W. Irwin" :

I am looking into the practicality of using clusters containing ~5
cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
and test software (since even with ccache I am currently spending a
lot of my time waiting for tests to complete as I develop my
software). Such clusters would give you ~40 cores which is a lot of
computer power for ~$500 or so. But, of course, the issue with
clusters is how to use them efficiently?


1. For compilation you often need good single-core performance more
than parallelism, and I doubt that embedded-grade ARMv8 can provide it

I guess exception is compiling projects written in C which don't have huge
source files. Still, overhead of passing data between nodes may be
substanital.

2. You should probably use icecc or distcc, unless you have MPI as a
requirement.


Most of the work is run-time tests (configured with
add_custom_command/target) rather than compilations.  It have just
discovered that it looks like "distmake" or equivalent is what I need
(see the post I just wrote).

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Alan W. Irwin]

On 2018-01-12 11:14+0100 Eric Noulard wrote:


2018-01-12 10:58 GMT+01:00 Alan W. Irwin :


I am looking into the practicality of using clusters containing ~5
cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
and test software (since even with ccache I am currently spending a
lot of my time waiting for tests to complete as I develop my
software).  Such clusters would give you ~40 cores which is a lot of
computer power for ~$500 or so.  But, of course, the issue with
clusters is how to use them efficiently?

For one 8-core M3 you would expect

make -j target

to scale linearly with NNN up to the saturation limit of roughly NNN=8
(where target depends on ~hundreds of different and mostly independent
tests). But my understanding from recent Google searching is that the
-j option for GNU Make is not MPI-aware, i.e., only works for the
cores for a single computer (node) in a cluster.  So in a cluster you
would have to assign the above task to just one node and simply accept
that you cannot scale NNN up to the total number of cores in that
cluster.






AFAIU what you want is "distributed ctest" not really distributed build
right?


Not ctest, but actually distributed "build", but with "build" in
quotes because instead of exclusively building software, most of the
targets that are built are configured with
add_custom_command/add_custom_target and actually run tests with
proper dependencies on other test targets as well as targets that
actually build software.  So this testing paradigm makes initial
testing of one particular test much quicker because you don't have to
build the "all" target before the test like is required with ctest.


For distributed build you may have a look at distcc (http://distcc.org/)
which seems already used
by CMake-enabled project like ccache is
(
https://stackoverflow.com/questions/32752446/using-compiler-prefix-commands-with-cmake-distcc-ccache/34317588
)


It looks like distcc could speed up compilations tremendously on a
cluster, but only a minority of my targets are actually builds of
software.  So I need a "distmake".  Just after writing that sentence I
decided I liked that made-up name so on the off-chance someone else
had liked it as well, I did a google search and found the SF distmake project
that apparently does exactly what I need!  Instead of using MPI (Message
Passing Interface) standard to coordinate the targets that are built
between the various nodes, it looks like it uses something a lot
simpler.


Now I'm not quite sure to understand why you speak of MPI-awareness ?


I thought to coordinate target builds over a cluster, that the make
programme itself would have to have MPI directives inserted into its source
code so the whole build coordination could be handled by MPI.  But it
appears that complexity is not necessary (see distmake, above).


Are your test including MPI which would mean that some executable are
already eating up more than one core?


No.


Or do you have a big collection of mono-processor tests you want to run on
the cluster?


Yes, but with complete dependencies as explained above.

In sum, it looks like this question now boils down to whether anyone
here has used distmake (or something equivalent to it) on a cluster,
and if so, are there any gotcha's?

Alan

__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Konstantin Tokarev]

12.01.2018, 12:58, "Alan W. Irwin" :
> I am looking into the practicality of using clusters containing ~5
> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
> and test software (since even with ccache I am currently spending a
> lot of my time waiting for tests to complete as I develop my
> software). Such clusters would give you ~40 cores which is a lot of
> computer power for ~$500 or so. But, of course, the issue with
> clusters is how to use them efficiently?

1. For compilation you often need good single-core performance more
than parallelism, and I doubt that embedded-grade ARMv8 can provide it

I guess exception is compiling projects written in C which don't have huge
source files. Still, overhead of passing data between nodes may be
substanital.

2. You should probably use icecc or distcc, unless you have MPI as a
requirement.

>
> For one 8-core M3 you would expect
>
> make -j target
>
> to scale linearly with NNN up to the saturation limit of roughly NNN=8
> (where target depends on ~hundreds of different and mostly independent
> tests). But my understanding from recent Google searching is that the
> -j option for GNU Make is not MPI-aware, i.e., only works for the
> cores for a single computer (node) in a cluster. So in a cluster you
> would have to assign the above task to just one node and simply accept
> that you cannot scale NNN up to the total number of cores in that
> cluster.
>
> I did find one reference to an effort (called the pmake project at
> Lawrence Livermore, see )
> to modify GNU make to become MPI-aware so the above command would
> roughly scale NNN up to the number of cores in the cluster. But the
> single developer of the project seemed unable to overcome a bug in the
> result so that parallel make project was abandoned.
>
> I hope someone here can point me to a make variant (does Ninja
> qualify?) that does have the capability of scaling NNN up to the
> number of cores in the cluster since that would make clusters with a
> large number of cores become extremely useful for the purpose of quick
> testing by software developers.
>
> Alan
> __
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __
>
> Linux-powered Science
> __
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake

-- 
Regards,
Konstantin
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Eric Noulard]

2018-01-12 11:14 GMT+01:00 Eric Noulard :

>
>
> 2018-01-12 10:58 GMT+01:00 Alan W. Irwin :
>
>> I am looking into the practicality of using clusters containing ~5
>> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
>> and test software (since even with ccache I am currently spending a
>> lot of my time waiting for tests to complete as I develop my
>> software).  Such clusters would give you ~40 cores which is a lot of
>> computer power for ~$500 or so.  But, of course, the issue with
>> clusters is how to use them efficiently?
>>
>> For one 8-core M3 you would expect
>>
>> make -j target
>>
>> to scale linearly with NNN up to the saturation limit of roughly NNN=8
>>
>
A side note on that.
Sometimes you want to control the 'load' more than the "parallelism".
With make (or ninja) you can ask for **load** limitation using:

make -l 
and
ninja -l 

i.e. if you have 8 core and you want to keep 1 of them "calm" then
make -l 7

is interesting.


>
> AFAIU what you want is "distributed ctest" not really distributed build
> right?
> For distributed build you may have a look at distcc (http://distcc.org/)
> which seems already used
> by CMake-enabled project like ccache is
> (https://stackoverflow.com/questions/32752446/using-
> compiler-prefix-commands-with-cmake-distcc-ccache/34317588)
>
> Now I'm not quite sure to understand why you speak of MPI-awareness ?
> Are your test including MPI which would mean that some executable are
> already eating up more than one core?
> Or do you have a big collection of mono-processor tests you want to run on
> the cluster?
>

Concerning high-level distributed execution on a cluster I would have a
look at something
like OpenPBS: http://www.rguha.net/writing/pub/opbs.html (now apparently
http://pbspro.org/).

Once you have your PBS cluster setup I would say that you would need to
write a small test driver
that would launch the tests on the cluster instead of launching them
locally.


-- 
Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Eric Noulard]

2018-01-12 10:58 GMT+01:00 Alan W. Irwin :

> I am looking into the practicality of using clusters containing ~5
> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
> and test software (since even with ccache I am currently spending a
> lot of my time waiting for tests to complete as I develop my
> software).  Such clusters would give you ~40 cores which is a lot of
> computer power for ~$500 or so.  But, of course, the issue with
> clusters is how to use them efficiently?
>
> For one 8-core M3 you would expect
>
> make -j target
>
> to scale linearly with NNN up to the saturation limit of roughly NNN=8
> (where target depends on ~hundreds of different and mostly independent
> tests). But my understanding from recent Google searching is that the
> -j option for GNU Make is not MPI-aware, i.e., only works for the
> cores for a single computer (node) in a cluster.  So in a cluster you
> would have to assign the above task to just one node and simply accept
> that you cannot scale NNN up to the total number of cores in that
> cluster.
>

>
AFAIU what you want is "distributed ctest" not really distributed build
right?
For distributed build you may have a look at distcc (http://distcc.org/)
which seems already used
by CMake-enabled project like ccache is
(
https://stackoverflow.com/questions/32752446/using-compiler-prefix-commands-with-cmake-distcc-ccache/34317588
)

Now I'm not quite sure to understand why you speak of MPI-awareness ?
Are your test including MPI which would mean that some executable are
already eating up more than one core?
Or do you have a big collection of mono-processor tests you want to run on
the cluster?

-- 
Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

Re: [CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Nils Gladitz]

On Fri, Jan 12, 2018 at 10:58 AM, Alan W. Irwin 
wrote:

> I am looking into the practicality of using clusters containing ~5
> cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
> and test software (since even with ccache I am currently spending a
> lot of my time waiting for tests to complete as I develop my
> software).  Such clusters would give you ~40 cores which is a lot of
> computer power for ~$500 or so.  But, of course, the issue with
> clusters is how to use them efficiently?
>

I don't know anything about MPI but would distributed compilation via icecc
(icecream) [1] or distcc be an option?
Should work with both the Makefile and Ninja generators (replace compilers
with icecc/distcc compiler wrappers).

Nils

[1] https://github.com/icecc/icecream
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[CMake] Do any CMake back ends have support for parallel builds that efficiently use clusters?

[Alan W. Irwin]

I am looking into the practicality of using clusters containing ~5
cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build
and test software (since even with ccache I am currently spending a
lot of my time waiting for tests to complete as I develop my
software).  Such clusters would give you ~40 cores which is a lot of
computer power for ~$500 or so.  But, of course, the issue with
clusters is how to use them efficiently?

For one 8-core M3 you would expect

make -j target

to scale linearly with NNN up to the saturation limit of roughly NNN=8
(where target depends on ~hundreds of different and mostly independent
tests). But my understanding from recent Google searching is that the
-j option for GNU Make is not MPI-aware, i.e., only works for the
cores for a single computer (node) in a cluster.  So in a cluster you
would have to assign the above task to just one node and simply accept
that you cannot scale NNN up to the total number of cores in that
cluster.

I did find one reference to an effort (called the pmake project at
Lawrence Livermore, see )
to modify GNU make to become MPI-aware so the above command would
roughly scale NNN up to the number of cores in the cluster.  But the
single developer of the project seemed unable to overcome a bug in the
result so that parallel make project was abandoned.

I hope someone here can point me to a make variant (does Ninja
qualify?) that does have the capability of scaling NNN up to the
number of cores in the cluster since that would make clusters with a
large number of cores become extremely useful for the purpose of quick
testing by software developers.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake