I agree with Bart here.
Minimizing toolchain dependencies is key. Don't build something with
Intel + OpenMPI + CUDA if it's something that should be built with GCC
(for example CMake or Autoconf)... it is just wasting time and space.
That's why I dislike the "intel" and "foss" toolchains and we typically
do not use them at Compute Canada, they include too many things. We have
a very large number of packages that don't care about MPI and should not
be built with a toolchain that requires MPI. Most packages are that way
in fact.
My 2 cents...
Maxime
On 18-03-21 11:48, Bart Oldeman wrote:
You can mitigate this issue by installing as much as possible at the
compiler level -- we do that at Compute Canada. I have some pending
work on the framework that could make that possible for Python too.
The major incompatibility between goolfc and goolf=foss is in the MPI
libraries, one with and one without CUDA support. But anything that
does not need MPI should not need recompilation.
Basically for goolfc you'd have this shiny alphabet soup:
gcccuda - gompic - goolfc
/ \ /
GCCcore - GCC - golf - golfc
\ \
gompi - goolf
ie:
goolf has two direct subtoolchains (golf=GNU+OpenBLAS+LAPACK+FFTW, and
gompi)
goolfc has two direct subtoolchains (golfc and gompic)
golfc has two direct subtoolchains (golf and gcccuda)
Then you can install Python (except for mpi4py) for the golf toolchain.
I got one headache when trying to implement this though: what about
FFTW in a hierarchy? There need to be two FFTW modules, one with (in
goolf) and one without MPI (in golf), and the majority of applications
(even those that use MPI) use the serial FFTW.
1. using a -serial suffix for the serial one, and no suffix for the
MPI one?
2. using a -mpi suffix for the MPI one, no suffix for serial.
3. using pFFTW and FFTW?
4. anything else? Should the MPI FFTW only contain the parallel
libraries (libfftw3*_mpi.*) or also a copy of the non-MPI ones?
Bart
On 21 March 2018 at 10:55, Jakob Schiøtz <[email protected]
<mailto:[email protected]>> wrote:
I very strongly agree with Jack on this. If only a single program
/ python module uses CUDA, it is wasteful to have to build and
install a new toolchain, and to rebuild everything on the system,
including Python and perhaps even X11 (if using matplotlib).
But there may be something I have overlooked.
Best regards
Jakob
> On 19 Mar 2018, at 15:49, Jack Perdue <[email protected]
<mailto:[email protected]>> wrote:
>
>
> On 03/19/2018 09:43 AM, Joachim Hein wrote:
>> Hi,
>>
>> I am currently installing tensorflow via easybuild (I assume
many of us do these days) and am trying to understand EasyBuild’s
ideas on toolchains supporting cuda.
>>
>> I looked at TensorFlow-1.5.0-goolfc-2017b-Python-3.6.3.eb,
which builds ontop of a toolchain containing GCC, Cuda (installed
as a compiler module), and OpenMPI, Blas, FFTW etc.
>>
>> I now noticed that there is a new
TensorFlow-1.6.0-foss-2018a-Python-3.6.4-CUDA-9.1.85.eb, which is
accepted into the development branch (PR 6016). This builds
ontop a “vanilla” foss-2018a toolchain, using a Cuda and cuDNN
modules installed as a core module (system compiler).
>>
>> I am wondering how do we want to organise us in future? Do we
want to continue with the goolfc idea or do we go for a “core”
cuda and cuDNN? I feel this needs standardising soonish. It is
also something I feel I need to document for my users, who want to
build their own cuda based software. What models should be loaded
to build software.
>>
>> Any comments, how we take this further?
>>
>> Best wishes
>> Joachim
>>
>
> FWIW ($.02) I'm partial to the latter approach since it allows
> more flexibility of CUDA version without redefining an entire
> toolchain (which then requres everything to be rebuilt (e.g. Python)
> whether they need CUDA or not).
>
> Jack Perdue
> Lead Systems Administrator
> High Performance Research Computing
> TAMU Division of Research
> [email protected] <mailto:[email protected]> http://hprc.tamu.edu
> HPRC Helpdesk: [email protected] <mailto:[email protected]>
>
>
>
--
Jakob Schiøtz, professor, Ph.D.
Department of Physics
Technical University of Denmark
DK-2800 Kongens Lyngby, Denmark
http://www.fysik.dtu.dk/~schiotz/
<http://www.fysik.dtu.dk/%7Eschiotz/>
--
Dr. Bart E. Oldeman | [email protected]
<mailto:[email protected]> | [email protected]
<mailto:[email protected]>
Scientific Computing Analyst / Analyste en calcul scientifique
McGill HPC Centre / Centre de Calcul Haute Performance de McGill |
http://www.hpc.mcgill.ca
Calcul Québec | http://www.calculquebec.ca
Compute/Calcul Canada | http://www.computecanada.ca
Tel/Tél: 514-396-8926 | Fax/Télécopieur: 514-396-8934
--
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique
