Re: [cmake-developers] FindMPI take 2
On 12/11/2014 05:08 AM, Alin Marin Elena wrote: > I have attached a checked patch for linker. flags with the blocks merged. Thanks. I've applied the combined patch for -Xlinker options here: FindMPI: Extract -Xlinker options http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=a0792c48 I'll wait for more iterations on the other side of this thread for the MPI compiler selection work. Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
Hi KT, > In general, I think this is a good idea because I have encountered situation > were the cmake compiler was a different flavor (intel, gnu, pgi, etc) than > the selected mpi compiler wrapper. However, if the chosen mpi compiler > wrapper is the same as CMAKE__COMPILER, we don't need the try_compile > because cmake will have already verified that the CMAKE__COMPILER works. The nornal test will not test any MPI feature...to be sure that actually the wrapper is an MPI compiler we will need to test a MPI test. Also we need to add a test to see if the found mpi has mpi.mod and can be used. This is a Fortran usage only. Also I like Brad's suggestion on MPICC&friends. regards, Alin -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
Alin Elena wrote: > In addition to that I have noticed that in the interrogate > function we never check that we can actually generate a binary > with the findings, shall we have a try_mpi_compile at the end > of interrogate function? In general, I think this is a good idea because I have encountered situation were the cmake compiler was a different flavor (intel, gnu, pgi, etc) than the selected mpi compiler wrapper. However, if the chosen mpi compiler wrapper is the same as CMAKE__COMPILER, we don't need the try_compile because cmake will have already verified that the CMAKE__COMPILER works. -kt -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
Hi Kelly
> I have been looking into this. For my setup, I found it useful to compare
> the CMAKE__COMPILER to the list of known mpi wrapper names and prepend
> _MPI_${lang}_COMPILER_NAMES with CMAKE__COMPILER so that the remaining
> logic in FindMPI.cmake chooses the user supplied compiler/mpi-wrapper over
> any of the other choices. Something like this:
>
> # append vendor-specific compilers to the list if we either don't know the
> compiler id,
> # or if we know it matches the regular compiler.
> foreach (lang C CXX Fortran)
> foreach (id GNU Intel PGI XL)
> if (NOT CMAKE_${lang}_COMPILER_ID OR CMAKE_${lang}_COMPILER_ID STREQUAL
> id)
> list(APPEND _MPI_${lang}_COMPILER_NAMES
> ${_MPI_${id}_${lang}_COMPILER_NAMES})
> endif()
> unset(_MPI_${id}_${lang}_COMPILER_NAMES)# clean up the namespace here
> endforeach()
>
> + # If cmake_$lang_compiler matches a known mpi compiler wrapper name,
> + # prefer the provided value.
> + get_filename_component( compiler_wo_path "${CMAKE_${lang}_COMPILER}" NAME )
> + set( ${lang}_compiler_is_mpiwrapper false )
> + foreach( mpiwrapper ${_MPI_${lang}_COMPILER_NAMES} )
> +if( "${mpiwrapper}" STREQUAL "${compiler_wo_path}" )
> + set( ${lang}_compiler_is_mpiwrapper true )
> +endif()
> + endforeach()
> + if( ${lang}_compiler_is_mpiwrapper )
> +list( REMOVE_ITEM _MPI_${lang}_COMPILER_NAMES ${compiler_wo_path} )
> +list( INSERT _MPI_${lang}_COMPILER_NAMES 0 ${compiler_wo_path} )
> + endif()
> endforeach()
checked your patch and indeed works.
In addition to that I have noticed that in the interrogate function we
never check that we can actually generate a binary with the findings,
shall we have a try_mpi_compile at the end of interrogate function?
regards
Alin
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
Hi Brad,
>> 0001-correctly-detect-MPI-linker-flags-for-Intel-MPI-mayb.patch
>
> Thanks. That adds a block much like one above it. Can the preceding
> block be modified to match both -Wl, and -Xlinker? Something like:
>
> -string(REGEX MATCHALL "(^| )-Wl,([^\" ]+|\"[^\"]+\")"
> MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
> +string(REGEX MATCHALL "(^| )(-Wl,|-Xlinker )([^\" ]+|\"[^\"]+\")"
> MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
>
> may work (untested).
Indeed the merging of the two sections is the elegant decision... I
kept it in two separate blocks maybe due to my phobia for debugging
regexes.
I have attached a checked patch for linker. flags with the blocks merged.
> The check for whether the CMAKE__COMPILER is already a MPI
> compiler requires a try_compile. Is there some other way to check
> to see if there is any chance before the full test?
I do not know the answer to this. When I did my changes I tried to
minimise them.
> Also I'm not sure we fully support using a MPI compiler as the main
> compiler. I can't think of anything that definitely won't work, but
> we don't have nightly testing for the case.
>
> Generally I've viewed the MPI compiler interrogation to be much like
> using pkg-config or $pkg-config tools to get the info needed to use
> a package. Once that is done then MPI should be treated like any
> other library.
Unfortunately MPI wrapper is abused as main compiler in some projects.
Practically my patch just checks for that. As it involves a pretty
active decision from the user, I think we shall be fine.
> The user should be able to specify MPI__COMPILER explicitly.
> Perhaps it could be initialized by ENV{MPICC}, ENV{MPICXX}, and
> ENV{MPIFC} environment variables? This selection is much like
> picking the primary compiler in the first place.
This will the the best situation but will involve changes to the
FindMPI module that may be above my station to do.
regards,
Alin
From 9a8b2c7e0e39bef0e3aad84c8f1ba7e3489ab262 Mon Sep 17 00:00:00 2001
From: Alin Marin Elena
Date: Thu, 11 Dec 2014 09:56:39 +
Subject: [PATCH 2/2] Linking flags correctly detected for intel compilers
---
Modules/FindMPI.cmake | 13 +
1 file changed, 1 insertion(+), 12 deletions(-)
diff --git a/Modules/FindMPI.cmake b/Modules/FindMPI.cmake
index 5f23c02..840b714 100644
--- a/Modules/FindMPI.cmake
+++ b/Modules/FindMPI.cmake
@@ -363,18 +363,7 @@ function (interrogate_mpi_compiler lang try_libs)
endif()
# Extract linker flags from the link command line
-string(REGEX MATCHALL "(^| )-Wl,([^\" ]+|\"[^\"]+\")" MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
-set(MPI_LINK_FLAGS_WORK)
-foreach(FLAG ${MPI_ALL_LINK_FLAGS})
- if (MPI_LINK_FLAGS_WORK)
-set(MPI_LINK_FLAGS_WORK "${MPI_LINK_FLAGS_WORK} ${FLAG}")
- else()
-set(MPI_LINK_FLAGS_WORK ${FLAG})
- endif()
-endforeach()
-
-# Extract linker flags from the link command line Intel compilers.. others maybe
-string(REGEX MATCHALL "(^| )-Xlinker ([^\" ]+|\"[^\"]+\")" MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
+string(REGEX MATCHALL "(^| )(-Wl,|-Xlinker )([^\" ]+|\"[^\"]+\")" MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
set(MPI_LINK_FLAGS_WORK)
foreach(FLAG ${MPI_ALL_LINK_FLAGS})
if (MPI_LINK_FLAGS_WORK)
--
2.2.0
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
Brad King wrote:
>> 0002-first-try-to-see-if-what-user-provided-produces-an-m.patch
> The check for whether the CMAKE__COMPILER is already a
> MPI compiler requires a try_compile. Is there some other way
> to check to see if there is any chance before the full test?
I have been looking into this. For my setup, I found it useful to compare the
CMAKE__COMPILER to the list of known mpi wrapper names and prepend
_MPI_${lang}_COMPILER_NAMES with CMAKE__COMPILER so that the remaining
logic in FindMPI.cmake chooses the user supplied compiler/mpi-wrapper over any
of the other choices. Something like this:
# append vendor-specific compilers to the list if we either don't know the
compiler id,
# or if we know it matches the regular compiler.
foreach (lang C CXX Fortran)
foreach (id GNU Intel PGI XL)
if (NOT CMAKE_${lang}_COMPILER_ID OR CMAKE_${lang}_COMPILER_ID STREQUAL id)
list(APPEND _MPI_${lang}_COMPILER_NAMES
${_MPI_${id}_${lang}_COMPILER_NAMES})
endif()
unset(_MPI_${id}_${lang}_COMPILER_NAMES)# clean up the namespace here
endforeach()
+ # If cmake_$lang_compiler matches a known mpi compiler wrapper name,
+ # prefer the provided value.
+ get_filename_component( compiler_wo_path "${CMAKE_${lang}_COMPILER}" NAME )
+ set( ${lang}_compiler_is_mpiwrapper false )
+ foreach( mpiwrapper ${_MPI_${lang}_COMPILER_NAMES} )
+if( "${mpiwrapper}" STREQUAL "${compiler_wo_path}" )
+ set( ${lang}_compiler_is_mpiwrapper true )
+endif()
+ endforeach()
+ if( ${lang}_compiler_is_mpiwrapper )
+list( REMOVE_ITEM _MPI_${lang}_COMPILER_NAMES ${compiler_wo_path} )
+list( INSERT _MPI_${lang}_COMPILER_NAMES 0 ${compiler_wo_path} )
+ endif()
endforeach()
There is probably a better way to do this, but you get the idea. In my setup,
I'm trying to get FindMPI.cmake to select mpiicpc over mpicc without manually
setting the MPI_${lang}_COMPILER manually.
> Also I'm not sure we fully support using a MPI compiler
> as the main compiler. I can't think of anything that
> definitely won't work, but we don't have nightly testing
> for the case.
My project does nightly regressions with and without the use of MPI compiler
wrappers. I have never had a problem using the compiler wrappers (e.g.: mpicc,
mpiicpc, Cray's CC). Additionally, I think Trilinos (trilinos.org) requires
the use of compiler wrappers to build their code (cmake build system).
-kt
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] FindMPI take 2
On 12/09/2014 06:59 AM, Alin Marin Elena wrote:
> 0001-correctly-detect-MPI-linker-flags-for-Intel-MPI-mayb.patch
Thanks. That adds a block much like one above it. Can the preceding
block be modified to match both -Wl, and -Xlinker? Something like:
-string(REGEX MATCHALL "(^| )-Wl,([^\" ]+|\"[^\"]+\")"
MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
+string(REGEX MATCHALL "(^| )(-Wl,|-Xlinker )([^\" ]+|\"[^\"]+\")"
MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
may work (untested).
> 0002-first-try-to-see-if-what-user-provided-produces-an-m.patch
The check for whether the CMAKE__COMPILER is already a MPI
compiler requires a try_compile. Is there some other way to check
to see if there is any chance before the full test?
Also I'm not sure we fully support using a MPI compiler as the main
compiler. I can't think of anything that definitely won't work, but
we don't have nightly testing for the case.
Generally I've viewed the MPI compiler interrogation to be much like
using pkg-config or $pkg-config tools to get the info needed to use
a package. Once that is done then MPI should be treated like any
other library.
> 3. This is a more complex issue and my understanding is that FindMPI
> was designed with mpicc/mpicxx/mpif90 being the workhorse wrappers for
> MPI. While I do not propose to change that I think that implementing
> the change at 2. will somehow mitigate the case of multiple wrappers
> and compilers.
The user should be able to specify MPI__COMPILER explicitly.
Perhaps it could be initialized by ENV{MPICC}, ENV{MPICXX}, and
ENV{MPIFC} environment variables? This selection is much like
picking the primary compiler in the first place.
Thanks,
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] FindMPI take 2
Dear All,
recently I have run into some issues with FindMPI and intel mpi[1],
more intel mpi issue rather than cmake. The original issue reported
in[1] was solved. However in the process I made few points related
mainly to FindMPI, that I summarise in here:
1. the linker information is not parsed, see below
2. Failure to use the wrapper as a compiler.
3. Using wrong wrappers to interrogate for intel mpi and intel
compilers. eg. mpif90 instead of mpiifort (NB: If a user uses
I_MPI_F90=ifort, or c/c++ equivalents this issue is somehow
mittigated)
1. Linker issue
-- MPI_C_INCLUDE_PATH
/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/include
-- MPI_C_LINK_FLAGS
-- MPI_C_LIBRARIES
/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib/libmpifort.so;/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib/release/libmpi.so;/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib/libmpigi.a;/usr/lib64/libdl.so;/usr/lib64/librt.so;/usr/lib64/libpthread.so
-- Configuring done
-- Generating done
-- Build files have been written to: /ichec/home/staff/alin/findmpi/5.0.1
[alin@fionn2:~/findmpi/5.0.1]: mpicc -compile-info
gcc -I/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/include
-L/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib/release
-L/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib
-Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker
/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib/release
-Xlinker -rpath -Xlinker
/ichec/packages/intel/intel-cluster-studio-2015/impi_5.0.1/intel64/lib
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/5.0/intel64/lib/release
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/5.0/intel64/lib -lmpifort
-lmpi -lmpigi -ldl -lrt -lpthread
Patch attached
0001-correctly-detect-MPI-linker-flags-for-Intel-MPI-mayb.patch
2. We first use whatever is provided in FC to detect the
libs/includes/linker flags. I think the correct order shall be first
check if what is provided in FC build mpi binaries, if fails go to
interrogation.
patch attached
0002-first-try-to-see-if-what-user-provided-produces-an-m.patch
3. This is a more complex issue and my understanding is that FindMPI
was designed with mpicc/mpicxx/mpif90 being the workhorse wrappers for
MPI. While I do not propose to change that I think that implementing
the change at 2. will somehow mitigate the case of multiple wrappers
and compilers.
[1] http://www.cmake.org/Bug/view.php?id=15182
Without Questions there are no Answers!
__
Dr. Alin Marin ELENA
http://alin.elenaworld.net/
__
From 4462ada3909c5ab023800b0f389038dcada9f91d Mon Sep 17 00:00:00 2001
From: Alin Marin Elena
Date: Tue, 9 Dec 2014 10:52:33 +
Subject: [PATCH 1/2] correctly detect MPI linker flags for Intel MPI maybe
others too
---
Modules/FindMPI.cmake | 11 +++
1 file changed, 11 insertions(+)
diff --git a/Modules/FindMPI.cmake b/Modules/FindMPI.cmake
index 6f6dcf3..75f645f 100644
--- a/Modules/FindMPI.cmake
+++ b/Modules/FindMPI.cmake
@@ -373,6 +373,17 @@ function (interrogate_mpi_compiler lang try_libs)
endif()
endforeach()
+# Extract linker flags from the link command line Intel compilers.. others maybe
+string(REGEX MATCHALL "(^| )-Xlinker ([^\" ]+|\"[^\"]+\")" MPI_ALL_LINK_FLAGS "${MPI_LINK_CMDLINE}")
+set(MPI_LINK_FLAGS_WORK)
+foreach(FLAG ${MPI_ALL_LINK_FLAGS})
+ if (MPI_LINK_FLAGS_WORK)
+set(MPI_LINK_FLAGS_WORK "${MPI_LINK_FLAGS_WORK} ${FLAG}")
+ else()
+set(MPI_LINK_FLAGS_WORK ${FLAG})
+ endif()
+endforeach()
+
# Extract the set of libraries to link against from the link command
# line
string(REGEX MATCHALL "(^| )-l([^\" ]+|\"[^\"]+\")" MPI_LIBNAMES "${MPI_LINK_CMDLINE}")
--
2.2.0
From 08cd4aacf7fb7e26d4a0f39c0db4631afec87a4d Mon Sep 17 00:00:00 2001
From: Alin Marin Elena
Date: Tue, 9 Dec 2014 11:55:14 +
Subject: [PATCH 2/2] first try to see if what user provided produces an mpi
binary
---
Modules/FindMPI.cmake | 15 +++
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/Modules/FindMPI.cmake b/Modules/FindMPI.cmake
index 75f645f..5f23c02 100644
--- a/Modules/FindMPI.cmake
+++ b/Modules/FindMPI.cmake
@@ -486,7 +486,7 @@ endfunction()
# This function attempts to compile with the regular compiler, to see if MPI programs
-# work with it. This is a last ditch attempt after we've tried interrogating mpicc and
+# work with it. This is a first ditch attempt befre interrogating mpicc and
# friends, and after we've tried to find generic libraries. Works on machines like
# Cray XE6, where the modules environment changes what MPI version cc, CC, and ftn use.
function(try_regular_compiler lang success)
@@
