On 2020-06-01 22:50, Alastair McKinstry wrote:
Openmpi is in 64-bit mode when -m64 is set, which it is by default on
amd64 (and I believe the equivalent for lp64 on arm64, mips64).
Oh ok, I didn't realise OpenMPI was already built with 64-bit
capability. I was thinking it would have to be provided as a separate
build, libmpi64.so rather than libmpi.so.
Might be a couple of things to unravel here. When you say -m64 is set by
default on amd64, do you mean openmpi was explicitly built with it? I
can't see it in the openmpi build log.
amd64 in general uses 32-bit int not 64-bit (it's LP64 not ILP64).
I made a test code to probe the integers we're dealing with:
============ ptest.cpp =================
#include <petsc.h>
#include <iostream>
int main()
{
std::cout << "Testing PETSc types\n";
int plainint(0);
int32_t int32(0);
int64_t int64(0);
PetscInt petscint(0);
PetscMPIInt petscmpiint(0);
PetscBLASInt petscblasint(0);
std::cout << "plain int " << plainint << " size " << sizeof(plainint)
<< std::endl;
std::cout << "int32 " << int32 << " size " << sizeof(int32) <<
std::endl;
std::cout << "int64 " << int64 << " size " << sizeof(int64) <<
std::endl;
std::cout << "petscint " << petscint << " size " << sizeof(petscint)
<< std::endl;
std::cout << "petscmpiint " << petscmpiint << " size " <<
sizeof(petscmpiint) << std::endl;
std::cout << "petscblasint " << petscblasint << " size " <<
sizeof(petscblasint) << std::endl;
return 0;
}
=========================================
Compiling it with or without -m64 gives the same output,
mpic++ `pkg-config --cflags --libs PETSc` ptest.cpp -o ./ptest
mpic++ -m64 `pkg-config --cflags --libs PETSc` ptest.cpp -o ./ptest64
both give
Testing PETSc types
plain int 0 size 4
int32 0 size 4
int64 0 size 8
petscint 0 size 4
petscmpiint 0 size 4
petscblasint 0 size 4
So with or without -m64, int is int32_t on amd64. Perhaps I
misunderstand what -m64 is doing. gcc docs says it's a machine-specific
flag for Nvidia PTX.
The only explicit setting for integer type that I could see in openmpi's
./configure is OMPI_MPI_INTEGER_KIND.
(I should mention that PETSc isn't actually doing anything clever to set
PetscMPIInt equal to MPI_INT. It's just typedefing PetscMPIInt as int,
in petscsystypes.h. It does use some preprocessor logic to define
PetscBLASInt though)
The question becomes : do we want m64 set on i386, (and is the
equivalent even possible for arm7, mips , etc) ?
Switching to m64 on i386 means breaking the ABI which i'm reluctant to
do given the above.
Breaking the ABI is probably sensible grounds for not doing it. If it
was worth it, could put it on a TODO list to do later when there's an
OpenMPI ABI transition happening anyway.
Keeping sight on the goal, we want 64 bit to enable calculation of large
systems. If anyone does want to run such large computations on an i386
(or other 32-bit) cluster, then they've still got access to 64-bit
indexing in MUMPS and PETSc to do it over 32-bit MPI.
Drew
