Hello Yevgeny, hello all,Yevgeny, first of all thanks for explaining what the MTT parameters do and why there are two of them! I mean this post:
http://www.open-mpi.org/community/lists/devel/2012/08/11417.php
Well, the official recommendation is "twice the RAM amount".And here we are: we have 2 nodes with 2 TB (that with a 'tera') RAM and a couple of nodes with 1TB, each with 4x Mellanox IB adapters. Thus we should have raised the MTT parameters in order to make up to 4 TB memory registrable.
I've tried to raise the MTT parameters in multiple combinations, but the maximum amount of registrable memory I was able to get was one TB (23 / 5). All tries to get more (24/5, 23/6 for 2 TB) lead to not responding InfiniBand HCAs.
Is there any another limits in the kernel have to be adjusted in order to be able to register that a bunch of memory?
Best, Paul Kapinos -- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, Center for Computing and Communication Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
smime.p7s
Description: S/MIME Cryptographic Signature
