Hi Eli,Thanks a lot for this explanation of the MTT and how log_mtt_per_seg log_num_mtt work. I will play around with these values and figure which one works best for our cluster + application environment.
-- Bharath On 1/21/2013 11:57 AM, Eli Cohen wrote:
Hi Bharath, each MTT entry is used to cover a whole page. Assuming page size in you system equals 4K, the max total of registered memory is 2^(12+24) which is 64 GB. The log_mtt_per_seg param denotes the granularity of the allocator managing MTTs. So, in your case, MTTs are allocated in granularity of 2. So if you try to allocate 3 MTTs, you'll end up consuming 4. Moreover, the MTT allocator is a buddy allocator that allocates power of 2 MTT segments. So if you register memory that requires (2^15 + 1) MTTs, you'll actually consume 2^16 MTTs! In your case I would try to use 2^25 MTTs and log_mtt_per_seg=1. If the driver fails to load (probably because the allocator fails to allocate memory), try to use 2^24 and 2^2 etc. I hope that helps. On Mon, Jan 21, 2013 at 5:08 AM, Bharath Ramesh <[email protected]> wrote:I am trying to find the correct setting for log_num_mtt and log_mtt_per_seg should be for our cluster. Each node has 64G of RAM. We are having issues when running MPI applications, the error is related to registering memory. The current setting log_num_mtt=24 and log_mtt_per_seg=1. There are lot of conflicting documentations available with regards how these settings should be changed. I was wondering if the community could explain how these settings works so that we can come up with the correct settings that will work in our environment. This document [1] specifically says that log_mtt_per_seg should always be 1. However, OpenMPI mailing list post [2] talks about different value. Any help on this is appreciated. I am not subscribed to the list so would really appreciate if I am copied in the replies. [1] http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem [2] http://www.open-mpi.org/community/lists/users/2011/09/17222.php -- Bharath
-- Bharath
smime.p7s
Description: S/MIME Cryptographic Signature
