Read the source code/answered my own question. Likely bogus since 2012,
confirmed at least in 3.10.0-957.21.3 — this was the relevant fix:
/*
* We want to scale the number of MTTs with the size of the
* system memory, since it makes sense to register a lot of
* memory on a system with a lot of memory. As a heuristic,
* make sure we have enough MTTs to cover twice the system
* memory (with PAGE_SIZE entries).
*
* This number has to be a power of two and fit into 32 bits
* due to device limitations, so cap this at 2^31 as well.
* That limits us to 8TB of memory registration per HCA with
* 4KB pages, which is probably OK for the next few months.
*/
si_meminfo(&si);
request->num_mtt =
roundup_pow_of_two(max_t(unsigned, request->num_mtt,
min(1UL << (31 - log_mtts_per_seg),
si.totalram >> (log_mtts_per_seg -
1))));
> On Jul 9, 2019, at 4:12 PM, Ryan Novosielski <[email protected]> wrote:
>
> Hi all,
>
> There seems to be a whole lot of misinformation out there about the
> appropriate setting for the log_mtts_per_seg parameter in the mlx4 driver.
> Some folks suggest that it’s different between the RHEL/kernel.org provided
> driver and Mellanox OFED, some suggest it’s been fixed such that the default
> can assign 2x the total memory on a node since at least RHEL6.6 (and so one
> would assume even longer ago in OFED), and some places seem to still be
> carrying it forward because “maybe it matters.”
>
> Is anyone here sure of the current state? I’ll probably read the source code
> if not, but I’d like to spare myself the hassle.
>
> --
> ____
> || \\UTGERS,
> |---------------------------*O*---------------------------
> ||_// the State | Ryan Novosielski - [email protected]
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> || \\ of NJ | Office of Advanced Research Computing - MSB C630,
> Newark
> `'
>
> _______________________________________________
> Beowulf mailing list, [email protected] sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
