I am running into similar issues with both Mellanox and IBM HCAs.
On a node installed with RHEL6.2 and MLNX_OFED-1.5.3-3.0.0, there is a
significant hit to locked memory when going with the device's max_cqe.
Here, for comparison's sake is the memory utilization for a simple MPI
process when using
>> I mentioned on the call that for Mellanox devices (+OFA verbs) this resource
>> is really cheap. Do you run mellanox hca + OFA verbs ?
>
> (I'll reply because I know Terry is offline for the rest of the day)
>
> Yes, he does.
I asked because SUN used to have own verbs driver.
>
> The
On Jul 5, 2012, at 3:53 PM, Shamis, Pavel wrote:
> I mentioned on the call that for Mellanox devices (+OFA verbs) this resource
> is really cheap. Do you run mellanox hca + OFA verbs ?
(I'll reply because I know Terry is offline for the rest of the day)
Yes, he does.
The heart of the
> So if I do a run of -np 2 across two separate node than the use of the
> max_cqe of my ib device (4194303) is ok. Once I go beyond 1 process on the
> node I start getting the memlocked limits message. So how much memory does a
> cqe take? Is it 1k by any chance? I ask this because the
With Jeff's latest changes to how we set up the cq_size I am now seeing
error messages saying that my machine's memlocked limits are too low. I
am concerned that it might be something else because my max'd locked
memory is unlimited on my machine.
So if I do a run of -np 2 across two
