pandit ib wrote:
In iwch_provider.c, line 718, iwch_get_dma_mr() there is a comment
which seems to indicate that the Chelsio card supports registering
only 4GB of physical memory.
Is that correct?
Most of our systems have way greater memory. In fact, the HPC7000
blades we are testing on has 64GB of memory.
The problem we are running into is that our driver calls
ib_get_dma_mr() to register all physical memory.
When the driver attempts to post recv buffers it's getting -EINVAL error.
I traced it down to line 253 in iwch_qp.c.
if (sg_list[i].addr + ((u64) sg_list[i].length) >
mhp->attr.va_fbo + ((u64) mhp->attr.len)) {
PDBG("%s %d\n", __func__, __LINE__);
return -EINVAL;
}
when i printed out those values I got the following:
Jul 6 19:13:17 lv2 kernel: sg_list[i].addr faca80000,
sg_list[i].length 7360 mhp->attr.va_fbo 0 mhp->attr.len 4294967295
The length of the registration indeed seems to be 4GB in size.
Is there a way to make it work on systems with > 4GB of physical
memory. Is there a configuration parameter?
Thanks,
Ranjit
Hey Ranjit,
Sorry, this is limitation in the current Chelsio HW (T3).
You cannot use get_dma_mr to register memory above 4GB. It just doesn't
work for T3. In fact, get_dma_mr just maps dma addresses 0..4GB-1. So
if your system is using a iommu and mapping bus addresses above 4GB, it
won't work at all. Chelsio's next part will overcome this limitation.
In the meantime, you have to use reg_phys_mr or the fast_register work
request to register specific regions on demand. You can see how this is
done by investigating the NFS RDMA transport code.
Steve.
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
