On 11/15/2018 05:37 PM, Hefty, Sean wrote:
However, the verbs extensions in the header verbs_exp.h provide
additional functionality. Among other things this includes the
capability to register a physical address as MR instead of a virtual
one. For Mellanox drivers this is described in [1]. The call is
ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
Using the physical address for the reg_mr call we can then transfer
data directly from our custom PCIe card to the network adapter,
without any copies. Pretty nice!
To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
[2]. Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
is available, so just the flag needs to be added.
As for my feature request, can this be added as official feature to
libfabric? I am not sure how many other providers could actually
support physical address memory registration, but it is a nice feature
when working with custom hardware and drivers. My patch is of course a
bit "crude" and this would need to be properly implemented.
Thanks.
Cheers,
Jörn
[1] https://community.mellanox.com/docs/DOC-2480
[2]
https://github.com/joerns/libfabric/compare/v1.6.x...joerns:phys_addr_
mr
Looking at the changes, I see a couple of options. We can add this as an
extension for verbs only. This could make use of the provider specific flag
bits passed into the mr_reg call. Or, we can define this as a generic mr_reg
flag.
I'm hesitant to make this a more integrated feature, like defining it as a
capability bit, since it's completely non-standard and experimental.
Is there a reliable way for the libfabric provider to determine what type of
memory this is, such that the flag passed into verbs can be set automatically
as needed?
I am happy with this being a provider-specific extension for now. The
project that requires this will anyway only use the verbs provider.
I do not think there is a fully reliable way of distinguishing
physical/virtual addresses. I spoke with our driver expert here who
cannot think of a way either. The provider-specific flag seems to be the
way to go, what do you think?
Cheers,
Jörn
_______________________________________________
ofiwg mailing list
[email protected]
https://lists.openfabrics.org/mailman/listinfo/ofiwg
