> However, the verbs extensions in the header verbs_exp.h provide > additional functionality. Among other things this includes the > capability to register a physical address as MR instead of a virtual > one. For Mellanox drivers this is described in [1]. The call is > ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag. > > Using the physical address for the reg_mr call we can then transfer > data directly from our custom PCIe card to the network adapter, > without any copies. Pretty nice! > > To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see > [2]. Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h > is available, so just the flag needs to be added. > > As for my feature request, can this be added as official feature to > libfabric? I am not sure how many other providers could actually > support physical address memory registration, but it is a nice feature > when working with custom hardware and drivers. My patch is of course a > bit "crude" and this would need to be properly implemented. > > Thanks. > > Cheers, > Jörn > > > [1] https://community.mellanox.com/docs/DOC-2480 > [2] > https://github.com/joerns/libfabric/compare/v1.6.x...joerns:phys_addr_ > mr
Looking at the changes, I see a couple of options. We can add this as an extension for verbs only. This could make use of the provider specific flag bits passed into the mr_reg call. Or, we can define this as a generic mr_reg flag. I'm hesitant to make this a more integrated feature, like defining it as a capability bit, since it's completely non-standard and experimental. Is there a reliable way for the libfabric provider to determine what type of memory this is, such that the flag passed into verbs can be set automatically as needed? - Sean _______________________________________________ ofiwg mailing list [email protected] https://lists.openfabrics.org/mailman/listinfo/ofiwg
