> One solution is to change the IB device driver interface so that > kernel virtual addresses are passed to the IB device driver and > the device driver is responsible for calling dma_map_single(), etc. > I believe this will be unacceptable to the OpenFabrics community
Actually it's worse than unacceptable -- I don't see how this can work at all. The problem is that addresses are not just passed directly to the local HCA; they also might be embedded in protocol messages that are sent to a remote HCA and then used by the remote HCA to initiate RDMA. For example, the SRP driver uses ib_get_dma_mr() to get an R_Key, which it then sends to the target along with a DMA address. The target uses that R_Key/address to RDMA data directly to or from the host. There's no good way for the low-level driver to handle the DMA mapping, since the address is embedded in a protocol message that the low-level driver knows nothing about. > Another solution is to change the IB device driver interface to add > a function which tells the caller what type of addresses the device > expects. Kernel modules would then be required to pass either a > dma_map_xxx() address or a kernel virtual address based on the > driver's preference. > The current set of IB consumers either start with kmalloc/vmalloc > memory (such as the MAD layer) or a list of physical pages > (such as ISER and SRP). The current code could therefore be > fairly easily changed except for ISER/SRP when a struct page > doesn't have a direct kernel address (high pages) and would > need to call kmap()/kunmap() in that case. I have a few problems with this: first, it's unfortunate that every consumer needs two code paths to handle the two possibilities; second, I don't see how it handles the case of SRP's use of the ib_get_dma_mr() R_Key as above anyway; third, expecting consumers to kmap pages for a long time across work request execution is a bad idea. Maybe the least bad solution would be to add rdma_XXX wrappers around the dma mapping functions that RDMA consumers use; then most low-level drivers could just pass them through to the DMA mapping API, while the ipath driver could handle things itself. The problem with that is that it ends up wrapping a huge API -- for example, you need both dma_map_single and dma_map_sg at least, plus someone might want to use dma_alloc_coherent memory, not to mention the dma_pool stuff, etc. A cleaner solution would be to make the dma_ API really use the device it's passed anyway, and allow drivers to override the standard PCI stuff nicely. But that would be major surgery, I guess. (BTW, vmalloc memory should not be used for DMA, since it's not guaranteed to be DMA-able -- so anyone doing that is just wrong) - R. _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
